Paper deep dive
NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference
Aurélien Lac, Tony Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/3/2026, 2:55:44 AM
Summary
The paper introduces NeoMME, a family of single-tower, multimodal-native, multilingual bidirectional encoders (260M and 800M parameters) designed for efficient fine-tuning and inference. Unlike dual-tower or VLM-based encoders, NeoMME processes raw image patches and multilingual text in a single bidirectional Transformer using masked discrete-diffusion pretraining. It supports a 16,384-token context and achieves state-of-the-art performance on visual document retrieval benchmarks (ViDoRe v3) while offering significantly higher throughput and compression capabilities via hierarchical token pooling and asymmetric quantization.
Entities (12)
Relation Signals (10)
NeoMME-Retriever 260M → achievesscore → 0.523 nDCG@10
confidence 95% · NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10
NeoMME-Retriever 800M → achievesscore → 0.556 nDCG@10
confidence 95% · NeoMME-Retriever 800M reaches 0.556
NeoMME-Retriever → evaluatedon → ViDoRe v3
confidence 95% · On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models
NeoMME → hasvariant → NeoMME-260M
confidence 95% · We introduce NeoMME, a family of 260M and 800M-parameter... Encoders
NeoMME → hasvariant → NeoMME-800M
confidence 95% · We introduce NeoMME, a family of 260M and 800M-parameter... Encoders
NeoMME → releasedon → Hugging Face Transformers
confidence 95% · We contribute NeoMME to Hugging Face Transformers
NeoMME-260M → hasthroughputadvantage → ColModernVBERT
confidence 90% · NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT
NeoMME → pretrainedwith → Masked Discrete-Diffusion
confidence 90% · Both models are pretrained from scratch with a masked discrete-diffusion text objective
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.01657v1
- Canonical: https://arxiv.org/abs/2609.01657v1
Trouble viewing inline? Open PDF directly →
Full Text
150,122 characters extracted from source content.
Expand or collapse full text
https://hf.co/collections/Hcompany/neomme NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Aurélien Lac * Tony Wu * H Company * Equal contribution. Multimodal models often build on architectures designed for generative vision–language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late- interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048×2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2×the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255×while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 athttps: //hf.co/collections/Hcompany/neomme. DUAL-TOWER ENCODER (SigLIP2 B/16-512) VLM AS ENCODER (ColQwen2.5-v0.2) MODERNVBERT (250M) NEOMME (NeoMME-260M) ImageText Vision encoder Text encoder Similarity matrix ImageText Vision encoder Projection Tokenizer Embedding Input sequence LLM decoder Hidden states ImageText Vision encoder Projection Tokenizer Embedding Input sequence LM encoder Hidden states ImageText Patchify 2-layer MLP Tokenizer Embedding Input sequence Multimodal encoder Hidden states LEGEND Image vector Text vector Causal attention Bidirectional attention Figure 1|Unlike dual-tower and VLM encoders, NeoMME processes image patches and text tokens in one bidirectional Transformer, without a pretrained vision tower or causal decoder. aurelien.lac.pro@gmail.com, tonywu.ai@outlook.com arXiv:2609.01657v1 [cs.IR] 31 Aug 2026 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 1. Introduction Bidirectional encoders are strong models for learning text representations that transfer across tasks. BERT established masked bidirectional pretraining (Devlin et al., 2019), while ModernBERT brought long-context and efficiency improvements to encoder architectures (Warner et al., 2025). Large Language Models (LLMs) can also be converted into encoders similarly to LLM2Vec and LFM2.5- Encoder (BehnamGhader et al., 2024; Liquid AI, 2026). In matched experiments, Ettin (Weller et al., 2026) finds that native masked encoders remain stronger than causal decoders and decoder-to- encoder adaptations on classification and retrieval tasks, while native causal decoders remain stronger on generation. The contrast between encoders and decoders also extends to multimodal architectures. CLIP and SigLIP align independent image and text towers (Radford et al., 2021; Zhai et al., 2023), while generative Visual Language Models (VLMs) often project the output of a pretrained visual encoder to a causal language model (Alayrac et al., 2022; Li et al., 2023; Beyer et al., 2024). Visual document retrieval systems reuse both architectural patterns. DSE produces dense embeddings from PDF page screenshots (Ma et al., 2024), while ColPali keeps the image patch granularity for finer representations using late-interaction (Faysse et al., 2025). ModernVBERT is a 250M-parameter model that replaces the causal decoder with a bidirectional encoder while retaining a pretrained SigLIP2 tower (Teiletche et al., 2026; Tschannen et al., 2025). Its retrieval results show that a model of this size can perform visual document retrieval. An alternative is to process image and text tokens with a single shared Transformer. ViLT, OneR, and M3AE instantiate this design in bidirectional architectures (Kim et al., 2021; Jang et al., 2023; Geng et al., 2022), while recent tower-free VLMs feed projected image patches directly to generative backbones (Chen et al., 2024b; Diao et al., 2024). Sharing the Transformer gives both modalities the same computational path, rather than separate towers with potentially asymmetric architectures and forward passes. It also unifies the model lifecycle: the same backbone can be pretrained, fine-tuned, parallelized, and served across modalities. With these motivations in view, we introduce NeoMME, a family of bidirectional multimodal encoders trained entirely from scratch. Text embeddings and raw 32×32 pixel patches enter through modality-specific projections and then share every Transformer layer. Dynamic-resolution images retain their aspect ratio, while the 32-pixel patches and long-context architecture keep high-resolution inputs tractable. For text-only examples, pretraining uses a masked discrete-diffusion objective (Sahoo et al., 2024; Shi et al., 2024; Nie et al., 2025). For image–text pairs, the same text-denoising objective is conditioned on visible patches from the corresponding natural or document image. As a downstream evaluation of the backbone, we fine-tune NeoMME for visual document retrieval, yielding NeoMME-Retriever. A single backbone forward pass produces both a dense pooled represen- tation and a late-interaction multi-vector representation. We also release Late-Interaction Kernels (LIK), a suite of fused MaxSim kernels that reduces peak VRAM and runtime for late-interaction inference and training (Lac and Wu, 2026). On ViDoRe v3 (Loison et al., 2026), the 260M and 800M models respectively achieve 0.523 and 0.556 nDCG@10, as shown in Table 5. NeoMME-260M encodes 2048×2048 pages at 51.3 pages per second on an NVIDIA L40S, with 1.97×the throughput of ColModernVBERT at the same input size, as shown in Figure 13. Hierarchical token pooling at factor 8, combined with asymmetric quantization to int8 queries and binary documents, makes high-resolution visual document retrieval tractable for large corpora. In our experiments, the two together shrink the NeoMME-260M embedding from∼1.5 MB per ViDoRe v3 document to 6 kB, a 255×compression that retains more than 95% of the original retrieval quality, as shown in Figure 12. Contribution 1: NeoMME, an efficient Multilingual and Multimodal-native foundational Encoder. 2 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference We train a tokenizer and 260M and 800M bidirectional single-tower Transformer backbones from scratch that take raw image patches as input. The evaluated recipe combines multilingual text, code, mathematics, natural images, and document images with long-context dynamic-resolution encoding, and image-conditioned masked-diffusion pretraining. We also contribute NeoMME to the Hugging Face Transformers library (Wolf et al., 2020). The NeoMME model documentation describes the public architecture and API. Contribution 2: NeoMME-Retriever, an efficient visual document retrieval embedder. 1 The model can output both late-interaction and dense representations for deployment flexibility. It is compatible with Sentence Transformers for dense and multi-vector retrieval (Reimers and Gurevych, 2019). We evaluate both 260M and 800M models on visual-document and text retrieval and show that NeoMME-Retriever is Pareto-optimal on ViDoRe v1, v2, and v3 (Macé et al., 2025). We also study input resolution, representation storage and compression, indexing throughput, and query-encoding latency. 2. Related work 2.1. Text representation models BERT established masked bidirectional pretraining (Devlin et al., 2019), and Sentence-BERT, DPR, and ColBERT adapted encoder representations to dense and late-interaction retrieval (Reimers and Gurevych, 2019; Karpukhin et al., 2020; Khattab and Zaharia, 2020). ModernBERT, EuroBERT, and mmBERT update this family of models with longer contexts, newer Transformer components, and broader language coverage (Warner et al., 2025; Boizard et al., 2025; Marone et al., 2025). Ettin compares masked encoders and causal decoders with matched model shapes, data order, and training recipes (Weller et al., 2026). Its results are task-specific: the tested encoders are stronger on classification and retrieval, while the decoders are stronger on generation. Causal decoder models can also be converted into encoders. LLM2Vec enables bidirectional attention before masked and contrastive adaptation (BehnamGhader et al., 2024); LFM2.5-Encoder also converts its causal convolutions (Liquid AI, 2026); and BidirLM extends this strategy across model scales and modalities (Boizard et al., 2026). 2.2. Masked diffusion for representations Masked diffusion extends masked-language modeling from one corruption rate to a sampled noise trajectory. DiffusionBERT connected discrete diffusion to BERT-style denoising (He et al., 2023); MDLM and MD4 simplified the objective (Sahoo et al., 2024; Shi et al., 2024); and LLaDA demonstrated its large-scale generative use (Nie et al., 2025). Diffusion-pretrained hidden states also support retrieval: DiffEmbed learns text embeddings (Zhang et al., 2025), PPLX-Embed converts causal models into multilingual bidirectional encoders (Eslami et al., 2026), and DiffRetriever reads multiple retrieval representations from masked positions (Wang et al., 2026a). Multimodal work spans several related settings. Masked Diffusion Captioning and LaViDa recon- struct text conditioned on separate image encoders (Feng et al., 2025; Li et al., 2025b). UniDisc and MMaDA move diffusion into a shared Transformer, but operate on externally tokenized images and primarily target generation (Swerdlow et al., 2025; Yang et al., 2025). LaViDa and MMaDA have also been contrastively adapted into dense multimodal embedding models (Wang et al., 2026b). Among the systems reviewed, none combines continuous raw patches, a vision-tower-free shared bidirec- 1 spaces/tonywu71/neomme-retriever-demo 3 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference tional Transformer trained from random initialization, image-conditioned masked-text diffusion, and subsequent dense and late-interaction visual-document retrieval. 2.3. Vision–language architectures CLIP and SigLIP scale image and text retrieval through separate towers (Radford et al., 2021; Zhai et al., 2023). This allows offline image indexing, but with limited interaction between modalities. Generative VLMs introduce deeper fusion while usually retaining a separate visual model. Flamingo inserts gated cross-attention between vision and language streams (Alayrac et al., 2022); BLIP-2 uses a lightweight Querying Transformer to connect a frozen image encoder and language model (Li et al., 2023); and LLaVA, PaliGemma and Qwen2-VL project or merge vision-tower features into causal language models (Liu et al., 2023; Beyer et al., 2024; Wang et al., 2024). Other studies also developed architectures that share more of the model backbone among modali- ties. ViLT processes raw projected patches and text with the same bidirectional layers, but initializes them from a pretrained ViT (Kim et al., 2021; Dosovitskiy et al., 2021). UFO and Uni-Perceiver reuse one Transformer across unimodal and multimodal tasks (Wang et al., 2021; Zhu et al., 2022), while OneR and M3AE train shared representation backbones from scratch (Jang et al., 2023; Geng et al., 2022). VLMo, BEiT-3, and the masked-prediction EVE share attention while retaining modality-specific experts or visual targets (Bao et al., 2022; Wang et al., 2023; Chen et al., 2024a). That EVE uses fixed-rate masked reconstruction and pretrained initialization, and BEiT-3 relies on a separate visual tokenizer. Recent vision tower-free VLMs fed patches directly to decoder-oriented backbones. This line of work includes Fuyu, the encoder-free EVE (an unrelated model that shares the EVE name), SOLO, EVEv2, and NEO (Bavishi et al., 2023; Diao et al., 2024; Chen et al., 2024b; Diao et al., 2025b,a). Chameleon instead uses discrete image codes (Chameleon Team, 2024), while Gemma 4 Unified includes a from-scratch raw-patch decoder (Gemma Team, 2026). These works establish the individual components of shared multimodal processing. 2.4. Visual document representation and retrieval Document encoders traditionally combine extracted words, layout coordinates, and image pixels. LayoutLMv3, for example, requires OCR tokens and boxes and uses a pretrained visual tokenizer for masked-image targets (Huang et al., 2022). Visual document retrieval can instead avoid OCR by treating each page as an image and embedding it directly. DSE produces dense page embeddings (Ma et al., 2024), while ColPali and ColQwen2 preserve page-token representations for MaxSim retrieval (Faysse et al., 2025). In practice, visual document retrieval can be used for visual retrieval-augmented generation (Visual RAG): the top-푘retrieved pages are fed to a VLM along with a user query so it can answer from them. This keeps layout, tables, figures, and other evidence that text extraction may lose (Yu et al., 2024; Cho et al., 2024; Sun et al., 2025). Recent systems improve this recipe without removing inherited visual components. ModernVBERT combines a compact bidirectional encoder with a pretrained SigLIP2 tower (Teiletche et al., 2026; Tschannen et al., 2025); Jina Embeddings v4 derives dense and multi-vector representations from Qwen2.5-VL (Günther et al., 2025); and Nemotron ColEmbed V2 scales the model size and embedding space dimension (Moreira et al., 2026). 4 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 2.5. Late-interaction capacity and efficiency Single-vector embeddings have a fixed representational capacity. Their dimension limits which top-푘 document sets can be separated by a fixed score margin, and current embedding models exhibit related failures on the LIMIT benchmark (Weller et al., 2025). Multi-vector embeddings can have strictly greater capacity. Some relevance matrices require exponentially large single-vector embeddings but admit polynomial-size multi-vector embeddings, and models using multi-vector representations retain an advantage on the associated ANDOR benchmark after task-specific fine-tuning (Agarwal et al., 2026). These mathematical constructions and synthetic text benchmarks do not establish that late-interaction outperforms dense retrieval on every task or that the same mechanism explains visual document retrieval. Late-interaction can provide greater expressive capacity than dense retrieval, but it also increases storage and scoring costs: storage scales with the number of input tokens, and MaxSim scoring is more costly than cosine similarity. PLAID prunes candidates through centroids (Santhanam et al., 2022b), while hierarchical token pooling and MUVERA reduce or transform stored multi-vector representations (Clavié et al., 2024; Dhulipala et al., 2024). FLASH-MAXSIM (Pony et al., 2026) and MaxSim 2 are kernels that fuse MaxSim without materializing the full token-similarity tensor. 3. Architecture NeoMME is a multimodal encoder built around a single bidirectional Transformer optimized for long-context. Modality-specific input layers map text tokens and RGB image patches into a shared hidden space, where the encoder processes them jointly. Figure 1 compares how dual-tower encoders, decoder-based visual language models, ModernVBERT, and NeoMME route image and text through their architectures. 3.1. Tokenizer NeoMME’s tokenizer was trained from scratch for efficiency and multilingual coverage. Data. The training mixture used to train NeoMME’s tokenizer comprises multilingual web text, code, math, and machine-produced image transcripts. It draws English from FineWeb-Edu, 20 additional languages from FineWeb2-HQ, mathematical text from FineMath, and 10 programming languages from StarCoderData. Its 131,072-entry vocabulary uses byte fallback, splits every digit, and reserves 64 identifiers for fixed special tokens. Methodology. Motivated by the compression results reported for SuperBPE (Liu et al., 2025), we trained a whitespace-unconstrained byte-level byte-pair encoding (BPE) tokenizer. Allowing merges to cross word boundaries enables the vocabulary to capture subwords, common multiword expressions, and formatting patterns such as code indentation. Tokens are limited to 48 bytes. Performance Aggregated by total token count over 14 target languages from FLORES-200 devtest (NLLB Team, 2024), NeoMME emits 44.4% fewer tokens than ModernBERT (Warner et al., 2025), 39.4% fewer than LFM2.5-Encoder-230M (Liquid AI, 2026), 6.3% fewer than mmBERT-base (Marone et al., 2025), and 16.9% fewer than EuroBERT-210m (Boizard et al., 2025). However, the analysis over the 204 languages in FLORES-200 exposes a weaker coverage outside the original target set. Table 16 and Table 17 in subsection B.1 report the per-language results, evaluation protocol, and comparison limitations. 2 https://github.com/erikkaum/maxsim 5 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 3.2. Shared multimodal inputs 3.2.1. Text tokens To minimize NeoMME’s model size, text tokens use an ALBERT-style factorized embedding (Lan et al., 2020): a 256-dimensional lookup followed by a linear projection to the model width. Let퐸 ∈ ℝ 푉×푑 푒 be the token table and푃 ∈ ℝ 푑×푑 푒 the projection, where푉is the vocabulary size,푑 푒 the embedding rank, and 푑 the model width. For token 푥 푖 , the input path is 풉 (0) 푖 = 푃퐸 ⊤ 푥 푖 ,: .(1) For final hidden state 풉 푖 , the masked-token output path is ℓ 푖 = 퐸푃 ⊤ 풉 푖 .(2) Because the tied, factorized masked-token decoder reuses both factors, it adds no output-specific parameters. 3.2.2. Image patches Each input image is converted to RGB and partitioned into non-overlapping 32×32 patches. An image patch contains 3×32×32=3,072 values. These values are then projected to the model input space dimension using layer normalization followed by a 2-layer MLP. The MLP is trained jointly from scratch; no patch-merging module or pretrained vision encoder is used. Structural tokens delimit inputs, image grids, and patch rows, while the segment offsets prevent attention across packed inputs. At fixed resolution, 32-pixel patches produce approximately one quarter as many image tokens as 16-pixel patches. An earlier Gemma 4-inspired variant combined direct 48-pixel patches, a linear stem, and learned coordinate embeddings (Gemma Team, 2026). In the 260M-scale experiments, this configuration appeared to weaken text-reading performance, possibly because each token had to compress a larger image region. During pretraining and fine-tuning, the image pipeline randomly samples a longest-side cap between 1,024 and 2,048 pixels for each example. Varying the cap exposes the model to different image sizes and reduces overfitting to a fixed resolution. The pipeline preserves aspect ratio and downsamples only when the image exceeds the sampled cap. The resulting patch count varies with image dimensions and resolution, following dynamic-resolution vision–language models such as Qwen2-VL (Wang et al., 2024). Figure 2 illustrates this trade-off with dimensions that are illustrative rather than benchmark averages. SigLIP So400m/14(patch size: 14px) Fixed-size square resizing 768 × 1024 px Resize 384 × 384 px N = 729 (fixed) NeoMME(patch size: 32px) Dynamic resolution 384 × 512 px N = 192 768 × 1024 px N = 768 1536 × 2048 px N = 3072 Figure 2 | Dynamic-resolution image processing with a variable side-length cap. 6 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 3.2.3. 2D rotary position embeddings NeoMME extends rotary position embeddings (RoPE) (Su et al., 2024) to two coordinate axes, following multimodal RoPE designs such as Qwen2-VL. Consecutive rotary frequency pairs alternate between the two axes. This construction preserves the usual one-dimensional ordering of text by assigning token푖the coordinate(푖, 푖), whereas image inputs use the axes to represent rows and columns. For an image whose coordinate base is푏, the input and image markers receive(푏, 푏)and(푏+1, 푏+1), respectively. A patch at row푟and column푐then receives(푏+2+ 푟, 푏+2+ 푐), while the row marker appended to each patch row occupies one additional grid column. Text following the image grid returns to diagonal coordinates, starting beyond both grid axes. Global-attention layers use partial RoPE (Khan et al., 2026), rotating 25% of each query and key head as in Qwen3-Next (Qwen Team, 2025). Following the sliding-window–global RoPE split used in Gemma 4, global layers use base 10 6 , whereas sliding-window layers apply full RoPE with base 10 4 . 3.3. Modern bidirectional backbone Long-context attention. The encoder uses bidirectional attention. Both 260M and 800M NeoMME models support a maximum context length of 16,384 tokens, chosen to accommodate up to two standard 3,840×2,160 4K UHD images after 32-pixel patching. To reduce attention computation at this limit, most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention. Among the sliding-window layers, the half-window alternates between 256 and 1,024 tokens. This design combines Longformer’s symmetric sliding-window attention (Beltagy et al., 2020) with the interleaved sliding-window–global layouts of ModernBERT and Gemma 2 (Gemma Team, 2024); the short–long schedule follows modded-nanogpt (Jordan and modded- nanogpt contributors, 2026), although the specific 256- and 1,024-token half-windows are NeoMME design choices. Figure 3 shows the layer sequence for both model sizes. NeoMME-260M · 17 layers NeoMME-800M · 20 layers Sliding window ±256Sliding window ±1024Full attention Figure 3 | Alternating sliding-window and global-attention layers in the NeoMME encoder stack. Attention heads and query–key normalization. Both sliding-window and global layers use grouped- query attention (GQA) (Ainslie et al., 2023). The 260M model has 16 query heads and 4 key-value heads, whereas the 800M model has 28 query heads and 7 key-value heads. Queries and keys are independently root-mean-square normalized before RoPE and, consequently, before the attention dot product, implementing query–key (QK) normalization (Dehghani et al., 2023). 7 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 1 | NeoMME architecture settings. ConfigurationNeoMME-260M 3 NeoMME-800M 4 Exact parameters262,937,906793,715,032 Embedding rank256256 Hidden width1,0241,792 Encoder layers1720 Global layers34 Query heads1628 Key-value heads47 Head dimension6464 MLP width3,5846,400 Context length16,38416,384 Vocabulary131,072131,072 Patch size3232 Patch stemMLPMLP Figure 5 | Parameter allocation by module group for NeoMME-260M and NeoMME-800M. Counts include the backbone and input paths but exclude the contrastive-retrieval heads. Figure 4|Pre-normalized attention and MLP paths in one NeoMME en- coder layer. Block structure. Each block uses parameter-free root-mean- square pre-normalization (Zhang and Sennrich, 2019) and a squared-ReLU MLP (So et al., 2021). Its attention output is further modulated by a query-dependent elementwise sigmoid gate (Qiu et al., 2025). Meanwhile, each attention and MLP residual branch is multiplied by(2퐿) −1/2 , counting the two branches in each of the퐿layers separately. This is the branch-scaling component of Depth-휇P (Yang et al., 2024; Bordelon et al., 2023), although NeoMME does not otherwise claim to implement the complete Depth-휇P pa- rameterization. The same parameter-free root-mean-square normalization closes the encoder stack. Figure 4 summa- rizes the normalization, attention, MLP, and residual paths. Token-indexed value embeddings. At the first and last global-attention layers, a shared embedding table indexed by the input token identifiers is added to the value vec- tors. This design is inspired by ResFormer’s value-residual learning (Zhou et al., 2025) and follows the token-indexed value-embedding implementation introduced in modded- nanogpt; unlike ResFormer, however, it learns a dedicated table rather than reusing values projected in an earlier layer. Residual mixing. Following the embedding shortcut used in modded-nanogpt, each block forms a learned scalar mixture of the current residual stream and the initial normalized multimodal input before its attention mixer. NeoMME additionally uses a learnable variant of exclusive self-attention (Zhai, 2026), scaling each head’s value-aligned subtraction by a separate coefficient. Model sizes. To keep NeoMME as efficient as possible, we chose model sizes of 260M and 800M parameters. ModernVBERT showed that visual document retrieval was possible with 250M parameters, close to the size of our smaller model (Teiletche et al., 2026). NeoMME is released as NeoMME-260M 3 and NeoMME-800M 4 . The 800M configuration scales mainly through width, keeping the backbone to 20 layers and 4 global-attention layers at the 16,384-token context length. Table 1 lists the architecture settings, and Figure 5 shows the parameter allocation by module group. 8 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 3.4. Weight initialization NeoMME initializes the attention output and MLP down projections to zero. This applies the broader zero-last-layer principle introduced by Fixup (Zhang et al., 2019), making both residual branches exact no-ops and each block an identity at initialization, without otherwise adopting the complete Fixup parameterization. The factorized word table is drawn from a zero-mean normal distribution with standard deviation 푑 −1/2 푒 , where푑 푒 =256. The same dimension-dependent embedding scale appears in T-Fixup (Huang et al., 2020); in NeoMME, it keeps the logits produced by the tied factorized decoder in Equation 2 at order-one scale. The token-indexed value table and per-head exclusive-self-attention coefficients are initialized to zero, while the input-shortcut mixture begins with coefficients(1,0)on the current and initial residual streams, respectively. All remaining backbone linear layers use the default PyTorch 2.8 torch.n.Linear initialization. 4. Pretraining The pretraining phase trains the shared backbone from random initialization on text-only and multimodal examples, including document images and natural images paired with text. Its data pipeline and masking objective reduce text-only shortcuts and encourage the model to use image evidence. 4.1. Dataset mixture Pretraining samples come from separate text-only and multimodal streams. The text stream uses 14 datasets across 58 configurations, spanning web and PDF text, synthetic educational material, encyclopedic text, mathematics, question answering, and code. The multimodal stream combines mixed visual–text collections, document content, OCR data, and natural images. Sources are sampled according to the weights defined in Table 2, with 55% of packed input tokens coming from the text-only stream and 45% from the multimodal stream. The planned total is about 524B packed input tokens. subsection A.1 reports every configuration, its sampling weight, and the source descriptions. 4.2. Data pipeline and systems optimization At this model scale, input processing can take longer than the forward and backward passes, making it the main bottleneck. We therefore pack documents into 16,384-position streams while keeping them isolated through variable-length attention boundaries. Images are decoded and patchified in background workers, and packed batches are prefetched during the preceding training step. A compute-aware schedule balances multimodal work across data-parallel ranks, while shape-stable batch construction supports compiled training with dynamic image resolution. With FlashAttention-3 (Shah et al., 2024), Liger Kernel’s fused linear cross-entropy (Hsu et al., 2025), andtorch.compile, these optimizations keep GPU utilization high during steady-state training. 4.3. Pretraining objective NeoMME was pretrained as a discrete masked-diffusion denoiser over text, optionally conditioned on visible image patches. Its objective follows the general absorbing-mask training paradigm exemplified by MDLM, MD4, and LLaDA (Sahoo et al., 2024; Shi et al., 2024; Nie et al., 2025). 3 Hcompany/NeoMME-260M. 4 Hcompany/NeoMME-800M. 9 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 2 | Configured pretraining mixture by logical source family. GroupDatasetWithin-stream weight Expected tokens k Text only: 55% of packed input tokens Web and PDF FineWeb-Edu0.16766048.346B Nemotron-C-v2.10.07289621.020B FinePDFs0.13121437.837B FineWeb2-HQ0.349900100.897B Synthetic, reference, and math Cosmopedia-v20.10205429.428B Wikipedia0.08747925.225B FineMath0.05102714.714B Question answering MMLU0.0061961.787B MedMCQA0.0040091.156B CommonsenseQA0.0002190.063B QASC0.0001820.052B OpenBookQA0.0001090.031B AI2 ARC0.0000870.025B CodeStarCoderData0.0269727.778B Text-only total1.000000288.358B ë Multimodal: 45% of packed input tokens Mixed collectionFineVision0.500000117.965B Document and OCR PDFA +LightOnOCR0.30000070.779B DocAtlas0.07000016.515B Synthetic multilingual OCR0.0300007.078B Natural imagesPixelProse0.10000023.593B Multimodal total1.000000235.930B Let푠index a real document segment in a packed sequence. Text-only segments draw a corruption rate휌 푠 ∼ U(0,1), whereas multimodal segments draw휌 푠 ∼ U(0.30,1). The higher minimum corruption rate reduces reliance on visible textual context and encourages predictions to use the image patches. Given휌 푠 , each eligible text position is independently replaced by the mask token with probability휌 푠 . Eligible positions exclude padding, image patches, and structural markers. Unlike BERT, which uses a fixed 15% selection rate and an 80–10–10 mixture of mask tokens, random tokens, and unchanged tokens (Devlin et al., 2019), NeoMME always replaces selected positions with the mask token and computes the loss only at those positions. Image patches remain visible and condition the prediction of masked text. Neither masked image modeling nor a pixel-prediction head is used, so no pixel-reconstruction, latent-image, or generative- image target contributes to the loss. The same text denoiser can be used at inference time by starting from masked text and iteratively predicting and revealing tokens, optionally conditioned on the visible image patches. For data-parallel rank푗, letM 푗 contain the masked eligible positions in the packed batch assigned to that rank. The loss for rank 푗 is L 푗 = Í 푖∈M 푗 푤 푖 CE 푓 휃 ( ̃풙 푗 , 풑 푗 ) 푖 , 푥 푖 Í 푖∈M 푗 푤 푖 , 푤 푖 = 1 max(푟 푠(푖) , 푟 0 ) , 푟 0 = 0.05,(3) where푠(푖)is the segment containing position푖, ̃풙 푗 is the corrupted token sequence,풑 푗 denotes the image patches when present, and푥 푖 is the original token. We setL 푗 =0 whenM 푗 is empty. With푅 10 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 3 | Pretraining hardware, batch, and optimization settings. SettingNeoMME-260M NeoMME-800M Hardware and batch Nodes24 H100 accelerators1632 Packed sequences per GPU42 Multimodal sequence share45%45% Compute precisionbfloat16bfloat16 Packed input tokens per global step1,048,5761,048,576 Patch cap per rank57,34432,768 Optimization NorMuon peak learning rate0.0120.010 AdamW peak learning rate0.00130.00075 Warmup steps300300 Final decay fraction0.100.10 Final learning-rate floor1% of peak1% of peak Gradient clipping norm1.01.0 Cautious-decay coefficient0.010.01 Embedding weight decay0.010.01 One-dimensional weight decay00 data-parallel ranks, gradient averaging optimizes L pretrain = 1 푅 푅 ∑︁ 푗=1 L 푗 .(4) The inverse-rate weighting follows the standard linear absorbing-mask formulation. In practice, NeoMME caps the reciprocal weight at 20 and normalizes by the realized weight mass on each rank. This limits the influence of rare low-corruption samples and maintains a stable loss scale during packed distributed training. 4.4. Training settings Pretraining configuration. Table 3 summarizes the pretraining hardware, batch, and optimization settings for both 260M and 800M models. The two pretraining runs use AWSp5.48xlarge 5 instances with Elastic Fabric Adapter (EFA) 6 networking. We used SkyPilot (Yang et al., 2023) 7 to launch and manage these AWS training jobs. Each run processes 1,048,576 packed input tokens per global step for 500,000 steps, for a planned total of 524.288 billion packed input tokens. Optimizers. NeoMME routes parameters according to module type and tensor rank. Embedding tables use a custom MasterAdamW implementation, while all other matrix-valued parameters use NorMuon (Li et al., 2026b), which augments Muon’s orthogonalized matrix updates (Jordan et al., 2024) with per-neuron second-moment normalization. The latter group includes the factorized embedding projection, attention projections, and MLP matrices. One-dimensional parameters use MasterAdamW without weight decay. Both optimizers retain full-precision master weights and moments while the model parameters remain in bfloat16. NorMuon applies cautious weight decay, 5 https://aws.amazon.com/ec2/instance-types/p5/ 6 https://aws.amazon.com/hpc/efa/ 7 https://skypilot.ai/ 11 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference restricting decay to coordinates where the parameter and optimizer-update signs agree (Chen et al., 2025). MasterAdamW applies decoupled weight decay to embedding tables (Loshchilov and Hutter, 2019). To reduce optimizer overhead, same-shaped NorMuon matrices are processed in batches and their updates are sharded across GPUs within each node. Since gradients are synchronized globally, nodes can repeat this computation independently without an additional inter-node optimizer collective. Learning-rate scheduler. We use a warmup–stable–decay (WSD) learning-rate schedule (Hu et al., 2024). Both learning rates increase linearly over 300 updates, remain at their peaks, and decay linearly over the final 10% of training to 1% of their peak values. We release the predecay checkpoints of NeoMME-260M 8 and NeoMME-800M 9 at the stable-to-decay boundary, following Pythia and ModernBERT (Biderman et al., 2023; Warner et al., 2025). Researchers can restart training from these checkpoints and anneal on domain-appropriate data for their intended use. The 260M and 800M contrastive runs, however, initialize from the step-500,000 checkpoints rather than from the predecay ones. 4.5. Text masking and image sensitivity Low corruption rates can leave many masked targets predictable from text alone, a known limitation of multimodal masked-language modeling (Bitton et al., 2021). To weaken this shortcut, multimodal segments draw their corruption rates fromU(0.30,1). Prior work similarly finds that vision–language pretraining benefits from substantially higher text masking rates (Verma et al., 2022). With higher corruption rates, the model receives less textual context, so image evidence can contribute more. We measure this behavior using a cross-modal input-ablation probe (Frank et al., 2021). At each probe event푡, we mask 30%, 60%, or 90% of eligible text positions and denote the masked positions byM 푡 . We run the model twice on the same masked text, first with the page patches퐼 푡 and then with a zero tensor0 of the same shape. For either patch input푍, let푝 휃 (푣 | e 풙 푡 , 푍) 푖 be the predicted probability of vocabulary token푣at masked position푖, where e 풙 푡 is the masked text and푥 푡,푖 is the original token. We define masked-token accuracy 퐴 푡 (푍), event-level image gain 퐺 푡 , and mean image gain퐺 as 퐴 푡 (푍)= 1 |M 푡 | ∑︁ 푖∈M 푡 ퟙ arg max 푣∈V 푝 휃 (푣 | e 풙 푡 , 푍) 푖 = 푥 푡,푖 , 퐺 푡 = 퐴 푡 (퐼 푡 )− 퐴 푡 (0), 퐺= 1 푇 푇 ∑︁ 푡=1 퐺 푡 . (5) The logged value퐺 푡 compares exact token recovery for one probe event, and퐺averages those event- level differences over푇probe events. A positive퐺 푡 means that visible page patches improve recovery of the masked text. Over the final 50 probe events, both models have positive mean image gain at every tested corruption rate. Image gain increases as textual context is removed, reaching 38.4 percentage points for NeoMME-260M and 40.5 percentage points for NeoMME-800M at 90% masking. Positive image gain shows that the models use page patches to improve masked-token recovery. We examine image-conditioned generation separately in subsection 4.6. Figure 6 shows the training loss and 90% image gain for both models. The NeoMME-260M loss enters a lower regime around step 36,000, while the NeoMME-800M loss has a visible change around step 25,000. Image gain rises from near zero during early training for both models. The shared 8 Hcompany/NeoMME-260M-Pretrain-predecay-s450000. 9 Hcompany/NeoMME-800M-Pretrain-predecay-s450000. 12 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 2.75 3.00 3.25 3.50 3.75 4.00 Training loss NeoMME-260M 2.75 3.00 3.25 3.50 3.75 4.00 Training loss NeoMME-800M 0100000200000300000400000500000 Training step 0.0 0.2 0.4 0.6 0.8 Image gain 0100000200000300000400000500000 Training step 0.0 0.2 0.4 0.6 0.8 Image gain Figure 6|Pretraining loss and image gain at 90% masking for both model sizes. Lines show raw and smoothed measurements, with dashed lines marking observed loss transitions. training-step axis shows when the observed loss transitions occur relative to when image gain appears. The final loss decline coincides with the scheduled WSD learning-rate decay to 1% of its peak. Such a decline is expected during WSD annealing. 4.6. Image-conditioned generation Predicted text: "This image displays: A wooden patio with lounge chairs and rocking chairs. There are trees and mountains in the background." Figure 7|Natural-image input and gener- ated text from the step-500,000 pretrained NeoMME-260M model. We tested the step-500,000 pretrained NeoMME-260M 3 model on unprompted image-conditioned text genera- tion without captioning or optical character recognition fine-tuning. Starting from visible image patches and a fully masked text canvas, the model iteratively predicts and reveals tokens through the tied vocabulary head used during pretraining. We tested uniform and blockwise LLaDA-style de- coding. We paired cosine remaining-mask schedules with MaskGIT confidence ranking (Chang et al., 2022), using either greedy or nucleus sampling (Holtzman et al., 2020). The examples shown here and in the appendix use fully masked, unprompted runs. The can- vas matches the reference length up to 48 tokens for captions and 64 tokens for document text, so the model receives the output length but no reference tokens. Fig- ure 7 shows one natural-image caption, with additional natural-image and document examples in Table 18 in subsection B.2. Together with the reading probe, the se- lected outputs show that the model uses image content, but they do not measure average generation quality. 13 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 5. Retrieval To evaluate the representations learned from the pretrained NeoMME encoder backbone, we fine-tune NeoMME for document retrieval with joint late-interaction and dense objectives. We then evaluate the resulting NeoMME-Retriever 260M and 800M models on both visual document and text retrieval benchmarks, and compare document-indexing throughput and query-encoding latency against other retrievers. 5.1. Problem formulation Retrieval. A retrieval system scores how relevant a document푑from a corpusC=푑 1 , . . . , 푑 푁 C is to a query푞from a query spaceQ. Computing the similarity score푠(푞, 푑) ∈ ℝfor every푑 ∈ Cproduces a ranking from which the system returns the most relevant documents. A document is an atomic item that the system can index and return. In text information retrieval, text is usually split into chunks (Karpukhin et al., 2020). On the other hand, visual document retrieval use screenshots of a PDF page as the retrieval unit (Ma et al., 2024; Faysse et al., 2025). Indexing and querying. Retrieval systems generally have an offline indexing phase and an online query phase. The system encodes and indexes corpus documents offline. At query time, it encodes푞, scores the query against the indexed documents, and returns a ranking. The offline representation must therefore preserve evidence that a query can use without another document-encoder forward pass. Retrieval architectures. We compare three scoring architectures that differ in when query and document representations interact and whether document representations can be indexed offline, as shown in Figure 8. Dense bi-encoder. A dense bi-encoder maps푞and푑independently to normalized vectors풆 푞 ,풆 푑 ∈ ℝ 푚 and scores their dot product (Reimers and Gurevych, 2019): 푠 dense (푞, 푑)=⟨풆 푞 ,풆 푑 ⟩.(6) The document vector can be computed offline and reused. Its one-vector form also allows approximate nearest-neighbor (ANN) indexes to search large corpora (1B+ documents) at high speeds (down to 13.3휇푠/query with Faiss) (Indyk and Motwani, 1998; Jégou et al., 2011; Johnson et al., 2019). Cross-encoder. A cross-encoder processes every query–document pair jointly, allowing full token interaction and often strong relevance scoring, but requiring a separate forward pass for each pair (Nogueira and Cho, 2019). Cross-encoders therefore cannot produce reusable document representa- tions for offline indexing and are generally used to rerank a small document set returned by a faster first-stage retriever. The same multistage design includes encoder–decoder rerankers such as monoT5 (Nogueira et al., 2020). Neural reranking now also covers multimodal retrieval with models that rank both text documents and page images (Ananya and Chatelain, 2026). Late-interaction. Late-interaction encodes푞and푑independently as normalized token-vector matrices 푸=[풒 1 , . . . ,풒 퐿 푞 ] ⊤ ∈ ℝ 퐿 푞 ×푟 and푫=[풅 1 , . . . ,풅 퐿 푑 ] ⊤ ∈ ℝ 퐿 푑 ×푟 , where푟=128 is the representation width. Retaining one vector per token preserves token-level matches that dense pooling removes. Standard ANN indexes directly support the one-vector dense score, while the set-to-set late-interaction score requires direct scoring or a specialized multi-vector retrieval method. Each query vector풒 푠 ∈ ℝ 푟 for 푠 ∈ 1, . . . , 퐿 푞 and each document vector풅 푡 ∈ ℝ 푟 for푡 ∈ 1, . . . , 퐿 푑 has unit length. ColBERT and ColPali use MaxSim (Khattab and Zaharia, 2020). mLateOn uses MeanMaxSim, which normalizes MaxSim by the query length, because its authors observed slightly better performance in their 14 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Bi-encoder Dense head q₁q₂q₃d₁d₂d₃d₄ EncoderEncoder Score Cross-encoder q₁q₂q₃d₁d₂d₃d₄ Joint encoder Score Late-interaction MaxSim head q₁q₂q₃d₁d₂d₃d₄ EncoderEncoder Score Figure 8 | Dense, cross-encoder, and late-interaction retrieval scoring patterns. experiments (Sourty et al., 2026): 푠 late (푞, 푑)= 1 퐿 푞 퐿 푞 ∑︁ 푠=1 max 1≤푡≤퐿 푑 ⟨풒 푠 ,풅 푡 ⟩.(7) 5.2. Retrieval architecture Two retrieval heads. The final hidden states from the shared backbone feed a jointly trained late- interaction head and a dense head as shown in Fig- ure 9. An ablation shows that joint training improves late-interaction retrieval scores on both evaluated benchmark suites. The ablation details are shown in subsection C.4. Late-interaction head. A learned linear layer projects every final hidden state to 128 dimensions and normalizes the resulting vector. Similarly to Col- BERT and ColPali, text inputs generate one vector per token, and images have one token per patch. Dense head. The dense head uses mean pooling to average the final hidden state vectors and then applies L2 normalization. The dense head is trained using Matryoshka (Kusupati et al., 2022): NeoMME- Retriever 260M uses widths of 128, 256, 512, and 1,024 dimensions and the 800M model uses the same four widths and a fifth width of 1,792 dimen- sions. INPUTS Text tokensImage patches OR NeoMME backbone Shared multimodal encoder BACKBONE HIDDEN STATES Late-interaction head 128d per vector L2 normalization Late-interaction embedding Dense head Mean pooling L2 normalization Dense embedding Figure 9|Late-interaction and dense retrieval heads for both NeoMME model sizes. Deployment flexibility. A single backbone pass produces both representations for a query or docu- ment, and the parameter-free dense head adds only mean pooling and normalization to the compu- tation already required by late-interaction. A system can use late-interaction directly on a smaller corpus. On a very large corpus, it can use the dense representation with an ANN index to retrieve a shorter list and then rerank that list with late-interaction, following a common multistage retrieval design (Hofstätter et al., 2022; Formal et al., 2024). 15 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 4 | Configured retrieval training mixture by logical source family. GroupDatasetWithin-stream weight Expected batch share k Text only LightOn retrieval Filtered English retrieval0.093.00% mLateOn multilingual0.7324.33% Code retrieval0.134.33% Other retrieval Additional multilingual retrieval0.031.00% Organic retrieval (en) andOrganic retrieval (fr) 0.020.67% Text-only total1.001/3 ë Multimodal Document images ColPali0.2516.67% Multilingual document images0.2516.67% VisRAG0.2516.67% VisRAG synthetic0.2516.67% Multimodal total1.002/3 5.3. Dataset mixture Retrieval training combines separate text-only and multimodal streams of query–document pairs. Training on both streams lets NeoMME-Retriever retrieve over mixed-modality corpora, where doc- uments are either text chunks or page images, though we do not evaluate that setting here. The text stream uses five source pools spanning English retrieval, multilingual retrieval, and code search. The multimodal stream uses four page-image datasets, including VisRAG (Yu et al., 2024). Sources are sampled according to configured weights that define their intended shares within each stream. The weights in Table 4 are normalized within each stream, and the expected batch shares use one text batch for every two multimodal batches. We decontaminated the training mixture against every retrieval benchmark we evaluate by masking matched queries and documents. subsection A.2 reports source counts and the query-generation procedure. Query augmentation. Before hard-negative mining, we augmented the multimodal stream with queries generated from page images. Qwen3.5-9B 10 generated two queries for each of 100,000 sampled pages, and 191,915 queries survived parsability and self-containment filtering, raising the multimodal stream from 760,826 to 952,741 queries without adding pages. Qwen3-VL-Reranker-8B 11 then filters the generated pairs during hard-negative mining. The generated queries add six query types, three output formats, and six languages. subsection A.2 gives the sampling and filtering procedure. This dataset augmentation is separate from the 10 learned query-expansion tokens described in subsection A.3. Hard negatives. Text examples use the hard negatives provided with the mLateOn multilingual dataset referenced in Table 4. Visual hard negatives are self-mined in two stages: a first fine-tuning run without explicit hard negatives mines a 32-candidate window for each visual query, and a second run repeats the protocol with seven negatives sampled from that window. subsection A.5 details the mining and filtering steps. 10 Qwen/Qwen3.5-9B. 11 Qwen/Qwen3-VL-Reranker-8B. 16 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 5.4. Contrastive objective For each training query푞, letC B (푞) ⊆ Cbe the in-batch corpus scored for푞, including its positive document 푑 + . The late-interaction loss is L late =−log exp(푠 late (푞, 푑 + )/휏) Í 푑∈C B (푞) exp(푠 late (푞, 푑)/휏) , 휏= 0.02.(8) LetKcontain the trained Matryoshka widths, and let푠 (푘) dense score vectors truncated to width푘 and renormalized. The dense head uses the same in-batch corpus and temperature at every width: L dense =− 1 |K| ∑︁ 푘∈K log exp(푠 (푘) dense (푞, 푑 + )/휏) Í 푑∈C B (푞) exp(푠 (푘) dense (푞, 푑)/휏) .(9) The retrieval objective is L retrieval =L late +L dense .(10) Both terms have weight one. The recipe uses no teacher logits and no knowledge distillation. 5.5. Optimization Large-batch contrastive training. The retrieval objective scores each query against the full in-batch corpus, which makes a direct backward pass memory intensive. Ordinary gradient accumulation over independent micro-batches would change the InfoNCE objective because each query would see fewer in-batch negatives (van den Oord et al., 2018; Gao et al., 2021). GradCache preserves the full in-batch corpus while recomputing encoder activations in smaller chunks. subsection A.2 gives the micro-batch sizes and the rest of the training settings. Late-Interaction Kernels (LIK). A direct implementation of MaxSim materializes the full query-token by document-token similarity tensor in high-bandwidth memory. That tensor dominates scoring memory in this recipe, because a 2048×2048 page produces a 4,162-long multi-vector embedding and every query is scored against the entire in-batch corpus. LIK instead computes the score in tiles and keeps only a running maximum per query token, storing the winning document-token index so that the backward pass can route gradients to it (Lac and Wu, 2026). The computation is exact, so scores and the loss match the direct implementation. At 4,096 document tokens, LIK lowers the MaxSim memory peak from 672 MB to 193 MB and the backward pass from 1.82 ms to 0.54 ms, while the direct implementation runs out of memory at 8,192 tokens. At the NeoMME-260M training shape, the throughput difference is within run-to-run variation, so the benefit here is memory headroom rather than speed. subsection D.3 gives the kernel details and the full measurements. 5.6. Evaluation protocol We evaluate each retrieval setting on its standard benchmarks: ViDoRe v3, v2, and v1 for visual document retrieval (Macé et al., 2025; Loison et al., 2026), and BEIR-15 for text retrieval (Thakur et al., 2021). We use only the eight public tasks from ViDoRe v3. Each benchmark task contains queries, a document corpus, and relevance judgments. The evaluator first averages each metric over the judged queries within a task, then averages the task scores so each task has equal weight. Each model uses its official processor and the scoring settings recommended by its authors. NeoMME evaluation settings. ViDoRe uses a downscale-only longest-side cap of 2,048 pixels and exact all-pairs MeanMaxSim scoring. BEIR-15 uses 8,192-token query and document limits, with each 17 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference document represented by its title followed by its body. The late-interaction head uses a compressed FastPLAID 12 index for approximate retrieval followed by exact rescoring of retrieved documents (Santhanam et al., 2022b), while the dense head uses the same pipeline with one normalized vector per input. Both NeoMME models use the output at step 20,000. subsection C.1 gives the omitted BEIR datasets and further evaluation details. Reported comparison results. External benchmark results come from the MTEB Leaderboard or the cited model reports. The ViDoRe v3 comparisons use per-task MTEB scores aggregated over the same eight tasks as our results, and we report each benchmark’s default metric (Muennighoff et al., 2023). 5.7. Main results On visual document retrieval, NeoMME is competitive for its size. The 260M model leads all models below 300M parameters, and the 800M model scores within 0.9 nDCG@10 points of the similarly sized Vultron Flash on ViDoRe v3. Table 5 compares NeoMME with selected visual retrievers across three parameter ranges, while Table 6 shows how its dense and late-interaction heads transfer to text retrieval. Table 19 reports every model in Figure 10, including ViDoRe v1 and v2 scores where reported and the ViDoRe v3 aggregate. We use ViDoRe v3 as the primary visual benchmark because it was built with a more rigorous methodology and is less saturated than ViDoRe v1 and v2. 2505001,0002,0004,0008,000 Model size (M parameters) 0.2 0.3 0.4 0.5 0.6 8-task public mean nDCG@10 +0.132 +0.117 NeoMME late-interaction dense Baselines late-interaction dense Figure 10|ViDoRe v3 nDCG@10 versus model size. NeoMME-Retriever 260M and 800M sit on the Pareto frontier. Arrows mark the gain of each NeoMME late-interaction head over its dense head. Table 19 gives the full results. Small visual retrievers. NeoMME-260M is the strongest model below 300M parameters and strictly below 800M across all ViDoRe benchmarks as shown in Table 5. Compared with the best other model below 300M parameters on each benchmark, it improves nDCG by 26.1 points on ViDoRe v3, 11.5 points on v2, and 5.4 points on v1. Its ViDoRe v3 score of 0.523 is within 0.2 points of the 12 https://github.com/lightonai/fast-plaid 18 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 3.75B-parameter ColQwen2.5 score while being 14.4× smaller by parameter count. Scaling visual retrievers to 800M. Increasing the model size improves every NeoMME aggregate and both retrieval heads. The ViDoRe gains are 3.3 points on v3, 3.7 points on v2, and 1.5 points on v1. On ViDoRe v3, NeoMME-800M reaches 0.556, within 0.9 point of the slightly larger Vultron Flash (∼850M) and 3.2 points above the 3.75B-parameter ColQwen2.5. We report document-encoding throughput separately in subsection 5.10. On BEIR-15, scaling adds 2.5 points to late-interaction and 6.3 points to dense retrieval. subsection C.2 gives the full metric and domain breakdowns. Text retrieval. Late-interaction exceeds dense retrieval by 18.3 points for NeoMME-260M and 14.4 points for NeoMME-800M. The 800M model reaches 0.513 with late-interaction and 0.369 with dense retrieval on BEIR-15. These results show that the same encoder backbone can retrieve both page images and text, with late-interaction the stronger of the two representations. Table 5 | Visual document retrieval performance on the ViDoRe benchmarks. Model detailsViDoRe (nDCG@푘) ModelParams. v3 (@10) v2 (@5) v1 (@5) <300M ColModernVBERT 13 250M0.261 † 0.407 ‡ 0.806 ‡ ColSmol-256M 14† 256M0.2070.3480.797 NeoMME-260M 15‡ 260M0.523 0.522 0.860 300M to 1B ColSmol-500M 16 500M0.340 ‡ 0.455 † 0.825 † Vultron Flash 17† 850M0.565 0.604 0.882 NeoMME-800M 18‡ 800M0.5560.5590.874 >1B ColQwen2.5-v0.2 19† 3.75B0.524 0.601 0.895 ColPali v1.3 20† 2.92B0.4300.5470.848 Table 6 | BEIR-15 text retrieval results. ModelParams. nDCG@10 Dense GTE-ModernBERT 21† 149M0.5519 DenseOn 22§ 149M0.5620 NeoMME-260M 15‡ 260M0.3055 NeoMME-800M 18‡ 800M0.3686 Late-interaction ColBERTv2 23¶ 110M0.4863 LateOn 24§ 149M0.5722 NeoMME-260M 15‡ 260M0.4881 NeoMME-800M 18‡ 800M0.5126 † Scores from MTEB. § Scores from Sourty et al. (2026). ¶ Scores from LightOn’s blog post. ‡ Results from our evaluation. 5.8. Resolution impact Image resolution trades retrieval quality against indexing cost. Higher longest-side caps produce more image tokens, which require more encoder FLOPs and more late-interaction storage. Across ViDoRe v1, v2, and v3, reducing the cap from 2,048 to 1,536 pixels lowers MeanMaxSim nDCG@10 by at most 1.8% for either model, while a square page uses about 57% as many vectors and raw float32 bytes. At 1,024 pixels, the relative quality loss ranges from 3.0 to 12.7%, and at 768 pixels it ranges from 11.7 to 36.1%. Table 21 reports the full quality grid and square-page representation sizes. 13 ModernVBERT/colmodernvbert. 14 vidore/colSmol-256M. 15 Hcompany/NeoMME-260M-Retriever. 16 vidore/colSmol-500M. 17 vultr/VultronRetrieverFlash-Qwen3.5-0.8B (Georgiou, 2026b). 18 Hcompany/NeoMME-800M-Retriever. 19 vidore/colqwen2.5-v0.2. 20 vidore/colpali-v1.3. 21 Alibaba-NLP/gte-modernbert-base. 22 lightonai/DenseOn. 23 colbert-ir/colbertv2.0 (Santhanam et al., 2022a). 24 lightonai/LateOn. 19 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 15101520 Pool factor 96 97 98 99 100 101 Retained relative nDCG@10 (%) Comp. sci. Energy Fin. EN Fin. FR HR Industrial Pharma. Physics 260M avg. 800M avg. 15101520 Pool factor (a) Quality by pool factor for both model sizes. 255075100 Relative storage footprint (%) 98.0 98.5 99.0 99.5 100.0 Retained relative nDCG@10 (%) NeoMME-260M NeoMME-800M (b) Quality versus storage for both model sizes. Figure 11|Hierarchical token pooling on ViDoRe v3: impact of the pool factor on retained retrieval quality for both NeoMME-Retriever models. We keep a downscale-only 2,048-pixel cap as the default because it preserves the best measured retrieval quality without adding tokens to smaller native pages. Forcing every page to a 2,048-pixel longest side through upsampling also does not improve either head. A 1,536-pixel cap is a lower-cost option when a small quality loss is acceptable. 5.9. Compression and storage While a dense embedding stores one fixed-width vector per document, late-interaction storage scales linearly with the number of document vectors. A larger processed image has more patches, so the late-interaction embedding has more vectors, and the index uses more storage. subsection 5.6 uses a 2,048-pixel longest-side cap for retrieval performance, but compression is required to make this setup tractable for large document corpora, because high-resolution images are encoded into large late-interaction embeddings. Hierarchical token pooling reduces the number of stored vectors, and asymmetric quantization reduces the number of bytes per vector. We evaluate each method separately, then combine them for the 260M model to measure their joint quality–storage trade-off. Hierarchical token pooling. Hierarchical token pooling clusters similar document vectors and replaces each cluster with its mean (Clavié et al., 2024). At pool factor 7, both NeoMME models reduce the storage used by their document vectors by about 7×while retaining more than 99% of factor-one ViDoRe v3 nDCG@10. In the paper’s unquantized text-retrieval experiments, factor three is the practical near-lossless boundary: it reduces average quality by less than 1%, while factors four and six reduce it by about 3% and 10%. In our visual document experiments, factor 10 retains 99.2% of NeoMME-260M quality and 98.9% of NeoMME-800M quality, while factor 20 still retains 97.8% and 98.0%, respectively. The models, datasets, and evaluation protocols differ, so this is not a controlled modality comparison. However, the wider range of low-loss pooling factors suggests that token pooling is especially effective for visual document retrieval, possibly because page images contain more redundant patch representations than text contains redundant token representations. Figure 11a shows all measured factors for both sizes, and Figure 11b shows the corresponding storage trade-off. The experiment does not measure index overhead or serving latency. 20 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 7 | Int8 document representations reduce packed storage by 3.9× while retaining at least 99.96% of float32 nDCG@10, and binary documents reduce storage by 32×. Arrows give the change from the float32 baseline in nDCG@10 points (1 point = 0.01). All results use the evaluation protocol in subsection 5.6. ModelQuery Document nDCG@10 Storage (kB) NeoMME-260M 15 float32 float320.5226 1536.7± 201.1 int8int80.5224 ↓−0.02 390.2± 51.1 int8binary0.5068 ↓−1.58 48.0± 6.3 binary binary0.4960 ↓−2.66 48.0± 6.3 NeoMME-800M 18 float32 float320.5560 1536.7± 201.1 int8int80.5559 ↓−0.01 390.2± 51.1 int8binary0.5369 ↓−1.91 48.0± 6.3 binary binary0.5253 ↓−3.07 48.0± 6.3 10 0 10 1 10 2 10 3 Index compression ratio 92 94 96 98 100 Retained relative nDCG@10 (%) Query / documents float32 / float32 int8 / int8 int8 / binary uncompressed control Figure 12 | Quality–storage frontier for the NeoMME-260M late-interaction index on ViDoRe v3. Labels show pool factor, retained quality, compression, and storage. Asymmetric quantization. Asymmetric quantization stores document vectors at low precision because stored documents dominate index size while queries are encoded once per search (Mixedbread, 2026). We test float32 query–document scoring, matched int8 scoring, int8 queries with binary documents, and matched binary scoring. The packed indexes use per-token symmetric int8 codes with float16 scales or sign-bit binary codes and are built from released late-interaction embeddings. We verified that packed scores match the float reference within 5×10 −5 on every query, with an observed maximum difference of 1.8×10 −7 . The float32 baselines match the main retrieval scores. Storage counts vector codes, int8 scales, document offsets, and identifiers, while excluding query bytes, approximate nearest-neighbor routing, and filesystem overhead. Table 7 shows that quantizing queries to int8 costs at most 0.0002 nDCG@10, while int8 and binary document indexes are 3.9×and 32×smaller than float32. Matched binary query–document scoring can also reduce retrieval latency because packed binary dot products can use bitwise XOR and population-count instructions. Realizing that speedup requires specialized kernels that operate directly on packed bits; dequantizing before scoring removes the advantage. Developing and benchmarking these kernels is outside the scope of this study, so we report quality and storage rather than optimized retrieval latency. Combined compression. We also measure every combination of token pooling and query and document quantization for NeoMME-260M. The study reports task-macro nDCG@10 relative to the float32 baseline and computes compression from deployed bytes. Quantizing queries to int8 alongside the documents costs at most 0.0002 nDCG@10 at any document setting. Pool factor 10 with int8 queries and int8 documents reduces storage from 1536.7 kB to 39.0 kB (39.4×) and retains 99.16% of the baseline nDCG@10. Pool factor 8 with int8 queries and binary documents reduces storage from 1536.7 kB to 6.0 kB (255.5×) and retains 95.19%. Figure 12 shows the three selected precision pairs across all measured pool factors. 21 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 5.10. Indexing efficiency NeoMME-260MNeoMME-800M VultronRetrieverFlash-0.8B Qwen3-VL-Embedding-2B ColModernVBERT ColSmol-256M ColQwen2.5-v0.2 Nemotron-ColEmbed-VL-4B-v2 0 50 100 150 200 250 300 350 400 Optimized indexing throughput (pages/s) 768 1024 1536 2048 768 1024 1536 2048 768 1024 1344 capped 768 1024 1344 capped 2048 fi xed 2048 fi xed 756 capped 768 896 capped Figure 13|Compiled document-encoding throughput over the resolution grid on one NVIDIA L40S. Figure 13 compares model-side document-encoding throughput. We run each model’s official image processor before timing to separate processor runtime from model and accelerator execution. The processor still determines each model’s input shape and token count. The timer covers device transfer, the model forward pass, output unpadding, and contiguous float32 serialization. Each row uses a separately calibrated batch, and the figure includes only the image sizes that each processor actually encodes. subsection D.1 gives the complete protocol and cross-hardware results. At 2048×2048, NeoMME-260M reaches 51.3 pages per second, which is 1.97×ColModernVBERT’s throughput of 26.0 pages per second. NeoMME-800M reaches 39.2 pages per second at 1536×1536, 94% faster than Vultron Flash’s 20.2 pages per second at its documented 1344×1344 square. At the 2048×2048 resolution used for its retrieval-quality results, NeoMME-800M still indexes faster, at 21.2 pages per second. The unmatched comparison favors Vultron because NeoMME-800M processes 31% more image pixels. Table 27 reports the largest effective square for every model. Document encoding happens offline, while query encoding runs for every request. In interactive applications such as Visual RAG, described in subsection 2.4, query-encoding latency adds to the time between a user query and the generated answer. The measured NeoMME latency is small on the tested hardware. subsection D.2 reports the mean and 95th-percentile latency. 22 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 5.11. Interpretability Query: "Which hour of the day had the highest overall electricity generation in 2019?" Figure 14|Similarity map for the query to- kenhour, overlaid on a document from the ViDoRe syntheticDocQA energy test set. 25 We reuse ColPali’s similarity-map methodology to visu- alize late-interaction scoring. For each query token, NeoMME identifies the most similar document posi- tions, and MeanMaxSim averages these per-token max- ima into a query-to-page score. The map colors each document patch by its cosine similarity to the token representation. Figure 14 asks “Which hour of the day had the highest overall electricity generation in 2019?” and shows the map for the underlined token hour. The strong responses around the chart title, the horizontal-axis label, and the numbered hours suggest that NeoMME-260M has learned emergent OCR-like be- havior without a separate OCR pipeline. The visualization does not capture the full scoring process because MeanMaxSim also considers nonspatial structural positions that cannot be placed on the page. Some responses on background patches may also reflect the high-norm register-like tokens observed in Vision Transformers (Darcet et al., 2024), rather than local visual evidence. We therefore treat the map as a qualitative diagnostic, not a complete explanation of the retrieval score. 6. Other downstream tasks Retrieval is the principal downstream application of NeoMME. Nevertheless, we test whether the same compact backbone can serve as an initialization for language classification, token labeling, document classification, and natural-image tasks. The following experiments use the NeoMME-260M and measure adaptability rather than superiority over models specialized for a single modality, and assess whether one shared backbone can support task-specific text, document-image, and multimodal fine-tuning. 6.1. Language transfer We evaluate the 17-task suite used for LFM2.5-Encoder, comprising eight GLUE tasks, four SuperGLUE tasks, and five multilingual tasks (Wang et al., 2018, 2019; Liquid AI, 2026). We follow its task definitions and disjoint selection and reporting seeds, while retaining NeoMME’s matched NorMuon– AdamW optimization (Li et al., 2026b; Loshchilov and Hutter, 2019). The comparison is therefore closely aligned, but not a same-harness reproduction of Liquid’s training recipe. NeoMME reaches a 17-task mean of 75.3, within 4 points of the 79.3 reported by the dedicated LFM2.5-Encoder-230M text model. The gap is small for a similarly sized backbone whose capacity and pretraining are shared between multilingual text and raw image patches rather than optimized exclusively for language understanding. NeoMME exceeds that peer on PAWS-X, MASSIVE Intent, SeaHorse, MRPC, and WSC, although its 46.4 Matthews correlation on CoLA shows a clear weakness in acceptability judgments. The same backbone reaches 85.5±0.2 accuracy on the 8K-context LEDGAR task and 89.3 entity F1 in a one-seed CoNLL-2003 experiment (Tjong Kim Sang and De Meulder, 2003). This is additional evidence of transfer to long-document classification and token labeling. 25 vidore/syntheticDocQA_energy_test. 23 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 6.2. Visual transfer Pretraining does not optimize a direct target over image tokens: patches remain visible and contribute through masked transcript recovery, without pixel reconstruction, image classification, or an image- level contrastive objective. Unlike vision encoders trained explicitly to organize a global image representation (Radford et al., 2021; Zhai et al., 2023), NeoMME is therefore not expected to expose a strong frozen natural-image embedding space without adaptation. Frozen 16-shot probes across 10 natural-image classification tasks average 13.2 accuracy (Xiao et al., 2025), while task-specific fine-tuning reaches 77.1±0.3 on Food101, 63.9±0.2 on Oxford Pets, and 46.8±6.3 on Stanford Cars. These natural-image experiments use an exploratory evaluation protocol and are reported as adaptation diagnostics rather than leaderboard results. Transfer is stronger for document images, which are closer to the pretraining data and objective. On RVL-CDIP (Harley et al., 2015), a frozen first-token probe reaches 51.6±0.3 accuracy, while fine-tuning on only 6,000 examples, less than 2% of the available training set, raises accuracy to 81.5±0.6. RVL-CDIP contains known annotation noise and train–test overlap (Larson et al., 2026), so this result is best interpreted as evidence of rapid document adaptation rather than clean out-of-distribution generalization. 7. Limitations This work was completed under limited time and compute. While we believe the NeoMME results are interesting, the study at hand has several limitations. Pretraining scale. NeoMME uses much less pretraining data than ModernBERT. Each model processes about 524B packed input tokens, including about 290B from the text-only stream, while ModernBERT trains on about 2T text tokens (Warner et al., 2025). Although the accounting methods differ, ModernBERT sees roughly seven times more text-only data. Longer runs could test how NeoMME responds to more text, more images, and more training steps. Visual objective. NeoMME has no objective that predicts or reconstructs image content. Pretraining predicts masked text while image patches remain visible, so the model receives no direct image-token or global image-representation target. The weak frozen natural-image results in section 6 may follow from this choice. Matched studies of image-token prediction, pixel reconstruction, and image-level contrastive objectives could separate the effect of the objective from the data mixture and architecture. Retrieval supervision. NeoMME uses much less text retrieval supervision than SOTA retrieval system such as mLateOn. During retrieval training, each model processes about 430K sampled pure-text query examples and 850K sampled image-query examples. In comparison, mLateOn processes about 660M contrastive query–document examples followed by about 16M multilingual hard-negative examples (Sourty et al., 2026). Larger matched training sets could show how much of the BEIR difference comes from data scale rather than model architecture, objectives, or evaluation code. Mixed-modality corpora. Training NeoMME-Retriever on both text and page-image retrieval data places text chunks and page images in one embedding space, so we can rank both from a single candidate pool. However, we did not run this evaluation, though two benchmarks in the literature cover it: UniDoc-Bench scores joint retrieval over a unified corpus of 70,000 PDF pages (Peng et al., 2025), and MixBench targets mixed-modality search over collections of images, texts, and multimodal documents (Li et al., 2025a). Future work could measure NeoMME-Retriever on both, then extend them to the multilingual page-image corpora that the model targets. Visual document data. Visual retrieval lacks a large multilingual dataset with broad coverage of document types and layouts. Real-world PDFs are difficult to translate while preserving their layout and visual content. Future work could generate documents from code, translate their source content, 24 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference and render matched pages in many languages. Releasing the resulting dataset would extend the translation augmentation used for mLateOn from text retrieval to visual document retrieval. Language coverage for retrieval. Visual retrieval data has much narrower language coverage than text retrieval data. The text stream draws heavily on mLateOn’s translate-train corpus, which covers English plus Arabic, French, German, Italian, Norwegian, Portuguese, Spanish, and Swedish, while another multilingual pool covers 13 languages. The visual stream contains pages in English, French, German, Italian, and Spanish, while generated queries add Portuguese. Synthetic document generation could expand page and query coverage across more scripts and regions, especially for low-resource languages. Ablations. The current results do not isolate the effects of major architecture, model-scaling, data, and training choices. Future work could compare these factors under matched compute and evaluation settings to identify which choices drive pretraining, retrieval, and transfer performance. Distillation. Neither pretraining nor retrieval fine-tuning uses teacher supervision. Future work could distill the 800M model into the 260M model at both stages and compare against matched training without distillation. Data bias. Training data from the web and machine-generated sources can reproduce social, geo- graphic, topical, and language biases (Singh et al., 2025). A dedicated evaluation could measure how often the biases affect representations, retrieval rankings, and generated text, then compare data filtering and balancing methods. Safety. Iterative masked-token sampling can produce harmful, private, or biased text even though NeoMME is not a conversational model. Future work could measure this behavior across text and image inputs and different decoding methods. 8. Conclusion NeoMME addresses common limitations of multimodal encoders built by repurposing separate image and text encoders. The model maps multilingual text tokens and raw 32×32 image patches into one long-context bidirectional Transformer encoder trained from scratch. Dynamic resolution preserves image aspect ratios without patch merging, while the same masked-diffusion objective supports text-only and image-conditioned denoising. The 260M and 800M models share this architecture and are trained on multilingual text, code, mathematics, natural images, and document images. NeoMME-Retriever validates the shared backbone on visual document and text retrieval. One forward pass produces both dense and late-interaction representations for flexibility at inference time. The 260M model outperforms all evaluated models strictly below 800M parameters and, at a matched 2048×2048 input size on an NVIDIA L40S, encodes pages at about 2×ColModernVBERT’s throughput. The 800M model is competitive with models several times its size, and it encodes pages at much higher resolution for comparable throughput. Moreover, we show that hierarchical token pooling and asymmetric quantization reduce the late-interaction multimodal embeddings from roughly 1.5 MB to 6 kB per page, a 255× compression, while retaining more than 95% of the original retrieval quality. We release the NeoMME model checkpoints and a day-zero Hugging Face Transformers implemen- tation to give practitioners a foundation to build efficient multimodal and multilingual representation models and advance research on the topic. 25 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Acknowledgements We thank H Company for supporting our work and providing the compute to train NeoMME. References Mihir Agarwal, Viraj Agrawal, Sabyasachi Basu, Ankit Garg, and Kirankumar Shiragur. Retrieval needs multivectors: An exponential separation. arXiv preprint arXiv:2608.21494, 2026. URL https://arxiv.org/abs/2608.21494. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of EMNLP, 2023. URL https://arxiv.org/abs/2305.13245. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, et al. Flamingo: A visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2204.14198. Ishrat Jahan Ananya and Amelie Chatelain. One adapter, both modalities: Field notes from building and serving a multimodal reranker.https://huggingface.co/blog/lightonai/lighto n-rerank, 2026. Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. VLMo: Unified vision-language pre-training with mixture- of-modality-experts. In Advances in Neural Information Processing Systems, 2022. URLhttps: //arxiv.org/abs/2111.02358. Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Introducing our multimodal models, 2023. URLhttps://w.adept.ai/blo g/fuyu-8b. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2Vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024. URLhttps://openreview.net/forum?id=IW1PR7 vEBf. Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. URL https://arxiv.org/abs/2004.05150. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, et al. PaliGemma: A versatile 3b vision-language model for transfer. arXiv preprint arXiv:2407.07726, 2024. URLhttps://arxiv.org/abs/2407.077 26. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, et al. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of ICML, 2023. URL https://arxiv.org/abs/2304.01373. Yonatan Bitton, Michael Elhadad, Gabriel Stanovsky, and Roy Schwartz. Data efficient masked language modeling for vision and language. In Findings of the Association for Computational Linguistics: EMNLP 2021, 2021. doi: 10.18653/v1/2021.findings-emnlp.259. URLhttps: //aclanthology.org/2021.findings-emnlp.259. 26 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, et al. EuroBERT: Scaling multilingual encoders for european languages. arXiv preprint arXiv:2503.05500, 2025. URL https://arxiv.org/abs/2503.05500. Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef, Céline Hudelot, and Pierre Colombo. BidirLM: From text to omnimodal bidirectional encoders by adapting and composing causal LLMs. arXiv preprint arXiv:2604.02045, 2026. URLhttps://arxiv.org/abs/2604.0 2045. Blake Bordelon, Lorenzo Noci, Mufan Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hy- perparameter transfer in residual networks: Dynamics and scaling limit, 2023. URLhttps: //arxiv.org/abs/2309.16620. Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. URL https://arxiv.org/abs/2405.09818. Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022. URL https://arxiv.org/abs/2202.04200. Junyi Chen, Longteng Guo, Jia Sun, Shuai Shao, Zehuan Yuan, Liang Lin, and Dongyu Zhang. EVE: Efficient vision-language pre-training with masked prediction and modality-aware MoE. Proceedings of the AAAI Conference on Artificial Intelligence, 38(2):1110–1119, 2024a. doi: 10.1609/aaai.v38i2 .27872. URL https://arxiv.org/abs/2308.11971. Lizhang Chen, Jonathan Li, Kaizhao Liang, Baiyu Su, Cong Xie, Nuo Wang Pierse, Chen Liang, Ni Lao, and Qiang Liu. Cautious weight decay. arXiv preprint arXiv:2510.12402, 2025. Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. SOLO: A single transformer for scalable vision-language modeling. Transactions on Machine Learning Research, 2024b. URLhttps: //arxiv.org/abs/2407.06438. Jaemin Cho, Debanjan Mahata, Ozan İrsoy, Yujie He, and Mohit Bansal. M3DocRAG: Multi- modal retrieval is what you need for multi-page multi-document understanding. arXiv preprint arXiv:2411.04952, 2024. URL https://arxiv.org/abs/2411.04952. Benjamin Clavié, Antoine Chaffin, and Griffin Adams. Reducing the footprint of multi-vector retrieval with minimal performance impact via token pooling. arXiv preprint arXiv:2409.14683, 2024. URL https://arxiv.org/abs/2409.14683. Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In International Conference on Learning Representations, 2024. URLhttps://arxiv.or g/abs/2309.16588. Mostafa Dehghani, Josip Djolonga, Basil Mustafa, et al. Scaling vision transformers to 22 billion parameters. In Proceedings of ICML, 2023. URL https://arxiv.org/abs/2302.05442. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, 2019. URL https://aclanthology.org/N19-1423/. Laxman Dhulipala, Majid Hadian, Rajesh Jayaram, Jason Lee, and Vahab Mirrokni. MUVERA: Multi- vector retrieval via fixed dimensional encodings. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2405.19504. 27 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2406.11832. Haiwen Diao, Mingxuan Li, Silei Wu, Linjun Dai, Xiaohua Wang, Hanming Deng, Lewei Lu, Dahua Lin, and Ziwei Liu. From pixels to words—towards native vision-language primitives at scale. arXiv preprint arXiv:2510.14979, 2025a. URL https://arxiv.org/abs/2510.14979. Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. EVEv2: Improved baselines for encoder-free vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21014–21025, 2025b. URL https://arxiv.org/abs/2502.06788. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2010.11929. Sedigheh Eslami, Maksim Gaiduk, Markus Krimmel, Louis Mark Milliken, Bo Wang, and Denis Bykov. Diffusion-pretrained dense and contextual embeddings. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics: Industry Track, pages 990–1004, 2026. doi: 10.186 53/v1/2026.acl-industry.69. URL https://aclanthology.org/2026.acl-industry.69/. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. ColPali: Efficient document retrieval with vision language models. In International Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2407.01449. Chao Feng, Zihao Wei, and Andrew Owens. Masked diffusion captioning for visual feature learning. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 25247–25263, 2025. doi: 10.18653/v1/2025.findings-emnlp.1376. URLhttps://aclanthology.org/2025. findings-emnlp.1376/. Thibault Formal, Stéphane Clinchant, Hervé Déjean, and Carlos Lassance. SPLATE: Sparse late interaction retrieval. In Proceedings of SIGIR, pages 2635–2640, 2024. doi: 10.1145/3626772.36 57968. Stella Frank, Emanuele Bugliarello, and Desmond Elliott. Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9847–9857. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.emnlp-main.775. URLhttps://aclanthology.org /2021.emnlp-main.775. Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup. In Proceedings of RepL4NLP, 2021. URLhttps://arxiv.org/ab s/2101.06983. Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. URL https://arxiv.org/abs/2408.00118. Gemma Team. Gemma 4 technical report, 2026. URL https://arxiv.org/abs/2607.02770. Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205.14204, 2022. URL https://arxiv.org/abs/2205.14204. 28 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Athos Georgiou. VultronRetrieverCore-Qwen3.5-4.5B: Mid-tier late-interaction visual document retrieval at 320 dimensions.https://huggingface.co/vultr/VultronRetrieverCore-Q wen3.5-4.5B, 2026a. Athos Georgiou. VultronRetrieverFlash-Qwen3.5-0.8B: Small-tier late-interaction visual document retrieval at 320 dimensions.https://huggingface.co/vultr/VultronRetrieverFlash -Qwen3.5-0.8B, 2026b. Athos Georgiou. VultronRetrieverPrime-Qwen3.5-8B: Late-interaction visual document retrieval at 320 dimensions.https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B, 2026c. Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, and Han Xiao. jina-embeddings-v4: Universal embeddings for multimodal multilingual retrieval. In Proceedings of the Workshop on Multilingual Representation Learning, pages 531–550, 2025. doi: 10.18653/v1/2025.mrl-main.36. Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In International Conference on Document Analysis and Recognition, 2015. Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusion- BERT: Improving generative masked language models with diffusion models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 4521–4534, 2023. doi: 10.18653/v1/2023.acl-long.248. URL https://aclanthology.org/2023.acl-long.248/. Sebastian Hofstätter, Omar Khattab, Sophia Althammer, Mete Sertkan, and Allan Hanbury. Introducing neural bag of whole-words with ColBERTer: Contextualized late interactions using enhanced reduction. In Proceedings of CIKM, pages 737–747, 2022. doi: 10.1145/3511808.3557367. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URLhttps://arxi v.org/abs/1904.09751. Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, Yanning Chen, and Zhipeng Wang. Liger-Kernel: Efficient triton kernels for LLM training. In Championing Open-source Development in Machine Learning Workshop at ICML, 2025. URL https://openreview.net/forum?id=36SjAIT42G. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024. URLhttps://arxiv.org/abs/2404.06395. Xiao Shi Huang, Felipe Pérez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4475–4483. PMLR, 2020. URL https://proceedings.mlr.press/v119/huang20f.html. Xin Huang and Kye Min Tan. Beyond text: Unlocking true multimodal, end-to-end RAG with tomoro ColQwen3, 2025. URLhttps://tomoro.ai/insights/beyond-text-unlocking-true-m ultimodal-end-to-end-rag-with-tomoro-colqwen3. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. LayoutLMv3: Pre-training for document AI with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, 2022. doi: 10.1145/3503161.3548112. 29 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: Towards removing the curse of dimensionality. In Proceedings of STOC, pages 604–613, 1998. doi: 10.1145/276698.276876. Jiho Jang, Chaerin Kong, Donghyeon Jeon, Seonhoon Kim, and Nojun Kwak. Unifying vision-language representation space with single-tower transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023. URL https://arxiv.org/abs/2211.11153. Hervé Jégou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128, 2011. doi: 10.1109/TPAMI.2010.57. Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019. doi: 10.1109/TBDATA.2019.2921572. Keller Jordan and modded-nanogpt contributors. modded-nanogpt: Speedrunning the NanoGPT baseline, 2026. URL https://github.com/KellerJordan/modded-nanogpt. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URLhttps://kell erjordan.github.io/posts/muon/. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP, 2020. URL https://arxiv.org/abs/2004.04906. Mohammad Aflah Khan, Krishna P. Gummadi, Manish Gupta, and Abhilasha Ravichander. Fractional rotation, full potential? investigating performance and convergence of partial rope, 2026. URL https://arxiv.org/abs/2603.11611. Omar Khattab and Matei Zaharia. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of SIGIR, 2020. URLhttps://arxiv.org/abs/2004 .12832. Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and-language transformer without convolution or region supervision. In Proceedings of ICML, 2021. URLhttps://arxiv.org/abs/2102.033 34. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In Advances in Neural Information Processing Systems, 2022. URLhttps: //arxiv.org/abs/2205.13147. Aurélien Lac and Tony Wu. late-interaction-kernels: Fused triton kernels for late-interaction scoring. Software,§ https://github.com/hcompai/late-interaction-kernels, 2026. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL https://arxiv.org/abs/1909.11942. Stefan Larson, Attila Nagy, Sam Desai, Cyrus Desai, Nicole C. Lima, Yixin Yuan, Siddharth Betala, Kaushal K. Prajapati, Jamiu T. Suleiman, Sharad Duwal, and Kevin Leach. Revising RVL-CDIP: Quantifying errors and test-train overlap. In ACM Symposium on Document Engineering, 2026. doi: 10.1145/3820755.3821486. 30 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Binxu Li, Yuhui Zhang, Xiaohan Wang, Weixin Liang, Ludwig Schmidt, and Serena Yeung-Levy. Closing the modality gap for mixed modality search. arXiv preprint arXiv:2507.19054, 2025a. URL https://arxiv.org/abs/2507.19054. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 19730–19742, 2023. URL https://arxiv.org/abs/2301.12597. Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Qwen3-VL-Embedding and Qwen3-VL- Reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720, 2026a. URL https://arxiv.org/abs/2601.04720. Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. LaViDa: A large diffusion language model for multimodal understanding. In Advances in Neural Information Processing Systems, 2025b. URL https://arxiv.org/abs/2505.16839. Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. NorMuon: Making muon more efficient and scalable. In Proceedings of ICML, 2026b. URLhttps://arxiv.org/abs/2510.0 5491. Liquid AI. LFM2.5-Encoders: Fast at long context, even on CPU. Liquid AI Blog, 2026. URL https://huggingface.co/blog/LiquidAI/lfm2-5-encoders. Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. SuperBPE: Space travel for language models. In Conference on Language Modeling, 2025. URLhttps: //arxiv.org/abs/2503.13423. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2304.08485. António Loison, Quentin Macé, Antoine Edy, Victor Xing, Tom Balough, Gabriel de Souza P. Moreira, Bo Liu, Manuel Faysse, Celine Hudelot, and Gautier Viaud. ViDoRe v3: A comprehensive evaluation of retrieval augmented generation in complex real-world scenarios. In Proceedings of ACL, pages 16570–16600, 2026. doi: 10.18653/v1/2026.acl-long.755. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1711.05101. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. In Proceedings of EMNLP, 2024. URLhttps: //arxiv.org/abs/2406.11251. Quentin Macé, António Loison, and Manuel Faysse. ViDoRe benchmark v2: Raising the bar for visual retrieval. arXiv preprint arXiv:2505.17166, 2025. URLhttps://arxiv.org/abs/2505.17166. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. mmBERT: A modern multilingual encoder with annealed language learning. arXiv preprint arXiv:2509.06888, 2025. URL https://arxiv.org/abs/2509.06888. Mixedbread. Asymmetric quantization for late interaction retrieval. Blog post, 2026. URLhttps: //w.mixedbread.com/blog/asymmetric-quant. 31 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Gabriel de Souza P. Moreira, Ronay Ak, Mengyao Xu, Oliver Holworthy, Benedikt Schifferer, Zhiding Yu, Yauhen Babakhin, Radek Osmulski, Jiarui Cai, Ryan Chesler, Bo Liu, and Even Oldridge. Nemotron ColEmbed V2: Top-performing late interaction embedding models for visual document retrieval. arXiv preprint arXiv:2602.03992, 2026. URLhttps://arxiv.org/abs/2602.03992. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Proceedings of EACL, 2023. URLhttps://aclanthology.org/2023.eacl-m ain.148/. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2502.09992. NLLB Team. Scaling neural machine translation to 200 languages. Nature, 630:841–846, 2024. doi: 10.1038/s41586-024-07335-x. Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085, 2019. URL https://arxiv.org/abs/1901.04085. Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, 2020. doi: 10.18653/v1/2020.findings-emnlp.63. URLhttps://aclant hology.org/2020.findings-emnlp.63/. Nomic Team. Nomic embed multimodal: Interleaved text, image, and screenshots for visual document retrieval, 2025. URL https://w.nomic.ai/news/nomic-embed-multimodal. OpenSearch-AI. Ops-ColQwen3: State-of-the-art multimodal embedding model for visual document retrieval. https://huggingface.co/OpenSearch-AI/Ops-Colqwen3-4B, 2026. Xiangyu Peng, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, and Chien-Sheng Wu. UniDoc-Bench: A unified benchmark for document-centric multimodal RAG. arXiv preprint arXiv:2510.03663, 2025. URL https://arxiv.org/abs/2510.03663. Roi Pony, Daniel Ezer, Adi Raz Goldfarb, Idan Friedman, Oshri Naparstek, and Udi Barzelay. FLASH- MAXSIM: IO-aware fused kernels for late-interaction retrieval. arXiv preprint arXiv:2605.29517, 2026. URL https://arxiv.org/abs/2605.29517. Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2505.06708. Qwen Team. Qwen3-next: Towards ultimate training and inference efficiency, 2025. URLhttps: //qwen.ai/blog?id=qwen3-next. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, et al. Learning transferable visual models from natural language supervision. In Proceedings of ICML, 2021. URL https://arxiv.org/abs/2103.00020. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. In Proceedings of EMNLP-IJCNLP, 2019. URLhttps://aclanthology.org/D19-141 0/. 32 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, 2024. URLhttps://arxiv.org/ abs/2406.07524. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of NAACL, 2022a. URL https://arxiv.org/abs/2112.01488. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. PLAID: An efficient engine for late interaction retrieval. In Proceedings of CIKM, 2022b. URLhttps: //arxiv.org/abs/2205.09707. Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention- 3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2407.08608. Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2406.04329. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Proceedings of ACL, pages 18761–18799, 2025. doi: 10.18653/v1/2025.acl-long.919. David R. So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Primer: Searching for efficient transformers for language modeling. In Advances in Neural Information Processing Systems, 2021. URL https://arxiv.org/abs/2109.08668. Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, and Amélie Chatelain. DenseOn with the LateOn: Fully open dense and late-interaction models for multilingual, long-context, and code search. In arXiv preprint arXiv:2607.27178, 2026. URLhttps://arxiv.org/abs/2607.27178. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. URLhttps: //arxiv.org/abs/2104.09864. Yubo Sun, Chunyi Peng, Yukun Yan, Shi Yu, Zhenghao Liu, Sen Mei, Chi Chen, and Maosong Sun. VisRAG2.0: Mitigating visual hallucinations via evidence-guided multi-image reasoning in visual retrieval-augmented generation. arXiv preprint arXiv:2510.09733, 2025. URLhttps: //arxiv.org/abs/2510.09733. Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragkiadaki. Unified multimodal discrete diffusion. arXiv preprint arXiv:2503.20853, 2025. doi: 10.48550/arX iv.2503.20853. URL https://arxiv.org/abs/2503.20853. Paul Teiletche, Quentin Macé, Max Conti, Antonio Loison, Gautier Viaud, Pierre Colombo, and Manuel Faysse. ModernVBERT: Towards smaller visual document retrievers. In Proceedings of ICML, 2026. URL https://arxiv.org/abs/2510.01149. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of NeurIPS Datasets and Benchmarks, 2021. URL https://arxiv.org/abs/2104.08663. 33 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language- independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning, pages 142–147, 2003. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, et al. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. URL https://arxiv.org/abs/2502.14786. Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. URL https://arxiv.org/abs/1807.03748. Siddharth Verma, Yuchen Lu, Rui Hou, Hanchao Yu, Nicolas Ballas, Madian Khabsa, and Amjad Alma- hairi. Uniform masking prevails in vision-language pretraining. arXiv preprint arXiv:2212.05195, 2022. URL https://arxiv.org/abs/2212.05195. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP, pages 353–355, 2018. doi: 10.18653/v1/W18-5446. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32, 2019. Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. UFO: A unified transformer for vision-language representation learning. arXiv preprint arXiv:2111.10023, 2021. URL https://arxiv.org/abs/2111.10023. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. In arXiv preprint arXiv:2409.12191, 2024. URL https://arxiv.org/abs/2409.12191. Shuai Wang, Yu Yin, Shengyao Zhuang, Bevan Koopman, and Guido Zuccon. DiffRetriever: Parallel representative tokens for retrieval with diffusion language models. arXiv preprint arXiv:2605.07210, 2026a. URL https://arxiv.org/abs/2605.07210. Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEiT pretraining for all vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19175–19186, 2023. URLhttps: //arxiv.org/abs/2208.10442. Zihang Wang, Siyue Zhang, Yilun Zhao, Jingyi Yang, Tingyu Song, Anh Tuan Luu, and Chen Zhao. Analyzing diffusion and autoregressive vision language models in multimodal embedding space. arXiv preprint arXiv:2602.06056, 2026b. URL https://arxiv.org/abs/2602.06056. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of ACL, 2025. URL https://aclanthology.org/2025.acl-long.127/. Orion Weller, Michael Boratko, Iftekhar Naim, and Jinhyuk Lee. On the theoretical limitations of embedding-based retrieval. arXiv preprint arXiv:2508.21038, 2025. URLhttps://arxiv.org/ abs/2508.21038. 34 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. Seq vs seq: An open suite of paired encoders and decoders. In International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2507.11412. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clément Delangue, Anthony Moi, Pierric Cistac, Tim Rault, et al. Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP: System Demonstrations, pages 38–45, 2020. URLhttps://arxiv.org/ab s/1910.03771. Chenghao Xiao, Isaac Chung, Imene Kerboua, Jamie Stirling, Xin Zhang, Márton Kardos, Roman Solomatin, Noura Al Moubayed, Kenneth Enevoldsen, and Niklas Muennighoff. MIEB: Massive image embedding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. URL https://arxiv.org/abs/2504.10471. Mengyao Xu, Gabriel Moreira, Ronay Ak, Radek Osmulski, Yauhen Babakhin, Zhiding Yu, Benedikt Schifferer, and Even Oldridge. Llama NeMoRetriever ColEmbed: Top-performing text-image retrieval model. arXiv preprint arXiv:2507.05513, 2025. URLhttps://arxiv.org/abs/2507 .05513. Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.02244. Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. MMaDA: Multimodal large diffusion language models. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2505.15809. Zongheng Yang, Zhanghao Wu, Michael Luo, Wei-Lin Chiang, Romil Bhardwaj, Woosuk Kwon, Siyuan Zhuang, Frank Sifei Luan, Gautam Mittal, Scott Shenker, and Ion Stoica. SkyPilot: An intercloud broker for sky computing. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 437–455, 2023. URLhttps://w.usenix.org/conferenc e/nsdi23/presentation/yang-zongheng. Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. VisRAG: Vision-based retrieval-augmented generation on multi-modality documents, 2024. URL https://arxiv.org/abs/2410.10594. Shuangfei Zhai. Exclusive self attention. arXiv preprint arXiv:2603.09078, 2026. URLhttps: //arxiv.org/abs/2603.09078. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of ICCV, 2023. URL https://arxiv.org/abs/2303.15343. Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32, 2019. URLhttps://arxiv.org/abs/1910.07467. Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representations, 2019. URLhttps://ar xiv.org/abs/1901.09321. Siyue Zhang, Yilun Zhao, Liyuan Geng, Arman Cohan, Anh Tuan Luu, and Chen Zhao. Diffusion vs. autoregressive language models: A text embedding perspective. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4273–4303, 2025. doi: 10.18653/v1/2025.emnlp-main.213. URLhttps://aclanthology.org/2025.emnlp-main. 213/. 35 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, Fares Obeid, and Zhenzhong Lan. Value residual learning. In Proceedings of ACL, 2025. URL https://arxiv.org/abs/2410.17897. Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni- Perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16804– 16815, 2022. doi: 10.1109/CVPR52688.2022.01630. URLhttps://arxiv.org/abs/2112.0 1522. 36 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference A. Training and implementation details This section describes the data mixtures, optimization settings, hardware, input formats, and sampling procedures used for pretraining and retrieval training. A.1. Pretraining data mixture A.1.1. Text-only stream Table 8 and Table 9 report the 58 datasets used in the pretraining text mixture, and the dataset names link to their Hugging Face dataset cards. The listed weights are configured source-sampling weights before document packing, not observed shares of tokens, documents, or examples. Table 10 groups related language and configuration rows into dataset families while the two weight tables keep those rows separate. Table 8 | Pretraining text-source weights for English-dominant and multilingual web and PDF text. English-dominant web and multilingual PDF text Dataset and configWeight HuggingFaceFW/fineweb-edu default0.167660 nvidia/Nemotron-C-v2.1 High-Quality0.072896 HuggingFaceFW/finepdfs eng_Latn0.072896 fra_Latn0.014579 deu_Latn0.014579 spa_Latn0.007290 rus_Cyrl0.007290 arb_Arab0.007290 cmn_Hani0.007290 Multilingual web text epfml/FineWeb2-HQ ConfigWeight deu_Latn 0.021869 fra_Latn 0.021869 spa_Latn 0.021869 rus_Cyrl 0.021869 cmn_Hani 0.021869 jpn_Jpan 0.021869 ita_Latn 0.021869 por_Latn 0.021869 pol_Latn 0.014579 nld_Latn 0.014579 ind_Latn 0.014579 tur_Latn 0.014579 ces_Latn 0.014579 vie_Latn 0.014579 swe_Latn 0.014579 fas_Arab 0.014579 arb_Arab 0.014579 ell_Grek 0.014579 dan_Latn 0.014579 hun_Latn 0.014579 37 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 9|Pretraining text-source weights for synthetic, reference, mathematics, question answering, and code. Synthetic, reference, and mathematics Dataset and configWeight HuggingFaceTB/smollm-corpus cosmopedia-v20.102054 wikimedia/wikipedia 20231101.en0.021869 20231101.fr0.007290 20231101.de0.007290 20231101.es0.007290 20231101.ru0.007290 20231101.zh0.007290 20231101.ja0.007290 20231101.it0.007290 20231101.pt0.007290 20231101.ar0.007290 HuggingFaceTB/finemath finemath-4plus 0.051027 Question answering and code Dataset, configWeight MMLU, auxiliary_train 0.006196 MedMCQA, default0.004009 CommonsenseQA, default 0.000219 QASC, default0.000182 OpenBookQA, main0.000109 AI2 ARC, ARC-Easy0.000058 AI2 ARC, ARC-Challenge 0.000029 bigcode/starcoderdata python0.005103 javascript0.003645 typescript0.002916 java0.002916 c0.002187 cpp0.002187 go0.002187 rust0.002187 sql0.001822 shell0.001822 Table 10 | Pretraining text source descriptions. GroupDataset familyDescription Web and PDF FineWeb-EduEnglish web text filtered for educational quality. Nemotron-C-v2.1High-quality English Common Crawl text curated by NVIDIA. FinePDFsExtracted PDF text in English, French, German, Span- ish, Russian, Arabic, and Chinese. FineWeb2-HQHigh-quality web text in 20 languages. Synthetic, reference, and math Cosmopedia-v2Synthetic educational text generated from web and reference material. WikipediaEncyclopedic articles in 10 languages. FineMathMathematical web text from thefinemath-4plus configuration. Question answering MMLUMultiple-choice questions spanning academic and pro- fessional subjects. MedMCQAMultiple-choice medical entrance-exam questions. CommonsenseQAMultiple-choice questions that test everyday common- sense knowledge. QASCMultiple-choice science questions paired with support- ing facts. OpenBookQAElementary science questions based on an open book of scientific facts. AI2 ARCGrade-school science questions from the Easy and Chal- lenge configurations. Code StarCoderDataSource code in Python, JavaScript, TypeScript, Java, C, C++, Go, Rust, SQL, and shell. 38 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference A.1.2. Visual–text stream The visual–text stream combines five source families covering natural images, rendered documents, PDF pages, charts, diagrams, and synthetic images paired with captions, optical character recognition text, recaptions, Markdown transcriptions, or synthetic text. Table 11 reports each family’s configured sampling weight and role. These weights define the distribution within the visual–text stream, not proportions of the complete pretraining mixture or observed shares of images, pixels, or tokens. Table 11 | Pretraining visual–text source weights and descriptions. Dataset familyWeightDescription FineVision0.50A selected mixture of 45 vision–language and document datasets, predomi- nantly in English. PDFA andLightOnOCR0.30PDF page images paired with extracted text and model-generated recaptions where available. PixelProse0.10URL-fetched natural images paired with Gemini-generated captions. DocAtlas0.07Rendered document images paired with Markdown targets in 82 languages. Synthetic multilingual OCR0.03Synthetic document images and text in Japanese, Korean, simplified Chi- nese, and traditional Chinese. Total1.00 A.2. Retrieval training settings NeoMME-Retriever was fine-tuned using onep5.48xlarge 5 node with eight H100 accelerators. Table 12 summarizes the retrieval-training settings for both model sizes. Table 3 reports the more detailed pretraining settings in the main pretraining section. Table 12 | Retrieval-training settings. SettingNeoMME-260MNeoMME-800M Initialization and schedule Pretraining initializationStep 500,000 3 Step 500,000 4 Nodes11 H100 accelerators88 Steps20,00020,000 Sequences per rank88 Optimization NorMuon learning rate0.00150.0010 AdamW learning rate0.000250.0001667 Warmup100 steps100 steps ScheduleCosine to 1% of peakCosine to 1% of peak Representations Dense poolingMean poolingMean pooling Late-interaction width128128 Dense widths128, 256, 512, 1,024128, 256, 512, 1,024, 1,792 Batch construction Hard negatives7 from a 32-candidate window 7 from a 32-candidate window Query expansion tokens1010 Temperature0.020.02 GradCache micro-batch2412 Text:image batch ratio1:21:2 39 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference A.3. Processor The processor notation uses푞 푠 for a query token,푑 푡 for a document token,퐿 푞 and퐿 푑 for the query and document lengths, and 푝 푟,푐 for the patch at grid position (푟, 푐) in an 퐻-row by 푊-column patch grid. Table 13 | NeoMME processor format. Colors mark structural roles rather than model inputs. ContextFormatting Query prefix [QUERY] Document prefix [DOC] Image stream prefix [IMG] Patch-row boundary [ROW] Query expansion [MASK] Image patch at grid position (푟, 푐)푝 푟,푐 Resulting model input sequence ® Query [QUERY] 푞 1 · 푞 퐿 푞 [MASK] ×10 P Text document [DOC] 푑 1 · 푑 퐿 푑 ë Page screenshot [DOC] [IMG] [(푝 푟,0 · 푝 푟,푊−1 ) [ROW]] 퐻−1 푟=0 The processor inserts learned structural tokens into each model input.[QUERY]and[DOC] distinguish the two retrieval sides, while image documents add[IMG]before the row-major patch grid and[ROW]after each patch row to preserve its two-dimensional structure. Queries also append 10 learned[MASK]tokens, following ColBERT’s query augmentation (Khattab and Zaharia, 2020); their effect is not isolated in an ablation. Table 13 gives the full layouts. A.4. Training mixture details Text-only stream. The text stream contains 1,581,039 queries across five source pools. The LightOn sources provide the English, mLateOn, code, and organic pools (Sourty et al., 2026), while an additional multilingual pool supplies the rest. The translated mLateOn pools add further language coverage. The weights in Table 14 define the sampling distribution within this stream. Table 14 | Retrieval text-source weights. Source poolWeight Coverage Filtered English retrieval0.09 English, 10 candidates per query mLateOn multilingual0.73 Multilingual retrieval splits Code retrieval0.13 Code-search splits Additional multilingual retrieval0.0313 languages, eight candidates per query Organic retrieval (en) Organic retrieval (fr) 0.02 English and French Total1.00 Multimodal stream. The multimodal stream combines the ColPali train set, a multilingual document- image dataset, and the VisRAG in-domain and synthetic sets (Faysse et al., 2025; Yu et al., 2024). Each source has a configured weight of 0.25. Table 15 gives the final mixture sizes, and multiple queries can point to the same document. The original pairs cover English, French, German, Italian, and Spanish. 40 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 15 | Image retrieval mixture. SourceWeight # Queries # Documents Languages ColPali0.25148,123118,195 en Multilingual document images0.25351,651280,679 en, it, fr, de, es VisRAG0.25154,187122,752 en VisRAG synthetic0.25298,780239,200 en Total1.00952,741760,826 6 Query augmentation adds Portuguese, bringing the total to six languages. Query augmentation. Before hard-negative mining, we sampled 100,000 of the 760,826 pages and asked Qwen3.5-9B 10 to generate two queries per page. For each query, we uniformly sampled one of six types: multi-hop, compare-contrast, open-ended, boolean, enumerative, or numerical. We sampled question, instruction, and keyword formats with probabilities 0.60, 0.25, and 0.15. We sampled English with probability 0.35, French with probability 0.25, and Spanish, German, Italian, and Portuguese with probability 0.10 each. The prompt required a precise grounded answer and wording that differed from the page. After filtering for parsable, self-contained queries, 191,915 queries remained, increasing the multimodal query count from 760,826 to 952,741 without adding pages. During hard-negative mining, Qwen3-VL-Reranker-8B 11 kept generated pairs whose positive page scored at least 0.1. A.5. Contrastive training details Hard-negative sampling. Text examples use the hard negatives provided with the mLateOn multilin- gual dataset referenced in Table 4. Visual hard negatives are self-mined in two stages. The first stage trains NeoMME-Retriever with positive pages and in-batch negatives but no explicit hard negatives. The resulting model checkpoint then mines a fixed 32-candidate window for each visual query. The second stage repeats the same retrieval fine-tuning protocol with one positive and seven negatives sampled from that window. Cross-rank gathering adds positives and negatives from the other ranks to the candidate pool. Negatives with stored mining scores at least 0.98×the positive, and nonfinite candidates, are removed. The gate does not recompute the score with the current model during training. Memory-efficient training. GradCache separates representation computation from the contrastive loss, so the encoder can process the gathered candidate pool in smaller activation chunks (Gao et al., 2021). LIK fuses token dot products with an online maximum reduction, which avoids writing the full token-similarity tensor to high-bandwidth memory and reduces the MaxSim memory peak (Lac and Wu, 2026). The NeoMME-260M throughput control averaged 1.6% higher throughput with LIK, although run-to-run variation overlaps. Table 28 and Table 29 report the measurements. B. Supporting pretraining results B.1. Tokenizer evaluation All tokenizers encode identical texts without added special tokens. We compare NeoMME with ModernBERT (Warner et al., 2025), LFM2.5-Encoder-230M (Liquid AI, 2026), mmBERT-base (Marone et al., 2025), and EuroBERT-210m (Boizard et al., 2025). The language evaluation uses all 1,012 aligned sentences per language in FLORES-200 devtest (NLLB Team, 2024), which keeps the compared content fixed across languages. The aggregate reductions reported in section 3 are ratios of token 41 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference counts summed over the 14 languages, not averages of the per-language rates below. The domain evaluation uses 2,000 documents per source, each capped at 4,000 characters. Flag icons come from Twemoji under C-BY 4.0. Table 16|Tokenizer compression on the 14 target languages in FLORES-200 devtest. All tokenizers encode identical texts without added special tokens. Values are tokens per UTF-8 byte, and lower values are better. SourceNeoMME 3 ModernBERT 26 LFM2.5 27 mmBERT 28 EuroBERT 29 Vocabulary131,07250,36865,536256,000128,256 English0.19300.20750.20970.20440.2057 French0.21220.29260.24340.22430.2652 German0.23060.31940.24520.21650.2748 Spanish0.21540.29420.24780.21360.2617 Italian0.22650.31070.26850.23130.2830 Portuguese0.22220.30640.27250.22500.2713 Russian0.15330.27790.18120.14320.1687 Arabic0.16610.38650.24820.19040.2126 Persian0.16220.41650.38950.17420.1851 Greek0.15760.32090.50660.21540.2112 Turkish0.23440.40790.42090.25580.2576 Vietnamese 0.17900.47980.55650.20190.2059 Chinese0.24190.46710.29910.24560.2881 Japanese0.18960.35720.22930.19410.2457 Table 17|Tokenizer compression on document, math, and code sources. All tokenizers encode the same 2,000 documents per source without added special tokens. Values are tokens per UTF-8 byte, and lower values are better. SourceNeoMME 3 ModernBERT 26 LFM2.5 27 mmBERT 28 EuroBERT 29 Vocabulary131,07250,36865,536256,000128,256 P English PDF0.24390.25350.25520.26160.2390 y Math0.25100.28650.28890.29890.2792 3 Python0.21480.30920.28690.40590.2428 â JavaScript0.20920.30860.28500.39640.2462 õ SQL0.26650.36050.35330.38520.2924 Across all 204 FLORES-200 language-script pairs, NeoMME emits 65.29% more tokens than mmBERT and 15.28% more than EuroBERT in aggregate, with the largest deficits in Tibetan, South Asian, and Southeast Asian scripts. B.2. Image-conditioned generation examples The selected pretraining generations below use NeoMME-260M 3 and are qualitative examples rather than task-level captioning or optical character recognition measurements. 26 answerdotai/ModernBERT-base. 27 LiquidAI/LFM2.5-Encoder-230M. 28 jhu-clsp/mmBERT-base. 29 EuroBERT/EuroBERT-210m. 42 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 18|Selected image inputs and text generated by the NeoMME-260M model.[...]marks truncation. ImagePredicted text This image displays: A pair of red Nike sneakers with black laces and white Nike logo accents. The sneakers are displayed on a black background. This image displays: A sunset over a river. The sky is orange and yellow, with a few clouds. The river is blue and still. There are no people in the image. The image is a photograph. Further Suspension and Restriction of Hearings June 8, 2021 Page 2 ## Out of Custody Trials All out of custody trials will be suspended. Counsel should discuss re-setting these matters for trial and advise the Court of the new trial date on the date currently set for trial [...] # Main Terms of TERP Settlement Proposal ## Settlement effective at closing of a jointly approved strategic transaction ### TERP shareholders participate ratably in transaction - After transaction, regardless of form, SUNE and all pre-transaction shareholders receive equal value per share –- 9 [...] 43 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference C. Retrieval evaluation and complete results This section describes the retrieval evaluation protocol, complete metric and domain breakdowns, all trained Matryoshka widths, and supporting ablations (Kusupati et al., 2022). C.1. Evaluation protocol details Benchmarks. ViDoRe v1, v2, and v3 test document retrieval (Macé et al., 2025; Loison et al., 2026). BEIR-15 follows the MTEB 2.18.7 benchmark definition and matches the mLateOn evaluation task list (Thakur et al., 2021; Muennighoff et al., 2023). It contains 14 of the original 18 zero-shot BEIR datasets and adds the MS MARCO development set. BioASQ, Signal-1M, TREC-NEWS, and Robust04 are omitted because their corpora are not distributed as downloadable BEIR datasets and require separate reconstruction. CQADupstack’s 12 child corpora are averaged through the official aggregate task and counted once. Scoring. The visual evaluation uses downscale-only resizing with a maximum image side length of 2,048 pixels and bfloat16 execution. MeanMaxSim uses exact all-pairs scoring for ViDoRe. BEIR-15 uses the official MTEB scorer over each full corpus, with 8,192-token query and document limits, title followed by body, a compressed multi-vector index, and exact rescoring of top candidates. The dense head uses the same pipeline with one normalized vector per input. Metric aggregation and evaluation. For each task, we average the metric over judged queries and then give every task equal weight in the reported aggregate. ViDoRe v3 contains eight reported tasks, and BEIR-15 contains 15 tasks. The reported results use the 260M and 800M models at step 20,000. C.2. Complete retrieval results ViDoRe benchmark comparison. Table 19 lists every model and retrieval head included in Figure 10. The ViDoRe v3 column is the task-macro mean over the benchmark tasks. We reran ColModernVBERT on v1 to match the exact protocol used for NeoMME, and on v2 because MTEB does not provide a full-benchmark result (Teiletche et al., 2026). We reran ColSmol-500M on v3 because MTEB does not provide a full-benchmark result there either. Aggregate metrics. Table 20 reports all aggregate visual-retrieval metrics at step 20,000. Visual retrieval uses a maximum image side length of 2,048 pixels. Late-interaction uses the full 128- dimensional token representation, and dense uses the full model width. 30 ModernVBERT/bimodernvbert (Teiletche et al., 2026). 31 google/siglip-so400m-patch14-384 (Zhai et al., 2023). 32 Qwen/Qwen3-VL-Embedding-2B (Li et al., 2026a). 33 vidore/colqwen2-v1.0 (Faysse et al., 2025). 34 nvidia/llama-nemoretriever-colembed-1b-v1 (Xu et al., 2025). 35 nomic-ai/colnomic-embed-multimodal-3b (Nomic Team, 2025). 36 TomoroAI/tomoro-colqwen3-embed-4b (Huang and Tan, 2025). 37 jinaai/jina-embeddings-v4 (Günther et al., 2025). 38 nvidia/llama-nemoretriever-colembed-3b-v1 (Xu et al., 2025). 39 nvidia/llama-nemotron-colembed-vl-3b-v2 (Moreira et al., 2026). 40 vultr/VultronRetrieverCore-Qwen3.5-4.5B (Georgiou, 2026a). 41 OpenSearch-AI/Ops-Colqwen3-4B (OpenSearch-AI, 2026). 42 nvidia/nemotron-colembed-vl-4b-v2 (Moreira et al., 2026). 43 nomic-ai/colnomic-embed-multimodal-7b (Nomic Team, 2025). 44 Qwen/Qwen3-VL-Embedding-8B (Li et al., 2026a). 45 TomoroAI/tomoro-colqwen3-embed-8b (Huang and Tan, 2025). 46 vultr/VultronRetrieverPrime-Qwen3.5-8B (Georgiou, 2026c). 47 nvidia/nemotron-colembed-vl-8b-v2 (Moreira et al., 2026). 44 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 19|Visual document retrieval performance on the ViDoRe benchmarks for all compared models. A dash marks a result that was not reported. Emb. dim. is the per-token vector width for late- interaction models and the single-vector width for dense models. Model detailsViDoRe ModelLate-interaction Params. (B) Emb. dim. v1 nDCG@5 v2 nDCG@5 v3 nDCG@10 <300M BiModernVBERT 30† 0.25768–0.1541 ColModernVBERT 13 ✓0.251280.8058 ‡ 0.4068 ‡ 0.2612 † ColSmol-256M 14† ✓0.261280.79740.34820.2073 NeoMME-260M, dense 15‡ 0.261,0240.75520.40750.3907 NeoMME-260M, late-interaction 15‡ ✓0.261280.85980.52180.5226 300M to 1B ColSmol-500M 16 ✓0.501280.8249 † 0.4550 † 0.3397 ‡ SigLIP-So400M 31† 0.881,1520.56380.33000.1723 NeoMME-800M, dense 18‡ 0.801,7920.79930.44750.4391 NeoMME-800M, late-interaction 18‡ ✓0.801280.87440.55910.5560 Vultron Flash (0.8B) 17† ✓0.853200.88150.60360.5649 >1B Qwen3-VL (2B) 32† 2.132,048–0.5289 ColQwen2-v1.0 33† ✓2.211280.89230.56040.4418 ColPali-v1.3 20† ✓2.921280.84750.54720.4295 NemoRetriever (1B) 34† ✓2.422,0480.90500.62960.5548 ColNomic (3B) 35† ✓3.001280.89860.55680.5640 ColQwen2.5-v0.2 19† ✓3.751280.89540.60060.5244 Tomoro (4B) 36† ✓4.003200.90570.64690.6016 Jina v4 37† 3.932,0480.90350.58230.4961 NemoRetriever (3B v1) 38† ✓4.413,0720.91000.63320.5707 Nemo ColEmbed (3B v2) 39† ✓4.413,0720.91740.63380.5970 Vultron Core (4.5B) 40† ✓4.543200.92210.66120.6372 Ops ColQwen3 (4B) 41† ✓4.802,5600.91360.68660.6127 Nemotron (4B) 42† ✓4.802,5600.91620.64490.6142 ColNomic (7B) 43† ✓7.001280.89720.60250.5764 Qwen3-VL (8B) 44† 8.144,096–0.5829 Tomoro (8B) 45† ✓8.003200.90760.65400.6160 Vultron Prime (8B) 46† ✓8.393200.92080.68180.6472 Nemotron (8B) 47† ✓8.704,0960.92650.65160.6354 † Scores from MTEB. ‡ Results from our evaluation. 45 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 20|Visual retrieval aggregates. A dash marks a metric that was not reported. All results use the evaluation protocol in subsection 5.6. ViDoRe v3 ModelHeadnDCG@5 nDCG@10 Recall@5 Recall@10 Recall@100MAP NeoMME-260M 15 Late-interaction0.49910.52260.46370.56640.8326 0.4555 Dense0.36800.39070.34840.44190.7478 0.3274 NeoMME-800M 18 Late-interaction 0.53290.5560 0.49060.59700.8607 0.4878 Dense0.41450.43910.38770.49040.7937 0.3719 ViDoRe v2 ModelHeadnDCG@5 nDCG@10 Recall@5 Recall@10 Recall@100MAP ColModernVBERT 13 Late-interaction0.40680.4364–0.8475– NeoMME-260M 15 Late-interaction0.52180.5505–0.9232– Dense0.40750.43340.39380.49790.8091 0.3662 NeoMME-800M 18 Late-interaction 0.55910.5814–0.9259– Dense0.44750.4697 0.43470.53190.8434 0.4075 ViDoRe v1 ModelHeadnDCG@5 nDCG@10 Recall@5 Recall@10 Recall@100MAP ColModernVBERT 13 Late-interaction0.80580.8177–0.9805– NeoMME-260M 15 Late-interaction0.85980.86660.90110.92220.9752 0.8511 Dense0.75520.76910.83520.87760.9603 0.7378 NeoMME-800M 18 Late-interaction 0.87440.8801 0.91750.93530.9786 0.8641 Dense0.79930.80980.86850.90060.9681 0.7836 Resolution impact. Table 21 reports MeanMaxSim nDCG@10 under a downscale-only longest-side cap. The cost columns show the maximum for a square page at each cap; pages with smaller native dimensions or nonsquare aspect ratios use fewer vectors. Vector counts include image patches and structural positions. Raw storage assumes a 128-dimensional float32 late-interaction representation and excludes compression and index overhead. Table 21|Retrieval quality and uncompressed square-page representation size across image resolutions. All results use the evaluation protocol in subsection 5.6. Longest side (pixels) Square-page representationViDoRe v1ViDoRe v2ViDoRe v3 Vectors Raw float32 (MB)260M 800M 260M 800M 260M 800M 7686020.31 0.7231 0.7771 0.3897 0.4454 0.3340 0.4049 1,024 1,0580.54 0.8278 0.8533 0.4843 0.5251 0.4562 0.5029 1,536 2,3541.21 0.8633 0.8775 0.5419 0.5728 0.5134 0.5496 2,048 4,1622.13 0.8666 0.8801 0.5505 0.5814 0.5226 0.5560 ViDoRe v3 domains. Table 22 reports nDCG@10 for every task and a subset of the compared models. The NeoMME-800M row has a mean score of 0.5560. Task-level results. Table 23 reports task-level results for the NeoMME-260M late-interaction head on ViDoRe v3 and BEIR-15. 46 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 22 | ViDoRe v3 nDCG@10 results by task. ModelParams. ViDoRe v3 domains HR Fin.-EN Industrial Pharma.CS Energy Physics Fin.-FR <300M ColModernVBERT 13† 250M 0.1982 0.28760.15670.3219 0.3761 0.3251 0.2262 0.1977 ColSmol-256M 14† 256M 0.1646 0.23230.12870.2785 0.2880 0.2483 0.1614 0.1568 NeoMME-260M 15‡ 260M 0.5520 0.56390.3989 0.5963 0.6743 0.5922 0.4246 0.3782 300M to 1B ColSmol-500M 16 500M 0.2552 0.36970.19850.4126 0.5054 0.4052 0.3053 0.2656 Vultron Flash 17† 850M 0.5837 0.60600.4619 0.6309 0.7382 0.6114 0.4795 0.4079 NeoMME-800M 18‡ 800M 0.5946 0.60540.44550.6187 0.7026 0.6180 0.4482 0.4151 >1B ColQwen2 v1.0 33† 2.21B 0.4511 0.39030.38340.5221 0.6861 0.4856 0.4162 0.1997 ColPali v1.3 20† 2.92B 0.4480 0.34440.3557 0.5311 0.6528 0.4692 0.4174 0.2177 † Scores from MTEB. ‡ Results from our evaluation. Table 23|Selected task-level NeoMME-260M 15 late-interaction results. All results use the evaluation protocol in subsection 5.6. ViDoRe v3 TasknDCG@5 nDCG@10 Recall@100 Human resources0.52550.55200.8884 Finance, English0.53890.56390.8665 Industrial0.38830.39890.6530 Pharmaceuticals0.58220.59630.8657 Computer science 0.64640.67430.9409 Energy0.56510.59220.8938 Physics0.39770.42460.7903 Finance, French0.34850.37820.7621 Task macro mean0.49910.52260.8326 BEIR-15 TasknDCG@10 ArguAna0.4188 ClimateFEVER0.2400 CQADupstack0.3529 DBPedia0.3848 FEVER0.9176 FiQA-20180.3657 HotpotQA0.7222 MS MARCO0.3852 NFCorpus0.3173 NQ0.5336 Quora0.6945 SCIDOCS0.1514 SciFact0.7181 TREC-COVID0.7635 Touché-20200.3556 Task macro mean0.4881 47 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference C.3. Matryoshka results Matryoshka training trains each model to produce dense representations at several widths. Table 24 reports the retrieval quality at every trained width for NeoMME-260M and NeoMME-800M on ViDoRe v3, v2, and v1, using task-macro nDCG@10. Table 24|nDCG@10 results on ViDoRe v3, v2, and v1 at every trained Matryoshka representation size. All results use the evaluation protocol in subsection 5.6. Model detailsViDoRe (nDCG@10) ModelDense widthv3v2v1 NeoMME-260M 15 1280.36070.40630.7461 2560.37610.41330.7568 5120.38640.42280.7668 1,0240.39070.43340.7691 NeoMME-800M 18 1280.39740.43320.7786 2560.42170.45190.7999 5120.43180.46270.8074 1,0240.43520.45710.8089 1,7920.43910.46970.8098 C.4. Dual-head training ablation NeoMME-Retriever jointly trains a late-interaction head and a dense head on one shared backbone. We test whether optimizing both objectives improves either representation compared with optimizing its corresponding objective alone. The NeoMME-260M experiment compares late-interaction-only, dense-only, and joint training from the same initial checkpoint. All three runs use the same data seed, query order, negative draws, data mixture, schedule, and evaluation protocols. They differ only in the active losses. Table 25 reports the task-macro results, and Figure 15 shows nDCG@10 over training. Table 25 | Retrieval-head objective ablation for NeoMME-260M. Arrows show nDCG@10-point changes from the corresponding single-objective run. All results use the same evaluation protocol. ViDoRe v3BEIR-15 ArchitectureLateDenseLateDense LI head0.5088–0.4774– Dense head–0.3906–0.3240 Dual-head0.5226 0.3907 0.4881 0.3055 ↑+1.38 ↑+0.01 ↑+1.07 ↓−1.85 5000100001500020000 Training step 0.35 0.40 0.45 0.50 nDCG@10 ViDoRe v3 5000100001500020000 Training step 0.350 0.375 0.400 0.425 BEIR-15 HeadLate-interactionDense TrainingSingle-headDual-head HeadLate-interactionDense TrainingSingle-headDual-head Figure 15 | nDCG@10 during joint and single-objective NeoMME-260M retrieval training on ViDoRe v3 and BEIR-15. Joint training raises late-interaction task-macro nDCG@10 by 1.38 points on ViDoRe v3, while dense retrieval changes by 0.01 point. A paired analysis across all 14,514 judged queries estimates a 1.39-point late-interaction gain with a 95% confidence interval of 1.11 to 1.67 points (푝=10 −4 ). The paired dense change is 0.03 point with a confidence interval of−0.26 to 0.32 points (푝=0.81), so the experiment does not show a dense improvement. On BEIR-15, joint training raises late- interaction by 1.07 points and reduces dense retrieval by 1.85 points. These results are consistent with one-directional transfer from the dense objective to late-interaction on ViDoRe v3, not a general 48 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference improvement to both heads. Each run uses one training seed, so the paired intervals do not measure run-to-run variation. D. Retrieval efficiency D.1. Efficiency measurement details NeoMME-260MNeoMME-800M Vultron Flash 0.8B Qwen3-VL 2B ColModernVBERT ColSmol 256M ColQwen2.5 v0.2 Nemotron ColEmbed 4B v2 0 100 200 300 400 500 600 700 Optimized indexing throughput (pages/s) 768 1024 1536 2048 768 1024 1536 2048 768 1024 1344 capped 768 1024 1344 capped 2048 fi xed 2048 fi xed 756 capped 768 896 capped (a) NVIDIA H100 80GB HBM3. NeoMME-260MNeoMME-800M Qwen3-VL 2B ColModernVBERT ColSmol 256M ColQwen2.5 v0.2 0 10 20 30 40 Optimized indexing throughput (pages/s) 768 1024 1536 2048 768 1024 1536 2048 768 1024 1344 capped 2048 fi xed 2048 fi xed 756 capped (b) Apple M5 Pro. Figure 16|Image indexing throughput over the resolution grid on NVIDIA H100 and Apple M5 Pro. Timed path and benchmark inputs. We generate deterministic RGB squares and run each model’s official image processor before timing. This separates processor runtime from model and accelerator execution, while the processor still determines each model’s input shape and token count. We time device transfer, document encoding, output unpadding, conversion to contiguous float32 embeddings, and in-memory serialization. We do not time image processing, image decoding, storage input and output, compression, or index construction. On H100 and L40S, we use 128 images and 10 timed observations. On M5 Pro, we use 16 images and three observations, so those intervals are less precise. Each model and image size has a separately calibrated batch size. The tested square sizes are 768, 1,024, 1,536, and 2,048 pixels, plus a model’s native square when it is not already in the grid. We report each processor’s effective image sizes in Table 26. NeoMME encodes every requested square, while the external processors rescale each input to a fixed pixel or token budget. ColSmol and ColModernVBERT use fixed 2,048-pixel tiling at every supplied size. We omit repeated inputs that map to the same model input and do not measure complete page ingestion. The processor settings for ColModernVBERT and Nemotron ColEmbed VL 4B v2 follow their respective model reports (Teiletche et al., 2026; Moreira et al., 2026). Hardware and software setup. For CUDA measurements, we use bfloat16 andtorch.compile. The L40S measurements use four OpenMP threads. We benchmark NeoMME with the Transformers implementation at revisionc0f8594 48 and use FlashAttention 2 on CUDA. On M5 Pro, NeoMME and Qwen3-VL use compiled execution. The ColPali-engine models use eager execution because the PyTorch 2.13 MPS compiler cannot generate their graphs. For compiled runs, we set PyTorch 48 https://github.com/huggingface/transformers/commit/c0f8594234be10908e3e588f7a8e784d8cf bee33 49 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Dynamo’s recompile limit to 256 49 and exclude any run whose log reports that the limit was reached. Vultron Flash and Nemotron ColEmbed do not support MPS. Within each device, every model uses the same machine settings and benchmark implementation. Because the execution environments differ across devices, we compare models only within the same device. Table 26 | Processor behavior and effective image settings in the indexing benchmark. ModelProcessor behaviorSquares shown (pixels)Vectors at cap NeoMME-260M 15 Encodes each requested square768 / 1,024 / 1,536 / 2,0484,162 NeoMME-800M 18 Encodes each requested square768 / 1,024 / 1,536 / 2,0484,162 ColModernVBERT 13 Fixed 2,048-pixel tiling2,0481,149 ColSmol-256M 14 Fixed 2,048-pixel tiling2,0481,139 Vultron Flash 17 Caps at its visual-token budget768 / 1,024 / 1,3441,775 Qwen3-VL-Embedding-2B 50 Caps at its visual-token budget768 / 1,024 / 1,3441 ColQwen2.5-v0.2 19 Caps at 602,112 pixels756740 Nemotron ColEmbed VL 4B v2 51 Caps at 802,816 pixels768 / 896798 Table 27|Document-encoding throughput at each model’s largest effective square. All values use the same measurement protocol. Model Square (pixels) Vectors per page H100 L40S M5 Pro NeoMME-260M 15 2,0484,162 76.8 51.33.2 NeoMME-800M 18 2,0484,162 40.4 21.21.4 Vultron Flash 17 1,3441,775 35.2 20.2– Qwen3-VL-Embedding-2B1,3441 15.5 8.10.4 ColModernVBERT 13 2,0481,149 53.3 26.02.9 † ColSmol-256M 14 2,0481,139 50.7 24.82.6 † ColQwen2.5-v0.2 19 756740 42.7 16.11.1 † Nemotron ColEmbed VL 4B v2896798 31.3 16.8– Values are median pages per second. A dash means that the model does not support the hardware target. † means that the M5 Pro measurement uses eager execution because compilation could not generate the model graph. D.2. Query-encoding latency We measure batch-one query-encoding latency using a fixed set of 649 NanoBEIR queries. For each reported row, we run one complete warmup pass and then three measured passes. We time tokenization or query processing, the model forward pass, and transfer of the output embedding to host memory. For the CPU measurements, we use a SkyPilot-provisioned CPU-only Kubernetes pod with 128 CPU cores and 1 TB RAM, matching ModernVBERT’s query-latency setup. We set OMP_NUM_THREADS=128and use float32. The L40S and M5 Pro measurements use bfloat16. The indexing and query-encoding benchmarks use the same Transformers implementation for NeoMME. We report each device separately because the execution environments differ. The measurements cover query encoding only and exclude candidate generation, approximate search, MeanMaxSim scoring, and end-to-end request latency. NeoMME-260M is second only to ColModernVBERT on every device in Figure 17. Its mean latency is 21.0 ms on the L40S, 78.3 ms on the CPU host, and 15.9 ms on the M5 Pro. The other measured visual language model retrievers range from 30.6 to 52.5 ms, 113.8 to 899.1 ms, and 20.1 to 129.1 ms, 49 https://docs.pytorch.org/docs/2.11/user_guide/torch_compiler/torch.compiler_troublesho oting.html 50 Qwen/Qwen3-VL-Embedding-2B. 51 nvidia/nemotron-colembed-vl-4b-v2. 50 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference 050010001500 NeoMME-260M NeoMME-800M ColQwen2.5-v0.2 ColSmol-256M ColModernVBERT Qwen3-VL-Embedding-2B Nemotron-ColEmbed-VL-4B-v2 VultronRetrieverFlash-0.8B 78.3 214.7 899.1 113.8 62.0 416.4 CUDA only CUDA only CPU, 128 cores 0204060 21.0 25.4 52.5 44.5 13.6 30.6 35.7 36.1 NVIDIA L40S 0100200300400 15.9 31.8 129.1 20.1 8.2 51.7 CUDA only CUDA only Apple M5 Pro Batch-1 query-encoding latency (ms, mean with p95 whisker) Figure 17|Batch-one text query-encoding latency for visual retrievers on a 128-core CPU host, one NVIDIA L40S, and an Apple M5 Pro. Bars show the mean, and error bars extend to the 95th percentile. respectively. NeoMME-800M is also faster than the larger visual language model retrievers on every device. Across the tested CUDA, MPS, and CPU execution paths, NeoMME has low query-encoding latency. D.3. Late-Interaction Kernels (LIK) Method. LIK computes exact MaxSim scores in tiles and keeps only running token maxima instead of writing the full query-token by document-token similarity tensor to high-bandwidth memory (Lac and Wu, 2026). During training, it saves the winning document-token indices needed to route gradients. The LIK design guide 52 gives the full derivation and kernel diagrams. Measurement scope. Table 28 combines NeoMME-260M throughput from matched release runs with selected long-document kernel measurements. Table 29 selects end-to-end training results from the complete LIK benchmark report 53 , which records the full sweeps, software versions, baselines, and timing protocol. All measurements use one H100 80 GB SXM and float32 accumulation. Training uses bfloat16 inputs. The NeoMME-260M values are the mean and standard deviation from the 10 to 70 minute measurement window. ColQwen2 uses LoRA rank 32, gradient checkpointing, and ColPali training pages (Faysse et al., 2025). PyLate uses gradient checkpointing and MS MARCO triplets. Its batch-1,024 baseline usesscore_mini_batch_size=64, PyLate’s own chunking mitigation. Long-document measurements use float16 inputs with 16 queries, 32 documents, and 32 query tokens. The results cover memory and short-run step time, not convergence or downstream retrieval quality. 52 https://hcompai.github.io/late-interaction-kernels/how-it-works.html 53 https://github.com/hcompai/late-interaction-kernels/blob/main/docs/benchmarks.md 54 vidore/colqwen2-base. 55 lightonai/GTE-ModernColBERT-v1. 51 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference Table 28|Selected NeoMME-260M throughput and long-document MaxSim results. † results come from the LIK benchmark report, and ‡ results come from matched release runs. Training throughput NeoMME-260M 15‡ Configuration Tokens/s Std. dev. With LIK383,614 21,404 Without LIK 377,439 19,576 Long-document MaxSim † Document length Method Forward Backward Peak memory 4,096 Naive 0.65 ms 1.82 ms672 MB LIK0.12 ms 0.54 ms193 MB 8,192 NaiveOOMOOMOOM LIK0.18 ms 0.44 ms321 MB Table 29 | Selected LIK training results from the LIK benchmark report. SystemWorkloadMeasureBaselineLIK Improvement ColQwen2 54 MaxSim, batch 128Peak memory7.81 GiB61 MiB 130× lower Whole trainingMaximum batch641282× higher PyLate 55 Whole training, batch 1,024 Peak memory62.18 GiB 57.67 GiB7% lower Whole training, batch 1,024 Step time6.02 s4.81 s 1.25× faster E. Retrieval case demonstrations The three cases below show how NeoMME-Retriever 260M ranks pages on three ViDoRe v3 tasks (Loison et al., 2026). Each case reports one query, the top-3 retrieved pages of the full task corpus, their MeanMaxSim scores, and each page’s query relevance. ViDoRe v3 grades a page (1) fully relevant when it answers the query on its own, (2) critically relevant when it holds facts the answer needs but leaves the rest to other pages, or (3) unjudged when no annotator graded it for that query. Green frames mark relevant pages and red frames mark unjudged ones. 52 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference ViDoRe v3 retrieval case 1 Query (French) En quelle année les activités du réseau PES ont-elles été marquées par la célébration de son dixième anniversaire, et à travers quel segment la commémoration a-t-elle été diffusée ? Task:vidore/vidore_v3_hr (1,110 pages) Model: NeoMME-Retriever 260M (late-interaction) Rank 1 MeanMaxSim: 0.7205 Query relevance: Critically relevant Rank 2 MeanMaxSim: 0.7068 Query relevance: Critically relevant Rank 3 MeanMaxSim: 0.6989 Query relevance: Critically relevant ViDoRe v3 retrieval case 2 Query (English) Can crazing be caused by the paints used in a plastic panel paint restoration process? Task:vidore/vidore_v3_industrial (5,244 pages) Model: NeoMME-Retriever 260M (late-interaction) Rank 1 MeanMaxSim: 0.8502 Query relevance: Fully relevant Rank 2 MeanMaxSim: 0.7275 Query relevance: Unjudged Rank 3 MeanMaxSim: 0.7146 Query relevance: Unjudged 53 NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference ViDoRe v3 retrieval case 3 Query (German) Wie unterscheidet sich der Ansatz von ISMP zur Fehlermeldung von dem der FDA? Task:vidore/vidore_v3_pharmaceuticals(2,313 pages) Model: NeoMME-Retriever 260M (late- interaction) Rank 1 MeanMaxSim: 0.7850 Query relevance: Unjudged Rank 2 MeanMaxSim: 0.7804 Query relevance: Unjudged Rank 3 MeanMaxSim: 0.7572 Query relevance: Critically relevant 54