Paper deep dive
MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/9/2026, 8:39:07 AM
Summary
The paper introduces MetaEncoder, a bi-encoder model fine-tuned from the Muse-Glimmer 30B decoder for multimodal System One decision-making. It utilizes a natural language interface where both requests and candidates are expressed in text, images, or video. Trained via unidirectional contrastive learning, MetaEncoder scales to both closed-set (<256 options) and open-set (millions of candidates) scenarios, outperforming state-of-the-art multimodal encoders in decision-making tasks while identifying limitations in complex reasoning compared to decoder-based models.
Entities (10)
Relation Signals (8)
MetaEncoder → usesarchitecture → Bi-encoder
confidence 98% · MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning
MetaEncoder → finetunedfrom → Muse Glimmer 30B
confidence 95% · MetaEncoder fine-tunes a pre-trainedMuse-Glimmer 30B decoder into an instruction-following decision-making encoder.
MetaEncoder → trainedwith → Unidirectional Contrastive Learning
confidence 95% · MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment.
MetaEncoder → evaluatedon → MMLU
confidence 92% · MMLU (Hendrycks et al., 2021)... We evaluate 4,000 two-option rows.
MetaEncoder → evaluatedon → JevBench
confidence 92% · We conduct extensive evaluations across 11 benchmark suites... JevBench
MetaEncoder → evaluatedon → NanoBEIR
confidence 92% · NanoBEIR (AI, 2025): 13 multilingual text retrieval tasks... For NanoBEIR we report nDCG@10
Bi-encoder → supports → System One
confidence 90% · In this work we revisit bi-encoders and explore how far they can go as general-purpose, instruction-following encoders... repurposing general-purpose encoders as System One models
MetaEncoder → outperforms →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trained Muse-Glimmer 30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks.
Tags
Links
- Source: https://arxiv.org/abs/2610.11316v1
- Canonical: https://arxiv.org/abs/2610.11316v1
Trouble viewing inline? Open PDF directly →
Full Text
68,558 characters extracted from source content.
Expand or collapse full text
MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao 1 Meta AI System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trainedMuse-Glimmer30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks. Date: October 9, 2026 Hugging Face: https://huggingface.co/facebook/meta-encoder Github: https://github.com/facebook/meta-encoder-eval 1 Introduction There is a recent trend of repurposing general-purpose encoders as System One models for decision mak- ing (TypeSafe AI, 2026). An encoder-only model answers fast and in a constrained form. It returns a choice among candidates, together with a probability distribution over them, instead of generating free-form text. Many production decisions fit this form: routing a request to an agent, selecting a tool, classifying content against a policy, ranking items, picking the next action in an environment, or retrieving a memory. For these, a calibrated score over candidates from a single forward pass is more useful than a generated answer. Encoder models that score candidates are not a new topic. First, sentence-BERT (Reimers and Gurevych, 2019) and dense retrievers (Karpukhin et al., 2020) popularized bi-encoders. These embed a query and a candidate independently and compare them by vector similarity, and they were later scaled into universal text embedders (Wang et al., 2022; Xiao et al., 2024; Su et al., 2023; Wang et al., 2024). In vision, two-tower models such as CLIP (Radford et al., 2021) align images with texts through contrastive learning. More recently, multimodal embedders built on vision-language backbones cast classification, question answering, and grounding as retrieval over candidates (Jiang et al., 2024, 2025; Zhang et al., 2025a). Second, Cross- encoders (Nogueira and Cho, 2019; Nogueira et al., 2020) instead encode the query and the candidate together, trading efficiency for accuracy, and remain the standard choice for reranking. Finally, constrained decoders can also score candidates directly. The options are listed in the prompt under identifiers (A, B, C, . . . ), and the model picks the identifier with the highest next-token probability (Hendrycks et al., 2021; Robinson et al., 2023; Gangi Reddy et al., 2024). This model compares all options jointly, but the candidate cardinality is bounded by the context window and results are sensitive to option order (Pezeshkpour and Hruschka, 2024; Zheng et al., 2024). We review all above work in Section 2. When such a model is repurposed as a System One model, a common interface is a structured request object. 1 arXiv:2610.11316v1 [cs.CL] 8 Oct 2026 It contains separate fields for the context or state, the questions or instructions, the options and scoring criteria, and each task has its own schema. Schemas are precise, but they are rigid. In this work we revisit a fully natural-language interface. The prompt is free-form text, optionally with images or video, that describes the state, the instruction, and the criteria for the task. The output stays constrained to a set of candidate options, each of which is also described in natural language. Our goal is a general-purpose encoder that follows instructions and makes decisions, similar to prompting a decoder-only model. A key consideration in choosing the encoder architecture is balancing efficiency against scalability. Scalability here means both the number of candidates that can be scored and whether candidate representations can be cached. Cross-encoders and constrained decoders are powerful because the request attends to candidates directly. However, they cannot cache efficiently, so they do not scale to large candidate sets. Bi-encoders encode candidates independently of the request. This makes the candidate cardinality scale from a closed set of <256 options to millions of items. However, the price is that all interaction between request and candidate passes through a single vector. Whether that bottleneck can still support instruction generalization is an open question. In this work we revisit bi-encoders and explore how far they can go as general-purpose, instruction-following encoders. Empirically, we demonstrate that bi-encoders can also achieve high accuracy in closed-set decision-making by incorporating candidate options directly into the request’s prompt, supplying the model with information equivalent to that of a constrained decoder. We train MetaEncoder, a general-purpose multimodal encoder for instruction-following decision making, understanding and retrieval. MetaEncoder fine-tunes a pre-trainedMuse-Glimmer(Meta Superintelligence Lab, 2026) 30B decoder into an encoder. For training, MetaEncoder adopts unidirectional contrastive learning to align each request with its target candidate, differentiating closed- and open-set problems through the negative selection strategy (i.e., using a provided option list vs. gathering across all devices). For inference, the same model handles small closed sets of fewer than 256 options and open corpora of millions of candidates, which differ only in how the request prompt is constructed. MetaEncoder adopts the natural-language interface described above, so users interact with it similar to a decoder-only LLM. Both the request and the candidates can be multimodal. We evaluate MetaEncoder across 11 benchmark suites comprising 190 tasks. These span closed-set multimodal decision-making and understanding (JevBench (Standhartinger and Contributors, 2026), ImaJevBench (Mohit and Contributors, 2026), MMLU (Hendrycks et al., 2021), MMMU (Yue et al., 2024), VideoMMMU (Hu et al., 2025), NaturalBench (Li et al., 2024), and TempCompass (Liu et al., 2024)) as well as open-corpus retrieval (NanoBEIR (AI, 2025), MMEB-v3 image, video, and visdoc (Huang et al., 2026)). MetaEncoder outperforms state-of-the-art (SOTA) multimodal embedding models (Zhou et al., 2026; Li et al., 2026) on all decision-making tasks while achieving near-SOTA results on multimodal retrieval benchmarks which is beyond the scope of Jev models (Wortega and Contributors, 2026). On the other hand, we also identify the inherent limitations of bi-encoder architectures on complex decision-making tasks relative to decoders, where the lack of explicit reasoning capabilities remains the primary bottleneck. In summary, our contributions are: • We revisit a fully natural-language System One interface. The request is free-form text with auxiliary images and video, and the output is constrained to candidate options that are themselves described in natural language. •We train MetaEncoder, a bi-encoder fine-tuned from a 30B multimodal decoder with request-to-candidate contrastive learning. It scales from closed sets of fewer than 256 options to open corpora of millions of candidates. • We conduct an extensive evaluation over 11 benchmark suites and 190 tasks. It identifies both where a bi-encoder can act as a general-purpose, instruction-following encoder and where it does not yet generalize. 2 2 Related Work A general-purpose encoder maps an instruction, a context (text, image, video, etc.), and a set of candidates to a decision. Classification, retrieval routing, reranking, and judgment all fit this one interface. They differ only in where the candidate set comes from: a global corpus, a fixed label space, or a set defined by each example, as in Jev-style tasks where every row carries its own options and criteria. Early universal embedders focused on text retrieval and semantic similarity (Reimers and Gurevych, 2019; Karpukhin et al., 2020; Wang et al., 2022; Li et al., 2023; Xiao et al., 2024). Later work added instructions to the input (Su et al., 2023; Wang et al., 2024; Lee et al., 2025; Muennighoff et al., 2025; BehnamGhader et al., 2024) and extended the recipe to multimodal inputs using vision-language backbones (Jiang et al., 2024, 2025; Zhang et al., 2025a; Lin et al., 2025; Gu et al., 2026; Chen et al., 2026; Zhang et al., 2025a), and augmenting encoder input with reasoning (Cui et al., 2026, 2025; Lan et al., 2026; Cheng et al., 2026; Jiang et al., 2026). Benchmarks such as MTEB (Muennighoff et al., 2023) and MMEB (Jiang et al., 2025) already cast classification, VQA, and grounding as ranking over candidates. Broadly, a model built on a language-model backbone can score candidates in 3 ways. They differ in how much the prompt and the candidates interact, and therefore in cost, scalability, and expressiveness: (1) Bi-encoders. A bi-encoder embeds the prompt (instruction, state, context, etc.) and each candidate separately and scores them by cosine similarity. It is trained with a CLIP-style contrastive objective that pulls the prompt toward its gold candidate and pushes it away from in-batch or hard negatives (Radford et al., 2021; Jia et al., 2021; Zhai et al., 2023; van den Oord et al., 2018). This is the dominant design for universal embedders (Wang et al., 2022; Su et al., 2023; Wang et al., 2024; Lee et al., 2025; Jiang et al., 2025; Zhang et al., 2025a). Its main advantage is efficiency: since candidate embeddings do not depend on the prompt, they can be precomputed and indexed. ScoringKcandidates then costs a single prompt forward pass plusKdot products, and this scales to candidates of millions. However, the tradeoff lies in expressiveness: since all interaction happens through a single dot product, the prompt embedding must anticipate every candidate it might be compared against. This is a known weakness on fine-grained or compositional distinctions (Yuksekgonul et al., 2023; Thrush et al., 2022). It is especially limiting when what a label means is defined by the prompt itself, for example by row-specific criteria, since the candidate embedding cannot see those criteria. Late-interaction models such as ColBERT (Khattab and Zaharia, 2020) relax the single-vector bottleneck but still encode the two sides independently. (2) Cross-encoders. A cross-encoder concatenates the prompt with one candidate at a time. A shared transformer processes the pair jointly, and a scalar relevance score is read from a classification head or from the logit of a yes/true token (Nogueira and Cho, 2019; Nogueira et al., 2020; Ma et al., 2024; Zhang et al., 2025b). Full token-level attention between prompt and candidate makes cross-encoders consistently more accurate than bi-encoders. That is why they serve as the second-stage reranker in retrieve-then-rerank pipelines (Nogueira and Cho, 2019; Xiao et al., 2024; Cui et al., 2025). LLM-as-a-judge pointwise scoring follows the same pattern (Zheng et al., 2023). Two costs follow from the design: First, nothing can be cached: scoringKcandidates takesKfull forward passes over the shared prompt. Second, each candidate is scored in isolation, so the model never compares candidates directly. Scores must be calibrated across independent passes, and the model cannot use the structure of the option set (none of the above, mutually exclusive labels, near-duplicate options) to decide. (3) One-step constrained decoding. The third approach puts the instruction, the context, and all candidates into a single prompt. Each candidate is labeled with an identifier (A, B, . . . , Z), and a generative model is asked to emit one identifier token. The prediction is the argmax of the next-token distribution restricted to the valid identifiers. Since the vocabulary projection can be pruned to constrain outputs, the approach acts as a cheap encoder prefilling pass topped with a light linear decision layer. This is the standard multiple-choice format for evaluating LLMs (Hendrycks et al., 2021; Robinson et al., 2023). It underlies zero-shot classification through verbalizers (Schick and Schütze, 2021) and has been adopted for listwise reranking, where the first generated identifier’s logits act as the ranking score (Sun et al., 2023; Gangi Reddy et al., 2024). Because all candidates share one context, the model can compare them jointly. Inference takes a single forward pass regardless ofK, and no new head or training objective is needed beyond the language-modeling loss. On the other hand, the limitations come from treating candidates as tokens in a sequence. Accuracy is sensitive to option order and identifier choice (Pezeshkpour and Hruschka, 2024; Zheng et al., 2024). The candidate set is 3 bounded by the size of the identifier alphabet and by the context window, so the approach cannot support large candidates. Besides, the output is a distribution over positions, but not a reusable representation of the candidates. As a result, nothing can be indexed, cached across prompts, or reused for retrieval and clustering. Summary. The three designs trade off along one axis: how much the prompt and the candidates interact. Bi- encoders allow none interactions, which buys indexing and scale. Cross-encoders let the prompt interact with each candidate individually, which buys accuracy atO(K) cost and no representation caching. Constrained generation lets all candidates interact with each other and with the prompt in one pass, but it gives up open-ended candidate sets and reusable embeddings completely. In this work, we adopt a cachable bi-encoder design to explore its limit in general-purpose decision making. 3 MetaEncoder 3.1 Natural-language interface MetaEncoder scores a set of candidatesC=c 1 ,...,c K against a requestq. The request is a free-form prompt, optionally interleaved with images or video. It describes everything the decision depends on, such as the context and state; the instruction and question; the criteria which define what each option means. Each candidatec k is also expressed in natural language: a text description with optionally an image or video. Closed-set decision making and open-set retrieval therefore share one interface. They differ only in whetherC is a small set which can be added into the request as additional context, or a large corpus which has to be indexed in advance. For large open-set candidates, it is not possible to add them as additional context into the request due to context window restrictions. 3.2 Architecture MetaEncoder is a bi-encoder initialized from the pre-trainedMuse-Glimmer30B multimodal decoder. The same networkf θ encodes requests and candidates. Images and video are processed by the backbone’s native vision tower and interleaved with the text tokens. Last-token representation. We keep the backbone’s causal attention and take the final-layer hidden state of the last token as the representation, normalized to unit length: q = h last (q) ∥h last (q)∥ 2 ,c k = h last (c k ) ∥h last (c k )∥ 2 .(1) Under causal attention, the last token attends to every token in the input. It therefore sees the full context for the request, including state, instruction, criteria and options (for closed-set only), as well as all visual tokens, and can therefore summarize the request. Candidates in the prompt. For small closed sets, the candidate options can be listed in the request itself. Listing them in request gives the last token a global view, so that the model sees which alternatives exist and can contrast options. This recovers some of the joint-reasoning-over-candidate ability offered by cross-encoders and decoder-only models. However, for open-set retrieval, the candidate pool is too large to list in request anyway. To support both cases, during training we dropout the option list from the prompt for a random fraction of closed-set examples. The model then cannot purely rely on matching option text in the prompt and must still embed the request meaningfully. Scoring. Each request-candidate is scored by cosine similarity,s(q,c) = q ⊤ c. So a softmax overs(q,c k )/τ K k=1 gives a probability distribution over the candidate set. In this way, candidate embeddings do not depend on the request, so they can be computed once and cached. This scoring mechanism scales from a closed set of a few options to an index of millions of items. 3.3 Unidirectional Contrastive Training We train MetaEncoder to align each request with its correct candidate. Each example is a pair (q i ,c + i ). For a batch ofBrequests, the loss is an InfoNCE objective (van den Oord et al., 2018) applied in the 4 request-to-candidate direction only: L =− 1 B B X i=1 log exp s(q i ,c + i )/τ P c∈N(q i )∪c + i exp s(q i ,c)/τ ,(2) The negative set N(q i ) depends on the training task type: • Open-set (retrieval).N(q i ) contains the gathered candidates of the other requests in the global batch, plus hard negatives where available. Any in-batch candidate that is identical toc + i is removed from N(q i ). •Closed-set (decision making).N(q i ) is the request’s own option set, minus the gold option. The softmax therefore runs over exactly the candidates the task offers, so training matches evaluation, where each example is scored against its own options. 4 Experiment 4.1 Benchmarks We evaluate MetaEncoder on 11 benchmarks comprising 190 tasks (Table 1). We group them into three capabilities: decision making, multimodal understanding, and retrieval. They span both candidate regimes: in the closed-set regime, each request comes with its own small option set (fewer than 256 candidates). In the open-set regime, the request is scored against a shared larger candidate pool. Decision making. These benchmarks are recently established to evaluate System One models. Each example in the benchmarks consists of a task state, an instruction, decision criteria that define the labels, and a set of label options. •JEVBench (Standhartinger and Contributors, 2026) contains 231 text-only decision rows in three splits: original (72 rows), easy (48), and hard (111). Each row defines its own label set of 2 to 6 options, and the meaning of each label is given only by the row’s criteria. The splits contain 6, 13, and 80 distinct task instructions, respectively. •ImaJEV-Bench (Mohit and Contributors, 2026) extends the Jev format to multimodal states. It has 173 dev entries in three subsets. In visual, the answer must be read from an image, e.g., the current price of an item on a menu board. In text, the state is given entirely in text, e.g., checking an invoice’s lines against its stated total under a rule. In joint, a rule stated in text must be applied to values that appear only in the image, e.g., approving an order whose total is computed from the prices on the board. Multimodal understanding. We cast standard multiple-choice understanding benchmarks as candidate scoring. The question and any image or video form the request, and each answer option is embedded as a candidate. • MMLU (Hendrycks et al., 2021): text-only knowledge-related multi-choice questions. There are 57 subjects in total. • MMMU (Yue et al., 2024): college-level image-text multi-choice questions across 30 subjects. • Video-MMMU (Hu et al., 2025): multi-choice questions about professional lecture videos, organized into 3 tracks (Perception, Comprehension, Adaptation). •NaturalBench (Li et al., 2024): adversarially paired natural image-question examples that penalize answering from language priors alone. We evaluate 4,000 two-option rows. •TempCompass (Liu et al., 2024): fine-grained temporal perception in video. We evaluate its multiple- choice format, using the full option sentence as the candidate rather than the option letter. Retrieval. • NanoBEIR (AI, 2025): 13 multilingual text retrieval tasks of BEIR (Thakur et al., 2021). Each query is scored against the task’s full passage corpus. 5 CapabilityBenchmarkModalityCandidates Tasks Metric Decision making JEVBenchTextClosed3 Accuracy ImaJEV-BenchMultimodal Closed3 Accuracy Multimodal understanding MMLUTextClosed57 Hit@1 MMMUMultimodal Closed30 Hit@1 Video-MMMUMultimodal Closed3 Hit@1 NaturalBenchMultimodal Closed1 Hit@1 TempCompassMultimodal Closed1 Hit@1 Retrieval NanoBEIRTextOpen13 nDCG@10 MMEB-V3 Image Multimodal Open37 Hit@1 MMEB-V3 VideoMultimodal Open18 Hit@1 MMEB-V3 VisDoc Multimodal Open24 nDCG@5 Total190 Table 1 Evaluation benchmarks. Tasks counts the scored task units for each suite. Closed-set tasks score each request against its own option set (fewer than 256 candidates). Open-set tasks score against a shared and larger candidate pool. •MMEB-V3 (Huang et al., 2026): the image (37 tasks), video (18 tasks), and visual-document (24 tasks) partitions of the MMEB-V3. They cover multimodal classification, question answering, grounding, and retrieval, all formulated as ranking candidates for a multimodal query. 4.2 Evaluation All benchmarks are scored the same way. The request is rendered in natural language and encoded once with the last-token representation (Section 3.2), and each candidate is encoded independently. Candidates are then ranked by cosine similarity. The prompt used for each benchmark is provided in App. A. Closed set. Each request is scored only against its own options, and the prediction is the highest-scoring option. This small set of options is added to the request so that the model has a global picture for decision making. We report accuracy, which is Hit@1 over the request’s option set. Open set. Each request is scored against the full candidate pool of its task. The prompt contains only the state and task description, but not the options. For NanoBEIR we report nDCG@10 with linear gain. For MMEB-V3 we follow the benchmark’s protocol and report Hit@1 on the image and video partitions and nDCG@5 on visual documents. Results Table 2 reports the results MetaEncoder across all evaluation suites. As baselines we include the SOTA decision making model OpenJev-27B (Wortega and Contributors, 2026), and two SOTA multimodal retrieval models leading the MMEB benchmark, including WeMM-Embedding-9B (Zhou et al., 2026) and Qwen3-VL-Embedding-8B (Zhang et al., 2025b). The results of MetaEncoder demonstrates that a unified bi-encoder architecture can effectively master both mul- timodal decision-making and retrieval tasks. On decision-making and multimodal multiple-choice understand- ing benchmarks, MetaEncoder outperforms current SOTA retrieval models, including WeMM-Embedding-9B, which holds first place on the MMEB-V3 leaderboard (TIGER-Lab, 2026). However, MetaEncoder slightly lags behind OpenJev-27B in text-only decision-making. OpenJev-27B is a SOTA System One model that jointly encodes the request and all options using a single backbone paired with a decision scoring head. We attribute this performance gap to the structural differences between bi-encoders and full-attention models, reflecting the classic tradeoff between bi-encoder caching efficiency and cross-encoder ranking accuracy. However, OpenJev-27B does not have the ability for retrieval tasks nor multimodal tasks. 4.3 Analysis 4.3.1 Whether to Include Options in Prompt Bi-encoders encode requests and candidates independently, computing scores via inner product instead of cross-attention. This architecture is effective when candidates are semantically distinct, but multiple- 6 CategoryBenchmarkMetaEncoder OpenJev (27B) WeMM-Embedding (9B) Qwen3-VL-Embedding (8B) Decision making JevBench (orig)0.94440.98610.62500.6389 JevBench (easy)1.01.00.97920.9375 JevBench (hard)0.73870.76580.48650.4324 Jev Intelligence81.484.545.842.1 Multimodal decision making ImaJev (dev)0.8555–0.81170.7792 General knowledge understanding MMLU0.77670.86010.73250.6544 Multimodal understanding MMMU0.5788–0.56560.5062 VideoMMMU0.59–0.48460.4187 NaturalBench0.813–0.80170.7097 TempCompass0.7346–0.75270.7172 Multilingual text retrieval NanoBEIR0.6634–0.61750.6061 Multimodal retrieval MMEB V3 (image)0.7897–0.80990.7796 MMEB V3 (video)0.6486–0.74310.6715 MMEB V3 (visdoc)0.8149–0.83340.8236 Table 2 Benchmark Evaluation Results for MetaEncoder compared to SOTA System One decision makers and SOTA retrieval models. BenchmarkWithout options With options∆ MMLU (57 subjects)0.56450.7456+0.1811 MMMU (30 subjects)0.48750.5629+0.0754 MMMU-Pro (4 options)0.46990.5553+0.0854 MMMU-Pro (10 options)0.35030.4237+0.0734 Table 3 Effect of appending candidate option lists to the request prompt. Scores reflect unweighted subject means (MMLU, MMMU) or Hit@1 (MMMU-Pro). choice benchmarks frequently present options differing by a single attribute—such as"red, oxidative, slow contracting" versus"red, oxidative, fast contracting". To equip the encoder with a global picture of what candidates are available, one solution we adopt is to put the option list directly into the request if the cardinality is small. In this ablation study, we compare evaluation results with and without the options in the prompt. The ablation is conducted on MMLU and MMMU. Table 3 summarizes the main results, with detailed subject breakdowns provided in Tables 9 and 10. We find that appending option lists to request yields substantial, uniform gains: performance improves across 56 of 57 MMLU subjects and 27 of 30 MMMU subjects. These gains are more significant on text-only benchmarks than on multimodal ones. Recommendation: Always append option list to the request for smaller candidate pool. 4.3.2 Candidate Format in Target In bi-encoders the candidate string is the entire signal on the candidate side and the only interaction between a request-candidate pair is an inner product between the two vectors. In this section, we measures the impact of the candidate-side format to the final result. We compare 3 variants of candidate renderings on MMLU and MMMU. In all variants, the request is always the options-in-prompt form (option list and its letters are enumerated in the request); only the candidate strings change: RenderingExample C0 description only"red, oxidative, slow contracting." C1 bare label"C" C2 label and description "C. red, oxidative, slow contracting." The results of the three variants are in Table 4. In summary, including both labels and descriptions (C2) 7 BenchmarkC0C1C2∆ C1 ∆ C2 MMLU (57 subjects) 0.7456 0.6512 0.7767−0.0944+0.0311 MMMU (30 subjects) 0.5629 0.4983 0.5788−0.0646+0.0159 MMMU-Pro0.4916 0.3803 0.5042−0.1113+0.0126 Table 4 Candidate rendering, with the request held fixed at the options-in-prompt form. C0 description only. C1: bare label. C2: label and description. Hit@1 is the metric over all rows; ∆ is against C0. C2 | label + description σ = 0.0014 C0 | description only σ = 0.0030 C1 | bare label σ = 0.0028 0.600.650.700.750.80 Figure 1 10 random permutations of option order across three variants on MMLU. Standard deviation is measured. works the best. Meanwhile, using only positional labels (C1) in the candidate side regresses all benchmarks. Despite so, it should be noted that using label only (C1) still reaches 0.6512 on MMLU against a chance floor of 0.2500. This is a clean demonstration that the model resolves a pointer: the letter"C"carries no information about the option semantics and yet it recovers most of the task. However, resolution also degrades as the option set grows: on MMMU the C1 deficit widens monotonically with the number of options, from +0.0281 at three options to−0.0792 at four,−0.1344 at five and−0.1725 at ten, against the C0 baseline. A pointer is harder to resolve more options. Recommendation: Render each candidate as its identifier and its description. 4.3.3 Sensitivity to candidate order The previous sections analyze the request and candidate representations which can include labels (A...Z). A natural question to ask is the robustness of option ordering in both the request and candidate sides. On MMLU, we permute the option order for each candidate which is reflected inside the request and also at the candidate side for C1 and C2. For C0, it affects request only since the candidate is label free. We run 10 independent permutations for that the candidate orders are shuffle randomly in each permutation. We then visualize the score distribution for C0, C1 and C2. Figure 1 shows the results of all 10 runs. In summary, candidate order barely matters for all variants. Across the 10 permutations, none of the three standard deviation exceeds 0.003. Conclusion The bi-encoder based model is robust to option orders in the prompt and at the target side. 4.3.4 Video understanding All above analysis focuses on text and image benchmarks. In this section we deep analyze MetaEncoder video understanding on Video MMMU benchmark, focusing on the impact of the number of frames for encoding, as well as the rendering of the request and of the candidates. Setup. We follow the official Video-MMMU protocol: each of the three tracks (Perception, Comprehension, Adaptation) is scored over its 300 questions and the overall score is their mean. Frames are sampled uniformly from 64 extracted per clip. 8 Candidate Request(A) text A. text text(A)A Default instruction, (A) list0.5800 0.5656 0.4933 0.4789 0.4556 Select the correct option., (A) list 0.5711 0.5667 0.5011 0.5178 0.4789 Select the correct option., A. list0.5544 0.5556 0.4989 0.4100 0.3744 Table 5 Video-MMMU overall accuracy (official protocol, 32 frames) by request and candidate rendering. Rows vary the request: the default instruction (Appendix A.3) orSelect the correct option., with options listed as(A) . . .or A. . . . . Columns vary the candidate. 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 816324864 Frames per clip (log scale) Accuracy Perception 0.753 Overall 0.590 Comprehension 0.580 Adaptation 0.437 Figure 2 Video-MMMU accuracy as a function of the number of frames sampled per clip, with the request and candidate rendering fixed at the defaults of Appendix A.3. Accuracy follows the official protocol: each track is scored over its 300 questions (open-ended Adaptation questions count as wrong) and Overall is the mean of the three tracks. Doubling from 8 to 16 and from 16 to 32 frames adds 0.021 and 0.027 overall. More frames help. As shown in Figure 2, accuracy rises monotonically with the frame budget, from 0.5322 at 8 frames to 0.5800 at 32 and 0.5900 at 64. Most of the gain arrives by 32 frames. Repeated runs of the same configuration agree within 0.0011 standard deviation, so the trend is well above run-to-run noise. Because video tokens grow linearly with the number of frames, the 64-frame setting costs roughly eight times for a 0.058 gain. The gain is perceptual. The improvement is concentrated where the answer is on screen. Perception rises by 0.127 (from 0.6267 to 0.7533) and Comprehension by 0.070, while Adaptation falls slightly (0.4600 to 0.4367). By question type, OCR improves from 0.6173 to 0.7509; concept and problem-solving comprehension improve by 0.064 and 0.078. However, the two Adaptation types do not improve (problem-solving adaptation regressed from 0.4071 to 0.4000; case-study analysis regressed from 0.5827 to 0.5396). This indicates that for tasks requiring more complex reasoning, the performance bottleneck of encoder-only architectures becomes more pronounced. Request and candidate representations also matter for video understanding. Table 5 varies the request and candidate rendering at 32 frames. The pattern matches the closed-set findings of Sections 4.3.1 and 4.3.2. Removing the option list from the request leads to a large regression. With the list present, the best candidate is the option’s letter together with its text. Dropping the letter costs 0.087, because the letter is what binds the candidate to its entry in the request, and dropping the text costs 0.101 to 0.124, because a bare letter carries no content to match. Conclusion. Increasing frame count enhances video understanding, significantly boosting perception and modestly improving comprehension. However, it yields no benefit for more complex adaptation tasks. Prompt 9 BenchmarkDecoder Encoder Agreement MMLU0.77430.77400.8195 MMMU0.49620.57300.7154 MMMU-Pro0.44410.50390.7221 4 options0.49760.56010.7200 10 options0.39060.44770.7242 Video-MMMU0.52670.54150.7531 Perception0.64000.6333- Comprehension0.46670.5000- Adaptation0.46950.4875- Table 6 Encoder versus constrained decoder. Video uses 8 frames. Agreement denotes the fraction of instances where both models are simultaneously correct or incorrect. rendering findings transfer directly from text and image domains to video: options should be explicitly included in request, and each candidate should use both its label and full content. 4.3.5 Encoder versus Constrained Decoder This section compares the MetaEncoder architecture against a constrained decoder baseline. For comparison, we use the pre-trainedMuse-Glimmerbackbone as a decoder, prompting it to generate an option label in a single constrained decoding step. Like the encoder setup, the decoder processes the request, including all candidate options, during prompt pre-filling. It then uses its output projection layer (restricted to valid option tokens A–Z) to emit logit scores for each option. Because the projection head is pruned to these candidate tokens, this approach is as computationally efficient as an encoder with a label projection head. Theoretically, this allows the decoder to evaluate all options jointly using its pre-trained next-token prediction capability while having full attention to request and options. Table 6 presents the comparison. On MMLU, MetaEncoder and the constrained decoder perform comparably. On MMMU and MMMU-Pro and Video-MMMU, MetaEncoder all outperforms the decoder. We attribute the encoder’s better performance to three factors: 1) MetaEncoder is fine-tuned on contrastive data targeting this exact scoring task, whereas the decoder is trained for general next-token prediction; 2) MetaEncoder incorporates option candidates directly into the prompt, giving it full context parity with the decoder; and 3) constrained single-token decoding deprives the decoder of intermediate chain-of-thought (CoT) reasoning. 4.3.6 Missing Reasoning Ability To test the third hypothesis above, we measured the total probability mass ofMuse-Glimmerassigned to valid label tokens prior to the argmax step. Candidate letters account for 0.8679 of the probability mass on MMLU and 0.8434 on Video-MMMU, but only 0.7424 on MMMU, indicating that the decoder is reluctant to emit a direct option letter on complex reasoning tasks without prior step-by-step generation. To confirm this hypothesis, we disabled constrained decoding for theMuse-Glimmerdecoder, allowing unconstrained generation. As shown in Figure 3, accuracy on MMMU jumped from 0.4962 to 0.7202. This improvement indicates that the primary performance bottleneck lies not in the scoring mechanism itself, but in whether intermediate reasoning is provided. Neither MetaEncoder nor the constrained decoder performs explicit reasoning, resulting in a ∼ 20% gap compared to an unconstrained decoder on MMMU. As shown in Figure 3, we run further experiments on MMMU to show that providing pre-generated reasoning traces can improve MetaEncoder as well. Prepending CoT reasoning generated byMuse-Glimmerto the encoder prompt boosts MetaEncoder’s MMMU accuracy from 0.5730 to 0.7155. Because the generated CoT explicitly reveals the answer label in 93.7% of cases, we also evaluated a setting where final labels were truncated from the CoT to eliminate direct label leakage. Even without label leakage, MetaEncoder reaches 0.6871 accuracy on MMMU. This demonstrates that intermediate reasoning is the primary bottleneck for scoring models regardless of architecture, and incorporating reasoning traces effectively bridges this gap. 10 Decoder (Muse-Glimmer-30B) MetaEncoder 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.4962 0.5730 0.6871 0.7202 0.7155 MMMU Accuracy Direct (No CoT) + Truncated CoT+ Full CoT Figure 3 Impact of Chain-of-Thought (CoT) reasoning on MMMU accuracy. Disabling constrained single-token decoding for the Decoder or prepending CoT traces to MetaEncoder substantially bridges the performance gap (∼ 20%). 5 Conclusion We present MetaEncoder, a multimodal bi-encoder that exposes a single natural language interface for closed-set decision-making, understanding, and open-set retrieval tasks. By utilizing a bi-encoder architecture, MetaEncoder enables efficient representation caching. Furthermore, we demonstrate that appending candidate options directly to the prompt allows bi-encoders to execute decision-making tasks effectively, narrowing the performance gap with cross-encoders and constrained decoders. Comprehensive evaluation across 11 benchmarks spanning multimodal decision-making, understanding, and retrieval demonstrates competitive performance. Notably, candidate-pool size poses minimal difficulty for MetaEncoder: retrieving from a pool of 1,000 candidates is not harder for MetaEncoder than resolving a four-way multiple-choice question whose options differ by only a single attribute. Instead, task difficulty is dictated by whether an answer possesses clear semantic boundaries or requires pre-retrieval reasoning. Consequently, while MetaEncoder excels at reading, perception, classification, and single-step judgment, it shows weaker performance on complex comprehension and adaptation. To resolve this decision-accuracy bottleneck, reasoning capabilities have to be injected by either leveraging a decoder’s CoT as additional context, or making the encoder model be able to reason directly. 11 References Sionic AI. Nano-beir: A multilingual information retrieval benchmark with quality-enhanced queries, 2025.https: //huggingface.co/sionic-ai. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2Vec: Large language models are secretly powerful text encoders. In Conference on Language Modeling (COLM), 2024. Haonan Chen, Chu Li, Zhicheng Wang, Yuanwei Liu, Yuanjiang Wang, Shaohua Jiang, and Zhicheng Dou. Douyin multimodal embedding model technical report. arXiv preprint arXiv:2608.02148, 2026. Jianpeng Cheng, Xian Wu, Jiangfan Zhang, Wentao Bao, Chaitanya Ahuja, Shlok Kumar Mishra, Hanchao Yu, Yang Gao, Fan Xia, Qi Guo, et al. Tte-flash: Accelerating reasoning-based multimodal representations via think-then-embed tokens. arXiv preprint arXiv:2605.16638, 2026. Xuanming Cui, Hong-You Chen, Hao Yu, Hao Yuan, Zihao Wang, Shlok Kumar Mishra, Hanchao Yu, Yonghuan Yang, Jun Xiao, Ser-Nam Lim, et al. Reason to contrast: A cascaded multimodal retrieval framework. arXiv preprint arXiv:2602.23369, 2025. Xuanming Cui, Jianpeng Cheng, Hong-you Chen, Satya Narayan Shukla, Abhijeet Awasthi, Xichen Pan, Chaitanya Ahuja, Shlok Mishra, Taipeng Tian, Qi Guo, et al. Think then embed: Generative context improves multimodal embedding. In International Conference on Learning Representations, volume 2026, pages 2690–2709, 2026. Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. FIRST: Faster improved listwise reranking with single token decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8642–8652, 2024. Tiancheng Gu, Kaicheng Yang, Kaichen Zhang, Xiang An, Ziyong Feng, Yueyi Zhang, Weidong Cai, Jiankang Deng, and Lidong Bing. Unime-v2: Mllm-as-a-judge for universal multimodal embedding learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 21378–21386, 2026. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021. Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-MMMU: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025. Haohang Huang, Xuan Lu, Mingyi Su, Xuan Zhang, Ziyan Jiang, Ping Nie, Kai Zou, Tomas Pfister, Wenhu Chen, Wei Zhang, et al. Mmeb-v3: Measuring the performance gaps of omni-modality embedding models. arXiv preprint arXiv:2604.23321, 2026. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021. Haonan Jiang, Yuji Wang, Yongjie Zhu, Xin Lu, Wenyu Qin, Meng Wang, Pengfei Wan, and Yansong Tang. Embed-rl: Reinforcement learning for reasoning-driven multimodal embeddings. In European Conference on Computer Vision, pages 39–56. Springer, 2026. Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-V: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580, 2024. Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. VLM2Vec: Training vision-language models for massive multimodal embedding tasks. In International Conference on Learning Representations (ICLR), 2025. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, 2020. Omar Khattab and Matei Zaharia. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 39–48, 2020. 12 Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. Ume-r1: Exploring reasoning-driven generative multimodal embeddings. In International Conference on Learning Representations, volume 2026, pages 115779– 115806, 2026. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NV-Embed: Improved techniques for training LLMs as generalist embedding models. In International Conference on Learning Representations (ICLR), 2025. Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. NaturalBench: Evaluating vision-language models on natural adversarial samples. In Advances in Neural Information Processing Systems (NeurIPS), 2024. Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, et al. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720, 2026. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023. Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms. In International Conference on Learning Representations, volume 2025, pages 44215–44234, 2025. Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do video LLMs really understand videos? In Findings of the Association for Computational Linguistics: ACL 2024, pages 8731–8772, 2024. Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning LLaMA for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421–2425, 2024. Meta Superintelligence Lab. Muse glimmer-30b model card.https://huggingface.co/meta-models/Muse-Glimmer-30B, august 2026. Hugging Face Model Repository. Mohit and Contributors. Imajevbench: Benchmark for typed decisions from visual evidence and rules.https: //huggingface.co/datasets/mohit67890/imajev-bench, september 2026. Dataset and Multimodal Evaluation Suite. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, 2023. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In International Conference on Learning Representations (ICLR), 2025. Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085, 2019. Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, 2020. Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, 2024. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, 2019. Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations (ICLR), 2023. Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, 2021. 13 Florian Standhartinger and Contributors. Jevbench: Metamorphic coherence testing for typed probabilistic decision models.https://github.com/fstandhartinger/jevbench, september 2026. Benchmark Repository and Evaluation Suite. Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction-finetuned text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1102–1121, 2023. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14918–14937, 2023. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021. Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5228–5238, 2022. TIGER-Lab. Massive multimodal embedding benchmark (mmeb) leaderboard.https://huggingface.co/spaces/ TIGER-Lab/MMEB-Leaderboard, 2026. Accessed: 2026-10-07. TypeSafe AI. Introducing system one models & jev.https://typesafe.ai, september 2026. Technical Announcement and Early Access Release. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, 2024. Alex Wortega and Contributors. Openjev: Qwen3.5-based system one cross-encoder decision model.https:// huggingface.co/openjev/openjev, 2026. Accessed: 2026-10-07. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 641–649, 2024. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9556–9567, 2024. Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations (ICLR), 2023. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952, 2023. Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Bridging modalities: Improving universal multimodal retrieval by multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9274–9285, 2025a. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025b. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations (ICLR), 2024. 14 Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, pages 46595–46623, 2023. Junjie Zhou, Ke Mei, Lei Li, Tianyi Wang, Fengyun Rao, and Jing Lyu. Wemm-embedding: Wechat multi-modal embedding technical report. arXiv preprint arXiv:2608.24053, 2026. 15 Appendix A Prompts This appendix gives the exact prompt used for every evaluation suite, aligning with MetaEncoder’s natural language interface. A.1 Common template All requests and candidates are assembled into a two-turn chat similar to a instruction-following decoder: system: Represent the user’s input. user: request (plus any image or video content parts) The system turn is a constant, always applied to both prompt and candidate encodings. The request goes to the user turn, which includes user instructions, context descriptions, criteria etc. Media are attached in the MetaEncoder input interface. The chat template is applied withadd_generation_prompt=False, so the sequence ends at the user turn’s<|eot|>, which is what last-token pooling reads. Candidates are encoded with the same template above, unless noted below, carry only their own text. A.2 Decision making JEVBench and ImaJEV-Bench. Each row carries its own task definition and label ontology, rendered by a single shared function: state Task: instruction Criteria: - label: meaning of the label Options: a, b, c The criteria block is what makes these rows transferable: the label vocabulary is defined in the prompt, so a row whose label set the model has never seen is still answerable. Boolean fields are rendered asyes: description andno: descriptionrather thanTrueandFalse, since a bare"True"is an opaque token that carries no information about what the row is asking. S1MB (Additional Eval). The instruction is taken from each decision’sinstructions_jsonfield and the case from state_json, rendered askey: valuelines. Candidates are the natural-languagecriteria[].description_json strings. The source’s criterion identifiers ("B","option_3","0") are used only to resolve which candidate is correct and never reach the model. One row is emitted per (case, decision) pair, because a single case may carry several decisions with different answer spaces. An agentic control example: Choose one permitted tactical maneuver using the measured scene. The simulator retains its collision reflex and flight controller. adapter: drone observation: "control_policy": "braking_deceleration_mps2": 3, . . . with candidates such as"climb: Climb over an obstacle within verified clearance"and"gap_right: Use the observed safe gap on the right". For ordinal (score) decisions each criterion additionally carries a numericvalue, which is retained in a sidecar so that predictions can receive graded credit: constraint: Do not infer additional numbers from associations question: How much of the claim does the source support? unit: Explicit numerical claims 16 claim: Revenue rose 20% and profit rose 20% source: Revenue rose 20%. Profit figures were not disclosed. with candidates"Source mentions neither revenue nor profit"(value 0),"Mentions revenue or profit but confirms neither claimed number"(1),"Confirms exactly one claimed number"(2) and"Confirms both claimed numbers" (3). A.3 Multimodal understanding Video-MMMU and TempCompass. These three share one template, with the options rendered into the request: Given a video and a question, select the most accurate answer from the provided candidates. Return only the exact text of your chosen answer. Question:question Options: (A) option (B) option . . . Candidates are the full option sentence together with its letter marker, for example"(A) To get her to trust him". A verbatim MVBench request: Given a video and a question, select the most accurate answer from the provided candidates. Return only the exact text of your chosen answer. Question:Why did Castle dress like a fairy when he was speaking to Emily? Options: (A) To get her to trust him (B) He secretly loved fairies . . . TempCompass Options arrive letter-prefixed in the source ("B. A man is dunking a basketball."). The gold is resolved by matching that prefix, after which the prefix is stripped, so the candidate is the sentence alone. Video inputs use 8 uniformly sampled frames drawn from 64 extracted per clip. Video-MMMU Adaptation track additionally references a still image. Frames are passed through a separate video channel and the still as its own image content part, so that both reach the model, and the<image 1> placeholder in the question text is reduced to the word image. MMLU and MMMU. The base request is: Answer the multiple-choice question by selecting the correct option. question: question In the options-in-prompt variant, marked∗in the results tables, the alternatives are additionally rendered into the request: Options: A. option B. option . . . Candidates are the full option text in both variants, never the letter. MMMU’s<image N>placeholders are reduced to the wordimage, since the referenced images travel as content parts; rows whose candidates consist only of an image reference are excluded, because such a candidate carries no matchable text. A verbatim MMLU ∗ request: Answer the multiple-choice question by selecting the correct option. question: Type I muscle fibres have the following characteristics: Options: A. white, glycolytic, slow contracting. B. red, oxidative, fast contracting. C. red, oxidative, slow contracting. D. white, oxidative, slow contracting. NaturalBench. The dataset’s own yes/no question forms the request, with the image attached as a content part; the candidates areYesandNo. Each group pairs two images with two questions whose answers flip between the images, so answering from the question alone cannot succeed on all four cells. 17 A.4 Retrieval NanoBEIR. Queries carry the task instruction for their subset; passages are encoded without an instruction. The instructions are, verbatim: SubsetQuery instruction NanoMSMARCOGiven a web search query, retrieve relevant passages that answer the query NanoNQGiven a question, retrieve passages that answer the question NanoHotpotQAGiven a multi-hop question, retrieve documents that can help answer the question NanoFEVERGiven a claim, retrieve documents that support or refute the claim NanoNFCorpusGiven a question, retrieve relevant documents that best answer the question NanoFiQA2018Given a financial question, retrieve user replies that best answer the question NanoQuoraRetrieval Given a question, retrieve questions that are semantically equivalent NanoArguAnaGiven a claim, find documents that refute the claim NanoClimateFEVER Given a claim about climate change, retrieve documents that support or refute the claim NanoDBPediaGiven a query, retrieve relevant entity descriptions from DBPedia NanoSCIDOCSGiven a scientific paper title, retrieve paper abstracts that are cited by the given paper NanoSciFactGiven a scientific claim, retrieve documents that support or refute the claim NanoTouche2020Given a question, retrieve detailed and persuasive arguments that answer the question MMEB-V3. For MMEB-V3 each task ships its own query instruction in the benchmark metadata, and we use it directly in prompt. The request is assembled asqry_instuctionfollowed by the row’sqry_text, with the image attached as a content part; candidates are built fromtgt_textandtgt_img_path. Instructions are per task, and their form follows the task type: classification tasks ask for a representation of the image, retrieval tasks describe what should be retrieved, and grounding tasks describe the region to select. Taskqry_inst ImageNet-1KRepresent the given image for classification Country211Identify the country depicted in the image DocVQARepresent the given image with the following question: RefCOCOSelect the portion of the image that follows the language expressions. OVENRetrieve a Wikipedia image-description pair that provides evidence for the question of this image: FashionIQFind an image to match the fashion image and style note: MSCOCO_t2iFind me an everyday image that matches the given caption: VisDialRepresent the given dialogue about an image, which is used for image retrieval: 18 TaskTypeAcc open_jev__browser_control_v1choice1 laya__enron_spamnoul0.96 miqachoice0.96 open_jev__amount_extraction_control_v1noul0.96 open_jev__mailroom_control_v1noul0.95 winobiaschoice0.94 followir_robust04noul0.935 workflow_controls__customer_servicenoul0.93 generalization_contextual_choicechoice0.93 open_jev__silent_failure_control_v1noul0.92 hansnoul0.91 go_emotionsnoul0.9085 workflow_controls__invoice_processingnoul0.9 workflow_controls__security_incidentsnoul0.9 hatechecknoul0.89 implinoul0.89 workflow_controls__agent_trace_observabilitynoul0.89 laya__ag_newschoice0.88 sdoh_nlinoul0.88 open_jev__painting_geometry_v1noul0.87 generalization_diverse_choicechoice0.87 defeasible_nlichoice0.86 open_jev__context_retention_control_v1noul0.86 creaknoul0.85 dbpediachoice0.85 quartzchoice0.85 snlichoice0.85 synth_rel__NanoFEVERscore0.84 synth_rel__NanoSciFactscore0.8387 pawsnoul0.83 scitailnoul0.83 arcchoice0.81 babi_nlinoul0.81 open_jev__customer_control_v1score0.81 open_jev__vizdoom_basic_v1score0.803 synth_rel__NanoNQscore0.8025 synth_rel__nanocoirscore0.8 clincchoice0.7748 ethosnoul0.77 open_jev__phone_extraction_control_v1choice0.77 claddernoul0.76 followir_news21noul0.76 synth_rel__NanoHotpotQAscore0.755 lonlichoice0.75 gsm8kchoice0.74 open_jev__ir_control_v1score0.74 massivechoice0.735 openbookqachoice0.73 synth_rel__NanoSCIDOCSscore0.7288 synth_rel__NanoNFCorpusscore0.7275 followir_core17noul0.725 few_nerdchoice0.7235 civil_commentsnoul0.7229 hwu64choice0.72 mtopchoice0.715 canttalknoul0.71 generalization_diverse_noulnoul0.71 aegis2noul0.7027 synth_rel__NanoMSMARCOscore0.6925 open_jev__sponsor_segment_control_v1choice0.69 synth_rel__NanoClimateFEVERscore0.69 synth_rel__NanoArguAnascore0.6725 open_jev__citation_control_v1choice0.67 open_jev__email_selection_control_v1noul0.67 synth_rel__NanoQuoraRetrievalscore0.6687 arctchoice0.66 open_jev__entity_alignment_control_v1noul0.66 generalization_contextual_noulnoul0.66 synth_rel__NanoDBPediascore0.6462 open_jev__reasoning_control_v1score0.64 ruletakernoul0.64 sconechoice0.64 ethicsnoul0.63 synth_rel__NanoFiQA2018score0.6275 lexcompnoul0.62 planenoul0.59 aqua_ratchoice0.58 bbqchoice0.58 scicitechoice0.58 crows_pairschoice0.57 robust_lrchoice0.57 contract_nlichoice0.5676 gretel_piichoice0.5666 banking77choice0.56 open_jev__snake_v1noul0.56 laya__typed_decisionsscore0.556 logical_entailmentnoul0.55 generalization_diverse_scorescore0.55 temporal_nlichoice0.55 generalization_contextual_scorescore0.53 tracienoul0.53 h_rlhfchoice0.52 winowhynoul0.52 sgdchoice0.51 nlsatnoul0.5 ud_ewtchoice0.4944 qaspernoul0.49 synth_rel__NanoTouche2020score0.4809 boardgameqachoice0.48 open_jev__drone_control_v1noul0.44 fol_nlichoice0.43 19 TaskTypeAcc wiqachoice0.42 corr2causechoice0.41 open_jev__tic_tac_toe_v1choice0.38 spartqachoice0.37 patent_similarityscore0.36 escichoice0.33 poem_sentimentchoice0.33 stepgamechoice0.28 argument_qualitynoul0.27 Table 7 Full results on S1MB (110 tasks). TaskScore scene_transition0.925 unexpected_action0.775 moving_attribute0.755 object_existence0.695 character_order0.675 action_prediction0.645 object_interaction0.615 action_sequence0.585 state_change0.565 fine_grained_pose0.56 action_antonym0.55 fine_grained_action0.52 moving_direction0.48 episodic_reasoning0.465 action_localization0.425 counterfactual_inference0.395 object_shuffle0.375 action_count0.37 moving_count0.35 egocentric_navigation0.35 Table 8 Full results on MVBench (20 tasks). SubjectScore (options not in prompt)Score (options in prompt) security_studies0.29390.702 international_law0.4050.7769 human_sexuality0.47330.8168 nutrition0.43790.7778 moral_disputes0.43350.7139 jurisprudence0.49070.7685 high_school_microeconomics0.56720.8361 sociology0.5970.8607 high_school_physics0.47020.7285 college_biology0.60420.8542 econometrics0.40350.6404 high_school_chemistry0.44830.6798 high_school_government_and_politics0.67880.9067 clinical_knowledge0.54340.766 high_school_statistics0.54170.7639 computer_security0.560.78 professional_accounting0.48230.7021 high_school_geography0.67170.8889 astronomy0.59210.8092 high_school_macroeconomics0.5410.7564 electrical_engineering0.46210.6759 us_foreign_policy0.580.79 high_school_computer_science0.630.84 high_school_world_history0.6540.8565 abstract_algebra0.50.7 philosophy0.49520.6945 high_school_biology0.66770.8645 high_school_us_history0.70590.902 high_school_european_history0.60610.8 college_mathematics0.450.64 professional_law0.39370.5815 professional_psychology0.54740.732 management0.64080.8252 public_relations0.52730.7091 medical_genetics0.70.88 college_medicine0.55490.7283 prehistory0.6080.7685 college_computer_science0.530.69 marketing0.74790.906 high_school_psychology0.73580.8936 conceptual_physics0.62980.766 human_aging0.56050.6951 professional_medicine0.71320.8456 college_physics0.67650.8039 machine_learning0.55360.6786 moral_scenarios0.36760.4883 college_chemistry0.530.65 formal_logic0.42060.5397 miscellaneous0.78930.908 virology0.40360.5181 high_school_mathematics0.54440.6556 anatomy0.57780.6889 logical_fallacies0.73620.8466 world_religions0.75440.848 business_ethics0.610.7 elementary_mathematics0.76720.8439 global_facts0.570.5 20 SubjectScore (options not in prompt)Score (options in prompt) Table 9 Full results for MMLU (57 tasks). SubsetScore (options not in prompt)Score (options in prompt) Electronics0.3020.5302 History0.52920.7468 Psychology0.4970.7134 Design0.66830.8492 Manage0.36130.5393 Sociology0.47690.6299 Math0.36710.4968 Chemistry0.43260.5603 Diagnostics_and_Laboratory_Medicine0.32290.4427 Agriculture0.41460.5316 Art_Theory0.64190.7533 Biology0.4740.5836 Physics0.40510.5118 Materials0.36270.4481 Clinical_Medicine0.58470.6695 Geography0.38110.4656 Marketing0.6250.7067 Basic_Medical_Science0.53470.6152 Mechanical_Engineering0.32960.4081 Pharmacy0.54180.6058 Energy_and_Power0.37640.4245 Economics0.52740.5719 Art0.73950.7816 Public_Health0.5920.6315 Literature0.82390.8551 Computer_Science0.48030.5046 Music0.34440.365 Architecture_and_Engineering0.37540.3483 Finance0.56550.5329 Accounting0.54860.495 Finance0.56550.5329 Accounting0.54860.495 Table 10 Full results for MMMU (30 tasks). SuiteDatasetScore imageEDIS0.98 imageVisual7W-Pointing0.972 imageDocVQA0.965 imageRefCOCO0.969 imageRefCOCO-Matching0.942 imageImageNet-R0.943 imageTextVQA0.913 imageWebQA0.904 imageInfographicsVQA0.882 imageVisDial0.874 imageObjectNet0.857 imageVOC20070.855 imageImageNet-1K0.85 imageWiki-S-NQ0.841 imageVisualNews_i2t0.856 imageChartQA0.837 imageOVEN0.836 imageSUN3970.816 imageVisualNews_t2i0.839 imageN24News0.816 imageHatefulMemes0.794 imageMSCOCO_t2i0.801 imageMSCOCO_i2t0.78 imageScienceQA0.745 imageOK-VQA0.757 imageGQA0.756 imageImageNet-A0.721 imageMSCOCO0.732 imageCIRR0.685 imageVizWiz0.674 imageA-OKVQA0.665 imageNIGHTS0.672 imageVisual7W0.592 imageMCMR0.6685 imagePlace3650.525 imageCountry2110.477 imageFashionIQ0.427 videoUCF1010.941 videoActivityNetQA0.799 videoSmthSmthV20.8 videoNExTQA0.7477 videoHMDB510.716 videoK7000.718 videoMSVD0.7239 videoQVHighlight0.7156 videoDiDeMo0.5916 videoMVBench0.5533 videoVideo-MME0.5485 videoMSR-VTT0.526 videoEgoSchema0.522 videoVATEX0.5013 videoMomentSeeker0.4744 videoBreakfast0.4665 videoYouCook20.2693 21 SuiteDatasetScore videoCharades-STA0.2682 visdocViDoSeek-doc0.9974 visdocViDoRe_syntheticDocQA_artificial_intelligence0.94 visdocVisRAG_SlideVQA0.9514 visdocViDoRe_tabfquad0.8964 visdocViDoRe_syntheticDocQA_energy0.88 visdocViDoRe_syntheticDocQA_healthcare_industry0.9 visdocVisRAG_InfoVQA0.883 visdocViDoRe_infovqa0.8583 visdocViDoRe_syntheticDocQA_government_reports0.84 visdocViDoRe_arxivqa0.842 visdocVisRAG_ArxivQA0.8333 visdocVisRAG_MP-DocVQA0.8359 visdocVisRAG_ChartQA0.8571 visdocMMLongBench-doc0.7554 visdocViDoSeek-page0.7145 visdocViDoRe_shiftproject0.65 visdocViDoRe_biomedical_lectures_v2_multilingual0.6234 visdocViDoRe_esg_reports_v2_multilingual0.6053 visdocVisRAG_PlotQA0.5817 visdocViDoRe_esg_reports_human_labeled_v20.6731 visdocViDoRe_tatdqa0.5765 visdocViDoRe_economics_reports_v2_multilingual0.6422 visdocMMLongBench-page0.5215 visdocViDoRe_docvqa0.5366 Table 11 Full results on MMEB-V3 image, video and visdoc (79 tasks). TaskNDCG@10 Touche2020 *0.4609 NQ0.8311 QuoraRetrieval0.949 ClimateFEVER0.2869 FiQA20180.7123 MSMARCO0.5819 DBPedia *0.6235 ArguAna0.6537 FEVER0.9534 NFCorpus0.3789 HotpotQA *0.8943 SCIDOCS *0.4885 SciFact0.8075 Table 12 Full results on Nano-BEIR (13 tasks). 22