Paper deep dive
PACE: Progressive Angular-to-Norm Contrastive Embedding
Yanping Li, Wei Zhou, Yawen Liu, Yibo Wang, Ke Zhu, Guangda Huzhang, Qing-Guo Chen, Zhao Xu, Jun Zhang, Wei Wei
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal embedding models encode heterogeneous inputs into a shared embedding space, enabling efficient similarity computation across modalities and tasks. Most existing methods optimize cosine-based contrastive objectives, which promote stable training but restrict semantic compatibility to angular geometry, precluding embedding norms from serving as an additional semantic signal. However, directly optimizing the more expressive dot-product similarity, which leverages both angular and norm information, underperforms cosine-based training and exhibits unstable training dynamics. We attribute this discrepancy to premature optimization-space expansion, manifested as angular--norm entanglement and directional anisotropy in the representation space and further compounded by full-parameter fine-tuning. In this paper, we propose PACE, a two-stage framework that progressively expands both the representation and trainable parameter spaces. Stage I combines cosine-based objective with low-rank adaptation to establish a reliable angular geometry within constrained optimization spaces. Stage II switches to dot-product similarity and full-parameter fine-tuning, enabling embedding directions and norms to jointly encode semantic information. We further introduce Focal Embedding Loss, a confidence-adaptive objective that downweights queries with high positive retrieval confidence while emphasizing ambiguous queries with competitive negatives. Experiments across multiple backbone scales and diverse multimodal embedding tasks consistently validate the effectiveness of PACE.
Tags
Links
- Source: https://arxiv.org/abs/2609.15152v1
- Canonical: https://arxiv.org/abs/2609.15152v1
Trouble viewing inline? Open PDF directly →
Full Text
66,173 characters extracted from source content.
Expand or collapse full text
PACE: Progressive Angular-to-Norm Contrastive Embedding Yanping Li Affiliation: The Hong Kong University of Science and Technology Affiliation: Alibaba Group Wei Zhou Affiliation: Huazhong University of Science and Technology Affiliation: Alibaba Group Yawen Liu Affiliation: Alibaba Group Yibo Wang Affiliation: Alibaba Group Affiliation: Nanjing University Ke Zhu Affiliation: Alibaba Group Guangda Huzhang Affiliation: Alibaba Group Qing-Guo Chen Affiliation: Alibaba Group Zhao Xu Affiliation: Alibaba Group Jun Zhang Affiliation: The Hong Kong University of Science and Technology Wei Wei Affiliation: Huazhong University of Science and Technology Abstract Multimodal embedding models encode heterogeneous inputs into a shared embedding space, enabling efficient similarity computation across modalities and tasks. Most existing methods optimize cosine-based contrastive objectives, which promote stable training but restrict semantic compatibility to angular geometry, precluding embedding norms from serving as an additional semantic signal. However, directly optimizing the more expressive dot-product similarity, which leverages both angular and norm information, underperforms cosine-based training and exhibits unstable training dynamics. We attribute this discrepancy to premature optimization-space expansion, manifested as angular–norm entanglement and directional anisotropy in the representation space and further compounded by full-parameter fine-tuning. In this paper, we propose PACE, a two-stage framework that progressively expands both the representation and trainable parameter spaces. Stage I combines cosine-based objective with low-rank adaptation to establish a reliable angular geometry within constrained optimization spaces. Stage I switches to dot-product similarity and full-parameter fine-tuning, enabling embedding directions and norms to jointly encode semantic information. We further introduce Focal Embedding Loss, a confidence-adaptive objective that downweights queries with high positive retrieval confidence while emphasizing ambiguous queries with competitive negatives. Experiments across multiple backbone scales and diverse multimodal embedding tasks consistently validate the effectiveness of PACE. 1 Introduction Multimodal embedding models (Li et al. 2026a; Shanbhogue et al. 2026; Lee et al. 2025; Zhang et al. 2025b; Radford et al. 2021; Girdhar et al. 2023; Zhu et al. 2024) encode heterogeneous inputs—including text, images, and arbitrary compositions thereof—into compact dense vectors in a shared embedding space, such that semantically corresponding instances receive high similarity scores regardless of their modalities. By providing a unified and modality-agnostic representation interface, these models enable efficient similarity computation across diverse input forms and support a broad spectrum of applications, including cross-modal retrieval, classification, visual question answering, and visual grounding (Radford et al. 2021; Jiang et al. 2025; Lin et al. 2025; Li et al. 2026c; Li et al. 2026b; Xue et al. 2025). Recent research (Jiang et al. 2025; Lin et al. 2025; Zhang et al. 2025a; Gu et al. 2026) has further shifted toward universal multimodal embedding models that generalize across both input modalities and task formulations within a single model. Learning representations that are well aligned across modalities, discriminative with respect to fine-grained semantics, and transferable across tasks has therefore become a fundamental problem in multimodal representation learning. Figure 1: Comparison of direct dot-product and cosine-based optimization over a training epoch. (a) Direct dot-product optimization achieves lower embedding performance (MMEB score). (b) Its loss trajectory exhibits recurrent late-stage spikes, whereas cosine-based optimization remains comparatively stable. Despite substantial advances in model architectures (Shanbhogue et al. 2026; Li et al. 2026a) and multimodal data curation (Gadre et al. 2023; Fang et al. 2024; Xu et al. 2024; Zhang et al. 2025a), the dominant training paradigm for multimodal embedding models remains geometrically restrictive. Most existing methods optimize an InfoNCE objective using cosine similarity (Radford et al. 2021; Jiang et al. 2025; Lin et al. 2025; Zhu et al. 2026), which computes similarity between ℓ2 _2-normalized embeddings. By eliminating radial degrees of freedom, normalization prevents norm variations from rescaling contrastive logits and thereby simplifies optimization. However, it restricts semantic compatibility to angular information, excluding embedding norms as a potential relevance signal. This restriction is increasingly consequential as multimodal large language models (MLLMs) are adapted into universal multimodal embedders, since an angular-only similarity geometry may underutilize their substantial representational capacity. Unnormalized inner products have long served as relevance scores in retrieval and recommendation, motivating extensive research on maximum inner-product search (MIPS) (Ram and Gray 2012; Shrivastava and Li 2014). Recent evidence further indicates that embedding norms can encode task-dependent relevance information (Feng and Watanabe 2026), motivating dot-product similarity as a more expressive alternative that leverages both angular and norm information. Nevertheless, our empirical results in Figure 1 show that directly optimizing dot-product similarity underperforms its cosine-based counterpart and exhibits unstable training dynamics, revealing a persistent gap between representational expressivity and optimization stability. To understand the performance gap, preliminary experiments reveal two challenges in direct dot-product optimization. First, jointly learning angular alignment and norm scaling from the outset can entangle directions and norms, causing them to encode redundant rather than complementary semantic information. Cosine similarity avoids this interaction by removing radial degrees of freedom. Second, dot-product training can induce directional anisotropy, concentrating embeddings within a narrow cone and assigning spuriously high similarity to semantically unrelated instances. Angular–norm entanglement and directional anisotropy help explain its unexpected underperformance and unstable training dynamics. Full-parameter fine-tuning further compounds the difficulty by substantially enlarging the trainable parameter space. Together, these findings motivate a coarse-to-fine strategy that establishes a reliable angular geometry within constrained representation and parameter spaces before progressively expanding both to exploit additional capacity. Based on this insight, we propose Progressive Angular-to-Norm Contrastive Embedding (PACE), a two-stage framework for stable and expressive multimodal embedding learning. PACE progressively expands both the representation and trainable parameter spaces. Stage I combines a cosine-based objective with LoRA (Hu et al. 2022), restricting semantic organization to angular geometry and parameter updates to a compact low-rank subspace. Initialized from this constrained solution, Stage I adopts dot-product similarity and full-parameter fine-tuning, enabling embedding directions and norms to jointly encode semantic information while exploiting the complete parameter space. A complementary first-order gradient analysis formalizes the radial–angular coupling induced by dot-product optimization and shows that, the progressive Stage I initialization yields locally weak cross-Jacobian interactions, providing a mechanism for mitigating angular–norm entanglement. Despite these benefits, multi-stage training may overemphasize already well-separated examples, increasing the risk of overfitting. To mitigate this issue, we further introduce Focal Embedding Loss, a confidence-adaptive objective inspired by focal learning (Lin et al. 2017). It assigns each query a difficulty-aware weight based on its positive retrieval confidence, downweighting examples whose positives are already well separated from negatives while emphasizing ambiguous examples with competitive negatives. This mechanism suppresses redundant updates on saturated examples and directs the expanded optimization capacity toward unresolved semantic distinctions. Extensive experiments across multiple model scales and a diverse suite of multimodal embedding benchmarks demonstrate that PACE consistently outperforms strong baselines and validating the effectiveness of each proposed component. Our main contributions are summarized as follows: • To our knowledge, we provide the first systematic diagnosis that attributes the unexpected underperformance and unstable training dynamics of direct dot-product optimization to angular–norm entanglement and directional anisotropy, supported by controlled experiments and a complementary theoretical analysis. • We propose PACE, a progressive training framework that combines a cosine-to-dot-product curriculum with a LoRA-to-full-fine-tuning schedule, progressively expanding the representation and trainable parameter spaces for stable and expressive multimodal embedding learning. • We further introduce Focal Embedding Loss, a confidence-adaptive objective that downweights well-separated examples and emphasizes ambiguous examples with competitive negatives. Experiments across multiple backbone scales and diverse multimodal embedding tasks demonstrate the effectiveness of the overall framework. Figure 2: Diagnostic analyses of direct dot-product training. (a) PCA projections of ℓ2 _2-normalized embeddings, colored by their original norm percentiles. Direct dot-product training produces a norm-stratified directional geometry absent under cosine-based training. (b) Cosine-similarity distributions between randomly paired queries and candidates, showing substantially greater directional concentration under direct dot-product training. (c) MMEB image-task performance under LoRA and full-parameter fine-tuning, where direct dot-product training exhibits a markedly larger degradation. 2 Related Work Existing work improves embedding models through four complementary directions: model development, query augmentation, data-side techniques, and similarity design. Multimodal embedding learning has progressed from vision–language alignment to universal representation learning. CLIP (Radford et al. 2021) establishes the image–text dual-encoder paradigm, while ImageBind (Girdhar et al. 2023) and LanguageBind (Zhu et al. 2024) extend shared-space learning to additional modalities. More recently, MLLMs have been adapted into universal embedders through instruction-aware training and broader task and modality coverage (Jiang et al. 2025; Lin et al. 2025; Zhang et al. 2025a). Native multimodal embedding models further scale model capacity and modality coverage (Shanbhogue et al. 2026; Li et al. 2026a). Query augmentation improves retrieval by enriching the input representation. Query2doc (Wang, Yang, and Wei 2023) and HyDE (Gao et al. 2023) generate pseudo-documents, BRIGHT (Su et al. 2025) studies explicit reasoning for retrieval, and ExpandR (Yao et al. 2025) aligns generated expansions with the retriever. These methods generally require additional LLM inference for each query. Data-side techniques encompass data curation and difficulty-aware supervision. Existing work improves training data through standardized selection, learned filtering, metadata-guided balancing, and task-diverse synthesis (Gadre et al. 2023; Fang et al. 2024; Xu et al. 2024; Zhang et al. 2025a). For difficulty-aware training, M-Embed (Lin et al. 2025) mines modality-aware hard negatives, while Focal-InfoNCE (Hou and Li 2023) applies pair-level focal modulation. Query-level reweighting based on the positive retrieval probability over the candidate set remains underexplored. Our Focal Embedding Loss addresses this gap by downweighting well-separated queries and emphasizing those with competitive negatives. Most multimodal embedders use cosine similarity over normalized representations, excluding embedding norms from similarity computation. Unnormalized inner products have long served as retrieval and recommendation scores, motivating maximum inner-product search (Ram and Gray 2012; Shrivastava and Li 2014). Feng and Watanabe (Feng and Watanabe 2026) further show that query and candidate norms play distinct, task-dependent roles in optimization and inference. However, reliably learning a multimodal embedding geometry that jointly exploits angular and norm information remains underexplored. 3 Methodology Figure 3: Overview of PACE. Stage I learns a stable angular geometry using cosine similarity and LoRA within constrained representation and parameter spaces. Stage I expands to dot-product similarity and full-parameter fine-tuning, while Focal Embedding Loss emphasizes ambiguous queries with competitive negatives. 3.1 Problem Formulation Given a multimodal query xiqx_i^q and a candidate set i=xi+,xi,1−,…,xi,K−C_i=\x_i^+,x_i,1^-,…,x_i,K^-\, both queries and candidates may consist of text, images, or interleaved image–text sequences. We employ a shared MLLM-based encoder fθf_θ to map each heterogeneous input into a d-dimensional dense vector in a shared embedding space: i=fθ(xiq),=fθ(x),x∈i.q_i=f_θ(x_i^q), =f_θ(x), x _i. Candidate relevance is measured by a similarity function s(i,)s(q_i,c), according to which the candidate set is ranked. The learning objective is to assign the positive candidate xi+x_i^+ a higher similarity score than all negative candidates, thereby supporting modality-agnostic retrieval across heterogeneous input compositions. 3.2 Motivation To diagnose why direct dot-product optimization exhibits unstable training dynamics, we examine whether it exploits angular and norm information complementarily. Using cosine-based training as a controlled reference under matched backbones, data, and training settings, we analyze 5,000 randomly sampled training instances and assess whether full-parameter fine-tuning amplifies the optimization difficulty. Angular–Norm Entanglement. After ℓ2 _2-normalizing the learned embeddings, we apply PCA to visualize their directional structure and color each point by its original norm. As shown in Figure 2(a), embeddings with substantially different norms are intermixed under cosine-based training, indicating little visible dependence between directions and norms. Direct dot-product training instead produces a norm-stratified structure in which embeddings with similar norms occupy nearby projected regions. This qualitative pattern suggests that direct optimization entangles angular and norm information rather than exploiting them complementarily. Directional Anisotropy. We next examine cosine similarities between randomly paired queries and candidates. Figure 2(b) shows that cosine-based training yields lower random-pair similarities and a more dispersed directional geometry. Direct dot-product training shifts the distribution toward higher similarity values, indicating that embedding directions concentrate within a narrower region of the unit hypersphere. Such directional concentration can increase spurious similarities between semantically unrelated instances and potentially exacerbate hubness (Radovanovic, Nanopoulos, and Ivanovic 2010; Angiulli 2018). Effect of Parameter-Space Expansion. Finally, Figure 2(c) compares cosine- and dot-product training under LoRA and full-parameter fine-tuning. Full-parameter fine-tuning underperforms LoRA for both similarity functions, with a substantially larger degradation under dot-product similarity. This interaction suggests that simultaneously enlarging the representation and trainable parameter spaces compounds early-stage optimization difficulty. Overall, direct dot-product optimization does not necessarily realize its greater representational capacity. These findings motivate a coarse-to-fine strategy that first establishes a reliable angular geometry within a compact parameter subspace and subsequently introduces radial degrees of freedom and full-parameter updates. 3.3 PACE As illustrated in Figure 3, PACE is a two-stage framework that progressively expands both the representation and trainable parameter spaces. Stage I restricts optimization to angular similarity and a compact low-rank parameter subspace, providing a stable initialization with a reliable semantic geometry. Stage I subsequently introduces radial representational freedom and full-parameter optimization, enabling the model to exploit the increased capacity of both spaces. Focal Embedding Loss is applied in both stages. Using a consistent cosine-normalized confidence function across both stages mitigates cross-stage shifts in the focal-weight distribution and prevents embedding-norm scaling from distorting query difficulty. Stage I: Constrained Optimization Space. In the first stage, we restrict the similarity function to angular information by ℓ2 _2-normalizing query and candidate embeddings: s(1)(,)=scos(,)=⊤‖2‖2.s^(1)(q,c)=s_cos(q,c)= q c\|q\|_2\|c\|_2. (1) The resulting scale invariance removes radial degrees of freedom from the contrastive objective, allowing the model to first establish a well-structured angular geometry. We simultaneously restrict parameter updates using LoRA (Hu et al. 2022). For a pretrained weight matrix 0W_0, LoRA parameterizes its update as Δ= =BA, where rank(Δ)≤rrank( )≤ r and r is substantially smaller than the dimensions of 0W_0. Optimizing only these low-rank updates constrains training to a compact parameter subspace, reducing early-stage optimization difficulty and providing a stable, semantically meaningful initialization for the subsequent stage. Stage I: Expanded Optimization Space. The second stage initializes from the Stage I solution and expands both optimization spaces. In the representation space, we remove ℓ2 _2 normalization and adopt dot-product similarity: s(2)(,)=sdot(,)=⊤.s^(2)(q,c)=s_dot(q,c)=q c. (2) This transition releases the radial degrees of freedom, allowing embedding directions and norms to jointly encode semantic information. In the parameter space, we merge the learned LoRA updates into the pretrained backbone and perform full-parameter fine-tuning. Starting from the constrained Stage I solution allows the model to exploit the greater representational and parameter capacity of Stage I without directly optimizing both expanded spaces from the beginning. Focal Embedding Loss. Although progressive expansion improves representational capacity, both stages may overemphasize queries whose positive candidates are already well separated from the negatives. We introduce Focal Embedding Loss, which adaptively reweights each query according to its current retrieval difficulty. For Stage T∈I,IT∈\I,I\, the positive probability used by the contrastive objective is πi(t)=exp(s(t)(i,i+)/τt)∑∈iexp(s(t)(i,)/τt), _i^(t)= (s^(t)(q_i,c_i^+)/ _t ) _c _i (s^(t)(q_i,c)/ _t ), (3) where s(t)s^(t), t∈1,2t∈\1,2\, denotes the similarity employed in Stage T; specifically, s(1)s^(1) and s(2)s^(2) are the cosine and dot-product similarities used in Stages I and I, respectively. And τt _t is the corresponding contrastive temperature. The per-query contrastive loss is ℓi(t)=−logπi(t) _i^(t)=- _i^(t). To obtain a norm-invariant estimate of query difficulty, we compute the focal confidence using cosine-normalized logits in both stages: pi(t)=exp(scos(i,i+)/τw)∑∈iexp(scos(i,)/τw),p_i^(t)= (s_cos(q_i,c_i^+)/ _w ) _c _i (s_cos(q_i,c)/ _w ), (4) where τw _w is the weighting temperature. Decoupling the weighting confidence from the stage-specific training similarity prevents embedding-norm scaling in Stage I from artificially inflating the estimated confidence and provides a consistent difficulty measure across stages. We first compute a detached focal coefficient ai(t)=sg[(1−pi(t))γ],a_i^(t)=sg\! [ (1-p_i^(t) )^γ ], (5) where sg[⋅]sg[·] denotes stop-gradient and γ determines the strength of difficulty modulation. We normalize these coefficients to have unit mean over a minibatch of N queries: w~i(t)=ai(t)1N∑k=1Nak(t). w_i^(t)= a_i^(t) 1N _k=1^Na_k^(t). (6) The focal-weighted objective for Stage t is then ℒPACE(t)=1N∑i=1Nw~i(t)ℓi(t)=−∑i=1Nai(t)logπi(t)∑i=1Nai(t).L_PACE^(t)= 1N _i=1^N w_i^(t) _i^(t)=- _i=1^Na_i^(t) _i^(t) _i=1^Na_i^(t). (7) Queries whose positives are angularly well separated from the negatives yield high confidence and receive smaller weights, whereas ambiguous queries with competitive negatives receive larger weights. This mechanism suppresses redundant updates on saturated examples and directs optimization toward unresolved semantic distinctions in both training stages. 3.4 Theoretical Analysis Radial–Angular Gradient Decomposition. We analyze the first-order embedding dynamics of the unweighted InfoNCE objectives in both stages. Let =r^q=r q, where ‖^‖2=1\| q\|_2=1, and define ^⟂=−^^⊤P_ q =I- q q . Differentiating the cosine objective through ℓ2 _2 normalization gives ∇ℒcos=1r^⟂∇^ℒcos,^⊤∇ℒcos=0. _qL_cos= 1rP_ q _ qL_cos, q _qL_cos=0. (8) Hence, cosine-based training has no first-order radial embedding gradient. Assume candidates share a common norm and write j=R^jc_j=R c_j and uj=^⊤^ju_j= q c_j. For dot-product InfoNCE, let pj(2)=exp(rRuj/τ2)/∑kexp(rRuk/τ2)p_j^(2)= (rRu_j/ _2)/ _k (rRu_k/ _2). Its gradient decomposes as ∇ℒdot _qL_dot =ad^+d, =a_d q+b_d, (9) ad a_d =Rτ2(p(2)[u]−u+), = R _2 (E_p^(2)[u]-u_+ ), d _d =Rτ2^⟂(∑jpj(2)^j−^+). = R _2P_ q ( _jp_j^(2) c_j- c_+ ). Here, ada_d and db_d govern radial and angular evolution, respectively; because p(2)p^(2) depends jointly on r and u, the two dynamics are coupled. Proposition 1 (Confidence-Dependent Dynamic Decoupling). Under the common-norm setting above, consider the Stage I initialization 0=r0^q_0=r_0 q and let β=r0R/τ2β=r_0R/ _2. If its positive posterior satisfies p+(2)≥1−ρp_+^(2)≥ 1-ρ, then ‖∂d∂r‖2≤4R2τ22ρ,‖D^ad‖op≤Rτ2(4β+2)ρ. \| _d∂ r \|_2≤ 4R^2 _2^2ρ, \|D_ qa_d \|_op≤ R _2(4β+2)ρ. (10) where D^ad[]D_ qa_d[h] denotes the directional derivative of the radial coefficient ada_d with respect to q along a tangent perturbation h, and ‖D^ad‖op=sup⟂^,‖2=1|D^ad[]|\|D_ qa_d\|_op= _h q,\,\|h\|_2=1|D_ qa_d[h]|. These off-diagonal blocks characterize the local coupling between radial and angular dynamics. For bounded β and R/τ2R/ _2, both cross-Jacobian norms vanish as ρ→0ρ→ 0, yielding local first-order decoupling at the Stage I transition. This result is consistent with the reduced angular–norm dependence observed empirically but does not imply global statistical independence. A complete derivation is provided in the supplementary material. 4 Experiment Models #Params Per Meta-Task Score Average Score Classification VQA Retrieval Grounding IND OOD Overall # of Datasets – 10 10 12 4 20 16 36 Zero-shot on MMEB CLIP (ViT-L) 0.4B 42.8 9.1 53.0 51.8 37.1 38.7 37.8 OpenCLIP (ViT-L) 0.4B 47.8 10.9 52.3 53.3 39.3 40.2 39.7 MagicLens (ViT-L) 0.4B 38.8 8.3 35.4 26.0 31.0 23.7 27.1 SigLIP (So/14) 0.9B 40.3 8.4 31.6 59.5 32.3 38.0 35.0 BLIP-2 (ViT-L) 1.2B 27.0 4.2 33.9 47.0 25.3 25.1 28.0 CLIP (ViT-BigG/14) 2.5B 52.3 14.0 50.5 60.3 38.9 45.8 44.3 EVA-CLIP 8B 56.0 10.4 49.2 58.9 38.1 45.6 43.7 E5-V (Phi-3.5-V) 4.2B 39.1 9.6 38.0 57.6 33.1 31.9 36.1 E5-V (LLaVA-1.6) 7B 39.7 10.8 39.4 60.2 34.2 33.4 37.5 Fine-tuning on MMEB VLM2Vec (Qwen3.5-0.8b) 0.8B 51.0 56.8 53.1 73.2 61.1 49.2 55.8 VLM2Vec (Qwen3.5-2b) 2B 61.4 62.0 61.4 73.7 67.2 57.6 62.9 UniME-V2 (Qwen3.5-0.8b) 0.8B 51.7 59.0 53.8 75.0 60.9 52.2 57.0 UniME-V2 (Qwen3.5-2b) 2B 60.7 64.1 63.8 80.3 67.0 62.1 64.9 PACE (Qwen3.5-0.8b) 0.8B 52.3 (+0.6) 60.4 (+1.4) 57.4 (+3.6) 77.2 (+2.2) 63.8 (+2.7) 53.0 (+0.8) 59.0 (+2.0) PACE (Qwen3.5-2b) 2B 63.6 (+2.2) 66.0 (+1.9) 66.5 (+2.7) 81.8 (+1.5) 71.2 (+4.0) 62.4 (+0.3) 67.3 (+2.4) Table 1: Results on the MMEB benchmark. IND: in-distribution, OOD: out-of-distribution. Scores are average Precision@1 (%). The best are highlighted in bold. Δ values in parentheses are computed against the highest-scoring baseline of the same model. Detailed results are in the supplementary materials. 4.1 Implementation We train PACE using two MLLMs, Qwen3.5-0.8B-Base and Qwen3.5-2B-Base (Qwen Team 2026). Training consists of two stages. In the first stage, we implement LoRA (rank=32) and adopt Focal Embedding Loss with global weight normalization, setting the focal parameter γ to 2.0 and using a learning rate of 2×10−52× 10^-5. In the second stage, we initialize the model from the step-2,000 checkpoint obtained in the first stage and perform full-parameter language-model fine-tuning with Focal Embedding Loss, where the similarity metric is calculated using the dot product. The learning rate is reduced to 2×10−62× 10^-6. In both stages, we use 5 hard negatives per query. Detailed hyperparameters and hardware configurations are provided in the supplementary material. 4.2 Datasets and Evaluation Training Data. We use the mmE5-Synthetic dataset (Chen et al. 2025) as our training data. After preprocessing, the resulting corpus comprises approximately 2.0 million training instances constructed under an MMEB-oriented design for general-purpose multimodal embedding learning. It provides task-relevant supervision across MMEB-style task formats and modality combinations, aligning with the MMEB benchmark, which consists of 20 in-distribution and 16 out-of-distribution datasets spanning classification, visual question answering, multimodal retrieval, and visual grounding. Evaluation. We evaluate PACE on both the in-distribution and out-of-distribution splits of MMEB (Jiang et al. 2025), comprising 20 and 16 test sets, respectively, to assess its general-purpose multimodal embedding capabilities across diverse tasks. Following the standard MMEB evaluation protocol, we report Precision@1, defined as the proportion of queries for which the correct candidate is ranked first. Unlike conventional cosine-similarity evaluation, PACE ranks candidates using the dot product between unnormalized query and candidate embeddings, consistent with the dot-product objective employed during the second training stage. 4.3 Baselines We compare PACE against both zero-shot and fine-tuned multimodal embedding models. The zero-shot baselines include general-purpose vision–language encoders, namely CLIP (Radford et al. 2021), OpenCLIP (Cherti et al. 2023), MagicLens (Zhang et al. 2024), SigLIP (Zhai et al. 2023), BLIP-2 (Li et al. 2023), and EVA-CLIP (Sun et al. 2023), as well as MLLM-based embedders, including E5-V (Jiang et al. 2024) and pretrained Qwen3.5 backboneswithout embedding-specific fine-tuning. For fine-tuned comparisons, we consider VLM2Vec (Jiang et al. 2025), which performs instruction-guided contrastive training with standard InfoNCE loss across diverse multimodal embedding tasks. UniME-V2 (Gu et al. 2026) further employs MLLM-derived semantic matching scores to improve contrastive supervision. 5 Main Results Table 1 presents the quantitative results on MMEB, where all methods are trained under identical hyperparameter settings for fair comparison. PACE achieves overall scores of 59.0 and 67.3 with the 0.8B and 2B backbones, respectively; the 0.8B model alone exceeds the strongest zero-shot baseline by 14.7 despite using fewer parameters. Against the strongest scale-matched fine-tuned baselines, PACE yields overall margins of 2.0 and 2.4, with per-category improvements of 0.6–3.6 (0.8B) and 1.5–2.7 (2B) across classification, VQA, retrieval, and grounding. Positive margins on both IND and OOD splits further confirm robustness across evaluation distributions. These consistent gains support the central premise of PACE: establishing a reliable angular geometry before progressively expanding the representation and parameter spaces allows MLLM backbones to realize their representational capacity without sacrificing optimization stability. 6 Geometric Analysis Figure 4: Angular-geometry diagnostics of direct dot-product training vs. PACE. (a) PACE weakens the negative correlation between cosine similarity and the standardized log norm product. (b) PACE lowers the mean cross-cosine similarity, indicating a more angularly dispersed embedding geometry. To assess whether PACE alleviates the representation-space distortions caused by direct dot-product training, we randomly analyze a subset of training data, focusing on the angular-norm coupling and directional anisotropy. Additional visual embedding geometry analysis are provided in the supplementary material. Reduced Angular–Norm Coupling. Figure 4(a) plots the standardized log(∥2∥+∥2) ( _2 ^+ _2) against cosine similarity for matched query–positive pairs. Direct dot-product training yields a negative Pearson correlation (r=−0.78r=-0.78), indicating strong angular–norm coupling, whereas PACE reduces it to r=−0.11r=-0.11. This near-zero correlation implies little mutual information between the two factors: instead of encoding redundant information as in the coupled regime, norm and direction carry complementary signals, enlarging the effective representational capacity of the embedding space. Directional Dispersion. Figure 4(b) compares the cosine-similarity distributions of randomly paired queries and candidates. PACE yields a lower mean similarity than direct dot-product training, consistent with reduced global directional concentration and a more angularly dispersed embedding geometry. Together, these results demonstrate that PACE mitigates both angular–norm coupling and directional anisotropy. 7 Ablation Studies Component Contributions. Table 2 evaluates the contribution of each PACE component. Removing Focal Embedding Loss decreases the overall score from 59.0 to 57.7. Disabling representation-space expansion by retaining cosine similarity in Stage I further reduces the score to 57.2, while retaining LoRA instead of transitioning to full-parameter fine-tuning yields the largest degradation, to 54.1. These results demonstrate that difficulty-aware weighting, cosine-to-dot representation-space expansion, and LoRA-to-full parameter-space expansion each contribute to PACE. Method ClS VQA RET GRD Overall PACE (Ours) 52.3 60.4 57.4 77.2 59.0 PACE w/o Focal loss 52.3 59.7 55.0 74.1 57.7 PACE w/o Dot 52.1 58.2 54.3 76.0 57.2 PACE w/ S2 LoRA 49.8 56.4 49.6 72.3 54.1 Table 2: Ablation study examining effects of training objectives and second-stage tuning strategy on MMEB. Abbreviations: CLS, Classification; RET, Retrieval; GRD, Grounding. S2 denotes Stage I. Order of Optimization-Space Expansion. To determine whether the improvement arises from the progressive expansion order rather than two-stage training alone, Figure 5 compares PACE with the corresponding reversed schedules. Cosine-to-dot training outperforms dot-to-cosine training by 2.9 overall, demonstrating the benefit of establishing an angular semantic geometry before introducing radial degrees of freedom. Similarly, LoRA-to-full fine-tuning exceeds the reversed full-to-LoRA schedule by 7.3. This result supports our hypothesis that early optimization within a compact parameter subspace provides a more reliable update direction for subsequent full-parameter optimization. Figure 5: Effect of progressive expansion order on MMEB. Cosine-to-dot and LoRA-to-full schedules outperform their reversed counterparts by 2.9 and 7.3 overall, respectively. Stage-Transition Checkpoint. Figure 6 (a) examines the effect of the Stage I initialization checkpoint. Initializing Stage I from the intermediate Stage I at steps 2000 and 4000 yields comparable scores of 58.3 and 58.5, both exceeding the 56.9 obtained from the fully converged checkpoint at step 7847 This suggests that fully converging in the constrained first-stage space can overspecialize the learned angular geometry and reduce plasticity when the optimization space is subsequently expanded. Intermediate checkpoints instead provide a more favorable balance between reliable semantic initialization and adaptability for Stage I. Further analyzes of Stage I initialization checkpoints are provided in the supplementary material. Focal Embedding Loss Variants. To assess whether the improvement arises from difficulty-aware weighting rather than arbitrary loss rescaling, Figure 6(b) compares Focal Embedding Loss with three alternatives: random sample weighting, query–positive similarity weighting, and query–negative similarity weighting based on the hardest negative. For simplicity, all variants are evaluated in Stage I under otherwise identical settings. Focal Embedding Loss performs best, indicating that query-level retrieval confidence over the complete candidate set—which jointly reflects positive alignment and negative competition—provides a more informative difficulty estimate than either signal alone. Sensitivity to the focal parameter γ and additional ablations are provided in the supplementary material. Figure 6: Ablations of Stage I design choices. (a) Intermediate checkpoints provide better initialization for Stage I than the final. (b) Focal Embedding Loss outperforms random, query-positive (QP) and query-negative (QN) weighting. 8 Conclusion In this work, we introduced PACE, a two-stage framework for stable and expressive multimodal embedding learning. PACE progressively expands both the representation and parameter spaces, transitioning from cosine similarity to dot-product similarity and from LoRA to full-parameter fine-tuning. This coarse-to-fine strategy first establishes a reliable angular semantic geometry before introducing radial and parameter degrees of freedom. Focal Embedding Loss further directs optimization toward ambiguous queries while reducing redundant updates on well-separated examples. Geometric analyses demonstrate that PACE alleviates angular–norm coupling and directional anisotropy, while experiments across multiple backbone scales and diverse multimodal embedding tasks consistently validate its effectiveness. These results establish progressive optimization-space expansion as an effective approach to better exploiting the representational capacity of MLLM-based embedders. References Angiulli (2018) Angiulli, F. 2018. On the behavior of intrinsically high-dimensional spaces: distances, direct and reverse nearest neighbors, and hubness. Journal of Machine Learning Research, 18(170): 1–60. Chen et al. (2025) Chen, H.; Wang, L.; Yang, N.; Zhu, Y.; Zhao, Z.; Wei, F.; and Dou, Z. 2025. mmE5: Improving Multimodal Multilingual Embeddings via High-Quality Synthetic Data. In Findings of the Association for Computational Linguistics: ACL 2025, 8254–8275. Cherti et al. (2023) Cherti, M.; Beaumont, R.; Wightman, R.; Wortsman, M.; Ilharco, G.; Gordon, C.; Schuhmann, C.; Schmidt, L.; and Jitsev, J. 2023. Reproducible Scaling Laws for Contrastive Language–Image Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2818–2829. Fang et al. (2024) Fang, A.; Jose, A. M.; Jain, A.; Schmidt, L.; Toshev, A.; and Shankar, V. 2024. Data Filtering Networks. In International Conference on Learning Representations. Feng and Watanabe (2026) Feng, X.; and Watanabe, T. 2026. When Does Embedding Magnitude Matter? A Cross-Task Functional-Symmetry Framework. arXiv preprint arXiv:2602.09229. Gadre et al. (2023) Gadre, S. Y.; Ilharco, G.; Fang, A.; Hayase, J.; Smyrnis, G.; Nguyen, T.; Marten, R.; Wortsman, M.; Ghosh, D.; Zhang, J.; Orgad, E.; Entezari, R.; Daras, G.; Pratt, S.; Ramanujan, V.; Bitton, Y.; Marathe, K.; Mussmann, S.; Vencu, R.; Cherti, M.; Krishna, R.; Koh, P. W. W.; Saukh, O.; Ratner, A. J.; Song, S.; Hajishirzi, H.; Farhadi, A.; Beaumont, R.; Oh, S.; Dimakis, A. G.; Jitsev, J.; Carmon, Y.; Shankar, V.; and Schmidt, L. 2023. DataComp: In Search of the Next Generation of Multimodal Datasets. In Advances in Neural Information Processing Systems, volume 36. Gao et al. (2023) Gao, L.; Ma, X.; Lin, J.; and Callan, J. 2023. Precise Zero-Shot Dense Retrieval without Relevance Labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 1762–1777. Girdhar et al. (2023) Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023. ImageBind: One Embedding Space To Bind Them All. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15180–15190. Gu et al. (2026) Gu, T.; Yang, K.; Zhang, K.; An, X.; Feng, Z.; Zhang, Y.; Cai, W.; Deng, J.; and Bing, L. 2026. Unime-v2: Mllm-as-a-judge for universal multimodal embedding learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 21378–21386. Hou and Li (2023) Hou, P.; and Li, X. 2023. Improving Contrastive Learning of Sentence Embeddings with Focal InfoNCE. In Findings of the Association for Computational Linguistics: EMNLP 2023, 4757–4762. Hu et al. (2022) Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations. Jiang et al. (2024) Jiang, T.; Song, M.; Zhang, Z.; Huang, H.; Deng, W.; Sun, F.; Zhang, Q.; Wang, D.; and Zhuang, F. 2024. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580. Jiang et al. (2025) Jiang, Z.; Meng, R.; Yang, X.; Yavuz, S.; Zhou, Y.; and Chen, W. 2025. VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. In International Conference on Learning Representations. Lee et al. (2025) Lee, J.; Chen, F.; Dua, S.; Cer, D.; Shanbhogue, M.; Naim, I.; Ábrego, G. H.; Li, Z.; Chen, K.; Vera, H. S.; et al. 2025. Gemini embedding: Generalizable embeddings from gemini. arXiv preprint arXiv:2503.07891. Li et al. (2023) Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. BLIP-2: Bootstrapping Language–Image Pre-Training with Frozen Image Encoders and Large Language Models. In Proceedings of the 40th International Conference on Machine Learning, 19730–19742. Li et al. (2026a) Li, M.; Zhang, Y.; Long, D.; Chen, K.; Song, S.; Bai, S.; Yang, Z.; Xie, P.; Yang, A.; Liu, D.; et al. 2026a. Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking. arXiv preprint arXiv:2601.04720. Li et al. (2026b) Li, S.; Huang, Y.; Liu, Z.; Li, Y.; Fu, J.; Zhao, L.; Bian, J.; Zhang, L.; Zhang, J.; and Wang, R. 2026b. GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation. arXiv preprint arXiv:2605.11853. Li et al. (2026c) Li, Y.; Liu, Z.; Li, Z.; Lin, Z.; and Zhang, J. 2026c. Token-level data selection for safe LLM fine-tuning. arXiv preprint arXiv:2603.01185. Lin et al. (2025) Lin, S.-C.; Lee, C.; Shoeybi, M.; Lin, J.; Catanzaro, B.; and Ping, W. 2025. Mm-embed: Universal multimodal retrieval with multimodal llms. In International Conference on Learning Representations, volume 2025, 44215–44234. Lin et al. (2017) Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; and Dollár, P. 2017. Focal Loss for Dense Object Detection. In Proceedings of the IEEE International Conference on Computer Vision, 2980–2988. Qwen Team (2026) Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, 8748–8763. Radovanovic, Nanopoulos, and Ivanovic (2010) Radovanovic, M.; Nanopoulos, A.; and Ivanovic, M. 2010. Hubs in space: Popular nearest neighbors in high-dimensional data. Journal of machine learning research, 11(sept): 2487–2531. Ram and Gray (2012) Ram, P.; and Gray, A. G. 2012. Maximum Inner-Product Search Using Cone Trees. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 931–939. Shanbhogue et al. (2026) Shanbhogue, M.; Li, Z.; Zhang, S.; Ábrego, G. H.; Huang, S.-C.; Jain, A.; Salz, D.; Goenka, S.; Hegde, C.; Ma, J.; et al. 2026. Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini. arXiv preprint arXiv:2605.27295. Shrivastava and Li (2014) Shrivastava, A.; and Li, P. 2014. Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS). In Advances in Neural Information Processing Systems, volume 27, 2321–2329. Su et al. (2025) Su, H.; Yen, H.; Xia, M.; Shi, W.; Muennighoff, N.; Wang, H.-y.; Liu, H.; Shi, Q.; Siegel, Z. S.; Tang, M.; Sun, R.; Yoon, J.; Arik, S. O.; Chen, D.; and Yu, T. 2025. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval. In International Conference on Learning Representations. Sun et al. (2023) Sun, Q.; Fang, Y.; Wu, L.; Wang, X.; and Cao, Y. 2023. EVA-CLIP: Improved Training Techniques for CLIP at Scale. arXiv preprint arXiv:2303.15389. Wang, Yang, and Wei (2023) Wang, L.; Yang, N.; and Wei, F. 2023. Query2doc: Query Expansion with Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 9414–9423. Xu et al. (2024) Xu, H.; Xie, S.; Tan, X.; Huang, P.-Y.; Howes, R.; Sharma, V.; Li, S.-W.; Ghosh, G.; Zettlemoyer, L.; and Feichtenhofer, C. 2024. Demystifying CLIP Data. In International Conference on Learning Representations. Xue et al. (2025) Xue, Y.; Huang, Y.; Shao, J.; Zhu, L.; Zhang, C.; Li, X.; and Zhang, J. 2025. Vlmq: Token saliency-driven post-training quantization for vision-language models. arXiv preprint arXiv:2508.03351. Yao et al. (2025) Yao, S.; Huang, P.; Liu, Z.; Gu, Y.; Yan, Y.; Yu, S.; and Yu, G. 2025. ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 19036–19054. Zhai et al. (2023) Zhai, X.; Mustafa, B.; Kolesnikov, A.; and Beyer, L. 2023. Sigmoid Loss for Language Image Pre-Training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 11975–11986. Zhang et al. (2024) Zhang, K.; Luan, Y.; Hu, H.; Lee, K.; Qiao, S.; Chen, W.; Su, Y.; and Chang, M.-W. 2024. MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions. In Proceedings of the 41st International Conference on Machine Learning, 59403–59420. Zhang et al. (2025a) Zhang, X.; Zhang, Y.; Xie, W.; Li, M.; Dai, Z.; Long, D.; Xie, P.; Zhang, M.; Li, W.; and Zhang, M. 2025a. Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9274–9285. Zhang et al. (2025b) Zhang, Y.; Li, M.; Long, D.; Zhang, X.; Lin, H.; Yang, B.; Xie, P.; Yang, A.; Liu, D.; Lin, J.; et al. 2025b. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176. Zhu et al. (2024) Zhu, B.; Lin, B.; Ning, M.; Yan, Y.; Cui, J.; Wang, H.; Pang, Y.; Jiang, W.; Zhang, J.; Li, Z.; Zhang, C.; Li, Z.; Liu, W.; and Yuan, L. 2024. LanguageBind: Extending Video-Language Pretraining to N-Modality by Language-Based Semantic Alignment. In International Conference on Learning Representations. Zhu et al. (2026) Zhu, L.; Huang, Y.; Ge, X.; Xue, Y.; Liu, Z.; Zhang, Y.; Lin, Z.; and Zhang, J. 2026. Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation. arXiv preprint arXiv:2602.19161. Appendix A Detail Experiment Setting A.1 Training Details We provide the full training configurations of PACE in Table 3, with both backbones, Qwen3.5-0.8B-Base and Qwen3.5-2B-Base, trained under identical hyperparameter settings. In the first stage, we perform parameter-efficient fine-tuning with LoRA (rank=32) under the Focal Embedding Loss with global weight normalization (γ=2.0γ=2.0). In the second stage, the model is initialized from the step-2,000 checkpoint of the first stage and undergoes full-parameter fine-tuning of the language model with dot-product similarity, with the learning rate reduced to 2×10−62× 10^-6. In both stages, the vision encoder and multimodal projector are frozen, and each stage is trained for one epoch on 8×NVIDIA H100 (80GB) GPUs using bfloat16 precision. Hyperparameter Stage 1 Stage 2 Training data 2.0M Training epochs 1 Batch size 32 Learning rate 2×10−52× 10^-5 2×10−62× 10^-6 Warmup ratio 0.05 Temperature 0.1 100 Trainable parameters LoRA (rank=32) Full LM Initialization Backbone Stage I step 2,000 Loss function Focal Embedding Loss (γ=2.0γ=2.0) Similarity metric Cosine Dot product Frozen modules Vision encoder, projector #Hard negatives 5 Max sequence length 8,192 Max image pixels 1,605,632 Optimizer AdamW Precision BF16 DeepSpeed Stage 2 GPU configuration 8×NVIDIA H100 (80GB) Table 3: Training hyperparameters and computational requirements for PACE (Qwen3.5-0.8B-Base and Qwen3.5-2B-Base). For the two fine-tuned baselines, VLM2Vec (Jiang et al. 2025) and UniME-V2 (Gu et al. 2026), both of which are cosine-similarity-based, we adopt the training configuration of our first stage, with both baselines trained under hyperparameter settings identical to those in Table 3. Their full training configurations are summarized in Table 4. GradCache is employed in neither PACE nor the two baselines; nevertheless, additional experiments presented later show that GradCache can be effectively applied to our approach. Hyperparameter VLM2Vec UniME-V2 Training data 2.0M Training epochs 1 Batch size 32 Learning rate 2×10−52× 10^-5 Warmup ratio 0.05 Temperature 0.1 Trainable parameters LoRA (rank=32) Initialization Backbone Loss function InfoNCE MLLM-judge InfoNCE Similarity metric Cosine Frozen modules Vision encoder, projector #Hard negatives 5 Max sequence length 8,192 Max image pixels 1,605,632 Optimizer AdamW Precision BF16 DeepSpeed Stage 2 GPU configuration 8×NVIDIA H100 (80GB) Table 4: Training hyperparameters and computational requirements for the retrained baselines VLM2Vec and UniME-V2. A.2 Instruction Prompts Every query and candidate is wrapped with the unified qwen3_5_emb chat template shown below, which injects the system-level instruction “Represent the user’s input.” automatically. Each query, positive, and hard negative is encoded independently under this template, and the hidden state at the last <|endoftext|> token is used for last-token pooling to obtain the embedding. Images and videos are inserted into the user content through <image> and <video> placeholders together with their corresponding media inputs. <|im_start|>system Represent the user’s input. <|im_end|> <|im_start|>user content <|im_end|> <|im_start|>assistant <|endoftext|> Task-level instructions are written directly into the user content of the query; Table 5 lists the instructions used for each task format. Unlike reranking-style prompting, which concatenates a query and a candidate into a single Yes/No judgment prompt, the embedding model encodes every instance separately and computes similarities between the resulting embeddings. All PACE models and retrained baselines share this unified template and instruction set. Task format Task-level instruction Visual question answering Represent the given image to answer the following question. Classification Represent the given image to answer the following question. Text→ retrieval Retrieve an image that best matches the following description. Image→ retrieval Find the caption that best describes this image. Composed retrieval Find an image matching the described changes. Table 5: Task-level instructions written into the user content of the query for each task format. The system-level instruction is injected automatically by the template. Appendix B Detailed Theoretical Analysis We provide the proof of Proposition 1. Under the common-norm setting, write =r^q=r q and j=R^jc_j=R c_j, where both directional vectors have unit norm, and let uj=^⊤^ju_j= q c_j. We treat the candidate embeddings as fixed. For notational simplicity, we omit the Stage I superscript and write pjp_j for pj(2)p_j^(2) throughout this section. We further define u¯=∑jpjuj u= _jp_ju_j; all probabilities and derivatives below are evaluated at the Stage I initialization after differentiation. The radial–angular gradient-component map (r,^)=(ad,d)G(r, q)=(a_d,b_d) has the block Jacobian D=[∂ad/∂rD^ad∂d/∂rD^d].DG= bmatrix∂ a_d/∂ r&D_ qa_d\\ _d/∂ r&D_ qb_d bmatrix. (B.1) The proposition bounds its two off-diagonal blocks at the Stage I initialization. Proof. Because pj=exp(rRuj/τ2)/∑kexp(rRuk/τ2)p_j= (rRu_j/ _2)/ _k (rRu_k/ _2), differentiating with respect to r while holding q fixed gives ∂pj∂r=Rτ2pj(uj−u¯). ∂ p_j∂ r= R _2p_j(u_j- u). (B.2) Since q and the candidate embeddings are held fixed, ^⟂P_ q and ^j c_j are independent of r. Therefore, ∂d∂r _d∂ r =Rτ2^⟂∑j∂pj∂r^j = R _2P_ q _j ∂ p_j∂ r c_j (B.3) =R2τ22^⟂∑jpj(uj−u¯)^j. = R^2 _2^2P_ q _jp_j(u_j- u) c_j. Since ∑jpj(uj−u¯)=0 _jp_j(u_j- u)=0, the centered sum can be rewritten as ∑jpj(uj−u¯)^j=∑j≠+pj(uj−u¯)(^j−^+). _jp_j(u_j- u) c_j= _j≠+p_j(u_j- u)( c_j- c_+). (B.4) The unit-norm assumptions imply |uj−u¯|≤2|u_j- u|≤ 2 and ‖^j−^+‖2≤2\| c_j- c_+\|_2≤ 2. Moreover, p+≥1−ρp_+≥ 1-ρ gives ∑j≠+pj≤ρ _j≠+p_j≤ρ. Because orthogonal projection is nonexpansive, ‖∂d∂r‖2≤4R2τ22ρ. \| _d∂ r \|_2≤ 4R^2 _2^2ρ. (B.5) For the reverse coupling, let ⟂^h q with ‖2=1\|h\|_2=1, and define vj=⊤^jv_j=h c_j and v¯=∑jpjvj v= _jp_jv_j. Along a smooth curve on the unit sphere with tangent h, D^uj[]=vjD_ qu_j[h]=v_j. Holding r=r0r=r_0 fixed, define the Stage I logit zj=βujz_j=β u_j, where β=r0R/τ2β=r_0R/ _2. Since D^zj[]=βvjD_ qz_j[h]=β v_j, the softmax differential gives D^pj[] D_ qp_j[h] =pj(βvj−β∑kpkvk) =p_j (β v_j-β _kp_kv_k ) (B.6) =βpj(vj−v¯). =β p_j(v_j- v). Applying the product rule to ad=(R/τ2)(∑jpjuj−u+)a_d=(R/ _2)( _jp_ju_j-u_+) gives D^ad[] D_ qa_d[h] =Rτ2[∑jD^pj[]uj = R _2 [ _jD_ qp_j[h]u_j (B.7) +∑jpjD^uj[]−D^u+[]] + _jp_jD_ qu_j[h]-D_ qu_+[h] ] =Rτ2[β∑jpj(vj−v¯)uj+∑jpjvj−v+]. = R _2 [β _jp_j(v_j- v)u_j+ _jp_jv_j-v_+ ]. Since ∑jpj(vj−v¯)uj=p[uv]−p[u]p[v]=Covp(u,v) _jp_j(v_j- v)u_j=E_p[uv]-E_p[u]E_p[v]=Cov_p(u,v) and ∑jpjvj=p[v] _jp_jv_j=E_p[v], this becomes D^ad[]=Rτ2[βCovp(u,v)+p[v]−v+].D_ qa_d[h]= R _2 [ _p(u,v)+E_p[v]-v_+ ]. (B.8) Under p+≥1−ρp_+≥ 1-ρ, Varp(u) _p(u) ≤∑j≠+pj(uj−u+)2≤4ρ, ≤ _j≠+p_j(u_j-u_+)^2≤ 4ρ, (B.9) Varp(v) _p(v) ≤∑j≠+pj(vj−v+)2≤4ρ. ≤ _j≠+p_j(v_j-v_+)^2≤ 4ρ. By Equation B.9 and the Cauchy–Schwarz inequality, |Covp(u,v)|≤4ρ|Cov_p(u,v)|≤ 4ρ. In addition, |p[v]−v+|=|∑j≠+pj(vj−v+)|≤2ρ. |E_p[v]-v_+ |= | _j≠+p_j(v_j-v_+) |≤ 2ρ. (B.10) Consequently, for every unit tangent direction h, |D^ad[]|≤Rτ2(4β+2)ρ. |D_ qa_d[h] |≤ R _2(4β+2)ρ. (B.11) Taking the supremum over such h proves ‖D^ad‖op≤Rτ2(4β+2)ρ, \|D_ qa_d \|_op≤ R _2(4β+2)ρ, (B.12) which completes the proof. The result controls the local off-diagonal Jacobian blocks of the radial–angular gradient-component map under the common-norm assumption. It does not imply global statistical independence or a global convergence guarantee. Appendix C Additional Experiments C.1 Effect of GradCache Method CLS VQA RET GRD Overall VLM2Vec 51.0 56.8 53.1 73.2 55.8 VLM2Vec‡ 50.7 61.6 55.6 73.8 57.9 PACE 52.3 60.4 57.4 77.2 59.0 PACE‡ 53.7 63.7 58.4 74.5 60.4 Table 6: Effect of GradCache on MMEB with Qwen3.5-0.8B-Base. ‡ denotes training with GradCache enabled. CLS: Classification, RET: Retrieval, GRD: Grounding. GradCache improves the overall score of both methods, with PACE retaining its lead. Best results are in bold and second-best are underlined. Effect of GradCache. As noted in the training details, GradCache is employed in neither PACE nor the retrained baselines. To verify its compatibility with our approach, we retrain PACE and VLM2Vec with GradCache enabled, with all hyperparameter settings kept unchanged except the temperature, which is reduced to one fifth of its original value, and report the per-category results in Table 6. GradCache lifts the overall score of both methods—from 59.0 to 60.4 for PACE and from 55.8 to 57.9 for VLM2Vec—with gains concentrated in VQA and retrieval, where the enlarged effective batch supplies more informative in-batch negatives. PACE with GradCache retains a 2.5-point overall lead over its VLM2Vec counterpart, which indicates that the benefit of the staged training is complementary to memory-efficient large-batch optimization rather than subsumed by it. The only regression is the grounding score of PACE (77.2 to 74.5), which nevertheless remains the highest among all configurations. C.2 Additional Geometry Analysis Figure 7: PCA visualizations of ℓ2 _2-normalized embeddings on four randomly resampled subsets, colored by the L2L_2-norm percentile of the original embeddings; circles and triangles denote queries and candidates. Across all subsets, cosine-based training yields intermixed colors, indicating low mutual information between directions and norms, while direct dot-product training reproduces the same norm-stratified layout, confirming that the entanglement pattern in Figure 2(a) is robust to sampling. PCA Visualization across Resampled Subsets. To verify that the qualitative pattern in Figure 2(a) is not an artifact of a particular sample, Figure 7 repeats the analysis on four subsets independently drawn from the evaluation data with different random seeds, applying the same procedure: embeddings are ℓ2 _2-normalized, projected by PCA, and colored by their original norm percentile. The observation is consistent across all four subsets. Under cosine-based training, embeddings with different norms remain intermixed in the projected space, and no norm-dependent structure emerges in any subset—directions carry little information about norms, implying low mutual information between the two factors. Under direct dot-product training, every subset reproduces the same norm-stratified layout, with high-norm embeddings concentrated in one projected region and low-norm embeddings in another. Moreover, the norm percentile varies smoothly along a coherent low-dimensional structure in the projected directional space, so that embedding magnitude is largely predictable from direction alone—a direct signature of the coupling between norms and angles. This stability across resampled subsets confirms that the angular–norm entanglement induced by direct dot-product optimization is a systematic property of the learned embedding space rather than a sampling artifact. Figure 8: Stage I training dynamics under different Stage I initializations. MMEB score is tracked along the second-stage dot-product training trajectory, with Mid-step start initialized from the Stage I checkpoint at step 4,000 and Final-step start from the fully converged checkpoint at step 7,847. Mid-step start converges stably to 58.3, while Final-step start deteriorates after step 4,500 (shaded region), leaving an 8.6-point gap at convergence. Staged Training Alleviates the Coupling. Figure 9: PCA visualizations of ℓ2 _2-normalized embeddings under direct dot-product training (left) and staged cosine→ training (right), colored by L2L_2-norm percentile. Direct dot-product training exhibits the norm-stratified low-dimensional structure analyzed above, while staged training restores intermixed colors—norms are no longer predictable from directions—indicating reduced mutual information between the two factors. Figure 9 compares direct dot-product training with the staged cosine→ schedule under the same visualization protocol. As in the preceding experiments, focal weighting is disabled, isolating the effect of the stage schedule itself. Direct dot-product training reproduces the coherent low-dimensional structure identified above, with the norm percentile varying smoothly along the projected directions. Under the staged schedule, this structure disappears: embeddings of different norms are intermixed across the projected space, and no norm-aligned gradient remains. The cosine→ transition alone therefore suffices to alleviate the angular–norm coupling induced by direct dot-product optimization, letting the model encode magnitude as a complementary signal rather than binding it to particular directions. C.3 Analysis of Stage Transition Stage-Transition Training Dynamics. To complement the summary in the main text, Figure 8 traces the full Stage I training trajectory for two representative initializations: Mid-step start (from the Stage I checkpoint at step 4,000) and Final-step start (from the fully converged checkpoint at step 7,847). As in the corresponding main-text ablation, this experiment disables focal weighting; the Stage I checkpoint steps therefore differ from the step-2,000 initialization used in the full PACE configuration. Mid-step start yields a smooth, monotonically increasing MMEB curve that plateaus around 58.3 after approximately 4,500 second-stage steps. The Final-step start, by contrary, peaks early at 55.8 within the first 1,000 steps, oscillates around 55 thereafter, and deteriorates with pronounced fluctuations after step 4,500, ultimately falling to 49.7, an 8.6-point gap relative to the Mid-step start at convergence. This divergence corroborates the hypothesis of overspecialization outlined in the main text. Stage I optimizes a cosine-based InfoNCE loss that projects embeddings onto the unit hypersphere, learning angular alignment while discarding norm information. A fully converged Stage I checkpoint has thoroughly committed to this norm-invariant geometry. When the objective switches to dot-product similarity in Stage I—requiring the model to additionally leverage embedding magnitude as a relevance signal—the optimization must simultaneously restructure the angular layout and recover norm sensitivity. The conflicting gradient signals destabilize training and lead to the degradation observed after step 4,500. In contrast, an intermediate Stage I checkpoint has acquired sufficient directional structure to provide a reliable semantic initialization, yet retains enough representational plasticity for the model to incorporate magnitude information smoothly under the expanded optimization space of Stage I. C.4 Sensitivity to the Focal Parameter γ After Stage I After Stage I 0.5 56.3 58.2 2.0 56.9 59.0 4.0 56.9 58.7 Table 7: Sensitivity to the focal parameter γ on MMEB using Qwen3.5-0.8B-Base. Scores denote average Precision@1 (%) after each training stage; the best result in each column is highlighted in bold. Table 7 reports the sensitivity of PACE to the focal parameter γ under otherwise identical training settings. Stage I performance varies by only 0.6 across the evaluated values. Stage I improves every configuration by 1.8–2.1, yielding final scores between 58.2 and 59.0. These results indicate that PACE exhibits limited sensitivity to γ within the evaluated range. The setting γ=2.0γ=2.0 achieves the highest final score of 59.0, while stronger modulation provides no additional benefit. We therefore adopt γ=2.0γ=2.0 as the default. Appendix D External Results Table 8 reports the per-dataset results of PACE and all compared methods on the MMEB benchmark. VLM2Vec VLM2Vec† UniME-V2 UniME-V2† PACE PACE† Classification (10 tasks) ImageNet-1K 55.6 72.6 58.2 72.0 59.6 72.6 N24News 65.7 69.8 59.4 59.3 72.0 75.5 HatefulMemes 49.9 57.3 45.8 43.2 49.1 58.4 VOC2007 74.4 71.1 81.3 83.4 77.3 83.6 SUN397 67.4 73.0 67.2 74.9 65.5 77.5 Place365∗ 35.0 42.3 34.7 41.0 35.0 40.3 ImageNet-A∗ 20.9 49.0 22.5 51.2 21.8 50.0 ImageNet-R∗ 72.9 88.6 75.6 87.4 76.0 88.5 ObjectNet∗ 61.6 75.3 63.4 77.9 59.0 74.6 Country-211∗ 6.8 14.6 8.7 17.0 7.6 15.2 All Classification 51.0 61.4 51.7 60.7 52.3 63.6 VQA (10 tasks) OK-VQA 47.2 56.7 48.7 54.2 51.2 60.9 A-OKVQA 41.0 48.9 43.9 49.6 44.3 52.7 DocVQA 91.6 92.4 93.5 94.4 94.0 95.2 InfographicsVQA 67.6 72.6 66.1 71.5 70.4 75.7 ChartQA 58.9 63.5 55.0 61.8 61.2 64.4 Visual7W 55.0 60.6 53.9 57.7 59.6 63.8 ScienceQA 29.9 40.5 31.9 45.6 29.2 43.7 VizWiz∗ 50.0 53.6 51.1 55.3 51.0 54.9 GQA∗ 52.3 48.0 65.1 65.4 62.0 62.1 TextVQA∗ 74.6 82.9 81.1 85.5 81.4 86.3 All VQA 56.8 62.0 59.0 64.1 60.4 66.0 Retrieval (12 tasks) VisDial 68.3 71.5 71.6 77.3 77.9 81.8 CIRR 42.6 55.5 32.1 49.3 41.8 51.6 VisualNewst2i 46.8 61.4 50.9 64.5 50.8 65.4 VisualNewsi2t 50.4 64.2 54.8 67.5 53.9 70.0 MSCOCOt2i 66.5 73.2 70.8 76.2 71.7 76.5 MSCOCOi2t 65.0 67.4 66.4 70.0 69.8 72.8 NIGHTS 64.7 65.3 62.5 66.6 65.1 67.1 WebQA 82.0 85.7 74.0 82.0 78.2 88.2 FashionIQ∗ 16.9 18.7 9.8 22.5 19.5 21.1 Wiki-S-NQ∗ 58.6 63.3 58.1 66.5 61.3 66.8 OVEN∗ 35.0 46.4 45.6 62.7 44.2 65.7 EDIS∗ 40.6 64.0 49.2 60.3 54.5 71.5 All Retrieval 53.1 61.4 53.8 63.8 57.4 66.5 Visual Grounding (4 tasks) MSCOCO∗ 60.8 60.3 62.1 65.1 63.2 69.7 RefCOCO∗ 81.1 85.0 84.2 89.5 86.5 91.7 RefCOCO-matching∗ 83.4 80.6 88.4 87.6 80.8 85.1 Visual7W-pointing∗ 67.7 68.9 65.4 78.9 78.2 80.8 All Visual Grounding 73.2 73.7 75.0 80.3 77.2 81.8 Final Score (36 tasks) All IND 59.5 66.2 59.4 66.0 62.1 69.9 All OOD 51.1 58.8 54.1 63.4 55.1 64.0 All 55.8 62.9 57.0 64.9 59.0 67.3 Table 8: Per-dataset results on the MMEB benchmark, with “All” denoting the average over all 36 datasets. Datasets marked with ∗ denote OOD. † indicates the Qwen3.5-2B-Base backbone; all other methods use Qwen3.5-0.8B-Base.