Paper deep dive
Representation Alignment from a Few Paired Data via Distilled Optimal Transport
Julien Kordes, Reuben Dorent
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/12/2026, 6:20:12 AM
Summary
The paper introduces AnchorDOT, a method for aligning two independently pretrained unimodal encoders using only a small number of paired anchors (N~100-500) while leveraging abundant unpaired data. The method projects representations into a common anchor-relative coordinate system, solves a global entropic optimal transport problem to match point clouds while mitigating hubness, and distills the resulting transport plan into lightweight inductive encoders. It outperforms existing approaches on image-text and molecule-text alignment benchmarks.
Entities (11)
Relation Signals (8)
AnchorDOT → evaluatedon → MS-COCO
confidence 95% · We evaluate on image-text ... alignment ... on MS-COCO
AnchorDOT → evaluatedon → ChEBI-20
confidence 95% · We evaluate on ... molecule-text alignment ... on ChEBI-20
AnchorDOT → evaluatedon → Flickr30K
confidence 95% · We evaluate on image-text ... alignment ... on Flickr30k
AnchorDOT → uses → Optimal Transport
confidence 95% · Our method ... matches the two point clouds with a single entropic optimal transport problem
Julien Kordes → affiliatedwith → Sorbonne Université
confidence 90% · Julien Kordes ... Affiliation: Sorbonne Université
Reuben Dorent → affiliatedwith → Sorbonne Université
confidence 90% · Reuben Dorent ... Affiliation: Sorbonne Université
AnchorDOT → mitigates → Hubness
confidence 90% · marginal constraints counteract the hubness that defeats nearest-neighbor matching
ASIF → comparedwith → AnchorDOT
confidence 85% · ASIF ... extended across modalities ... However, these non-parametric methods require very large anchor collections
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only $N \sim 100$--$500$ pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the $N$ paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same anchor budget on Flickr30k, MS-COCO and ChEBI-20.
Tags
Links
- Source: https://arxiv.org/abs/2610.12781v1
- Canonical: https://arxiv.org/abs/2610.12781v1
Trouble viewing inline? Open PDF directly →
Full Text
75,646 characters extracted from source content.
Expand or collapse full text
Representation Alignment from a Few Paired Data via Distilled Optimal Transport Julien Kordes Reuben Dorent Affiliation: Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière, 75013 Paris, France reuben.dorent@inria.fr Abstract Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only N∼100N 100–500500 pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the N paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same anchor budget on Flickr30k, MS-COCO and ChEBI-20. 1 Introduction Vision-language models such as CLIP (Radford et al., 2021) learn a shared cross-modal space from hundreds of millions of paired examples. However, such paired multimodal data is scarce, requiring costly manual annotation or specialized paired measurements. In contrast, unimodal data can be abundant, and strong unimodal encoders are readily available off the shelf (Caron et al., 2021; Reimers and Gurevych, 2019; Ross et al., 2022). Leveraging these encoders for multimodal tasks requires aligning their embedding space, a fundamental challenge: trained independently, on different objectives and data distributions, their spaces differ in dimension, scale, and geometry, with no known mapping between them. In this work, we investigate the extreme low-resource regime: aligning two pre-trained unimodal encoders using only a few hundred paired examples (N∼100–500N 100--500). Prior approaches to low-resource cross-modal alignment fall into three categories. 1) Anchor-relative methods express each representation as its vector of similarities to a set of anchors, i.e. paired multimodal representations, casting both modalities into a shared coordinate system. Moschella et al. (2022) introduced this approach for a single modality, and ASIF (Norelli et al., 2023) extended across modalities. However, these non-parametric methods require very large anchor collections, e.g. 1.6×1061.6× 10^6 anchors for ASIF. Moreover, inference is costly since the large anchor set must be stored to compute the anchor-relative coordinates. 2) Graph-based methods such as SUE (Yacobi et al., 2025) embed each modality independently through the spectrum of its neighborhood graph, whose edges are weighted by intra-modal distances, and use the anchors only to provide the initial linear alignment. While SUE operates at a low anchor budget (N≈500N≈ 500), it assumes that independently trained unimodal encoders converge toward latent spaces with comparable local geometry, a local form of the Platonic Representation Hypothesis (Huh et al., 2024). This assumption is strong: cross-modal representations agree locally on which points are close, but not on local distances (Gröger et al., 2026). 3) Recently, SOTAlign (Roschmann et al., 2026) also performs a linear alignment using the anchors and uses it as a teacher, regularizing a student on unpaired batches with an optimal-transport divergence derived from the linear alignment. Since this regularization enforces consistency with the teacher, SOTAlign inherits the limits of the linear alignment. To overcome these limitations, we introduce AnchorDOT, a three-stage alignment framework (Figure 1) that combines (i) the expressiveness of anchor-relative coordinates with (i) the global matching guarantees of optimal transport (OT) and (i) the inference efficiency of parametric distillation. First, we project unpaired collections from both modalities into a common anchor-relative coordinate space. Next, rather than performing nearest-neighbor matching, which suffers from the hubness problem in which a small number of embeddings are the nearest neighbors of many queries (Radovanović et al., 2010; Dinu et al., 2014), we formulate cross-modal alignment as a global entropic optimal transport problem over the unpaired pool. The transport problem’s marginal constraints determine the total mass each point receives, avoiding hubs. Finally, because the optimal transport plan is transductive and defined only on the unpaired pool, we distill the computed assignment plan into two compact, modality-specific student encoders. At inference time, new queries are embedded in a single forward pass, with no dependence on stored memory banks. Our contributions can be summarized as follows: (i) we align two frozen unimodal encoders from N∼100N 100–500500 paired anchors and distill the resulting transport plan into inductive encoders that require no reference collection at inference; (i) we show that pairing is needed only for the anchors, i.e. the two unpaired collections may be sampled independently from their marginals, with no latent pairing in the pool; and (i) holding the anchor-relative cost fixed, we show that enforcing both transport marginals leads to more accurate and significantly less hub-dominated correspondences than nearest neighbor and the one-step corrections used in retrieval. Figure 1: Overview of AnchorDOT. Step 1: modality-specific representations are expressed by their similarity profile to a small set of shared anchors, embedding them in a common anchor-relative coordinate space. Step 2: an entropic optimal transport coupling is solved between the two anchor-relative embedding spaces, recovering a soft cross-modal correspondence between points. Step 3: this coupling is used as a pseudo-pairing signal to train two lightweight encoders fθf_θ and gψg_ψ, via contrastive learning directly on the original pretrained embeddings, distilling the transport plan into a shared aligned space that requires no anchors and no optimal transport at inference time. 2 Related work Anchor-relative representations. Moschella et al. (2022) represent a point by its cosine similarities to a fixed anchor set, which makes the representation invariant to angle-preserving transformations. Since networks trained on the same data from different seeds differ approximately by such transformations, this enables zero-shot model stitching within a single modality. Similarly, Cannistraci et al. (2023) discover new anchors by solving a Sinkhorn problem between relative representations at each optimization step. Their method is validated only on same-modality and cross-lingual communication. Their coupling is used to discover anchors rather than to align. Relative representations have since been used across modalities. ASIF (Norelli et al., 2023) builds a common space without training and performs retrieval via a sparsified nearest-neighbor rule. Because each query is decided independently against the reference collection, the collection has to describe the space densely enough for the correct neighbor to be present and to match on raw similarity: ASIF uses 1.61.6 million reference pairs, and at 10410^4 pairs reaches only 18%18\% zero-shot ImageNet accuracy. It is also non-parametric, i.e. the whole collection is retained at inference. RELIT (Chen et al., 2023) builds the same anchor-relative representation, but only to mine pseudo-pairs by nearest-neighbor retrieval for pre-training a vision-language model, which is then fine-tuned on each task’s full labelled data (the full Flickr30k training split, ∼29 29k images). Our N anchors are instead the only paired supervision, and our matching is solved jointly over the pool. Concurrently, Kim and Park (2026) align frozen encoders under scarce paired data using token-level relative representations with learnable anchors, which are optimized such that cross-modal similarity patterns agree. Our anchors are instead a given set of paired examples, and the matching is a transport problem rather than a similarity-matching objective. Outside vision-language, Gu et al. (2022) guide an OT problem in heterogeneous domain adaptation with a relation profile over a handful of keypoints. Appendix C.3.2 compares its construction to ours in detail. In summary, anchor-relative methods work with a few hundred anchors within a single modality, but have so far relied on far more paired data across modalities. Alignment with little or no supervision. SUE (Yacobi et al., 2025) works at a comparable pair budget to ours but along a different axis: it computes a spectral embedding of each modality’s random-walk graph independently, uses the pairs only to fit a linear alignment between the two spectral embeddings via Canonical Correlation Analysis (CCA), and refines with an MMD residual network. Its premise is that the two point clouds carry similar intra-domain structure, so the pairs are asked only to fix an ambiguity between two spaces that already correspond; where that structural correspondence is weak, little supervision remains to recover it. CSA (Li et al., 2025) applies CCA directly to the paired unimodal features and compares items by a cosine weighted by the canonical correlations. It trains no network, but fits on the pairs alone, leaves the unpaired data unused, and is validated with thousands of pairs (55k on Flickr30k), an order of magnitude above our budget. SOTAlign (Roschmann et al., 2026) fits a linear teacher on the same few pairs and regularizes a student network toward it on unpaired batches via an OT-based divergence. SOTAlign thus refines an existing alignment rather than discovering one, unlike ours. Gholamzadeh and Sajid (2025) build a cross-modal cost from intra-modal distances routed through the known pairs, an inter-modal bridge cost, which consults the anchors only through a per-pair minimum and so discards the profile of similarities to all anchors. At tens of thousands of pairs, two to nearly three orders of magnitude above the budget we target, STRUCTURE (Gröger et al., 2025) aligns frozen encoders by regularizing towards each encoder’s neighborhood geometry, using intra-modal structure as a penalty rather than to build the cost. With no pairs at all, Gromov-Wasserstein alignment (Alvarez-Melis and Jaakkola, 2018) matches points by comparing intra-domain distances, MUSE (Conneau et al., 2017) aligns adversarially, and Wasserstein Procrustes (Grave et al., 2019) jointly estimates an orthogonal and a permutation matrix initialized by a convex relaxation. Its dictionary-seeded counterpart is the offline setting of Smith et al. (2017). All of these were developed and validated on two instances of the same modality (word embedding spaces of different languages). Gromov-Wasserstein does not require a shared dimension, but it matches on intra-domain geometry alone. Optimal transport as a distillation target. Using a Sinkhorn coupling as soft supervision for a network has already been proposed. OTTER (Wu et al., 2021) computes an entropic plan within each batch of noisy web-collected image-text pairs and distils it as soft labels for contrastive pre-training. SwAV (Caron et al., 2020) enforces a Sinkhorn equipartition on cluster assignments within each batch, using the code of one augmented view to supervise the prediction from the other. S-CLIP (Mo et al., 2023) transports unpaired images onto a few captioned images to build soft caption targets for CLIP. In each case, the similarity is available by construction: OTTER’s teacher encoders are trained on paired data, SwAV compares two augmentations of a single image, and S-CLIP compares images with images in a pretrained CLIP space. Between independently trained unimodal encoders no such similarity exists, and the anchor-relative coordinates supply it. Hubness. In high dimension, a few points, the hubs, are the nearest neighbors of many others (Radovanović et al., 2010). The usual corrections are one-step rescalings of the similarity matrix, applied at retrieval time (Smith et al., 2017; Conneau et al., 2017; Bogolin et al., 2022). Enforcing transport marginals instead constrains all assignments jointly. Section 3.3 relates the two. 3 Method 3.1 Problem setup and notation Let x∈ℝdxx ^d_x and y∈ℝdyy ^d_y denote representations obtained from two pre-trained, modality-specific encoders, and let p(x,y)p(x,y) denote their joint distribution, with marginals p(x)p(x) and p(y)p(y). Our supervision is a set of N paired anchors drawn from the joint distribution, =(xak,yak)k=1N∼p(x,y)⊗N,N∈[100,500].A=\(x_a_k,y_a_k)\_k=1^N p(x,y) N, N∈[100,500]. (1) Beyond the anchors we assume access to two unpaired collections drawn independently from the two marginals, u=xii=1n∼p(x)⊗n,u=yjj=1m∼p(y)⊗m,X_u=\x_i\_i=1^n p(x) n, _u=\y_j\_j=1^m p(y) m, (2) with N≪n,mN n,m. The two collections may differ in size and require no hidden pairing. Ground-truth correspondence is therefore limited to the N anchors. No map between ℝdxR^d_x and ℝdyR^d_y is given. The three steps below construct one: first a common coordinate system, then a correspondence within it, then an inductive model of that correspondence. Algorithm 1 presents the complete procedure. 3.2 Anchor-relative coordinate space A point’s absolute coordinates carry no meaning across modality-specific encoders. However, its similarity profile relative to a paired anchor set provides a modality-agnostic representation on a unified scale. We therefore express all cross-modal quantities through these anchors. Definition 1 (Relative representation). For a similarity simsim on ℝdR^d and an ordered anchor set Z=z1,…,zN⊂ℝdA_Z=\z_1,…,z_N\ ^d, the relative representation of z∈ℝdz ^d is ϕZ(z)=[sim(z,z1),…,sim(z,zN)]∈ℝN. _Z(z)= [\,sim(z,z_1),\,…,\,sim(z,z_N)\, ] ^N. (3) Applied to each modality over the anchor sets X=xakk=1NA_X=\x_a_k\_k=1^N and Y=yakk=1NA_Y=\y_a_k\_k=1^N: ϕX(x)=[sim(x,xa1),…,sim(x,xaN)],ϕY(y)=[sim(y,ya1),…,sim(y,yaN)]. _X(x)= [\,sim(x,x_a_1),\,…,\,sim(x,x_a_N)\, ], _Y(y)= [\,sim(y,y_a_1),\,…,\,sim(y,y_a_N)\, ]. (4) Since xakx_a_k and yaky_a_k form a multimodal anchor pair, coordinate k of ϕX(x) _X(x) and coordinate k of ϕY(y) _Y(y) measure similarity to the same concept. The two vectors are therefore directly comparable coordinate by coordinate, regardless of the dimensionality, scale, or geometry of the underlying encoders, and all subsequent cross-modal computations take place in ℝNR^N. Anchor-orthonormalized similarity. With unit-normalized embeddings and cosine similarity, coordinate k of the profile is the projection of x onto anchor k: ϕX(x)=AXx _X(x)=A_Xx, where AX∈ℝN×dxA_X ^N× d_x stacks the anchors as rows. Since the anchors are sampled from the data, many of them are expected to be along the same dominant directions, with the profile counting them several times. Formally, ⟨ϕX(x),ϕX(x′)⟩=Nx⊤MXx′ _X(x), _X(x ) =N\,x M_Xx with MX=AX⊤AX/NM_X=A_X A_X/N, instead of x⊤x′x x : directions along which many anchors lie are amplified relative to the others, and, for large N, this squares the spectrum of the embedding covariance (Appendix B.1). We therefore compensate for MXM_X within each modality, to a degree set by γ: simγ(x,x′)=x⊤MX−γ/2x′,ϕXγ(x)=AXMX−γ/2x,γ∈[0,2].sim_γ(x,x )=x M_X^-γ/2\,x , _X^γ(x)=A_XM_X^-γ/2\,x, γ∈[0,2]. (5) Coordinate k remains the similarity to anchor k. Setting γ=0γ=0 recovers plain cosine similarity. At γ=1γ=1, when MXM_X is invertible, ϕX1/N _X^1/ N is an exact isometric embedding of ℝdxR^d_x: ⟨ϕX1(x),ϕX1(x′)⟩=Nx⊤x′ _X^1(x), _X^1(x ) =N\,x x for every x,x′x,x , an orthonormalization (Löwdin) of the anchor matrix itself rather than of the embedding covariance (Appendix B.1). At γ=2γ=2, the profile instead compensates MXM_X twice, which whitens the embedding covariance and makes the comparison equivalent to an unregularized canonical correlation across modalities. Larger γ relies more heavily on directions poorly estimated from only N anchors. Small eigenvalues of MXM_X are clipped for numerical stability, and Y is treated in the same way with its own MYM_Y. We select γ with the procedure of Appendix D and use sim=simγsim=sim_γ for the remainder of this work. Placing the two modalities in one space makes them comparable, but does not say which unpaired yj∈uy_j _u corresponds to a given unpaired xi∈ux_i _u. That assignment is the matching problem of the next section, and the obvious rule for it, nearest neighbor, as used by ASIF’s retrieval rule and by RELIT’s pair-mining step (Norelli et al., 2023; Chen et al., 2023), typically fails due to hubness. 3.3 Global entropic optimal transport in the anchor-relative space Once both modalities are expressed in the common anchor-relative space ℝNR^N, cross-modal alignment reduces to a matching problem between two point clouds. Let ϕX(xi) _X(x_i) and ϕY(yj) _Y(y_j) denote the anchor-relative representations of the unpaired points of Equation 2, and define the cross-modal cost Cij=1−k(ϕXγ(xi),ϕYγ(yj)),i∈[n],j∈[m],C_ij=1-k ( _X^γ(x_i), _Y^γ(y_j) ), i∈[n],\ j∈[m], (6) where k is a similarity on ℝNR^N, such as cosine similarity. It acts on anchor-relative representations and is therefore distinct from the simsim of Section 3.2, which acts on modality-specific representations. We then solve a single entropy-regularized optimal transport problem over the entire unpaired pool, π⋆=argminπ∈Π(μ,ν)⟨π,C⟩−εH(π),Π(μ,ν)=π∈ℝ+n×m:πm=μ,π⊤n=ν,π \;=\; _π∈ (μ,ν)\; π,C \;-\; H(π), (μ,ν)= \π _+^n× m\;:\; 1_m=μ,\;\;π 1_n=ν \, (7) where H(π)=−∑ijπij(logπij−1)H(π)=- _ij _ij( _ij-1) is the entropy (Peyré and Cuturi, 2019), ε>0 >0 the regularization strength, and μ,νμ,ν the uniform marginals on the two sides. Why a global problem. Training-free retrieval methods compare each item against the reference collection (Norelli et al., 2023). Since nearest-neighbor matching in high dimension is dominated by hubs, a few points are typically matched with many queries (Radovanović et al., 2010). Equation 7 couples all assignments instead: its marginal constraints fix the total mass each point receives, such that every yjy_j receives exactly νj _j. Existing methods used to mitigate hubness in retrieval can be related to optimal transport. The inverted softmax (Smith et al., 2017) is the first half-step of Sinkhorn’s algorithm and normalizes a single marginal (Pan and others, 2025). CSLS (Conneau et al., 2017) instead subtracts a fixed penalty on each side, the average similarity to each point’s K nearest neighbors (Appendix B.2). Our coupling is thus a converged solution to these methods. Solving Equation 7 at collection scale. Since the entropic solution factorizes as π⋆ij=aie−Cij/εbjπ _ij=a_i\,e^-C_ij/ \,b_j, it is fully determined by the dual potentials (u,v)=(εloga,εlogb)(u,v)=( a, b), which occupy O(n+m)O(n+m) memory even when C does not. We thus compute C block-by-block on demand and never materialize it (Ye et al., 2026). The exact global coupling is therefore attainable for large pool sizes, and the problem stays rectangular, with independent marginals and no assumption that the two collections share an indexing, as Equation 2 requires. 3.4 From a transductive coupling to inductive encoders via distillation The coupling π⋆π is transductive: it is defined only over the points jointly considered when it was solved, so it cannot score an unseen point without re-solving Equation 7 against a reference collection. We therefore train two lightweight, modality-specific encoders, fθ:ℝdx→ℝdz,gψ:ℝdy→ℝdz,f_θ:R^d_x ^d_z, g_ψ:R^d_y ^d_z, (8) mapping raw (not relative) embeddings into a shared dzd_z-dimensional space, so that at inference time a new modality-specific representation can be embedded with a single forward pass through fθf_θ or gψg_ψ, with no dependence on the anchor set or on any reference collection. Training in chunks. The encoders are trained contrastively, so each point must be scored against a set of candidates, and the pool is too large for one softmax over all of it. We therefore draw chunks c=(cX,cY)c=(c_X,c_Y), with cX⊆[n]c_X [n] and cY⊆[m]c_Y [m] indexed independently and |cX|=|cY|=B|c_X|=|c_Y|=B, and take one gradient step per chunk. The transport itself is not chunked. Because the potentials determine π⋆π everywhere, the block a chunk needs is available in closed form, πij⋆=exp((ui+vj−Cij)/ε),i∈cX,j∈cY,π _ij= \! ((u_i+v_j-C_ij)/ ), i∈ c_X,\ j∈ c_Y, (9) requiring no further Sinkhorn iterations. Each chunk is therefore supervised by an exact restriction of the global plan, and the chunking is confined to the gradient estimate. Algorithm 1 Representation alignment from a few paired examples via distilled OT 1: Unlabeled pools u,uX_u,Y_u (drawn independently, Eq. 2); paired anchor set A, ||=N|A|=N; chunk size B; similarity simsim; regularization ε 2: Compute relative representations ϕX(xi) _X(x_i), ϕY(yj) _Y(y_j) for all unpaired points, using A ⊳ Sec. 3.2 3: Solve the global entropic OT problem over the full pool by streaming C in tiles, obtaining the dual potentials (u,v)(u,v) ⊳ Sec. 3.3, Eq. 7 4: Initialize encoders fθf_θ, gψg_ψ 5: while not converged do 6: Draw the next chunk c=(cX,cY)c=(c_X,c_Y) of size B, or the anchor chunk A with π()=Iπ^(A)=I ⊳ Sec. 3.4 7: Recover the exact block πij⋆π _ij, i∈cXi∈ c_X, j∈cYj∈ c_Y, in closed form from (u,v)(u,v) ⊳ Eq. 9 8: Compute distillation loss ℒ(c)(θ,ψ)L^(c)(θ,ψ) ⊳ Eq. 11–12 9: Update θ,ψθ,ψ by gradient descent 10: end while 11: return inductive encoders fθf_θ, gψg_ψ Soft-target distillation. Each point of the chunk is supervised by its own row or column of the restricted plan, normalized within the chunk. Define the row- and column-normalized plans π¯ij=πij⋆∑j′∈cYπij′⋆,π¯ij′=πij⋆∑i′∈cXπi′j⋆, π_ij= π _ij _j ∈ c_Yπ _ij , π _ij= π _ij _i ∈ c_Xπ _i j, (10) which redistribute, for each xix_i (respectively each yjy_j), the transport mass it receives over the candidates present in the chunk. A point that is ambiguous or poorly covered by the anchors simply yields a more diffuse target distribution rather than a degenerate one; every point of the chunk is both a source of supervision and a candidate for every other point’s softmax. Direction-asymmetric distillation loss. We minimize a soft-target contrastive loss in each direction separately, using the full chunk both as the set of supervised points and as the candidate pool against which the softmax is computed: ℒX→Y(c)(θ,ψ,s) ^(c)_X→ Y(θ,ψ,s) =−1|cX|∑i∈cX∑j∈cYπ¯ijlogexp(sfθ(xi)⊤gψ(yj))∑j′∈cYexp(sfθ(xi)⊤gψ(yj′)), =- 1|c_X| _i∈ c_X _j∈ c_Y π_ij\, (s\,f_θ(x_i) g_ψ(y_j) ) _j ∈ c_Y (s\,f_θ(x_i) g_ψ(y_j ) ), (11) ℒY→X(c)(θ,ψ,s) ^(c)_Y→ X(θ,ψ,s) =−1|cY|∑j∈cY∑i∈cXπ¯′ijlogexp(sgψ(yj)⊤fθ(xi))∑i′∈cXexp(sgψ(yj)⊤fθ(xi′)), =- 1|c_Y| _j∈ c_Y _i∈ c_X π _ij\, (s\,g_ψ(y_j) f_θ(x_i) ) _i ∈ c_X (s\,g_ψ(y_j) f_θ(x_i ) ), (12) with s=1/τ>0s=1/τ>0 a learned inverse-temperature scalar clamped to s≤100s≤ 100, following CLIP (Radford et al., 2021), and total objective ℒ(c)=12(ℒX→Y(c)+ℒY→X(c))L^(c)= 12(L^(c)_X→ Y+L^(c)_Y→ X). The anchor set enters as one further chunk with π()=Iπ^(A)=I, so the N known pairs also supervise the encoders directly as one-hot targets, recovering a standard InfoNCE loss on this chunk. Appendix A gives the training schedule and Appendix C reports further ablations. 4 Experiments In this section, we evaluate our method across a diverse set of experiments. We benchmark cross-modal retrieval against established baselines on two structurally different modality pairs, including image-text and molecule-text, at low anchor budget (N≤500N≤ 500). We isolate the contribution of the marginal constraint by comparing it against one-step hubness corrections from the bilingual lexicon induction literature, and we test whether our method needs a hidden pairing in the unpaired pool. Image-text. Following the experimental setting of Yacobi et al. (2025), we use Flickr30k (Young et al., 2014) and MS-COCO (Lin et al., 2014) image-caption pairs. Image embeddings are extracted with a frozen DINOv2 ViT-S/14 backbone (Caron et al., 2021). Caption embeddings are extracted with a frozen SBERT model, all-MiniLM-L6-v2 (Reimers and Gurevych, 2019). Molecule-text. We additionally use ChEBI-20 (Edwards et al., 2021), a dataset of molecule-description pairs combining PubChem compounds with ChEBI textual annotations. Molecule embeddings are extracted with a frozen MoLFormer-XL (Ross et al., 2022) encoder. Description embeddings are extracted with the same frozen SBERT model used for the image-text experiments, so that the text-side encoder is held fixed across modality pairs. General protocol. In every setting, all backbone encoders are frozen and embeddings are precomputed once. Only the lightweight encoders fθ,gψf_θ,g_ψ of Section 3.4 are trained. The architecture of the encoders is kept identical to the ones used by SOTAlign (a single LayerNorm with a Linear projection layer). We sample N=500N=500 paired anchors (Section 3.2) and hold out a test split never used during OT coupling or distillation. Unless otherwise noted, we use a pool of ∼29 29k items split into disjoint image-only and caption-only halves, and a held-out test set of 400400 pairs, following SUE’s protocol on Flickr30k. This test set is smaller than the standard Flickr30k 1K and MS-COCO 5K test sets, so our numbers are not comparable to the CLIP-era retrieval literature. We report recall@k in both retrieval directions for k∈1,5,10k∈\1,5,10\ on the held-out split, using nearest-neighbor search between fθ(xtest)f_θ(x_test) and gψ(ytest)g_ψ(y_test). Table 1: Cross-modal retrieval at N=500N=500 anchors, recall@k (%). All baselines are re-run under our protocol, with the same frozen backbones, anchors and test split, averaged over five seeds. Our re-run of SUE exceeds its published Flickr30k results at this budget (I2T R@1 7.807.80 vs. 4.254.25). Flickr30k MS-COCO Image → Text Text → Image Image → Text Text → Image Method R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 Contrastive 22.05 49.10 60.45 22.30 48.05 59.60 14.60 40.65 54.95 15.00 40.55 54.30 ASIF 10.35 29.10 42.45 9.55 29.80 43.05 15.15 48.65 69.95 14.20 48.80 69.70 CSA 0.90 3.15 5.60 0.95 3.55 6.10 1.30 5.55 9.40 1.35 5.70 8.95 SUE 7.80 24.50 37.50 7.35 26.80 38.05 9.90 36.60 55.80 9.10 37.25 56.35 SOTAlign 35.60 65.00 76.20 35.20 65.10 75.55 30.50 70.10 84.50 30.90 69.70 84.50 Ours 36.55 67.90 79.25 36.05 68.10 79.40 29.05 71.05 85.35 30.30 70.95 86.45 Retrieval performance. All competing methods are tested under our protocol, with the same backbones, anchors, unpaired pool (∼80 80k items on MS-COCO) and test split. Three use the N anchors only: ASIF (Norelli et al., 2023), CSA (Li et al., 2025), and a contrastive baseline, i.e. our encoders and loss trained on the anchor chunk alone, which isolates the contribution of the coupling. SUE (Yacobi et al., 2025) and SOTAlign (Roschmann et al., 2026) also use the unpaired pool, whose image and text parts are disjoint. On image-text (Table 1), our method is best in 1010 of 1212 columns, and SOTAlign and ours hold the top two positions in every column. Against SOTAlign, we lead by 0.90.9 R@11 points on Flickr30k and SOTAlign leads by 1.41.4/0.60.6 on MS-COCO, i.e. a few test queries per seed, so we consider the two tied at R@11. At R@55 and R@1010, we lead in all eight entries (+0.9+0.9 to +3.9+3.9). Distilling the coupling multiplies the contrastive baseline’s R@11 by 1.61.6–2.02.0, a gain that comes entirely from unpaired data. ASIF, which retrieves each query independently by nearest neighbor, is 1414–2727 R@11 points below ours. SUE is below the contrastive baseline at R@11 and R@55, and CSA, validated with thousands of pairs, stays near chance. Table 2: Molecule-text retrieval on ChEBI-20 at N=500N=500. Molecule → Text Text → Molecule Method R@1 R@5 R@10 R@1 R@5 R@10 Contrastive 16.10 37.90 48.20 15.10 36.15 47.05 ASIF 8.40 28.20 40.90 14.55 35.85 49.40 CSA 0.20 1.10 2.55 0.65 2.45 4.40 SUE 3.25 14.85 25.05 3.35 15.10 26.10 SOTAlign 20.30 47.15 60.60 20.30 47.80 62.20 Ours 22.10 49.75 63.25 22.05 51.00 63.70 Table 3: Zero-shot classification accuracy (%) on 700700 ImageNet images from 7 classes. Methods trained on Flickr30k or MS-COCO. Mean (std) over 5 seeds. Method Flickr30k MS-COCO SUE 72.26 (2.59) 63.97 (8.99) SOTAlign 87.51 (2.73) 95.34 (1.73) Ours 89.91 (3.89) 93.63 (3.33) CLIP 98.43 98.43 Molecule-text retrieval. On ChEBI-20 (Table 3), where no prior method operates at N≤500N≤ 500, our method is first in all six columns, 1.51.5–3.23.2 points ahead of SOTAlign. The gain over the contrastive baseline is smaller than on image-text (R@11 ×1.4× 1.4–1.51.5), and SUE falls furthest, to a fifth of the contrastive baseline’s R@11. Figure 2: Recall@10 (I2T/T2I mean) under two independent ablations on the pool composition. (a)–(b) Recall@1010 vs. number of anchor pairs N, at fixed pool and test split, over the full and low-N ranges respectively: (a) restricted to our method and the contrastive baseline, (b) all six methods of Table 1. (c) Recall@1010 vs. unpaired pool size at fixed N=500N=500 and test split. Figure 3: Recall@1010 on Flickr30k as a function of the overlap o, for SUE, SOTAlign and our method. Anchor budget and pool size. On Flickr30k (Figure 2b), our method consistently outperforms SOTAlign and the contrastive baseline across the evaluated anchor budgets. The gap with SOTAlign remains relatively stable as the number of paired anchors increases, while both methods improve with additional paired data. When the number of anchors becomes very large (a), our method and the contrastive baseline approach similar performance from N∼8kN 8k, reaching approximately 0.90.9 R@1010. This panel isolates the contribution of our OT-learned coupling on unpaired data which decreases as the pool of unpaired is replaced by paired data. In contrast, increasing the size of the unpaired pool (c) consistently benefits our method, which improves from approximately 0.630.63 R@1010 with 10210^2 unpaired samples to 0.800.80 with 3×1043× 10^4 samples. Hidden pairing influence. We vary the overlap o∈[0,1]o∈[0,1], the fraction of the unpaired pool whose image and caption come from the same underlying pair, at fixed N=500N=500, pool size and test split, and retrain all methods at every value (Figure 3). At o=1o=1, every unpaired item has its counterpart on the other side, although the pairing is not revealed. At o=0o=0, our default in the rest of the paper, the image-only and caption-only pools are disjoint, as when each modality is collected independently and no cross-modal pair exists beyond the N anchors. Recall@1010 is essentially unchanged across the whole range, so the methods are as effective without a hidden correspondence as with one. This is consistent with the fact that the unpaired collections can be drawn independently from the two marginals (Equation 2). Hubness reduction. We compare four matching rules on the same anchor-relative cost: (i) nearest neighbor, as in ASIF (Norelli et al., 2023), (i) the inverted softmax (Smith et al., 2017), (i) CSLS (Conneau et al., 2017), and (iv) our converged Sinkhorn coupling. The inverted softmax uses the same entropic kernel as our coupling but normalizes only the gallery side, so the gap between the two isolates the effect of enforcing both marginals. The rules are applied directly to the 400400 held-out test pairs, before any distillation. We measure hubness in both retrieval directions by the skewness of the k-occurrence distribution N10N_10 (Radovanović et al., 2010), the number of queries whose top-1010 contains each gallery item. On Flickr30k (Figure 4), our coupling has the lowest skewness in both directions (0.320.32/0.370.37, against 2.402.40/1.521.52 for nearest neighbor, 1.681.68/2.182.18 for the inverted softmax and 1.121.12/0.660.66 for CSLS) and the highest recall@1010 (0.800.80/0.800.80, against 0.670.67/0.640.64 for nearest neighbor and 0.760.76–0.770.77 for the two one-step corrections). Normalizing a single marginal is not enough: the inverted softmax even increases hubness over nearest neighbor in the text-to-image direction. Figure 4: (a) N10N_10 skewness (lower is better) and (b) recall@1010 of nearest neighbor, the inverted softmax, CSLS and our converged coupling, on the same anchor-relative cost between the 400400 held-out Flickr30k test pairs, in both retrieval directions. Error bars denote ±1± 1 std over 55 seeds. Zero-shot classification. Following Yacobi et al. (2025), we classify 700700 ImageNet images from seven categories (cat, dog, man, woman, mountain, shore, snow) with the encoders trained on Flickr30k or MS-COCO, without further training, by cosine similarity to the embeddings of the prompts “A photo of a ”. Our encoders never see these images, categories or the task during training. At the same anchor budget, they outperform SUE by 1818–3030 points and are comparable to SOTAlign, 2.42.4 points ahead on Flickr30k and 1.71.7 points below on MS-COCO (Table 3, examples in Appendix C.5.1). Further experiments. Beyond retrieval, we analyze the evolution of the participation ratio of the learned representation with γ (Appendix C.1), compare our method against classical unsupervised approaches (Appendix C.2), and ablate two choices underlying our matching cost (Appendix C.3): injecting a structural term via Fused Gromov–Wasserstein and replacing the N-anchor relative representation itself. We then show that the learned shared representation supports qualitative cross-modal generation (Appendix C.4). Finally, we evaluate zero-shot retrieval (Appendix C.5.2) and provide additional examples for zero-shot classification and classical retrieval (Appendices C.5.1 and C.6). 5 Conclusion and future work We investigated the extreme low-resource regime of aligning two frozen unimodal encoders from only N∼100N 100–500500 paired examples. Our framework expresses both modalities in anchor-relative coordinates, matches the unpaired collections with a global entropic optimal transport problem whose marginal constraints reduce hubness more than one-step corrections, and distills the resulting plan into two compact, modality-specific encoders. At the same anchor budget, it outperforms anchor-relative and graph-based methods by a wide margin, matches SOTAlign at R@11 on image-text and outperforms it elsewhere, and requires no hidden pairing in the unpaired pool. Zero-pair methods reach near-random recall on our encoders, and a Gromov-Wasserstein term on intra-modal geometry never improves our coupling: consistent with Gröger et al. (2026), independently trained encoders do not share enough structure to be aligned without supervision. Our method remains bounded by the quality of these encoders, and its uniform marginals assume that both collections come from the same joint distribution (Equation 2), which unbalanced optimal transport would relax. Finally, our anchors are drawn at random, and choosing which items to pair is a promising direction for domains such as medical imaging, where each pair requires expert annotation. Acknowledgments This work was supported by ANR through the “Investissements d’avenir” program (ANR-10-IAIHU-06, ANR-19-P3IA-0001, PRAIRIE 3IA Institute), the “France 2030” program (ANR-23-IACL-0008, PRAIRIE-PSAI). The ARAMIS Lab is affiliated with DIM C-BRAINS, funded by the Conseil Régional d’Ile-de-France. This work was performed using HPC resources from GENCI-IDRIS (Grant 2026-AD011017987). RD received Marie Skłodowska-Curie grant No. 101154248 (SafeREG). AI use statement In this work, we used generative AI tools to aid or polish writing: every paragraph of this paper was checked by AI and by us to look for writing mistakes or wrong phrasing. We also used generative AI tools for retrieval and discovery: the pipeline was entirely designed by us without the use of AI. We performed an initial literature review and analyzed relevant papers. However, we used AI to confirm implementation details of competing methods and to search for potential missing related work. Our related work section was drafted by us. We also used generative AI tools for research ideation and execution: we mainly used AI to test and execute our ideas. A significant part of our code was generated by AI and then manually checked to verify that it was consistent with our intent. Finally, we used generative AI tools to draft sections of the paper: some sections were first drafted by AI (results, appendix) and then revised by us to ensure they reflected our own thinking. Tables were done manually. Figures were generated locally. We take full responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI. Ethics statement We do not anticipate any ethical concerns arising from this work. References Alvarez-Melis and Jaakkola (2018) D. Alvarez-Melis and T. S. Jaakkola Gromov-wasserstein alignment of word embedding spaces. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §2. Bogolin et al. (2022) S. Bogolin, I. Croitoru, H. Jin, Y. Liu, and S. Albanie Cross modal retrieval with querybank normalisation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. Cannistraci et al. (2023) I. Cannistraci, L. Moschella, V. Maiorca, M. Fumero, A. Norelli, and E. Rodolà Bootstrapping parallel anchors for relative representations. In Tiny Papers Track, International Conference on Learning Representations (ICLR), Cited by: §2. Caron et al. (2020) M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Caron et al. (2021) M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin Emerging properties in self-supervised vision transformers. In International Conference on Computer Vision (ICCV), Cited by: §1, §4. Chen et al. (2023) C. Chen, P. Li, M. Sun, and Y. Liu Weakly supervised vision-and-language pre-training with relative representations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §2, §3.2. Conneau et al. (2017) A. Conneau, G. Lample, M. Ranzato, L. Denoyer, and H. Jégou Word translation without parallel data. arXiv preprint arXiv:1710.04087. Cited by: §B.2, §2, §2, §3.3, §4. Dinu et al. (2014) G. Dinu, A. Lazaridou, and M. Baroni Improving zero-shot learning by mitigating the hubness problem. arXiv preprint arXiv:1412.6568. Cited by: §1. Edwards et al. (2021) C. Edwards, C. Zhai, and H. Ji Text2Mol: cross-modal molecule retrieval with natural language queries. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §4. Gholamzadeh and Sajid (2025) A. Gholamzadeh and N. Sajid Model alignment using inter-modal bridges. arXiv preprint arXiv:2505.12322. Cited by: §C.3.1, §C.3, §2. Grave et al. (2019) E. Grave, A. Joulin, and Q. Berthet Unsupervised alignment of embeddings with wasserstein procrustes. In The 22nd International Conference on Artificial Intelligence and Statistics, p. 1880–1890. Cited by: §2. Gröger et al. (2026) F. Gröger, S. Wen, and M. Brbić Revisiting the platonic representation hypothesis: an aristotelian view. In International Conference on Machine Learning (ICML), Cited by: §C.3.3, §1, §5. Gröger et al. (2025) F. Gröger, S. Wen, H. Le, and M. Brbić With limited data for multimodal alignment, let the STRUCTURE guide you. External Links: 2506.16895 Cited by: §2. Gu et al. (2022) X. Gu, Y. Yang, W. Zeng, J. Sun, and Z. Xu Keypoint-guided optimal transport with applications in heterogeneous domain adaptation. Advances in Neural Information Processing Systems 35, p. 14972–14985. Cited by: §C.3.2, §C.3.2, §C.3, §2. Huh et al. (2024) M. Huh, B. Cheung, T. Wang, and P. Isola The platonic representation hypothesis. arXiv preprint arXiv:2405.07987. Cited by: §C.3.3, §1. Kim and Park (2026) S. Kim and Y. R. Park Learning relative representations for fine-grained multimodal alignment with limited data. External Links: 2605.16834 Cited by: §2. Li et al. (2025) P. Li S. Chinchali et al. Csa: data-efficient mapping of unimodal features to multimodal features. In International Conference on Learning Representations, Vol. 2025, p. 99459–99478. Cited by: §2, §4. Lin et al. (2014) T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick Microsoft COCO: common objects in context. In European Conference on Computer Vision (ECCV), Cited by: §4. Mo et al. (2023) S. Mo, M. Kim, K. Lee, and J. Shin S-CLIP: semi-supervised vision-language learning using few specialist captions. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Moschella et al. (2022) L. Moschella, V. Maiorca, M. Fumero, A. Norelli, F. Locatello, and E. Rodolà Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430. Cited by: §1, §2. Norelli et al. (2023) A. Norelli, M. Fumero, V. Maiorca, L. Moschella, E. Rodolà, and F. Locatello ASIF: coupled data turns unimodal models to multimodal without training. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2, §3.2, §3.3, §4, §4. Pan et al. (2025) Z. Pan et al. Hubness reduction with dual bank sinkhorn normalization for cross-modal retrieval. In Proceedings of the 33rd ACM International Conference on Multimedia, Cited by: §3.3. Peyré and Cuturi (2019) G. Peyré and M. Cuturi Computational optimal transport. now publishers, Inc. eBooks. External Links: Document Cited by: §3.3. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), Cited by: §1, §3.4. Radovanović et al. (2010) M. Radovanović, A. Nanopoulos, and M. Ivanović Hubs in space: popular nearest neighbors in high-dimensional data. Journal of Machine Learning Research 11, p. 2487–2531. Cited by: §1, §2, §3.3, §4. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using Siamese BERT-networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §1, §4. Roschmann et al. (2026) S. Roschmann, P. Krzakala, S. Mazelet, Q. Bouniot, and Z. Akata SOTAlign: semi-supervised alignment of unimodal vision and language models via optimal transport. arXiv preprint arXiv:2602.23353. Cited by: §1, §2, §4. Ross et al. (2022) J. Ross, B. Belgodere, V. Chenthamarakshan, I. Padhi, Y. Mroueh, and P. Das Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence 4, p. 1256–1264. Cited by: §1, §4. Smith et al. (2017) S. L. Smith, D. H. Turban, S. Hamblin, and N. Y. Hammerla Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03859. Cited by: §2, §2, §3.3, §4. Tov et al. (2021) O. Tov, Y. Alaluf, Y. Nitzan, O. Patashnik, and D. Cohen-Or Designing an encoder for stylegan image manipulation. ACM Transactions on graphics (TOG) 40 (4), p. 1–14. Cited by: §C.4. Vayer et al. (2019) T. Vayer, L. Chapel, R. Flamary, R. Tavenard, and N. Courty Optimal transport for structured data with application on graphs. In International Conference on Machine Learning (ICML), Cited by: §C.3. Wu et al. (2021) B. Wu, R. Cheng, P. Zhang, T. Gao, P. Vajda, and J. E. Gonzalez Data efficient language-supervised zero-shot recognition with optimal transport distillation. arXiv preprint arXiv:2112.09445. Cited by: §2. Yacobi et al. (2025) A. Yacobi, N. Ben-Ari, R. Talmon, and U. Shaham Learning shared representations from unpaired data. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: Figure 11, Figure 11, §C.4, §1, §2, §4, §4, §4. Ye et al. (2026) F. X.-F. Ye, X. Li, A. Yu, M. Chang, L. Chu, and D. Wertheimer FlashSinkhorn: IO-aware entropic optimal transport on GPU. External Links: 2602.03067 Cited by: §3.3. Young et al. (2014) P. Young, A. Lai, M. Hodosh, and J. Hockenmaier From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, p. 67–78. Cited by: §4. Appendix A Distillation details Anchors as a distinguished chunk. The anchor pairs A are appended as an additional, distinguished chunk with plan π()=Iπ^(A)=I (their correspondence being exactly known rather than inferred by transport). Properties of the distillation objective. ℒX→Y(c)L^(c)_X→ Y and ℒY→X(c)L^(c)_Y→ X (Equations 11-12) form a direction-asymmetric soft-target loss: the two directions use row- and column-normalized targets π¯ π, π¯′ π respectively, which differ unless π⋆π is symmetric. ℒX→Y(c)L^(c)_X→ Y reduces to a standard InfoNCE loss with hard, one-hot targets when π¯i,: π_i,: is one-hot for every i∈cXi∈ c_X, and more generally distills the residual uncertainty the transport plan assigns to ambiguous matches (e.g. two visually similar images with semantically close captions). Each chunk (including the anchor chunk) is visited exactly once per epoch, in an order reshuffled every epoch, and contributes a single stochastic-gradient step from its own ℒ(c)L^(c). The plan π⋆π is computed once before optimization begins and held fixed throughout training. Because fθf_θ and gψg_ψ operate directly on raw embeddings rather than on relative representations, the trained encoders no longer require access to the anchor set A, or to any reference collection, at inference time. This is the precise sense in which the distillation step removes the transductive constraint introduced in Section 3.3: the anchors are used only to supervise training, not to compute the final representation of a query. Figure 5: Recall comparison of the transductive OT teacher with how much is retained after distillation to the inductive encoders Appendix B Theory details B.1 Spectral distortion of anchor-relative profiles We clarify the spectrum-squaring effect referenced in Section 3.2 and its correction by anchor orthonormalization (Eq. 5). Setup. Let x∼p(x)x p(x) with second-moment matrix ΣX=[xx⊤]=USU⊤ _X=E[x ]=USU , S=diag(s1,…,sdx)S=diag(s_1,…,s_d_x), assumed positive definite, so that si>0s_i>0; for unit-normalized embeddings, ∑isi=trΣX=1 _is_i=tr\, _X=1. As in Section 3.2, the anchors xa1,…,xaN∼iidp(x)x_a_1,…,x_a_N p(x) are stacked as rows of AX∈ℝN×dxA_X ^N× d_x, and MX=AX⊤AX/NM_X=A_X A_X/N. Then [MX]=ΣXE[M_X]= _X and, by the strong law of large numbers, MX→ΣXM_X→ _X almost surely as N→∞N→∞. Since ΣX _X is positive definite, almost surely MXM_X is invertible for all sufficiently large N (for every N≥dxN≥ d_x if p has a density on the unit sphere). Eigenvalue clipping. In practice the eigenvalues of MXM_X are clipped below at τ=10−6τ=10^-6 times their mean, i.e. τ=10−6/dxτ=10^-6/d_x for unit-normalized anchors, before taking powers. Write max(MX,τI) (M_X,τ I) for MXM_X with its eigenvalues clipped at τ. It is a spectral function of MXM_X and therefore commutes with MXM_X, so the proof below goes through verbatim with MX1−γM_X^1-γ replaced by MXmax(MX,τI)−γM_X (M_X,τ I)^-γ. The two coincide whenever λmin(MX)≥τ _ (M_X)≥τ, which holds almost surely for large N if smin>τs_ >τ. We therefore state the results without clipping. Anchor orthonormalization at γ=1γ=1. AXMX−1/2=NAX(AX⊤AX)−1/2A_XM_X^-1/2= N\,A_X(A_X A_X)^-1/2 is N N times the polar factor of AXA_X, its symmetric (Löwdin) orthonormalization: its columns are orthonormal and, when N>dxN>d_x, its rows (the transformed anchors) form a tight frame of ℝdxR^d_x rather than an orthonormal set. Consequently ⟨ϕX1(x),ϕX1(x′)⟩=Nx⊤x′ _X^1(x), _X^1(x ) =N\,x x for all x,x′x,x : ϕX1/N _X^1/ N is an isometry of ℝdxR^d_x into ℝNR^N, exactly at every N with MXM_X invertible. Proposition 1 (Spectrum interpolation). For γ∈[0,2]γ∈[0,2], let ϕXγ(x)=AXMX−γ/2x∈ℝN _X^γ(x)=A_XM_X^-γ/2x ^N denote the profile induced by simγsim_γ of Eq. 5, and ΓXγ=x[ϕXγ(x)ϕXγ(x)⊤] _X^γ=E_x [ _X^γ(x) _X^γ(x) ] its second-moment matrix at fixed anchors. (i) For any anchors with MXM_X invertible, ΓXγ _X^γ has exactly dxd_x nonzero eigenvalues, counted with multiplicity, equal to those of NΣX1/2MX1−γΣX1/2N\, _X^1/2M_X^1-γ _X^1/2. In particular, for γ=1γ=1 they are exactly Nsii=1dx\Ns_i\_i=1^d_x. (i) With iid anchors as above, as N→∞N→∞, the sorted nonzero eigenvalues of ΓXγ/N _X^γ/N converge almost surely to the sorted si2−γi=1dx\s_i^2-γ\_i=1^d_x. Proof. (i) By definition of ΓXγ _X^γ, the anchors are fixed and the expectation is over x only, so ΓXγ=AXMX−γ/2[x⊤]MX−γ/2AX⊤=P⊤,P=AXMX−γ/2ΣX1/2∈ℝN×dx. _X^γ=A_XM_X^-γ/2\,E[x ]\,M_X^-γ/2A_X =P , P=A_XM_X^-γ/2 _X^1/2 ^N× d_x. The matrices PP⊤∈ℝN×NPP ^N× N and P⊤P∈ℝdx×dxP P ^d_x× d_x share their nonzero eigenvalues. Using AX⊤AX=NMXA_X A_X=NM_X and the fact that MX−γ/2M_X^-γ/2 commutes with MXM_X, P⊤P=ΣX1/2MX−γ/2(AX⊤AX)MX−γ/2ΣX1/2=NΣX1/2MX1−γΣX1/2,P P= _X^1/2M_X^-γ/2(A_X A_X)M_X^-γ/2 _X^1/2=N\, _X^1/2M_X^1-γ _X^1/2, which is positive definite as a congruence of the positive definite MX1−γM_X^1-γ by the invertible ΣX1/2 _X^1/2, so all its dxd_x eigenvalues are nonzero. This proves the identity. For γ=1γ=1, MX0=IM_X^0=I and P⊤P=NΣXP P=N _X, whose eigenvalues are Nsi\Ns_i\. (i) Almost surely, MXM_X is positive definite for all large N and MX→ΣXM_X→ _X. The map M↦M1−γM M^1-γ is continuous on positive definite matrices for any real exponent, so ΣX1/2MX1−γΣX1/2→ΣX2−γ _X^1/2M_X^1-γ _X^1/2→ _X^2-γ almost surely. By Weyl’s inequality, |λi(B)−λi(B′)|≤‖B−B′‖2| _i(B)- _i(B )|≤\|B-B \|_2 for the sorted eigenvalues of symmetric matrices B,B′B,B . Hence the sorted eigenvalues of ΓXγ/N _X^γ/N converge to those of ΣX2−γ=US2−γU⊤ _X^2-γ=US^2-γU , namely the sorted si2−γi=1dx\s_i^2-γ\_i=1^d_x. ∎ Proposition 1 recovers the reference points used above: γ=0γ=0 squares the spectrum (si2s_i^2), matching plain cosine similarity. γ=1γ=1 preserves its shape (si1=sis_i^1=s_i), exactly at every N, the exact isometry described above. γ=2γ=2 flattens it entirely in the limit (si0=1s_i^0=1), equivalent to fully whitening ΣX _X. Only the γ=1γ=1 case is exact at every finite N (Statement (i)). For γ≠1γ≠ 1, Statement (i) is an asymptotic approximation, accurate for the leading eigenvalues once N is large relative to dxd_x Remark 1 (Effective dimensionality). The participation ratio PR(λi)=(∑iλi)2/∑iλi2PR(\ _i\)=( _i _i)^2/ _i _i^2 of the limiting spectrum si2−γ\s_i^2-γ\ increases monotonically with γ∈[0,2]γ∈[0,2]. B.2 Relation to CSLS. CSLS [Conneau et al., 2017], another one-step hubness correction used in retrieval, shares the additive structure of one full Sinkhorn iteration. Writing ui=εlogaiu_i= a_i and vj=εlogbjv_j= b_j, taking εlog(⋅) (·) of the factorized coupling πij=aiKijbj _ij=a_iK_ijb_j with Kij=exp(simij/ε)K_ij= (sim_ij/ ) gives, at any iteration t, εlogπij(t)=simij+vj(t)+ui(t). π^(t)_ij=sim_ij+v_j^(t)+u_i^(t). (13) This matches the shape of CSLS(xi,yj)=2cos(xi,yj)−rT(xi)−rS(yj)CSLS(x_i,y_j)=2 (x_i,y_j)-r_T(x_i)-r_S(y_j): a raw similarity corrected by one additive term per side, rSr_S and rTr_T being the average similarity to the K nearest neighbors in the opposite space. We make this resemblance precise by writing out one full Sinkhorn iteration both potentials updated in parallel from a(0)≡b(0)≡1a^(0)≡ b^(0)≡ 1 (absorbing the constant logμi,logνj _i, _j terms): vj(1)=−εLSEi(simij/ε),ui(1)=−εLSEj(simij/ε),v_j^(1)=- \,LSE_i\! (sim_ij/ ), u_i^(1)=- \,LSE_j\! (sim_ij/ ), (14) with LSEi(zi)=log∑iexp(zi)LSE_i(z_i)= _i (z_i). Substituting into Equation 13 at t=1t=1 gives εlogπij(1)=simij−εLSEj(simij/ε)−εLSEi(simij/ε), π^(1)_ij=sim_ij- \,LSE_j(sim_ij/ )- \,LSE_i(sim_ij/ ), Fix j, write zi=simijz_i=sim_ij and let Mj=maxiziM_j= _iz_i. Factoring exp(Mj/ε) (M_j/ ) out of the sum inside the LSELSE, εLSEi(zi/ε)=Mj+εlogSj(ε),Sj(ε)=∑iexp(zi−Mjε)∈[1,n]. \,LSE_i(z_i/ )=M_j+ S_j( ), S_j( )= _i \! ( z_i-M_j )∈[1,n]. Since logSj(ε) S_j( ) is bounded uniformly in ε , εlogSj(ε)→0 S_j( )→ 0 and εLSEi(zi/ε)→Mj \,LSE_i(z_i/ )→ M_j as ε→0 → 0. The same argument applies symmetrically to ui(1)u_i^(1), giving vj(1)→−maxisimijv_j^(1)→- _isim_ij and ui(1)→−maxjsimiju_i^(1)→- _jsim_ij. Substituting back into Equation 13 at t=1t=1, εlogπij(1)=simij⏟cos(xi,yj)−maxj′simij′−maxi′simi′j⏟−rT(xi)−rS(yj)−(εlogSi(ε)+εlogSj(ε))⏟→0 as ε→0 π^(1)_ij= sim_ij_ (x_i,y_j) -max_j sim_ij -max_i sim_i j_-r_T(x_i)-r_S(y_j)- ( S_i( )+ S_j( ) )_→~0 as → 0 exactly the CSLS shape with coefficient 11 instead of 22, K=1K=1 and an additive f(ε)=−εlogSi(ε)−εlogSj(ε)f( )=- S_i( )- S_j( ) function that vanishes as ε→0 → 0. Appendix C Additional experiments C.1 Participation ratio (a) N=500N=500 (b) N=2000N=2000 (c) N=4000N=4000 Figure 6: Participation ratio as a function of γ evaluated on Flickr30k for three anchor budgets N. Figure 6 reports the participation ratio of the anchor-relative profile on Flickr30k, for both modalities, as γ is varied from 00 to 22, at three anchor budgets N∈500,2000,4000N∈\500,2000,4000\. At N=2000N=2000 and N=4000N=4000, PRPR increases monotonically with γ for both modalities, as predicted by the Remark 1, saturating near its asymptotic ceiling dxd_x as γ→2γ→ 2. At N=500N=500, however, the curve peaks around γ≈1.5γ≈ 1.5 and drops sharply toward γ=2γ=2. This is consistent with a finite-N estimation bias rather than a failure of the underlying result: the negative exponent on MXM_X at γ>1γ>1 disproportionately amplifies its smallest eigenvalues, which are also the ones N anchors estimate least reliably (Appendix B.1). At N=500N=500 against dx≈384d_x≈ 384, a handful of poorly-estimated directions can dominate the empirical spectrum once inverted, depressing PRPR well below its asymptotic value. As N grows, MXM_X estimates ΣX _X more accurately in exactly these tail directions, and the curve converges to the monotonic prediction of Proposition 1. This is precisely why we restrict γ to [0,1][0,1] in practice (Appendix D): at our target budget of a few hundred anchors, γ>1γ>1 risks amplifying anchor-estimation noise rather than correcting a real bias, well before reaching its asymptotic behavior. C.2 Comparison to unsupervised methods Figure 7 evaluates the zero-pair setting: Gromov-Wasserstein, Wasserstein Procrustes and Iterative Closest Point align the two point clouds using only their intra-modal geometry, with no anchors at all. All three collapse to near-random recall. This gap is not a failure of the matching algorithms themselves, which are well established. However, it indicates that the intra-modal geometries of our frozen image and text encoders are not yet close enough to be matched on structure alone. Unlike two instances of the same modality (e.g. word embeddings of different languages), where such zero-pair methods are known to succeed, image and text encoders trained independently offer not enough shared geometric signal for these methods to exploit. A handful of paired anchors is therefore a necessity in this regime. Figure 7: Recall comparison of our method to existing unsupervised baselines on Flickr30k with the same usual setting C.3 Choice of the optimal transport plan We now focus on the cost used to build our coupling, and on how effectively it exploits the information available beyond the N anchors. Sections C.3.1 and C.3.2 respectively compare our anchor-relative representation against the bridge cost of Gholamzadeh and Sajid [2025], an alternative construction that routes cross-modal distance through the anchors, and the cost of Gu et al. [2022], another construction that represents each point by a softmax-normalized distribution over its distances to the anchors, compared across modalities via Jensen-Shannon divergence. Then the section C.3.3 studies the effect of adding a structural constraint on top of our coupling through a Fused Gromov-Wasserstein [Vayer et al., 2019] formulation, which interpolates between our anchor-relative cost and a Gromov-Wasserstein penalty on the intra-modal geometry. C.3.1 Bridge-cost Figure 8: Hubness and recall@10 comparison on Flickr30k and MS-COCO for the Bridge cost against our method. The N10N_10 skewness gives similar results while the recall@10 drops considerably when using the Bridge cost suggesting that our method benefits from representing a point from its similarity to all anchors and not only the closest one. As an alternative to our anchor-relative profile, we consider the bridge cost of Gholamzadeh and Sajid [2025], which also builds a cross-modal transport cost solely from distances to the N anchors, but combines them differently. For a query x in modality X and y in modality Y, the bridge cost is Cbridge(x,y)=mini∈1,…,NdX(x,xai)+dY(yai,y),C_bridge(x,y)= _i∈\1,…,N\\;d_X(x,x_a_i)+d_Y(y_a_i,y), the length of the shortest two-hop path from x to y routed through a single anchor xaix_a_i. This differs from our profile representation, which embeds every point into the full N-dimensional coordinate system given by its distance to every anchor and compares two such coordinate vectors directly (e.g. by cosine distance), rather than selecting a single best anchor per pair. We plug this cost into the same Sinkhorn coupling used throughout the paper, changing nothing else, and compare the resulting retrieval performance and hubness to our own cost in Figure 8. C.3.2 KPG-RL cost Figure 9: Hubness and recall@10 comparison on Flickr30k and MS-COCO for the KPG-RL cost against our method. Both N10N_10 skewness and recall@10 give similar results with our method getting slightly better performances on the retrieval task As a further alternative, we consider the relation profile of Gu et al. [2022], which also builds a cross-modal transport cost from distances to the N anchors, but combines them differently. For a point z in either modality, the relation score is the softmax-normalized distribution over its distances to the anchors, RZ(z)k=exp(−dZ(z,ak)/τ)∑k′=1Nexp(−dZ(z,ak′)/τ),k∈1,…,N,R_Z(z)_k= (-d_Z(z,a_k)/τ) _k =1^N (-d_Z(z,a_k )/τ), k∈\1,…,N\, and the cross-modal cost compares the two distributions by Jensen-Shannon divergence, Ckpg(x,y)=JS(RX(x)∥RY(y))C_kpg(x,y)=JS (R_X(x)\,\|\,R_Y(y) ). This differs from our profile representation in two respects: the anchor coordinates are pre-normalized into a probability distribution rather than compared as a raw similarity vector, and the two profiles are compared by a divergence between distributions rather than a similarity. Gu et al. [2022] was designed for a different task and a far smaller anchor budget: heterogeneous domain adaptation between two feature extractors on the same visual data, guided by only 11 to 33 keypoints per class, rather than cross-modal retrieval with a few hundred paired anchors. It further imposes a mask constraining the coupling to match each anchor pair exactly within the transport problem itself, a hard constraint our transport plan does not impose. We plug the cost CkpgC_kpg, without the mask, into the same Sinkhorn coupling, doing the same analysis as in section C.3.1 and results are reported on Figure 9. C.3.3 Fused-Gromov-Wasserstein Recent critiques of the Platonic Representation Hypothesis [Huh et al., 2024] argue that the predicted convergence holds only partially in practice, and that the two modalities may have fundamentally different intra-modal geometries [Gröger et al., 2026]. Our Fused Gromov-Wasserstein ablation offers a targeted piece of evidence for this view within our own setting. We interpolate between the structural and feature-cost terms via ℒFGW(α)=(1−α)ℒEGW+αℒEOT-anchors,α∈[0,1],L_FGW(α)=(1-α)\,L_EGW+α\,L_EOT-anchors, α∈[0,1], where ℒEGWL_EGW is the entropic Gromov-Wasserstein term matching intra-modal pairwise distances, and ℒEOT-anchorsL_EOT-anchors is our anchor-relative feature-cost objective. The structural term never improves retrieval (Figure 10): recall increases with α and is highest at or near α=1α=1, i.e. when the structural term is dropped entirely and the coupling reduces exactly to our feature-cost OT. This suggests that the intra-modal geometries of images and text are not shared closely enough to be exploited through structural alignment alone. Figure 10: Recall@k as a function of the weight α in the FGW setting C.4 Qualitative cross-modal generation To probe whether the shared embedding space supports generation and not only retrieval, we reproduce the “(almost-) text-free” generation protocol of Yacobi et al. [2025] on caption-FFHQ. A pretrained StyleGAN2 generator and its GAN-inversion encoder (e4e, Tov et al., 2021) are kept entirely fixed; we train only a lightweight MLP converter that maps a universal image embedding fθ(x)f_θ(x) to the corresponding StyleGAN2 latent code, using images only. At inference, a text query is instead projected by gψg_ψ into the same shared space and passed through this same, image-only-trained converter and then to the GAN decoder. Generation from text alone is possible only because the coupling that aligns fθf_θ and gψg_ψ makes a text embedding indistinguishable, to the converter, from an image embedding. Figure 11: Examples of images generated from a custom caption following Yacobi et al. [2025]’s generation protocol Generated faces (Figure 11) are visually coherent and capture the coarse semantic content of the caption (hair color, expression, apparent age, glasses, facial hair), but fine or spatially localized attributes are imperfect: captions mentioning a hat rarely yield a recognizable texture, though the decoder consistently adds a silhouette above the head. C.5 Zero-shot transfer C.5.1 Zero-shot classification Figure 12 shows examples for the zero-shot classification experiment of Section 4 (Table 3). Figure 12: Examples of zero-shot classification on ImageNet comparing our method to SUE, SOTAlign (all trained on Flickr30k) and a pretrained CLIP checkpoint C.5.2 Zero-shot retrieval This experiment tests whether the alignment learned by our method transfers to a new data distribution. We fit each method on ∼29 29k unpaired sampled from MS-COCO and N=500N=500 anchors and then evaluate recall@10 directly on Flickr30k, a dataset never seen during training or hyperparameter selection. No further fine-tuning, calibration, or optimization is performed on Flickr30k: the frozen unimodal encoders and the learned alignment heads (fitted on MS-COCO) are directly applied to encode its images and captions, and retrieval is measured on a held-out query set. This is a substantially harder test than the standard held-out split, since MS-COCO and Flickr30k differ in image content, caption style, length and vocabulary. As expected, all three methods degrade under this distribution shift but the recall remains comparable to that of Table 1 (3/4 of it is kept), meaning that the alignment learned on a different distribution can be transferred to a new one. Our method remains the strongest in absolute terms in the zero-shot setting (59.7%59.7\% against 56.7%56.7\% and 24.3%24.3\%) , preserving the same ranking observed in-distribution (79.3%79.3\% against 75.9%75.9\% and 37.8%37.8\%). Figure 13: Comparison of our method, SUE and SOTAlign on zero-shot retrieval on Flickr30k after being trained on 30k30k samples from MS-COCO C.6 Retrieval examples Figure 14: Text to Image retrieval examples on the dataset Flickr30k Figure 15: Image to Text retrieval examples on the dataset MS-COCO Figure 16: Text to Molecule retrieval examples on the dataset ChEBI-20 C.7 Complement Table 4: Complement of Table 1 with std over 5 seeds for Flickr30k. Image → Text Text → Image Method R@1 R@5 R@10 R@1 R@5 R@10 Contrastive 22.05 (2.21) 49.10 (2.48) 60.45 (2.99) 22.30 (2.76) 48.05 (3.07) 59.60 (3.20) ASIF 10.35 (0.57) 29.10 (2.68) 42.45 (3.24) 9.55 (0.82) 29.80 (2.17) 43.05 (1.56) CSA 0.90 (0.45) 3.15 (1.28) 5.60 (1.27) 0.95 (0.65) 3.55 (0.84) 6.10 (0.96) SUE 7.80 (1.51) 24.50 (1.71) 37.50 (0.85) 7.35 (0.70) 26.80 (1.00) 38.05 (1.08) SOTAlign 35.60 (2.05) 65.00 (1.92) 76.20 (3.29) 35.20 (2.31) 65.10 (2.06) 75.55 (2.27) Ours 36.55 (1.66) 67.90 (2.96) 79.25 (2.47) 36.05 (3.99) 68.10 (2.23) 79.40 (2.10) Table 5: Complement of Table 1 with std over 5 seeds for MS-COCO. Image → Text Text → Image Method R@1 R@5 R@10 R@1 R@5 R@10 Contrastive 14.60 (0.58) 40.65 (2.41) 54.95 (1.45) 15.00 (1.43) 40.55 (2.69) 54.30 (2.64) ASIF 15.15 (1.04) 48.65 (2.93) 69.95 (2.42) 14.20 (2.28) 48.80 (2.26) 69.70 (2.00) CSA 1.30 (0.73) 5.55 (1.47) 9.40 (1.39) 1.35 (1.01) 5.70 (1.68) 8.95 (1.91) SUE 9.90 (0.72) 36.60 (1.34) 55.80 (3.95) 9.10 (1.07) 37.25 (1.56) 56.35 (2.31) SOTAlign 30.50 (1.93) 70.10 (1.86) 84.50 (2.46) 30.90 (1.19) 69.70 (2.78) 84.50 (2.08) Ours 29.05 (1.32) 71.05 (1.16) 85.35 (2.08) 30.30 (1.27) 70.95 (2.09) 86.45 (1.60) Table 6: Complement of Table 3 with std over 5 seeds for ChEBI-20. Molecule → Text Text → Molecule Method R@1 R@5 R@10 R@1 R@5 R@10 Contrastive 16.10 37.90 48.20 15.10 36.15 47.05 ASIF 8.40 (0.91) 28.20 (1.52) 40.90 (2.28) 14.55 (0.60) 35.85 (1.46) 49.40 (0.91) CSA 0.20 (0.20) 1.10 (0.84) 2.55 (0.51) 0.65 (0.52) 2.45 (1.08) 4.40 (1.52) SUE 3.25 (0.77) 14.85 (2.71) 25.05 (3.83) 3.35 (1.01) 15.10 (1.65) 26.10 (3.52) SOTAlign 20.30 (1.02) 47.15 (2.33) 60.60 (0.76) 20.30 (1.43) 47.80 (2.03) 62.20 (0.93) Ours 22.10 (1.27) 49.75 (1.68) 63.25 (2.09) 22.05 (0.92) 51.00 (2.35) 63.70 (1.81) Table 7: Training wall-clock time (seconds) on Flickr30k, averaged over 3 seeds. We vary either the number of anchors N while fixing M=2000M=2000, or the number of unpaired points M while fixing N=500N=500. Speedup is measured relative to SOTAlign. Setting Value Ours SOTAlign Speedup Varying number of anchors (M=2000M=2000) N=100N=100 5.9 ± 3.4 77.1 ± 3.5 13.07× N=500N=500 5.0 ± 0.4 75.0 ± 0.2 15.00× N=1000N=1000 7.9 ± 0.5 76.0 ± 0.2 9.62× N=5000N=5000 29.2 ± 6.8 99.6 ± 0.3 3.41× Varying number of unpaired points (N=500N=500) M=2000M=2000 11.05 ± 7.12 80.26 ± 7.76 7.26× M=5000M=5000 21.93 ± 1.33 305.99 ± 0.75 13.95× M=15000M=15000 180.0 ± 23.67 2305.78 ± 4.06 12.81× Appendix D Hyperparameters Beyond the anchor budget N and the pool size, our method involves three groups of hyperparameters: standard encoder-training hyperparameters, the entropic regularization of the Sinkhorn coupling, and the anchor-relative profile itself. The first group is fixed across datasets and seeds and summarized in Table 8; only some of the Sinkhorn and profile hyperparameters depend on the seed and dataset and are selected with the held-out procedure described below. Table 8: Training hyperparameters, fixed across datasets and seeds. Hyperparameter Value Projection dimension 1024 Optimizer Adam Learning rate 10−310^-3 Weight decay 10−210^-2 Warmup steps 200 Epochs 200 Temperature initialization 0.070.07 Chunk size 2000 Sinkhorn max iterations 1000 Sinkhorn tolerance 0.0001 Sinkhorn regularization. The Sinkhorn coupling built on the anchor-relative profile is regularized by ε=η⋅median(C) =η·median(C), where C is the anchor-relative cost matrix and η a single scalar multiplier, selected per dataset and seed (Table 9). Because median(C)median(C) is recomputed from data every time, η alone suffices to adapt ε to the scale of C for every dataset and every candidate profile – ε itself is never tuned directly. Profile hyperparameters. The anchor-relative profile is built from γ-interpolated orthonormalized embeddings: each modality is whitened independently before a point is compared to each anchor by cosine similarity, with γ∈[0,1]γ∈[0,1] interpolating between the untouched profile (γ=0γ=0) and an unmodified spectrum (γ=1γ=1). Together with η above, γ is the only other hyperparameter we select per dataset and seed (Table 9); every other aspect of the profile construction is fixed across datasets. Selecting the profile hyperparameters. For every seed, we split the N anchors 90/1090/10: the larger split defines the anchor-relative representation, while the OT coupling is solved, for each candidate (η,γ)(η,γ) on a small grid, over the unpaired pool together with the held-out 10%10\% of anchors, with everything else about the profile fixed, so that only (η,γ)(η,γ) vary between candidates. For every held-out anchor we record the rank of its true match in the resulting coupling, in both directions, and keep the candidate minimizing the mean held-out rank (Fig. 17). With (η,γ)(η,γ) fixed, we use this same configuration for the corresponding distillation run, over the full set of N anchors. Table 9 reports the resulting choice per dataset for one seed (results are very consistent across different seeds of the same dataset but may vary). Figure 17: Held-out rank (mean, both retrieval directions), N10N_10 skewness and recall@1 as a function of the epsilon multiplier η, one curve per γ value. The selected (γ⋆,η⋆)(γ ,η ) minimizes this held-out rank. Table 9: Selected profile hyperparameters per dataset (seed 0) Dataset γ⋆γ η⋆η Flickr30k 0.75 0.02 MS-COCO 0.75 0.02 ChEBI-20 1.00 0.02