Paper deep dive
Controlling Embedding Spaces with Text-Conditioned Transformations
Joseph Fioresi, Fabian Caba Heilbron, Pankaj Nathani, Mubarak Shah, Kushal Kafle
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification. These embeddings compress high-level semantics into a single vector, which comes at the cost of primarily expressing a dominant semantics like main object while suppressing other important attributes such as camera angle or color tone. We propose a text-conditioned transformation of visual embeddings that makes such attributes explicitly accessible. Given a natural language description of an attribute category (e.g., "color" or "art style"), a network generates an affine transformation that emphasizes the specified attribute. Conditioning on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface. The network is trained to align transformed embeddings with the frozen latent space, enabling retrieval using existing large-scale embeddings without any re-encoding. When applied to a full set, the same mechanism transforms the latent space for attribute disentanglement tasks such as multi-clustering. By operating directly in latent space, our method provides a unified and efficient framework for controlling embedding spaces, demonstrating state-of-the-art performance across both attribute-based retrieval and multi-attribute organization tasks with near-zero inference cost. Project page: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2607.22919v1
- Canonical: https://arxiv.org/abs/2607.22919v1
Trouble viewing inline? Open PDF directly →
Full Text
84,500 characters extracted from source content.
Expand or collapse full text
Controlling Embedding Spaces with Text-Conditioned Transformations Joseph Fioresi 1⋆ , Fabian Caba Heilbron 2 , Pankaj Nathani 2 , Mubarak Shah 1 , and Kushal Kafle 2 1 Institute of Artificial Intelligence, University of Central Florida, USA 2 Adobe Research, USA joseph.fioresi@ucf.edu, caba,pankajn,kkafle@adobe.com, shah@crcv.ucf.edu https://joefioresi718.github.io/ControlEmbed_webpage/ Abstract. Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification. These embeddings compress high-level seman- tics into a single vector, which comes at the cost of primarily express- ing a dominant semantics like main object while suppressing other im- portant attributes such as camera angle or color tone. We propose a text-conditioned transformation of visual embeddings that makes such attributes explicitly accessible. Given a natural language description of an attribute category (e.g., “color” or “art style”), a network generates an affine transformation that emphasizes the specified attribute. Condition- ing on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface. The network is trained to align transformed embeddings with the frozen latent space, enabling retrieval using existing large-scale embeddings without any re- encoding. When applied to a full set, the same mechanism transforms the latent space for attribute disentanglement tasks such as multi-clustering. By operating directly in latent space, our method provides a unified and efficient framework for controlling embedding spaces, demonstrating state-of-the-art performance across both attribute-based retrieval and multi-attribute organization tasks with near-zero inference cost. Keywords: Vision-Language Models· Embedding Spaces 1 Introduction Large-scale vision-language models like CLIP [42] and SigLIP [74] have become ubiquitous in computer vision, learning powerful multimodal embeddings from web-scale data. Such embeddings enable zero-shot task transfer, cross-modal retrieval, and semantic similarity search, and are increasingly deployed in prac- tice. However, their versatility comes from compressing diverse visual concepts— objects, attributes, styles, and contexts—into a single high-dimensional space. ⋆ Majority of work done as an intern at Adobe Research arXiv:2607.22919v1 [cs.CV] 24 Jul 2026 2J. Fioresi et al. Attributes Subject: cow Count: 2 Color: brown and white Location: natural environment Large Visual Embedding Set Embedding Index Attribute-aware Query Embeddings Retrieve Top 3 “color” Top 3 “count” Text Encoder Generating Transforms “colors” “same location” “color” Transform “location” Transform Input Text: “object count” Hypernetwork “count” Transform Query Transformation - Attribute-aware Retrieval “color” Transform “count” Transform Embedding Model Visual Embedding Transformed Attribute Embeddings Applying Transformation Global Transformation - Clustering “location” Transform “location” Embedding Set Natural Environment Urban Area/Street Residential Area Cluster Text Encoder Generating Transforms Modifier Input: E.g, “color”, “object count”, “location” etc.. Hypernetwork Query Transformation - Attribute-aware Retrieval Global Transformation - Attribute-aware Clustering Cluster1: Natural Env Cluster2: Urban Area/Street Cluster3: Residential Area Precomputed Embedding Set Top-3 retrieval Top-3 retrieval Clustering Query Text Encoder Generating Transforms Modifier Input: E.g, “color”, “object count”, “location” etc.. Hypernetwork Query Transformation - Attribute-aware Retrieval Global Transformation - Attribute-aware Clustering Cluster1: Natural Env Cluster2: Urban Area/Street Cluster3: Residential Area Precomputed Embedding Set Top-3 retrieval Top-3 retrieval Clustering Query Text Encoder Generating Transforms Modifier Input: E.g, “color”, “object count”, “location” etc.. Hypernetwork Query Transformation - Attribute-aware Retrieval Global Transformation - Attribute-aware Clustering Cluster1: Natural Env Cluster2: Urban Area/Street Cluster3: Residential Area Precomputed Embedding Set Top-3 retrieval from the set Top-3 retrieval from the set Clustering the entire set Query Query Transform Global Transform Kushal’s v2. Tried to put some more text in idk if we need it Fig. 1: Overview of our text-conditioned generation of embedding-space transforma- tions. (Left) We propose a hypernetwork that maps natural-language attribute descrip- tions to a modification function. Applying this transformation to a query embedding produces an attribute-aware representation that emphasizes the requested property. (Right) By controlling the embedding space with attribute transforms, the same mech- anism enables both attribute-aware retrieval and attribute-specific clustering. As a result, important attributes such as color, camera angle, or mood are en- coded only implicitly, entangled with more dominant object-level semantics. This entanglement limits the kinds of similarity queries that can be expressed. When retrieving images from a reference image with CLIP, we can neither predict nor control whether the similarity will be based on the main subject, style, or other attributes. For example, suppose a reference image is a pastel-colored watercolor painting of a Corgi as viewed from a low angle. In this case, CLIP can retrieve images that look similar, but the specific properties are nebulous [53]. Will a photograph of a Corgi against a plain background rank higher or lower than a similarly styled painting of a Labrador viewed from a similar angle? Not only can we not readily know the answer, but we also cannot easily specify our preference. This limitation restricts the feasibility of attribute-aware retrieval, where users search based on specific attributes rather than broad semantic categories. For instance, given a large collection of photos, one user might want to retrieve images with similar composition or lighting, while another might care more about color or artistic style. Yet, because such attributes are entangled within the latent space, a single query embedding cannot easily emphasize them. This challenge extends beyond retrieval to other tasks such as multi-clustering, where different users may wish to group the same media collection by different attributes. To- gether, these limitations highlight the need for controllable transformations of embedding spaces that adapt dynamically to user-specified attributes. Existing methods for attribute disentanglement often rely on one of two major strategies: (1) finetuning the encoder to explicitly separate desired at- tributes [10,60,72,73]; or (2) scaling up inference-time requirements by compos- ing knowledge from large VLMs and LLMs [22,27,32]. The first approach enables efficient inference once trained, but demands costly finetuning and re-embedding. Controlling Embedding Spaces with Text-Conditioned Transformations3 This causes linear memory footprint scaling and sacrifices the generalization of the original model. The second requires no retraining, but introduces high infer- ence latency and limited scalability for real-time applications. In this paper, we propose a novel approach to address the above limitations. At the core of our approach is a simple but powerful idea: instead of retraining the encoder or designing separate attribute-specific heads, we learn to reshape the latent space itself through a text-conditioned transformation. Concretely, we train an MLP-based hypernetwork that takes as input a text embedding representing an attribute category (e.g., “color” or “style”) and outputs a sin- gle global affine transformation. This transform acts as a learned operator that modifies the existing embedding space so that the specified attribute becomes more salient. Figure 1 demonstrates this idea, showcasing its use in two comple- mentary applications: a query transformation, where only the query embedding is modified to emphasize a desired attribute for attribute-based retrieval from a static embedding index; and a global transformation, which transforms all em- beddings for downstream analysis such as unsupervised clustering. Because the encoder remains frozen and the hypernetwork operates directly on latent em- beddings, inference is near-instant and the method can be seamlessly integrated into existing multimodal pipelines. Our main contributions are summarized as follows: – We introduce a text-conditioned hypernetwork that generates trans- formations of visual embeddings, enabling controlled emphasis of diverse attributes such as color, mood, and camera angle. Alignment with base em- beddings enables a query mode for attribute-aware retrieval. – These transformations can be applied to disentangle entire embedding sets in a global setting. This induces an attribute-specific similarity metric, where the pairwise distances reflect closeness along the chosen category axis. – Extensive experiments demonstrate that our lightweight transformations achieve performance matching or surpassing existing approaches across mul- tiple attribute control tasks with minimal computational cost. A single hy- pernetwork is trained across all attributes/datasets simultaneously, for a total coverage of 19 different attributes with 187 unique subclasses. 2 Related Work Foundation embedding models. Foundation models for vision-text embed- ding have demonstrated that high-dimensional representations can capture rich semantic relationships from large-scale text data [36,41]. This principle has been extended to multimodal contexts by models such as CLIP [42], ALIGN [16], and SigLIP [74], which learn a joint embedding space for images and text. Subsequent work has further scaled and refined such embeddings to improve representation quality and generalization [21, 39, 50, 54, 58]. Our approach operates on these frozen embeddings, allowing seamless integration into existing pipelines without recomputing features, and avoids fine-tuning the encoder, thereby mitigating forgetting or performance degradation. 4J. Fioresi et al. Powerful multimodal embeddings have enabled applications such as semantic image retrieval [5,37,46,75] and zero-shot classification [34,42,47,49,68,76]. Many works proposed lightweight task adaptions of such models to improve down- stream performance [14, 17]. Composed or conditional image retrieval methods extend these capabilities by allowing a user-specified text modification to retrieve images along different semantic axes [1–3,12,28,29,52,59,61,63]. In contrast, our method focuses on controlling embeddings according to a text query, either by transforming the query alone for retrieval or by transforming the full embedding space, rather than retrieving a specific modified image. Attribute disentanglement/attribute-aware embeddings. While founda- tion models like CLIP and ALIGN provide rich multimodal embeddings, fine- grained attributes such as color, pose, and lighting are often entangled with higher-level semantics [53]. Recent approaches have sought to disentangle vi- sual embeddings into separate attribute and object components. For exam- ple, [13,44] learn disentangled representations to support compositional reason- ing and fashion-based retrieval, respectively, but require retraining, which can be computationally expensive. Other works adopt lightweight approaches using frozen embeddings [55, 67], but focus primarily on personalization for text-to- image generation. Another line of research aims to learn interpretable attribute bits for hash-code-based retrieval [6,7,31,45,62,65,66]. In contrast, our method proposes a hypernetwork to generate controlled transformations of embeddings, enabling robust disentanglement without retraining the base encoder. Multi-clustering methods. Multi-clustering aims to cluster the same dataset in various meaningful ways. Traditional methods often require careful feature selection and design [15, 35, 43, 57, 64, 71]. Recent works like Multi-Map [73] and Multi-Sub [72] leverage powerful pretrained models like CLIP but fine- tune the model on each user query, optimizing clustering-specific losses that can disturb the natural relational structure. Another branch of methods uti- lizes large multimodal models (LMMs) such as LLaVA [25] to caption images, then employs large language models (LLMs) for iterative clustering based on text-based user inputs [22, 27, 32]. While these methods avoid retraining, they introduce significant inference costs. In contrast, our method learns to generate a text-conditioned projection that operates on frozen encoder features, offering a scalable and efficient solution for multi-clustering tasks. 3 Method The main objective of this work is to construct meaningful, text-conditioned transformations of foundation model embedding spaces. Our approach learns a single hypernetwork that maps a natural language input (e.g., “color” or “camera angle”) to the parameters of an affine transformation, reshaping the embedding space to emphasize the queried attribute. This transformation can be applied either to (1) the retrieval query only for efficient attribute-aware retrieval without changing the pre-extracted visual features for the whole gallery, or (2) the full embedding set for global tasks such as clustering. Controlling Embedding Spaces with Text-Conditioned Transformations5 blowing bubbles urban area/street holding umbrella natural environment holding umbrella urban area/street blowing bubbles natural environment Vision Encoder Aligned to original embedding space: Multi-attribute training images Action: blowing bubbles Location: urban area/street Action: holding umbrella Location: natural environment Action: holding umbrella Location: urban area/street Action: blowing bubbles Location: natural environment Action: holding umbrella Location: natural environment Vision Encoder “actions” Text-conditioned transform generation Text Encoder “blowing bubbles” Multi-Positive Contrastive Loss: Action: blowing bubbles Location: urban area/street Action: holding umbrella Location: natural environment Action: holding umbrella Location: urban area/street Action: blowing bubbles Location: natural environment Transform “holding umbrella” Aligned to original embedding space: Multi-attribute training images Highlight = contrastive loss positive “actions” Text-conditioned transform generation Multi-Positive Contrastive Loss: Action: blowing bubbles Location: urban area/street Action: holding umbrella Location: natural environment Action: holding umbrella Location: urban area/street Action: blowing bubbles Location: natural environment Transform Aligned to original embedding space: Multi-attribute training images Highlight = contrastive loss positive “actions” Text-conditioned transform generation Action: holding umbrellaAction: blowing bubbles “holding umbrella” “blowing bubbles” Multimodal Alignment Multi-modal guidance “natural environment” “urban area/street” Action: blowing bubbles Location: urban area/street Action: holding umbrella Location: natural environment Action: holding umbrella Location: urban area/street Action: blowing bubbles Location: natural environment Transform Aligned to original embedding space: Multi-attribute training images “same locations” Text-conditioned transform generation Multi-Positive Contrastive Loss: Multi-modal Text Alignment = Hypernetwork = Transformed embedding = Vision embedding = Attract/Pull = Repel/Push = Text embedding Action: holding umbrella Location: natural environment Vision Encoder “same locations” Transform Same images, different positives/negatives “urban area/street” “natural environment” Multi-attribute Training User Input: Next Batch: Sample from all attributes Vision Encoder Vision Encoder Vision Encoder Vision Encoder shared Batch 1: Action Batch 2: Location blowing bubbles urban area/street holding umbrella natural environment holding umbrella urban area/street blowing bubbles natural environment Fig. 2: Overview of our multi-attribute training workflow. An attribute category is sampled (e.g., “action” or “location”) and passed through the proposed hypernetwork to generate a transformation function, which is applied to a query embedding to pro- duce attribute-focused representations aligned with base embeddings of images sharing the same category attribute label via a multi-positive contrastive loss. An additional multimodal guidance loss further aligns transformed features with the text embedding of the corresponding attribute label. The process iterates over the full set of valid at- tributes, where the set of positive/negative samples varies based on selected category. 3.1 Notation We denote the vision and text encoders as f v and f t , respectively. Given an image I and a text input T, the encoders produce modality-specific embeddings: v = f v (I), t = f t (T ), where v, t∈R D . Unless otherwise specified, we use the CLIP ViT-L/14 model [42] with frozen weights. All operations are performed directly in the embedding space, eliminating the need for finetuning the base encoders or feature re-extraction. We consider a dataset D = (I i , y i ) N i=1 , where each image I i is annotated with labels y i across M semantic taxonomies. Each category C (m) (e.g., “color”) defines a set of K m mutually exclusive classes c (m) 1 ,...,c (m) K m . 3.2 Text-Conditioned Affine Transformation Hypernetwork We propose a hypernetwork H φ that generates an affine transformation condi- tioned on a textual query. Given a category text embedding t, the hypernetwork outputs the parameters of an affine transformation: W, b = H φ (t),(1) 6J. Fioresi et al. where W ∈R D×D and b∈R D . Each image embedding v is then projected as: v ′ = vW T + b.(2) This can be applied to a single query embedding for attribute-aware retrieval in the query setting, or to the full embedding set V in the global setting to create a new embedding space V ′ specialized to the input attribute. Note that the conditioning signal does not fundamentally require text embeddings, it is chosen as a natural interface choice for specifying attributes. Query-based Training Objective. The hypernetwork is trained to generate transformations that reorganize embeddings based on a user-specified category while maintaining consistency with the foundation model’s original embedding structure. An attribute category is sampled for each batch, defining the label set for the loss objectives. Diverse sampling results in a single hypernetwork capable of optimizing for multiple attributes simultaneously, shown in Figure 2. We use a multi-positive contrastive loss based on StableRep [51] to encourage transformed samples to be near the base embeddings from the same subclass: L SR =− 1 N N X i,j=1 1 i,j P N k=1 1 i,k log exp(v ′ ⊤ i v j /τ ) P N k=1 exp(v ′ ⊤ i v k /τ ) , (3) where v ′ i ∈ V ′ , v j ∈ V are ℓ 2 -normalized feature embeddings, τ is a temperature parameter, N is the batch size, and 1 i,j = 1 if i and j share a class label under the sampled attribute category and 0 otherwise. This objective encourages each transformed embedding v ′ i to align closely with embeddings v j from the same attribute label within the base space. The query-only similarity score⟨W v ′ i +b, v⟩ factors as v ′⊤ i W ⊤ v +b ⊤ v. The product v ′⊤ i W ⊤ v shows that the transformation is applied to the gallery side v via the inner product at score time rather than rewritten into the stored gallery vectors. Thus, the gallery can remained fixed while the retrieval score is axis-aware. Multimodal Objective. To maintain cross-modal consistency, we also include a symmetric multimodal alignment loss between transformed visual embeddings and text embeddings of the corresponding attribute labels: L i2t =− 1 N N X i=1 log exp(v ′ ⊤ i ̄ t i /τ ) P N j=1 exp(v ′ ⊤ i ̄ t j /τ ) ,(4) where ̄ t i represents an averaged text embedding over descriptive templates (e.g., “a photo of c”, , “an image in the c style”).L t2i is defined in a similar fashion, just with v ′ swapped with ̄ t. The final loss combines the multi-positive visual term and the multimodal guidance terms: L =L SR + ω∗ (L i2t +L t2i ),(5) where ω balances text alignment and category structure. Dual Inference Capability. Although the hypernetwork is trained only in the query setting, where transformed embeddings v ′ are aligned to the frozen base Controlling Embedding Spaces with Text-Conditioned Transformations7 space, it can be used for a dual purpose during inference time. The first straight- forward application is to transform the query image, yielding an attribute-aware retrieval along the direction of the transformation. The second ability is to achieve an emergent global transformation that reorganizes the entire distri- bution along the specified attribute axis without additional training. This trans- formed distribution shapes clustering to identify meaningful groupings based on the provided attribute. Concretely, the same learned transformation can be applied directly to the entire embedding set (V → V ′ ) at inference time. To summarize, the model learns a general operator that adapts to any visual em- bedding collection in an attribute-aware manner, enabling both attribute-aware retrieval and multi-attribute clustering within a single unified framework. Table 1: Frozen gallery attribute-aware retrieval results reported as mAP averaged across dataset taxonomies. Our method si- multaneously handles multiple categories with one set of weights while drastically improving performance.† denotes MLP adapter trained with multi-positive loss. Method Multi-Attr. Capability Clevr4 mAP S40 mAP ShotBench mAP Base Features✗33.8 43.7 25.2 LCR✓15.9 19.9 18.9 CWP✓43.8 49.8 21.4 ArcFace [8]†✗12.3 18.8 17.5 SupCon [18]†✗12.7 18.8 17.7 StableRep+ [51]†✗49.5 21.3 20.9 MaPLe [17]✗33.5 47.8 23.7 LoRA [14]✗42.9 49.3 22.0 SEARLE-XL [2]✗24.6 33.9 20.3 Ours-Query✓77.0 86.3 41.0 Table 2: Multi-clustering results averaged over 14 attribute categories. Inference time is reported assuming a small evaluation set of 5k images. The proposed approach im- proves performance while offering minimal compute overhead and near-instant infer- ence. (*reported on full action set.) Method Inference Time (s) Clevr4 c. ACC S40 c. ACC ShotBench c.ACC Base Features0.5948.8 65.220.8 LCR0.6256.4 64.830.2 CWP0.6860.3 64.932.2 Multi-MAP [73] 75062.8 62.832.6 Multi-Sub [72] 15300 72.2 66.132.4 IC|TC [22]37680 57.9 76.121.7 SSD-LLM [32] 27600 56.5 74.125.0 X-Cluster [27] 104760 64.9 68.3*- SEARLE-XL [2] 8130.7 46.924.5 Ours-Space0.64 73.0 80.0 51.6 4 Experiments We first describe the experimental setup, then present results on attribute-aware retrieval, clustering, and show comprehensive ablation studies and analysis. Ad- ditional experimental results can be found in Supplementary Section C. 4.1 Datasets ShotBench: ShotQA/ShotBench [26] are datasets designed for training and evaluating vision-language models on question-answering related to fine-grained photographic attributes, namely camera angle, composition, lens size, lighting, lighting type, shot framing, and shot size. From these, we extract attribute an- swers as labels, resulting in a multi-label dataset consisting of 31k training images and 2.7k testing images across 7 categories. 8J. Fioresi et al. Clevr-4 [56] is a synthetic dataset of 10k image samples, where each image contains a set of shapes with 4 axes of variation: color, count, shape, and texture. Each attribute category contains 10 subclasses and a mainly even distribution, so no subclass is over/underrepresented. Stanford 40 Actions [70] contains 9,532 images of humans performing one of 40 actions, along with attribute annotations for 1,000 images with 10 location categories and 4 mood categories [22], resulting in three distinct categories: ac- tion, location, and mood. For training, we use a subset of 532 images drawn from the original action training split, annotating them with labels consistent with the existing taxonomy. Evaluation follows the same setup as prior work, using the full mood and location subsets and the remaining 9,000 images for action. MS-COCO [24]. We additionally use the COCO 2017 test split (41k images) for qualitative examples, serving as a diverse, real-world benchmark to illustrate cross-domain generalization of attribute transformations learned on synthetic or domain-specific datasets. Additional information on datasets used may be found in Supp. Section A. 4.2 Baselines Few existing methods allow controllable adaptation of frozen embedding spaces, so we construct two lightweight, training-free baselines to contextualize the benefits of our learned transformation. These baselines operate entirely using CLIP text and image embeddings and provide inexpensive, interpretable at- tribute–specific projections. Given an attribute category with K class labels, we form a concept matrix A = [t 1 ,..., t K ]∈R K×D , where each row t k is the normalized text embedding of one attribute label. Both baselines project image embeddings V ∈R N×D into the subspace spanned by these concept vectors. (1) Linear Concept Reconstruction (LCR). A simple reconstruction of each visual embedding in the concept span: V LCR = (V A T )A.(6) LCR treats the attribute directions as a basis and re-expresses each image em- bedding as a weighted combination of them. (2) Concept-Weighted Projection (CWP). Because the concept vectors are not orthogonal, we correct for overlap via the inverse Gram matrix (A T ) −1 : V CWP = (V A T )(A T ) −1 A.(7) This performs the closest-point projection onto the attribute subspace, effectively removing redundancy between overlapping concept directions. Both baselines offer parameter-free, low-cost attribute awareness, but lack the learnable expressive ability of our proposed text-conditioned hypernetwork. Controlling Embedding Spaces with Text-Conditioned Transformations9 4.3 Implementation Details All experiments are implemented in PyTorch [40] and trained on a single NVIDIA A100 GPU. The hypernetwork is a three-layer MLP with ReLU activations, which takes as input the CLIP text embedding and predicts the parameters of the affine transformation (W, b). Dimensions depend on the chosen CLIP variant (e.g., 768 for ViT-L/14). We train with the AdamW [19,30] optimizer, a learning rate of 1e−4, and multimodal loss weight ω = 0.1. All reported runs use frozen encoder weights; only the hypernetwork parameters are updated during training. Further implementation details are provided in Supp. Section B. 4.4 Attribute-aware Image-to-Image Retrieval We begin with the attribute-aware retrieval setting, where only the query embed- ding is transformed while retrieval is performed over the original frozen gallery. This query mode preserves compatibility with large-scale search systems, as no re-encoding per attribute is required. Table 1 shows that CLIP’s base fea- tures do not offer control and struggle to surface attribute-consistent matches. Lightweight adapter-based approaches such as LoRA [14] and MaPLe [17] fine- tune CLIP to generally improve alignment. However, these require separate sets of parameters per attribute category, causing memory and storage costs to scale linearly with attribute count, and fail to handle the frozen gallery. In contrast, our method achieves state-of-the-art performance in this setting, while natively handling multiple categories and keeping memory footprint constant: a single hy- pernetwork handles all categories jointly, without growing in size as the number of attributes increases. Expanded results are provided in Supp. Section C. 4.5 Unsupervised Multi-Clustering We next evaluate the global setting, where the learned transformation is applied to the entire embedding set and unsupervised k-means clustering is performed on the transformed features. For each attribute, we run Faiss k-means [9] and compute clustering accuracy using Hungarian matching [20]. Table 2 compares our approach with state-of-the-art multi-clustering methods including Multi- MaP [73], Multi-Sub [72], IC|TC [22], SSD-LLM [32], and X-Cluster [27]. Meth- ods that rely on image captioning and LLM iteration [22, 27, 32] incur high inference cost because each image must be captioned and repeatedly queried through a large language model, which are operations that scale linearly with dataset size and dominate runtime. [72,73] reduce this reliance but still require per-attribute finetuning after user input, yielding moderate performance but still with substantial compute and memory overhead. In contrast, our proposed query-focused training method naturally handles this attribute disentanglement setting and achieves the highest average clustering accuracy across datasets while using a single transformation network trained once for all attributes. 10J. Fioresi et al. Action: skiing Count: 3 CA: high angle Action: skiing Action: skiing Action: skiing Action: skiing Action: skiing Base CLIP Features: “count” Transformed Query: “camera angle” Transformed Query: Query Image Retrieve Retrieve Retrieve Action: skiing Count: 3 CA: high angle Action: skiing Count: 3 CA: high angle Count: 3 CA: high angle Count: 3 CA: high angle Count: 3 CA: high angle Count: 3 CA: high angle Count: 3 CA: high angle Ours (CA = camera angle) Control over retrieval attribute! Action: skiing Count: 3 CA: high angle Action: skiing Count: 1 CA: eye level Action: skiing Count: 1 CA: low angle Action: skiing Count: 1 CA: eye level Action: skiing Count: 1 CA: eye level Action: skiing Count: 1 CA: eye level Base CLIP Features: “count” Transformed Query: “camera angle” Transformed Query: Query Image Retrieve Retrieve Retrieve Action: skiing Count: 3 CA: high angle Action: skiing Count: 3 CA: high angle Action: skiing Count: 3 CA: eye level Action: skiing Count: 3 CA: high angle Action: skiing Count: 3 CA: eye level Action: skiing Count: 1 CA: high angle Action: skiing Count: 3 CA: eye level Action: skiing Count: 5 CA: high angle Action: skiing Count: 3 CA: eye level Action: skiing Count: 1 CA: high angle Action: fishing Count: 3 CA: eye level Action: surfing Count: 3 CA: high angle Ours Image Attributes: (CA = camera angle) Control over retrieval attribute! No Transformation Same Attribute, Similar SemanticsSame Attribute, Less Semantic Adherence Fig. 3: Qualitative attribute-aware retrieval results under different attribute trans- formation functions. Our text-conditioned query transformations selectively retrieve images that match the specified attribute without losing the scene semantics at first, then prioritize the attribute over general semantics. From top to bottom, we show re- trievals for "action", "count", then fine-grained perceptual property "camera angle". 4.6 Ablations and Analysis Transformation Type. Attributes are entangled in CLIP space, yet they re- main linearly separable [4]. We demonstrate that controllable similarity is achiev- able with an asymmetric affine transformation applied to the query vector alone. A simple offset vector (translation) does not have the ability to scale and rotate sub-spaces, which is needed to satisfy our setting constraints (Table 3). Table 3: Affine transforms effectively enable per-attribute similarity. DatasetVariantCluster ACC Query mAP Global mAP Clevr-4 Base32.233.8– Translation-only32.235.434.5 Affine75.4 (+43.2) 77.6 (+42.2) 85.1 (+50.6) Stanford-40 Base46.443.6– Translation-only55.643.844.5 Affine79.4 (+23.8) 84.4 (+40.6) 87.7 (+43.2) Compositional Retrieval. Our framework directly supports compositional re- trieval without retraining. We compose a multi-attribute query by linearly blend- ing the projected queries into q blend = α (W a q + b a ) + (1−α) (W b q + b b ), where a and b are attributes such as action and location. Table 4 shows that a com- posed query (α = 0.5) improves joint-label (Both) retrieval over the raw baseline and either single-axis projection, indicating that the learned projections can be combined for multi-attribute user queries. Controlling Embedding Spaces with Text-Conditioned Transformations11 Table 4: Compositional retrieval on Stanford40 action (A) + location (L). Uniform blending improves joint-label retrieval w/o retraining. VariantmAP(Both) Both@5 A-only@5 L-only@5 Neither@5 Base48.545.121.515.917.5 Action only70.057.440.70.91.0 Location only21.215.62.876.45.3 Linear Blend (α = 0.5) 80.964.830.23.61.4 Embedding Space Analysis. We analyze how the learned transformation sup- ports attribute-aware retrieval from a frozen gallery. For each attribute, we de- compose W with SVD and identify the top left-singular directions, which repre- sent the axes most amplified by the transform. Even though gallery embeddings are unchanged, the query-gallery score is dominated by these amplified direc- tions, while suppressed directions contribute little. We provide two findings: 1) The transformed space is lower-dimensional and attribute-specific. Stable rank, an effective-dimensionality measure, shows that projected features v i W T occupy a smaller subspace than raw embeddings (Tab. 5). We further project features onto the dominant singular subspace of W (retaining 95% of singular-value energy) and its orthogonal complement. Retrieval in the kept sub- space greatly exceeds the raw baseline, while retrieval in the orthogonal comple- ment collapses, indicating that W isolates a compact attribute-specific subspace. 2) Query-only projection improves score discriminability. Although ab- solute cosine similarities decrease after projection (Tab. 6), same-label and different- label pairs become much better separated, and AUC improves along the target axis. This suggests that the transform suppresses irrelevant components rather than simply increasing all similarities. Table 5: Stable rank and retrieval anal- ysis. “Kept subspace” projects onto the dominant left-singular directions ofW . “Orth. Comp.” indicates the orthogo- nal complement. The transform selects a compact attribute-specific subspace. Axis Stable Rank (base→trans) Base mAP Kept Space Orth. Comp. Color 11.3→ 5.0 11.9 98.4 4.9 Action 22.1→ 14.3 48.9 92.1 15.3 Table 6: Average cosine similarities for same/diff.-label pairs on Clevr4. AUC mea- sures whether same-label pairs rank above different-label pairs. Transformed features drastically improve inter-class separation. Axis Mode Same/Diff. Cohen’s d AUC Color Raw 0.76 / 0.720.70.703 Query 0.19 / -0.02 4.0 0.993 Count Raw 0.74 / 0.720.30.579 Query 0.10 / 0.01 1.4 0.829 Generalization to Unseen Classes. Table 7 evaluates color-conditioned re- trieval when specific color subclasses are held out during training. Our trans- formed embeddings still retrieve images of the same color as the query, despite not seeing it during training, indicating that the transformation successfully 12J. Fioresi et al. emphasizes the general color property instead of only learning the specific seen colors. However, similar colors (blue/cyan) struggle to fully disentangle. Table 7: Color generalization evaluation on Clevr4. Ours (seen) is the upper bound where all colors are present during training. Ours (unseen) indicates training with one color held out (e.g., no green), then evaluating color-based retrieval (mAP) on that color. Our transformation functions learn to generalize beyond the training subclasses. Clevr4 – Color Retrieval (mAP) MethodBlue Brown Cyan Gray Green Orange Pink Purple Red Yellow Avg. Ours (seen)99.693.999.699.399.686.288.599.698.474.293.9 Baseline23.3 16.9 26.5 21.0 28.6 21.0 20.4 30.1 21.5 19.4 22.9 Ours (unseen) 33.1 +9.8 74.5 +57.6 32.6 +6.1 47.4 +26.4 91.9 +63.3 84.2 +63.2 44.4 +24.0 90.2 +60.1 42.3 +20.8 39.5 +20.1 58.0 +35.1 Effect of Multimodal Guidance. Beyond improving retrieval and clustering, multimodal alignment also enhances text-similarity classification accuracy (Ta- ble 8), reflecting tighter correspondence between transformed embeddings and attribute text features. Even without explicit supervision, the transformation exhibits meaningful alignment due to CLIP’s pretrained cross-modal structure. Table 8: Ablation on multimodal guidance across three tasks using native metrics. The multimodal objective (Eq. 4) enables text-similarity classification and modestly im- proves retrieval and clustering performance. Stanford40 abbreviated as S40 for brevity. Multiple Clustering Attr.-Aware Retrieval Text-based Classification ObjectiveClevr4 S40 ShotBench Clevr4 S40 ShotBench Clevr4 S40 ShotBench Ours73.0 80.0 51.6 77.1 91.2 41.1 88.7 83.2 51.6 − multimodal 72.7 78.4 53.276.8 87.3 40.771.7 78.531.3 Data Efficiency. Figure 4 ablates the number of training samples used per class on the Stanford 40 Action [70] dataset. Remarkably, our global affine transforma- tion achieves near peak performance across all taxonomies given just six training samples per class. Scalability Across Attributes. We assess how well the model scales as more attribute categories are trained within a single hypernetwork. Starting from one model per attribute, we progressively expand to one per dataset and finally to a unified model trained across all datasets. The results in Figure 5 show that performance remains stable even as the number of jointly trained attributes increases, varying by ≤1.5%. Notably, final results are presented using a single hypernetwork trained across 19 attributes with 187 subclasses, indicating that the hypernetwork can generalize across many attributes without interference. Controlling Embedding Spaces with Text-Conditioned Transformations13 # of Training Samples Per Class Clustering Acc (%) 55 65 75 85 0246810 Action (k=40) Location (k=10) Mood (k=4) Action-AllLocation-AllMood-All Fig. 4: Multi-clustering accuracy on Stanford40 under varying number of training images per class. Category- All line indicates performance using all available training samples. Our trans- form approach demonstrates extreme data efficiency, achieving near peak re- sults with just six images per class. 134714 # Attribute Categories Supported per Hypernetwork 40 50 60 70 80 90 Mean mAP (%) Clevr-4Stanford40ShotBench Fig. 5: Mean category mAP across datasets (each line) as the number of jointly trained attribute categories increases from a single category per model to unified training across all datasets. Performance remains consistent even as more attributes are added, demon- strating that a single hypernetwork can scale across 14+ attributes without degradation. CLIP-B/16CLIP-L/14 SigLIP-G/16 DINOv2-L/14Qwen3-VL- Embed-2B 0 25 50 75 100 mAP +42.2 +42.6 +37.2 +25.8 +48.9 +53.2 +51.8 +54.7 +48.2 +55.8 Clevr4 CLIP-B/16CLIP-L/14 SigLIP-G/16 DINOv2-L/14Qwen3-VL- Embed-2B mAP +46.3 +43.4 +38.7 +42.8 +41.5 +53.3 +52.0 +43.4 +47.2 +43.8 Stanford40 Base FeaturesOurs-QueryOurs-Global Fig. 6: Ablation on backbone architecture, evaluated using retrieval mAP. Our method is effective across all types/sizes of encoders, including non-text aligned DINOv2. Backbone Encoder Choice. Figure 6 shows that the proposed transformation consistently improves attribute-based retrieval across encoder architectures and model sizes. Gains are observed for both vision–language models (CLIP, SigLIP, Qwen3-VL-Embed) and purely visual backbones (DINOv2), demonstrating that the method generalizes beyond text-aligned representations. Note that for DI- NOv2, the CLIP H/14 text encoder is used for consistent feature dimensionality. 4.7 Discussion We next analyze qualitative and emergent properties of learned transformations. How can a single transformed query embedding retrieve from a frozen gallery set? The transformation is trained with a multi-positive contrastive loss which optimizes relative similarity—pulling the transformed query closer to pos- itives while repelling negatives. Because most concepts are partially entangled (e.g., two images may share an object but differ in color), minimizing absolute 14J. Fioresi et al. distance to positives would inadvertently bring it near some negatives. To re- solve this, the network learns to move the query in a new direction within the high-dimensional space that separates it from the general visual manifold while aligning it with the target attribute. This broadly resembles the text–image “modality gap” [23], allowing the transformed query to behave similar to a text embedding. Even without explicit multimodal supervision, Table 8 shows that transformed queries naturally align with text features, enabling accurate retrieval from a frozen gallery using similarity in the new space. Fig. 7: t-SNE visualization of embeddings colored by shot-size labels before and after transformation. The baseline space (left) shows entangled clusters. After applying our learned transformation (right), samples align intuitively along a continuous gradient that reflects the shot-size taxonomy. Can the transformation capture perceptual attributes and how does it affect embedding geometry? Contrastive image–text models such as CLIP are trained to capture broad semantic meaning, yet they provide little explicit separation for perceptual attributes like shot size, composition, or camera angle. As a result, embeddings that differ in these subtle properties often remain close in the feature space. Our transformation isolates these attributes by reorienting the space toward the specified taxonomy while preserving global structure. As shown in Figure 7, applying the transformation introduces a clear, continuous axis of variation corresponding to the target attribute (here, shot size). Samples form a smooth progression from close-up to wide shots, reflecting a structured and interpretable organization. Quantitatively, this behavior yields strong im- provements on fine-grained perceptual benchmarks like ShotBench, confirming that the transformation reshapes latent geometry to make attribute-specific re- lationships explicit without distorting the original feature layout. Do attribute-specific transforms learned on synthetic data transfer to real images? Our transformations generalize cleanly across domains. When trained on the synthetic Clevr-4 dataset, which contains simple geometric ob- Controlling Embedding Spaces with Text-Conditioned Transformations15 Fig. 8: Our attribute-conditioned similarity metric helps characterize unique relation- ships that CLIP similarity alone cannot. (Left) Baseline CLIP similarity prioritizes overall semantics. (Middle, Right) Applying attribute-specific transforms to the query (e.g., color or location) selectively re-scores relationships based on the input attribute. jects with uniform backgrounds, the model still produces meaningful attribute disentanglement for real, diverse MS-COCO images (Figs. 3 and 8). Despite the distribution shift, transforms learned for color and count correctly align COCO embeddings along the intended dimensions. We conjecture that this generaliza- tion arises from optimizing a single affine transform for the entire embedding space, which discourages shortcut memorization and instead forces the trans- form to properly disentangle embeddings to optimize the contrastive loss. Attribute-aware similarity. Figure 8 illustrates how the proposed similarity metric handles relationships that are not captured by standard CLIP similar- ity. Images that share the attribute are pulled closer while others are pushed away, irrespective of their original CLIP similarity. Our transform effectively re- structures the embedding space to emphasize the selected attribute, enabling controlled comparison of images along different attribute dimensions. Composed Image Retrieval (CIR) Compatibility. Our transformations can also be integrated into existing CIR pipelines to improve attribute discrim- inability. We provide an illustrative experiment in Supp. Section C. 5 Conclusion We presented a text-conditioned transformation framework that enables fine- grained control over pretrained embedding spaces without modifying encoder weights or recomputing gallery features. A single hypernetwork learns to gener- ate attribute-specific affine transformations from natural-language descriptions, supporting both attribute-aware retrieval and multi-attribute organization with near-zero inference overhead. While the method generalizes across diverse at- tributes, its expressiveness is currently bounded by the range of categories seen during training. Future work may explore scaling up training to find emergent transformation properties across attributes completely unseen during training. 16J. Fioresi et al. References 1. Agnolucci, L., Baldrati, A., Del Bimbo, A., Bertini, M.: isearle: Improving textual inversion for zero-shot composed image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence (2025) 2. Baldrati, A., Agnolucci, L., Bertini, M., Del Bimbo, A.: Zero-shot composed image retrieval with textual inversion. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 15338–15347 (2023) 3. Baldrati, A., Bertini, M., Uricchio, T., Del Bimbo, A.: Conditioned and composed image retrieval combining and partially fine-tuning clip-based features. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 4959–4968 (2022) 4. Bhalla, U., Oesterling, A., Srinivas, S., Calmon, F.P., Lakkaraju, H.: Interpreting clip with sparse linear concept embeddings (splice). Advances in Neural Informa- tion Processing Systems 37, 84298–84328 (2024) 5. Cao, B., Araujo, A., Sim, J.: Unifying deep local and global features for image search. In: European conference on computer vision. p. 726–743. Springer (2020) 6. Chen, Z.D., Zhao, L.J., Zhang, Z.C., Luo, X., Xu, X.S.: Characteristics matching based hash codes generation for efficient fine-grained image retrieval. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition. p. 17273–17281 (2024) 7. Cui, Q., Jiang, Q.Y., Wei, X.S., Li, W.J., Yoshie, O.: Exchnet: A unified hashing network for large-scale fine-grained image retrieval. In: European Conference on Computer Vision. p. 189–205. Springer (2020) 8. Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 4690–4699 (2019) 9. Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library. arXiv preprint arXiv:2401.08281 (2024) 10. Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., Qiao, Y.: Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision 132(2), 581–595 (2024) 11. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.H., et al.: Challenges in representation learning: A report on three machine learning contests. In: International conference on neural information processing. p. 117–124. Springer (2013) 12. Gupta, A., Parmar, J., Dave, I.R., Shah, M.: From play to replay: Composed video retrieval for temporally fine-grained videos. arXiv preprint arXiv:2506.05274 (2025) 13. Hou, Y., Vig, E., Donoser, M., Bazzani, L.: Learning attribute-driven disentangled representations for interactive fashion retrieval. In: Proceedings of the IEEE/CVF International conference on computer vision. p. 12147–12157 (2021) 14. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR 1(2), 3 (2022) 15. Hu, J., Qian, Q., Pei, J., Jin, R., Zhu, S.: Finding multiple stable clusterings. Knowledge and Information Systems 51(3), 991–1021 (2017) 16. Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. p. 4904–4916. PMLR (2021) Controlling Embedding Spaces with Text-Conditioned Transformations17 17. khattak, M.U., Rasheed, H., Maaz, M., Khan, S., Khan, F.S.: Maple: Multi-modal prompt learning. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023) 18. Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in neural information processing systems 33, 18661–18673 (2020) 19. Kingma, D.P.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 20. Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly 2(1-2), 83–97 (1955) 21. Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., et al.: Matryoshka representa- tion learning. Advances in Neural Information Processing Systems 35, 30233–30249 (2022) 22. Kwon, S., Park, J., Kim, M., Cho, J., Ryu, E.K., Lee, K.: Image clustering condi- tioned on text criteria. arXiv preprint arXiv:2310.18297 (2023) 23. Liang, V.W., Zhang, Y., Kwon, Y., Yeung, S., Zou, J.Y.: Mind the gap: Under- standing the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems 35, 17612–17625 (2022) 24. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. p. 740–755. Springer (2014) 25. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36, 34892–34916 (2023) 26. Liu, H., He, J., Jin, Y., Zheng, D., Dong, Y., Zhang, F., Huang, Z., He, Y., Li, Y., Chen, W., Qiao, Y., Ouyang, W., Zhao, S., Liu, Z.: Shotbench: Expert-level cinematic understanding in vision-language models (2025), https://arxiv.org/ abs/2506.21356, accessed: 21 July 2025 27. Liu, M., Zhong, Z., Li, J., Franchi, G., Roy, S., Ricci, E.: Organizing unstructured image collections using natural language. arXiv preprint arXiv:2410.05217 (2025) 28. Liu, Z., Rodriguez-Opazo, C., Teney, D., Gould, S.: Image retrieval on real- life images with pre-trained vision-and-language models. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 2125–2134 (2021) 29. Liu, Z., Sun, W., Teney, D., Gould, S.: Candidate set re-ranking for composed image retrieval with dual multi-modal encoder. arXiv preprint arXiv:2305.16304 (2023) 30. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017) 31. Lu, X., Chen, S., Cao, Y., Zhou, X., Lu, X.: Attributes grouping and mining hashing for fine-grained image retrieval. In: Proceedings of the 31st ACM International Conference on Multimedia. p. 6558–6566 (2023) 32. Luo, Y., An, R., Zou, B., Tang, Y., Liu, J., Zhang, S.: Llm as dataset analyst: Subpopulation structure discovery with large language model. In: European Con- ference on Computer Vision. p. 235–252. Springer (2025) 33. Malkov, Y.A., Yashunin, D.A.: Efficient and robust approximate nearest neigh- bor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42(4), 824–836 (2018) 34. Menon, S., Vondrick, C.: Visual classification via description from large language models. arXiv preprint arXiv:2210.07183 (2022) 18J. Fioresi et al. 35. Miklautz, L., Mautz, D., Altinigneli, M.C., Böhm, C., Plant, C.: Deep embedded non-redundant clustering. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, p. 5174–5181 (2020) 36. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word repre- sentations in vector space. arXiv preprint arXiv:1301.3781 (2013) 37. Noh, H., Araujo, A., Sim, J., Weyand, T., Han, B.: Large-scale image retrieval with attentive deep local features. In: Proceedings of the IEEE international conference on computer vision. p. 3456–3465 (2017) 38. Ohm, T., Karjus, A., Tamm, M.V., Schich, M.: fruit-salad: A style aligned artwork dataset to reveal similarity perception in image embeddings. Scientific Data 12(1), 254 (2025). https://doi.org/10.1038/s41597-025-04529-4, https://doi.org/ 10.1038/s41597-025-04529-4 39. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023) 40. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chillemi, G., De- Pasquale, L., Gitman, T., Li, J., Lin, Z., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Process- ing Systems 32 (2019) 41. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word repre- sentation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). p. 1532–1543 (2014) 42. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. p. 8748–8763. PmLR (2021) 43. Ren, L., Yu, G., Wang, J., Liu, L., Domeniconi, C., Zhang, X.: A diversified atten- tion model for interpretable multiple clusterings. IEEE Transactions on Knowledge and Data Engineering 35(9), 8852–8864 (2022) 44. Saini, N., Pham, K., Shrivastava, A.: Disentangling visual embeddings for at- tributes and objects. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 13658–13667 (2022) 45. Shen, Y., Sun, X., Wei, X.S., Jiang, Q.Y., Yang, J.: Semicon: A learning-to-hash solution for large-scale fine-grained image retrieval. In: European conference on computer vision. p. 531–548. Springer (2022) 46. Siméoni, O., Avrithis, Y., Chum, O.: Local features and visual words emerge in activations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 11651–11660 (2019) 47. Sirnam, S., Yang, J., Neiman, T., Rizve, M.N., Tran, S., Yao, B., Chilimbi, T., Shah, M.: X-former: Unifying contrastive and reconstruction learning for mllms. In: Computer Vision – ECCV 2024. p. 146–162. Springer Nature Switzerland (2025) 48. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 conference on empirical methods in natural language processing. p. 1631–1642 (2013) 49. Swetha, S., Meng, R., Ram, S., Neiman, T., Tran, S., Shah, M.: Smpro: Self- supervised visual preference alignment via differentiable multi-preference multi- group ranking. In: Proceedings of the AAAI Conference on Artificial Intelligence (2026) Controlling Embedding Spaces with Text-Conditioned Transformations19 50. Swetha, S., Rizve, M.N., Shvetsova, N., Kuehne, H., Shah, M.: Preserving modal- ity structure improves multi-modal learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). p. 21993–22003 (October 2023) 51. Tian, Y., Fan, L., Isola, P., Chang, H., Krishnan, D.: Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems 36, 48382–48402 (2023) 52. Tian, Y., Newsam, S., Boakye, K.: Fashion image retrieval with text feedback by additive attention compositional learning. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. p. 1011–1021 (2023) 53. Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., Xie, S.: Eyes wide shut? exploring the visual shortcomings of multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 9568–9578 (2024) 54. Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al.: Siglip 2: Multilingual vision-language encoders with improved semantic understanding, lo- calization, and dense features. arXiv preprint arXiv:2502.14786 (2025) 55. Valevski, D., Lumen, D., Matias, Y., Leviathan, Y.: Face0: Instantaneously con- ditioning a text-to-image model on a face. In: SIGGRAPH Asia 2023 Conference Papers. p. 1–10 (2023) 56. Vaze, S., Vedaldi, A., Zisserman, A.: No representation rules them all in category discovery. Advances in Neural Information Processing Systems 37 (2023) 57. Veit, A., Belongie, S., Karaletsos, T.: Conditional similarity networks. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. p. 830–838 (2017) 58. Venkataramanan, S., Pariza, V., Salehi, M., Knobel, L., Gidaris, S., Ramzi, E., Bursuc, A., Asano, Y.M.: Franca: Nested matryoshka clustering for scalable visual representation learning. arXiv preprint arXiv:2507.14137 (2025) 59. Vo, N., Jiang, L., Sun, C., Murphy, K., Li, L.J., Fei-Fei, L., Hays, J.: Composing text and image for image retrieval-an empirical odyssey. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 6439–6448 (2019) 60. Wang, J., Xu, Y., Hu, J., Yan, M., Sang, J., Qian, Q.: Improved visual fine-tuning with natural language supervision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 11899–11909 (2023) 61. Wang, L., Ao, W., Boddeti, V.N., Lim, S.N.: Generative zero-shot composed im- age retrieval. In: Proceedings of the Computer Vision and Pattern Recognition Conference. p. 29690–29700 (2025) 62. Wang, P., Li, Y., Zhao, L., Wei, X.S.: Learning attribute-aware hash codes for fine- grained image retrieval via query optimization. arXiv preprint arXiv:2509.17049 (2025) 63. Wang, T., Zhao, Y., Kong, S.: Generating a paracosm for training-free zero-shot composed image retrieval. arXiv preprint arXiv:2602.00813 (2026) 64. Wei, S., Wang, J., Yu, G., Domeniconi, C., Zhang, X.: Multi-view multiple cluster- ings using deep matrix factorization. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, p. 6348–6355 (2020) 65. Wei, X.S., Shen, Y., Sun, X., Wang, P., Peng, Y.: Attribute-aware deep hashing with self-consistency for large-scale fine-grained image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(11), 13904–13920 (2023) 20J. Fioresi et al. 66. Wei, X.S., Shen, Y., Sun, X., Ye, H.J., Yang, J.: A2-net: Learning attribute-aware hash codes for large-scale fine-grained image retrieval. Advances in Neural Infor- mation Processing Systems 34, 5720–5730 (2021) 67. Wei, Y., Zhang, Y., Ji, Z., Bai, J., Zhang, L., Zuo, W.: Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 15943–15953 (2023) 68. Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Lopes, R.G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al.: Robust fine-tuning of zero- shot models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 7959–7971 (2022) 69. Wu, H., Gao, Y., Guo, X., Al-Halah, Z., Rennie, S., Grauman, K., Feris, R.: Fash- ion iq: A new dataset towards retrieving images by natural language feedback. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. p. 11307–11317 (2021) 70. Yao, B., Jiang, X., Khosla, A., Lin, A.L., Guibas, L., Fei-Fei, L.: Human action recognition by learning bases of action attributes and parts. In: 2011 International conference on computer vision. p. 1331–1338. IEEE (2011) 71. Yao, J., Liu, E., Rashid, M., Hu, J.: Augdmc: Data augmentation guided deep multiple clustering. Procedia Computer Science 222, 571–580 (2023) 72. Yao, J., Qian, Q., Hu, J.: Customized multiple clustering via multi-modal subspace proxy learning. Advances in Neural Information Processing Systems 37, 82705– 82725 (2024) 73. Yao, J., Qian, Q., Hu, J.: Multi-modal proxy learning towards personalized visual multiple clustering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 14066–14075 (2024) 74. Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF international conference on computer vision. p. 11975–11986 (2023) 75. Zhang, Z., Wang, L., Zhou, L., Koniusz, P.: Learning spatial-context-aware global visual feature representation for instance image retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 11250–11259 (2023) 76. Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision- language models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p. 16816–16825 (2022) Controlling Embedding Spaces with Text-Conditioned Transformations21 Supplementary Overview Section A: Dataset details Section B: Implementation details Section C: Additional experiment details A Dataset Details ShotBench [26]. ShotBench provides annotations for perceptual, camera-related attributes such as camera angle, shot size, composition, and lighting. Although the dataset was originally introduced for visual question answering, we convert the answer strings into categorical labels for each attribute. A small number of answers contain multiple labels for the same category (e.g., “low angle & dutch angle”); these ambiguous cases are removed to ensure a single ground-truth la- bel per category. We use the official split of 31k training images and 2.7k test images. Clevr-4 [56]. Clevr-4 is a synthetic benchmark constructed to isolate four inde- pendent visual attributes: color, count, shape, and texture. Each rendered image depicts multiple 3D objects, and each attribute category contains 10 uniformly sampled subclasses, giving the dataset a clean, balanced structure ideal for learn- ing controlled transformations. We use the official split of 8.4k training images and 2.1k test images. Stanford 40 Actions [70]. Stanford40 consists of 9,532 images spanning 40 human actions. Following prior multi-attribute clustering works [22], we also use the additional annotations provided for two auxiliary categories: location (10 classes) and mood (4 classes), each defined for a disjoint subset of 1,000 images. To supply supervision for all three categories during training, we annotate a small set of 532 images from the original Stanford40 training split with labels consistent with the released location and mood definitions. These annotations do not overlap with the evaluation subsets. For evaluation, we follow the same pro- tocol as prior work: action is evaluated on the remaining 9,000 images, location and mood are evaluated on their respective 1,000 image annotated subsets. FruitSALAD [38]. FruitSALAD is a style–aligned artwork dataset designed to disentangle object identity from artistic appearance. It contains 10 fruit cat- egories and 10 artistic styles, with each fruit–style pairing instantiated by 100 images (for a total of 10,000 samples). This structure makes it well suited for evaluating attribute-aware retrieval. We train with even splits, taking 50 of each pairing instance as training images and the remaining as test images. FER-2013 [11]. FER-2013 is a facial expression recognition dataset containing 35,887 grayscale face images annotated with 7 emotion classes (anger, disgust, fear, happiness, sadness, surprise, neutral). We use the standard split: 28,709 training images and 3,589 public test images. Card [71]. Card is a playing-card recognition dataset with 8,029 images, anno- tated along two independent categories: suit (clubs, spades, diamonds, hearts) 22J. Fioresi et al. and rank (2–10, jack, queen, king, ace). We follow the official split of 7,624 training images, 265 validation images, and 265 test images. MS-COCO [24]. We additionally evaluate on images from the MS-COCO 2017 test2017 split (41k images) to study cross-domain generalization. Since COCO does not provide annotations for the attribute taxonomies used in this work, we generate labels using Qwen3-VL-32B with a structured prompt aligned to the attribute subclasses defined by the source datasets. The hypernetwork is trained only on CLEVR-4, Stanford40, and ShotBench, and applied to COCO without further training. This setting therefore evaluates zero-shot transfer of the learned attribute transformations to a diverse real-world image distribution. FashionIQ [69]. FashionIQ is a benchmark for composed fashion image re- trieval, where each query is defined by a reference image and a natural-language change text describing how to modify it to obtain a target image. The dataset is organized into three product categories (Dresses, Shirts, Tops&Tees). We report results on the official validation split of 6,016 reference–text–target queries. SST-5 [48]. The Stanford Sentiment Treebank (SST) is a sentence-level senti- ment dataset derived from movie reviews, where each sentence is annotated with one of five ordered sentiment classes ranging from very negative to very positive. We use the standard SST-5 splits of 8,544 training and 2,210 test sentences, ex- tracting text embeddings for each sentence and learning sentiment-conditioned transformations in the same manner as our visual-attribute experiments. B Implementation Details Memory Footprint Estimation. Table S6 includes a memory footprint es- timation for maintaining embeddings across 14 distinct attriubtes. The raw storage cost for 1M (N = 10 6 ) vectors, each of dimensionality D = 768 and stored with 32-bit floating-point precision (4 bytes), is N × D × 4 bytes = 1, 000, 000× 768× 4 ≈ 3.07 ≈ 3 GB. Additionally, we assume an HNSW (Hier- archical Navigable Small World) [33] index for fast search is used, incurring a memory overhead estimated at a factor of 2× over the raw vector storage. This results in a total estimated in-memory footprint of 3 GB× 1.5≈ 6 GB for each 1M embeddings. C Additional Experiments Composed Image Retrieval (CIR) Compatibility. To demonstrate that our query-only transformations can enhance existing CIR pipelines, we augment SEARLE-XL [2] on FashionIQ [69]. We keep the gallery fixed by precomputing normalized image embeddings v j = f v (I j ) for all candidates. Given a CIR query (reference image I r and relative caption T), SEARLE-XL produces a composed query embedding q ∈R D in the same CLIP space. Baseline retrieval ranks images by s base (j) = q ⊤ v j . We then apply our query-only transformation using the color attribute text to obtain q ′ , and define a color-aware auxiliary score s color (j) = q ′ ⊤ v j . A lightweight intent parser (Llama 3.2 3B-Instruct) maps the Controlling Embedding Spaces with Text-Conditioned Transformations23 caption T to a signed weight w ∈ [−1, 1] indicating whether color should be preserved (w > 0) or changed (w < 0). Final ranking uses s(j) = α∗ s base (j) + (1− α)∗ w ∗ s color (j),(8) with α = 0.8 and no training on FashionIQ. As shown in Table S1, this sim- ple plug-in reranking consistently improves SEARLE-XL across all categories, supporting that our attribute-consistency metric is compatible with CIR and enhances performance by injecting an explicit attribute prior. Table S1: Composed image retrieval results on FashionIQ. +Ours denotes augment- ing SEARLE-XL [2] with our attribute-consistency metric scoring for just "color", leaving the gallery index unchanged and using no FashionIQ training. DressShirtTopteeAverage MethodR@10 R@50R@10 R@50R@10 R@50R@10 R@50 SEARLE-XL [2]25.9 42.918.2 37.226.5 45.123.5 41.8 + Ours26.3 44.919.8 39.027.8 46.124.6 43.3 Text-based Attribute-aware Retrieval Results. Table S2 shows that our embedding-space manipulation framework is not restricted to visual attributes. When applied to text embeddings on SST-5, learning a sentiment-conditioned transformation consistently improves performance across all evaluation modes. These gains indicate that the proposed mechanism generalizes across embedding spaces independent of the modality. Table S2: Sentiment categorization on SST-5 using text embeddings (E5-large-v2). Our embedding-space transformation improves performance in all three attribute-aware tasks, demonstrating that our approach generalizes beyond vision to textual attributes. SST5 - Sentiment Method Retrieval Clustering Zero-Shot Baseline 30.941.729.6 Ours44.551.354.3 Attribute Abbreviations. For readability in tables, attribute categories are abbreviated as follows: – Clevr-4: Col = color, Cnt = count, Shp = shape, Tex = texture. – Stanford 40: Act = action, Loc = location, Md = mood. – ShotBench: CA = camera angle, CM = composition, LS = lens size, LI = lighting, LT = lighting type, SF = shot framing, S = shot size. 24J. Fioresi et al. Table S3: Category-wise attribute-aware retrieval mAP across multiple datasets. Cat- egory names are abbreviated (full names in Section C). Highlighted columns denote dataset averages. Our approach retrieves meaningful samples while transforming just the query embedding. Clevr-4Stanford 40 ActionShotBench MethodCol Cnt Shp Tex Avg. Act Loc Md Avg.CA CM LS LI LT SF S Avg. Base Features23.0 13.9 79.5 19.033.849.8 40.2 41.243.734.521.232.418.222.524.922.725.2 LCR18.4 11.3 19.2 14.815.96.3 24.8 28.519.922.5 17.9 29.6 15.0 11.9 16.2 19.018.9 CWP67.4 14.2 73.1 20.443.872.244.9 32.349.824.6 19.4 29.6 17.0 15.2 21.8 22.621.4 ArcFace [8]†10.9 10.3 17.5 10.7 12.33.7 24.0 28.818.821.7 17.6 26.3 13.6 11.9 16.0 15.617.5 SupCon [18]†13.2 13.0 14.5 10.212.73.2 24.5 28.718.822.8 17.8 27.4 13.8 11.4 15.0 15.817.7 StableRep+ [51]† 79.6 27.462.2 28.649.510.2 24.7 29.221.323.4 17.6 29.6 14.8 14.2 23.7 23.120.9 MaPLe [17]23.9 16.2 70.2 23.633.545.6 44.7 53.347.832.5 20.3 30.6 17.4 20.8 23.0 21.423.7 LoRA [14]34.4 17.7 83.436.042.946.5 46.255.349.330.1 17.9 31.1 14.7 12.4 24.1 23.822.0 Ours-Query 93.4 39.3 99.4 75.977.089.3 83.9 85.786.352.7 28.0 48.4 22.8 31.8 57.8 45.841.0 Expanded Attribute-aware Retrieval Results. To supplement the dataset- level retrieval results in the main paper, we also report mAP for each attribute category in Table S3. These per-attribute measurements reveal how retrieval per- formance varies across attributes and demonstrate that the query transformation consistently improves attribute alignment across a broad range of properties. Table S4: Mean average precision (mAP) retrieval performance across other bench- marks. Top section is the query (frozen gallery) setting, bottom section is the global transformation setting. Scores are averaged across taxonomies within each dataset. Our method strongly improves retrieval across the board. Method Cards FERFruitsAll Avg.Rank SuitEmotionFruit Style Base Features 37.9 44.625.667.7 58.843.4 LCR10.4 31.018.128.1 23.121.5 CWP55.5 48.018.597.6 53.348.6 Ours-Query 86.0 96.252.799.8 99.781.2 Base Features 37.9 44.625.667.7 58.843.4 LCR54.6 85.4 18.796.8 83.459.6 CWP80.2 84.919.199.1 84.564.5 Ours-Global 90.9 97.9 56.099.8 99.883.4 Additional Datasets: Attribute-aware Retrieval. We report results on three additional benchmarks (Card, FER, and FruitSALAD) to demonstrate broad attribute coverage in Table S4. Across both query-only mode and full space inference, our method maintains strong mAP performance. Additional Retrieval Metrics. To further validate retrieval behavior beyond mean Average Precision, we report Recall@1 and Recall@5 in Table S5. These metrics highlight how early each method surfaces an attribute-consistent result when retrieving from a frozen gallery. Across all datasets, our text-conditioned Controlling Embedding Spaces with Text-Conditioned Transformations25 Table S5: Recall@1 and Recall@5 attribute-aware retrieval performance using a frozen gallery. Results are averaged over all categories within each dataset. Our method con- sistently surfaces an attribute-matching image within the top retrieved results. Clevr4Stanford40ShotBench MethodR@1 R@5R@1 R@5R@1 R@5 Base Features70.1 88.672.7 92.440.8 77.2 LCR19.2 34.69.8 48.417.5 47.5 CWP51.0 69.761.1 81.721.2 50.9 ArcFace [8]†7.9 29.4 20.2 45.019.3 57.6 SupCon [18]†11.4 43.116.7 42.914.5 56.9 StableRep+ [51]† 62.0 89.1 22.1 55.825.3 67.0 MaPLe [17]64.7 85.686.7 96.836.9 74.6 LoRA [14]72.3 90.6 87.6 96.828.9 64.0 Ours-Query 83.8 93.5 98.7 99.851.0 78.1 transformation maintains strong early-rank performance, reliably retrieving an attribute-matching image within the first few items. Table S6: Mean average precision (mAP) retrieval performance across all benchmarks allowing for transformed galleries. Scores are averaged across taxonomies within each dataset. Our method remains competitive with attribute-specific trained models. Mem- ory footprint details explained in Section B. Method Memory Footprint Clevr4 mAP Stanford40 mAP ShotBench mAP Base Features6GB 33.843.725.2 LCR84GB 52.846.722.0 CWP84GB 55.653.223.2 ArcFace [8]†84GB 86.461.138.8 SupCon [18]†84GB 89.166.342.6 StableRep+ [51]† 84GB 88.860.843.1 MaPLe [17]84GB 50.448.623.9 LoRA [14]84GB 79.954.825.1 Ours-Global6GB 85.6 95.741.9 Modified Gallery Attribute-aware Retrieval. In this setting, we remove the frozen-gallery constraint and allow each method to transform both the query and the gallery embeddings. Although this setting is less practical than the query mode, it provides a direct comparison to approaches that rely on full- space adaptation. As shown in Table S6, our learned transform, despite being trained for a query transformation setting, remains competitive with methods that train separate attribute-specific adapters. Importantly, these adapter-based approaches require one model per attribute and a corresponding recomputation 26J. Fioresi et al. Table S7: Category-wise clustering accuracy results across multiple datasets. Category names are abbreviated (full names in Section C). Highlighted columns denote averages across taxonomies. Our approach achieves strong, consistent performance across both semantic and perceptual visual attributes. Clevr-4Stanford 40 ActionShotBench MethodCol Cnt Shp Tex Avg.Act Loc Md Avg.CA CM LS LI LT SF S Avg. Base Features13.1 13.1 89.2 13.932.359.4 58.6 75.864.651.928.238.523.3 26.8 30.3 28.432.5 LCR85.9 28.3 81.9 29.4 56.464.8 65.5 64.264.835.8 24.2 37.5 25.825.5 27.8 34.830.2 CWP80.2 35.6 97.3 28.060.374.8 59.1 60.864.937.2 27.1 38.3 25.829.8 33.933.432.2 Multi-Map CVPR’24 75.3 53.9 65.5 56.5 62.857.9 59.4 71.062.845.1 28.0 38.2 23.8 31.029.9 31.932.6 Multi-Sub NeurIPS’25 84.763.374.9 65.972.260.4 63.1 74.766.144.0 28.0 37.9 24.0 29.9 31.4 31.332.4 IC|TC ICLR’24 53.4 43.5 71.9 62.8 57.977.7 75.0 75.576.127.5 18.0 31.6 20.3 13.7 18.0 23.121.7 SSD-LLM ECCV’24 49.1 44.2 72.0 60.856.581.4 70.470.674.126.7 23.5 29.1 23.2 20.1 29.0 23.425.0 X-Cluster arxiv’25 70.3 65.7 58.4 65.364.982.869.8 52.368.3-------- Ours-Global72.1 59.0 87.673.573.084.4 67.3 88.380.065.9 32.6 61.9 25.9 38.9 74.6 61.851.6 Table S8: Multiple clustering performance across more benchmarks, averaged across taxonomies within each dataset. Our method strongly improves clustering accuracy. Method CardsFERFruitsAll Avg.Rank SuitEmotionFruit Style Base Features 30.4 63.127.052.7 33.138.9 LCR56.5 93.523.086.4 86.561.5 CWP71.9 92.323.971.3 80.860.7 Ours-Global 78.5 99.265.785.3 100.082.4 of gallery embeddings, leading to substantially higher complexity and memory that scale linearly with the number of attributes. Expanded Multiple-clustering Results. To complement the averaged clus- tering results reported in the main paper, we provide per-attribute accuracies for all datasets in Table S7. These finer-grained scores reveal how performance varies across individual attributes, offering a clear view of attribute difficulty and the consistency of the learned transformation. Across nearly all attributes, the global transform maintains stable clustering performance. Additional Datasets: Multiple-clustering. We additionally evaluate cluster- ing on Card, FER, and FruitSALAD to demonstrate broad attribute coverage in Table S8. In all three benchmarks, the global transformation produces stable and competitive clustering accuracy, confirming that the same attribute-aware transform generalizes effectively across diverse visual domains. Conditional Similarity Networks Comparison. Conditional Similarity Net- works (CSN) [57] trains attribute masks while tuning ResNet-18, not satisfying our frozen-gallery constraint. We compare against CSN’s published performance on Zappos50k [57] and against an implementation of their method with a frozen- CLIP-backbone. Table S9 shows our method wins each comparison, with the widest margins on the more semantic Stanford-40 dataset. Hyperparameter Sensitivity Analysis. We evaluate the effect of the multi- modal guidance weight ω on retrieval and clustering performance in Table S10. Moderate guidance improves retrieval while maintaining similar clustering per- Controlling Embedding Spaces with Text-Conditioned Transformations27 Table S9: Comparison to Conditional Similarity Networks adapted baseline. Our method wins each comparison. Dataset CSN (reported) w/ResNet-18 CSN (our impl.) w/ CLIP Ours Clevr-4 (mAP)—70.677.6 Stanford-40 (mAP)—46.584.4 Zappos (triplet acc.)78.880.985.4 formance, with ω = 0.1 performing best overall and therefore used in our ex- periments. As ω increases further, clustering accuracy improves, while retrieval performance begins to degrade. This behavior suggests that stronger text align- ment increasingly reshapes the embedding geometry, which can benefit clustering structure but reduce the fine-grained similarity relationships needed for retrieval. Table S10: Effect of multimodal guidance weight ω on CLEVR-4 retrieval (mAP) and clustering accuracy. Moderate guidance provides the best overall performance, while larger values slightly improve clustering at the expense of retrieval. Loss Weight Retrieval Clustering ω = 072.378.8 ω = 0.0172.479.6 ω = 0.173.479.6 ω = 173.081.5 Table S11: Zero-shot classifica- tion and k-N evaluation (k=5) on CIFAR10. The original class names are not captured in unre- lated "color" transformation, yet k-N evaluation shows that rel- ative similarities are preserved. CIFAR10 MethodZS Acc. k-N Base Features 86.06 97.66 Ours + "color" 52.57 97.48 Does the transformation preserve gen- eral visual structure beyond the target attribute? Although the transformed embed- dings primarily emphasize the desired attribute, we find that meaningful global relationships re- main intact. Table S11 shows that our transfor- mation does not collapse or distort the under- lying feature space merely to satisfy the con- trastive loss. Text-based classification for un- seen class names drops only slightly, and k-N evaluations confirm that relative similarity is preserved—samples that are visually and se- mantically similar remain close. For example, a “blue airplane” remains closer to other blue airplanes than to blue cars, although it is closer to both compared to a "yel- low airplane". This indicates that the transformation sharpens attribute-specific structure without erasing general semantics. 28J. Fioresi et al. Cross-dataset generalization on MS-COCO. To further evaluate general- ization, we test the trained model on images from the MS-COCO test2017 split without any additional training. Since COCO does not contain the attribute taxonomies used in our experiments, we automatically annotate images with the relevant attributes and subclasses using Qwen3-VL-32B with a structured prompting scheme. The trained hypernetwork remains unchanged and is applied directly to the COCO embeddings. Table S12 reports attribute-aware retrieval performance under this setting. Despite the relatively small size of the training datasets and a notable distri- bution shift between the training datasets and COCO (e.g., Clevr-4 provides synthetic object images for learning color and count attributes), our method consistently improves over the baseline across all attribute categories. These results suggest that the learned transformations capture transferable attribute structure encoded in the foundation model embeddings rather than overfitting to the specific training datasets. Table S12: Zero-shot attribute retrieval performance on MS-COCO. A model trained only on the source datasets (CLEVR-4, Stanford40, and ShotBench) is evaluated on COCO images annotated with attribute labels for evaluation. Results show consistent improvements across all attributes, indicating that the learned transformations capture transferable attribute structure beyond the training distributions. MethodAction Location Mood Color Count Camera Angle Shot Size Baseline21.9 36.6 34.9 19.7 30.536.430.7 Ours - Query 27.4 47.1 37.4 24.7 34.842.039.3 Qualitative Results. Beyond quantitative evaluations, we visualize the be- havior of our text-conditioned transformations across a range of attributes and visual domains. Figure S1 visualizes ranked retrieval results on the Fruit-SALAD dataset using the “style” attribute. Our transform causes the retrieval to respect the input attribute first, retrieving different fruit images in the same style before retrieving same fruit images in a different style. Figure S2 highlights retrievals driven by the shot framing attribute, where our method prioritizes compositional structure over object identity. Figure S3 demonstrates robustness to stylistic variation: even with a stylized drawing as input, the “emotion” transform re- trieves semantically aligned expressions. Finally, Figure S4 illustrates emergent compositional control by showing that color, location, and their combination produce distinct and semantically coherent retrieval sets. Collectively, these ex- amples show that the learned transformations cleanly isolate visual attributes, generalize across domains, and support multi-attribute control. Controlling Embedding Spaces with Text-Conditioned Transformations29 Retrieve Same Style “style” Transformed Query: Fruit: Strawberry Style: Crayon Top Retrieval Retrieve Query Image Fruit: Strawberry Style: Crayon Wrong Style (same fruit) Same Style Base CLIP Features: Goal: retrieve images with similar art styles Top Retrieval Fruit: Strawberry Style: Comic Fruit: Strawberry Style: Watercolor Fruit: Strawberry Style: Lomo Fruit: Apple Style: Crayon Fruit: Orange Style: Crayon Fruit: Apple Style: Crayon Fruit: Orange Style: Crayon Fruit: Avocado Style: Crayon Fruit: Strawberry Style: Watercolor Fruit: Strawberry Style: Comic Fig. S1: Retrieval ranked by similarity using base CLIP embeddings and our trans- formed query (“style”). CLIP prioritizes object semantics (e.g., fruit type), whereas our transformation maintains attribute alignment—retrieving different fruits rendered in the same artistic style. Base CLIP Features “shot framing” Transformed Query: Retrieve Retrieve Fig. S2: Top retrieval results comparing CLIP embeddings and our text-guided trans- formation using the query “shot framing.” Our method surfaces images with matching three-subject shot-framing structure. angryangryangryangry angry angry happy disgust fear neutral angry Query Image angry angryangryangryangry angry angry happydisgust fear neutral angry Base Features CLIP L\14 Query Image Transformed Query - “emotion” “shot framing” Transformed Query: Retrieve Retrieve “emotion” Transformed Query: Retrieve angry Base CLIP Features Retrieve Fig. S3: Attribute-aware retrieval using the “emotion” query. Despite the input being a stylized drawing, the learned transform isolates the intended emotion and retrieves aligned “angry” expressions across diverse visual styles. 30J. Fioresi et al. Base CLIP Features “location” Transformed Query: Retrieve Retrieve “color” Transformed Query: “location and color” Transformed Query: Retrieve Retrieve Emergent Compositional Query Handling! Fig. S4: Retrieval comparison illustrating compositional attribute control. CLIP pri- marily retrieves dog-related images, whereas our transformations isolate specific at- tributes: the “color” transform finds dogs of similar color, the “location” transform retrieves dogs on beaches, and the combined query “location and color” retrieves beach scenes with dogs of similar coloration, demonstrating emergent multi-attribute control- lability.