Paper deep dive
StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues
Zanxi Ruan, Songqun Gao, Qiuyu Kong, Yiming Wang, Marco Cristani
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 3:11:49 PM
Summary
StructXLIP is a fine-tuning paradigm for vision-language models that enhances cross-modal retrieval by aligning structural cues. It extracts edge maps from images and filters captions to remove appearance-based terms (color/material), creating 'structure-centric' representations. The method introduces three auxiliary losses: global structure-centric alignment, local structure-centric alignment, and consistency regularization to prevent representation drift. Theoretical analysis suggests this approach maximizes mutual information between structural representations, leading to more robust and semantically stable minima compared to standard CLIP fine-tuning.
Entities (8)
Relation Signals (7)
StructXLIP → uses → Structure-Centric Captions
confidence 95% · filters the corresponding captions to emphasize structural cues, making them 'structure-centric'
StructXLIP → uses → Edge Maps
confidence 95% · StructXLIP... extracts edge maps... treating them as proxies for the visual structure
StructXLIP → improves → Cross-modal Retrieval
confidence 92% · showing that isolating and aligning structural cues across modalities can greatly benefit fine-tuning... with a specific focus on improving cross-modal retrieval
StructXLIP → employs → InfoNCE
confidence 90% · standard alignment methods... minimize the InfoNCE contrastive loss... StructXLIP extends this objective
Canny → istypeof → Edge Maps
confidence 90% · edge maps (e.g., Canny)
StructXLIP → extends → CLIP
confidence 88% · We extend this principle to vision-language alignment... while standard CLIP maximizes...
StructXLIP → uses → SAM
confidence 85% · we apply the Segment Anything Model (SAM) on the original color image to generate a set of visually and semantically coherent masks
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Edge-based representations are fundamental cues for visual understanding, a principle rooted in early vision research and still central today. We extend this principle to vision-language alignment, showing that isolating and aligning structural cues across modalities can greatly benefit fine-tuning on long, detail-rich captions, with a specific focus on improving cross-modal retrieval. We introduce StructXLIP, a fine-tuning alignment paradigm that extracts edge maps (e.g., Canny), treating them as proxies for the visual structure of an image, and filters the corresponding captions to emphasize structural cues, making them "structure-centric". Fine-tuning augments the standard alignment loss with three structure-centric losses: (i) aligning edge maps with structural text, (ii) matching local edge regions to textual chunks, and (iii) connecting edge maps to color images to prevent representation drift. From a theoretical standpoint, while standard CLIP maximizes the mutual information between visual and textual embeddings, StructXLIP additionally maximizes the mutual information between multimodal structural representations. This auxiliary optimization is intrinsically harder, guiding the model toward more robust and semantically stable minima, enhancing vision-language alignment. Beyond outperforming current competitors on cross-modal retrieval in both general and specialized domains, our method serves as a general boosting recipe that can be integrated into future approaches in a plug-and-play manner. Code and pretrained models are publicly available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.20089v3
- Canonical: https://arxiv.org/abs/2602.20089v3
Trouble viewing inline? Open PDF directly →
Full Text
114,605 characters extracted from source content.
Expand or collapse full text
StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues Zanxi Ruan 1 Songqun Gao 2 Qiuyu Kong 1,3 * Yiming Wang 4 Marco Cristani 1,5 1 University of Verona 3 University of Trento 2 Sapienza University of Rome 4 Fondazione Bruno Kessler 5 Reykjavik University Abstract Edge-based representations are fundamental cues for vi- sual understanding, a principle rooted in early vision re- search and still central today. We extend this principle to vision-language alignment, showing that isolating and aligning structural cues across modalities can greatly ben- efit fine-tuning on long, detail-rich captions, with a spe- cific focus on improving cross modal retrieval. We intro- duce StructXLIP, a fine-tuning alignment paradigm that extracts edge maps (e.g., Canny), treating them as proxies for the visual structure of an image, and filters the corre- sponding captions to emphasize structural cues, making them “structure-centric”. Fine-tuning augments the stan- dard alignment loss with three structure-centric losses: (i) aligning edge maps with structural text, (i) matching lo- cal edge regions to textual chunks, and (i) connecting edge maps to color images to prevent representation drift. From a theoretical standpoint, while standard CLIP maxi- mizes the mutual information between visual and textual embeddings, StructXLIP additionally maximizes the mu- tual information between multimodal structural represen- tations. This auxiliary optimization is intrinsically harder, guiding the model toward more robust and semantically stable minima, enhancing vision-language alignment. Be- yond outperforming current competitors on cross-modal re- trieval on both general and specialized domains, our method serves as a general boosting recipe that can be integrated into future approaches in a plug-and-play manner. Code and pretrained models are publicly available at:https: //github.com/intelligolabs/StructXLIP. * Corresponding author. CLIP-family retrieval The picture shows a rugged natural scene centered on a sharp rock emerging from the ground. The rock has angular, pointed edges and casts a distinct shadow that emphasizes its form. Its surface is rough, with visible cracks and mineral textures catching the light. QueryInput ℰ(I) StructXLIP ℱ(T) GenericCLIP Framework + Training images The image shows an adult elephant in a savanna. Its large grey-brown body stands firmly on sturdy legs, and long ivory tusks curve outward. The elephant’s wrinkled skin contrasts with the warm ochre grass and green vegetation in the background. The image shows an adult elephant in a savanna. Its large grey-brown body stands firmly on sturdy legs, and long ivory tusks curve outward. The elephant’s wrinkled skin contrasts with the warm ochre grass and green vegetation in the background. The image shows an adult elephant in a savanna. Its large grey-brown body stands firmly on sturdy legs, and long ivory tusks curve outward. The elephant’s skin contrasts with the warm ochre grass and green vegetation in the background. Training captions Edge maps The image shows an adult elephant in a savanna. Its large grey-brown body stands firmly on sturdy legs, and long ivory tusks curve outward. The elephant’s wrinkled skin contrasts with the warm ochre grass and green vegetation in the background. The image shows an adult elephant in a savanna. Its large grey-brown body stands firmly on sturdy legs, and long ivory tusks curve outward. The elephant’s wrinkled skin contrasts with the warm ochre grass and green vegetation in the background. The image shows an adult elephant in a savanna. Its large grey-brown body stands firmly on sturdy legs, and long ivory tusks curve outward. The elephant’s skin contrasts with the warm ochre grass and green vegetation in the background. StructXLIPretrieval Inference + Fine-tuning 123 123 Structure-centrictext analysis 3 Figure 1. StructXLIP performs fine-tuning by adding to the stan- dard image-text alignment with multimodal structural cue: edge maps and structure-centric captions. StructXLIP consistently im- proves downstream vision–language retrieval inference. 1. Introduction Vision-language models (VLMs) [26,49] have become foun- dational ingredients in modern computer vision, thanks to their aligned representations across visual and textual modal- ities. While the pre-trained VLMs, such as CLIP [26], have demonstrated impressive generalization capability, fine- tuning is often required to be competitive on downstream tasks in specific domains [5,8–10,12,13,17,22,45,47,48, 51,53,56,57,59]. Yet, fine-tuning VLMs remains challeng- ing when images contain rich visual structures and captions arXiv:2602.20089v3 [cs.CV] 2 Mar 2026 are long and semantically dense. Pre-trained VLMs [26,49] are limited in such scenarios due to the token-length con- straint [3,44,51]. To overcome this limitation, Long- CLIP [51] extends CLIP through positional interpolation to accommodate longer captions. More recent advances, such as GOAL [3] and SmartCLIP [44], improve the effi- ciency of long-text alignment by reinforcing global-local correspondences and modular token-level interactions. Yet, these approaches rely on the semantic patterns learned from pre-training corpora, and their generalization can degrade in domains where visual or linguistic cues are scarce or poorly represented. Departing from such dependency on semantic alignment, we introduce StructXLIP, a vision-language fine-tuning framework that shifts the focus toward a more fundamen- tal and accessible signal: the geometrical structure of the image, expressed through its edges. In decades of com- puter vision research, edges have long been recognized as essential primitives for visual understanding [20,31], en- coding object boundaries and spatial layout. Building on this foundation, StructXLIP extracts edge maps to capture visual structures (see Fig. 1). Crucially, for the first time, we extend this structural perspective to language counter- part. We filter structure-centric captions by removing tokens tied to appearance-driven cues as color and material, aiming to retain only linguistic elements that feature shape, geom- etry, and spatial relations. We pair edge maps with such structure-centric text, enforcing multimodal alignment on the basis of structural cues rather than appearance. The alignment process supplements existing text-image align- ment with three structure-centric losses. The first aligns edge maps with the filtered text, the second establishes fine- grained correspondences between local edge regions and chunks of the filtered text, and the third links edge maps to the original color images to prevent representational drift. After fine-tuning, StructXLIP yields encoders that process simple images and text and produce embeddings enriched with structure-centered information, resulting in stronger alignment. Importantly, inference requires no additional processing and introduces no extra computational overhead. From a theoretical standpoint, StructXLIP can be inter- preted through an information-theoretic lens. Standard align- ment methods [26,51] minimize the InfoNCE contrastive loss, which corresponds to maximizing a mutual information lower bound between images and text [24,25]. StructXLIP extends this objective by adding a second maximization composite term focused on structure-centric multimodal representations. This auxiliary objective acts as a corre- lated, information-reduced task that introduces gradient di- versity [58] without interfering with the main optimization. In fact, information-theoretic analyses of multitask learn- ing [28] show that auxiliary losses with lower mutual infor- mation act as an implicit regularizer, expanding the effective search space and improving convergence. StructXLIP sets new state-of-the-art cross-modal retrieval results on both general-purpose dense-caption benchmarks and domain-specific datasets dominated by fine visual de- tails, such as fashion and fine-grained classification. Its three structure-centric losses also demonstrate potential as a “universal” booster: when plugged into existing fine-tuning frameworks, performance improves across the board while maintaining competitive generalization. Ablations show that any edge extractor is effective, with classical opera- tors like Canny and LoG performing the best. Our explain- ability analyses also confirm that the model consistently at- tends to human-interpretable, shape-driven regions. Notably, StructXLIP excels even in low-data regimes, benefiting from a strong structure-centric inductive bias that enables robust learning with limited data and supervision. Our main contributions are as follows: • We show that injecting structure-centric, multimodal cues into contrastive learning substantially improves long-text vision-language alignment. •StructXLIP outperforms recent fine-tuning methods with- out adding architectural complexity. • Our mutual-information and optimization analysis ex- plains why structure-centric alignment stabilizes fine- tuning and accelerates convergence. •Our structure-centric alignment losses serve as a plug- and-play booster to diverse fine-tuning frameworks, while maintaining competitive generalization. • Extensive experiments confirm the effectiveness of our loss designs, and the robustness to different edge/textual extraction choices and qualities. 2. Related Works Vision-language alignment. Contrastive learning on image- text pairs remains the dominant paradigm for VLMs, with two main loss formulations: CLIP’s InfoNCE objective [26] and SigLIP/SigLIP2’s pairwise sigmoid matching [35,49]. Subsequent dual-encoder variants mostly improve this frame- work through debiasing, adaptive weighting, or geometric regularization (DeCLIP [38], ALIP [46], GOAL [3], Smart- CLIP [44], CoAPT [39], AVSE [18]). Architectural scal- ing (EVA-CLIP [30], Florence-2 [42]) and text-capacity extensions such as Long-CLIP [51] to enable long-text handling. Fine-tuning strategies span full-model adapta- tion to parameter-efficient tuning: Double-Filter, FATE, MLMP [9,22,45], text-only tuning (LiT [48]), prompt learn- ing (CoOp, CoCoOp, MaPLe, ProGrad [13,56,57,59]), and lightweight adapters or low-rank modules (CLIP-Adapter, Tip-Adapter, LoRA, DoRA, Mma, Mmrl, UCDR-Adapter [5, 8,10,12,17,47,53]). These methods retain the standard contrastive setup and mainly enhance alignment efficiency or generalization. VLMs for long-text alignment. Despite recent advances in visual-textual alignment, most VLMs remain constrained by the∼77-token limit of CLIP-style text encoders, hinder- ing their ability to model rich detail descriptions. This has motivated a line of methods targeting long-text alignment in VLMs. Position-encoding extensions (Long-CLIP [50], TULIP [21], V2PE [6], FineLIP [1]) expand effective con- text length while preserving pretrained semantics. Caption decomposition approaches (DreamLIP [55], longAlign [16]) align sub-captions with local regions for finer ground- ing. Dataset- or structure-driven strategies introduce long- text corpora and learnable structural tokens (LoTLIP [41]) or region-level masked captioning [36].LLM-assisted pipelines such as MATE [11] further project long texts into the VLM embedding space through multi-stage transforma- tion. These methods operate on CLIP-based foundations and differ mainly in how rich semantics can be better aligned via fine-tuning among both modalities. Differently, StructXLIP augments standard contrastive fine-tuning with structural cues, serving as general booster to further enhance vision- language alignment. 3. StructXLIP StructXLIP performs fine-tuning in two stages (Fig. 3). In the structure-centric multimodal extraction stage, it pro- duces complementary structural views by generating edge maps and filtering captions to retain object layout and geo- metric cues by removing appearance terms. In the structure- centric alignment stage, these structural representations are aligned at both global and local levels together with the orig- inal image-text pairs, using dedicated losses to strengthen alignment under semantically rich conditions. The structure- centric signals are used only during fine-tuning. Inference solely involves original images and texts, thus no additional cost compared to CLIP, and no need for edge extraction nor structure-centric text filtering. 3.1. Structure-centric multimodal extraction Let us consider a color imageI ∈ Vwhich is visually de- scribed by a textual descriptionT ∈ L. Each imageI i and its visual description are paired (I i , T i ). Visual extraction. For each imageI i , we extract its struc- tural view asI ′ i =E(I i ), whereE(·) denotes any filter- based [2,20] or learning-based [19,43,52] edge detector. StructXLIP is edge-extractor-agnostic as shown in Sec. 4. Textual extraction. Complementary to the edge-based vi- sual cues, we extract a structure-centric textual description that emphasizes layout, geometric attributes, and spatial re- lations, without appearance-related terms. While directly prompting the LLM to perform rewriting of the original T i seems a straightforward solution, it is fragile to halluci- nations [32]. Alternatively, we obtain the structure-centric textual description by lexicon filtering to remove appearance terms, specifically, color and materials, from the original SKETCHY DOCCI a miniblue gem-studdedmodalfabriczip-up dress with a scoop neckline, lining, and a symmetrical design, featuring a normal waist, ruched and straight elements, with a plain pattern and a gem, and without beads. ...The carriage has fourredwoodenwheels. The hood of the carriage is folded upandisblack. The carriage is sitting on awooden porch... A whitewooden house is beside the carriage with a double door.Brownwooden planks are beside the house... Figure 2. Examples of Visual (left) and Textual (right) extraction. captions. We leverage the strong semantic understanding capability of a large language model (LLM), prompting it to construct a set of appearance-related vocabularyV a to be filtered from the original description. Specifically, V a =LLM(P), wherePis the prompt that queries the LLM to produce semantic concepts related to colors and materials, resulting in the vocabularyV a . We then obtain the structure- centric textT ′ i by filtering out terms that are insideV a from T i :T ′ i =F(T i ,V a ), whereF(·) is the filtering function based on regular expression matching to remove any semantic con- cept are withinV a , we refer to this process as the Lexicon Filter. Please refer to Supp. Mat. for the promptPand vo- cabularyV a used in the filtering processes. Fig. 2 illustrates examples of the extracted edge maps and the corresponding structure-centric captions. 3.2. Structure-centric multimodal alignment Let us consider a VLMf VLM with two main elements: the visual encoderf img and the text encoderf txt . As shown in Fig. 3, the encoders map their respective inputs into a sharedd-dimensional embedding space, i.e.,f img :V → R d andf txt :L→ R d . The resulting features arei i =f img (I i ) andt i =f txt (T i ), with structure-centric counterpartsi ′ i and t ′ i . The original image-text alignment lossL I,T , serving as the alignment objective basis, is expressed as a symmetric In- foNCE loss [24] with cosine similarity as similarity measure. In addition, we introduce three auxiliary learning objectives aligns structure-centric multimodal contents at multiple lev- els: the Structure-centric image-text alignment lossL I ′ ,T ′ enforces global structural alignment, the Local structure- centric image-text alignment lossL local I ′ ,T ′ captures part-level compositional semantics, and the Consistency regularization lossL I,I ′ aligns raw images and structure-centric views to prevent modality drift. Note that our auxiliary structure- centric losses, could be flexibly integrated to any finetuning framework, as we show in Sec. 4.1. Structure-centric image-text alignmentL I ′ ,T ′ . Using the structure-oriented visual and textual views, we introduce a contrastive objective that alignsI ′ andT ′ . We compute their cosine similarity with a dedicated temperatureτ ′ , and apply a symmetric InfoNCE loss over the batch. This loss has the same form as the standard image-text contrastive loss, but operates purely on structural cues to enforce multimodal A large rounded boulder balanced on grainy gray rocks on a beach. A large rounded boulder balanced on grainy gray rocks on a beach. Edge Detector Structure-centric image-text alignment (a) Structure-centric multimodal extraction Consistency regularization (b) Structure-centric multimodal alignment SAM On RGB image SAM regions on edge map Boulder + A large rounded boulder balanced on grainy gray rocks on a beach. Local structure- centric image-text alignment Local structure- centric image-text alignment INPUT A large rounded boulder balanced on grainy gray rocks on a beach. A large rounded boulder balanced on grainy gray rocks on a beach. Edge Detector Appearance Vocabulary Lexicon Filtering Structure-centric multimodal extraction Consistency regularization (b) Structure-centric multimodal alignment SAM On RGB image SAM regions On Edge map Boulder + A large rounded boulder balanced on grainy gray rocks on a beach. Local structure- centric image- text alignment Local structure- centric image-text alignment INPUT (a) (i) StructXLIP(i) Zoom-in on Structure-centric image-text alignment Figure 3. (i) Overview of StructXLIP fine-tuning which operates in two stages. The first structure-centric multimodal extraction stage extracts structural views by generating edge maps from original images via edge detector and performing lexicon filtering on original captions to remove appearance-related terms. The second structure-centric multimodal alignment stage fine-tunes the encodersf img and f txt by joining original image-text alignmentL I,T with our newly introduced structure-centric alignment objectives: the structure-centric image-text alignment lossL I ′ ,T ′ enforces global structural alignment, the Local structure-centric image-text alignment lossL local I ′ ,T ′ captures local compositional semantics, and the Consistency regularization lossL I,I ′ aligns raw images and edge maps to prevent representation drift. At inference, only color image (I) and captions (T) are input to the fine-tunedf img andf txt , neither edge extraction nor lexicon filtering is required. (i) Zoom-in on the Local structure-centric image-text alignment lossL local I ′ ,T ′ . alignment between the structure-preserving representations. Local structure-centric image-text alignmentL local I ′ ,T ′ . In addition toL I ′ ,T ′ , which features the global alignment be- tween betweenI ′ withT ′ , we further introduce a local-level alignment to capture the fine-grained correspondences of atomic structural semantic, as depicted in Fig. 3. Inspire by Local Image–Sentence Matching (LISM) [3], we first segment both the structure-centric imageI ′ and the textT ′ , where the segments from both modalities are then aligned during fine-tuning. Specifically, we apply the Segment Any- thing Model (SAM) [14] on the original color imageI i to generate a set of visually and semantically coherent masks K i . With these visual masks, we then obtain the set of lo- cal structure-centric visual regionsR i . On the textual side, we segment the textual counterpartT ′ into multiple phrases based on sentence delimiters (e.g., periods or semicolons), forming a set of text chunksC i of sizeM i that varies based on each textual description. We employ a multi-positive con- trastive learning strategy to jointly model the one-to-many correspondence between each local phrasec m ∈ C i and a total ofKstructure-centric visual regions that are most semantically alignedR K m based on the pre-trained VLM en- coders. For each text chunkc m of structure-centric imageI ′ i , the local alignment loss ℓ m is: ℓ m =− log P r k ∈R K n exp γ f txt (c m )· f img (r k ) P r j ∈R B exp γ f txt (c m )· f img (r j ) ,(1) whereγis the temperature andR B represents the set of visual local regions within a batch. The final lossL local I ′ ,T ′ is computed by averagingℓover the visual regions of all samples in the batch of size N: L local I ′ ,T ′ = 1 N N X i=1 1 M i M i X m=1 ℓ m .(2) L local I ′ ,T ′ does not require additional annotations, and its multi- positive formulation allows each text phrase to align with multiple structure-centric visual regions simultaneously, en- hancing the model’s capacity to capture compositional se- mantics and multi-part descriptions. Consistency regularizationL I,I ′ . While the structure- centric alignment provides additional supervision, its repre- sentation may gradually deviate from the original semantic manifold during the VLM fine-tuning. To mitigate this is- sue, we introduce the consistency regularization lossL I,I ′ , using the visual representation of the original image to an- chor the updated representation within the pre-trained vision- language latent space. L I,I ′ in the N-batch is computed as: L I,I ′ = 1 N N X i=1 1− < f img (I i ), f img (I ′ i ) > .(3) L I,I ′ serves as a regularization to ensure fine-tuning stability and semantic consistency. Full Learning Objective. Our overall structure-centric lossesL ∗ can be combined in a weighted manner: L ∗ = λ 1 L I ′ ,T ′ + λ 2 L I,I ′ + λ 3 L local I ′ ,T ′ ,(4) whereλ 1–3 are weight coefficients that are empirically set. The full objective can then be expressed byL total =L I,T +L ∗ . Note thatL ∗ are axillary and additional, which can be easily integrated into any fine-tuning framework and losses, substi- tutingL I,T . Through this fine-tuning objective, StructXLIP can effectively enhance the multimodal alignment in appear- ance and structural semantics, especially for long texts with rich semantic details. 3.3. Theoretical view We analyze StructXLIP from an information-theoretic per- spective [25,40], following common practice in contrastive representation learning. Our analysis is supported by nu- merical simulations performed across all four experimental datasets (see Sec. 4), multiple training data regimes (Sec. 4, Sample efficiency experiment), and considering StructXLIP a s plug-in to diverse alignment strategies (Finetuning- agnostic improvement experiment). Here we report one representative study (Sec. 4 Tab. 1, Sketchy dataset), with the remaining experiments provided in the Supp. Mat. The overall objectiveL total maximizes the mutual infor- mationI MI (I,T) between the image representationIand its corresponding textT. Our analysis focuses on the three lossesL I,T ,L I ′ ,T ′ andL I,I ′ , since the local lossL local I ′ ,T ′ can be seen as a specification ofL I ′ ,T ′ . LetE(·) andF(·) be 010k20k30k40k50k60k70k80k 0.00 0.05 0.10 0.15 0.20 0.25 Loss (a) I, T I ′ , T ′ Convergence I, T Convergence I ′ , T ′ 010k20k30k40k50k60k70k80k 0.00 0.02 0.04 0.06 Gradient Norm (b) I, T I ′ , T ′ Ratio I ′ , T ′ I, T 010k20k30k40k50k60k70k80k Fine-tuning Step 0.4 0.6 0.8 1.0 Cosine Similarity (c) I, T , I ′ , T ′ I, TI ′ , T ′ 0 5 10 15 Ratio Ratio=1 Figure 4. Empirical analysis confirming the information-theoretic view of StructXLIP. a) Loss convergence; b) Gradient norms and ratio evolution; c) Inter-task cosine similarity. deterministic mappings that produce information-reduced counterpartsI ′ =E(I) andT ′ =F(T), respectively. In prac- tice,I ′ is the edge map contains binary0, 1values instead of RGB triplets, andT ′ is stripped of color and material terms. By the Data Processing Inequality [33], the compressed rep- resentations satisfy I MI (I ′ , T ′ )≤ I MI (I, T). 1 As commonly 1 We also verify numerically by KSG mutual information neural es- done in information-theoretic representation learning, we approximate the mutual information quantities using their InfoNCE lower bounds [25,40]:I MI (I,T)≥ log N−E[L I,T ] andI MI (I ′ ,T ′ )≥ log N−E[L I ′ ,T ′ ], whereNis the batch size. Fig. 4 a) shows the two lossesL I,T andL I ′ ,T ′ . We observe that the gradients of the two losses,∇ θ L I,T and∇ θ L I ′ ,T ′ , are positively correlated in their direction (Fig.4 c)), consid- ering cosine similarity (μ= 0.6872,σ= 0.1028, min= 0.57, max= 0.89), indicating that the two optimization objectives pursue compatible optima in the parameter space. Empiri- cally, we observed that the consistency lossL I,I ′ strengthens this correlation, while disabling it does not significantly re- duce the correlation, indicating that the two problems are inherently related. However, sinceI ′ andT ′ encode less information, the aux- iliary maximizationI MI (I ′ ,T ′ ) is inherently more difficult, leading to a later convergence ofL I ′ ,T ′ , as evidenced from Fig.4 a) where we show the iteration after which we mea- sured convergence of the two losses, and Fig.4 b) where we plot the magnitude of the gradients of∇ θ L I,T and∇ θ L I ′ ,T ′ and their ratio. Convergence has been evaluated by adopting the two-regime learning-curve method of Gaussian-process theory, which defines the onset of convergence as the point where model error becomes comparable to or lower than the estimated noise level [29, 37]. As a result,L I ′ ,T ′ provides a persistent and informa- tive gradient even whenL I,T flattens, effectively steering optimization toward semantically coherent minima. This mechanism can be interpreted via multitask optimization and gradient diversity. Information-theoretic analyses of multitask learning [28], multi-objective optimization the- ory [58] and contrastive representation learning [40] show that auxiliary objectives with lower mutual information act as implicit regularizers, introducing controlled gradient di- versity that expands the effective search space and improves convergence stability. Our structure-centric auxiliary align- mentL I ′ ,T ′ plays exactly this role: it introduces a correlated, information-reduced objective that contributes complemen- tary gradient directions, maintaining active updates when the main contrastive gradients approach zero. 4. Experiments Datasets. We evaluate StructXLIP on both general-domain and specific-domain benchmarks featuring long, information- rich descriptions. For the general domain, we consider DCI [36] and DOCCI [23], two human-annotated dense captioning datasets with long, detailed descriptions. DCI contains 7.8k images with mask-aligned captions exceeding 1000 words on average, while DOCCI provides 15k diverse scenes paired with highly discriminative descriptions aver- aging 136 words. For specific-domain evaluation, we adopt timator [15] thatI MI (I ′ ,T ′ )≃0.20;I MI (I,T)≃0.52 on the DOCCI dataset [23] in the experiment of Tab. 1. SKETCHY [7], a 46k multimodal fashion dataset with fine visual attributes, and Insect [34], a 6k fine-grained biology dataset with expert-verified morphological descriptions (81 words on average). For the dataset splits, we follow the protocols of [3] for DCI and DOCCI, use the original split for SKETCHY, and adopt an 8:2 train/test split for Insect. Further dataset details in the Supp. Mat. Compared methods. We consider sota CLIP finetuning methods for long-text alignment: Long-CLIP [50] extends the text encoder via positional interpolation, FineLIP [1] improves long-text understanding through multi-stage fine- tuning with adaptive attention fusion, SmartCLIP [44] uses dynamic token routing and progressive context extension to emphasize salient segments, and GOAL [3] jointly models global/local correspondences for fine-grained alignment. Performance metrics. We report Recall@K (K=1/5/10) of the cross-modal retrieval task in bothText→Image (T→I) andImage→Text (I→T )settings. Recall@K measures the proportion of queries for which the corresponding ground- truth match appears within the top-K retrieved results. Each image is unique and paired to one textual description. Implementation details. As visual encoderf img we use ViTB/16 [4] and as text encoderf txt , we employ the Knowledge Preserving Stretching (KPS) strategy from Long- CLIP [50] to extend CLIP’s text encoder for long-text inputs. We report in main comparison with edges extracted by the Canny detector. Visual inputs are resized to 224×224. For fair comparison, all methods share the same tokenizer, batch size, and fine-tuning budget. We empirically setλ 1 = 0.25, λ 2 = 0.1, andλ 3 = 0.1, as this configuration consistently provided strong overall performance. Both models are fine- tuned for 10 epochs on an NVIDIA RTX 5090 GPU, with each run taking approximately 1–3 hours depending on the dataset. Results are averaged over 3 random seeds; we report mean values (and±std in the appendix). For GOAL [3], the results on the DOCCI and DCI datasets reported in Tab. 1 are taken directly from the original paper. For reference, the per- batch runtime of our method is 0.17s. The underlying CLIP variants used in our experiments span a similar range, with Long-CLIP running at 0.10s, GOAL at 0.18s, SmartCLIP at 0.07s, and FineLIP at 0.05s. Inference time is equivalent across all approaches, since it only involves projecting data into the embedding space. Additional implementation details and computational analysis in the Supp. Mat. 4.1. Main comparisons Best-performing on cross-modal retrieval. As shown in Tab. 1, our StructXLIP consistently achieves the best recall at every rank on both directions (T→I / I→T), among all com- pared methods, except on DCI R@5 I→T. On the SKETCHY dataset, the improvement obtained by StructXLIP is most significant: R@1 on the increases by +6.65% and +5.78% compared to the second-best method GOAL, on the T→I SKETCHY DOCCI ↑3.28 ↑0.70 35.75 36.70 ↑3.02 49.48 ↑4.14 59.41 ↑5.78 ↑6.57 53.20 ↑5.18 60.97 ↑6.65 69.86 68.22 ↑2.47 ↑1.91 69.76 68.92 ↑3.53 76.47 ↑2.72 78.45 ↑2.16 81.59 ↑5.75 77.18 ↑4.53 79.78 ↑3.57 83.04 Figure 5. Data amount vs. performance: Retrieval recall R@1 in case of 5%, 20%, 50% and the standard 100% of training data. The improvement line indicates the margin of StructXLIP against the second best approach, trained with the same amount of data. and I→T task, respectively. Despite being with the short- est word length, SKETCHY dataset contains rich structure- centric multimodal content concerning clothing items (e.g., shape, layering, designs, contour, and outfit compositions), making our structure-centric multimodal alignment most effective to enhance the representation alignment on fine- grained details. On the other hand, the performance of all models on the Insect dataset is relatively low as the dataset belongs to a rare domain, i.e. specific families of insects, that is underrepresented in the pre-training dataset for VLMs. Nevertheless, StructXLIP still improves R@1 in the T→I direction by +1.12% (a relative gain of 12.7%) with stable improvements for I→T. We believe this is attributed to the edge maps that provide a degree of morphological invariance across species highlighting subtle visual differences during the text-visual alignment. Finally, StructXLIP also shows consistent gains over compared methods on the general- domain datasets DOCCI and DCI, demonstrating its broad effectiveness. We also perform strongly at deeper ranks, detailed results are provided in the Supp. Mat. Finetuning-agnostic improvement. Our introduced losses L ∗ are additional auxiliary losses, which can be flexibly in- tegrated on top of any CLIP-based finetuning methods. To investigate its effectiveness, we experiment with both classic parameter-efficient finetuning (PEFT) techniques, including LoRA [10] and DoRA [17], and the state-of-the-art (sota) CLIP-based finetuning methods presented in the main com- parison (Tab. 1). As shown in Tab. 2,L ∗ is plug-and-play on any CLIP-based finetuning techniques, demonstrating XLIP-agnostic improvements on both general and specific domains 2 . The benefits also extend to the adapter-based PEFT techniques, when finetuning models with a few addi- tional learnable parameters. In PEFT,L ∗ with LoRA and DoRA achieves about +4-5% R@1 improvement, showing that it can serve as a lightweight and plug-and-play enhance- ment module to improve multimodal alignment with negligi- ble additional computational cost. Notably, on SKETCHY we get an average improvement of 6.20% (relative improve- ment of 12.5%) at R@1 and on DOCCI an average improve- 2 Results on the other datasets will be shown in the Supp. Mat. Table 1. Crossmodal retrieval performance of CLIP-based finetuning methods on multiple datasets. We report Recall@K (%) on both Text→ImageandImage→Textsettings. Each row represents one method evaluated on two datasets (left–right paired). Best results in bold; second best underlined.∆ denotes the margin over our best competitor, with gain in↑ green. Method SKETCHYINSECTDOCCIDCI R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10 Long-CLIP[ECCV’24] 54.32 80.14 88.43 52.76 80.31 88.088.20 23.83 34.97 9.4124.78 37.3164.49 87.67 93.43 63.08 87.45 93.1459.23 80.89 87.04 60.13 81.44 87.54 FineLIP[CVPR’25]40.59 71.16 81.78 40.33 72.11 82.388.46 23.32 33.596.86 23.75 34.4667.80 90.22 94.84 66.39 89.12 94.4766.13 85.34 89.79 64.58 84.59 89.54 SmartCLIP[CVPR’25] 50.73 81.09 94.56 51.30 80.83 94.044.84 16.84 34.634.66 15.46 34.0274.92 94.08 97.31 74.91 94.04 97.2969.88 86.64 94.0570.94 87.04 92.77 GOAL[CVPR’25]63.2187.1393.44 62.4487.8292.318.8124.3535.848.55 25.9136.1879.4796.6598.6979.4396.1497.2572.6489.8993.70 72.8490.50 93.20 StructXLIP69.86 90.85 95.42 68.22 90.67 95.689.93 26.60 38.34 9.50 26.60 39.6483.04 97.06 98.96 81.59 96.94 98.7875.90 90.00 95.15 74.39 89.9094.30 ∆↑ 6.65 ↑ 3.72 ↑ 0.86 ↑ 5.78 ↑ 2.85 ↑ 1.64↑ 1.12 ↑ 2.25 ↑ 2.50 ↑ 0.09 ↑ 0.69 ↑ 2.33↑ 3.57 ↑ 0.41 ↑ 0.27 ↑ 2.16 ↑ 0.80 ↑ 1.49↑ 3.26 ↑ 0.11 ↑ 1.10 ↑ 1.55 ↓ 0.60 ↑ 1.10 Table 2. Plug-and-play enhancement of ourL ∗ on CLIP-based finetuning. Results on SKETCHY and DOCCI forText→Image andImage→Textretrieval. Upper: full-parameter finetuning; lower: parameter-efficient tuning. Our method consistently boosts diverse CLIP variants. Best in bold, with gain in↑ green. Method SKETCHYDOCCI R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10 Long-CLIP 54.3280.1488.4352.7680.3188.0864.4987.6793.4363.0887.4593.14 +ourL ∗ 59.2485.3291.4559.5984.3791.0267.6790.8295.5967.9290.1695.10 ∆↑ 4.92↑ 5.18↑ 3.02↑ 6.83↑ 4.06↑ 2.94↑ 3.18↑ 3.15↑ 2.16↑ 4.84↑ 2.71↑ 1.96 FineLIP40.5971.1681.7840.3372.1182.3867.8090.2294.8466.3989.1294.47 +ourL ∗ 59.1585.2391.2858.5584.5490.0774.0694.2497.3572.9493.2796.55 ∆ ↑ 18.56↑ 14.07↑ 9.50↑ 18.22↑ 12.43↑ 7.69↑ 6.26↑ 4.02↑ 2.51↑ 6.55↑ 4.15↑ 2.08 SmartCLIP 50.7381.0994.5651.3080.8394.0474.9294.0897.3174.9194.0497.29 +ourL ∗ 52.9481.2694.7752.3380.9294.0477.3995.5798.6677.1095.4998.34 ∆↑ 2.21↑ 0.17↑ 0.21↑ 1.03↑ 0.090.00↑ 2.47↑ 1.49↑ 1.35↑ 2.19↑ 1.45↑ 1.05 GOAL63.2187.1393.4462.4487.8292.3179.4796.6598.6979.4396.1498.51 +ourL ∗ 67.8890.3395.1668.4889.8194.8280.9696.9098.9680.3196.7398.84 ∆↑ 4.67↑ 3.20↑ 1.72↑ 6.04↑ 1.99↑ 2.51↑ 1.49↑ 0.25↑ 0.27↑ 0.88↑ 0.59↑ 0.33 SigLIP268.9190.8595.1666.7590.2493.9671.8092.5395.8871.5192.4196.06 +ourL ∗ 73.4991.9795.7770.3891.5495.7775.4794.8297.6773.5994.3397.43 ∆↑ 4.58↑ 1.12↑ 0.61↑ 3.63↑ 1.3↑ 1.81↑ 3.67↑ 2.29↑ 1.79↑ 2.08↑ 1.92↑ 1.37 LoRA57.0884.7291.8056.7485.3291.7177.8096.4598.5577.0096.0298.05 +ourL ∗ 62.0986.7993.9559.4185.9292.7579.3596.6198.5778.6596.3198.45 ∆↑ 5.01↑ 2.07↑ 2.15↑ 2.67↑ 0.60↑ 1.04↑ 1.55↑ 0.16↑ 0.02↑ 1.65↑ 0.29↑ 0.40 DoRA61.7786.1891.8860.9487.3392.3166.2790.5095.1865.6589.5594.78 +ourL ∗ 65.2090.2694.9164.9488.3593.5270.6592.4796.5769.2291.8095.98 ∆↑ 3.43↑ 4.08↑ 3.03↑ 4.00↑ 1.02↑ 1.21↑ 4.38↑ 1.97↑ 1.39↑ 3.57↑ 2.25↑ 1.20 ment of 3.28% (relative improvement of 4.72%). Also, it seems that in general the positive margin diminishes as the recall ranking does augment; nonetheless, it is important to say that no negative margins (=worsening) happen at higher recall ranks, as we show in the Supp. Mat.. Generalization across general domains. Following the evaluation of [3], we also investigate the finetuned models in cross-domain settings on the two general-domain DCI and DOCCI dataset. Tab. 3 shows that StructXLIP demonstrates decent cross-domain generalization capability between DCI and DOCCI, with only a marginal decrease compared to in- domain results. Nonetheless, StructXLIP still outperforms GOAL and Long-CLIP by a clear margin. This indicates that the auxiliary structure-centric alignment does not bias the model toward domain-specific representation, but instead promotes generalizable representation that can transfer well across different image-text distributions. In the case of cross general-specific domain generalization (see Supp. Mat.), the decrease in performance is evident, with StructXLIP still showing comparable if not better robustness w.r.t. the Table 3. Cross-domain generalization between DCI and DOCCI. Train on one dataset and test on another. Values are Recall@K (%), usingText→ImageandImage→Textretrieval. In-domain best in italic bold, cross-domain best in bold. SettingR@1R@5R@10R@1R@5R@10 Train on DCI→ Test on DCI vs DOCCI Long-CLIP (DCI→DCI)59.2380.8987.0460.1381.4487.54 Long-CLIP (DCI→DOCCI)58.7384.6690.7558.0883.5390.47 GOAL (DCI→DCI)72.6489.8993.7072.84 90.5093.20 GOAL (DCI→DOCCI)71.2292.3996.47 72.1892.8896.49 StructXLIP (DCI→DCI)75.90 90.00 95.1574.3989.99 94.30 StructXLIP (DCI→DOCCI)75.4793.4197.2272.9893.2496.75 Train on DOCCI→ Test on DOCCI vs DCI Long-CLIP (DOCCI→DOCCI)64.4987.6793.4363.0887.4593.14 Long-CLIP (DOCCI→DCI)51.2373.3980.0950.7372.8980.29 GOAL (DOCCI→DOCCI)79.4796.6598.6979.4396.1497.25 GOAL (DOCCI→DCI)64.1382.6987.2965.8883.4487.89 StructXLIP (DOCCI→DOCCI) 83.04 97.06 98.9681.59 96.94 98.78 StructXLIP (DOCCI→DCI)65.1883.8988.0466.1383.8489.09 competitors. 4.2. Ablation studies Structure-centric alignment losses. We ablate the im- pact of our introduced structure-centric alignment losses L ∗ , which consists ofL I ′ ,T ′ ,L I,I ′ , andL local I ′ ,T ′ . As shown in Tab. 4, all three losses contribute positively to the retrieval improvement. The global structure alignment lossL I ′ ,T ′ alone already yields a clear gain. The consistency regular- izationL I,I ′ further stabilizes learning while maintaining coherence with the original semantic space. Finally, adding the local structure-centric alignment lossL local I ′ ,T ′ achieves the best results. Global alignment enforces holistic shape consistency, while local alignment strengthens fine-grained correspondences, leading to best multimodal alignment. Types of extraction methods. We evaluate several visual and textual extraction strategies, all of which yield con- sistent gains, showing that the model adapts robustly to structure-centric extraction. On the visual extraction side, we consider two categories of structure extraction methods: (i) classic filter-based (indicated by [F]) edge extraction This is a walkway made of stones and bricks. Dozens of people are walking on the walkway. On the left side of the walkway are five buildings that are a similar light brown color. At the end of the last building is a small green tree. You can see the light blue colored sky past the buildings. Str uctX LIP GOAL 1234567 1234567 ...[A below-the-knee yellow dress with a straight across neckline, applique details, a dot pattern, pencil silhouette]... GOAL StructXLIP ...[The branch has a broken end seen up close on the bottom right, and the other end is on the left]... (a)(b) Figure 6. (a) Retrieval results; (b) GRAD-CAM [27] highlighting cross-modal attention between the object-centric text and image. Table 4. Ablation on losses. Results onText→Imageand Image→Textretrieval for SKETCHY and DOCCI. Each row adds one additional loss term to the previous configuration. L I ′ ,T ′ L I,I ′ L local I ′ ,T ′ SKETCHYDOCCI R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10 66.7589.5593.6164.5188.7693.4480.3896.4598.7679.4396.5198.69 67.8890.8595.3467.1889.2194.9181.2096.8398.9580.6997.0498.73 68.5790.5095.6068.2289.6495.1681.9297.0198.9881.0597.1098.75 69.8690.8595.4268.2290.6795.6883.0497.0698.9681.5996.9498.78 methods (e.g., Canny [2] and LoG [20]), and (i) learning- based (indicated by [L]) methods (e.g., HED [43], LineArt- Detector (LAD) [52], and Photo2Sketch (P2S) [19]). The filter-based approaches achieve the best overall performance, while learning-based extractors yield comparably strong re- sults with only marginal differences. For the textual extrac- tion, the Original Text approach retains appearance-related vocabulary, while the LLM-Extraction prompts a LLM to rewrite the description only with structure-centric seman- tics, but may introduce hallucinated attributes. Our Lexicon Filter balances factual grounding and structural relevance, achieving the best results. To further assess the model’s robustness, we introduce a Noise Injection setting. On the visual side, Incorrect Match randomly pairs an image with edge maps from a different dataset, i.e. matching DOCCI edges with SKETCHY images, vice versa. On the textual side, Random Word Mask (RWM) randomly removes all non-structural and non-functional words outside the vocabularyV a , and Word Reduction re- tains only about 10% of the original words to simulate severe semantic loss. As shown in Tab. 5, both the visual and textual noise experiments cause only minor degradation, exhibiting strong robustnesss as the model remains stable and reliable even under structural noises and misalignment inL ∗ . The 90% reduction variant underperforms on SKETCHY, as the short captions make it difficult for the remaining 10% of words to cover most objects. Sample efficiency. We present in Fig. 5 showing how StructXLIP performs with varying ratios of the fine-tuning dataset, in comparison with other sota methods. Our ap- proach outperforms the competitors at every regime. No- tably, the margin seems to be maximized around 20-50% of the fine-tuning data, especially with the DOCCI dataset. Table 5. Ablation on extraction methods.Text→Imageand Image→Textretrieval on SKETCHY and DOCCI datasets. [F] indicates filter-based and [L] indicates learning-based edge extrac- tion methods. Bold numbers indicate the best results within each subtable. * denotes the option in our StructXLIP. (a) Visual Extraction SKETCHYDOCCI R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10 Methods Canny* [2] [F] 69.8690.8595.4268.2290.6795.6883.0497.0698.9681.5996.9498.78 LoG [20] [F]68.4889.6494.9167.3690.2494.4783.5997.3699.2081.9897.4199.04 HED [43] [L]68.9190.6795.4366.9390.4194.4783.1497.1898.9881.7397.0898.69 LAD [52] [L]69.1790.9395.4266.9390.1694.9182.6997.1098.9081.0896.8498.84 P2S [19] [L]68.1490.7694.9967.5390.1694.0481.4997.0298.7380.2096.4398.47 Noise Injection Incorrect match 66.2088.9593.3964.3788.2693.7880.2596.3398.7679.1596.1098.41 (b) Textual Extraction SKETCHYDOCCI R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10 Methods Original Text66.9389.3893.3564.5188.0093.0181.0296.8898.6780.2996.6198.73 LLM-Extraction 67.3690.2494.6567.7089.1294.3081.0897.0298.5980.2296.6198.65 Lexicon Filter* 69.8690.8595.4268.2290.6795.6883.0497.0698.9681.5996.9498.78 Noise Injection RWM68.9189.9893.9667.1988.9594.0481.4996.5398.7480.4996.5598.62 Word Reduction 62.0086.9692.6660.6286.0192.5781.0096.4898.7679.6196.5398.62 4.3. Qualitative analysis Fig. 6 (a) shows a qualitative retrieval comparison on the DOCCI dataset between StructXLIP and the second-best method GOAL [3]. Apart from the better retrieval of StructXLIP, there is a substantial difference on the gist of the retrieved images: StructXLIP extracts images that match more structure details (lamps and people are more present). One may think we are forgetting colors due to our emphasis on structure, but this is not the case: the brown, green, and light azure patterns are also evident. In the attention map of Fig. 6 (b), we visualize the cross-modal attention between two textual descriptions of an object and the image after fine- tuning. On the SKETCHY dataset (first row), it is evident how we match the dress with a very sharp attention map, in contrast to the one of GOAL. The DOCCI example (sec- ond row) showcases the StructXLIP’s ability in capturing the correct layout of the image, with the GOAL completely ignoring the tree branch on the floor. 5. Conclusion We introduced StructXLIP, a structure-centric fine-tuning framework that leverages edge-based visual cues and their textual counterparts to strengthen vision-language align- ment for long, detail-rich captions. Our mutual informa- tion analysis provides theoretical insight into its stabilizing effect on optimization. Extensive experiments across gen- eral and specialized domains demonstrate consistent gains by StructXLIP, its robustness to variations in structural ex- traction, and plug-and-play compatibility with diverse fine- tuning frameworks. StructXLIP highlights the value of in- troducing structural cues at VLM fine-tuning, which gives rise to an inviting future direction: What would emerge if we extend our framework to train a VLM from scratch?. How- ever, we acknowledge that exploring this direction is very computationally demanding. References [1]Mothilal Asokan, Kebin Wu, and Fatima Albreiki. Finelip: Extending clip’s reach via fine-grained alignment with longer text inputs. In Proceedings of the Computer Vision and Pat- tern Recognition Conference, pages 14495–14504, 2025. 3, 6 [2]John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelli- gence, (6):679–698, 2009. 3, 8, 2, 4, 5 [3] Hyungyu Choi, Young Kyun Jang, and Chanho Eom. Goal: Global-local object alignment learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4070–4079, 2025. 2, 4, 6, 7, 8, 1, 9 [4]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions (ICLR), 2021. 6 [5]Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip- adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581–595, 2024. 1, 2 [6]Junqi Ge, Ziyi Chen, Jintao Lin, Jinguo Zhu, Xihui Liu, Jifeng Dai, and Xizhou Zhu. V2pe: Improving multimodal long- context capability of vision-language models with variable visual position encoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21070– 21084, 2025. 3 [7]Federico Girella, Davide Talon, Ziyue Liu, Zanxi Ruan, Yim- ing Wang, and Marco Cristani. Lots of fashion! multi- conditioning for image generation via sketch-text pairing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19711–19720, 2025. 6, 1 [8]Yuncheng Guo and Xiaodong Gu. Mmrl: Multi-modal repre- sentation learning for vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 25015–25025, 2025. 1, 2 [9]Yaoqin He, Junchen Fu, Kaiwen Zheng, Songpei Xu, Fuhai Chen, Jie Li, Joemon M Jose, and Xuri Ge. Double-filter: Efficient fine-tuning of pre-trained vision-language models via patch layer filtering. In Forty-second International Con- ference on Machine Learning, 2025. 2 [10] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 1, 2, 6 [11] Young Kyun Jang, Junmo Kang, Yong Jae Lee, and Donghyun Kim. MATE: meet at the embedding - connecting images with long texts. In Findings of the Association for Compu- tational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 1659–1672. Association for Computational Linguistics, 2024. 3 [12]Haoyu Jiang, Zhi-Qi Cheng, Gabriel Moreira, Jiawen Zhu, Jingdong Sun, Bukun Ren, Jun-Yan He, Qi Dai, and Xian- Sheng Hua. Ucdr-adapter: Exploring adaptation of pre- trained vision-language models for universal cross-domain retrieval. In 2025 IEEE/CVF Winter Conference on Applica- tions of Computer Vision (WACV), pages 5429–5438. IEEE, 2025. 1, 2 [13]Muhammad Uzair khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi- modal prompt learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1, 2 [14]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 4 [15] Alexander Kraskov, Harald St ̈ ogbauer, and Peter Grass- berger. Estimating mutual information. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 69(6): 066138, 2004. 5 [16]Luping Liu, Chao Du, Tianyu Pang, Zehan Wang, Chongx- uan Li, and Dong Xu. Improving long-text alignment for text-to-image diffusion models. In The Thirteenth Interna- tional Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. 3 [17]Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: weight-decomposed low-rank adaptation. In Proceedings of the 41st International Confer- ence on Machine Learning. JMLR.org, 2024. 1, 2, 6 [18]Yang Liu, Mengyuan Liu, Shudong Huang, and Jiancheng Lv. Asymmetric visual semantic embedding framework for efficient vision-language alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5676–5684, 2025. 2 [19]Zhi-Song Liu, Wan-Chi Siu, and H Anthony Chan. Learn to sketch: A fast approach for universal photo sketch. In 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1450– 1457. IEEE, 2021. 3, 8, 2, 4, 5 [20]David Marr and Ellen Hildreth. Theory of edge detection. Proceedings of the Royal Society of London. Series B. Biolog- ical Sciences, 207(1167):187–217, 1980. 2, 3, 8, 4, 5 [21]Ivona Najdenkoska, Mohammad Mahdi Derakhshani, Yuki M. Asano, Nanne van Noord, Marcel Worring, and Cees G. M. Snoek. TULIP: token-length upgraded CLIP. In The Thir- teenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. 3 [22]Mehrdad Noori, David Osowiechi, Gustavo Adolfo Var- gas Hakim, Ali Bahri, Moslem Yazdanpanah, Sahar Das- tani, Farzad Beizaee, Ismail Ben Ayed, and Christian Desrosiers. Test-time adaptation of vision-language models for open-vocabulary semantic segmentation. arXiv preprint arXiv:2505.21844, 2025. 1, 2 [23]Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, and Jason Baldridge. Docci: Descriptions of connected and contrasting images. In European Conference on Computer Vision, 2024. 5, 1 [24]Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 2, 3 [25]Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual informa- tion. In International conference on machine learning, pages 5171–5180. PMLR, 2019. 2, 5, 7 [26]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PmLR, 2021. 1, 2 [27] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad- cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017. 8 [28]Ozan Sener and Vladlen Koltun. Multi-task learning as multi- objective optimization. In Proceedings of the 32nd Interna- tional Conference on Neural Information Processing Systems, page 525–536, Red Hook, NY, USA, 2018. Curran Associates Inc. 2, 5, 9 [29]Peter Sollich and Anason Halees. Learning curves for gaus- sian process regression: Approximations and bounds. Neural computation, 14(6):1393–1428, 2002. 5 [30]Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 2 [31]Rui Sun, Tao Lei, Qi Chen, Zexuan Wang, Xiaogang Du, Weiqiang Zhao, and Asoke K. Nandi. Survey of image edge detection. Frontiers in Signal Processing, Volume 2 - 2022, 2022. 2 [32] Davide Talon, Federico Girella, Ziyue Liu, Marco Cristani, and Yiming Wang. Seeing the abstract: Translating the ab- stract language for vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025. 3 [33]Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368–377, 2000. 5, 7, 9 [34] Thanh-Dat Truong, Hoang-Quan Nguyen, Xuan-Bac Nguyen, Ashley Dowling, Xin Li, and Khoa Luu. Insect-foundation: A foundation model and large multimodal dataset for vision- language insect understanding. International Journal of Com- puter Vision, pages 1–26, 2025. 6, 1 [35]Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. 2 [36]Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26700–26709, 2024. 3, 5, 1 [37] Tom Viering and Marco Loog. The shape of learning curves: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7799–7819, 2022. 5 [38]Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, and Zhuotao Tian. Declip: Decoupled learning for open- vocabulary dense perception. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14824– 14834, 2025. 2 [39]Zhangyun Wang, Ni Ding, and Aniket Mahanti. Learning robust vision-language models from natural latent spaces. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. 2 [40] Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual information in contrastive learning for visual representations, 2020. 5, 9 [41] Wei Wu, Kecheng Zheng, Shuailei Ma, Fan Lu, Yuxin Guo, Yifei Zhang, Wei Chen, Qingpei Guo, Yujun Shen, and Zheng- Jun Zha. Lotlip: Improving language-image pre-training for long text understanding. Advances in Neural Information Processing Systems, 37:64996–65019, 2024. 3 [42] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 4818–4829, 2024. 2 [43]Saining Xie and Zhuowen Tu. Holistically-nested edge detec- tion. In Proceedings of the IEEE international conference on computer vision, pages 1395–1403, 2015. 3, 8, 2, 4, 5 [44]Shaoan Xie, Lingjing Lingjing, Yujia Zheng, Yu Yao, Zeyu Tang, Eric P Xing, Guangyi Chen, and Kun Zhang. Smartclip: Modular vision-language alignment with identification guar- antees. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29780–29790, 2025. 2, 6 [45] Zhengqin Xu, Zelin Peng, Xiaokang Yang, and Wei Shen. Fate: Feature-adapted parameter tuning for vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9014–9022, 2025. 1, 2 [46] Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adap- tive language-image pre-training with synthetic caption. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2922–2931, 2023. 2 [47]Lingxiao Yang, Ru-Yuan Zhang, Yanchen Wang, and Xiaohua Xie. Mma: Multi-modal adapter for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23826–23837, 2024. 1, 2 [48]Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18123–18133, 2022. 1, 2 [49] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 1, 2 [50] Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European conference on computer vision, pages 310–325. Springer, 2024. 3, 6 [51] Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European conference on computer vision, pages 310–325. Springer, 2024. 1, 2, 9 [52] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023. 3, 8, 2, 4, 5 [53]Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kun- chang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip- adapter: Training-free adaption of clip for few-shot classifi- cation. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceed- ings, Part XXXV, page 493–510. Springer-Verlag, 2022. 1, 2 [54]Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything, 2023. 12 [55]Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language- image pre-training with long captions. In European Con- ference on Computer Vision, pages 73–90. Springer, 2024. 3 [56]Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816–16825, 2022. 1, 2 [57] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Zi- wei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), 2022. 1, 2 [58]Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, and Weinan E. Towards theoretically un- derstanding why sgd generalizes better than adam in deep learning. In Advances in Neural Information Processing Sys- tems, pages 21285–21296. Curran Associates, Inc., 2020. 2, 5, 9 [59]Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 15659–15669, 2023. 1, 2 StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues Supplementary Material In this supplementary material, we provide a comprehen- sive analysis and additional details to support the main paper. The content is organized as follows: •Dataset details (Sec. A): We provide visual samples and statistical breakdowns for all datasets (Sec. A). Specifi- cally, we detail the construction and curation process of the specific-domain INSECT dataset (Sec. A.1). •Lexicon filtering (Sec. B): We present the details about lexicon filtering in terms of the used prompts, the appearance-related vocabulary and the filtering process (Sec. B.1). We also provide a statistical analysis on the filtered texts (Sec. B.3). • Extended information-theoretic analysis (Sec. C): We provide detailed theoretical proof (Sec. C.1) and its full supporting empirical analyses (Sec. C.2) across all four datasets. •Additional implementation details (Sec. D): We extend the method description with details on text token extension (Sec. D.1) and the fine-tuning setup (Sec. D.2). Further- more, we provide computational analysis (Sec. D.3). •Additional experimental analyses (Sec. E): We present additional ablation analysis on the lexicon filtering (Sec. E.1), additional cross-domain evaluation (Sec. E.2), the extended experimental results at deeper ranks (Sec. E.3) and more qualitative results (Sec. E.4) to com- plement the experimental evaluation in the main paper. A. Dataset Details Figs. 1-4 shows sample images with their extracted edge maps, associated original textual descriptions and the filtered textual descriptions from each dataset. Our experiments are conducted on four datasets span- ning different domains and levels of semantic granularity: DCI [36], DOCCI [23], SKETCHY [7], and INSECT [34]. For the general-domain setting, we follow [3], utilize two human-annotated datasets, DCI and DOCCI, which are orig- inally designed for dense image captioning. DCI contains 7,805 natural images, each paired with highly detailed and information-dense descriptions. DOCCI consists of approx- imately 15k natural scene images collected across diverse geographic regions. Each image is annotated with long, highly compositional and discriminative descriptions, with an average length of 136 words. In the domain-specific setting, we use SKETCHY and INSECT, two datasets characterized by fine-grained visual concepts and prominent structural properties. SKETCHY is a large-scale fashion dataset containing roughly 46k out- fit images, each paired with detailed descriptions covering garment components, fabric, pattern shapes, and spatial re- lations between parts. Although the average text length is only about 56 words, the descriptions are semantically dense and rich in visual detail. INSECT contains 6k insect images with highly fine-grained and biologically rare categories that are underrepresented in general-purpose pretrained VLMs. Each image is paired with expert-verified biological descrip- tions covering coloration, wing structures, body-segment proportions, and species-level morphological traits, with an average length of 81 words. Since the original Insect-1M dataset contains substantial redundancy in both images and text, we construct a refined version suitable for fine-tuning on long-text understanding (see in the following). Regarding dataset splits, we follow the standard protocol for DOCCI and DCI as in [3], with 5,100 and 2,000 images in their test sets, respectively. For SKETCHY, we adopt the official test split of 1.2k images. For INSECT, since the original dataset provides no official split, we construct a split by dividing the curated dataset with an 8:2 train-test ratio. A.1. Condensed INSECT Dataset Construction The INSECT dataset is derived from the large-scale insect image repository Insect-1M, which contains over one mil- lion images covering approximately 34,000 species, along with hierarchical textual annotations ranging from Phylum down to Species. Despite its scale and richness, the orig- inal dataset is not readily suitable for image–text retrieval or cross-modal fine-tuning. First, the dataset does not as- sign unique descriptions to individual images; instead, it uses an indexed-description mechanism in which large num- bers of visually different images share the same short list of text tokens (e.g., descriptionA = [1,2,3]), resulting in exten- sive duplication of textual annotations. Then, the high-level textual descriptions (e.g., Order, Class) are shared across thousands of samples, leading to large textual overhead that are not very meaningful for fine-grained discrimination com- pared to those description on finer granularity concerning, e.g. Genus and Species. We therefore construct a more suitable version of IN- SECT for long-text fine-tuning. First, to reduce duplication and increase representational diversity, we design a greedy selection algorithm based on description-ID overlap and se- lect a core sample set using an overlap threshold of 6, effec- tively removing redundant images and repeated descriptions. Second, to create a more challenging cross-family generaliza- tion scenario, we apply a strict Family-level out-of-domain split, ensuring that the families in the test set are completely disjoint from those in the training set. Then, to ensure that the textual side focuses on biologically discriminative fea- OriginalCanny[2] LoG[20] HED[43] LAD[52]P2S[19] Figure 1. Illustration of SKETCHY dataset samples across different edge-map representations. For each example, we show the original RGB image together with its caption, followed by Canny, LoG, HED, LAD, and P2S edge maps, as well as the filtered captionT ′ that preserves only structure-centric information. tures, we retain only the Genus- and Species-level descrip- tions and discard higher-level labels, yielding more compact and semantically fine-grained image–text pairs. Through these steps, we obtain a curated 6,000-pair high-quality IN- SECT dataset. Both the dataset and the code used for its construction will be released publicly. B. Lexicon Filtering This section provides additional details on the lexicon filter- ing process described in the main paper (Sec. 3.1), including: i) how we use an LLM to construct the appearance vocab- ulary; i) the implementation of our filtering function; and i) statistical analyses of the filtering effect across the four OriginalCanny[2] LoG[20] HED[43] LAD[52]P2S[19] Figure 2. Illustration of INSECT dataset samples across different edge-map representations. OriginalCanny[2] LoG[20] HED[43] LAD[52]P2S[19] Figure 3. Illustration of DOCCI dataset samples across different edge-map representations. datasets. OriginalCanny[2] LoG[20] HED[43] LAD[52]P2S[19] Figure 4. Illustration of DCI dataset samples across different edge-map representations. B.1. Appearance Vocabulary Generation To systematically remove appearance-related terms such as color and material from the original text descriptions, we first construct a general-domain appearance lexicon, denoted asV a , which is later used for our lexicon filtering procedure. In the main paper, this lexicon is defined asV a =LLM(P). Here we provide additional details on how it is obtained. As shown in Fig. 5, we use a large language model (in practice, ChatGPT-5.1) and provide it with a structured promptP. The prompt instructs the model to consider a structure-centric representation and to produce a list of appearance-related terms that cannot be inferred from such structural information. The LLM returns a list of 600–800 appearance-only terms, including basic colors, color vari- ants, material categories. The set of returned terms is used to form the final appearance lexiconV a . The complete vocabu- lary is listed in Fig. 5. In all experiments, we use the same general-domain appearance lexiconV a as the filtering tem- plate for both general-domain and domain-specific datasets. The lexicon is built only once during pre-processing and is subsequently kept unchanged, enabling full reproducibility of the pipeline. (1): General Appearance Lexicon Prompt–Response USER: You are a vision-language expert. Consider an image representation that contains only edge information (i.e., contours, boundaries, structure, and geometric layout), with all color, shading, material, and surface cues removed. Please produce a comprehensive list of English words and short phrases whose meanings cannot be inferred from such an edge-only representation. Return the list as comma- separated items. LLM: [red, blue, green, yellow, black, blackish, white, gray, grey, orange, purple, pink, brown, beige, cyan, magenta, turquoise, teal, maroon, navy, violet, indigo, gold, silver, ivory, cream, olive, tan, peach, mint, burgundy, crimson, scarlet, lavender, lilac, azure, teal, aqua, aquamarine, navy blue, sky blue, baby blue, light blue, dark, dark blue, light green, dark green, forest green, lime green, light red, dark red, rose red, wine red, light pink, hot pink, dark gray, light gray, dark grey, light grey, colored, cotton, wool, silk, linen, denim, leather, suede, velvet, satin, chiffon, polyester, nylon, spandex, acrylic, rayon, cashmere, fleece, corduroy, lace, mesh, canvas, tweed, felt, rubber, plastic, metal, steel, iron, aluminum, bronze, brass, ceramic, glass, wood, bamboo, stone, marble, granite, concrete, clay, paper, fur, shearling, down, feather, denier, foam] Figure 5. Prompt and response used in our LLM-based construction of the general appearance vocabulary for lexicon filtering. This lexicon is generated only once during preprocessing; the pipeline consistently uses this fixed lexicon thereafter, ensuring full reproducibility. B.2. Lexicon Filtering Function In the main paper, we denote the filtering function asF(·), whose purpose is to remove all appearance-related terms contained inV a from the original textual description. Here we provide a detailed explanation of how this function is implemented in practice. We first convert all entries inV a into regular-expression matchers. During filtering, these ex- pressions are applied to the input text to identify and remove any appearance terms found in theV a . The matching process is case-insensitive and respects word boundaries to avoid un- intended partial matches. After removing appearance terms, we apply a lightweight grammatical cleanup. Since such removal can produce unnatural or fragmented text, for ex- ample, “a blue and white pattern” may temporarily become “a and pattern”. We first eliminate redundant spaces and punctuation to prevent repeated whitespace or stray commas. We then remove isolated conjunctions such as “and” or “or,” which lose their function once their associated tokens are deleted. Finally, we check whether the filtered sentence still contains sufficient semantic content. If too little meaningful text remains, we revert to the original description to avoid excessive information loss. B.3. Statistical Analysis on Lexicon Filtering We provide a statistical analysis of the effect of lexicon fil- tering across the four datasets. For consistency and ease of comparison, we coarsely categorizeV a into two classes: color words and material words, and analyze filtering cover- age (“What proportion of captions were modified?”), target specificity (“What types of words were removed?”), and modification intensity (“How many words were removed on average?”) based on this grouping. As shown in Fig. 6, A reports the proportion of captions that were modified at least once. SKETCHY and DOCCI exhibit nearly 100% intervention coverage, and DCI reaches 92.8%, indicating that captions in these datasets commonly contain identifiable color or material descriptors. In contrast, INSECT shows only 58.9% modified captions, reflecting that its descriptions inherently emphasize morphology and structure rather than appearance attribute, consistent with the style of biological taxonomic text. B summarizes the composition of removed words (computed over the modi- fied captions). In SKETCHY and INSECT, approximately 97% of removed terms are color-related, with material terms contributing only a negligible fraction. In DOCCI and DCI, material words account for 20% and 17%, respectively, align- ing with the fact that captions in these datasets often mention building materials or object surface composition. C shows the average number of removed words per caption and the proportion relative to the original caption length. For the shorter captions of SKETCHY and INSECT, only 2.6-2.9 words are removed on average (corresponding to 3.1% and 10.3% of total caption length). For the longer captions in DOCCI and DCI, 7.1-7.2 words are removed on average, but this corresponds to only 5-6% of the overall text. Our lexicon filtering removes appearance attributes effectively, without disrupting the structural or semantic core of the orig- inal caption. Overall, these three analyses demonstrate that lexicon filtering achieves high coverage, strong target speci- ficity, and mild editing intensity, making it both effective and stable across domains. 020406080100 Percentage of Dataset (%) DCI DOCCI Sketchy Insect 92.8% 99.0% 99.6% 58.9% What proportion of captions were modified? A. Intervention Scope 020406080100 Composition (%) 83%17% 80%20% 97%3% 97%3% What types of words were removed? B. Target Specificity 0246810 Words Removed per Caption 7.2 (5.4%) 7.1 (5.8%) 2.9 (10.3%) 2.6 (3.1%) How many words were removed on average? C. Modification Intensity Modified CaptionsColor WordsMaterial WordsAvg. Removed Words Figure 6. Statistical analysis of the lexicon filtering effect across the four datasets. For analysis, removed tokens are coarsely grouped into color words and material words. (A) Intervention scope: percentage of captions in which at least one word was removed. (B) Target specificity: composition of removed tokens (computed over the modified captions in Panel A), showing the proportion of color vs. material words. (C) Modification intensity: average number of words removed per caption, with the percentage relative to the original caption length shown in parentheses. C. Extended Information-Theoretic Analysis C.1. Information-Theoretic Analysis In this section we expand the theoretical view of Sec. 3.3 of the main paper. Specifically, we analyze the effect of the structure-centric objectivesL I ′ ,T ′ andL I,I ′ under the information-theoretic and optimization lens giving three lem- mas and a theorem. Lemma 1 (Information ordering for structure-centric views). LetIandTdenote the random variables associated with images and texts, and letI ′ =E(I) andT ′ =F(T) be their structure-centric counterparts, whereE(·) andF(·) are deter- ministic maps that remove appearance-related information and are not invertible. Then the following inequality holds: I MI (I ′ , T ′ ) < I MI (I, T), where I MI (·,·) is the mutual information operator. Proof sketch. The pair (I ′ ,T ′ ) is obtained from (I,T) through the deterministic channel (E,F). By the Data Processing Inequality [33], any such transformation cannot increase mutual information, henceI MI (I ′ ,T ′ )≤ I MI (I,T). The in- equality is strict whenever eitherEorFis not injective, which holds in our setting since edge extraction and lexicon filtering discard appearance cues, rather than introducing new information. Lemma 2 (InfoNCE lower bounds for the two objectives). Leti=f img (I) andt=f txt (T) be the image and text embeddings, and leti ′ =f img (I ′ ) andt ′ =f txt (T ′ ) be the structure-centric embeddings. Assume bothL I,T andL I ′ ,T ′ are symmetric InfoNCE losses with batch size N. Then I MI (i, t)≥ log N−E[L I,T ],I MI (i ′ , t ′ )≥ log N−E[L I ′ ,T ′ ]. Proof. This is the mere application of [25], stating that the InfoNCE loss is a standard variational lower bound on mutual information. Applying the result to the pairs (i,t) and (i ′ ,t ′ ) yields the two inequalities. The assumptions on symmetry and batch size match the formulation in Sec. 3.2 of the main paper. In the following, to the sake of clarity, we assume I MI (i, t)≈ I MI (I, T) and I MI (i ′ , t ′ )≈ I MI (I ′ , T ′ ). Lemma 3 (Directional compatibility of gradients). Letθ denote the parameters shared by the vision and text encoders. Consider the gradients∇ θ L I,T and∇ θ L I ′ ,T ′ . If the positive pairs in the two losses correspond to the same image–text instances and the encoders are shared, then the expected cosine similarity between the two gradients satisfies E cos(∇ θ L I,T ,∇ θ L I ′ ,T ′ ) > 0. Proof sketch. Both losses use the same positive pairs and differ only in the views fed to the encoders (full images and captions vs structure-centric counterparts). The correspond- ing positive logits are maximized in both objectives, while negatives are pushed apart. This induces aligned update di- rections for parameters that affect shared features. Under mild regularity assumptions on the encoders, the expected co- sine similarity between the two gradients is strictly positive. In practice, this is confirmed by the empirical measurements reported in Fig. 4(c). Theorem 1 (Effect of the structure-centric auxiliary losses). Consider the joint objective L total =L I,T + λ 1 L I ′ ,T ′ + λ 2 L I,I ′ , λ 1 ,λ 2 > 0. Under the assumptions of the lemmas above, the following properties hold: 1.The auxiliary alignment task between the pair (i ′ ,t ′ ) is information-reduced compared to (i,t), henceL I ′ ,T ′ op- timizes a harder objective in the sense of Lemma 1 and 2. 2.The gradients ofL I,T andL I ′ ,T ′ are directionally compat- ible, so the auxiliary loss does not conflict with the main alignment. 3.Due to the lower mutual information of (i ′ ,t ′ ), the gradi- ent norm of∇ θ L I ′ ,T ′ remains non-negligible even when ∇ θ L I,T starts to vanish, which provides persistent opti- mization signal. 4.The consistency termL I,I ′ bounds the drift betweeniand i ′ , so the structure-centric space stays anchored to the semantic manifold of i and fine-tuning remains stable. Proof sketch. Item 1 follows directly from Lemma 1 and the InfoNCE bounds in Lemma 2, which place the pair (i ′ ,t ′ ) at a lower mutual information level than (i,t). More in the detail, the auxiliary alignment task on (i ′ ,t ′ ) is strictly information- reduced compared to (i,t), and thereforeL I ′ ,T ′ optimizes a harder objective. More precisely, sinceI ′ =E(I) andT ′ = F(T) are obtained by applying non-invertible, deterministic maps that remove appearance-related variability, the entropy H(·) of both variables is reduced: H(I ′ ) < H(I),H(T ′ ) < H(T). By the Data Processing Inequality, this implies I MI (I ′ , T ′ )≤ I MI (I, T). In addition, the structure-centric views induce a contraction of the positive pair distribution: the space of valid matches becomes smaller, the intra-class variability is suppressed, and the negative samples become less separable in the em- bedding space. These effects lower the effective signal- to-noise ratio of the InfoNCE objective, which increases the difficulty of the optimization landscape associated with L I ′ ,T ′ . Consequently, the gradients generated byL I ′ ,T ′ tend to persist longer during fine-tuning, since reaching its mini- mum requires modeling more subtle, geometry-driven corre- spondences that remain unresolved after the full-information objectiveL I,T has already saturated. Item 2 follows from Lemma 3 and concerns the direc- tional compatibility of the gradients. Since bothL I,T and L I ′ ,T ′ operate on the same positive image–text instances and update the same encoder parameters, their contrastive objec- tives induce aligned update rules: both maximize the positive logits and suppress the negative logits associated with the same underlying pairs, even though the views differ. Let the gradients be: g =∇ θ L I,T ,g ′ =∇ θ L I ′ ,T ′ . Lemma 3 ensures that their expected cosine similarity satis- fies E cos(g, g ′ ) > 0. Therefore, the auxiliary gradient does not conflict with the semantic direction promoted by the main loss. Instead, it expands the set of admissible descent directions within a compatibility cone, enriching the optimization trajectory without introducing destructive interference. Item 3 relies on the fact that, owing to the information- reduced nature of (I ′ ,T ′ ), the lossL I ′ ,T ′ converges more slowly thanL I,T and produces non-vanishing gradients even when the main loss has already flattened. Indeed,L I,T op- timizes an InfoNCE objective associated with the higher mutual information quantityI MI (I,T), whose landscape typi- cally admits a faster descent: the positive pair (i,t) carries rich appearance and structural cues, which help discriminate positives from negatives early in fine-tuning. In contrast, L I ′ ,T ′ maximizes the lowerI MI (I ′ ,T ′ ), where bothi ′ and t ′ have reduced entropy because appearance information has been removed byE(·) andF(·). This contraction of the feature space has two effects: • the separation between positives and negatives becomes smaller, making the contrastive objective harder to opti- mize; •the gradients associated with hard positives and hard nega- tives decay more slowly, since the model must rely purely on geometric and structural cues to improve the logits. Formally, since the InfoNCE gradients satisfy ∇ θ L I,T = E g(i, t) , ∇ θ L I ′ ,T ′ = E g(i ′ , t ′ ) , and the score functiong(·) for (i ′ ,t ′ ) has larger relative vari- ance due to the reduced mutual information, the expected magnitude of∇ θ L I ′ ,T ′ remains positive for a longer portion of fine-tuning. Empirically, this manifests in a later conver- gence time forL I ′ ,T ′ compared toL I,T , and in a sustained gradient norm that continues to provide meaningful updates even after the main contrastive gradients approach zero. This behavior is consistent with the dynamics shown in Fig. 4(a,b) of the main paper, whereL I ′ ,T ′ displays a slower flattening and a more persistent gradient profile. Finally, Item 4 exploits the form ofL I,I ′ , which penal- izes large angular deviations betweeniandi ′ and therefore constrains the structure-centric representations to remain close to their full-information counterparts. Specifically, sinceL I,T andL I ′ ,T ′ correspond to two correlated but not identical contrastive objectives, their gradients span a larger subspace than either loss alone. Formally, let g =∇ θ L I,T ,g ′ =∇ θ L I ′ ,T ′ . From Item 2 we know thatcos(g,g ′ )>0 in expectation, which guarantees compatibility. However, because (i ′ ,t ′ ) belongs to an information-reduced space and responds dif- ferently to hard negatives and subtle geometric patterns, one has g ′ /∈ spang, so the matrix [g,g ′ ] has rank 2 almost everywhere. This implies that the combined update g tot = g + λ 1 g ′ + λ 2 ∇ θ L I,I ′ , explores descent directions that the main loss alone cannot access. This additional set of directions reduces the risk of premature convergence to shallow minima (a common issue in contrastive objectives), improves escape from flat regions of the loss surface, and increases the robustness of SGD trajectories. In the language of multi-objective optimiza- tion,L I ′ ,T ′ introduces a complementary gradient component that expands the effective feasible region of updates while maintaining alignment with the main semantic objective. The consistency lossL I,I ′ ensures that this expanded search space remains stable by preventingi ′ from drifting too far fromi. As a result, the combined gradient retains diver- sity without diverging from the semantic manifold defined by the original embeddings, yielding a balanced and stable optimization process. Taken together, these properties explain why the auxiliary structure-centric alignment acts as a beneficial regularizer: it introduces correlated but information-reduced gradients that improve convergence stability and lead to more robust alignment. C.2. Empirical Analyses In Sec. 3.3 of the main paper, we reported one representa- tive study of our information-theoretic analysis using the SKETCHY dataset. Here we provide the remaining exper- iments conducted on the other three datasets used in our evaluation: INSECT, DOCCI and DCI. To verify that the information-theoretic analysis holds consistently across dif- ferent data scales, we conduct the same experiments on reduced-scale versions of each dataset containing 5%, 20%, and 50% of the original data. We further include experiments performed under the finetuning-agnostic improvement set- ting combined with the second-best method GOAL [3], to confirm that the same behaviors arise consistently across different fine-tuning configurations. For each experiment, we show the same three quantities presented in Fig. 7 and Fig. 8: (a) the convergence behavior of the two contrastive lossesL I,T andL I ′ ,T ′ , (b) the evolution of their gradient norms and their ratio, and (c) the cosine similarity between the corresponding gradients. Across all datasets and settings, we observe the same consistent patterns as reported in Sec. 3.3. The auxiliary objectiveL I ′ ,T ′ which maximizes the mutual information I MI (I ′ ,T ′ ) under the information-reduction mappingsE(·) andF(·), exhibits a later convergence than the main objec- tiveL I,T . This is consistent with the Data Processing Inequal- ity [33], which impliesI MI (I ′ ,T ′ )≤ I MI (I,T), and makes the auxiliary alignment problem inherently more difficult. The gradient ofL I ′ ,T ′ remains informative even after the gradient ofL I,T flattens, providing persistent optimization signals that steer the model toward semantically coherent and structurally consistent minima. This behavior is further reflected in the positive cosine similarity between∇ θ L I,T and∇ θ L I ′ ,T ′ throughout the optimization trajectory, con- firming that the two tasks pursue compatible optima in the parameter space. The consistent observation across all datasets and set- tings provides a strong empirical support to our theoretical interpretation in Sec. 3.3: information-reduced auxiliary ob- jectives act as implicit regularizers that introduce controlled gradient diversity, as suggested by analyses in multitask learning, multi-objective optimization, and contrastive rep- resentation learning [28,40,58]. Such diversity expands the effective search space and stabilizes convergence, while preserving optimization compatibility through aligned gra- dient directions. The behavior of StructXLIP is agnostic to the dataset, and the findings drawn from the representative Sketchy experiment generalizes to other evaluated datasets. D. Additional implementation details D.1. Text Token Extension To enable base models like CLIP and SigLIP-2 to handle text sequences longer than their original 77 tokens limit, we fully follow the method proposed in Long-CLIP [51]. Specifically, we implement a positional encoding extension that expands the maximum sequence length of all models to 248 tokens. This process is completed before any fine-tuning and fol- lows a preserve-interpolate-extrapolate strategy. First, we keep the first 20 original position vectors unchanged. Then, we expand the middle part by pairwise linear interpolation. Finally, we use linear extrapolation based on the last two original encodings to fill the remaining positions. It is worth noting that Long-CLIP is originally designed and optimized for the CLIP architecture. We applied the same text token extension mechanism to all models in this study, including both the CLIP and SigLIP-2 baselines and their variants using our proposed losses. Nonetheless, we do acknowledge that directly applying this extension to SigLIP- 2 might not be the optimal way, as SigLIP-2 is trained dif- ferently from CLIP. Yet, as there exists no prior work with established techniques for extending SigLIP-2 to long-text scenarios, we opt to this option to ensure experimental con- sistency and fair comparison conditions. Figure 7. The convergence behavior of the two contrastive lossesL I,T andL I ′ ,T ′ (a), the evolution of their gradient norms and their ratio (b), and the cosine similarity between the corresponding gradients (c), evaluated on all two datasets under different percentages of dataset size. Figure 8. The convergence behavior of the two contrastive lossesL I,T andL I ′ ,T ′ (a), the evolution of their gradient norms and their ratio (b), and the cosine similarity between the corresponding gradients (c), evaluated on all two datasets under different percentages of dataset size. Table 1. Comparison of general vs. domain-specific appear- ance lexicons.Text→ImageandImage→Textretrieval on SKETCHY and INSECT. Bold denotes the best performance. SettingR@1R@5R@10R@25R@50R@1R@5R@10R@20R@50 SKETCHY General Lexicon Filter69.86 90.8595.4298.3699.2268.22 90.6795.6897.7599.22 Domain-Specific Lexicon Filter 69.00 90.8795.6098.3399.2268.14 90.5395.2197.4099.10 INSECT General Lexicon Filter9.9326.6038.3456.9969.349.5026.6039.6454.9268.65 Domain-Specific Lexicon Filter9.6728.5039.4655.0967.709.0728.0739.8154.3268.48 D.2. Fine-tuning Details We fine-tune the model for 10 epochs using the AdamW optimizer with a batch size of 16. The initial learning rate is set to 5×10 −6 , and we employ a Cosine Annealing scheduler that decays the learning rate to 0 over the course of fine-tuning, without restarts. The weight decay is set to 0.05. For the contrastive objectives, the global temperature parameterτis learnable (initialized from the pre-trained CLIP and clamped to a maximum logit scale of 3.5), while the local structure alignment temperatureγis fixed at 0.07 to encourage sharp local correspondences. D.3. Computational Analysis To evaluate the additional computational overhead intro- duced by our method, we measure two key preprocessing steps: the local-region segmentation used inL local I ′ ,T ′ and the edge-map extraction. In practice, we use FastSAM [54] for local segmentation. Due to the large variation in image reso- lution across datasets, the average per-image inference time also varies: 3.6 ms/image on the lower-resolution Sketchy dataset, 17.1 ms/image on INSECT, and 17.3 ms / 15.9 ms on the higher-resolution DOCCI and DCI datasets, respectively. For edge extraction, we use the Canny detector, which is lightweight, with timings of 3.77 ms/image on Sketchy, 4.59 ms/image on INSECT, and 19.05 ms/16.18 ms on DOCCI and DCI. Higher image resolutions lead to more processing time. Importantly, both pre-processing steps are executed only once before fine-tuning. During fine-tuning, the overall per-batch runtime of our method is approximately 0.17 s, which is comparable to existing CLIP fine-tuning approaches such as Long-CLIP (0.10 s), GOAL (0.18 s), SmartCLIP (0.07 s), and FineLIP (0.05 s). The inference time is the same as as standard CLIP-based methods. Overall, StructXLIP introduces very lightweight (and affordable) additional computation, and maintains the same inference efficiency as standard CLIP- based methods. E. Additional Experimental Analyses E.1. Ablation on Appearance Lexicons We also conducted the ablation investigating whether con- structing dataset-specific appearance vocabularies would Table 2. Cross-domain generalization from General (DOCCI) to Specific (Sketchy).Models are trained on the general dense-captioned dataset (DOCCI) and tested on abstract sketches (Sketchy). Due to the significant domain shift, all models ex- perience a performance drop, but StructXLIP maintains bet- ter robustness. Values are Recall@K (%). In-domain results (DOCCI→DOCCI) are in italic for reference. Best cross-domain results are in bold. SettingR@1R@5R@10R@1R@5R@10 Fine-tune on DOCCI (General)→ Test on Sketchy (Specific) Long-CLIP (In-domain)64.4987.6793.4363.0887.4593.14 Long-CLIP (Cross-domain)8.1220.0328.24 10.7124.2733.68 GOAL (In-domain)79.4796.6598.6979.4396.1497.25 GOAL (Cross-domain)8.7219.6027.469.7626.8638.17 StructXLIP (In-domain)83.04 97.06 98.9681.59 96.94 98.78 StructXLIP (Cross-domain)8.9621.8530.5712.2632.2142.23 lead to better performance, compared to a general lexicon. To this end, we built two domain-specific appearance lex- icons for the fashion dataset SKETCHY and the biologi- cal dataset INSECT. For each dataset, examples shown in Fig. 10, instructing the LLM to generate appearance terms particularly relevant to that domain. We then substituted the generalV a with these domain-specific vocabularies and re-trained our method under exactly the same fine-tuning configuration. From Tab. 1, we observe that across both the fashion do- main (SKETCHY) and the biological domain (INSECT), the domain-specific lexicons yields on-par performance, com- pared to the general appearance lexicon on most retrieval metrics. This is because the general lexicon already has a broad coverage over appearance-related attributes. Instead, the domain-specific lexicon, despite being more specialized, adds only a small number of additional terms into the vocab- ulary, offering negligible marginal benefit in practice. For this reason, in all our experiments, the appearance vocabu- laryV a used in the lexicon filter is obtained from the general prompt described in Sec. B.1, and the sameV a is applied across all datasets. E.2. Additional Cross-domain Evaluation Complementing to the cross-domain evaluation in the main paper, Table 2 reports the cross-domain evaluation from General (DOCCI) to Specific (SKETCHY). When models are fine-tuned on the general-domain dataset DOCCI and tested on the fashion-focused Sketchy dataset, all methods exhibit a clear performance drop due to the large domain gap. DOCCI contains diverse real-world scenes with dense captions, whereas Sketchy mainly features fashion items. Despite this strong domain shift, StructXLIP consistently achieves the best cross-domain performance across all Re- call@K metrics. E.3. Results at Deeper Ranks We report the full experimental analyses with Recall@K also at deeper ranks (K = 25, 50) on all four datasets. Table 3 shows the cross-modal retrieval at full ranks up to K=50. Our method consistently maintains positive margins over the strongest competitors, even at deeper ranks. Table 4 reports the Recall@K at full ranks for the plug- and-play effectiveness ofL ∗ onto different finetuning meth- ods. At top ranks (R@1/R@5/R@10),L ∗ yields consistent and often substantial improvements, with particularly large gains for lightweight or parameter-constrained methods such as FineLIP, LoRA, and DoRA (improvements ranging from 5% to 18%). On deeper ranks (R@25 and R@50),L ∗ con- tinues to provide stable and uniform benefits across nearly all model–dataset combinations. For DOCCI and DCI, the baselines already achieve extremely high R@50 (>97%), yetL ∗ still contributes an additional 0.1–1.4% improvement. Finally, Tab. 5 reports the data efficiency analysis at full ranks. We observe that StructXLIP consistently shows strong data efficiency across all fine-tuning dataset sizes (5%, 20%, 50%). In the most challenging 5% low-data regime, StructXLIP achieves the best overall results. When the data size increases to 20%, all methods improve, yet StructXLIP remains the best performer with noticeable gains on most metrics. At 50% data, although the baselines begin to saturate, StructXLIP still delivers the best results, provid- ing 1–5% improvements on R@1 and R@5, and enhancing deeper-rank performance (R@25 / R@50) on DOCCI and DCI. Overall, StructXLIP demonstrates strong generaliza- tion and high data efficiency across all four datasets and all data-scale settings. E.4. More Qualitative Results Figure 9 presents additional qualitative results of StructXLIP and the second-best method GOAL on all four datasets. Specifically, we highlight with color some parts of the long texts that are describing some visual objects, and showcase the attention maps between such object-centric texts on the image. It is clear that StructXLIP, compared to the second- best method GOAL, demonstrates a better correspondence between the visual objects and their rich textual descriptions. For instance, in the DOCCI example, given the textual de- scription regarding “the daisy”, StructXLIP produces more visible attention map on the daisy petals compared to GOAL, and it also yields more localized attention map regarding the “tree line”. Similar patterns can be observed in the DCI samples too, where StructXLIP is more capable in captur- ing accurate and localized attention map close to the visual object being described. On the specific-domain Sketchy dataset, while StructXLIP exhibits a better coverage on the described visual part, overall both StructXLIP and GOAL are able to capture the correct object. We hypothesize that this might be due to the fact Sketchy dataset contains mostly object-centric, where the model wearing outfits are dominant and mostly centered, which can be a bias that models can easily capture. On the other hand, the INSECT dataset is less represented by pre-trained VLMs, thus their attention map are generally less accurate and localized. Yet, StructXLIP still demonstrates a better alignment between the rich text and the visual counterpart, as evidenced by the more local- ized attention map on the “antennae of females” and the “thorax” compared to GOAL. Table 3. Crossmodal retrieval at full ranks on SKETCHY, INSECT, DOCCI and DCI. We report mean Recall@K (%)±standard deviation for K = 1, 5, 10, 25, 50 on bothText→ImageandImage→Text. All results are averaged over three independent runs with random seeds 42, 1337 and 3407. Bold indicates the best result, while underlinedenotes the second-best. SKETCHY Method R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50 Long-CLIP[ECCV’24] 54.32±0.46 80.14±2.27 88.43±1.88 95.25±0.78 98.27±0.7552.76±1.18 80.31±1.89 88.08±1.50 95.16±1.03 97.75±1.00 FineLIP[CVPR’25]40.59±1.91 71.16±1.28 81.78±0.63 91.27±1.72 95.94±0.5740.33±1.04 72.11±0.05 82.38±0.11 91.45±1.85 95.77±0.69 SmartCLIP[CVPR’25] 50.73±1.11 81.09±0.93 94.56±1.6796.11±0.44 99.05±1.2851.30±0.52 80.83±1.89 94.04±0.7395.51±1.21 98.96±0.36 GOAL[CVPR’25]63.21±0.4787.13±1.5893.44±0.35 97.67±1.0999.05±0.1962.44±1.3787.82±0.9592.31±1.44 96.98±0.2999.00±0.54 StructXLIP69.86±0.46 90.85±0.09 95.42±0.07 98.61±0.35 99.22±0.0068.22±0.45 90.67±0.12 95.68±0.13 98.03±0.39 99.08±0.20 INSECT Method R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50 Long-CLIP[ECCV’24] 8.20±0.87 23.83±0.68 34.97±0.25 51.21±0.31 63.30±0.279.41±0.8424.78±0.62 37.31±0.8853.18±0.4763.39±0.98 FineLIP[CVPR’25]8.46±0.59 23.32±0.81 33.59±0.44 51.21±1.93 66.58±0.696.86±0.74 23.75±1.41 34.46±0.53 52.42±1.26 65.37±0.88 SmartCLIP[CVPR’25] 4.84±0.66 16.84±1.52 34.63±0.37 39.03±1.18 57.60±0.44 4.66±0.91 15.46±0.33 34.02±1.74 39.72±0.85 58.38±1.29 GOAL[CVPR’25]8.81±0.0724.35±0.9935.84±1.6255.44±0.7267.46±0.918.55±0.38 25.91±0.3636.18±0.66 53.02±1.14 66.41±0.52 StructXLIP9.93±0.90 26.60±1.20 38.34±1.02 56.99±0.61 69.34±0.909.50±0.80 26.60±0.29 39.64±0.64 54.92±0.46 68.65±0.67 DOCCI Method R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50 Long-CLIP[ECCV’24] 64.49±0.38 87.67±0.43 93.43±0.38 97.73±0.26 99.14±0.1163.08±0.32 87.45±0.44 93.14±0.30 97.45±0.17 99.02±0.12 FineLIP[CVPR’25]67.80±1.28 90.22±0.56 94.84±0.31 98.22±1.49 99.45±0.5266.39±0.44 89.12±1.22 94.47±0.67 97.90±0.38 99.20±0.93 SmartCLIP[CVPR’25] 74.92±0.66 94.08±0.29 97.31±1.12 99.37±0.44 99.82±0.1874.91±0.53 94.04±0.72 97.29±0.36 99.32±0.91 99.84±0.27 GOAL[CVPR’25]79.47±0.4196.65±1.3398.69±0.0299.69±0.5799.92±0.1479.43±0.8896.14±0.3897.25±0.7399.61±1.1299.90±0.19 StructXLIP83.04±0.05 97.06±0.20 98.96±0.04 99.84±0.02 99.98±0.0181.59±0.34 96.94±0.04 98.78±0.03 99.76±0.03 99.92±0.02 DCI Method R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50 Long-CLIP[ECCV’24] 59.23±0.72 80.89±0.55 87.04±0.48 92.60±0.31 95.10±0.2760.13±0.69 81.44±0.52 87.54±0.46 92.85±0.33 95.60±0.30 FineLIP[CVPR’25]66.13±1.77 85.34±0.44 89.79±1.06 94.14±0.59 96.35±0.5264.58±0.89 84.59±0.71 89.54±1.68 94.00±0.34 96.40±1.08 SmartCLIP[CVPR’25] 69.88±1.41 86.64±0.63 94.05±1.1895.00±0.31 97.25±0.7770.94±0.52 87.04±1.29 92.77±0.84 95.75±0.48 97.05±1.03 GOAL[CVPR’25]72.64±0.55 89.89±1.2293.70±0.33 95.75±0.66 97.25±0.4172.84±1.1190.50±0.4093.20±0.8196.60±0.5797.60±0.22 StructXLIP75.90±0.50 90.00±0.40 95.15±0.39 95.95±0.1397.85±0.1574.39±0.16 89.90±0.0594.30±0.23 96.75±0.23 97.75±0.10 Table 4. Plug-and-play enhancement of ourL ∗ on CLIP-based finetuning. Results on SKETCHY, INSECT, DOCCI, and DCI for Text→ImageandImage→Textretrieval. We report R@1, R@5, R@10, R@25, and R@50. Upper: SKETCHY and INSECT; Lower: DOCCI and DCI. Best in bold, with gain in↑ green. Method SKETCHYINSECT R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50 Long-CLIP 54.3280.1488.4395.2598.2752.7680.3188.0895.1697.758.2023.8334.9751.2163.309.4124.7837.3153.1863.39 +ourL ∗ 59.2485.3291.4596.2098.5359.5984.3791.0296.2998.459.2425.3936.3652.3366.849.3827.2938.6055.3566.15 ∆↑ 4.92↑ 5.18↑ 3.02↑ 0.95↑ 0.26↑ 6.83↑ 4.06↑ 2.94↑ 1.13↑ 0.70↑ 1.04↑ 1.56↑ 1.39↑ 1.12↑ 3.54↓ 0.03↑ 2.51↑ 1.29↑ 2.17↑ 2.76 FineLIP40.5971.1681.7891.2795.9440.3372.1182.3891.4595.778.4623.3233.5951.2166.586.8623.7534.4652.4265.37 +ourL ∗ 59.1585.2391.2896.6399.4058.5584.5490.0796.2099.058.8923.8335.7553.6367.186.5623.7636.0153.4566.84 ∆↑ 18.56↑ 14.07↑ 9.50↑ 5.36↑ 3.46↑ 18.22↑ 12.43↑ 7.69↑ 4.75↑ 3.28↑ 0.43↑ 0.51↑ 2.16↑ 2.42↑ 0.60↓ 0.30↑ 0.01↑ 1.55↑ 1.03↑ 1.47 SmartCLIP 50.7381.0994.5696.1199.0551.3080.8394.0495.5198.964.8416.8434.6339.0357.604.6615.4634.0239.7258.38 +ourL ∗ 52.9481.2694.7796.1399.2052.3380.9294.0495.6098.965.6116.8934.8040.1660.615.1815.8034.0739.7259.20 ∆↑ 2.21↑ 0.17↑ 0.21↑ 0.02↑ 0.15↑ 1.03↑ 0.090.00↑ 0.090.00↑ 0.77↑ 0.05↑ 0.17↑ 1.13↑ 3.01↑ 0.52↑ 0.34↑ 0.050.00↑ 0.82 GOAL63.2187.1393.4497.6799.0562.4487.8292.3196.9899.008.8124.3535.8455.4467.468.5525.9136.1853.0266.41 +ourL ∗ 67.8890.3395.1698.5399.6568.4889.8194.8298.2799.318.8128.0738.3656.6568.838.7527.1239.2154.4968.05 ∆↑ 4.67↑ 3.20↑ 1.72↑ 0.86↑ 0.60↑ 6.04↑ 1.99↑ 2.51↑ 1.29↑ 0.310.00↑ 3.72↑ 2.52↑ 1.21↑ 1.37↑ 0.20↑ 1.21↑ 3.03↑ 1.47↑ 1.64 SigLIP268.9190.8595.1698.7999.5766.7590.2493.9698.1099.316.3721.0731.8748.1960.366.5621.8531.9549.2260.19 +ourL ∗ 73.4991.9795.7798.7999.6070.3891.5495.7798.1099.357.6923.4933.4250.5263.046.9923.6635.0652.1663.21 ∆↑ 4.58↑ 1.12↑ 0.610.00↑ 0.03↑ 3.63↑ 1.30↑ 1.810.00↑ 0.04↑ 1.32↑ 2.42↑ 1.55↑ 2.33↑ 2.68↑ 0.43↑ 1.81↑ 3.11↑ 2.94↑ 3.02 LoRA57.0884.7291.8096.4698.7056.7485.3291.7196.2998.365.7916.8425.8242.4956.224.8416.4924.9640.3356.30 +ourL ∗ 62.0986.7993.9597.3299.0559.4185.9292.7597.2498.975.8718.3127.6344.9959.675.3218.6528.4144.8256.99 ∆↑ 5.01↑ 2.07↑ 2.15↑ 0.86↑ 0.35↑ 2.67↑ 0.60↑ 1.04↑ 0.95↑ 0.61↑ 0.08↑ 1.47↑ 1.81↑ 2.50↑ 3.45↑ 0.48↑ 2.16↑ 3.45↑ 4.49↑ 0.69 DoRA61.7786.1891.8897.3298.8860.9487.3392.3196.4698.467.1720.2129.8846.2061.926.0420.8131.5246.2960.02 +ourL ∗ 65.20 90.2694.9198.0199.3164.9488.3593.5297.5898.967.0824.0135.0652.5967.018.2924.3535.1552.7666.58 ∆↑ 3.43↑ 4.08↑ 3.03↑ 0.69↑ 0.43↑ 4.00↑ 1.02↑ 1.21↑ 1.12↑ 0.50↓ 0.09↑ 3.80↑ 5.18↑ 6.39↑ 5.09↑ 2.25↑ 3.54↑ 3.63↑ 6.47↑ 6.56 Method DOCCIDCI R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50 Long-CLIP 64.4987.6793.4397.7399.1463.0887.4593.1497.4599.0259.2380.8987.0492.6095.1060.1381.4487.5492.8595.60 +ourL ∗ 67.6790.8295.5998.4399.3967.9290.1695.1098.2299.4763.1384.1489.6994.2096.5564.3386.1989.1494.2096.50 ∆↑ 3.18 ↑ 3.15↑ 2.16↑ 0.70↑ 0.25↑ 4.84↑ 2.71↑ 1.96↑ 0.77↑ 0.45↑ 3.90↑ 3.25↑ 2.65↑ 1.60↑ 1.45↑ 4.20↑ 4.75↑ 1.60↑ 1.35↑ 0.90 FineLIP67.8090.2294.8498.2299.4566.3989.1294.4797.9099.2066.1385.3489.7994.1496.3564.5884.5989.5494.0096.40 +ourL ∗ 74.0694.2497.3599.2399.8072.9493.2796.5598.7899.6168.8886.6491.1095.2496.9067.3386.6990.7594.8597.00 ∆↑ 6.26↑ 4.02↑ 2.51↑ 1.01↑ 0.35↑ 6.55↑ 4.15↑ 2.08↑ 0.88↑ 0.41↑ 2.75↑ 1.30↑ 1.31↑ 1.10↑ 0.55↑ 2.75↑ 2.10↑ 1.21↑ 0.85↑ 0.60 SmartCLIP 74.9294.0897.3199.3799.8274.9194.0497.2999.3299.8469.8886.6494.0595.0097.2570.9487.0492.7795.7597.05 +ourL ∗ 77.3995.5798.6699.4799.9477.1095.4998.3499.8699.8969.9386.9494.3595.1097.3071.1487.6494.1095.8098.60 ∆↑ 2.47↑ 1.49↑ 1.35↑ 0.10↑ 0.12↑ 2.19↑ 1.45↑ 1.05↑ 0.54↑ 0.05↑ 0.05↑ 0.30↑ 0.30↑ 0.10↑ 0.05↑ 0.20↑ 0.60↑ 1.33↑ 0.05↑ 0.55 GOAL79.4796.6598.6999.6999.9279.4396.1497.2599.6199.9072.6489.8993.7095.7597.2572.8490.5093.2096.6097.60 +ourL ∗ 80.9696.9098.9699.7699.9280.3196.7398.8499.7599.9472.8989.7994.4096.1597.6573.8989.9393.5096.8098.15 ∆↑ 1.49↑ 0.25↑ 0.27↑ 0.070.00↑ 0.88↑ 0.59↑ 0.33↑ 0.14↑ 0.04↑ 0.25↓ 0.10↑ 0.7↑ 0.20↑ 0.40↑ 1.05↓ 0.57↑ 0.30↑ 0.20↑ 0.25 SigLIP271.8092.5395.8898.7899.4371.5192.4196.0698.5199.3966.1384.4989.3494.0096.4065.0884.5489.8494.6596.95 +ourL ∗ 75.47 94.8297.6799.3099.7873.5994.3397.4399.3799.8467.1486.5490.6594.8596.8566.7886.0990.5094.7596.90 ∆↑ 3.67↑ 2.29↑ 1.79↑ 0.52↑ 0.35↑ 2.08↑ 1.92↑ 1.37↑ 0.86↑ 0.45↑ 1.01↑ 2.05↑ 1.31↑ 0.85↑ 0.45↑ 1.7↑ 1.55↑ 0.66↑ 0.10↓ 0.05 LoRA77.8096.4598.5599.1199.1477.0096.0298.0599.3299.1472.0488.6992.3595.5097.1572.0489.2493.2095.0097.40 +ourL ∗ 79.3596.6198.5799.6199.4078.6596.3198.4599.6399.8674.4489.4492.8595.9597.2073.8989.2492.6096.4097.70 ∆↑ 1.55↑ 0.16↑ 0.02↑ 0.50↑ 0.26↑ 1.65↑ 0.29↑ 0.40↑ 0.31↑ 0.72↑ 2.40↑ 0.75↑ 0.50↑ 0.45↑ 0.05↑ 1.850.00↓ 0.60↑ 1.40↑ 0.30 DoRA66.2790.5095.1898.4999.6765.6589.5594.7898.4199.3660.3382.1988.0993.4595.7560.5381.8487.7993.4095.30 +ourL ∗ 70.6592.4796.5799.0299.7369.2291.8095.9898.9899.6162.5882.3488.3493.7596.1062.7882.9988.3994.2596.75 ∆↑ 4.38↑ 1.97↑ 1.39↑ 0.53↑ 0.06↑ 3.57↑ 2.25↑ 1.20↑ 0.57↑ 0.25↑ 2.25↑ 0.15↑ 0.25↑ 0.30↑ 0.35↑ 2.25↑ 1.15↑ 0.60↑ 0.85↑ 1.45 Table 5. Data Efficiency Analysis. We report mean Recall@K (%) on SKETCHY, INSECT, DOCCI, and DCI using 5%, 20%, 50% respectively of the training data. Results are reported for K = 1, 5, 10, 25, 50 on bothText→ImageandImage→Text. Bold indicates the best result, while underlinedenotes the second-best. SettingR@1R@5R@10R@25R@50R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50R@1R@5R@10R@25R@50 5% DataSKETCHYINSECT Long-CLIP[ECCV’24] 21.42 47.58 59.5976.7786.1823.40 49.14 62.4478.2487.312.506.9910.8818.8327.982.688.0312.6120.5530.66 FineLIP[CVPR’25]30.92 60.28 71.6885.4991.8831.00 59.50 70.1284.9792.572.508.1211.8722.1332.993.288.3313.0422.3132.06 SmartCLIP[CVPR’25] 25.22 53.02 77.1281.3590.5927.98 54.15 75.7682.9091.972.687.4311.6619.8630.143.378.4413.1521.6632.25 GOAL[CVPR’25]33.4263.9974.1886.2392.4935.0564.0875.8288.6993.532.687.5111.8322.1131.353.348.2012.3321.2431.78 StructXLIP36.70 68.05 78.0788.4394.3935.75 68.05 77.4689.1294.913.338.4612.8722.2833.683.578.9813.5622.5433.16 ∆↑ 3.28 ↑ 4.06 ↑ 0.95 ↑ 2.20 ↑ 1.90↑ 0.70 ↑ 3.97 ↑ 1.64 ↑ 0.43 ↑ 1.38↑ 0.65 ↑ 0.34 ↑ 1.00 ↑ 0.15 ↑ 0.69↑ 0.20 ↑ 0.54 ↑ 0.41 ↑ 0.23 ↑ 0.91 5% DataDOCCIDCI Long-CLIP[ECCV’24] 61.20 86.33 92.6197.4399.0861.57 86.51 92.5197.1698.9853.73 75.34 82.2488.7992.2553.98 76.69 83.7490.0593.85 FineLIP[CVPR’25]66.00 89.73 94.7597.9299.1563.65 87.39 92.2097.0698.2058.08 79.74 85.7991.4595.1560.88 80.29 86.6493.3595.80 SmartCLIP[CVPR’25] 65.12 89.33 94.3798.2599.3165.71 89.76 95.2098.2799.0057.78 78.14 87.3390.6093.7856.13 78.84 88.1491.0595.30 GOAL[CVPR’25]66.45 91.2995.0098.1799.5767.8591.4795.4498.2099.0060.8881.5487.5992.2594.9063.1383.1488.9493.5596.15 StructXLIP68.92 91.84 95.8498.4999.6169.76 91.61 95.7698.4399.4962.53 82.99 88.2093.1595.1064.03 84.34 89.1993.8096.60 ∆↑ 2.47 ↑ 0.55 ↑ 0.84 ↑ 0.24 ↑ 0.04↑ 1.91 ↑ 0.14 ↑ 0.32 ↑ 0.16 ↑ 0.49↑ 1.65 ↑ 1.45 ↑ 0.61 ↑ 0.90 ↓ 0.05↑ 0.9 ↑ 1.20 ↑ 0.25 ↑ 0.25 ↑ 0.45 20% DataSKETCHYINSECT Long-CLIP[ECCV’24] 35.75 64.85 75.7385.9293.9636.01 65.72 76.8688.0894.394.0511.05 18.7430.9241.544.1514.68 20.2932.8243.70 FineLIP[CVPR’25]38.17 69.17 78.8489.8195.5135.75 67.18 78.4188.3294.993.6312.44 19.0830.3143.094.2313.64 20.8131.6943.52 SmartCLIP[CVPR’25] 38.00 68.48 88.0890.8597.2438.95 69.86 88.4390.8596.293.459.5415.2828.1440.393.2012.15 19.3929.7440.01 GOAL[CVPR’25]46.63 77.3785.0692.4596.9046.4676.9486.1092.6696.433.2112.8018.8333.5145.374.3313.99 21.1633.1546.06 StructXLIP53.20 80.22 88.2694.3997.5049.48 79.45 88.5194.7397.154.5813.30 19.6034.1147.584.9214.1622.1234.2246.72 ∆↑ 6.57 ↑ 2.85 ↑ 0.18 ↑ 1.94 ↑ 0.26↑ 3.02 ↑ 2.51 ↑ 0.08 ↑ 2.07 ↑ 0.72↑ 0.53 ↑ 0.50 ↑ 0.52 ↑ 0.60 ↑ 2.21↑ 0.59 ↓ 0.52 ↑ 0.96 ↑ 1.07 ↑ 0.66 20% DataDOCCIDCI Long-CLIP[ECCV’24] 62.16 86.75 92.6197.1499.0061.61 86.67 92.0097.3799.0256.53 79.64 85.5491.0594.0057.33 79.14 85.8491.0094.40 FineLIP[CVPR’25]67.18 89.92 94.0098.1099.2263.96 87.80 92.4097.4198.3763.98 83.79 88.6993.0595.8562.83 83.79 88.6993.0595.85 SmartCLIP[CVPR’25] 70.00 91.69 95.3198.8099.4172.12 92.64 96.2499.0099.3864.7882.84 89.0092.8995.9762.13 83.09 89.4493.4596.15 GOAL[CVPR’25]71.4393.1296.9099.1899.7572.9493.6396.8499.0099.6764.63 84.8490.0593.9096.2065.7385.2990.6594.6597.00 StructXLIP77.18 95.61 97.8299.4799.8276.47 95.31 97.8699.3999.8267.03 86.24 90.6094.5996.5069.03 87.34 91.3595.4597.50 ∆↑ 5.75 ↑ 2.49 ↑ 0.92 ↑ 0.29 ↑ 0.07↑ 3.53 ↑ 1.68 ↑ 1.02 ↑ 0.39 ↑ 0.15↑ 2.25 ↑ 1.40 ↑ 0.55 ↑ 0.69 ↑ 0.30↑ 3.30 ↑ 2.05 ↑ 0.70 ↑ 0.80 ↑ 0.50 50% DataSKETCHYINSECT Long-CLIP[ECCV’24] 47.93 74.01 82.9992.2396.2943.96 73.58 82.5691.8896.035.1815.80 25.4740.7653.284.7518.13 27.6340.5953.11 FineLIP[CVPR’25]39.98 69.44 79.9790.1595.6037.65 68.91 79.1088.9594.995.6117.10 24.9641.1154.755.0117.44 26.3441.0254.06 SmartCLIP[CVPR’25] 45.34 76.34 90.93 93.0197.2546.46 74.70 90.1692.0697.583.5711.92 19.7433.3143.713.6313.31 21.6732.9742.06 GOAL[CVPR’25]55.79 82.7388.3596.2098.1055.2782.0489.1295.4298.365.8718.1325.5141.0255.225.3518.7726.8443.0556.55 StructXLIP60.97 87.48 92.4996.8998.7059.41 85.58 91.8095.5598.706.5620.12 29.1743.7857.776.2220.64 29.8444.1357.94 ∆↑ 5.18 ↑ 4.75 ↑ 1.56 ↑ 0.69 ↑ 0.60↑ 4.14 ↑ 3.54 ↑ 1.64 ↑ 0.13 ↑ 0.34↑ 0.69 ↑ 1.99 ↑ 3.66 ↑ 2.67 ↑ 2.55↑ 0.87 ↑ 1.87 ↑ 2.21 ↑ 1.08 ↑ 1.39 50% DataDOCCIDCI Long-CLIP[ECCV’24] 63.49 87.49 93.2097.3399.0662.20 87.41 92.9097.4099.0258.63 80.79 85.9991.6594.6559.04 80.89 85.6992.2095.35 FineLIP[CVPR’25]67.20 90.06 94.7598.2299.3164.29 87.94 93.1097.7298.8665.43 84.44 88.1094.0096.0563.83 84.44 89.1993.1495.00 SmartCLIP[CVPR’25] 73.75 94.24 97.4199.3199.7774.00 94.04 97.1798.8499.6566.98 84.79 90.7793.4096.0066.20 84.84 91.1594.6597.12 GOAL[CVPR’25]75.2594.6197.5999.5199.8875.7394.5397.3799.4599.8069.0386.8491.1094.9596.5067.6386.6991.2095.6597.05 StructXLIP79.78 96.37 98.3199.6199.8878.45 95.98 98.4399.5799.8071.19 87.84 91.3095.5597.0571.64 87.39 92.6596.9097.50 ∆↑ 4.53 ↑ 1.76 ↑ 0.72 ↑ 0.100.00↑ 2.72 ↑ 1.45 ↑ 1.06 ↑ 0.120.00↑ 2.16 ↑ 1.00 ↑ 0.20 ↑ 0.60 ↑ 0.55↑ 4.01 ↑ 0.70 ↑ 1.45 ↑ 1.25 ↑ 0.38 Figure 9. Qualitative results of StructXLIP and the second-best method GOAL on all four datasets. We present the attention maps between the color-highlighted texts on the visual counterpart. Compared to GOAL, StructXLIP overall shows a better correspondence between the rich textual descriptions and the visual objects. (2): Domain-Specific Appearance Lexicon Prompt–Response USER: You are a vision-language expert specializing in fashion imagery. Consider a representation of a garment where only edge information is preserved (i.e., contours, stitching lines, panel boundaries, silhouettes, and geometric layout), with all color, shading, and material cues removed. Please provide a comprehensive list of English words and short phrases used in the fashion domain whose meanings cannot be inferred from such an edge-only representation. Return the list as comma-separated items. LLM: [red, blue, green, yellow, orange, purple, pink, brown, black, white, grey, gray, navy, sky blue, baby blue, royal blue, midnight blue, teal, turquoise, aqua, mint, forest green, lime, olive, khaki, sage, emerald, moss, maroon, burgundy, wine, cranberry, magenta, fuchsia, rose, blush, salmon, coral, peach, apricot, mustard, gold, lemon, amber, beige, cream, ivory, off-white, tan, camel, sand, taupe, stone, ecru, charcoal, slate, ash, lavender, violet, lilac, plum, eggplant, silver, bronze, copper, champagne, metallic, pastel, neon, bright, dark, light, neutral, multicolor, multi-color, multicolored, multi-colored, cotton, linen, hemp, ramie, jute, wool, merino, cashmere, mohair, alpaca, angora, silk, mulberry silk, tussah silk, denim, chambray, twill, gabardine, canvas, jersey, interlock, french terry, flannel, fleece, felt, velvet, velour, corduroy, tweed, boucle, jacquard, brocade, damask, viscose, rayon, modal, lyocell, tencel, acetate, cupro, polyester, nylon, spandex, elastane, lycra, acrylic, polyamide, microfiber, chiffon, georgette, organza, tulle, mesh, crepe, charmeuse, satin, taffeta, batiste, voile, lawn, leather, suede, nubuck, shearling, sheepskin, faux leather, vegan leather, faux suede, fur, faux fur, mink, fox fur, rabbit fur, knit, rib knit, cable knit, fine knit, softshell, hardshell, neoprene, Gore-Tex, ripstop, mesh, spacer fabric, lam ́ e, bonded fabric, coated fabric, sherpa, denier, pongee, jacquard knit] USER: You are an entomology-focused vision–language expert. Consider an insect image represented only by its edge map, that is, a structure-centric depiction containing contours, boundaries, wing venation geometry, and overall body segmentation, but with all color, shading, pigmentation, and optical surface cues removed. Please produce a comprehensive list of words and short phrases used in insect field guides or taxonomic descriptions whose meanings cannot be inferred from such an edge-only representation. These should strictly correspond to appearance attributes such as color terms, pigmentation qualities, optical reflectance (e.g., iridescence, metallic sheen), translucency/opacity, and other non-geometric visual properties. Return the list as comma-separated items. LLM: [black, brown, dark brown, light brown, tan, beige, cream, white, off-white, gray, grey, charcoal, slate, ash, red, reddish, orange, yellow, green, blue, purple, pink, magenta, violet, rust, russet, chestnut, mahogany, clay, ochre, umber, sienna, tawny, fawn, amber, honey-colored, golden, bronze, coppery, pale, dusky, washed-out, faded, dim, drab, rufous, testaceous, fulvous, ferruginous, castaneous, fuscous, livid, piceous, violaceous, cyaneous, glaucous, mottled, blotchy, flecked, freck- led, stained, tinged, tinted, smudged, clouded, diffuse, suffused, irregularly pigmented, unevenly pigmented, faintly pigmented, deeply pigmented, melanized, depigmented, discolored, frosted, pruinose, powdery, mealy, chalky, dusty, granular, velvety, shaded, darkened, lightened, somber, sooty, smoky, smeared, iridescent, metallic, submetallic, opalescent, pearlescent, rainbowlike, pris- matic, lustrous, holographic, shiny, glossy, subglossy, dull, matte, satiny, silky, polished, reflective, non-reflective, sheeny, mirrorlike, transparent, translucent, semi-translucent, opaque, hyaline, sub- hyaline, smoky-hyaline, warm-toned, cool-toned, earthy, vivid, dull-colored, bright, pale-colored, dark-colored, satiny, silken, glassy, resinous, lacquered, oily, greasy, waxy, glistening, gleaming] Figure 10. Prompt and response used in our LLM-based construction of the domain-specific appearance vocabulary lists for lexicon filtering.