Paper deep dive
Rethinking Text-Based Image Retrieval in Specific Domain
Jingyang Tan, Sheng Yang, Yuanpeng Chen, Jian Wang, Nianjin Ye, Chen Xing, Lanpeng Jia
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/16/2026, 2:44:06 AM
Summary
The paper introduces SecMM-TBIR, a new benchmark for Text-based Image Retrieval (TBIR) in specific domains like surveillance, addressing the limitation of single-match assumptions in existing datasets. It proposes the DSMM-TBIR data engine to construct a multi-match benchmark with 50k images and 200 queries. Additionally, it presents the Semantic-Aware Fine-Tuning (SAFT) framework, which combines Semantic-Aware Soft-Label Supervision (SASS) and Intra-modal Structural Distillation (ISD) to mitigate false negatives caused by semantic compression in specific domains, achieving significant performance gains over standard contrastive learning.
Entities (12)
Relation Signals (8)
DSMM-TBIR → constructs → SecMM-TBIR
confidence 95% · Leveraging this engine, we construct Security Multi-Match TBIR (SecMM-TBIR)
SecMM-TBIR → targetsdomain → surveillance
confidence 95% · a benchmark comprising 50k surveillance images
SAFT → includes → ISD
confidence 92% · incorporates Semantic-Aware Soft-Label Supervision (SASS) and Intra-modal Structural Distillation (ISD)
SAFT → includes → SASS
confidence 92% · incorporates Semantic-Aware Soft-Label Supervision (SASS) and Intra-modal Structural Distillation (ISD)
SAFT → improves → SecMM-TBIR
confidence 90% · SAFT yields an average mAP@20 gain of 7.8 points on SecMM-TBIR over standard image-text contrastive (ITC) fine-tuning
UniME-V2 → usedby → SAFT
confidence 88% · We employ UniME-V2 ... to generate reference visual embeddings ... for SASS
Flickr30K → isbaselinefor → general_domain_TBIR
confidence 85% · In the general domain, Flickr30K ... serve as the predominant testbeds
MS-COCO → isbaselinefor → general_domain_TBIR
confidence 85% · In the general domain, ... MS-COCO ... serve as the predominant testbeds
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Driven by the rapid advancement of vision-language representation learning, Text-based Image Retrieval (TBIR) has made notable progress. However, existing benchmarks are predominantly constructed on an exclusive single-match assumption between query and images. While effective in general scenarios, this assumption fails to reflect practical system performance in specific domains (e.g., surveillance), where a single query often corresponds to multiple relevant candidate images. To address this limitation, we design a Domain-Specific Multi-Match Text-based Image Retrieval (DSMM-TBIR) data engine. Leveraging this engine, we construct Security Multi-Match TBIR (SecMM-TBIR), a benchmark comprising 50k surveillance images with 200 comprehensive queries. Furthermore, we observe that vanilla contrastive learning in specific domains suffers from severe false negatives, forcing the model to push apart semantically similar pairs and thus degrading retrieval performance. We propose the Semantic-Aware Fine-Tuning (SAFT) framework to address semantic compression in specific domains, which incorporates Semantic-Aware Soft-Label Supervision (SASS) and Intra-modal Structural Distillation (ISD) to establish a promising paradigm for domain-specific TBIR tasks. Experiments across diverse CLIP-like models demonstrate that SAFT yields an average mAP@20 gain of 7.8 points on SecMM-TBIR over standard image-text contrastive (ITC) fine-tuning, while also improving general-domain performance. The entire benchmark will be released to facilitate further research.
Tags
Links
- Source: https://arxiv.org/abs/2608.10524v1
- Canonical: https://arxiv.org/abs/2608.10524v1
Trouble viewing inline? Open PDF directly →
Full Text
53,202 characters extracted from source content.
Expand or collapse full text
Rethinking Text-Based Image Retrieval in Specific Domain Jingyang Tan1 , Sheng Yang2 , Yuanpeng Chen4, Jian Wang2 , Nianjin Ye3, Chen Xing3, Lanpeng Jia3 Email: tanjingyang.wjf@gmail.com Abstract Driven by the rapid advancement of vision-language representation learning, Text-based Image Retrieval (TBIR) has made notable progress. However, existing benchmarks are predominantly constructed on an exclusive single-match assumption between query and images. While effective in general scenarios, this assumption fails to reflect practical system performance in specific domains (e.g., surveillance), where a single query often corresponds to multiple relevant candidate images. To address this limitation, we design a Domain-Specific Multi-Match Text-based Image Retrieval (DSMM-TBIR) data engine. Leveraging this engine, we construct Security Multi-Match TBIR (SecMM-TBIR), a benchmark comprising 50k surveillance images with 200 comprehensive queries. Furthermore, we observe that vanilla contrastive learning in specific domains suffers from severe false negatives, forcing the model to push apart semantically similar pairs and thus degrading retrieval performance. We propose the Semantic-Aware Fine-Tuning (SAFT) framework to address semantic compression in specific domains, which incorporates Semantic-Aware Soft-Label Supervision (SASS) and Intra-modal Structural Distillation (ISD) to establish a promising paradigm for domain-specific TBIR tasks. Experiments across diverse CLIP-like models demonstrate that SAFT yields an average mAP@20 gain of 7.8 points on SecMM-TBIR over standard image-text contrastive (ITC) fine-tuning, while also improving general-domain performance. The entire benchmark will be released to facilitate further research. 1 Introduction Text-based Image Retrieval (TBIR), which retrieves relevant images from large-scale databases given a textual query, has emerged as a fundamental vision-language task. This multi-modal synergy has powered wide spectrum of real-world applications, yielding substantial benefits for scenarios where the ability to rapidly and accurately retrieve targeted visual instances from massive galleries is the core requirement. The development of robust benchmarks is fundamental to the advancement of TBIR. In the general domain, Flickr30K (Plummer et al. 2017) and MS-COCO (Lin et al. 2014) serve as the predominant testbeds, characterized by their broad semantic coverage, providing an excellent proxy for evaluating generalized cross-modal alignment capabilities. However, these benchmarks are predominantly based on the single-match query-image mapping paradigm. While highly effective in general domains, this assumption fundamentally contradicts the practical requirements of specific scenarios. As illustrated in Fig 2, in such compressed semantic domain, textual descriptions tend to be simple and often naturally correspond to multiple relevant visual instances. Evaluating models with single-match labels in specific domains inherently biases standard metrics by treating potential positives as negatives, preventing them from reflecting actual retrieval performance. To facilitate the rapid construction of benchmarks across diverse vertical scenarios for evaluation, we design the Domain-Specific Multi-Match Text-based Image Retrieval (DSMM-TBIR) data engine. It leverages the generative and comprehension capabilities of Large Language Models (LLMs) and Vision-Language Models (VLMs), complemented by verification from multiple expert universal multi-modal embedding models. Using this engine, we construct Security Multi-Match TBIR (SecMM-TBIR), a multi-match benchmark comprising 50k surveillance images and 200 comprehensive queries. Figure 1: Consistent Improvement on SecMM-TBIR Benchmark Driven by SAFT. It yields an average gain of +7.8+7.8 mAP@20 over standard ITC fine-tuning, allowing MobileCLIP-S1 to outperform 2B pretrained embedders. In the context of multi-modal contrastive learning, the prevalence of false negatives (FNs) poses a critical bottleneck. Existing research has primarily diverged into two distinct paradigms: hard negative mining (HNM) and soft-label supervision. HNM is widely used to identify informative hard negatives (HNs) for enhanced contrastive learning (Gu et al. 2025a; Yuksekgonul et al. 2023b). Empirical evidence reveals that applying HNM to specific domains often yields suboptimal performance: as illustrated in Fig. 4, semantic distributions in domain-specific settings are highly compressed, resulting in a substantially higher density of FNs which are hard to identify. To mitigate the cross-modal misleading problem where one-hot labels falsely penalize potential positives, CUSA (Huang et al. 2024) and SoftCLIP (Gao et al. 2024) leverage soft labels derived from unimodal pre-trained models. However, this paradigm inherently creates a mismatch with cross-modal alignment, as unimodal similarity distributions cannot faithfully represent the cross-modal alignment required for TBIR, leading to inconsistent performance gains in domain-specific retrieval. To address these limitations, we propose the Semantic-Aware Fine-Tuning (SAFT) framework. SAFT extends standard contrastive learning with two complementary components: Semantic-Aware Soft-Label Supervision (SASS), which accommodates potential positive text-image matches via a soft cross-modal alignment distribution, and Intra-modal Structural Distillation (ISD), which preserves relative correlations among images within the visual modality. As shown in Fig 1, this integrated framework consistently improves domain-specific retrieval performance. In summary, the main contributions are as follows: • We introduce SecMM-TBIR, a multi-match benchmark for surveillance scenarios. Built via our DSMM-TBIR data engine, it comprises 50k images and 200 queries across pedestrian and vehicle domains, and will be publicly released to facilitate further research. • We propose SAFT, a systematic fine-tuning framework for CLIP-like (Radford et al. 2021) models to address performance degradation caused by domain-specific semantic compression. Integrating SASS and ISD, it effectively mitigates false negatives in contrastive learning. • Extensive experiments across various models demonstrate that SAFT achieves an average mAP@20 gain of 7.8 points on SecMM-TBIR, while also boosting general-domain performance. Moreover, SAFT generalizes well to other domains and enhances compositional reasoning. 2 Related Works 2.1 Text-Based Image Retrieval Benchmarks Rigorous benchmarks lay the foundation for evaluating TBIR systems. Existing testbeds are typically categorized into general-domain and specific-domain. General-domain benchmarks, such as MS-COCO and Flickr30K, encompass an extensive range of scenes, serving as the primary standard for evaluating cross-modal representation alignment. Conversely, domain-specific benchmarks concentrate on constrained vertical scenarios. Within this category, Text-based Person Retrieval (TBPR) represents a well-established task, evaluated on datasets such as CUHK-PEDES (Li et al. 2017), RSTPReid (Zhu et al. 2021), ICFG-PEDES (Ding et al. 2021) and SYNTH-PEDES (Zuo et al. 2024). Although traditional benchmarks have driven substantial progress, they are structurally limited by rigid single-match mappings and verbose captions, diverging from industrial realities that demand task-aligned concise queries and multi-match retrieval. To bridge this gap, several benchmarks have emerged, with InQuire (Vendrow et al. 2024) and FSIR-BD (Idan et al. 2026) being the closest to our work. InQuire focuses on expert-level ecological retrieval but demands intensive manual labeling, limiting its scalability to broader scenarios. FSIR-BD, built on the general-purpose Visual Genome dataset (Krishna et al. 2017), relies heavily on manual query annotation and struggles to provide comprehensive semantic coverage required for real-world scenarios. Consequently, balancing annotation scalability, public accessibility, and diverse semantic coverage remains an open challenge in the construction of DSMM-TBIR benchmarks. Figure 2: Single-Match Benchmarks. The single-match paradigm ignores multiple potential matches, leading to biased training and evaluation, particularly in specific domain. Figure 3: DSMM-TBIR Data Engine. Phase I constructs a curated multi-modal pool through Distribution-Aware Prompting (DAP) and Centroid-Guided Diversity Sampling (CGDS). Phase I employs multiple expert embedders to pre-annotate query-image pairs. Finally, Phase I manually filters these candidates to obtain verified multi-match labels for SecMM-TBIR. 2.2 False Negatives in Contrastive Learning Foundational vision-language models, such as CLIP and ALIGN (Jia et al. 2021), establish cross-modal representation alignment via contrastive pre-training on massive data (Schuhmann et al. 2021, 2022) with large batch sizes, which inevitably introduce FNs. Recent universal embedding models (Li et al. 2026; Jian et al. 2025a) utilize threshold-based HNM to mitigate FNs while capturing HNs. Yet, this yields suboptimal performance in domain-specific settings, as threshold-based methods struggle to disentangle HNs from FNs under concentrated semantic distributions. To address this, a prevalent alternative leverages soft-label supervision. Within specific domains, e-CLIP (Shin et al. 2022) introduced catalog-based soft labels to address duplicate products. While MedCLIP (Wang et al. 2022) proposed a soft semantic matching loss based on multi-hot labels, it is applicable only to closed-set retrieval. Ko and Park (2025) leveraged pre-trained models to extract domain-specific features and graph representations for joint supervision. Meanwhile, CellCLIP (Lu et al. 2026) employed DINOv2 (Oquab et al. 2023) to construct a continuous visual similarity matrix, adopting the Continuously Weighted Contrastive Loss (Srinivasa et al. 2023) for alignment guidance. Paradigms such as ICSD (Chen et al. 2025) and CUSA leverage unimodal soft labels to supervise cross-modal alignment. Notably, ICSD suffers from architectural coupling that limits deployment. In contrasct, CUSA remains architecture-independent but underperforms as unimodal statistics fail to reflect true cross-modal matching probabilities. In summary, existing methods rely heavily on rigid domain priors or approximate cross-modal relationships using unimodal statistics, leaving domain-specific TBIR still underexplored. 3 Methodology Figure 4: Dataset Statistics and Semantic Compression Analysis. Left: Statistics of training set and SecMM-TBIR benchmark. Right: Visualizations of cross-sample similarities (1,000 random samples), where specific domains exhibit higher similarities. 3.1 DSMM-TBIR Data Engine We introduce the DSMM-TBIR Engine, a versatile data pipeline designed for the automated construction of benchmarks across diverse vertical domains. It is designed to overcome four critical limitations in current benchmarks. General-domain benchmarks fail to represent vertical industrial scenarios; we addressed domain incompatibility by targeting constrained settings. Unlike traditional datasets that rely on detailed captions, we generate concise queries which are more compatible for practical applications to resolve caption-query disparity. We construct a large-scale testbed comprising 50k images and 200 comprehensive queries to address the evaluation saturation caused by the limited test scales of existing benchmarks. Evaluating models with single-match labels in specific domains prevents standard metrics from reflecting actual retrieval performance. To eliminate the metric bias, we utilize a multi-match paradigm and adopt mAP@K as the primary metrics. As illustrated in Fig 3, VLMs and LLMs enable efficient adaptation to specific scenarios, allowing rapid TBIR evaluation without extensive manual labeling. DSMM-TBIR data engine comprises three sequential phases: Phase 1: Adaptive Data Pool Curation We utilize both textual and visual branches to establish a representative domain-specific multi-modal data pool. For the textual branch, we develop Distribution-Aware Prompting (DAP) to generate queries that comprehensively cover the scenarios within the task domain. Specifically, we perform a statistical analysis of the textual distribution in the caption pool and incorporate the resulting statistics into prompt engineering to guide LLM-based query generation. Details of the prompting procedure are provided in the Supplementary Material. This process transforms raw metadata into natural queries that comprehensively reflect the semantic distributions of real-world environments. For the visual branch, Centroid-Guided Diversity Sampling (CGDS) is designed to construct a diverse image gallery. Under this strategy, the raw image pool ℐI is projected into a high-dimensional embedding space via a versatile visual representation model (Siméoni et al. 2025), and partitioned into K~ K distinct semantic regions using K-means clustering. Within each cluster, images are randomly sampled to capture intra-class variance and visual diversity, ensuring balanced and comprehensive coverage of the target domain. Phase 2: Multi-Expert Collaborative Filtering (MECF) Individual VLMs suffer from limited cross-domain generalization, delivering uneven performance across domains, with each model exhibiting varying performance in different scenarios. To alleviate this limitation, we employ an ensemble of four diverse embedders (Li et al. 2026; Jian et al. 2025b; günther2025jinaembeddingsv4universalembeddingsmultimodal; Tschannen et al. 2025). For each query q, we retrieve the top-K K visual candidates with the highest cosine similarity in the corresponding embedding space. To minimize false negatives, we define the preliminary candidate set (q)C(q) as the union of the top-K K retrieval results across all M models: (q)=⋃i=1Mx∈ℐ|rankℐ(sim(ϕi(q),ψi(x)))≤K^.C(q)= _i=1^M \x \; |\;rank_I (sim( _i(q), _i(x)) )≤ K \. (1) where q denotes the query, x is a candidate image, ϕi _i and ψi _i are the text and image embedders of the i-th model, and sim(⋅)sim(·) denotes cosine similarity. Phase 3: Label Refinement Annotators manually inspect the visual candidates in (q)C(q) and verify their relevance to the query q, filtering out residual false positives to obtain the final multi-match labels. 3.2 Rethinking Fine-Tuning Paradigms To characterize the semantic density of TBIR datasets, we randomly sample 1,000 instances from each dataset and compute pairwise cosine similarities in the embedding space. As shown in Fig 4, specific domains exhibit substantially more compressed latent spaces. Consequently, semantically related yet unpaired samples are more prevalent within a mini-batch and be treated as negatives under standard image-text contrastive (ITC) learning. This motivates us to revisit three common fine-tuning choices in TBIR tasks: Text Encoder Joint Fine-Tuning in specific domains often induces overfitting to the textual patterns of training sets. Standard Image Self-Supervised (ISS) Objectives force the separation of unpaired images in a mini-batch which are semantically near-identical, yielding distorted rather than discriminative representations. Hard Negative Mining defines a threshold α by shifting the similarity between the query embedding eqe_q and its paired positive target embedding et+e_t^+ with a margin parameter β: α=cos(eq,et+)+βα= (e_q,e_t^+)+β (2) Within a batch of candidate negatives et−\e_t^-\, any sample satisfying cos(eq,et−)>α (e_q,e_t^-)>α is excluded as a false negative, while the remaining instances are ranked by similarity to select the top-K hard negatives. However, this threshold-based mechanism enforces a rigid separation between FNs and HNs, failing to reliably distinguish them in semantically compressed training sets. Figure 5: Overview of SAFT Framework. A frozen universal multi-modal embedding model serves as the teacher, providing cross-modal soft targets for SASS and image-to-image structural targets for ISD, complementing the standard ITC objective. 3.3 Loss Functions The standard ITC loss in CLIP pre-training is formulated as: ℒITC=−12N∑i=1N∑j=1N(qi,jlogpi,j+qj,ilogpj,i)L_ITC=- 12N _i=1^N _j=1^N (q_i,j p_i,j+q_j,i p_j,i ) (3) where N represents batch size, q represents rigid one-hot labels. The predicted image-to-text (I2T) probability pi,jp_i,j measures the probability of matching the i-th image to the j-th text, while the text-to-image (T2I) probability pj,ip_j,i measures the probability of matching the i-th text to the j-th image: pi,j p_i,j =exp((eiv)⊤ejt/τ)∑k=1Nexp((eiv)⊤ekt/τ), = ((e^v_i) e^t_j/τ) _k=1^N ((e^v_i) e^t_k/τ), (4) pj,i p_j,i =exp((eit)⊤ejv/τ)∑k=1Nexp((eit)⊤ekv/τ) = ((e^t_i) e^v_j/τ) _k=1^N ((e^t_i) e^v_k/τ) where eive^v_i and ejte^t_j denote the normalized visual and textual embeddings for the i-th and j-th instances within a mini-batch respectively, and τ represents the temperature. We employ UniME-V2 (Gu et al. 2025b), a universal cross-modal embedder to generate reference visual embeddings EvE^v and textual embeddings EtE^t, which preserve deep semantic inter-dependencies. For a given training batch, these embeddings are used to derive soft-label over image-text pairs: ri,j r_i,j =exp((Eiv)⊤Ejt/τo)∑k=1Nexp((Eiv)⊤Ekt/τo), = ((E^v_i) E^t_j/ _o) _k=1^N ((E^v_i) E^t_k/ _o), (5) rj,i r_j,i =exp((Eit)⊤Ejv/τo)∑k=1Nexp((Eit)⊤Ekv/τo) = ((E^t_i) E^v_j/ _o) _k=1^N ((E^t_i) E^v_k/ _o) Let Si2t=[pi,j]S_i2t=[p_i,j] and St2i=[pj,i]S_t2i=[p_j,i] denote the predicted cross-modal distributions of the lightweight student model. Similarly, Ti2t=[ri,j]T_i2t=[r_i,j] and Tt2i=[rj,i]T_t2i=[r_j,i] denote the target distributions generated by the teacher embedder. All distributions are in ℝN×NR^N× N. The SASS loss is formulated as a bidirectional Kullback-Leibler (KL) divergence to encourage robust cross-modal alignment: ℒSASS=14( _SASS= 14 ( KL(Ti2t∥Si2t)+KL(Tt2i∥St2i) _KL(T_i2t S_i2t)+D_KL(T_t2i S_t2i) (6) + + KL(Si2t∥Ti2t)+KL(St2i∥Tt2i)) _KL(S_i2t T_i2t)+D_KL(S_t2i T_t2i) ) While SASS improves cross-modal alignment, it does not explicitly preserve the relational structure within the visual modality. We therefore propose Intra-modal Structural Distillation (ISD), which transfers the teacher’s fine-grained image-image relationships to the student. Specifically, for each mini-batch, we use the teacher’s visual embeddings EvE^v to construct an intra-modal soft targets: ri,jv=exp((Eiv)⊤Ejv/τo)∑k=1Nexp((Eiv)⊤Ekv/τo)r^v_i,j= ((E^v_i) E^v_j/ _o) _k=1^N ((E^v_i) E^v_k/ _o) (7) We define the teacher’s intra-modal structural distribution as Tv2v=[ri,jv]∈ℝN×NT_v2v=[r^v_i,j] ^N× N, the student’s predicted distribution is denoted as Sv2v=[pi,jv]∈ℝN×NS_v2v=[p^v_i,j] ^N× N, where pi,jvp^v_i,j is computed based on the visual embeddings. pi,jv=exp((eiv)⊤ejv/τ)∑k=1Nexp((eiv)⊤ekv/τ)p^v_i,j= ((e^v_i) e^v_j/τ) _k=1^N ((e^v_i) e^v_k/τ) (8) The ISD loss is then formulated as the KL divergence between these two intra-modal distributions: ℒISD=KL(Tv2v∥Sv2v)L_ISD=D_KL(T_v2v S_v2v) (9) Finally, the joint training objective is formulated as: ℒSAFT=ℒITC+α⋅ℒSASS+β⋅ℒISDL_SAFT=L_ITC+α·L_SASS+β·L_ISD (10) where α and β are balancing hyperparameters. Difference from Unimodal Proxy Supervision. Unlike prior works that use unimodal soft labels as proxies for cross-modal supervision, SAFT leverages cross-modal similarity distributions derived from a universal multi-modal embedder, enabling more faithful cross-modal supervision. Model SecMM-TBIR Pedestrian Vehicle m@10 m@20 m@30 m@10 m@20 m@30 TinyCLIP-22M/32 32.5 25.1 23.3 45.4 38.0 33.5 + ITC 38.0 29.8 27.8 51.7 42.5 37.0 + CUSA 36.2 29.3 27.6 48.7 40.7 34.8 + SAFT (ours) 40.7 32.1 29.6 60.1 51.3 45.8 TinyCLIP-45M/32 40.0 32.7 30.4 53.3 45.2 40.2 + ITC 45.4 37.6 35.3 67.6 56.7 50.7 + CUSA 42.3 34.5 32.4 60.2 51.1 46.6 + SAFT (ours) 50.5 41.5 38.4 73.3 64.6 58.5 MobileCLIP-S0 49.7 41.0 38.5 60.1 52.2 46.5 + ITC 53.1 44.2 41.3 65.8 57.4 51.6 + CUSA 53.7 45.8 42.7 69.2 59.4 53.3 + SAFT (ours) 59.8 49.7 46.7 77.7 67.9 61.7 MobileCLIP-S1 52.1 42.6 39.8 65.2 57.2 51.7 + ITC 58.0 48.4 44.9 64.8 59.4 54.8 + CUSA 59.6 49.1 46.3 69.0 62.2 56.8 + SAFT (ours) 65.0 54.9 52.1 79.2 73.0 66.8 OpenCLIP-B/32 34.0 27.4 25.5 55.6 46.5 41.6 + ITC 41.0 33.2 31.4 63.5 55.6 49.3 + CUSA 39.2 31.6 29.3 63.0 55.6 49.7 + SAFT (ours) 47.5 39.0 36.3 73.1 64.8 58.5 OpenCLIP-B/16 40.5 32.9 30.4 56.4 48.0 42.9 + ITC 47.8 38.8 36.3 65.4 56.8 51.6 + CUSA 49.3 40.1 37.6 65.8 57.0 52.2 + SAFT (ours) 57.9 46.9 44.0 75.4 68.7 62.8 Table 1: Main Results on SecMM-TBIR Benchmark. Fine-tuned Encoders Pedestrian (mAP@20 ↑ ) Vehicle (mAP@20 ↑ ) Image Text T-22 T-45 M-S0 M-S1 B/16 B/32 T-22 T-45 M-S0 M-S1 B/16 B/32 – – 25.1 32.7 41.0 42.6 32.9 27.4 38.0 45.2 52.2 57.2 48.0 46.5 ✓ – 29.829.8 37.637.6 44.244.2 48.448.4 38.838.8 33.233.2 42.542.5 56.756.7 57.957.9 61.461.4 56.856.8 57.357.3 ✓ ✓ 26.5−3.326.5 [rgb]1,0,0-3.3 35.8−1.835.8 [rgb]1,0,0-1.8 40.0−4.240.0 [rgb]1,0,0-4.2 43.5−4.943.5 [rgb]1,0,0-4.9 38.1−0.738.1 [rgb]1,0,0-0.7 29.2−4.029.2 [rgb]1,0,0-4.0 40.0−2.540.0 [rgb]1,0,0-2.5 54.5−2.254.5 [rgb]1,0,0-2.2 55.9−2.055.9 [rgb]1,0,0-2.0 59.7−1.759.7 [rgb]1,0,0-1.7 56.1−0.756.1 [rgb]1,0,0-0.7 55.1−2.255.1 [rgb]1,0,0-2.2 Table 2: Ablation Study on Encoder Optimization. Results obtained by fine-tuning different encoder modules are reported. Model General Benchmarks Flickr30K MS-COCO R@1 R@5 R@10 R@1 R@5 R@10 TinyCLIP-22M/32 51.8 77.2 85.3 28.4 53.0 64.5 + ITC 56.1 80.6 88.4 31.6 58.0 69.3 + CUSA 56.9 81.7 88.4 33.5 61.0 72.2 + SAFT (ours) 57.3 82.9 88.7 35.0 62.6 73.7 TinyCLIP-45M/32 64.0 86.0 92.1 36.9 62.6 73.1 + ITC 67.6 89.1 93.8 41.1 67.7 78.0 + CUSA 68.4 89.6 94.5 42.2 69.5 79.7 + SAFT (ours) 70.0 89.9 94.8 43.5 70.9 80.8 MobileCLIP-S0 66.8 88.7 93.1 39.6 65.9 75.6 + ITC 70.8 90.9 93.6 44.6 72.5 81.6 + CUSA 71.3 91.7 95.3 45.6 73.2 82.8 + SAFT (ours) 71.9 91.8 95.5 46.3 73.3 83.0 MobileCLIP-S1 71.4 90.8 95.0 43.7 69.1 78.2 + ITC 75.9 93.1 94.9 49.2 75.4 83.6 + CUSA 76.8 93.7 96.7 50.0 76.0 84.3 + SAFT (ours) 77.4 94.2 96.9 50.7 77.0 85.4 OpenCLIP-B/32 61.1 85.0 90.9 37.1 62.3 72.7 + ITC 66.3 88.6 93.1 42.8 71.2 81.1 + CUSA 69.3 90.2 94.5 43.7 71.2 80.7 + SAFT (ours) 69.3 90.5 94.7 45.0 72.1 81.6 OpenCLIP-B/16 67.5 88.4 93.0 40.1 66.0 75.6 + ITC 73.9 92.2 95.5 47.1 73.3 82.5 + CUSA 75.7 93.1 96.3 48.1 74.6 83.5 + SAFT (ours) 76.1 93.3 96.5 49.3 75.7 84.5 Table 3: Main Results on General Benchmarks. 4 Experiments 4.1 Experiment Setup Evaluation Benchmarks. As illustrated in Fig 4, we perform our evaluation on SecMM-TBIR, which comprises 50k surveillance images and 200 comprehensive queries across two subdomains: pedestrian and vehicle. More details are provided in the Supplementary Material. Training Sets. The training sets comprise two standard general datasets, augmented by a specialized internal dataset which are decoupled from SecMM-TBIR benchmark: • General Multi-modal Dataset: Comprising Flickr30K and MS-COCO training splits to ensure data balance, this dataset is explicitly utilized to preserve robust pretrained general cross-modal alignment. • Specialized Internal Dataset: Consisting of real-world image-text pairs from surveillance scenarios, the dataset focuses on pedestrian and vehicle domains to enhance discriminability. Specifically, it’s built via Rex-Omni (Jiang et al. 2025) for video frame object detection and Qwen3-VL (Bai et al. 2025) together with Qwen-3.5 (Qwen Team 2026) for high-fidelity captions. Baseline Configurations. To demonstrate broad applicability, we select a broad spectrum of CLIP-like models: TinyCLIP (Wu et al. 2023), MobileCLIP (Faghri et al. 2025), OpenCLIP (Cherti et al. 2023). As our first baseline, we freeze the text encoder and optimize only the visual branch via the standard ITC objective. Our second comparative baseline CUSA establishes cross-modal alignment via teacher-derived unimodal priors without introducing additional architectural components, and we upgrade its teacher to the stronger UniME-V2 to ensure fairness. Implementation Details. All experiments are conducted on a single NVIDIA RTX 5090 GPU (32GB VRAM). The model is fine-tuned for 25k iterations via AdamW (Loshchilov and Hutter 2019) with batch size of 128. Specifically, the learning rate linearly warms up from 5×10−75× 10^-7 to 5×10−65× 10^-6 over the first 10%10\% of steps, then follows cosine annealing with a weight decay of 0.050.05. Further details are provided in the Supplementary Material. Model General (R@1 ↑ ) SecMM-TBIR (m@20 ↑ ) Flickr COCO Pedestrian Vehicle Tiny-22M/32 51.8 28.4 25.1 38.0 Tiny-45M/32 64.0 36.9 32.7 45.2 MobileCLIP-S0 66.8 39.6 41.0 52.2 MobileCLIP-S1 71.4 43.7 42.6 57.2 OpenCLIP-B/32 61.1 37.1 27.4 46.5 OpenCLIP-B/16 67.5 40.1 32.9 48.0 Qwen3-VL-2B 90.2 77.3 45.1 64.3 UniMEV2-2B 89.9 65.1 57.4 68.9 Table 4: Zero-shot Generalization of Foundation Models. Backbone Baseline Pedestrian Dataset Vehicle Dataset K=32K=32 K=64K=64 K=96K=96 K=32K=32 K=64K=64 K=96K=96 (w/o) β=–.1 β=.0 β=.1 β=–.1 β=.0 β=.1 β=–.1 β=.0 β=.1 β=–.1 β=.0 β=.1 β=–.1 β=.0 β=.1 β=–.1 β=.0 β=.1 TinyCLIP-22M/32 29.8 / 42.5 30.0 28.4 30.2 29.6 27.6 30.2 30.1 29.9 30.3 42.2 40.0 42.4 42.7 39.3 42.4 42.5 42.5 42.4 TinyCLIP-45M/32 37.6 / 56.7 35.1 35.0 37.8 34.5 34.9 37.9 37.5 38.0 37.8 51.9 50.8 55.1 53.0 49.9 55.1 55.8 55.7 55.6 MobileCLIP-S0 44.2 / 57.9 42.6 44.4 44.3 42.7 44.5 44.2 43.9 43.5 43.9 48.8 56.2 57.4 50.6 56.0 57.6 57.5 57.8 57.9 MobileCLIP-S1 48.4 / 61.4 47.8 48.4 48.2 47.2 48.6 48.2 48.0 48.0 48.1 55.6 59.4 60.4 55.0 60.6 59.8 60.1 60.0 59.8 OpenCLIP-B/32 33.2 / 57.3 30.8 33.0 33.2 33.3 32.8 33.2 32.9 32.8 32.7 53.6 51.7 57.5 56.0 52.7 57.4 56.2 56.2 56.3 OpenCLIP-B/16 38.8 / 56.8 37.9 38.1 38.5 38.2 38.8 38.6 39.0 38.7 38.9 56.3 56.2 56.6 56.8 56.8 57.0 56.8 57.1 57.0 Table 5: Hyperparameter Sensitivity of HNM. We report mAP@20 (%) across varying margins β and candidate sizes K. The w/o column denotes the baseline on pedestrian and vehicle without HNM, illustrating the marginal gains of heuristic tuning. Loss Term Pedestrian (mAP@20 ↑ ) Vehicle (mAP@20 ↑ ) ITC SASS ISD T-22 T-45 M-S0 M-S1 B/16 B/32 T-22 T-45 M-S0 M-S1 B/16 B/32 – – – 25.1 32.7 41.0 42.6 32.9 27.4 38.0 45.2 52.2 57.2 48.0 46.5 ✓ – – 29.8 37.6 44.2 48.4 38.8 33.2 42.5 56.7 57.9 61.4 56.8 57.3 ✓ ✓ – 31.1 40.2 48.6 54.7 45.6 37.0 50.3 64.4 66.566.5 72.172.1 67.5 64.7 ✓ ✓ ✓ 32.132.1 41.541.5 49.749.7 55.055.0 46.946.9 39.039.0 51.351.3 64.664.6 67.967.9 73.073.0 68.768.7 64.864.8 Table 6: Ablation Study on Loss Functions. We report ITC baseline and incremental gains from adding SASS and ISD across CLIP-like models. SASS consistently improves retrieval performance, and ISD further boosts gains when combined with SASS. 4.2 Main Results Results on TBIR Benchmark. As summarized in Table 1, SAFT consistently outperforms both standard ITC and CUSA across all settings. The selected architectures span transformer and convolution-heavy variants, indicating that the gains of SAFT are not tied to a specific model design. Compared with ITC, SAFT improves average mAP@20 by 5.4 and 10.3 points on pedestrian and vehicle subsets, respectively. SAFT also surpasses CUSA by 5.6 points on pedestrian and 10.7 points on vehicle retrieval, demonstrating the advantage of direct cross-modal distribution supervision over unimodal proxy signals in semantically compressed domains. Results on General Benchmarks. As reported in Table 3, we report TBIR performance on Flickr30K and MS-COCO. On these general benchmarks, SAFT consistently improves over standard ITC fine-tuning across all reported metrics. It also outperforms CUSA in most settings. While CUSA yields broadly positive improvements over ITC, the gains are less stable across models and task domains. Extended Evaluations. Beyond the main results, we further validate SAFT’s general applicability on Fashion200K (Han et al. 2017) and its compositional reasoning ability on ARO (Yuksekgonul et al. 2023a). On Fashion200K, SAFT outperforms competing baselines with marked retrieval gains. On ARO, it improves reasoning on both Relation and Attribution subsets. Detailed results are provided in the Supplementary Material. Model Pedestrian Vehicle ITC ITC+ISS ITC ITC+ISS TinyCLIP-22M/32 29.8 27.2−2.627.2 [rgb]1,0,0-2.6 42.5 40.9−1.640.9 [rgb]1,0,0-1.6 TinyCLIP-45M/32 37.6 33.9−3.733.9 [rgb]1,0,0-3.7 56.7 53.6−3.153.6 [rgb]1,0,0-3.1 MobileCLIP-S0 44.2 44.0−0.244.0 [rgb]1,0,0-0.2 57.9 56.3−1.656.3 [rgb]1,0,0-1.6 MobileCLIP-S1 48.4 48.0−0.448.0 [rgb]1,0,0-0.4 61.4 59.8−1.659.8 [rgb]1,0,0-1.6 OpenCLIP-B/32 33.2 32.0−1.232.0 [rgb]1,0,0-1.2 57.3 57.30.057.3^0.0 OpenCLIP-B/16 38.8 38.3−0.538.3 [rgb]1,0,0-0.5 56.8 56.1−0.756.1 [rgb]1,0,0-0.7 Table 7: Ablation Study on Image Self-Supervision (ISS), evaluated by mAP@20. 4.3 Ablation Studies We conduct ablation experiments to analyze the contributions of our framework. To verify the general applicability of our findings, we evaluate these variants across mainstream models optimized with standard ITC objective, on pedestrian and vehicle retrieval sub-tasks. Foundation Model Generalization Gap. We first evaluate the zero-shot performance of pre-trained models. As shown in Table 4, models demonstrating strong performance on general-domain retrieval benchmarks can fail to maintain effectiveness on SecMM-TBIR, underscoring the necessity of benchmarking and fine-tuning in specific domains. Impact of Text Encoder Tuning. While joint cross-modal fine-tuning is conventionally assumed to increase alignment, we find it suboptimal across all models and sub-tasks. As reported in Table 2, tuning the text encoder consistently degrades performance. We attribute this to the distortion of pretrained textual space: in semantically compressed domains, the encoder overfits local patterns and degrades the generalization capability from large-scale pretraining. In contrast, exclusively optimizing the visual branch preserves pretrained capability while adapting to domain-specific visual details. Performance Degradation of Standard ISS. As discussed in Section 3.2, conventional ISS imposes an ill-posed constraint by separating semantically near-identical images. Table 7 confirms it causes consistent performance drops across nearly all models in vehicle and pedestrian domains. Ablation of Hard Negative Mining. We evaluate the HNM strategy by integrating it into our standard ITC baseline. Table 5 shows that HNM yields only marginal gains and can even degrade retrieval performance, as its rigid decision boundaries cannot reliably distinguish genuine hard negatives from false negatives under such dense semantic overlap, potentially leading to dataset-specific overfitting. Ablations of Loss Functions. As detailed in Table 6, we evaluate the contributions of the SASS and ISD components. SASS consistently improves retrieval performance over the baseline across all architectures, demonstrating the effectiveness of cross-modal soft-label supervision in alleviating false negatives. ISD further delivers additional gains, validating the complementary role of structural distillation in boosting domain-specific retrieval. 5 Conclusion In this paper, we propose a systematic approach for domain-specific TBIR that encompasses benchmarking model performance in specific scenarios and improving cross-modal alignment under severe semantic compression. Specifically, we design the DSMM-TBIR data engine to construct the SecMM-TBIR benchmark. Integrated with our SAFT framework, it establishes a robust training and evaluation pipeline. Extensive results demonstrate that our approach substantially improves domain-specific TBIR performance and further enhances general-domain representation capabilities. Future works may include extending this pipeline to more specialized domains, while exploring lightweight deployment on edge devices also remains an open research topic. References S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: 2nd item. D. Chen, Y. Wang, Y. Xie, S. Chen, W. Peng, M. Tang, M. Fang, C. P. Chen, P. Li, and W. Zhang (2025) Intra-modal consistency for image-text retrieval through soft-label distillation. Pattern Recognition, p. 112817. Cited by: §2.2. M. Cherti, R. Beis, B. He, A. Fang, C. Schuhmann, L. Schmidt, and J. Jitsev (2023) Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2818–2829. Cited by: §4.1. Z. Ding, C. Ding, Z. Shao, and D. Tao (2021) Semantically self-aligned network for text-to-image part-aware person re-identification. arXiv preprint arXiv:2107.12666. Cited by: §2.1. F. Faghri, P. K. A. Vasu, C. Koc, V. Shankar, A. T. Toshev, O. Tuzel, and H. Pouransari (2025) MobileCLIP2: improving multi-modal reinforced training. Transactions on Machine Learning Research. External Links: ISSN 2835-8856 Cited by: §4.1. Y. Gao, J. Liu, Z. Xu, T. Wu, E. Zhang, K. Li, J. Yang, W. Liu, and X. Sun (2024) Softclip: softer cross-modal alignment makes clip stronger. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 1860–1868. Cited by: §1. T. Gu, K. Yang, Z. Feng, X. Wang, Y. Zhang, D. Long, Y. Chen, W. Cai, and J. Deng (2025a) Breaking the modality barrier: universal embedding learning with multimodal llms. External Links: 2504.17432, Link Cited by: §1. T. Gu, K. Yang, K. Zhang, X. An, Z. Feng, Y. Zhang, W. Cai, J. Deng, and L. Bing (2025b) UniME-v2: mllm-as-a-judge for universal multimodal embedding learning. External Links: 2510.13515, Link Cited by: §3.3. X. Han, Z. Wu, P. X. Huang, X. Zhang, M. Zhu, Y. Li, Y. Zhao, and L. S. Davis (2017) Automatic spatially-aware fashion concept discovery. External Links: 1708.01311, Link Cited by: §4.2. H. Huang, Z. Nie, Z. Wang, and Z. Shang (2024) Cross-modal and uni-modal soft-label alignment for image-text retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 18298–18306. Cited by: §1. O. Idan, V. Vexler, G. Lederman, D. Sivov, A. C. Zada, and S. N. Komforti (2026) Few shots text to image retrieval: new benchmarking dataset and optimization methods. arXiv preprint arXiv:2603.25891. Cited by: §2.1. C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung, Z. Li, and T. Duerig (2021) Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Learning (ICML), p. 4904–4916. Cited by: §2.2. W. Jian, Y. Zhang, D. Liang, C. Xie, Y. He, D. Leng, and Y. Yin (2025a) Rzenembed: towards comprehensive multimodal retrieval. arXiv preprint arXiv:2510.27350. Cited by: §2.2. W. Jian, Y. Zhang, D. Liang, C. Xie, Y. He, D. Leng, and Y. Yin (2025b) RzenEmbed: towards comprehensive multimodal retrieval. External Links: 2510.27350, Link Cited by: §3.1. Q. Jiang, J. Huo, X. Chen, Y. Xiong, Z. Zeng, Y. Chen, T. Ren, J. Yu, and L. Zhang (2025) Detect anything via next point prediction. External Links: 2510.12798, Link Cited by: 2nd item. H. Ko and C. Park (2025) Bringing clip to the clinic: dynamic soft labels and negation-aware learning for medical analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 25897–25906. Cited by: §2.2. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei (2017) Visual genome: connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123, p. 32–73. External Links: Link, Document Cited by: §2.1. M. Li, Y. Zhang, D. Long, C. Keqin, S. Song, S. Bai, Z. Yang, P. Xie, A. Yang, D. Liu, J. Zhou, and J. Lin (2026) Qwen3-vl-embedding and qwen3-vl-reranker: a unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720. Cited by: §2.2, §3.1. S. Li, T. Xiao, H. Li, B. Zhou, and X. Wang (2017) Person search with natural language description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 1954–1962. Cited by: §2.1. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, p. 740–755. Cited by: §1. I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), Cited by: §4.1. M. Lu, E. Weinberger, C. Kim, and S. Lee (2026) CellCLIP-learning perturbation effects in cell painting via text-guided contrastive learning. Advances in Neural Information Processing Systems 38, p. 124505–124537. Cited by: §2.2. M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023) Dinov2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Cited by: §2.2. B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik (2017) Flickr30K entities: collecting region-to-phrase correspondences for richer image-to-sentence models. IJCV 123 (1), p. 74–93. Cited by: §1. Qwen Team (2026) Qwen3.5: towards native multimodal agents. External Links: Link Cited by: 2nd item. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, P. Askell, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, p. 8748–8763. Cited by: 2nd item. C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev (2022) LAION-5b: an open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, External Links: Link, 2210.08402 Cited by: §2.2. C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki (2021) LAION-400m: open dataset of clip-filtered 400 million image-text pairs. In Data Centric AI Workshop, NeurIPS 2021, External Links: Link, 2111.02114 Cited by: §2.2. W. Shin, J. Park, T. Woo, Y. Cho, K. Oh, and H. Song (2022) E-clip: large-scale vision-language representation learning in e-commerce. In Proceedings of the 31st ACM international conference on information & knowledge management, p. 3484–3494. Cited by: §2.2. O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, et al. (2025) Dinov3. arXiv preprint arXiv:2508.10104. Cited by: §3.1. R. S. Srinivasa, J. Cho, C. Yang, Y. M. Saidutta, C. Lee, Y. Shen, and H. Jin (2023) Cwcl: cross-modal transfer with continuously weighted contrastive loss. Advances in Neural Information Processing Systems 36, p. 78496–78513. Cited by: §2.2. M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, O. Hénaff, J. Harmsen, A. Steiner, and X. Zhai (2025) SigLIP 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. Cited by: §3.1. E. Vendrow, O. Pantazis, A. Shepard, G. Brostow, K. E. Jones, O. Mac Aodha, S. Beery, and G. Van Horn (2024) INQUIRE: a natural world text-to-image retrieval benchmark. NeurIPS. Cited by: §2.1. Z. Wang, Z. Wu, D. Agarwal, and J. Sun (2022) Medclip: contrastive learning from unpaired medical images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 3876–3887. Cited by: §2.2. K. Wu, H. Peng, Z. Zhou, B. Xiao, M. Liu, L. Yuan, H. Xuan, M. Valenzuela, X. (. Chen, X. Wang, H. Chao, and H. Hu (2023) TinyCLIP: clip distillation via affinity mimicking and weight inheritance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 21970–21980. Cited by: §4.1. M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou (2023a) When and why vision-language models behave like bags-of-words, and what to do about it?. External Links: 2210.01936, Link Cited by: §4.2. M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou (2023b) When and why vision-language models behave like bags-of-words, and what to do about it?. In International Conference on Learning Representations, Cited by: §1. A. Zhu, Z. Wang, Y. Li, X. Wan, J. Jin, T. Wang, F. Hu, and G. Hua (2021) DSSL: deep surroundings-person separation learning for text-based person retrieval. In Proceedings of the 29th ACM International Conference on Multimedia, p. 1201–1209. Cited by: §2.1. J. Zuo, J. Hong, F. Zhang, C. Yu, H. Zhou, C. Gao, N. Sang, and J. Wang (2024) PLIP: language-image pre-training for person representation learning. External Links: 2305.08386, Link Cited by: §2.1. Appendix A Benchmark Details We construct SecMM-TBIR with our proposed DSMM-TBIR pipeline, with the Adaptive Data Pool Curation stage serving as the core component of our data preparation workflow. In this stage, we build the image and caption pools by aggregating multi-source data drawn from three complementary channels: self-collected domain-specific datasets, targeted web crawling of in-domain visual-textual pairs, and a curated collection of publicly available benchmarks, notably encompassing CUHK-PEDES, RSTPReid, ICFG-PEDES, SYNTH-PEDES, and CUHK-CompCars. Textual Branch. We first analyze the text distribution of the collected data to derive corresponding statistical information, which are incorporated into prompts as detailed in Table 11. The resulting prompts are then fed into Qwen3 to generate comprehensive textual queries. Visual Branch. Images are first encoded by DINOv3 to obtain visual embeddings. CGDS then applies K-means clustering to the pedestrian and vehicle image pools separately in the embedding space, with K~=100 K=100 in each domain, followed by randomly sampling 100–1000 images from each cluster. Qwen3-VL subsequently filters the sampled images to retain only those relevant to the target object category. Figure 6: More SecMM-TBIR Samples. Multi-Expert Collaborative Filtering. A suite of universal multi-modal embedding models serve as expert retrievers, including Qwen3-VL-Embedding, Jina-v4, RZen-v2, and SigLIP. Each expert retrieves its top-K^=1000 K=1000 candidates for a given query. We then construct the final set of potential matching pairs by taking the union of all retrieved results, which proceeds to annotation. As shown in Figure 6, additional visualization examples of our SecMM-TBIR are provided for easier intuitive understanding, where each query is associated with multiple relevant visual candidates. Figure 7: Mini-batch Training Strategies. Hard Negative Mining (HNM) enforces a rigid decision boundary, whereas SASS utilizes soft cross-modal distributions to preserve continuous semantic similarities. Hyperparameter Value TinyCLIP, MobileCLIP, OpenCLIP Input resolution 2242224^2, 2562256^2, 2242224^2 Resize Crop Scale [0.8, 1.0] Resize Crop Ratio [0.2, 2.0] Training Iterations 25k Batch Size 128 Optimizer AdamW Min LR 5e-7 Max LR 5e-6 Warmup Ratio 0.1 LR Decay Schedule Cosine Annealing Weight Decay Rate 0.05 SASS Loss Weight α 1 ISD Loss Weight β 0.75 SASS Teacher UniME-V2-7B ISD Teacher UniME-V2-7B Table 8: Hyperparameter Configurations for the Fine-Tuning Framework. Appendix B Training Hyperparameters For completeness and reproducibility, we explicitly provide full supplementary implementation details on the hyperparameter configurations used in our main experiments. The corresponding hyperparameter settings for different experiments are fully listed in Table 8. We adopt a relatively small batch size and a conservative learning rate schedule to ensure stable optimization under severe semantic compression in domain-specific TBIR tasks. The scale and aspect ratio ranges for random resized cropping are adjusted to fit the typical visual profiles of pedestrian and vehicle targets. Concretely, the scale range is empirically fixed at [0.8,1.0][0.8,1.0] to effectively retain key foreground information, and the aspect ratio range is set to [0.2,2.0][0.2,2.0] to handle the varied shapes and orientations of pedestrians and vehicles in the real-world. Model Fashion200k R@@1 R@@5 R@@10 TinyCLIP-22M/32 35.7 65.0 76.3 + ITC 39.4 66.8 77.0 + CUSA 39.2 68.4 77.2 + SAFT (ours) 39.7 68.6 78.0 TinyCLIP-45M/32 42.2 71.0 80.2 + ITC 44.3 71.5 80.7 + CUSA 44.6 71.3 81.8 + SAFT (ours) 45.7 73.7 82.8 MobileCLIP-S0 45.1 74.8 84.6 + ITC 52.7 83.9 91.9 + CUSA 52.1 83.2 91.4 + SAFT (ours) 55.3 84.3 92.3 MobileCLIP-S1 53.7 81.7 89.4 + ITC 60.8 88.6 94.7 + CUSA 59.8 88.1 94.6 + SAFT (ours) 61.5 88.7 95.3 OpenCLIP-B/32 38.3 67.5 77.3 + ITC 42.4 69.1 79.4 + CUSA 42.7 71.1 80.9 + SAFT (ours) 42.8 71.2 81.5 OpenCLIP-B/16 40.2 69.1 78.2 + ITC 44.7 71.1 81.8 + CUSA 45.2 73.8 82.2 + SAFT (ours) 46.3 74.2 82.7 Table 9: Results on Fashion200K. Appendix C More Detailed Results Domain Generalization on Fashion200K. To further evaluate the domain generalization capability of our framework, we conduct additional experiments on the Fashion200K dataset, extending the evaluation scope from surveillance domain to commercial fashion scenarios. To adapt the benchmark to practical text-based image retrieval settings, we augment the original dataset by generating item descriptions using the Qwen3-VL together with Qwen-3.5, from which 2,000 of the resulting samples are selected to form the test set, while the remaining samples are used to form the training set. Quantitative results are summarized in Table 9. Our method consistently outperforms all baselines by a significant margin in this completely different domain. This validation confirms the broad applicability of our method beyond security-focused surveillance tasks and demonstrates its effectiveness and generalizability for domain-specific TBIR. Compositional Reasoning Capability. We further validate the compositional reasoning capability of our method on the ARO benchmark. The model is trained using the same training set described in Experiment Section. As reported in Table 10, SAFT improves compositional reasoning performance across most evaluated models on both the Relation and Attribution subsets. While the gains vary across different models, the overall trend suggests that SAFT enhances the model’s ability to capture fine-grained semantic structures and complex multi-modal compositional relations. Model ARO (Acc. ↑ ) Relation Attribution TinyCLIP-22M/32 48.2 56.0 + ITC 51.7 60.6 + CUSA 50.8 57.2 + SAFT (ours) 51.3 60.0 TinyCLIP-45M/32 49.8 59.1 + ITC 51.2 61.2 + CUSA 51.8 61.6 + SAFT (ours) 51.9 61.9 MobileCLIP-S0 49.5 62.6 + ITC 51.1 63.8 + CUSA 50.9 63.7 + SAFT (ours) 53.5 65.7 MobileCLIP-S1 50.2 65.7 + ITC 50.6 65.0 + CUSA 50.6 65.4 + SAFT (ours) 53.0 67.2 OpenCLIP-B/32 49.8 58.5 + ITC 50.9 60.8 + CUSA 50.6 60.0 + SAFT (ours) 51.0 60.8 OpenCLIP-B/16 46.6 56.8 + ITC 49.9 59.8 + CUSA 50.6 60.2 + SAFT (ours) 51.2 61.0 Table 10: Evaluation on ARO Benchmark. Task Prompt Category Prompt Pedestrian System You are a data engineer responsible for constructing a text-image retrieval dataset. Based on the following requirements, please generate a batch of English text retrieval queries centered around "pedestrians/people" for security camera scenarios. [Task Scenario] • All outputs must be English query sentences. • The content of the queries must center around "people/pedestrians." • The tone of the queries should match the realistic search behavior of security system users. [Template Structure] All query sentences must adhere to the following structure: [person] + [specific action and/or descriptor] + (optional) second or third complementary description [Stylistic Requirements] • Sentence Length: Approximately 5-15 English words (MS-COCO caption style, with natural variations in query length). • Tone & Style: Natural, concise, and aligned with real-world surveillance search queries, avoiding overly literary or descriptive language. • Diversity: Cover a wide range of human attributes, including actions, appearance, clothing, carried objects, and characteristics. • Safety Guardrails: Generation of any content related to danger, violence, pornography, weapons, or illegal activities is strictly prohibited. [Semantic Background] Common nouns, verbs, adjectives, and adverbs related to people have been statistically extracted from relevant datasets. Please randomly select and combine elements from these semantic scopes to ensure high diversity in the generated queries. [Output Format] • Output multiple English queries. • Each query should be on a new line, without any numbering or bullet points. • Do not output any explanations or extra commentary; output the queries only. [Output Examples] A little boy sitting alone on a motorcycle. A man rides a motorcycle down an empty street next to houses. The woman in the kitchen is holding a huge pan. A man standing in a kitchen while closing a cupboard door. Man in motorcycle leathers standing in front of a group of bikes. Two people riding on a moped with a bus in the next lane. A girl in bikini standing with surfboard on the beach. A man holds a hot dog while several people walk in the background. A man with a racket on a tennis court. A woman in a pink jacket and printed pants. A man with a red shirt is holding a cell phone up to his ear. User Please randomly select from the following keywords and generate [part_num] pedestrian-related descriptions (try to use most of the words): Nouns: [keywords_noun_part_str] Verbs: [keywords_verb_part_str] Adjectives: [keywords_adj_part_str] Please start generating queries: Table 11: Prompt Engineering in Distribution-Aware Prompting (DAP).