Paper deep dive
Generative Universal Multimodal Retrieval with Dual-role Identifiers
Kaipeng Li, Haitao Yu, Xuanchen Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/14/2026, 5:43:28 AM
Summary
The paper introduces DrIG, a novel generative framework for universal multimodal retrieval (GUMR) that utilizes dual-role identifiers. Each candidate is assigned a single residual-quantized identifier that serves two roles: a sequential role for autoregressive decoding and a set-based role for prefix-independent relevance estimation. This approach addresses challenges in generative information retrieval (GIR) such as prefix-level errors, local optima, and the gap between discrete identifier efficiency and dense-vector accuracy. DrIG incorporates hybrid reranking and is evaluated on benchmarks like M-BEIR, Flickr30K, and MSCOCO, demonstrating superior performance over state-of-the-art baselines.
Entities (13)
Relation Signals (11)
DrIG â uses â Dual-role Identifiers
confidence 95% · we propose DrIG, a novel Generative framework for universal multimodal retrieval featuring Dual-role Identifiers
Xuanchen Zhou â affiliatedwith â University of Tsukuba
confidence 90% · Affiliation: College of Knowledge and Library Sciences, University of Tsukuba
Kaipeng Li â affiliatedwith â University of Tsukuba
confidence 90% · Affiliation: Independent Researcher , Tsukuba , Ibaraki , Japan... Institute of Library, Information, and Media Science, University of Tsukuba
Haitao Yu â affiliatedwith â University of Tsukuba
confidence 90% · Affiliation: Institute of Library, Information, and Media Science, University of Tsukuba
DrIG â evaluatedon â M-BEIR
confidence 90% · Extensive experiments on the M-BEIR benchmark
Dual-role Identifiers â hasrole â Set-based Role
confidence 90% · In its set-based role, the same tokens are reinterpreted as an unordered set to provide a prefix-independent relevance prior
Dual-role Identifiers â hasrole â Sequential Role
confidence 90% · In its sequential role, the identifier is decoded autoregressively
DrIG â addresseschallenge â Local Optima
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generative information retrieval (GIR) has emerged as a compelling alternative to the conventional index-retrieve-then-rank retrieval pipeline by training a generator to produce the identifiers of relevant items directly. Despite its promise, a number of open challenges still remain. First, constrained left-to-right decoding is vulnerable to prefix-level errors and local optima. Second, most prior GIR research remains largely unimodal, leaving instruction-aware retrieval across text, image, and mixed image-text items underexplored. Third, although discrete identifier-based GIR offers higher efficiency, its retrieval accuracy still lags behind that of the cutting-edge dense-vector-based retrieval methods. Motivated by these challenges, we propose DrIG, a novel Generative framework for universal multimodal retrieval featuring Dual-role Identifiers, which supports diverse retrieval tasks across multiple modalities and domains. Each candidate is assigned a single residual-quantized identifier that serves two complementary roles. In its sequential role, the identifier is decoded autoregressively, where the first token explicitly models modality and the remaining tokens capture progressively finer semantics. In its set-based role, the same tokens are reinterpreted as an unordered set to provide a prefix-independent relevance prior, which guides constrained beam search and alleviates local-optimum errors. Extensive experiments on the M-BEIR benchmark and the text-to-image evaluation datasets show that:(1)DrIG consistently outperforms state-of-the-art generative multimodal baselines across diverse tasks, while hybrid reranking achieves a favorable efficiency-effectiveness trade-off against strong dense retrievers. (2)Ablation and scaling analyses reveal how the base LMM, beam size, reranking depth, and fusion strategy affect retrieval performance, providing practical guidance for system design.
Tags
Links
- Source: https://arxiv.org/abs/2608.12987v1
- Canonical: https://arxiv.org/abs/2608.12987v1
Trouble viewing inline? Open PDF directly â
Full Text
133,405 characters extracted from source content.
Expand or collapse full text
Generative Universal Multimodal Retrieval with Dual-role Identifiers Kaipeng Li OrcID: 0009-0004-4406-2956 Affiliation: Independent Researcher , Tsukuba , Ibaraki , Japan email: keeey.k@gmail.com , Haitao Yu Note: Corresponding author. OrcID: 0000-0002-1569-8507 Affiliation: Institute of Library, Information and Media Science, University of Tsukuba , Tsukuba , Ibaraki , Japan email: yuhaitao@slis.tsukuba.ac.jp and Xuanchen Zhou OrcID: 0009-0002-1659-2592 Affiliation: College of Knowledge and Library Sciences, University of Tsukuba , Tsukuba , Japan email: s2313451@u.tsukuba.ac.jp © none Abstract. Generative information retrieval (GIR) has emerged as a compelling alternative to the conventional index-retrieve-then-rank retrieval pipeline by training a generator to produce the identifiers of relevant items directly. Despite its promise, a number of open challenges still remain. First, constrained left-to-right decoding is vulnerable to prefix-level errors and local optima. Second, most prior GIR research remains largely unimodal, leaving instruction-aware retrieval across text, image, and mixed image-text items underexplored. Third, although discrete identifierâbased GIR offers higher efficiency, its retrieval accuracy still lags behind that of the cutting-edge dense-vector-based retrieval methods. Motivated by these challenges, we propose DrIG, a novel Generative framework for universal multimodal retrieval featuring Dual-role Identifiers, which supports diverse retrieval tasks across multiple modalities and domains. Each candidate is assigned a single residual-quantized identifier that serves two complementary roles. In its sequential role, the identifier is decoded autoregressively, where the first token explicitly models modality and the remaining tokens capture progressively finer semantics. In its set-based role, the same tokens are reinterpreted as an unordered set to provide a prefix-independent relevance prior, which guides constrained beam search and alleviates local-optimum errors. To further compensate for the information loss introduced by discretization, we incorporate a hybrid retrieval strategy that reranks the top generative candidates based on dense-vector-based similarity. Extensive experiments11 1 The code will be released. on the M-BEIR benchmark and the text-to-image evaluation datasets (Flickr30K and MSCOCO) show that: (1) DrIG consistently outperforms state-of-the-art generative multimodal retrieval baselines across diverse retrieval tasks. The hybrid reranking successfully combines the strengths of two paradigms and yields a favorable efficiency-effectiveness trade-off relative to strong dense baselines. (2) Comprehensive ablation and scaling analyses show that the base LMM, beam size, reranking depth, and fusion strategy significantly affect retrieval performance, providing practical guidance for future generative multimodal retrieval systems. 1. Introduction Nowadays, information retrieval (IR) systems play a crucial role in bridging the ever-expanding World Wide Web (W) with diverse user information needs, supporting activities that range from fact finding and media search to decision making and knowledge access. Classical IR methods (including lexical matching, learning-to-rank, and modern dense retrieval) have substantially advanced retrieval effectiveness over the past decades (57; 54; 2; 21; 44). Nevertheless, most of the prior methods boil down to the multi-stage index-retrieve-then-rank paradigm (44), suffering from objective inconsistency across stages. Motivated by the recent success of large language models (LLMs), generative information retrieval (GIR) has emerged as a promising alternative (61; 1; 31). Instead of exhaustively scoring all candidates, GIR first maps each candidate to a discrete identifier and then trains a generator to produce the identifiers of relevant items directly. This formulation shifts retrieval from similarity search in a large candidate space to conditional generation in a compact discrete space. Towards effective GIR, many methods have been proposed. Based on the modalities involved, we can categorize relevant studies into three groups: (1) generative unimodal retrieval (GUR), which focuses on retrieval within a single modality, such as (61; 42; 65; 48; 35; 71; 34; 58; 1; 8; 7; 55; 47; 78; 64; 59; 76; 43; 68; 60; 22); (2) generative cross-modal retrieval (GCMR), which addresses retrieval across two distinct modalities, such as (27; 32; 15; 49); and (3) generative universal multimodal retrieval (GUMR), which enables retrieval across multiple modalities within a unified framework, such as (66; 23; 36). A closer look at the previous studies on GIR shows that most prior studies focus on GUR, while GCMR and GUMR remain relatively underexplored. In particular, multimodal retrieval has rapidly evolved from conventional image-text matching to more general universal multimodal retrieval, where a query may contain text, images, or both, and the target can likewise belong to different modalities and domains (3; 66). This setting is particularly appealing because it provides a unified formulation for a wide range of tasks, including text-to-image retrieval, image-to-text retrieval, multimodal question answering retrieval, and multimodal evidence retrieval. Recent retrieval methods based on large multimodal models (LMMs) have demonstrated strong representation capacity in such settings (40; 36). Yet these methods still inherit the core limitations of dense retrieval: they require a candidate pool to be represented and searched in continuous space, and their computational cost scales with the retrieval corpus. Applying GIR to universal multimodal retrieval is therefore attractive, but it introduces a set of nontrivial challenges. First, most generative retrievers decode identifiers from left to right with constrained beam search. This makes the retrieval result highly sensitive to early prefix decisions: once the prefix of a relevant candidate is pruned, the candidate becomes irrecoverable, even if it is globally relevant. Second, multimodal retrieval requires the model to represent modality distinctions and fine-grained semantics simultaneously. A naive identifier design may fail to encode both aspects effectively. Third, discrete identifiers inevitably compress continuous embeddings, which improves efficiency but also discards information that is often crucial for fine-grained ranking. As a result, generative multimodal retrievers typically still lag behind strong dense baselines in effectiveness. In this work, we address these challenges with a simple but powerful idea: the same discrete identifier can serve two complementary roles. On the one hand, it acts as an ordered token sequence that can be generated autoregressively. On the other hand, the same tokens can be reinterpreted as an unordered set that supports order-invariant relevance estimation. Building on this idea, we propose DrIG, a generative framework for universal multimodal retrieval with dual-role identifiers. DrIG first learns instruction-aware multimodal embeddings using an LMM and contrastive fine-tuning. It then converts candidate embeddings into residual-quantized identifiers whose first token explicitly captures modality while later tokens encode progressively finer semantics. During inference, sequential decoding scores and order-invariant global relevance priors are combined to guide constrained beam search, reducing the risk that relevant candidates are discarded due to locally suboptimal prefixes. To further narrow the effectiveness gap between generative and dense retrieval, DrIG also incorporates a hybrid retrieval strategy. The generative retriever first produces a compact top-k candidate list efficiently; the top-k results are then reranked by continuous similarities in the original embedding space. This design preserves the scalability advantage of generative retrieval while recovering fine-grained distinctions that may be lost during quantization. In addition, we introduce query augmentation through queryâtarget interpolation and a discriminative ranking objective for decoder training, which together improve robustness and ranking consistency. To summarize, the main contributions of this paper can be listed as follows: âą We propose DrIG, a novel generative framework for universal multimodal retrieval that assigns each candidate a single residual-quantized identifier and reuses it in two complementary roles: a sequential role for autoregressive generation and a set-based role for prefix-independent relevance estimation. âą We introduce dual-guided constrained decoding that combines prefix-valid constrained beam search with an order-invariant global relevance prior, thereby mitigating the local-optimum problem of left-to-right identifier generation. We further integrate DrIG with dense top-k reranking and systematically analyze the resulting effectivenessâefficiency trade-off. âą We provide ablations and diagnostic analyses on codebook design, beam size, prior weight, reranking depth, decoder backbone, and key training objectives, providing practical insights into the design of generative multimodal retrieval systems. The remainder of this paper is organized as follows. Section 2 reviews prior work on multimodal retrieval and recent advances in GIR. Section 3 introduces the problem formulation of GUMR. Section 4 presents the proposed DrIG framework, including instruction-aware multimodal representation learning, dual-role identifier construction, dual-guided constrained decoding, decoder training objectives, and hybrid reranking. Section 5 describes the experimental setup and reports the main results on M-BEIR, together with additional text-to-image retrieval experiments on Flickr30K and MSCOCO. We further provide component ablations, visualization analyses, effectivenessâefficiency comparisons, hyperparameter studies, and qualitative case studies. Finally, Section 6 concludes the paper and discusses future research directions. 2. Related Work In this section, we first review non-generative multimodal retrieval methods, including shared-embedding approaches, cross-modal interaction models, and recent LMM-based methods. Due to space constraints, we refer the reader to the work (4) for a detailed overview. Then we describe recent advances in GIR and position our work with respect to closely related methods. 2.1. Non-generative Multimodal Retrieval Early multimodal retrieval studies mainly focused on imageâtext matching. A large body of work learns shared embedding spaces where images and texts can be compared directly, while another line of research introduces cross-modal interaction mechanisms to improve matching quality. Representative approaches include convolutional architectures, metric-learning objectives, cross-attention models, and Transformer-based matching networks (13; 26; 45; 29). These methods are often instantiated in one of three architectural families. Two-tower models encode different modalities separately and compare them with a lightweight similarity function, making them scalable for large candidate pools (50; 6; 77). Fusion-based or two-leg models introduce cross-modal interaction to improve matching quality, although usually at a higher computational cost (56; 70). One-tower models instead attempt to unify multimodal encoding within a single backbone (17; 62). More recently, multimodal retrieval has been extended from fixed imageâtext matching to universal multimodal retrieval, where queries and candidates may be text, images, or imageâtext pairs, and the retrieval intent is specified by natural-language instructions. UniIR/M-BEIR (66) provides a benchmark covering diverse retrieval tasks, modalities, and domains. Meanwhile, LMMs have been adapted for retrieval by leveraging their stronger language understanding and multimodal reasoning capacity. Examples include instruction-aware retrieval assistants and universal multimodal embedding models (66; 20; 36; 40). VLM2Vec (20) converts visionâlanguage models into general-purpose multimodal embedders through contrastive training on massive multimodal embedding tasks. M-Embed (36) fine-tunes multimodal LLMs as universal multimodal retrievers and further shows that MLLM-based rerankers can improve retrieval results for complex multimodal queries. LamRA (40) adapts LMMs through language-only pretraining and multimodal instruction tuning, enabling both retrieval and reranking across heterogeneous multimodal tasks. These non-generative embedding-based retrieval methods provide powerful representations. However, they still fundamentally rely on dense candidate scoring, approximate nearest-neighbor search, or expensive ranking over candidate lists. Our work complements this line of research by using LMMs to obtain strong multimodal representations, but it departs from continuous search at inference time through generative identifier decoding. 2.2. Generative Information Retrieval In this section, we introduce representative studies on GIR and then clarify how DrIG differs from closely related methods. We refer the reader to the work (31) for a detailed survey on GIR. 2.2.1. Generative Unimodal Retrieval Generative unimodal retrieval has been most extensively studied in text and document retrieval, where the goal is to retrieve textual documents by generating their identifiers. Representative early methods, such as DSI (61), NCI (64), and autoregressive search engines (1), demonstrate the feasibility of using sequence models as differentiable or generative indexes. Subsequent studies (61; 1; 64; 42; 8; 47; 78; 34; 58; 65; 55; 48; 35; 71; 72; 60; 7; 22; 76; 59; 43; 68; 73) have improved this paradigm from several perspectives, including identifier design, training objectives, decoding strategies, scalability, and adaptation to dynamic corpora. A central challenge in generative retrieval is how to construct identifiers that are both easy for a sequence model to generate and sufficiently discriminative for retrieval. Existing work has explored semantic strings, learned document tokens, interpretable identifiers, multi-view identifiers, and tree-structured identifiers [34, 56, 59, 66]. Recent studies on document identifier learning further emphasize that effective identifiers should be descriptive, discriminative, and aligned with relevance signals (34; 58; 65; 55). Another line of work focuses on improving the training and decoding objectives of generative retrievers. LTRGR (35) introduces a learning-to-rank objective to reduce the mismatch between token-level generation and document-level ranking. DiffusionRet (48) combines diffusion modeling with constrained decoding. RIPOR (71) proposes prefix-oriented ranking optimization and relevance-based identifier initialization, showing that prefix-level supervision is important for scalable generative retrieval. More recent studies investigate direct document relevance optimization (43), the limitations of constrained autoregressive decoding (68), and the behavior of generative retrievers under dynamic corpora (7; 22; 76). The aforementioned studies reveal a common limitation of purely left-to-right identifier generation: a relevant item can be irreversibly discarded once its prefix is pruned during constrained decoding. To cope with this challenge, TSGen (74) replaces purely sequential document identifiers with term sets and uses permutation-invariant decoding to reduce false-pruning errors caused by left-to-right generation. PAG (72) creates the set-based DocIDs under the bag-of-words assumption and sequential DocIDs based on the relevance-based document representations to support simultaneous and autoregressive decodings, respectively. Inspired these two studies, DrIG designs identifiers with two complementary roles: an ordered sequence to facilitate Trie-constrained autoregressive generation, and a prefix-independent set-based representation for global relevance guidance. Furthermore, DrIG targets a more heterogeneous setting. Rather than designing identifiers only for unimodal text retrieval, DrIG constructs modality-aware residual-quantized identifiers for text, image, and imageâtext candidates. 2.2.2. Generative Cross-modal Retrieval GCMR extends GIR beyond text-only retrieval, most commonly to text-to-image retrieval. Compared with unimodal document retrieval, this setting introduces additional challenges because the model must bridge modality gaps while still generating valid candidate identifiers. Recent methods have explored different ways to represent visual candidates as discrete identifiers and train multimodal generators to retrieve them (33; 75; 49; 32; 10; 30; 27). GRACE (33) assigns identifier strings to images and trains a multimodal language model to memorize and retrieve images by generating their identifiers. IRGen (75) recasts image retrieval as a sequence-to-sequence generative modeling problem. TIGeR (49) explores the connection between text-to-image generation and retrieval, using large multimodal models to unify the two tasks. AVG (32) formulates text-to-image retrieval as autoregressive token-to-visual-token generation. CART (10) proposes a coarse-to-fine generative cross-modal retrieval framework that combines clustering and residual vector quantization to construct multimodal identifiers. ComGTIR (30) further introduces dual identifiers and hybrid retrieval strategies for generative text-image retrieval, using a sequential identifier and an order-invariant identifier to mitigate local-optimum errors during constrained beam search. These methods demonstrate that generative retrieval can be successfully extended from text-only retrieval to cross-modal scenarios. However, most existing GCMR methods are designed for task-specific retrieval, especially text-to-image retrieval, where the query modality and target modality are predefined. DrIG differs from this line of work in both problem setting and identifier design. First, DrIG targets universal multimodal retrieval, where queries and candidates may be text, image, or imageâtext pairs, and the target modality must be inferred from the instruction. Second, DrIG does not maintain separate identifier spaces for sequential generation and global matching. Instead, it assigns each candidate a single residual-quantized identifier and reuses the same code tokens as both an ordered sequence and a prefix-independent set-based relevance representation. 2.2.3. Generative Universal Multimodal Retrieval GUMR is an emerging direction that aims to support retrieval across heterogeneous query and candidate modalities within a unified generative framework. This setting is more challenging than conventional text-to-image retrieval because the model must jointly infer semantic relevance, target modality, and task intent from the input instruction. Wei et al. (66) establish an important benchmark for universal multimodal retrieval by assembling diverse datasets across multiple tasks, modalities, and domains. However, they mainly focus on dense vector-based retrieval rather than generative identifier decoding. The closest prior work to DrIG is GENIUS (23), which proposes a generative framework for universal multimodal search. GENIUS introduces modality-decoupled semantic quantization to transform multimodal candidates into discrete identifiers and uses query augmentation to improve generalization across diverse query forms. Despite its effectiveness, GENIUS still primarily relies on sequential identifier generation. As a result, it remains vulnerable to prefix-level pruning errors during constrained beam search. Once the prefix of a relevant candidate is discarded, the candidate cannot be recovered in subsequent decoding steps. DrIG addresses this limitation by introducing dual-role identifiers for generative universal multimodal retrieval. This design preserves the validity and efficiency of constrained generative retrieval while reducing the risk of local-optimum errors caused by early prefix pruning. 3. Preliminaries In universal multimodal retrieval (66), a query is formulated as a pair q^=(qcon,qinst) q=(q_con,q_inst), where qconq_con denotes the query content and qinstq_inst is a natural-language instruction. The query content may be text qtq_t, image qiq_i, or a multimodal pair (qi,qt)(q_i,q_t). The instruction specifies the retrieval objective, including the desired target modality and the application domain. A candidate item c c can likewise be text ctc_t, image cic_i, or an imageâtext pair (ci,ct)(c_i,c_t). Under this formulation, a single retrieval model can support diverse tasks such as text-to-image, image-to-text, multimodal-to-text, and multimodal-to-multimodal retrieval in a unified manner. In generative universal multimodal retrieval (GUMR), a retrieval system typically contains two key components: a converter and a generative retriever. The converter maps each candidate to a discrete identifier =[m1,âŠ,mL]m=[m_1,âŠ,m_L], where mim_i is the i-th token and L is the identifier length. The generative retriever is then trained to generate the identifiers of relevant candidates conditioned on the input query. Different frameworks for GUMR can be constructed by varying the converter design, adopting different token selection strategies during the autoregressive generation process, and devising alternative loss functions for optimization. 4. The Proposed Framework Figure 1. An overview of the proposed DrIG framework. Figure 1 provides an overview of DrIG, which includes four core stages. (a) LMM-oriented representation for encoding queries and candidates into an instruction-aware shared embedding space. (b) Constructing dual-role identifiers for converting candidate embeddings into residual-quantized codes that support both sequential generation and set-based relevance estimation. (c) The training process for learning the autoregressive decoder with generative and discriminative objectives. (d) The inference process for retrieving candidates via dual-guided Trie-constrained beam search and optional dense reranking. We elaborate these stages in Sections 4.1â4.3. 4.1. Initial Representation in a Shared Embedding Space To cope with the diverse retrieval tasks in universal multimodal retrieval (as shown in Table 1), we first employ a large multimodal model (LMM) to obtain the initial dense representations of retrieval queries and candidate items by encoding them into a shared embedding space. Then a two-stage fine-tuning strategy is deployed to further improve the capability of the selected LMM for embedding multimodal items in retrieval tasks. The rationale behind this design is twofold. First, benefiting from interleaved visionâlanguage training, LMMs are able to represent multimodal information according to their meanings with prompt. Second, compared with methods that rely on separate encoders for different modalities (e.g., CLIP), LMMs possess stronger language understanding and reasoning capabilities. Consequently, this design not only facilitates similarity estimation between query content and candidate items across different modalities, but also enables accurate comprehension of the search instruction. 4.1.1. LMM-oriented Representation Inspired by the explicit one-word limitation (EOL) strategy proposed in (18; 19), we explicitly instruct the adopted LMM to represent multimodal inputs using one word. Following the work by Liu et al. (40), we append a task-specific summarization prompt to the input as follows: (1) for image-only inputs, the prompt is set to: <image> Summarize the above image in one word: <emb>; (2) for text-only inputs, the prompt is set to: <text> Summarize the above sentence in one word: <emb>; and (3) for a mixed image-text input, the prompt is set to: <image><text> Summarize the above image and sentence in one word: <emb>. In these prompts, <image> and <text> denote placeholders for the input image and sentence, respectively. Finally, we extract the last hidden state immediately preceding the <emb> token as the embedding vector of the input. 4.1.2. Fine-tuning for Improved Representation As demonstrated in prior studies (40; 24; 36), most pretrained LMMs are primarily optimized for generative tasks, such as next-token prediction. In contrast, retrieval tasks require distinguishing relevant items from non-relevant ones. Consequently, directly employing an LMM to embed multimodal items for retrieval often leads to inferior performance. After an in-depth comparison of three closely related studies (40; 24; 36), we finally follow the two-stage fine-tuning strategy proposed in (40) to further improve the capability of the selected LMM for embedding multimodal items in retrieval tasks. In the first stage, the LMM is adapted for text-to-text retrieval using a natural language inference dataset (12). In the second stage, we further fine-tune the model on a range of multimodal retrieval tasks based on the M-BEIR dataset (details are provided in Section 5.1.1). Across both stages, the training objective is contrastive learning with the InfoNCE loss (46). Due to space limitations, please refer to Section 3.2 of (40) for detailed descriptions of the training procedure. 4.2. Constructing Dual-role Identifiers The dense representations of queries and candidate items are continuous and high-dimensional, making them incompatible with autoregressive generation. To bridge this gap, prior generative information retrieval methods convert such representations into discrete identifiers. However, existing approaches typically treat identifiers purely as ordered token sequences, which introduces a critical limitation: the left-to-right generation process may bias the model toward local optima and restrict its ability to capture global similarity. To address this issue, we propose to construct dual-role identifiers, which enable complementary interpretations of discrete tokens. Specifically, each identifier simultaneously serves (i) as a sequential representation for autoregressive generation, preserving structural and semantic dependencies, and (i) as a set-based representation for order-invariant matching, mitigating the limitations of sequential decoding. Based on this design, we construct discrete identifiers for both queries and candidate items through a three-step process, as detailed in the following subsections. 4.2.1. Sequential Role: Identifier Construction for Autoregressive Generation We denote a dense representation vector in the shared embedding space described above as âz^*, and its corresponding identifier as :=(m1,âŠ,mL)m:=(m_1,âŠ,m_L). When necessary, we use subscripts such as q^âz^*_ q and c^âz^*_ c to distinguish the embeddings of a search query and a candidate item, respectively. Inspired by Kim et al. (23), the identifier m is particularly designed such that the first token m1m_1 captures modality distinctions, whereas the remaining tokens (m2,âŠ,mL)(m_2,âŠ,m_L) progressively encode increasingly fine-grained semantic information. Under the sequential role for autoregressive generation, we adopt a two-stage design for identifier construction. In the first stage, we introduce a lightweight residual MLP module (the Adapter block in Figure 1(b)) to refine the initial dense representation âz^* and facilitate the generation of an effective first token m1m_1. Specifically, this module is designed as: (1) =â+MLPâĄ(â),z=z^*+MLP(z^*), where MLPâĄ(â )MLP(·) denotes a learnable multi-layer perceptron, and z denotes the refined representation vector. In the second stage, we deploy the residual quantization (RQ) (25; 52) to convert the refined representation z into a sequential identifier. Let â°=1,âŠ,i,âŠ,LE=\E_1,âŠ,E_i,âŠ,E_L\ denote the set of codebooks, and i=kiââdâŁk=1,âŠ,|i|E_i= \e_k^i ^d k=1,âŠ,|E_i| \ represent the i-th codebook, where kie_k^i is the k-th code embedding in iE_i, and |i||E_i| is the codebook size. For the first codebook 1E_1, we set its size as |1|=3|E_1|=3, corresponding to images, text, and image-text pairs. We initialize the initial residual vector as 0=r_0=z and perform quantization recursively with L steps. At the i-th step, we search for the nearest neighbor in the i-th codebook iE_i as follows: (2) kiâ=argâĄmin1â€kâ€|i|âĄâiâ1âkiâ2,k_i^*= _1†kâ€|E_i| \|r_i-1-e_k^i \|^2, where kiâk_i^* denotes the selected index at the i-th step. Then the residual vector is updated as i=iâ1âkiâir_i=r_i-1-e_k_i^*^i. Finally, the refined representation vector z is approximated by summing the selected code embeddings across all quantization codebooks, yielding the quantized vector ^=âi=1Lkiâi z= _i=1^Le_k_i^*^i. A key property of residual quantization is that the code embedding selected at each level represents the residual information specific to that level, thus the information is progressively separated across quantization levels. To ensure the quantization quality, our training objective incorporates three types of loss functions that are computed over the same mini-batch of queryâcandidate pairs, denoted as Bsâr=(q^i,c^i)|1â€iâ€|Bsâr|B_sr=\(z_ q^\,i,z_ c^\,i)|1†iâ€|B_sr|\, where (q^i,c^i)(z_ q^\,i,z_ c^\,i) denotes the i-th pair, and |Bsâr||B_sr| denotes the corresponding batch size. The first is the residual quantization loss defined in Equation-3, (3) ârq=12â|Bsâr|âââBsârâi=1Lâiâ1âsgâĄ(kiâi)â2,L_rq= 12|B_sr| _zâ B_sr _i=1^L \|r_i-1-sg\! (e_k_i^*^i ) \|^2, Here, omitting the subscripts of z indicates that z may represent either a search query or a candidate item. This also explains why 2â|Bsâr|2|B_sr| is used as the denominator when computing the mean value. This loss encourages each residual vector to align with its assigned code embedding so that the quantization error is progressively reduced across quantization levels, where sgâĄ(â )sg(·) denotes the stop-gradient operator, preventing gradients from directly updating the code embeddings. Instead, the code embeddings are updated via an exponential moving average (EMA) (53) over training steps to ensure stable updates. The second loss function is the contrastive loss defined in Equation-4, (4) âsrcl=â1|Bsâr|âi=1|Bsâr|logexpâĄ(cosâĄ(q^i,c^i)/Ï)âj=1|Bsâr|expâĄ(cosâĄ(q^i,c^j)/Ï),L_sr^cl=- 1|B_sr| _i=1^|B_sr| \! ( (z_ q^\,i,z_ c^\,i)/Ï ) _j=1^|B_sr| \! ( (z_ q^\,i,z_ c^\,j)/Ï ), For the i-th query q^iz_ q^\,i in the mini-batch, we view c^iz_ c^\,i as the positive candidate, whereas the candidates paired with the remaining queries are treated as in-batch negative candidates. Since the query is guided by an instruction that specifies the desired target type, the pre-quantization embedding space should be structured so that each query is aligned with targets consistent with both its semantics and retrieval intent. This loss function encourages matched queryâtarget pairs to be close in the embedding space while pushing apart mismatched candidates, thereby inducing modality-level grouping and semantic alignment within each group. The third loss function is the MSE loss defined in Equation-5, (5) âmse=1|Bsâr|ââi=1|Bsâr|â^q^iâ^c^iâ22,L_mse= 1|B_sr| _i=1^|B_sr| \| z_ q^\,i- z_ c^\,i \|_2^2, where ^q^i z_ q^\,i and ^c^i z_ c^\,i denote the quantized representations obtained from q^iz_ q^\,i and c^iz_ c^\,i, respectively. This loss function further preserves pairwise consistency after quantization by keeping the quantized query representation close to its matched target candidate representation. Finally, the overall objective for optimizing the identifierâs sequential role is formulated by combining the above three loss functions as (6) âsr=âsrcl+ÎČ1âârq+ÎČ2ââmse,L_sr=L_sr^cl+ _1L_rq+ _2L_mse, Empirically, âsrclL_sr^cl is typically on the order of 10â110^-1, while ârqL_rq and âmseL_mse are on the order of 10â210^-2. To well balance their contributions, we introduce the weighting coefficients ÎČ1 _1 and ÎČ2 _2. 4.2.2. Set-based Role: Identifier Construction for Order-invariant Matching Under the set-based role for order-invariant matching, we aim to mitigate the limitations introduced by prefix-dependent autoregressive decoding. While the sequential identifier m enables token-by-token generation, inference in generative retrieval typically relies on constrained beam search, where identifiers are expanded based on prefix-level likelihoods. Consequently, once the prefix of a relevant identifier is pruned at an early decoding step, the corresponding target becomes irrecoverable. This prefix-sensitive behavior often leads to local optima, particularly when multiple candidates share similar prefixes. To address this issue, a key desideratum is to provide a global, prefix-independent relevance signal. Rather than introducing a separate identifier space, we reinterpret the sequential identifier =[m1,âŠ,mL]m=[m_1,âŠ,m_L] as an unordered set. This set-based reinterpretation discards positional information while preserving the underlying semantic tokens, thereby enabling order-invariant matching. To operationalize identifiers under the set-based role, we adopt a two-step design. Noting that identifiers consist of discrete code tokens selected from RQ codebooks, the first step aims to map the query representation into the same token space, thereby facilitating direct token-level relevance estimation. Specifically, given a query embedding q^z_ q, we refine it via an MLP module as ~q^=MLPoiâ(q^â) z_ q=MLP_oi(z^*_ q). Based on the refined representation, we compute a token-level score vector as (7) q^=logâĄ(1+ReLUâĄ(~q^âÂŻâ€))ââVT,s_ q= \! (1+ReLU( z_ q E ) ) ^V_T, where ÂŻââVTĂd E ^V_TĂ d denotes the fixed global codebook embedding table, constructed by aggregating code embeddings from all frozen RQ codebooks in â°E, and VTV_T is the size of the resulting code-token vocabulary. In particular, we interpret each dimension of q^s_ q as a relevance indicator measuring how well the semantic content encoded by the corresponding code token aligns with the query. Given this vocabulary-wise score vector and a candidate identifier =m1,âŠ,mLm=\m_1,âŠ,m_L\, the second step computes an order-invariant relevance score by aggregating the token-level scores associated with the tokens in m: (8) soiâ(q^,)=âl=1Lq^â[ÏâĄ(l,ml)],s_oi(z_ q,m)= _l=1^Ls_ q[Ï(l,m_l)], where the level-specific mapping function ÏâĄ(l,ml)=ml+âi=1lâ1|i|Ï(l,m_l)=m_l+ _i=1^l-1|E_i| maps the level-wise token index mlm_l to its global vocabulary index, and [â ][\,·\,] denotes the corresponding indexing operation. Finally, this formulation yields a set-level relevance score between the query and the candidate that is invariant to token order and independent of the decoding prefix. To optimize the identifierâs set-based role for order-invariant matching, we adopt two loss functions computed over a mini-batch of query-candidate pairs, denoted as Boi=(q^i,c^i)âŁ1â€iâ€|Boi|,B_oi=\(z_ q^\,i,z_ c^\,i) 1†iâ€|B_oi|\, where |Boi||B_oi| denotes the corresponding batch size. The first loss function is a contrastive objective, (9) âoicl=â1|Boi|âi=1|Boi|logexpâĄ(soiâ(q^i,i)/Ï)âj=1|Boi|expâĄ(soiâ(q^i,j)/Ï).L_oi^cl=- 1|B_oi| _i=1^|B_oi| \! (s_oi(z_ q^\,i,m_i)/Ï ) _j=1^|B_oi| \! (s_oi(z_ q^\,i,m_j)/Ï ). Analogous to Equation-4, the key difference lies in replacing the queryâcandidate relevance score with the order-invariant score defined in Equation-8. The second loss function is based on the margin ranking loss. To enhance discrimination against hard negatives, we identify the most challenging in-batch negative for each query according to dense vector-based cosine similarity: jiâ=argâĄmaxjâ iâĄcosâĄ(q^i,c^j)j_i^-= _jâ i (z_ q^\,i,z_ c^\,j). The corresponding identifier jiâm_j_i^- is used as the hard negative for the i-th query. Then a fixed margin Ï is enforced between the positive and hard-negative scores: (10) âoirank=1|Boi|ââi=1|Boi|maxâĄ(0,Ïâ(soiâ(q^i,i)âsoiâ(q^i,jiâ))).L_oi^rank= 1|B_oi| _i=1^|B_oi| \! (0,\,Ï- (s_oi(z_ q^\,i,m_i)-s_oi(z_ q^\,i,m_j_i^-) ) ). Finally, the overall objective for optimizing the identifierâs set-based role is defined as the combination of the two losses: âoi=âoicl+âoirankL_oi=L_oi^cl+L_oi^rank. Unlike the losses in the sequential role, the two objectives in the set-based role are empirically of comparable magnitude, and we therefore assign them equal importance. 4.3. Generative Retrieval with Dual-role Identifiers To fully exploit the complementary properties of dual-role identifiers, we use them in two ways: the sequential role supports autoregressive decoding, while the set-based role provides a global relevance prior during search. We first describe how the two roles are combined at inference time, and then introduce the corresponding training objectives. 4.3.1. Inference Process During inference, we exploit the two complementary roles of the same identifier to guide the generation process. Specifically, the sequential role provides fine-grained and order-sensitive token predictions through autoregressive decoding, while the set-based role provides a global, prefix-independent relevance prior. By combining sequential decoding scores with order-invariant global relevance signals, our method alleviates the local-optimum issue in constrained beam search. To condition the decoder on the query representation, we employ a lightweight projector that maps the query embedding into N prefix embeddings: (11) âĄ(q^)=ReshapeâĄ(MLPâĄ(q^))ââNĂdâČ,P(z_ q)=Reshape\! (MLP(z_ q) ) ^NĂ d , where dâČd denotes the hidden dimension of the decoder. Based on these prefix embeddings, decoding proceeds over the sequential identifier space using constrained beam search. More specifically, we build a Trie (prefix tree) from the sequential identifiers of all candidates and use it to constrain generation. A newly expanded prefix is considered valid only if it matches at least one candidate identifier prefix. We express this constraint by the validation function (12) ÎŽâĄ(â€i)=0,if â[t1,âŠ,ti]â is a valid prefix,ââ,otherwise,ÎŽ(t_†i)= cases0,&if [t_1,âŠ,t_i] is a valid prefix,\\ -â,&otherwise, cases where â€i=[t1,âŠ,ti]t_†i=[t_1,âŠ,t_i] denotes the prefix after expanding the current token. With the prefix constraint in place, we describe how the sequential decoding score is computed at each step. Given the query embedding q^z_ q and the current prefix <it_<i, the decoder produces the hidden state for the i-th decoding step as (13) i=DecoderâĄ(âĄ(q^),<i)ââdâČ.h_i=Decoder(P(z_ q),t_<i) ^d . Let idecââViĂdâČE^dec_i ^V_iĂ d denote the output embedding table for code tokens at the i-th identifier level, where ViV_i is the corresponding vocabulary size. To preserve the semantics encoded in the quantized space, we initialize the embeddings using the corresponding RQ codebook vectors projected into the decoder embedding space. The unnormalized autoregressive score for token tit_i is then idecâ[ti]â iE^dec_i[t_i]·h_i, and the full sequential decoding score for a generated identifier =[t1,âŠ,tL]t=[t_1,âŠ,t_L] is (14) ηâĄ(,q^)=âi=1Lidecâ[ti]â i,η(t;z_ q)= _i=1^LE^dec_i[t_i]·h_i, which accumulates order-sensitive evidence along the autoregressive decoding path. To incorporate the set-based role, we further introduce a global relevance prior derived from the order-invariant scoring function soiâ(â ,â )s_oi(·,·) defined in Section 4.2.2. For a partial prefix â€it_†i, we consider all candidate identifiers whose sequential forms share this prefix and compute (15) ÏâĄ(â€i,q^)=maxââ€iâĄsoiâ(q^,),Ï(t_†i;z_ q)= _m _t_†is_oi(z_ q,m), where â€iC_t_†i denotes the set of candidate identifiers whose sequential forms share the prefix â€it_†i. Each identifier in this set is evaluated by sois_oi under its set-based interpretation. Thus, ÏâĄ(â€i,q^)Ï(t_†i;z_ q) provides the strongest global relevance signal among candidates reachable from the current prefix. We then combine the prefix validity constraint, the sequential decoding score, and the set-based global prior into a unified expansion score: (16) fâĄ(â€i,q^)=ÎŽâĄ(â€i)+ηâĄ(<i,q^)+idecâ[ti]â i+λâÏâ(â€i,q^).f(t_†i;z_ q)=ÎŽ(t_†i)+η(t_<i;z_ q)+E^dec_i[t_i]·h_i+λÏ(t_†i;z_ q). Here, λ is a weighting hyperparameter that balances the contribution of the set-based global prior against the sequential decoding score. Each term in Equation-16 serves a distinct but complementary purpose. The validation score ÎŽâĄ(â€i)ÎŽ(t_†i) ensures that the newly expanded prefix remains consistent with at least one valid candidate identifier. The sequential component ηâĄ(<i,q^)+idecâ[ti]â iη(t_<i;z_ q)+E^dec_i[t_i]·h_i provides fine-grained, order-sensitive token-level evidence from the autoregressive decoder, but may still be vulnerable to local optima caused by early pruning. In contrast, the set-based prior ÏâĄ(â€i,q^)Ï(t_†i;z_ q) supplies a global relevance signal that is independent of the local decoding likelihood. By combining the two roles of the same identifier in this way, the decoder can maintain both local generation fidelity and global retrieval awareness throughout the search process. 4.3.2. Query Augmentation via Interpolation In multimodal generative retrieval, the limited representation capacity of discrete identifiers compared with continuous embeddings poses a challenge for model generalization, especially when training data is limited. To alleviate this issue, we introduce a query augmentation strategy based on queryâtarget interpolation in the continuous embedding space. Concretely, we construct an augmented query embedding by interpolating the query representation q^z_ q and the representation of its paired target c^z_ c: (17) q^âČ=ÎŒâ q^+(1âÎŒ)â c^,z _ q=Ό·z_ q+(1-ÎŒ)·z_ c, where ÎŒ is randomly sampled from a Beta distribution, i.e., ÎŒâŒBetaâĄ(α,α)ÎŒ (α,α). This strategy generates diverse augmented queries while preserving semantic relevance to the target, thereby improving robustness to query variation. 4.3.3. Generative Training Objective During training, the decoder is conditioned on the augmented query embedding through the projector âĄ(q^âČ)P(z _ q). The generative loss is defined as the token-level cross-entropy over the target sequential identifier of the paired candidate: (18) âGR((q^âČ),c^)=ââk=1Llogp(mkc^âŁ(q^âČ),m<kc^).L_GR\! (P(z _ q),m_ c )=- _k=1^L p\! (m c_k (z _ q),\,m c_<k ). This objective encourages the decoder to generate the target identifier sequence conditioned on the augmented query, thereby learning a robust mapping from continuous query semantics to discrete retrieval identifiers. 4.3.4. Discriminative Training Objective While generative learning focuses on predicting one correct identifier sequence, retrieval ultimately requires ranking candidates according to relevance. To reduce this discrepancy, we introduce a discriminative ranking objective that encourages the decoder to assign higher sequence-level scores to relevant target candidates than to hard negatives. Similar to the hard-negative selection used in the set-based role, we construct the discriminative objective over a mini-batch of queryâcandidate pairs, denoted as Bdec=(q^i,c^i,c^i)âŁ1â€iâ€|Bdec|B_dec=\(z_ q^\,i,z_ c^\,i,m_ c^\,i) 1†iâ€|B_dec|\, where c^im_ c^\,i is the sequential identifier of the paired target candidate. For each query i, we retain c^im_ c^\,i as the positive identifier and select the most similar non-matching candidate in the batch according to cosine similarity, i.e., jiâ=argâĄmaxjâ iâĄcosâĄ(q^i,c^j)j_i^-= _jâ i (z_ q^\,i,z_ c^\,j). The sequential identifier of the selected candidate, c^jiâm_ c^\,j_i^-, is then treated as the hard negative for query i. To compare candidates, we score each candidate sequential identifier under teacher forcing by averaging the token-level log-likelihoods assigned by the decoder. Given a candidate sequential identifier =(m1,âŠ,mL)m=(m_1,âŠ,m_L), we define its sequence-level score as (19) sseqâ(âĄ(q^âČi),)=1Lââk=1LlogâĄpâĄ(mkâŁâĄ(q^âČi),m<k).s_seq(P(z_ q \,i),m)= 1L _k=1^L p\! (m_k (z_ q \,i),\,m_<k ). Based on this scoring function, the positive and negative sequence-level scores are obtained as si+=sseqâ(âĄ(q^âČi),c^i)s_i^+=s_seq(P(z_ q \,i),m_ c^\,i) and siâ=sseqâ(âĄ(q^âČi),c^jiâ)s_i^-=s_seq(P(z_ q \,i),m_ c^\,j_i^-), respectively. We adopt an adaptive margin derived from the teacher similarity signal in the original embedding space: (20) âłi=maxâĄ(0,(cosâĄ(q^i,c^i)âcosâĄ(q^i,c^jiâ))â Îł),M_i= \! (0,\, ( (z_ q^\,i,z_ c^\,i)- (z_ q^\,i,z_ c^\,j_i^-) )·γ ), where Îł controls the scale of the margin. This design yields a softer constraint when the positive and negative candidates are semantically close, and a stronger one when they are well separated in the teacher embedding space. We then optimize a smooth pairwise ranking loss: (21) ârank=1|Bdec|ââi=1|Bdec|logâĄ(1+expâĄ(âłiâ(si+âsiâ))).L_rank= 1|B_dec| _i=1^|B_dec| \! (1+ (M_i-(s_i^+-s_i^-) ) ). The final decoder objective combines the generative loss and the discriminative ranking loss as âdecoder=âGR+ârank.L_decoder=L_GR+L_rank. Here, âGRL_GR provides token-level supervision for generating the correct identifier sequence, while ârankL_rank further encourages the decoder to assign higher sequence-level likelihood to relevant identifiers than to hard negatives. Compared with using only the generative loss, this combined objective better aligns autoregressive identifier generation with the retrieval objective, where semantically similar distractors should be ranked below the paired target. 4.4. Hybrid Retrieval Strategies A closer examination of the previous steps in Sections 4.2 and 4.3 reveals that discrete quantization is a crucial component of generative retrieval, which inevitably incurs information loss due to the approximation of continuous embeddings by finite codebooks. Although residual quantization preserves a coarse-to-fine semantic structure and enables efficient identifier generation, subtle distinctions between highly similar candidates may not be fully captured in the discrete space. To mitigate this limitation, we adopt a hybrid retrieval strategy that combines generative retrieval with dense vector-based reranking. Specifically, we first perform generative retrieval under dual guidance to obtain a ranked list of candidate identifiers and select the top-k candidates. These candidates are then reranked using the cosine similarity between the query embedding qz_q and candidate embeddings cz_c. From a computational perspective, generative retrieval avoids exhaustive similarity computation over the entire candidate pool, whose complexity would be âĄ(||âd)O(|C|d) for dense retrieval, where |||C| denotes the number of candidates and d the embedding dimension. Instead, the decoder explores only valid identifier prefixes via constrained beam search, significantly reducing the effective search space. The additional reranking stage operates only on the top-k candidates, incurring a cost of âĄ(kâd)O(kd) with kâȘ||k |C|, making the overhead negligible in practice. This hybrid design therefore preserves the scalability advantage of generative retrieval while compensating for quantization-induced information loss through continuous fine-grained scoring. 5. Experiments In this work, we investigate the following research questions: (1) RQ1: How does DrIG perform compared to existing dense vector-based approaches and generative retrieval methods in the context of universal multimodal retrieval? (2) RQ2: To what extent can DrIG generalize to standard text-to-image retrieval benchmarks under both zero-shot and in-domain training settings? (3) RQ3: How does each key component of DrIG contribute to its overall retrieval performance? (4) RQ4: How does DrIG balance retrieval effectiveness and inference efficiency relative to baseline dense vector-based and generative retrieval methods? Beyond these primary research questions, we further analyze the impact of key hyperparameters and conduct qualitative case studies to better understand the behaviors and capabilities of DrIG. 5.1. Experimental Setting 5.1.1. Datasets Table 1. An overview of M-BEIR. The columns refer to the task-specific retrieval setup, the constituent datasets, the involved domains, the number of queries across the Train/Validation/Test splits (#Query), the average number of relevant items per query (#Rel./Query), and the total number of candidate items (#Candid.). Task Dataset Domain #Query #Rel./Query #Candid. Train Val. Test Train Val. Test 1. qtâciq_tâ c_i VisualNews (38) News 99K 20K 20K 1.0 1.0 1.0 542K MSCOCO (37) Misc. 100K 24.8K 24.8K 1.0 1.0 1.0 5K Fashion200K (14) Fashion 15K 1.7K 1.7K 3.3 3.1 2.8 201K 2. qtâctq_tâ c_t WebQA (5) Wiki 16K 1.7K 2.4K 2.0 2.0 2.0 544K 3. qtâ(ci,ct)q_tâ(c_i,c_t) EDIS (39) News 26K 3.2K 3.2K 2.6 2.6 2.6 1M WebQA (5) Wiki 17K 1.7K 2.5K 1.4 1.4 1.4 403K 4. qiâctq_iâ c_t VisualNews (38) News 100K 20K 20K 1.0 1.0 1.0 537K MSCOCO (37) Misc. 113K 5K 5K 5.0 5.0 5.0 25K Fashion200K (14) Fashion 15K 4.8K 4.8K 1.0 1.0 1.0 61K 5. qiâciq_iâ c_i NIGHTS (11) Misc. 16K 2K 2K 1.0 1.0 1.0 40K 6. (qi,qt)âct(q_i,q_t)â c_t OVEN (16) Wiki 150K 50K 50K 8.5 10.0 9.9 676K InfoSeek (9) Wiki 141K 11K 11K 6.8 6.7 6.5 611K 7. (qi,qt)âci(q_i,q_t)â c_i FashionIQ (67) Fashion 16K 2K 6K 1.0 1.0 1.0 74K CIRR (41) Misc. 26K 2K 4K 1.0 1.0 1.0 21K 8. (qi,qt)â(ci,ct)(q_i,q_t)â(c_i,c_t) OVEN (16) Wiki 157K 14.7K 14.7K 17.8 17.5 17.7 335K InfoSeek (9) Wiki 143K 17.6K 17.6K 9.1 7.5 7.5 481K M-BEIR (66) 4 domains 1.1M 182K 190K 6.5 5.9 5.7 5.6M To evaluate universal multimodal retrieval, we adopt the widely recognized M-BEIR benchmark (66). M-BEIR integrates 10 datasets across 4 domains and encompasses 8 retrieval task types, where both queries and candidates can be text, images, or image-text pairs. Table 1 summarizes the key statistics of these datasets, including task type, source, domain, query splits, average relevant candidates per query, and total candidate pool size. We refer readers to the original paper (66) for detailed descriptions of the constituent datasets and their preprocessing procedures. Following previous studies (66; 40), our evaluation on M-BEIR is conducted under two distinct settings: (i) local-pool retrieval, where each query is searched only against the candidate pool of its corresponding dataset; and (i) global-pool retrieval, where candidates from all datasets are combined into a single unified pool. In addition, we include Flickr30K (69) and MSCOCO (the latter being a subset of M-BEIR) to facilitate additional comparisons with representative generative text-to-image retrieval methods. 5.1.2. Evaluation Metric We evaluate retrieval effectiveness using query-level mean Recall@K. (22) Recallâ@âK=1||ââqââĄ(TopKâĄ(q)â©qâ â ),Recall@K= 1|Q| _q I (TopK(q) _qâ ), where Q denotes the query set, qG_q is the set of ground-truth candidate items for query q, and TopKâĄ(q)TopK(q) denotes the set of top-K retrieved candidate items. âĄ(â )I(·) is the indicator function. Since a query may have multiple relevant candidates, we define a successful retrieval as the presence of at least one relevant candidate within the top-K retrieved items. Following prior studies, we evaluate the performance in terms of Recall@1, Recall@5, and Recall@10. Due to space constraints, Recall@k is abbreviated as R@k in some tables. 5.1.3. Implementation Details Table 2. The key hyperparameters and implementation settings. Parameter Value Parameter Value Random seed 2026 Modality codebook size 3 Identifier Construction Codebook vocabulary size 256,1024,2048,4096\256,1024,2048,4096\ RQ levels 4,6,8\4,6,8\ Training epochs / batch size 20 / 2000 Learning rate 1Ă10â41Ă 10^-4 Set-based and Sequential Roles Set-based epochs / batch size 10 / 512 Set-based learning rate 2Ă10â42Ă 10^-4 Sequential epochs / batch size 30 / 256 Sequential learning rate 1Ă10â41Ă 10^-4 Warmup steps 500 Evaluation Beam size 1,5,10,20,30,40,50\1,5,10,20,30,40,50\ Guidance scale λ 0.0,0.1,âŠ,1.0\0.0,0.1,âŠ,1.0\ Following LamRA (40), we adopt Qwen2-VL (63) as the multimodal encoder to produce 3584-dimensional embeddings for both queries and candidates. The generative retriever uses T5-small (51) as the autoregressive decoder. For identifier construction, we use modality-aware residual quantization. The default setting prepends a first-level modality codebook of size 3 to distinguish image, text, and imageâtext candidates, followed by residual codebooks for semantic encoding. The codebooks are initialized by k-means and updated with EMA (53) during training. DrIG is trained in three stages: (i) RQ-based identifier construction using Equation-6; (i) set-based role optimization for global, prefix-independent relevance guidance; and (i) sequential identifier generation by training the prefix projector and T5 decoder. The RQ module is trained for 20 epochs using AdamW with batch size 2000, learning rate 1Ă10â41Ă 10^-4, weight decay 0.01, loss weights ÎČ1=ÎČ2=100.0 _1= _2=100.0, and contrastive temperature Ï=0.01Ï=0.01. The set-based module is a two-layer MLP with hidden size 4096 and is trained for 10 epochs using AdamW with batch size 512, learning rate 2Ă10â42Ă 10^-4, weight decay 1Ă10â41Ă 10^-4, margin Ï=0.1Ï=0.1, and temperature Ï=0.01Ï=0.01. The sequential generation stage uses AdamW with batch size 256, learning rate 1Ă10â41Ă 10^-4, weight decay 1Ă10â41Ă 10^-4, 500 warmup steps, maximum gradient norm 1.0, and query interpolation coefficient α=2.0α=2.0. During inference, we perform Trie-constrained beam search over valid candidate identifiers. Unless otherwise specified, the beam size and guidance scale are set to 50 and 1.0, respectively. The generated top-k candidates can be further reranked using dense vector-based similarity. All experiments are conducted with mixed precision and distributed data parallel training on four NVIDIA A100 40GB GPUs. The main hyperparameters are summarized in Table 2. 5.1.4. Baseline Methods In this work, we compare our method against two distinct groups of baselines. The first group consists of dense vector-based methods, including CLIP (50), BLIP (28), their respective variants proposed in (66) (CLIP-SF, CLIP-F, BLIP-SF, and BLIP-F), and LamRA (40). Notably, LamRA represents the current state-of-the-art approach. The second group comprises generative retrieval methods. For universal multimodal retrieval, we consider GENIUS (23) as the current state-of-the-art approach. By GENIUS-C, we denote the variant of GENIUS augmented with CLIP-SF-based reranking. For text-to-image retrieval, representative methods include GRACE (33), IRGen (75), AVG (32), and ComGTIR-D (30), with ComGTIR-D serving as the current state-of-the-art. We evaluate three variants of our proposed model: (1) DrIG: The base generative retriever without additional reranking. (2) DrIG-C and DrIG-LT: Two extensions of DrIG that rerank the top candidates using CLIP-SF and LamRA embeddings, respectively. 5.2. Main Comparison on M-BEIR (RQ1) Table 3 summarizes the main comparison results on M-BEIR under the two evaluation settings introduced in Section 5.1.1: local-pool retrieval and global-pool retrieval. Overall, DrIG consistently improves over the generative baseline GENIUS in both settings, increasing the average score from 29.5 to 38.0 in local-pool retrieval and from 28.6 to 36.4 in global-pool retrieval. When combined with dense vector-based reranking, DrIG-C and DrIG-LT further improve the results, showing that generative candidate generation and dense vector-based reranking provide complementary signals. In the following subsections, we analyze these results in detail. Table 3. The comparison with state-of-the-art (SOTA) methods under two evaluation settings: local-pool retrieval and global-pool retrieval. Bold values indicate the best performance among all methods, and underlined values indicate the best performance within each respective retrieval strategy. Î% \% denotes the relative improvement over the corresponding generative baseline. Retrieval Strategy Dense Vector-based Retrieval LMM-based Reranking Generative Retrieval Generative Retrieval + Dense Vector-based Reranking #Parameters 428M 447M 7B 7B 30M 30M â â â Task Dataset Metric CLIP-SF BLIP-F LamRA-ret LamRA GENIUS DrIG (Î %) GENIUS-C DrIG-C (Î %) DrIG-LT (Î %) (a) Local-pool Retrieval Setting 1: qtâciq_tâ c_i VisualNews R@5 42.4 22.8 41.6 48.0 17.7 22.1(+24.9) 26.1 33.8(+29.5) 33.8(+29.5) MSCOCO R@5 80.6 79.4 81.5 85.2 61.3 70.2(+14.5) 66.0 78.8(+19.4) 79.6(+20.6) Fashion200K R@10 17.9 26.1 28.5 32.9 12.2 15.9(+30.3) 15.0 19.3(+28.7) 22.3(+48.7) 2: qtâctq_tâ c_t WebQA R@5 84.0 79.1 86.0 96.7 32.6 45.3(+39.0) 43.1 64.7(+50.1) 65.9(+52.9) 3: qtq_t â(ci,ct)â(c_i,c_t) EDIS R@5 53.6 49.8 62.4 75.8 35.7 38.7(+8.4) 43.3 50.4(+16.4) 56.1(+29.6) WebQA R@5 78.2 78.0 81.2 87.7 47.3 57.6(+21.8) 57.9 69.7(+20.4) 70.7(+22.1) 4: qiâctq_iâ c_t VisualNews R@5 42.5 23.0 39.6 48.6 17.9 21.5(+20.1) 25.5 33.7(+32.2) 32.7(+28.2) MSCOCO R@5 91.7 90.7 90.6 92.3 82.2 84.6(+2.9) 89.9 92.0(+2.3) 90.0(+0.1) Fashion200K R@10 18.1 28.5 30.4 36.1 12.1 16.7(+38.0) 16.7 20.1(+20.4) 24.7(+47.9) 5: qiâciq_iâ c_i NIGHTS R@5 31.0 31.7 32.1 33.5 9.5 17.7(+86.3) 30.0 31.7(+5.7) 32.0(+6.7) 6: (qi,qt)(q_i,q_t) âctâ c_t OVEN R@5 45.8 42.6 54.1 59.2 35.3 42.2(+19.5) 38.4 51.1(+33.1) 51.0(+32.8) InfoSeek R@5 27.2 23.2 52.1 64.1 11.4 25.0(+119.3) 19.8 35.0(+76.8) 44.1(+122.7) 7: (qi,qt)(q_i,q_t) âciâ c_i FashionIQ R@10 24.7 29.1 33.1 37.8 12.9 19.5(+51.2) 18.3 24.0(+31.1) 26.5(+44.8) CIRR R@5 44.6 50.5 53.1 63.3 21.8 33.7(+54.6) 37.3 45.3(+21.4) 49.0(+31.4) 8: (qi,qt)(q_i,q_t) â(ci,ct)â(c_i,c_t) OVEN R@5 68.7 56.4 76.2 79.2 32.6 48.7(+49.4) 35.1 63.1(+79.8) 63.2(+80.1) InfoSeek R@5 48.7 30.3 63.3 78.3 13.2 26.8(+103.0) 26.2 47.2(+80.2) 50.7(+93.5) â Average â 51.3 47.9 58.1 63.7 29.5 38.0(+28.8) 37.6 48.7(+29.5) 50.4(+34.0) (b) Global-pool Retrieval Setting 1: qtâciq_tâ c_i VisualNews R@5 42.1 22.3 41.3 46.9 17.7 22.0(+24.3) 26.1 33.8(+29.5) 33.8(+29.5) MSCOCO R@5 71.3 65.2 75.3 78.0 49.9 59.2(+18.6) 62.8 70.6(+12.4) 71.1(+13.2) Fashion200K R@10 17.9 26.0 28.5 32.5 12.2 15.8(+29.5) 14.2 19.3(+35.9) 22.3(+57.0) 2: qtâctq_tâ c_t WebQA R@5 83.4 78.4 85.8 96.5 31.7 44.3(+39.7) 43.0 64.3(+49.5) 65.1(+51.4) 3: qtq_t â(ci,ct)â(c_i,c_t) EDIS R@5 52.7 49.2 62.3 74.4 35.4 38.4(+8.5) 43.1 49.8(+15.5) 56.0(+29.9) WebQA R@5 77.2 77.0 81.0 87.1 47.0 56.8(+20.9) 57.5 68.5(+19.1) 70.0(+21.7) 4: qiâctq_iâ c_t VisualNews R@5 38.7 21.0 39.3 47.6 17.7 21.1(+19.2) 25.4 33.5(+31.9) 32.4(+27.6) MSCOCO R@5 91.3 89.7 90.4 92.4 81.5 84.5(+3.7) 89.9 91.9(+2.2) 90.0(+0.1) Fashion200K R@10 18.0 27.2 30.4 36.6 11.7 18.0(+53.8) 16.7 20.2(+21.0) 24.5(+46.7) 5: qiâciq_iâ c_i NIGHTS R@5 30.9 31.6 32.1 34.2 9.5 17.7(+86.3) 30.0 31.4(+4.7) 31.8(+6.0) 6: (qi,qt)(q_i,q_t) âctâ c_t OVEN R@5 39.5 39.4 48.4 54.0 34.0 40.4(+18.8) 37.8 46.3(+22.5) 47.1(+24.6) InfoSeek R@5 22.1 19.7 48.7 58.7 9.9 22.2(+124.2) 17.7 31.3(+76.8) 40.3(+127.7) 7: (qi,qt)(q_i,q_t) âciâ c_i FashionIQ R@10 24.4 28.8 33.1 37.4 12.8 19.1(+49.2) 18.2 23.7(+30.2) 26.3(+44.5) CIRR R@5 43.1 48.1 50.5 59.7 21.1 31.0(+46.9) 36.6 43.5(+18.9) 45.4(+24.0) 8: (qi,qt)(q_i,q_t) â(ci,ct)â(c_i,c_t) OVEN R@5 59.7 55.8 70.0 72.6 36.8 47.0(+27.7) 46.4 63.0(+35.8) 63.7(+37.3) InfoSeek R@5 44.1 26.1 60.0 74.0 12.4 25.3(+104.0) 25.3 44.4(+75.5) 47.8(+88.9) â Average â 48.6 45.7 56.3 61.4 28.6 36.4(+27.3) 37.8 47.1(+24.6) 48.9(+29.4) 5.2.1. Local-pool retrieval Table 3(a) reports the results under the local-pool setting. Compared with the generative baseline GENIUS, DrIG achieves consistent improvements across all retrieval tasks, increasing the average score from 29.5 to 38.0, corresponding to a relative gain of 28.8%. The gains are particularly pronounced on knowledge-intensive and compositional tasks, such as InfoSeek in Task 6, where the score increases from 11.4 to 25.0, and CIRR in Task 7, where the score increases from 21.8 to 33.7. These results suggest that the proposed dual-role identifier is effective for heterogeneous multimodal retrieval. Specifically, the sequential role preserves the autoregressive generation process, while the set-based role introduces a global, prefix-independent relevance signal that helps constrained beam search retain candidates that are semantically compatible with the full query. Dense vector-based reranking further improves performance, indicating that generative retrieval and dense vector-based retrieval provide complementary signals. Specifically, DrIG-C and DrIG-LT raise the average score to 48.7 and 50.4, respectively, outperforming both DrIG and GENIUS-C. This supports a natural two-stage interpretation in which the generative retriever efficiently produces a compact candidate set, and the dense vector-based reranker refines the final order using continuous, fine-grained semantic similarity. Nevertheless, a clear gap remains between the best DrIG variant and the strongest LMM-based reranker LamRA, especially on text-centric knowledge retrieval. For example, on WebQA in Task 2 (qtâctq_tâ c_t), DrIG-LT achieves 65.9, whereas LamRA reaches 96.7. This indicates that discretizing dense representations into short identifier sequences may lose information that is important for long, knowledge-dense text matching, and that reranking can mitigate but not fully eliminate this limitation. 5.2.2. Global-pool retrieval Table 3(b) reports the results under the global-pool setting. Compared with the local-pool setting, this setting is more challenging because all candidates are searched within a unified pool, requiring the model to infer the target modality, domain, and retrieval intent from the instruction. Under this more demanding setting, DrIG again consistently outperforms the generative baseline GENIUS across all retrieval tasks, improving the average score from 28.6 to 36.4, with a relative gain of 27.3%. Large improvements are observed on knowledge-intensive and multimodal tasks, such as InfoSeek in Task 6, where the score increases from 9.9 to 22.2, and InfoSeek in Task 8, where the score increases from 12.4 to 25.3. These results indicate that the dual-role identifier remains effective even when the retrieval space contains heterogeneous candidates from different datasets, domains, and modalities. The transition from local-pool to global-pool retrieval causes only a moderate decrease for DrIG, from 38.0 to 36.4 on average. A similar trend is observed for strong dense vector-based baselines, such as CLIP-SF, which decreases from 51.3 to 48.6, and LamRA-ret, which decreases from 58.1 to 56.3. This suggests that DrIG is reasonably robust to candidate-pool expansion. One possible explanation is that the modality-aware identifier and instruction-conditioned decoding process help constrain the search toward target-compatible candidate regions, while the set-based global prior provides an additional relevance signal for distinguishing candidates beyond local prefix likelihoods. Dense vector-based reranking remains beneficial in the global-pool setting. DrIG-C and DrIG-LT further improve the average score to 47.1 and 48.9, respectively, clearly outperforming GENIUS-C and surpassing BLIP-F. Notably, DrIG-LT also becomes competitive with CLIP-SF, showing that generative candidate generation combined with dense reranking can provide a favorable balance between retrieval effectiveness and scalability. Nevertheless, DrIG-LT still lags behind LamRA-ret and LamRA, whose average scores are 56.3 and 61.4, respectively. This gap indicates that although dual-role identifiers improve generative retrieval substantially, discretization still loses fine-grained semantic information, especially for tasks that require precise textual or knowledge-intensive matching. Overall, the global-pool results confirm the main findings from the local-pool setting: DrIG provides a stronger generative retriever than GENIUS, and hybrid reranking further narrows the effectiveness gap between generative and dense retrieval paradigms. 5.3. Text-to-image Generative Retrieval (RQ2) Table 4. The comparison of generative retrieval methods on text-to-image retrieval benchmarks. The best and second-best results for each metric are shown in bold and underlined, respectively. â indicates zero-shot performance, highlighting the ability of the model to generalize without task-specific fine-tuning. Method Training Data Recall@1 Recall@5 Recall@10 Flickr30K GRACE (33) Flickr30K 37.4 59.5 66.2 IRGen (75) Flickr30K 49.0 68.9 72.5 AVG (32) Flickr30K 40.8 75.1 84.2 ComGTIR-D (30) Flickr30K 42.6 75.7 85.3 ComGTIR-DHclip (30) Flickr30K 68.4 86.3 90.7 DrIG M-BEIR 59.0â 83.1â 88.2â DrIG-LT M-BEIR 75.8â 90.0â 91.6â DrIG Flickr30K 65.8 88.4 92.8 DrIG-LT Flickr30K 76.9 92.5 94.8 MSCOCO GRACE (33) MSCOCO 16.7 39.2 50.3 IRGen (75) MSCOCO 29.6 50.7 56.3 AVG (32) MSCOCO 19.3 45.7 59.7 ComGTIR-D (30) MSCOCO 20.6 46.7 61.0 ComGTIR-DHclip (30) MSCOCO 45.2 64.2 72.4 DrIG M-BEIR 41.8 70.2 79.4 DrIG-LT M-BEIR 56.1 79.6 86.0 DrIG MSCOCO 43.4 71.3 80.5 DrIG-LT MSCOCO 56.3 80.1 86.7 Table 4 compares DrIG with representative generative retrieval methods on Flickr30K and MSCOCO, two widely adopted benchmarks for text-to-image retrieval. These experiments complement the M-BEIR evaluation by examining whether the proposed generative retrieval framework remains effective in the conventional text-to-image setting. For Flickr30K, we evaluate both cross-benchmark transfer, where DrIG is trained on M-BEIR and directly tested on Flickr30K without Flickr30K-specific fine-tuning, and in-domain training on Flickr30K. For MSCOCO, since it is already included in M-BEIR, the M-BEIR-trained results should be interpreted as the performance of the universal model on a standard text-to-image benchmark rather than as a strict zero-shot evaluation. On Flickr30K, the M-BEIR-trained DrIG achieves strong zero-shot performance, reaching 59.0/83.1/ 88.2 in terms of Recall@1/Recall@5/Recall@10, respectively. This already outperforms several in-domain generative baselines, including GRACE, IRGen, AVG, and ComGTIR-D, indicating that instruction-conditioned training on heterogeneous M-BEIR tasks yields transferable visual-semantic identifiers. When combined with LamRA-based reranking, DrIG-LT further improves to 75.8/90.0/91.6, surpassing the strongest prior hybrid baseline, ComGTIR-DHclip. After in-domain training on Flickr30K, DrIG and DrIG-LT further improve to 65.8/88.4/92.8 and 76.9/92.5/94.8, respectively, showing that task-specific fine-tuning can still provide additional benefits on top of the universal retrieval training. On MSCOCO, DrIG trained on M-BEIR obtains 41.8/70.2/79.4 in terms of Recall@1/Recall@5/ Recall@10, respectively. Although its Recall@1 is slightly lower than ComGTIR-DHclip, it achieves higher Recall@5 and Recall@10 than all non-reranked and hybrid generative baselines. With LamRA-based reranking, DrIG-LT reaches 56.1/79.6/86.0, clearly outperforming existing generative retrieval methods across all three metrics. In-domain training on MSCOCO brings further but relatively modest gains, improving DrIG to 43.4/71.3/80.5 and DrIG-LT to 56.3/80.1/86.7. This suggests that the universal M-BEIR training already provides a strong foundation for MSCOCO-style text-to-image retrieval, while in-domain fine-tuning mainly refines dataset-specific matching patterns. The effect of reranking is also consistent across the two benchmarks. For example, under M-BEIR training, LamRA-based reranking improves DrIG by 16.8 Recall@1 points on Flickr30K and by 14.3 Recall@1 points on MSCOCO. The gains at Recall@10 are smaller, suggesting that the generative retriever often includes relevant candidates within the generated top-ranked set, while the dense reranker is particularly useful for promoting the most semantically matched image to the top positions. Overall, these results show that DrIG is not limited to the heterogeneous M-BEIR setting. It also serves as an effective generative candidate generator for standard text-to-image retrieval, and its combination with dense vector-based reranking yields strong performance under both transfer and in-domain evaluation settings. 5.4. Component Ablation and Representation Analysis (RQ3) We next analyze how different components of DrIG contribute to retrieval effectiveness. All ablations are conducted on the base DrIG model without dense vector-based reranking, so that the effect of each component can be isolated within the generative retrieval framework. We evaluate representative tasks from M-BEIR, including text-to-image retrieval, image-to-text retrieval, text-to-text retrieval, text-to-multimodal retrieval, multimodal-to-image retrieval, and image-to-image retrieval. 5.4.1. Component Ablation Table 5 reports the ablation results. We consider the following variants: (1) w/o Set-based Role removes the set-based interpretation of the identifier, so inference relies only on sequential identifier generation under constrained beam search. (2) w/o Trie removes the prefix-tree constraint, allowing the decoder to generate identifiers without prefix-level validity checking. (3) w/o Trie & Set-based removes both the structural validity constraint and the set-based global relevance prior. (4) w/o Query Augmentation disables the queryâtarget interpolation strategy used for decoder training. (5) w/o ârankL_rank removes the discriminative ranking objective in Equation-21, leaving only the generative sequence prediction objective. (6) w/o âsrclL^cl_sr removes the contrastive loss in Equation-4 during identifier construction. (7) Finally, w/o Modality removes the dedicated first-level modality codebook and uses only semantic residual codebooks for quantization. Table 5. The impact of different components in DrIG. The best and second-best results for each metric are shown in bold and underlined, respectively. MSCOCO WebQA CIRR NIGHTS qtâciq_tâ c_i qiâctq_iâ c_t qtâctq_tâ c_t qtâ(ci,ct)q_tâ(c_i,c_t) (qi,qt)âci(q_i,q_t)â c_i qiâciq_iâ c_i Component R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 Full 41.8 70.2 79.4 57.8 84.6 91.6 20.6 45.2 55.9 30.9 57.6 67.3 14.4 33.7 45.2 3.1 17.7 34.8 w/o Set-based Role 40.8 68.6 77.9 56.9 83.2 91.5 19.7 43.7 52.8 29.5 55.7 65.0 12.5 32.2 43.3 2.9 16.0 33.3 w/o Trie 20.6 21.6 21.6 50.7 60.2 60.5 18.9 29.8 30.3 27.9 41.2 43.1 10.7 14.6 14.9 2.9 14.0 19.8 w/o Trie & Set-based Role 0.1 0.1 0.1 1.0 1.0 1.0 0.3 0.4 0.4 3.6 4.6 4.6 0.1 0.1 0.1 0.9 1.9 2.7 w/o Query augmentation 37.1 66.8 77.7 43.9 72.9 83.3 14.9 36.9 49.5 27.4 52.2 63.7 11.0 28.8 40.1 2.8 15.0 29.6 w/o ârankL_rank in Equation-21 40.3 67.8 77.3 56.5 83.1 90.8 20.1 43.4 52.0 29.7 55.4 65.3 12.9 31.9 42.7 3.2 15.8 32.9 w/o âsrclL^cl_sr in Equation-4 0.5 1.9 3.2 0.5 2.5 3.9 0.0 0.0 0.1 0.0 0.2 0.3 0.0 0.1 0.4 0.0 0.3 0.4 w/o Modality 40.6 68.6 78.1 57.3 84.3 91.2 19.7 42.3 52.3 28.3 54.4 65.8 13.7 33.4 44.9 3.7 19.1 35.2 The most important observation is that contrastive learning before quantization is essential for constructing a usable discrete retrieval space. Removing âsrclL^cl_sr causes performance to collapse across almost all tasks. For example, MSCOCO text-to-image Recall@1 drops from 41.8 to 0.5, MSCOCO image-to-text Recall@1 drops from 57.8 to 0.5, and WebQA text-to-text Recall@1 drops from 20.6 to 0.0. This indicates that residual quantization alone is insufficient for retrieval-oriented identifier construction. The pre-quantization embedding space must first be organized according to semantic relevance and retrieval intent. Otherwise, the resulting identifiers cannot reliably preserve queryâcandidate matching relationships. The Trie constraint is another critical component. When the Trie is removed, the decoder is no longer restricted to valid candidate identifiers, leading to large drops in top-K retrieval performance. For instance, MSCOCO text-to-image Recall@10 decreases from 79.4 to 21.6, and CIRR Recall@10 decreases from 45.2 to 14.9. The degradation becomes even more severe when both the Trie constraint and the set-based role are removed, where most metrics approach zero. These results show that valid-prefix control and global relevance guidance are complementary. The Trie ensures that generated sequences correspond to existing candidates, while the set-based role helps rank valid prefixes using query-level relevance information beyond local token likelihoods. The set-based role itself provides consistent but more moderate gains. Removing it reduces MSCOCO text-to-image Recall@1 from 41.8 to 40.8, WebQA text-to-multimodal Recall@5 from 57.6 to 55.7, CIRR Recall@5 from 33.7 to 32.2, and NIGHTS Recall@5 from 17.7 to 16.0. These improvements are smaller than those brought by contrastive identifier learning or Trie-constrained decoding, but they are systematic across the evaluated tasks. This supports our motivation that prefix-independent relevance guidance can alleviate the local-optimum risk of purely sequential decoding, especially when multiple valid candidates share similar prefixes. Query augmentation also contributes substantially to decoder robustness. Without interpolation-based augmentation, performance drops on all evaluated tasks, with especially large decreases on MSCOCO image-to-text retrieval and WebQA text-to-text retrieval. For example, MSCOCO image-to-text Recall@1 decreases from 57.8 to 43.9, and WebQA text-to-text Recall@1 decreases from 20.6 to 14.9. This suggests that queryâtarget interpolation helps the decoder learn a smoother mapping from continuous query embeddings to discrete candidate identifiers, improving generalization to diverse query formulations. The discriminative ranking loss ârankL_rank further improves the alignment between autoregressive generation and retrieval ranking. Removing it generally lowers top-K performance, such as MSCOCO text-to-image Recall@5 from 70.2 to 67.8 and WebQA text-to-multimodal Recall@5 from 57.6 to 55.4. The gains are not uniformly large, and NIGHTS Recall@1 slightly increases from 3.1 to 3.2 without this loss. This indicates that ârankL_rank is mainly helpful for improving ranking consistency across most tasks, while its effect can be task-dependent when the absolute Recall@1 is very low or the candidates are visually fine-grained. The modality codebook is also generally beneficial, particularly for heterogeneous retrieval tasks. Removing the first-level modality token reduces performance on most text, image, and multimodal retrieval settings. For example, WebQA text-to-text Recall@5 decreases from 45.2 to 42.3, and WebQA text-to-multimodal Recall@5 decreases from 57.6 to 54.4. However, the effect is not universal. On NIGHTS image-to-image retrieval, removing the modality codebook slightly improves Recall@1/Recall@5/Recall@10 from 3.1/17.7/34.8 to 3.7/19.1/35.2. A possible explanation is that NIGHTS contains only image candidates, so an explicit modality token provides less discriminative benefit and may slightly reduce the capacity available for fine-grained visual similarity. Overall, the modality codebook is useful for universal multimodal retrieval, but its benefit depends on the degree of modality heterogeneity in the candidate pool. Figure 2. The t-SNE visualizations of multimodal code representations with and without the contrastive loss âsrclL^cl_sr. Colors indicate different modalities, including text, image, and imageâtext pairs. In summary, the ablation results show that DrIG depends on several complementary components. Contrastive identifier learning and Trie-constrained decoding are indispensable for building a valid and retrieval-oriented generative index. The set-based role provides additional prefix-independent relevance guidance, query augmentation improves decoder robustness, the ranking loss better aligns generation with retrieval, and the modality codebook helps organize heterogeneous candidate types. Together, these components enable DrIG to construct reliable dual-role identifiers for generative universal multimodal retrieval. 5.4.2. Visualization Analysis Figure 2 visualizes the learned code representations with and without the contrastive loss LsrclL^cl_sr. Without this contrastive objective, the representations still exhibit partial modality-related structure, but the clusters are scattered and overlap substantially, especially between text and imageâtext candidates. After introducing LsrclL^cl_sr, the representations become more compact and more clearly separated across text, image, and imageâtext modalities. This visualization provides qualitative evidence for the role of contrastive learning in identifier construction. By aligning matched queryâcandidate pairs and separating mismatched candidates before quantization, âsrclL^cl_sr encourages the residual quantizer to assign identifiers in a more semantically and modality-aware manner. This observation is consistent with the ablation results in Table 5, where removing âsrclL^cl_sr leads to severe performance degradation. Therefore, the contrastive objective is not merely an auxiliary training signal. It is a key prerequisite for learning discrete identifiers that preserve retrieval-relevant structure. 5.5. EffectivenessâEfficiency Trade-off (RQ4) Figure 3 compares representative retrieval methods from both efficiency and effectiveness perspectives. Figure 3(a) reports the online throughput, measured in queries per second (QPS), as the image candidate pool increases from 5K to 300K on a single A100 GPU. Under this evaluation setting, generative retrieval methods exhibit substantially more stable throughput than dense retrieval baselines. GENIUS and DrIG maintain nearly flat QPS curves as the candidate pool grows, because their main online cost comes from autoregressive decoding over a fixed-length identifier space rather than scoring every candidate in the pool. DrIG-C, which adds dense vector-based reranking over a fixed-size generated candidate set, also preserves this favorable scaling trend. In contrast, dense retrieval baselines such as CLIP-SF, BLIP-F, and LamRA-ret show decreasing throughput as the candidate pool becomes larger, since their retrieval process requires similarity computation over an expanding set of candidate embeddings in this implementation. LamRA has the lowest throughput, reflecting the additional cost of LMM-based reranking. Efficiency alone, however, is insufficient for evaluating retrieval systems. Figure 3(b) therefore compares retrieval effectiveness across different task types. DrIG consistently improves over GENIUS while retaining the scalability advantage of generative retrieval. For example, in the global-pool setting, DrIG improves the qtâctq^tâ c^t task from 31.7 to 44.3 and the averaged multimodal-to-multimodal task (qi,qt)â(ci,ct)(q^i,q^t)â(c^i,c^t) from 24.6 to 36.2. These gains indicate that the proposed dual-role identifiers improve the quality of generated candidates without sacrificing the efficient decoding behavior of generative retrieval. Hybrid reranking provides a further operating point in the effectivenessâefficiency trade-off. Compared with pure DrIG, DrIG-LT substantially improves effectiveness by using dense vector-based reranking to refine the generated candidate list, raising the average global-pool score from 36.4 to 48.9. This confirms that generative retrieval and dense vector-based reranking play complementary roles. The generator efficiently narrows the search space to a compact set of candidates, while the reranker performs fine-grained semantic matching within this set. Nevertheless, the strongest dense and LMM-based methods, such as LamRA-ret and LamRA, still achieve higher absolute effectiveness, indicating that discretization can lose information that is useful for fine-grained ranking. Overall, DrIG offers a favorable trade-off. It is substantially more effective than prior generative retrieval methods while preserving their scalability, and its hybrid variants allow users to exchange additional reranking cost for improved accuracy. 00505010010015015020020025025030030000551010151520202525Image set size (K)Queries per secondCLIPSFGENIUSLamRA-RetBLIPFFLamRADrIGDrIG-C (a) The efficiency under varying candidate-pool sizes. (b) The retrieval effectiveness across M-BEIR task types. Figure 3. The effectivenessâefficiency comparison across representative retrieval methods. 5.6. Design and Hyperparameter Sensitivity Analysis Beyond the main research questions, we further analyze how key design choices and hyperparameters affect DrIG. Unless otherwise specified, the analyses are conducted with the base DrIG model under the local-pool setting. We focus on five factors: residual-quantization codebook configuration, beam size, reranking depth, decoder backbone, and the weight of the set-based global prior. 5.6.1. Codebook Configuration Table 6. The ablation study on codebook configurations (Levels LĂLĂ Vocab K). QPS denotes the average number of queries processed per second across all tasks. Bold values indicate the best performance for each metric. MSCOCO WebQA CIRR NIGHTS QPS qtâciq_tâ c_i qiâctq_iâ c_t qtâctq_tâ c_t qtâ(ci,ct)q_tâ(c_i,c_t) (qi,qt)âci(q_i,q_t)â c_i qiâciq_iâ c_i Ave. LĂKLĂ K R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 (/s) 4 Ă 4096 32.3 61.3 73.1 49.4 79.0 88.2 7.3 20.9 31.0 14.3 36.1 48.4 9.0 24.7 34.8 3.1 14.8 26.0 63.3 6 Ă 4096 38.4 66.6 77.0 54.8 82.8 90.5 15.7 35.9 46.0 24.9 53.7 64.0 11.7 31.1 42.3 3.1 16.7 32.1 47.0 8 Ă 4096 41.8 70.2 79.4 57.8 84.6 91.6 20.6 45.2 55.9 30.9 57.6 67.3 14.4 33.7 45.2 3.1 17.7 34.8 37.6 4 Ă 2048 30.5 59.3 71.1 45.9 76.5 86.5 6.7 20.0 30.0 12.9 32.9 44.8 8.1 24.1 34.3 3.6 15.0 27.7 63.8 6 Ă 2048 36.5 65.8 76.4 53.2 82.3 90.3 14.1 33.8 44.8 23.6 49.3 60.4 11.9 30.0 41.2 3.9 16.4 31.4 47.4 8 Ă 2048 41.0 69.4 79.0 56.5 84.1 91.3 20.0 43.3 52.8 29.9 55.8 67.8 13.7 33.8 45.2 3.9 18.1 33.4 37.8 4 Ă 1024 28.3 56.9 68.8 42.1 74.2 85.3 6.0 17.4 25.9 10.7 29.0 39.2 7.7 21.7 31.3 2.6 14.6 27.3 36.7 6 Ă 1024 35.6 64.8 75.5 51.8 80.5 89.2 15.1 32.8 42.9 22.0 47.5 58.8 10.8 28.8 40.1 3.1 15.7 31.9 27.6 8 Ă 1024 39.8 68.4 78.3 55.5 83.4 90.7 19.3 42.6 51.2 27.7 55.5 66.6 12.5 31.8 43.8 3.0 17.4 34.0 37.9 4 Ă 256 22.6 49.3 62.4 35.1 65.1 78.1 3.0 10.7 14.9 5.1 14.1 21.1 5.0 15.6 23.8 2.1 9.7 18.7 39.1 6 Ă 256 31.7 60.4 71.9 47.4 77.4 87.0 8.6 25.8 35.2 15.4 36.9 47.9 7.9 24.3 34.8 3.0 16.4 29.9 48.5 8 Ă 256 36.1 65.1 75.4 52.7 80.7 89.3 15.4 36.1 45.6 22.7 47.5 59.4 11.0 28.8 39.4 3.4 17.6 31.5 26.3 Table 6 reports the effect of different residual-quantization configurations by varying the number of quantization levels L and the semantic codebook vocabulary size K. The first-level modality codebook is kept fixed, while the remaining codebooks are used to encode semantic residuals. Overall, increasing the quantization depth brings the most consistent improvement. For example, when K=4096K=4096, increasing L from 4 to 8 improves MSCOCO text-to-image Recall@1 from 32.3 to 41.8 and WebQA text-to-text Recall@1 from 7.3 to 20.6. This indicates that longer identifiers provide more capacity for capturing fine-grained multimodal semantics. Increasing the vocabulary size also tends to improve retrieval effectiveness, especially when the identifier is sufficiently deep. For instance, under L=8L=8, the configuration 8Ă40968Ă 4096 achieves the best performance on most MSCOCO and WebQA metrics. However, the benefit of a larger vocabulary is not uniform across all tasks. On visually fine-grained datasets such as CIRR and NIGHTS, 8Ă20488Ă 2048 performs slightly better on several metrics, including CIRR Recall@5 and NIGHTS Recall@1/Recall@5. This suggests that an excessively large codebook is not always optimal. For tasks with many visually similar candidates, a moderately sized vocabulary may provide useful regularization. The efficiency trend is also task- and implementation-dependent. Increasing the number of quantization levels generally reduces throughput because the decoder must generate longer identifiers. For example, with K=4096K=4096, QPS decreases from 63.3 for 4Ă40964Ă 4096 to 37.6 for 8Ă40968Ă 4096. In contrast, the effect of vocabulary size on QPS is less monotonic, since Trie-constrained decoding only expands valid prefixes rather than all tokens in the vocabulary. Therefore, codebook design introduces a trade-off between identifier expressiveness and inference efficiency. We use 8Ă40968Ă 4096 as the default setting because it provides the strongest and most stable overall effectiveness while maintaining acceptable retrieval throughput. 5.6.2. Beam Size Table 7. The comprehensive beam size ablation on decoding settings. QPS represents the average queries per second across all tasks. Bold values indicate the best performance for each metric. MSCOCO WebQA CIRR NIGHTS QPS qtâciq_tâ c_i qiâctq_iâ c_t qtâctq_tâ c_t qtâ(ci,ct)q_tâ(c_i,c_t) (qi,qt)âci(q_i,q_t)â c_i qiâciq_iâ c_i Ave. Beam R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 (/s) 1 25.1 25.1 25.1 43.4 43.5 43.5 8.1 8.2 8.2 14.9 15.9 15.9 8.1 8.2 8.2 3.1 3.3 3.3 972.6 5 38.9 57.6 57.6 57.2 79.8 80.1 17.2 30.9 31.0 26.5 43.4 44.6 13.5 26.9 27.2 3.4 17.3 18.9 275.2 10 40.8 65.0 70.3 57.6 83.1 89.2 19.7 38.1 43.0 29.4 51.0 58.0 13.8 31.2 39.3 3.2 17.8 32.8 162.8 20 41.5 68.4 76.7 57.8 84.1 91.0 20.4 42.8 50.6 30.5 55.2 63.5 14.4 32.8 43.2 3.2 18.0 34.5 87.9 30 41.7 69.5 78.4 57.9 84.6 91.4 20.6 44.3 53.5 30.8 56.7 66.0 14.5 33.2 44.6 3.1 17.7 34.4 61.3 40 41.8 69.9 79.1 57.8 84.7 91.6 20.6 45.3 55.2 30.9 57.1 66.6 14.6 33.4 44.7 3.1 17.7 34.9 46.6 50 41.8 70.2 79.4 57.8 84.6 91.6 20.6 45.2 55.9 30.9 57.6 67.3 14.4 33.7 45.2 3.1 17.7 34.8 37.6 Table 7 studies the effect of beam size during Trie-constrained decoding. Increasing the beam size mainly improves candidate coverage by allowing the decoder to explore more valid identifier paths. The improvement is especially large when moving from greedy decoding to moderate beam search. For example, MSCOCO text-to-image Recall@10 increases from 25.1 with beam size 1 to 70.3 with beam size 10, showing that greedy decoding is highly vulnerable to early prefix-level decisions. The performance gain becomes smaller as the beam size continues to increase. Most metrics improve rapidly from beam size 1 to 20 and then gradually saturate between 30 and 50. This pattern suggests that a moderate beam is already sufficient to preserve most relevant candidate paths, while a very large beam mainly refines the top-ranked candidate set. The improvement is not strictly monotonic for every metric. For instance, NIGHTS obtains its best Recall@1 at beam size 5, and several Recall@1 values fluctuate slightly at larger beam sizes. This is expected because a larger beam improves coverage but does not necessarily improve the final top-1 ranking, especially on visually fine-grained tasks with many similar candidates. Beam size also has a substantial impact on inference efficiency. QPS decreases from 972.6 with beam size 1 to 37.6 with beam size 50. Thus, beam size controls a direct effectivenessâefficiency trade-off. We set the default beam size to 50 in the main experiments to maximize retrieval effectiveness, while beam sizes 20 or 30 provide a practical alternative when lower latency is required. 5.6.3. Reranking Depth Figure 4. The comprehensive ablation study of reranking depth k across different datasets. Recall is reported for kâ3,5,âŠ,50kâ\3,5,âŠ,50\, highlighting the impact of retrieval-rerank trade-offs in representative tasks. Figure 4 analyzes the effect of reranking depth k for the hybrid variants DrIG-C and DrIG-LT. Here, k denotes the number of generated candidates passed to the dense vector-based reranker. The results show that reranking is most beneficial when k increases from a very small value to a moderate range. This is because the reranker can only refine the order of candidates that have already been generated by DrIG. When k is too small, the generated candidate set may not contain enough relevant or near-relevant candidates for reranking to be effective. This pattern is especially clear on visually challenging tasks. For example, on NIGHTS, DrIG-LT obtains a large improvement in Recall@5 when k increases from 3 to 10, indicating that a small reranking pool is insufficient for image-to-image retrieval, where many candidates are visually similar. After the main relevant candidates are included, further increasing k leads to diminishing returns. Similar saturation trends can be observed on MSCOCO, WebQA, and CIRR. Across most tasks, DrIG-LT outperforms DrIG-C, showing that LamRA-based embeddings provide stronger fine-grained reranking signals than CLIP-SF embeddings. These results confirm the complementary roles of generative retrieval and dense vector-based reranking. The generative retriever efficiently narrows the search space, while the reranker improves the final ordering within the generated candidate set. In the main experiments, we use k=50k=50 to maximize effectiveness. For efficiency-sensitive applications, however, moderate values such as k=20k=20 or k=30k=30 can provide a better trade-off between accuracy and reranking cost. 5.6.4. Decoder Backbone Table 8. The ablation results of model backbone combinations. Bold values indicate the best performance for each metric. MSCOCO WebQA CIRR NIGHTS qtâciq_tâ c_i qiâctq_iâ c_t qtâctq_tâ c_t qtâ(ci,ct)q_tâ(c_i,c_t) (qi,qt)âci(q_i,q_t)â c_i qiâciq_iâ c_i Model # Params R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 T5-small 30M 41.8 70.2 79.5 57.8 84.6 91.6 20.7 45.3 56.0 30.9 57.6 67.3 14.4 33.7 45.2 3.1 17.7 34.8 T5-base 110M 42.5 70.6 80.0 58.4 85.3 91.8 23.2 47.1 57.6 32.1 58.2 68.6 11.4 30.5 42.2 2.3 14.1 31.1 T5-large 400M 42.1 70.4 79.9 58.5 84.8 92.0 24.1 47.7 57.3 32.1 58.6 68.4 10.4 27.2 39.2 1.7 11.4 27.6 Table 8 compares different T5 decoder backbones while keeping the multimodal encoder and identifier construction procedure unchanged. Increasing the decoder size brings moderate gains on several text-centric or knowledge-intensive tasks. For example, T5-large improves WebQA text-to-text Recall@1 from 20.7 to 24.1, and T5-base slightly improves MSCOCO text-to-image Recall@1 from 41.8 to 42.5. These results suggest that a larger decoder can better model semantic and textual patterns during identifier generation. However, larger decoders do not improve all tasks. On visually grounded and compositional retrieval tasks, T5-small is more stable. For example, on CIRR, T5-small achieves 14.4/33.7/45.2 in Recall@1/Recall@5/Recall@10, whereas T5-large drops to 10.4/27.2/39.2. A similar trend appears on NIGHTS image-to-image retrieval. This indicates that simply increasing decoder capacity is not always beneficial for generative multimodal retrieval. For tasks that depend heavily on fine-grained visual similarity, the bottleneck may lie more in identifier construction and visual-semantic discretization than in the language modeling capacity of the decoder. Considering both retrieval effectiveness and model efficiency, we use T5-small as the default decoder backbone. It has only 30M parameters and provides the most balanced cross-task performance, making it suitable for scalable generative retrieval. 5.6.5. Global Prior Weight λ Figure 5. The effect of the global prior weight λ on Recall@5. Figure 5 studies the effect of the global prior weight λ in Equation-16. This parameter controls the contribution of the set-based global relevance prior during Trie-constrained beam search. When λ=0λ=0, inference relies only on the sequential decoding score and the Trie constraint. When λ>0λ>0, the decoder additionally uses the prefix-independent relevance signal derived from the set-based role of the identifier. The results show that introducing the global prior consistently improves Recall@5 over λ=0λ=0 on most tasks. For MSCOCO text-to-image, WebQA text-to-multimodal retrieval, NIGHTS image-to-image retrieval, and CIRR composed image retrieval, performance generally increases as λ becomes larger. This confirms that the set-based role provides useful global guidance beyond local autoregressive likelihoods. By scoring candidates reachable from a prefix using the full query representation, the global prior helps the decoder retain semantically promising branches that may otherwise be pruned early. The optimal value of λ is task-dependent. For example, text-centric retrieval can exhibit small fluctuations when the prior weight becomes large, suggesting that the set-based prior and sequential decoder score may have different calibration behaviors across modalities and tasks. Nevertheless, λ=1.0λ=1.0 provides strong and stable performance across most evaluated settings. We therefore use λ=1.0λ=1.0 as the default value in the main experiments, while noting that task-specific tuning may yield further improvements in specialized retrieval scenarios. 5.7. Qualitative Case Studies Figure 6. The qualitative case studies of DrIG. Figure 6 presents three representative cases that illustrate how DrIG behaves during dual-guided decoding and hybrid reranking. The red path denotes the identifier sequence selected during beam search, while the green result denotes the ground-truth candidate. Together, these examples show three typical outcomes: direct success by dual-guided generation, correction by dense vector-based reranking, and a remaining hard case involving fine-grained multimodal evidence matching. In the first case, DrIG retrieves the ground-truth image at rank 1 during beam search. The generated identifier follows the correct branch, and the candidate branch ending with token 173 receives an additional set-based relevance contribution. This example demonstrates the benefit of the dual-role identifier when the sequential decoding score and the set-based global prior are aligned. The autoregressive decoder provides a valid and semantically plausible identifier path, while the set-based role further strengthens the globally relevant branch, allowing the correct candidate to remain preferred over competing prefixes. The second case shows an error made by beam search that is corrected by reranking. During generation, the ground-truth text candidate remains in the top-ranked candidate list, but it is placed behind a competing branch because its combined decoding score is still lower. Specifically, although the correct branch ending with token 139 receives an additional set-based relevance contribution, it does not surpass the competing branch ending with token 140 during the generative stage. However, because the correct candidate is preserved within the generated top-k set, dense vector-based reranking can compare candidates in the continuous representation space and promote the ground-truth result to rank 1. This case supports the motivation of the hybrid retrieval strategy: generative retrieval provides an efficient candidate-generation mechanism, while dense reranking compensates for residual ranking errors caused by discrete identifier decoding. The third case highlights a remaining limitation. DrIG correctly infers the target modality as an imageâtext pair and retrieves candidates that are visually related to the query image. However, the task requires simultaneous alignment between visual evidence and fine-grained textual information. Although the retrieved candidates are visually similar railway-station examples, the model fails to distinguish the correct textual evidence, such as the specific answer about the opening year. The ground-truth branch associated with token 189 receives an additional set-based relevance contribution, but it still does not dominate the competing branch associated with token 187. Moreover, dense vector-based reranking is unable to move the ground-truth candidate to the top position, suggesting that both the discrete identifier and the dense reranker struggle when the distinction depends on subtle multimodal evidence. Overall, the case studies provide qualitative support for the proposed design. The dual-role identifier can guide beam search toward globally relevant candidates and reduce prefix-level decoding errors, while hybrid reranking can further correct cases where the ground-truth candidate is generated but not ranked first. At the same time, the failure case indicates that DrIG still has difficulty with retrieval scenarios requiring precise joint reasoning over visual content and textual evidence. Improving fine-grained multimodal grounding and strengthening reranking for imageâtext candidates remain important directions for future work. 6. Conclusion and Future Work In this paper, we proposed DrIG, a generative framework for universal multimodal retrieval with dual-role identifiers. The key idea is to assign each candidate a single residual-quantized identifier and reuse it in two complementary ways. In its sequential role, the identifier is generated autoregressively under a Trie constraint, enabling efficient retrieval over a discrete candidate space. In its set-based role, the same identifier tokens provide a prefix-independent global relevance prior, which helps guide constrained beam search beyond local token-level likelihoods. By combining these two roles during inference, DrIG reduces the risk that relevant candidates are discarded because of early prefix-level pruning. To address the information loss introduced by discrete quantization, we further introduced a hybrid retrieval strategy. DrIG first generates a compact list of candidate identifiers and then optionally reranks the corresponding candidates using continuous embedding similarity. This design allows the generative retriever to preserve its scalability advantage while using dense representations to recover fine-grained semantic distinctions that may be weakened during quantization. In addition, query augmentation through queryâtarget interpolation and a discriminative ranking objective further improve the robustness and retrieval-awareness of decoder training. Experiments on M-BEIR demonstrate that DrIG consistently outperforms the generative universal multimodal retrieval baseline GENIUS under both local-pool and global-pool settings. Additional text-to-image experiments on Flickr30K and MSCOCO further show that DrIG is effective beyond the heterogeneous M-BEIR benchmark. The ablation studies confirm the importance of contrastive identifier learning, Trie-constrained decoding, query augmentation, the set-based relevance prior, the ranking objective, and modality-aware quantization. The efficiency and sensitivity analyses further show how codebook configuration, beam size, reranking depth, decoder backbone, and prior weight affect the effectivenessâefficiency trade-off. Overall, these results suggest that dual-role identifiers provide a promising direction for scalable generative multimodal retrieval, especially when combined with lightweight dense vector-based reranking. Several directions remain for future work. First, although DrIG substantially improves generative retrieval effectiveness, a gap remains between hybrid generative retrieval and the strongest dense or LMM-based retrievers on some knowledge-intensive and text-centric tasks. Developing more expressive identifiers, better score calibration mechanisms, or stronger reranking strategies may further reduce this gap. Second, the current framework follows a staged training pipeline in which representation learning, identifier construction, set-based scoring, and decoder training are optimized separately. A more tightly coupled or end-to-end optimization strategy could further align discrete identifier learning with the final retrieval objective. Third, generative retrieval systems must support dynamic candidate collections in practical applications. Future work should investigate how DrIG can efficiently handle candidate insertion, deletion, and identifier updates without expensive retraining. Finally, it would be valuable to evaluate DrIG in larger and more diverse multimodal retrieval scenarios, such as web-scale imageâtext retrieval, video retrieval, and retrieval-augmented multimodal generation. References Bevilacqua et al. (2022) M. Bevilacqua, G. Ottaviano, P. Lewis, S. Yih, S. Riedel, and F. Petroni Autoregressive search engines: generating substrings as document identifiers. Advances in Neural Information Processing Systems 35, p. 31668â31683. Cited by: §1, §2.2.1. Burges et al. (2006) C. Burges, R. Ragno, and Q. Le Learning to rank with nonsmooth cost functions. Advances in neural information processing systems 19. Cited by: §1. Cao et al. (2024) H. Cao, C. Tan, Z. Gao, Y. Xu, G. Chen, P. Heng, and S. Z. Li A survey on generative diffusion models. IEEE transactions on knowledge and data engineering 36 (7), p. 2814â2830. Cited by: §1. Cao et al. (2022) M. Cao, S. Li, J. Li, L. Nie, and M. Zhang Image-text retrieval: a survey on recent research and development. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, p. 5410â5417. Note: Survey Track Cited by: §2. Chang et al. (2022) Y. Chang, M. Narang, H. Suzuki, G. Cao, J. Gao, and Y. Bisk Webqa: multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 16495â16504. Cited by: Table 1, Table 1. Chen et al. (2021) J. Chen, H. Hu, H. Wu, Y. Jiang, and C. Wang Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15789â15798. Cited by: §2.1. Chen et al. (2023a) J. Chen, R. Zhang, J. Guo, M. de Rijke, W. Chen, Y. Fan, and X. Cheng Continual learning for generative retrieval over dynamic corpora. In Proceedings of the 32nd ACM international conference on information and knowledge management, p. 306â315. Cited by: §1, §2.2.1, §2.2.1. Chen et al. (2023b) X. Chen, Y. Liu, B. He, L. Sun, and Y. Sun Understanding differential search index for text retrieval. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, p. 10701â10717. Cited by: §1, §2.2.1. Chen et al. (2023c) Y. Chen, H. Hu, Y. Luan, H. Sun, S. Changpinyo, A. Ritter, and M. Chang Can pre-trained vision and language models answer visual information-seeking questions?. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 14948â14968. Cited by: Table 1, Table 1. Fang et al. (2025) M. Fang, S. Ji, J. Zuo, H. Huang, Y. Xia, J. Zhu, X. Cheng, X. Yang, W. Liu, G. Wang, Z. Dong, and Z. Zhao CART: a generative cross-modal retrieval framework with coarse-to-fine semantic modeling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, External Links: Document Cited by: §2.2.2. Fu et al. (2023) S. Fu, N. Tamir, S. Sundaram, L. Chai, R. Zhang, T. Dekel, and P. Isola DreamSim: learning new dimensions of human visual similarity using synthetic data. External Links: 2306.09344, Link Cited by: Table 1. Gao et al. (2021) T. Gao, X. Yao, and D. Chen SimCSE: simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Online and Punta Cana, Dominican Republic, p. 6894â6910. External Links: Link, Document Cited by: §4.1.2. Gong et al. (2014) Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik Improving image-sentence embeddings using large weakly annotated photo collections. In Computer VisionâECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, p. 529â545. Cited by: §2.1. Han et al. (2017) X. Han, Z. Wu, P. X. Huang, X. Zhang, M. Zhu, Y. Li, Y. Zhao, and L. S. Davis Automatic spatially-aware fashion concept discovery. In Proceedings of the IEEE international conference on computer vision, p. 1463â1471. Cited by: Table 1, Table 1. Hendriksen et al. (2025) M. Hendriksen, S. Zhang, R. Reinanda, M. Yahya, E. Meij, and M. de Rijke Benchmark granularity and model robustness for image-text retrieval: a reproducibility study. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 3183â3193. Cited by: §1. Hu et al. (2023) H. Hu, Y. Luan, Y. Chen, U. Khandelwal, M. Joshi, K. Lee, K. Toutanova, and M. Chang Open-domain visual entity recognition: towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 12065â12075. Cited by: Table 1, Table 1. Jang et al. (2023) J. Jang, C. Kong, D. Jeon, S. Kim, and N. Kwak Unifying vision-language representation space with single-tower transformer. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37, p. 980â988. Cited by: §2.1. Jiang et al. (2024a) T. Jiang, S. Huang, Z. Luan, D. Wang, and F. Zhuang Scaling sentence embeddings with large language models. In Findings of the association for computational linguistics: EMNLP 2024, p. 3182â3196. Cited by: §4.1.1. Jiang et al. (2024b) T. Jiang, M. Song, Z. Zhang, H. Huang, W. Deng, F. Sun, Q. Zhang, D. Wang, and F. Zhuang E5-v: universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580. Cited by: §4.1.1. Jiang et al. (2025) Z. Jiang, R. Meng, X. Yang, S. Yavuz, Y. Zhou, and W. Chen VLM2Vec: training vision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations, Cited by: §2.1. Khattab and Zaharia (2020) O. Khattab and M. Zaharia Colbert: efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, p. 39â48. Cited by: §1. Kim et al. (2024) C. Kim, S. Yoon, H. Lee, J. Jang, S. Yang, and M. Seo Exploring the practicality of generative retrieval on dynamic corpora. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 13616â13633. Cited by: §1, §2.2.1, §2.2.1. Kim et al. (2025) S. Kim, X. Zhu, X. Lin, M. Bastan, D. Gray, and S. Kwak GENIUS: a generative framework for universal multimodal search. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 19659â19669. Cited by: §1, §2.2.3, §4.2.1, §5.1.4. Lee et al. (2025) C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping NV-embed: improved techniques for training llms as generalist embedding models. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 79310â79333. External Links: Link Cited by: §4.1.2. Lee et al. (2022) D. Lee, C. Kim, S. Kim, M. Cho, and W. Han Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 11523â11532. Cited by: §4.2.1. Lee et al. (2018) K. Lee, X. Chen, G. Hua, H. Hu, and X. He Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), p. 201â216. Cited by: §2.1. Li et al. (2025a) H. Li, Y. Bin, Y. Ma, G. Wang, Y. Yang, S. Ng, and T. Chua SemCORE: a semantic-enhanced generative cross-modal retrieval framework with mllms. arXiv preprint arXiv:2504.13172. Cited by: §1, §2.2.2. Li et al. (2022) J. Li, D. Li, C. Xiong, and S. Hoi Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, p. 12888â12900. Cited by: §5.1.4. Li et al. (2021) J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi Align before fuse: vision and language representation learning with momentum distillation. Advances in neural information processing systems 34, p. 9694â9705. Cited by: §2.1. Li et al. (2025b) K. Li, H. Yu, Y. Fang, and C. Lei A combination-based framework for generative text-image retrieval: dual identifiers and hybrid retrieval strategies. In Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP 2025, New York, NY, USA, p. 281â291. External Links: ISBN 9798400722189, Link, Document Cited by: §2.2.2, §5.1.4, Table 4, Table 4, Table 4, Table 4. Li et al. (2025c) X. Li, J. Jin, Y. Zhou, Y. Zhang, P. Zhang, Y. Zhu, and Z. Dou From matching to generation: a survey on generative information retrieval. ACM Transactions on Information Systems 43 (3), p. 1â62. Cited by: §1, §2.2. Li et al. (2025d) Y. Li, H. Cai, W. Wang, L. Qu, Y. Wei, W. Li, L. Nie, and T. Chua Revolutionizing text-to-image retrieval as autoregressive token-to-voken generation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 813â822. Cited by: §1, §2.2.2, §5.1.4, Table 4, Table 4. Li et al. (2024a) Y. Li, W. Wang, L. Qu, L. Nie, W. Li, and T. Chua Generative cross-modal retrieval: memorizing images in multimodal language models for retrieval and beyond. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 11851â11861. Cited by: §2.2.2, §5.1.4, Table 4, Table 4. Li et al. (2023) Y. Li, N. Yang, L. Wang, F. Wei, and W. Li Multiview identifiers enhanced generative retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6636â6648. Cited by: §1, §2.2.1, §2.2.1. Li et al. (2024b) Y. Li, N. Yang, L. Wang, F. Wei, and W. Li Learning to rank in generative retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 8716â8723. Cited by: §1, §2.2.1, §2.2.1. Lin et al. (2025) S. Lin, C. Lee, M. Shoeybi, J. Lin, B. Catanzaro, and W. Ping Mm-embed: universal multimodal retrieval with multimodal llms. The Thirteenth International Conference on Learning Representations. Cited by: §1, §1, §2.1, §4.1.2. Lin et al. (2014) T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. DollĂĄr, and C. L. Zitnick Microsoft coco: common objects in context. In European conference on computer vision, p. 740â755. Cited by: Table 1, Table 1. Liu et al. (2021a) F. Liu, Y. Wang, T. Wang, and V. Ordonez Visual news: benchmark and challenges in news image captioning. In Proceedings of the 2021 conference on empirical methods in natural language processing, p. 6761â6771. Cited by: Table 1, Table 1. Liu et al. (2023) S. Liu, W. Feng, T. Fu, W. Chen, and W. Wang Edis: entity-driven image search over multimodal web content. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 4877â4894. Cited by: Table 1. Liu et al. (2025) Y. Liu, Y. Zhang, J. Cai, X. Jiang, Y. Hu, J. Yao, Y. Wang, and W. Xie Lamra: large multimodal model as your advanced retrieval assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 4015â4025. Cited by: §1, §2.1, §4.1.1, §4.1.2, §5.1.1, §5.1.3, §5.1.4. Liu et al. (2021b) Z. Liu, C. Rodriguez-Opazo, D. Teney, and S. Gould Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the IEEE/CVF international conference on computer vision, p. 2125â2134. Cited by: Table 1. Mehta et al. (2023) S. V. Mehta, J. Gupta, Y. Tay, M. Dehghani, V. Q. Tran, J. Rao, M. Najork, E. Strubell, and D. Metzler Dsi++: updating transformer memory with new documents. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 8198â8213. Cited by: §1, §2.2.1. Mekonnen et al. (2025) K. A. Mekonnen, Y. Tang, and M. de Rijke Lightweight and direct document relevance optimization for generative information retrieval. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 1327â1338. Cited by: §1, §2.2.1, §2.2.1. Metzler et al. (2021) D. Metzler, Y. Tay, D. Bahri, and M. Najork Rethinking search: making domain experts out of dilettantes. In Acm sigir forum, Vol. 55, p. 1â27. Cited by: §1. Nam et al. (2017) H. Nam, J. Ha, and J. Kim Dual attention networks for multimodal reasoning and matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 299â307. Cited by: §2.1. Oord et al. (2018) A. v. d. Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §4.1.2. Pradeep et al. (2023) R. Pradeep, K. Hui, J. Gupta, A. Lelkes, H. Zhuang, J. Lin, D. Metzler, and V. Tran How does generative retrieval scale to millions of passages?. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, p. 1305â1321. Cited by: §1, §2.2.1. Qiao et al. (2023) S. Qiao, X. Liu, and S. Na DiffusionRet: diffusion-enhanced generative retriever using constrained decoding. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 9515â9529. Cited by: §1, §2.2.1, §2.2.1. Qu et al. (2025) L. Qu, H. Li, T. Wang, W. Wang, Y. Li, L. Nie, and T. Chua Tiger: unifying text-to-image generation and retrieval with large multimodal models. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §2.2.2. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748â8763. Cited by: §2.1, §5.1.4. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21 (140), p. 1â67. Cited by: §5.1.3. Rajput et al. (2023) S. Rajput, N. Mehta, A. Singh, R. Hulikal Keshavan, T. Vu, L. Heldt, L. Hong, Y. Tay, V. Tran, J. Samost, et al. Recommender systems with generative retrieval. Advances in Neural Information Processing Systems 36, p. 10299â10315. Cited by: §4.2.1. Razavi et al. (2019) A. Razavi, A. Van den Oord, and O. Vinyals Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems 32. Cited by: §4.2.1, §5.1.3. Robertson et al. (1994) S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, M. Gatford, et al. Okapi at trec. Cited by: §1. Si et al. (2024) Z. Si, Z. Sun, J. Chen, G. Chen, X. Zang, K. Zheng, Y. Song, X. Zhang, J. Xu, and K. Gai Generative retrieval with semantic tree-structured identifiers and contrastive learning. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, p. 154â163. Cited by: §1, §2.2.1, §2.2.1. Singh et al. (2022) A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela Flava: a foundational language and vision alignment model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15638â15650. Cited by: §2.1. Sparck Jones (1972) K. Sparck Jones A statistical interpretation of term specificity and its application in retrieval. Journal of documentation 28 (1), p. 11â21. Cited by: §1. Sun et al. (2023) W. Sun, L. Yan, Z. Chen, S. Wang, H. Zhu, P. Ren, Z. Chen, D. Yin, M. Rijke, and Z. Ren Learning to tokenize for generative retrieval. Advances in Neural Information Processing Systems 36, p. 46345â46361. Cited by: §1, §2.2.1, §2.2.1. Tang et al. (2025) Y. Tang, R. Zhang, J. Guo, M. de Rijke, S. Liu, S. Wang, D. Yin, and X. Cheng Generative retrieval for book search. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD â25, p. 2606â2617. External Links: ISBN 9798400712456 Cited by: §1, §2.2.1. Tang et al. (2024) Y. Tang, R. Zhang, J. Guo, M. Rijke, W. Chen, and X. Cheng Generative retrieval meets multi-graded relevance. Advances in Neural Information Processing Systems 37, p. 72790â72817. Cited by: §1, §2.2.1. Tay et al. (2022) Y. Tay, V. Tran, M. Dehghani, J. Ni, D. Bahri, H. Mehta, Z. Qin, K. Hui, Z. Zhao, J. Gupta, et al. Transformer memory as a differentiable search index. Advances in Neural Information Processing Systems 35, p. 21831â21843. Cited by: §1, §2.2.1. Tschannen et al. (2023) M. Tschannen, B. Mustafa, and N. Houlsby Clippo: image-and-language understanding from pixels only. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11006â11017. Cited by: §2.1. Wang et al. (2024) P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, et al. Qwen2-vl: enhancing vision-language modelâs perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Cited by: §5.1.3. Wang et al. (2022) Y. Wang, Y. Hou, H. Wang, Z. Miao, S. Wu, Q. Chen, Y. Xia, C. Chi, G. Zhao, Z. Liu, et al. A neural corpus indexer for document retrieval. Advances in Neural Information Processing Systems 35, p. 25600â25614. Cited by: §1, §2.2.1. Wang et al. (2023) Z. Wang, Y. Zhou, Y. Tu, and Z. Dou Novo: learnable and interpretable document identifiers for model-based ir. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, p. 2656â2665. Cited by: §1, §2.2.1, §2.2.1. Wei et al. (2024) C. Wei, Y. Chen, H. Chen, H. Hu, G. Zhang, J. Fu, A. Ritter, and W. Chen Uniir: training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, p. 387â404. Cited by: §1, §1, §2.1, §2.2.3, §3, §5.1.1, §5.1.4, Table 1. Wu et al. (2021) H. Wu, Y. Gao, X. Guo, Z. Al-Halah, S. Rennie, K. Grauman, and R. Feris Fashion iq: a new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, p. 11307â11317. Cited by: Table 1. Wu et al. (2025) S. Wu, Z. Ren, X. Xin, J. Yang, M. Zhang, Z. Chen, M. de Rijke, and P. Ren Constrained auto-regressive decoding constrains generative retrieval. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 2429â2440. Cited by: §1, §2.2.1, §2.2.1. Young et al. (2014) P. Young, A. Lai, M. Hodosh, and J. Hockenmaier From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics 2, p. 67â78. Cited by: §5.1.1. Yu et al. (2022) J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu CoCa: contrastive captioners are image-text foundation models. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §2.1. Zeng et al. (2024a) H. Zeng, C. Luo, B. Jin, S. M. Sarwar, T. Wei, and H. Zamani Scalable and effective generative information retrieval. In Proceedings of the ACM Web Conference 2024, p. 1441â1452. Cited by: §1, §2.2.1, §2.2.1. Zeng et al. (2024b) H. Zeng, C. Luo, and H. Zamani Planning ahead in generative retrieval: guiding autoregressive generation through simultaneous decoding. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 469â480. Cited by: §2.2.1, §2.2.1. Zhang et al. (2025a) F. Zhang, X. Liu, X. Jia, Y. Zhang, S. Zhang, X. Li, F. Zhuang, W. Lin, and Z. Zhang Multi-level relevance document identifier learning for generative retrieval. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, p. 10066â10080. External Links: Document Cited by: §2.2.1. Zhang et al. (2024a) P. Zhang, Z. Liu, Y. Zhou, Z. Dou, F. Liu, and Z. Cao Generative retrieval via term set generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, External Links: Document Cited by: §2.2.1. Zhang et al. (2024b) Y. Zhang, T. Zhang, D. Chen, Y. Wang, Q. Chen, X. Xie, H. Sun, W. Deng, Q. Zhang, F. Yang, et al. Irgen: generative modeling for image retrieval. In European Conference on Computer Vision, p. 21â41. Cited by: §2.2.2, §5.1.4, Table 4, Table 4. Zhang et al. (2025b) Z. Zhang, X. Ma, W. Sun, P. Ren, Z. Chen, S. Wang, D. Yin, M. de Rijke, and Z. Ren Replication and exploration of generative retrieval over dynamic corpora. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 3325â3334. Cited by: §1, §2.2.1, §2.2.1. Zheng et al. (2020) Z. Zheng, L. Zheng, M. Garrett, Y. Yang, M. Xu, and Y. Shen Dual-path convolutional image-text embeddings with instance loss. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16 (2), p. 1â23. Cited by: §2.1. Zhuang et al. (2023) S. Zhuang, H. Ren, L. Shou, J. Pei, M. Gong, G. Zuccon, and D. Jiang Bridging the gap between indexing and retrieval for differentiable search index with query generation. Proceedings of the First Workshop on Generative Information Retrieval (Gen-IR 2023). Cited by: §1, §2.2.1. Appendix Appendix A M-BEIR Instructions This section shows the natural-language instructions used for each retrieval task type over M-BEIR. These instructions specify the retrieval intent and target modality, allowing a unified model to handle heterogeneous queryâcandidate formats, such as text, image, and imageâtext pairs. Table 9 summarizes the instruction templates used in our experiments. Table 9. Summary of the M-BEIR instructions. Task Dataset Instruction qtâciq^tâ c^i VisualNews Identify the news-related image in line with the described event. Display an image that best captures the following caption from the news. Based on the caption, provide the most fitting image for the news story. I want you to retrieve an image of this news caption. MSCOCO Find me an everyday image that matches the given caption. Identify the image showcasing the described everyday scene. I want to retrieve an image of this daily life description. Show me an image that best captures the following common scene description. Fashion200K Based on the following fashion description, retrieve the best matching image. Match the provided description to the correct fashion item photo. Identify the fashion image that aligns with the described product. You need to identify the image that corresponds to the fashion product description provided. qtâctq^tâ c^t WebQA Retrieve passages from Wikipedia that provide answers to the following question. You have to find a Wikipedia paragraph that provides the answer to the question. I want to find an answer to the question. Can you find some snippets that provide evidence from Wikipedia? Iâm looking for a Wikipedia snippet that answers this question. qtâ(ci,ct)q^tâ(c^i,c^t) EDIS Find a news image and its caption that match the provided caption. Identify the news photo and its caption for the given caption. Can you pair the provided caption with the right image and its caption? Iâm looking for an image and its caption that aligns with the provided caption. WebQA Find a Wikipedia image and its description that answers the given question. Provide me with an image and its description from Wikipedia to answer the given question. I want to know the answer to the given question. Please find the related Wikipedia image and its description for me. You need to retrieve an evidence image with its description from Wikipedia to address the given question. qiâctq^iâ c^t VisualNews Find a caption for the news in the given photo. Based on the shown image, retrieve an appropriate news caption. Provide a news-related caption for the displayed image. I want to know the caption for this news image. MSCOCO Find an image caption describing the following everyday image. Retrieve the caption for the displayed day-to-day image. Can you find a caption talking about this daily life image? I want to locate the caption that best describes this everyday scene image. Fashion200K Find a product description for the fashion item in the image. Based on the displayed image, retrieve the corresponding fashion description. Can you retrieve the description for the fashion item in the image? I want to find a matching description for the fashion item in this image. qiâciq^iâ c^i NIGHTS Find a day-to-day image that looks similar to the provided image. Which everyday image is the most similar to the reference image? Find a daily life image that is identical to the given one. You need to identify the common scene image that aligns most with this reference image. (qi,qt)âct(q^i,q^t)â c^t OVEN Retrieve a Wikipedia paragraph that provides an answer to the given query about the image. Determine the Wikipedia snippet that identifies the visual entity in the image. I want to find a paragraph from Wikipedia that answers my question about this image. You have to find a Wikipedia segment that identifies this imageâs subject. InfoSeek Retrieve a Wikipedia paragraph that provides an answer to the given query about the image. Determine the Wikipedia snippet that matches the question of this image. I want to find a paragraph from Wikipedia that answers my question about this image. You have to find a Wikipedia segment that answers the question about the displayed image. (qi,qt)âci(q^i,q^t)â c^i FashionIQ Find a fashion image that aligns with the reference image and style note. With the reference image and modification instructions, find the described fashion look. Given the reference image and design hint, identify the matching fashion image. Iâm looking for a similar fashion product image with the described style changes. CIRR Retrieve a day-to-day image that aligns with the modification instructions of the provided image. Pull up a common scene image like this one, but with the modifications I asked for. Can you help me find a daily image that meets the modification from the given image? Iâm looking for a similar everyday image with the described changes. (qi,qt)â(ci,ct)(q^i,q^t)â(c^i,c^t) OVEN Retrieve a Wikipedia image-description pair that provides evidence for the question of this image. Determine the Wikipedia image-snippet pair that clarifies the entity in this picture. I want to find an image and subject description from Wikipedia that answers my question about this image. I want to know the subject in the photo. Can you provide the relevant Wikipedia section and image? InfoSeek Retrieve a Wikipedia image-description pair that provides evidence for the question of this image. Determine the Wikipedia image-snippet pair that matches my question about this image. I want to find an image and subject description from Wikipedia that answers my question about this image. I want to address the query about this picture. Please pull up a relevant Wikipedia section and image.