Paper deep dive
DEPT: Document Embedding Preservation Tuning for Unified Query Expansion and Retrieval
Jingyuan Wang, Richong Zhang, Zhijie Nie, Mingxin Li, Yanzhao Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/19/2026, 5:14:40 AM
Summary
The paper introduces DEPT (Document Embedding Preservation Tuning), a method for training a single decoder-only Large Language Model (LLM) end-to-end for both query expansion and dense retrieval. DEPT addresses the 'moving-target' problem where updating the model shifts document embeddings, by using a Document Embedding Preservation (DEP) loss to keep document embeddings close to cached initial embeddings, combined with a fixed whitening transform. This allows retrieval gradients to improve query-side expansion while maintaining stable retrieval targets, enabling index reuse and online hard-negative mining. Experiments on BEIR datasets show DEPT outperforms training-free, independently trained, and staged unified baselines.
Entities (14)
Relation Signals (10)
DEPT → evaluatedon → BEIR
confidence 95% · Experiments with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct on five datasets in BEIR benchmark show that DEPT improves average retrieval quality
DEPT → evaluatedwithmodel → Qwen3-4B-Instruct-2507
confidence 95% · Experiments with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct on five datasets in BEIR benchmark show that DEPT improves average retrieval quality
DEPT → evaluatedwithmodel → Llama 3.2-3B-Instruct
confidence 95% · Experiments with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct on five datasets in BEIR benchmark show that DEPT improves average retrieval quality
DEPT → uses → DEP Loss
confidence 95% · DEPT uses a Document Embedding Preservation (DEP) loss, fixed whitening, online hard-negative mining, and straight-through expansion training
DEPT → uses → Whitening
confidence 95% · DEPT uses a Document Embedding Preservation (DEP) loss, fixed whitening, online hard-negative mining, and straight-through expansion training
DEPT → solvesproblem → moving-target problem
confidence 92% · This unified setting creates a moving-target problem... We introduce Document Embedding Preservation Tuning (DEPT), which keeps tuned document embeddings close to cached initial embeddings
DEPT → outperforms → Promptagator
confidence 90% · DEPT improves average retrieval quality over training-free, independently trained, and staged unified baselines
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) can both expand underspecified queries and encode text as dense representations, suggesting a unified model for query expansion and retrieval. Existing systems usually rely on prompted expansions, independently trained modules, or staged optimization, leaving generated expansions only indirectly aligned with the retrieval loss that judges them. We train a single decoder-only LLM end to end, where the same model generates the expansion and encodes both the expanded query and candidate documents. This unified setting creates a moving-target problem: retrieval supervision should improve query-side expansion, but the same update also shifts the document embeddings that serve as retrieval targets. We introduce Document Embedding Preservation Tuning (DEPT), which keeps tuned document embeddings close to cached initial embeddings while allowing retrieval gradients to pass through straight-through decoding into the generator. DEPT converts joint query--document movement into query-side adaptation against approximately stable, whitened document embeddings that support index reuse and online hard-negative mining. Experiments with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct on five datasets in BEIR benchmark show that DEPT improves average retrieval quality over training-free, independently trained, and staged unified baselines, while ablations isolate the effects of preservation, whitening, end-to-end expansion training, and online negatives. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.17632v1
- Canonical: https://arxiv.org/abs/2608.17632v1
Trouble viewing inline? Open PDF directly →
Full Text
43,742 characters extracted from source content.
Expand or collapse full text
DEPT: Document Embedding Preservation Tuning for Unified Query Expansion and Retrieval Jingyuan Wang Richong Zhang Zhijie NieMingxin Li, Yanzhao Zhang Abstract Large language models (LLMs) can both expand underspecified queries and encode text as dense representations, suggesting a unified model for query expansion and retrieval. Existing systems usually rely on prompted expansions, independently trained modules, or staged optimization, leaving generated expansions only indirectly aligned with the retrieval loss that judges them. We train a single decoder-only LLM end to end, where the same model generates the expansion and encodes both the expanded query and candidate documents. This unified setting creates a moving-target problem: retrieval supervision should improve query-side expansion, but the same update also shifts the document embeddings that serve as retrieval targets. We introduce Document Embedding Preservation Tuning (DEPT), which keeps tuned document embeddings close to cached initial embeddings while allowing retrieval gradients to pass through straight-through decoding into the generator. DEPT converts joint query–document movement into query-side adaptation against approximately stable, whitened document embeddings that support index reuse and online hard-negative mining. Experiments with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct on five datasets in BEIR benchmark show that DEPT improves average retrieval quality over training-free, independently trained, and staged unified baselines, while ablations isolate the effects of preservation, whitening, end-to-end expansion training, and online negatives. Code is available at https://github.com/ILSparkle/DEPT. Introduction Query expansion and dense retrieval solve two complementary parts of retrieval (3; 13; 16). Expansion rewrites an underspecified query into text that exposes missing entities, constraints, or topical cues, while dense retrieval maps the expanded query and documents into a vector space where relevant evidence can be found efficiently. The combination is attractive because expansion makes the query more explicit and the dense retriever gives the expanded query a scalable search mechanism. The main obstacle is that these two parts are often optimized through different signals: expansion is judged by textual plausibility or indirect retrieval feedback, whereas the final system is judged by whether the resulting representation ranks the right documents. Figure 1: Qwen3-4B-Instruct-2507 retrieval quality versus expansion length. Stars mark our methods: DEPT has the best average nDCG@1010, while DEPT-K gives the short-expansion trade-off. Figure 2: Alternative routes for adapting a generative LLM to unified expansion and retrieval. The base model produces useful expansions but weak retrieval representations. Conventional contrastive learning updates both query and document representations, so retrieval improves substantially but degraded expansions can limit the final gain. DEPT instead adapts the query side while preserving document representations, retaining expansion quality and yielding stronger retrieval. LLMs make this mismatch more visible and more promising to address. The same model family can produce fluent expansion text and provide dense representations, so one might expect a single decoder-only LLM to learn expansions directly from the retrieval loss that evaluates them. Such a unified model would have two practical advantages. First, expansion would become a trainable retrieval action rather than a prompted preprocessing step. Second, the generated text would remain inspectable, so the retrieval decision could still be examined through the model’s textual output rather than only through an opaque vector. Existing LLM-based expansion systems move toward this goal but stop short of the end-to-end objective. Training-free methods prompt an LLM to produce pseudo-documents or query augmentations, leaving the generator unchanged even when the downstream retriever fails (9; 28). Independent-training methods such as InPars (1) and Promptagator (6) improve the training signal, but the generator and retriever are still optimized through separate interfaces. Sequential systems use retrieval feedback or preference-style supervision, as in ExpandR (32), yet the generator is not optimized by the final contrastive loss of the deployed retriever. Staged unified methods such as UniRAG (15) reduce architectural separation, but augmentation and representation remain separate training phases. The common limitation is therefore not that generated text is useless; it is that the text-producing component is not continuously shaped by the same retrieval objective and embedding geometry that determine final ranking. Directly training the unified model, however, creates a different failure mode. Figure 2 illustrates the choice. Before retrieval training, a base LLM can already produce useful expansions, but its dense retrieval representations are weak. Conventional contrastive learning fixes the representation problem by moving expanded queries toward relevant documents and away from irrelevant ones. Because the same parameters also define document embeddings and generation behavior, the update can improve ranking while changing the retrieval targets and the expansion policy at the same time. The outcome is a bottleneck rather than a complete failure: retrieval still improves, but the final gain is limited when expansion drifts away from the fluent explanatory behavior that made LLM augmentation useful in the first place. This failure mode suggests that unified expansion and retrieval should be asymmetric. The query side should remain plastic because the generated tokens and expanded-query representation are the parts that need to learn from retrieval feedback. The document side should instead serve as a stable target because it defines the contrastive comparison during training. As a practical benefit, the same stability can also support cached indexing at inference. With relatively fixed document targets, the retrieval objective places clearer pressure on the query path and expansion behavior. Without that stability, the same objective can be satisfied by moving documents rather than by improving the generated query. We introduce Document Embedding Preservation Tuning (DEPT) to implement this asymmetric training principle. DEPT keeps current document embeddings close to cached embeddings from the initial model, while allowing retrieval gradients to update expansion and the expanded-query representation. This preservation makes two additional design choices useful rather than fragile. Since raw LLM embeddings can be anisotropic and poorly calibrated for cosine retrieval (7), we fit a fixed whitening transform on cached document embeddings and train in the resulting coordinate system. Because document embeddings remain close to this reference, the same cached index can support online hard-negative mining during training and index reuse at inference. Figure 1 previews the resulting trade-off: the long-expansion model reaches the best average retrieval quality, while the keyword-style variant remains competitive with very short expansions. The main contributions of this work can be summarized as follows. First, we formulate query expansion and dense retrieval as a unified decoder-only LLM training problem, and identify document-embedding drift as the stability obstacle that prevents ordinary contrastive learning from fully exploiting expansion. Second, we propose Document Embedding Preservation Tuning (DEPT), which uses a Document Embedding Preservation (DEP) loss, fixed whitening, online hard-negative mining, and straight-through expansion training to make retrieval gradients act primarily on the query side. Third, experiments on two LLM backbones and five retrieval tasks show that DEPT achieves state-of-the-art average retrieval quality among the compared query-expansion paradigms while preserving generation ability and cached-index compatibility. Figure 3: Overview of DEPT training. A single decoder-only LLM handles expansion and retrieval under an end-to-end retrieval objective. The lower panels detail the three mechanisms: straight-through expansion passes retrieval gradients to generated tokens, the DEP loss preserves current document embeddings near cached document embeddings from the initial model, and a fixed whitening transform maps embeddings into a better-conditioned retrieval space. Related Work LLM-Based Query Expansion. HyDE encodes a generated hypothetical document to bridge a zero-shot query–document gap (9), while Query2Doc prompts an LLM to generate a pseudo-document that augments the original query (28). ExpandR uses LLM guidance to train a retriever beyond the literal query and aligns generated augmentations with retrieval preferences (32). UniRAG is the closest staged unified framework: it uses a decoder-only LLM for both query augmentation and representation, but optimizes augmentation and encoding through separate stages (15). These methods show that generated text can improve retrieval, but generation is usually optimized outside the final contrastive retrieval loss or through indirect feedback. They leave open how to stabilize retrieval targets when one shared model handles generation and encoding. Generative and Representational LLMs. Instruction-tuned LLM embeddings show that generative models can produce strong text representations (27). GritLM jointly trains generative and embedding objectives in a single model (18), establishing the broader feasibility of unifying generation and representation. Our setting adds a retrieval-specific constraint that is less visible in generic multitask formulations: corpus embeddings are retrieval targets during training and can be expensive to refresh in deployment. With one model for generation and embedding, shared updates can change both query behavior and document embeddings, so end-to-end expansion training must control target drift rather than treating document movement as harmless. Representation Geometry and Stability. Transformer representations are known to be anisotropic (7), and whitening can improve semantic similarity and retrieval by centering and decorrelating sentence embeddings (21). Whitening is commonly used as a post-processing transform for a fixed embedding model. In trainable retrieval systems, the transformed space remains valid only if embeddings stay close to the distribution used to estimate the transform. Method Preliminary We first describe the ordinary expansion-and-retrieval pipeline before introducing training. Given a query qiq_i, an expansion module generates a text sequence yi=(yi,1,…,yi,T)y_i=(y_i,1,…,y_i,T), where T is the expansion length. The retriever then encodes the concatenated text [qi;yi][q_i;y_i] and each document d from a corpus D into a shared vector space, and ranks documents by similarity to the expanded query. Let θ denote the parameters of the encoder used by the retriever. For any token sequence x, the model produces final-layer hidden states H(x)=(h1(x),…,h|x|(x))H(x)=(h_1(x),…,h_|x|(x)). We use one raw embedding function for both sides, defined by mean pooling followed by ℓ2 _2 normalization: h¯(x) h(x) =1|x|∑t=1|x|ht(x), = 1|x| _t=1^|x|h_t(x), (1) fθ(x) f_θ(x) =h¯(x)‖h¯(x)‖2. = h(x)\| h(x)\|_2. Given a generated expansion yiy_i, we abbreviate the raw expanded-query embedding and the raw embedding of document d as i=fθ([qi;yi]),=fθ(d),q_i=f_θ([q_i;y_i]), =f_θ(d), (2) where [qi;yi][q_i;y_i] denotes concatenation of the original query and its expansion. At inference, retrieval uses a score such as i⊤q_i d to rank documents. Existing methods differ mainly in how the expansion module and retriever are obtained: the expansion may be prompted, trained separately, or trained in a stage before the final retriever. In all cases, the basic interface is the same: generated text changes the query representation, and the retriever scores it against document embeddings. Figure 3 summarizes the resulting training graph and highlights the three mechanisms used by DEPT: straight-through expansion, document embedding preservation, and fixed whitening. Document Embedding Preservation Tuning DEPT uses one decoder-only LLM for both abilities in the pipeline: it generates the expansion and also encodes expanded queries and documents. This single-model formulation is important because the expansion and representation are expressed by the same parameters, but it also creates a conflict between two roles of the model. The query side should change so that generated expansions become better retrieval actions. The document side should remain stable because it defines the retrieval targets used by the contrastive loss; cached-index reuse is a downstream benefit. This conflict is asymmetric rather than a generic preference for small updates. The query-side generator is conditioned on a retrieval instruction and the input query, and the query representation is computed from the expanded input [qi;yi][q_i;y_i]. This path should be allowed to adapt, because retrieval supervision must teach the model what kind of expansion improves document ranking. In contrast, documents are fed to the same model as plain document text, and their embeddings serve as the retrieval targets against which all expanded queries are judged. This asymmetry also helps preserve generation: plain-document encoding shares the same token embeddings and transformer blocks used by document-conditioned language modeling, so keeping document embeddings close to their cached initial embeddings acts as lightweight functional rehearsal while the instruction-conditioned query path remains free to adapt. The conflict appears directly in the raw query–document score r(qi,yi,d)=i⊤r(q_i,y_i,d)=q_i d. Because query and document embeddings share parameters, this score has the gradient ∇θr(qi,yi,d) _θr(q_i,y_i,d) =(∇θi)⊤ = ( _θq_i ) d (3) +i⊤(∇θ). +q_i ( _θd ). The first term is the desired query-side update: it teaches the expansion and expanded-query representation to align with relevant documents. The second term moves the document representation, creating query–document gradient interference. If both sides are left unconstrained, the model can reduce retrieval loss by reorganizing document embeddings around the current minibatch rather than learning expansions that transfer to a stable retrieval target. DEPT resolves this single-model, dual-ability conflict by keeping document embeddings close to their cached references while leaving the query path trainable. This makes query optimization more consistent across steps: target document embeddings change slowly, so retrieval gradients are directed toward improving the expansion and query representation instead of chasing a moving corpus representation. Freezing a separate document encoder would stabilize targets but break the unified formulation. We instead regularize the quantity that matters to retrieval: the raw document embedding d produced from each document d. Let θ0 _0 denote the initial model before DEPT tuning. Before training, we precompute a cached reference embedding for each training document: 0=fθ0(d).d^0=f_ _0(d). (4) These cached embeddings are not updated during training. For any training document d, define its angular drift from the cached reference embedding as δ(d)=1−cos(,0).δ(d)=1- \! (d,d^0 ). (5) Controlling this drift keeps retrieval feedback tied to an adapting expanded query rather than to arbitrarily moving target document embeddings. In a training step with B queries, let iC_i be the candidate documents for query qiq_i, and let i⊆iS_i _i be the documents whose current embeddings are computed, including the positive document and sampled negatives. For scale s>0s>0 and exponent p>1p>1, we define the DEP loss as ℒDEP=1B∑i=1B1|i|∑d∈i(sδ(d))p.L_DEP= 1B _i=1^B 1|S_i| _d _i (s\,δ(d) )^p. (6) Here λ>0λ>0 will control the preservation strength in the final objective. The scale s keeps small cosine deviations numerically visible, while p>1p>1 emphasizes large departures. Reference document embeddings 0d^0 receive no gradient. Whitening Preserved Document Embeddings. The DEP loss preserves raw document embeddings, but retrieval still benefits from a better-conditioned coordinate system. Because decoder-only LLM embeddings can be anisotropic, using the raw embedding geometry inherits its defects. Figure 4 illustrates how whitening reduces dominant directional variance before cosine retrieval. Figure 4: Schematic illustration of whitening, visualized by projecting embeddings onto the top two principal components via PCA. Raw cached document embeddings can concentrate along dominant directions; the fixed whitening transform maps them into a better-conditioned retrieval space. We therefore estimate a fixed whitening transform from cached reference document embeddings and keep this transform fixed throughout tuning. This design is tied to preservation: whitening improves the structure of the initial document embedding distribution, and the DEP loss keeps later document embeddings close enough for the same coordinate system to remain useful. For independently trained baselines whose document embeddings continue to move, a whitening transform fitted before training would gradually cease to describe the current embedding distribution and would behave mainly as an extra fixed linear map. Let μ be the empirical mean of cached document embeddings 0d^0, and let UΛU⊤U U be the eigendecomposition of their covariance matrix. With small constant ε>0 >0 and whitening strength α∈[0,1]α∈[0,1], we define W=U(Λ+εI)−α.W=U( + I)^-α. (7) Here I is the identity matrix, and ε prevents unstable inversion of small eigenvalues. Any raw embedding z is mapped into the fixed whitened coordinate system by z^=norm(W⊤(z−μ)). z=norm\! (W (z-μ) ). (8) We write ^i q_i for the whitened version of iq_i, and d for the whitened version of d. Since (μ,W)(μ,W) is fitted from cached documents before tuning, whitening remains meaningful only if the DEP loss keeps current document embeddings close to their cached references. End-to-End Expansion and Retrieval Training With stable document embeddings in place, the retrieval objective can be used to train expansion and representation jointly. The role of DEPT is to make this end-to-end signal selective: retrieval gradients may update expansion logits and the expanded-query representation, while the DEP loss prevents the same updates from turning retrieval into a moving-target problem on the document side. The training set contains triples (qi,di+,i−)(q_i,d_i^+,D_i^-), where qiq_i is the input query, di+d_i^+ is a relevant document, and i−D_i^- is a pool of non-relevant or mined negative documents. For a minibatch of size B, the candidate set iC_i contains di+d_i^+, in-batch documents, and sampled negatives. The shared model first generates yiy_i from the retrieval instruction and query, then encodes [qi;yi][q_i;y_i] and every document in iC_i. Hard token selection would sever the path between the retrieval loss and the expansion logits. We retain discrete expansions in the forward pass while using a differentiable approximation in the backward pass. For one generation step, let ℓ(v) (v) be the logit of vocabulary token v∈v , let K be the top-k tokens under that logit vector, and let E(v)E(v) be the embedding of token v. We compute π(v) π(v) =exp(ℓ(v))∑u∈exp(ℓ(u)),v∈, = ( (v)) _u ( (u)), v , (9) e~ e =∑v∈π(v)E(v), = _v π(v)E(v), (10) eST e^ST =e~+sg(E(y^)−e~), = e+sg\! (E( y)- e ), (11) where y^=argmaxv∈ℓ(v) y= _v (v) is the greedy token and sg(⋅)sg(·) stops gradients. Thus, eSTe^ST equals the greedy token embedding in the forward computation, while its gradient is that of the top-k soft embedding. Applying this step at each generation position gives (ei,1ST,…,ei,TST)(e_i,1^ST,…,e_i,T^ST). The query encoder consumes these expansion embeddings together with the original query, allowing retrieval gradients to update both the representation and the logits that produced the expansion. This path trains the unified generator–encoder interface: an expansion is useful only if its embedding ranks the relevant document above candidates in iC_i. The retrieval loss is the InfoNCE objective in this whitened space: ℒRET=−1B∑i=1Blogexp(^i⊤^i+/τ)∑d∈iexp(^i⊤^/τ),L_RET=- 1B _i=1^B \! ( q_i d_i^+/τ ) _d _i \! ( q_i d/τ ), (12) where ^i+ d_i^+ is the whitened embedding of the positive document di+d_i^+. The final DEPT objective is ℒDEPT=ℒRET+λℒDEP.L_DEPT=L_RET+ _DEP. (13) Here ℒRETL_RET supplies the end-to-end learning signal, while ℒDEPL_DEP keeps that signal focused on query-side expansion and representation learning rather than arbitrary corpus movement. The loss does not match query embeddings to their initialization or freeze parameters; retrieval gradients still traverse query encoding, expansion, and any parameters whose updates keep document embeddings close to their cached references. Leveraging Stable Document Embeddings Index Reuse. Dense retrieval normally encodes the corpus once and stores its vectors in a nearest-neighbor index (13; 30). End-to-end tuning without preservation breaks this serving assumption: even if the tuned model improves retrieval after re-encoding all documents, the index built before tuning may no longer contain vectors compatible with the tuned queries. DEPT is designed to avoid this mismatch. Since document embeddings are kept close to their cached references and whitening is fitted on the same cached document embeddings, the tuned expanded-query embedding can be searched against the reference document index without rebuilding the corpus. This gives a direct deployment benefit for large collections, where document encoding and index construction are often much more expensive than updating the query-side model. We evaluate this compatibility by comparing retrieval with the cached reference index against retrieval after re-encoding documents with the final model. Online Hard-Negative Mining. The stable index also supports training-time hard-negative mining. At each step, the current model produces an expansion and a whitened query embedding ^i q_i. We use this query to search a fixed FAISS index built from cached whitened document embeddings, remove the positive document, and sample negatives from a high-ranking interval. The selected negative documents are then re-encoded by the current model before entering the InfoNCE denominator and the DEP loss. This procedure gives the retrieval loss query-conditioned negatives that track the current expansion behavior, while the expensive corpus index remains fixed. The sampling interval also avoids relying only on the nearest retrieved items, which can include mislabeled positives or overly ambiguous documents, and provides a controlled source of difficult but usable negatives throughout training. Backbone Method Exp. Tok. SciFact ArguAna NFCorpus FiQA SCIDOCS Avg. Qwen3-4B Instruct-2507 Original Query 0.00 66.26 32.68 31.34 32.92 17.48 36.14 Query2Doc 289.18 72.28 31.04 34.28 37.09 19.51 38.84 HyDE 364.50 72.16 38.91 34.16 35.16 18.86 39.85 ExpandR 174.82 72.66 40.74 35.62 37.41 19.02 41.09 UniRAG 56.86 67.28 39.08 33.54 35.35 17.83 38.62 DEPT-K (Ours) 9.15 73.62 40.30 35.81 37.36 20.14 41.45 DEPT (Ours) 161.87 74.35 41.12 36.99 41.66 18.82 42.59 LLaMA-3.2 3B-Instruct Original Query 0.00 65.85 33.98 32.55 31.94 16.60 36.18 Query2Doc 262.51 69.48 34.01 31.22 34.75 19.21 37.73 HyDE 307.38 65.30 34.84 26.89 26.15 15.61 33.76 ExpandR 164.86 69.28 38.02 32.40 34.17 16.96 38.17 UniRAG 59.18 67.73 35.61 30.29 33.57 16.35 36.71 DEPT-K (Ours) 8.98 69.83 37.14 34.45 34.62 17.63 38.73 DEPT (Ours) 148.82 71.36 38.56 35.22 35.53 17.31 39.60 Table 1: Main retrieval results on five BEIR tasks, reported as nDCG@1010. Exp. Tok. is the average number of generated expansion tokens. UniRAG denotes our reimplementation under the same backbone family and retrieval protocol. Within each backbone, the best result is bold and the second-best is underlined. Experiments Experimental Setup Training Data. We train on an ECHO retrieval mixture constructed from eight sources: ELI5 question answering (8), FEVER (23), HotpotQA (31), MS MARCO document retrieval, MS MARCO passage retrieval (19), Natural Questions (14), SQuAD (20), and TriviaQA (12). Each example is converted to an instruction, query, positive document, and negative document in an E5-style format (26). The expansion prompt contains the retrieval instruction and original query; we use a keyword-style prompt for DEPT-K and a longer explanatory prompt for DEPT. Evaluation Tasks and Metric. We evaluate zero-shot retrieval on five datasets in BEIR benchmark (22): SciFact (25), ArguAna (24), NFCorpus (2), FiQA (17), and SCIDOCS (5). We report nDCG@1010 (11) for every task and the unweighted average across tasks. Backbones and Baselines. We instantiate every method with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct. Training-free baselines include the original query, HyDE (9), and Query2Doc (28); HyDE and Query2Doc use the untrained instruction model only for expansion and the corresponding contrastive-tuned LLM retriever for encoding. ExpandR (32) represents independent expansion-and-retrieval training with separate same-backbone decoder and encoder instances. We also include UniRAG as a staged unified baseline, using our same-backbone reimplementation in which a distilled augmenter supplies text to a subsequently trained retriever but receives no final retrieval gradient. All methods use the same evaluation collections and metric. Implementation Details. We use mean pooling and ℓ2 _2 normalization for all raw embeddings. DEPT-K uses a keywords-style prompt with a max-32 output limit, while DEPT uses a passage-style prompt with a 128–512-token output range; realized expansion lengths are in Table 1. Table 2 summarizes the main hyperparameters. Unless stated otherwise, all reported results are from a single run, and preprocessing, sampling, and training use seed 42. Training uses LoRA adaptation (10) and is conducted on four NVIDIA RTX PRO 6000 GPUs with 96GB memory per GPU. Hyperparameter DEPT-K DEPT Expansion prompt Keywords-style Passage-style Output limit Max 32 tokens 128–512 tokens Input length 1,024 tokens ST decoding Top-6464 LoRA r=16r=16, α=32α=32, dropout 0.050.05 Training 1k steps, lr 10−410^-4, batch 256 Retrieval temp. 0.020.02 DEP loss λ=0.1λ=0.1, s=100s=100, p=2p=2 Whitening 1,024 cached docs, α=0.5α=0.5 Online mining Rank-5 negative Table 2: Main hyperparameters for DEPT variants. Shared values use merged cells. Results and Analysis The experiments examine five aspects: main retrieval quality, component causality, cached-index compatibility, general generation behavior, and qualitative expansion behavior. We distinguish DEPT-K, which generates short keyword-style expansions, from long-expansion DEPT. Main Results. Table 1 evaluates whether expansion improves when retrieval supervision trains the generator and encoder in one unified objective rather than through prompting, independent objectives, or staged unification. Long-expansion DEPT obtains the best average score with both backbones, while DEPT-K remains competitive using roughly nine generated tokens. Figure 1 shows the same trade-off on Qwen: DEPT is the highest-quality point, and DEPT-K lies on the short-expansion frontier. The gains are not a simple length effect. Training-free methods generate longer text, but it is not optimized for the embedding-based scorer that ranks it. DEPT lets the final retrieval loss update the expansion path inside the shared model; SCIDOCS remains the boundary case, where concise topical cues can be stronger than long semantic expansions. Ablation Study. Table 3 isolates DEPT components on Qwen. Removing the DEP loss hurts even when documents are re-encoded, showing that preservation improves optimization beyond cached-index reuse. Whitening is the largest individual factor, and removing expansion or detaching it as an offline input loses the benefit of straight-through training. Overall, the DEP loss and whitening stabilize document embeddings, while expansion training and hard negatives teach the query side to use those stable targets. Variant SciFact ArguAna NFCorpus Avg. DEPT (Ours) 74.35 41.12 36.99 50.82 w/o DEP Loss 71.88 35.75 33.21 46.95 w/o Whitening 66.31 30.58 29.37 42.09 w/o Online Neg. 72.79 37.81 34.20 48.27 w/ Detached Expansion 72.33 37.25 32.37 47.32 w/o Expansion 65.35 31.94 31.03 42.77 Table 3: Ablations on three BEIR tasks with Qwen3-4B-Instruct-2507, reported as nDCG@1010. Figure 5: Cached-index compatibility on Qwen3-4B-Instruct-2507. Arrows show movement from re-encoded to cached-index retrieval; thick arrows are averages and pale arrows are individual datasets. Index Reuse. Figure 5 tests whether a pre-tuning document index remains usable after end-to-end training. With DEPT, cached-index retrieval stays close to re-encoding all documents with the final model. Without the DEP loss, the original index no longer contains valid keys for the tuned document encoder; preservation prevents this failure mode while allowing the query path and LoRA parameters to adapt. Generation Ability. Figure 6 checks generation on three benchmarks (4; 33; 29). DEPT stays close to the base model, decreasing from 77.56 to 76.28 on average, whereas standard contrastive learning (CL) collapses to 9.32. Thus preservation improves retrieval without turning the same decoder into a narrow embedding-only model. Figure 6: General generation evaluation on GSM8K, IFEval, MMLU-Pro, and their average. CL denotes standard contrastive learning on the unified model. Case Study. Table 4 illustrates the expansion behavior behind the aggregate trends. The base model mentions relevant biomedical mechanisms but does not select them as retrieval cues, whereas standard CL and UniRAG produce degraded or repetitive strings. DEPT-K compresses the query into high-value cues, and full DEPT preserves the same mechanism in a readable explanatory form. This pattern matches the quantitative results: preservation lets the model improve retrieval without discarding the expansion behavior that makes LLM-based retrieval inspectable. Query: Starving Tumors of Their Blood Supply Method Generated expansion excerpt Base … cutting off the nutrients and oxygen that tumors need … anti-angiogenic drugs … vascular endothelial growth factor … CL keywords, Starving Task, Task, Starvation, StarvStarving UniRAG … Starving tumors Starving Starving tumor Star Star Star tumors … DEPT-K Starving tumors blood supply; tumor hypoxia mechanism; angiogenesis inhibition DEPT … deprive tumors of the oxygen and nutrients they need … anti-angiogenic drugs … angiogenesis … Table 4: Expansion excerpts on one NFCorpus query; three periods mark omitted text. Conclusion We studied unified LLM-based query expansion and dense retrieval, where the same decoder-only model generates retrieval-oriented expansions and encodes the candidate documents that judge them. The central issue is a moving-target training dynamic: ordinary contrastive updates can improve local retrieval scores while drifting document embeddings and degrading the generator that produces the expansion. DEPT addresses this problem with document embedding preservation, fixed whitening, online hard negatives, and straight-through expansion training. Across two LLM backbones and five BEIR tasks, it improves retrieval while preserving generation and cached-index compatibility, showing that unified generative and representational retrieval benefits from explicitly stabilizing document embeddings. This points to a simple design principle: train the query behavior aggressively, but keep document embeddings reusable. References Bonifacio et al. (2022) L. H. Bonifacio, H. Abonizio, M. Fadaee, and R. Nogueira InPars: unsupervised dataset generation for information retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 2387–2392. External Links: Document Cited by: Introduction. Boteva et al. (2016) V. Boteva, D. Gholipour, A. Sokolov, and S. Riezler A full-text learning to rank dataset for medical information retrieval. In Proceedings of the 38th European Conference on Information Retrieval, External Links: Link Cited by: Evaluation Tasks and Metric.. Carpineto and Romano (2012) C. Carpineto and G. Romano A survey of automatic query expansion in information retrieval. ACM Computing Surveys 44 (1), p. 1:1–1:50. External Links: Document Cited by: Introduction. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. External Links: Link Cited by: Generation Ability.. Cohan et al. (2020) A. Cohan, S. Feldman, I. Beltagy, D. Downey, and D. Weld SPECTER: document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 2270–2282. External Links: Document Cited by: Evaluation Tasks and Metric.. Dai et al. (2023) Z. Dai, V. Y. Zhao, J. Ma, Y. Luan, J. Ni, J. Lu, A. Bakalov, K. Guu, K. B. Hall, and M. Chang Promptagator: few-shot dense retrieval from 8 examples. In International Conference on Learning Representations, External Links: Link Cited by: Introduction. Ethayarajh (2019) K. Ethayarajh How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, p. 55–65. External Links: Document Cited by: Introduction, Representation Geometry and Stability.. Fan et al. (2019) A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli ELI5: long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 3558–3567. External Links: Document Cited by: Training Data.. Gao et al. (2023) L. Gao, X. Ma, J. Lin, and J. Callan Precise zero-shot dense retrieval without relevance labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1762–1777. External Links: Document Cited by: Introduction, LLM-Based Query Expansion., Backbones and Baselines.. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: Implementation Details.. Järvelin and Kekäläinen (2002) K. Järvelin and J. Kekäläinen Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20 (4), p. 422–446. External Links: Document Cited by: Evaluation Tasks and Metric.. Joshi et al. (2017) M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1601–1611. External Links: Document Cited by: Training Data.. Karpukhin et al. (2020) V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, p. 6769–6781. External Links: Document Cited by: Introduction, Index Reuse.. Kwiatkowski et al. (2019) T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7, p. 452–466. External Links: Document Cited by: Training Data.. Li et al. (2025) R. Li, L. He, Q. Liu, Z. Zhang, H. Yu, Y. Ye, L. Zhu, and Y. Su UniRAG: unified query understanding method for retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14163–14178. External Links: Document Cited by: Introduction, LLM-Based Query Expansion.. Lin et al. (2021) J. Lin, R. Nogueira, and A. Yates Pretrained transformers for text ranking: BERT and beyond. Synthesis Lectures on Human Language Technologies, Morgan & Claypool. External Links: Document Cited by: Introduction. Maia et al. (2018) M. Maia, S. Handschuh, A. Freitas, B. Davis, R. McDermott, M. Zarrouk, and A. Balahur W’18 open challenge: financial opinion mining and question answering. In The Web Conference 2018, External Links: Document Cited by: Evaluation Tasks and Metric.. Muennighoff et al. (2025) N. Muennighoff, H. Su, L. Wang, N. Yang, F. Wei, T. Yu, A. Singh, and D. Kiela Generative representational instruction tuning. In International Conference on Learning Representations, External Links: Link Cited by: Generative and Representational LLMs.. Nguyen et al. (2016) T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng MS MARCO: a human generated machine reading comprehension dataset. In NIPS Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches, External Links: Link Cited by: Training Data.. Rajpurkar et al. (2016) P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, p. 2383–2392. External Links: Document Cited by: Training Data.. Su et al. (2021) J. Su, J. Cao, W. Liu, and Y. Ou Whitening sentence representations for better semantics and faster retrieval. External Links: 2103.15316 Cited by: Representation Geometry and Stability.. Thakur et al. (2021) N. Thakur, N. Reimers, A. Rucklé, A. Srivastava, and I. Gurevych BEIR: a heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Advances in Neural Information Processing Systems, Vol. 34, p. 26804–26817. External Links: Link Cited by: Evaluation Tasks and Metric.. Thorne et al. (2018) J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p. 809–819. External Links: Document Cited by: Training Data.. Wachsmuth et al. (2018) H. Wachsmuth, S. Syed, and B. Stein Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 241–251. External Links: Document Cited by: Evaluation Tasks and Metric.. Wadden et al. (2020) D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi Fact or fiction: verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, p. 7534–7550. External Links: Document Cited by: Evaluation Tasks and Metric.. Wang et al. (2022) L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. External Links: Link Cited by: Training Data.. Wang et al. (2024a) L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 11897–11916. External Links: Document Cited by: Generative and Representational LLMs.. Wang et al. (2023) L. Wang, N. Yang, and F. Wei Query2doc: query expansion with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 9414–9423. External Links: Document Cited by: Introduction, LLM-Based Query Expansion., Backbones and Baselines.. Wang et al. (2024b) Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574. External Links: Link Cited by: Generation Ability.. Xiong et al. (2021) L. Xiong, C. Xiong, Y. Li, K. Tang, J. Liu, P. N. Bennett, J. Ahmed, and A. Overwijk Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, External Links: Link Cited by: Index Reuse.. Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2369–2380. External Links: Document Cited by: Training Data.. Yao et al. (2025) S. Yao, P. Huang, Z. Liu, Y. Gu, Y. Yan, S. Yu, and G. Yu ExpandR: teaching dense retrievers beyond queries with LLM guidance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 19036–19054. External Links: Document Cited by: Introduction, LLM-Based Query Expansion., Backbones and Baselines.. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. External Links: 2311.07911 Cited by: Generation Ability..