Paper deep dive
Working Notes on Late Interaction Dynamics: Analyzing Targeted Behaviors of Late Interaction Models
Antoine Edy, Max Conti, Quentin Macé
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/31/2026, 1:40:52 AM
Summary
This paper analyzes the dynamics of Late Interaction retrieval models, specifically focusing on length bias in multi-vector scoring and the similarity distribution beyond the MaxSim operator. The authors demonstrate that causal multi-vector models exhibit a strict monotonic length bias, while bi-directional models mitigate this but remain vulnerable at length extremes. Furthermore, they find no significant similarity trends beyond the top-1 document token, suggesting current models do not effectively leverage deeper token-level similarity distributions.
Entities (5)
Relation Signals (3)
MaxSim → usedin → Late Interaction
confidence 99% · In late-interaction retrieval, the MaxSim score between a query and a chunk is defined as...
jina-embeddings-v4 → exhibits → Length Bias
confidence 95% · The multi-vector causal model (jina-embeddings-v4) disproportionately retrieves false positives that are significantly longer than the relevant documents.
Bi-directional models → mitigates → Length Bias
confidence 90% · Bi-directional multi-vector models successfully dampen the aggressive bias of causal models.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While Late Interaction models exhibit strong retrieval performance, many of their underlying dynamics remain understudied, potentially hiding performance bottlenecks. In this work, we focus on two topics in Late Interaction retrieval: a length bias that arises when using multi-vector scoring, and the similarity distribution beyond the best scores pooled by the MaxSim operator. We analyze these behaviors for state-of-the-art models on the NanoBEIR benchmark. Results show that while the theoretical length bias of causal Late Interaction models holds in practice, bi-directional models can also suffer from it in extreme cases. We also note that no significant similarity trend lies beyond the top-1 document token, validating that the MaxSim operator efficiently exploits the token-level similarity scores.
Tags
Links
- Source: https://arxiv.org/abs/2603.26259v1
- Canonical: https://arxiv.org/abs/2603.26259v1
Trouble viewing inline? Open PDF directly →
Full Text
21,226 characters extracted from source content.
Expand or collapse full text
Working Notes on Late Interaction Dynamics: Analyzing Targeted Behaviors of Late Interaction Models Antoine Edy 1,*,† , Max Conti 1,† and Quentin Macé 1 1 Illuin Technology Abstract While Late Interaction models exhibit strong retrieval performance, many of their underlying dynamics remain understudied, potentially hiding performance bottlenecks. In this work, we focus on two topics in Late Interaction retrieval: a length bias that arises when using multi-vector scoring, and the similarity distribution beyond the best scores pooled by the MaxSim operator. We analyze these behaviors for state-of-the-art models on the NanoBEIR benchmark. Results show that while the theoretical length bias of causal Late Interaction models holds in practice, bi-directional models can also suffer from it in extreme cases. We also note that no significant similarity trend lies beyond the top-1 document token, validating that the MaxSim operator efficiently exploits the token-level similarity scores. Keywords Information Retrieval, Late-interaction, Multi-Vector Retrieval, Causal Encoders, Bi-directional Encoders 1. Introduction Neural late-interaction retrieval models, such as ColBERT [1], use a token-level interaction while computing similarity between text passages. While this approach allows for finer semantic matching between queries and documents, some of its underlying dynamics are yet to be thoroughly studied. In these notes, we analyze two key behaviors that provide elements for a better understanding of Late Interaction performance: (a) Length bias: Causal encoders, when used with multi-vector MaxSim scoring, exhibit a monotonic bias that favors longer chunks, regardless of their true relevance. (b) Similarity distribution: Given a query token, the MaxSim operator is insensitive to similarity scores of document tokens beyond the highest, collapsing information to a single maximum value. We perform small-scale experiments on the NanoBEIR [2] benchmark to bring more insights into how current state-of-the-art models behave along these axes. 2. Length Bias In Multi-Vector Retrieval In this section, we explore the length bias that arises in late-interaction retrieval frameworks. We highlight two key observations: multi-vector causal models appear to suffer from a strict, monotonic length bias, and while bi-directional architectures theoretically avoid this flaw, empirical insights suggest they remain sensitive to length disparities at extreme margins. Details on the experimental setup are available in Appendix A. 2.1. Theoretical Motivation Let a chunk푐be a sequence of tokens represented by contextualized embeddings. In late-interaction retrieval, the MaxSim score [1] between a query 푞 and a chunk 푐 is defined as: Late Interaction Workshop (LIR) @ ECIR 2026, April 02, 2026. Colocated with ECIR 2026. * Corresponding author. † These authors contributed equally. $ antoine.edy@illuin.tech (A. Edy); max.conti@illuin.tech (M. Conti); quentin.mace@illuin.tech (Q. Macé) © 2026 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (C BY 4.0). 푆 푞,푐 = ∑︁ 푖∈[|퐸 푞 |] max 푗∈[|퐸 푐 |] 퐸 푞 푖 · 퐸 푇 푐 푗 where퐸 푞 and퐸 푐 are the respective sets of query and chunk embeddings. When utilizing a causal encoder with a multi-vector representation, appending tokens to a chunk yields a strict superset of embeddings. As a result, the maximum inner product for each query token can only increase or remain constant. This dynamic introduces a theoretical monotonic length bias that artificially favors longer chunks regardless of their true relevance. Bi-directional and single-vector models theoretically avoid this strict bias. In bi-directional models, appending new tokens alters the attention context for all preceding tokens, allowing scores to naturally decrease if the semantic focus is diluted. Similarly, single-vector models aggregate tokens into a fixed- length representation that does not inherently benefit from added tokens. However, to understand their practical robustness to length differences, we complement this theoretical intuition with an empirical analysis. 2.2. Multi-Vector Architectures Induce A Length Bias 938577285104 128 151 190 211 245 276 313 346 374 466 626 1236 Quantile of relevant chunk length 0 200 400 600 800 1000 Length False positives (jina-embeddings-v4) False positives (Qwen3-Embedding-4B) Relevant chunks Global mean (corpus) Figure 1: Mean length comparison between the retrieved false positive chunks, the relevant ground-truth documents, and the global corpus average. Queries are grouped into quantiles on the x-axis based entirely on the average token length of their corresponding relevant chunks. Figure 1 isolates the impact of the pooling mechanism in causal architectures by comparing a multi- vector model (jina-embeddings-v4[3]) with a single-vector dense model (Qwen3-Embedding-4B [4]). Queries on the horizontal axis are partitioned into quantiles based on the token length of their true relevant documents. For each quantile, we compare the mean length of retrieved false positives against the true relevant documents, as well as the global corpus mean (199 tokens). While false positives are statistically expected to be slightly longer on average due to the inherently wider semantic scope of longer texts, this length gap should ideally remain marginal. However, the multi-vector causal model disproportionately retrieves false positives that are significantly longer than the relevant documents. Conversely, the single-vector causal model tracks the relevant document length much more closely. This confirms that within causal architectures, the multi-vector setup is the primary driver of length bias. 2.3. Bi-Directional Models Mitigate But Do Not Eliminate Bias Having identified the multi-vector setup as a primary driver of length bias in causal architectures, we now investigate whether bi-directional attention mechanisms can effectively neutralize this issue. We 413183147597183961081211391621902212573033554295802740 Added Chunk Length 0.0000 0.0005 0.0010 0.0015 0.0020 0.0025 0.0030 Expected nDCG@10 Decrease Actual Model Harm No Bias Expected Baseline 90% Random Variance Interval (a) jina-embeddings-v4 413183147597183961081211391621902212573033554295802740 Added Chunk Length 0.0000 0.0002 0.0004 0.0006 0.0008 0.0010 0.0012 0.0014 Expected nDCG@10 Decrease Actual Model Harm No Bias Expected Baseline 90% Random Variance Interval (b) Qwen3-Embedding-4B 413183147597183961081211391621902212573033554295802740 Added Chunk Length 0.000 0.001 0.002 0.003 0.004 0.005 Expected nDCG@10 Decrease Actual Model Harm No Bias Expected Baseline 90% Random Variance Interval (c) GTE-ModernColBERT-v1 413183147597183961081211391621902212573033554295802740 Added Chunk Length 0.0000 0.0005 0.0010 0.0015 0.0020 0.0025 Expected nDCG@10 Decrease Actual Model Harm No Bias Expected Baseline 90% Random Variance Interval (d) ColBERT-Zero Figure 2: Expected decrease in retrieval performance (nDCG@10) when a chunk of a specific length is added to the corpus. Chunks are categorized into equal-sized quantile bins by token length on the x-axis. The solid line plots the average nDCG penalty incurred by the presence of a chunk from that bin, evaluated against a random baseline (dashed line) and its 90% confidence interval (shaded area). evaluate the expected unconditional decrease in retrieval performance (nDCG) when a chunk of a specific length is added to the corpus. To isolate the effect of length, we compute a random baseline and a 90% confidence interval using a permutation test that assumes no correlation between chunk length and retrieval harm. Any deviation outside this interval demonstrates a statistically significant length bias, indicating that adding chunks of that size disproportionately harms ranking quality compared to random chance. The results reveal distinct architectural behaviors. The causal multi-vector model (Figure 2a) exhibits a near-monotonic length bias; adding longer chunks consistently translates to a greater expected decrease in nDCG. In contrast, the single-vector dense model (Figure 2b) displays no significant length bias, remaining safely within the random baseline and corroborating the findings of the previous section. Notably, while bi-directional multi-vector models successfully dampen the aggressive bias of causal models, they remain vulnerable at length extremes (Figures 2c and 2d). For these models, adding unusually short chunks is significantly less harmful than expected by random chance, whereas introducing exceptionally long chunks disproportionately degrades overall ranking quality. Thus, while bi-directional attention refines token representations, it remains unable to fully recover the systemic length bias introduced by the MaxSim operation across substantial token variations. 3. Similarity Distribution: What Happens Beyond The Top-1 Document Token The MaxSim operator described in subsection 2.1, by construction, only considers the single most similar document token for each query token, ignoring the number and density of relevant tokens in a chunk. This could lead to hypersensitivity to the highest similarity score of document tokens. Intuitively, a single query token that has several strong matches in a document A is more similar to it than to a document B, where only one token has a high similarity, a nuance that is lost during the MaxSim operation. To understand how document token similarity behaves beyond the top-1, we focus on queries where retrieval fails (i.e., when the positive document is outside of the top-10), to understand whether there are unseen similarity trends that could be exploited through alternative functions. We compare the sorted document token scores on these queries, aggregated over all query tokens, and across the failing queries on the NanoBEIR dataset, for both ColBERT-Zero and jina-embeddings-v4. We compare the similarity distribution for the positive sample that was not retrieved, and multiple negatives: the top-1 (best), the one ranked directly below the positive, and the worst. 0.011.122.233.344.455.666.777.888.9100.0 Relative document token rank (%) 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 Avg similarity Absolute Scores | 201 failed queries | All datasets (13) Positive (mean) Top-1 Neg. (mean) Adjacent Neg. (mean) Worst Neg. (mean) 0.011.122.233.344.455.666.777.888.9100.0 Relative document token rank (%) 0.5 0.6 0.7 0.8 0.9 1.0 Relative to max Relative Decline | 201 failed queries | All datasets (13) Positive (mean) Neg-1 (mean) Adjacent neg (mean) Worst neg (mean) 0500100015002000 Document length (tokens) 0 20 40 60 80 100 120 Count Pos =278 Neg =259 Doc Length Distribution | All datasets (13) Positive (201) Negative (603) (a)Similarities aggregated across datasets (201 queries). 0.011.122.233.344.455.666.777.888.9100.0 Relative document token rank (%) 0.50 0.55 0.60 0.65 0.70 0.75 0.80 Avg similarity Absolute Scores | 15 failed queries | NanoArguAna Positive (mean) Top-1 Neg. (mean) Adjacent Neg. (mean) Worst Neg. (mean) 0.011.122.233.344.455.666.777.888.9100.0 Relative document token rank (%) 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Relative to max Relative Decline | 15 failed queries | NanoArguAna Positive (mean) Neg-1 (mean) Adjacent neg (mean) Worst neg (mean) 0100200300400500600700800 Document length (tokens) 0 1 2 3 4 5 6 Count Pos =84 Neg =246 Doc Length Distribution | NanoArguAna Positive (15) Negative (45) (b) Similarities on NanoArguAna (15 queries). Figure 3:ColBERT-Zerodocument token similarities on failed queries. While some datasets exhibit interesting results (e.g., NanoArguAna, right), no clear tendency emerges for positive documents across NanoBEIR (left). As Figure 3b shows, on NanoArguAna, the positive document has better document similarity than the top-1 negative beyond the first tokens (starting around 10%). In such cases, leveraging score distribution beyond the best document token could help identify positive chunks. However, this result does not hold on average across all datasets of NanoBEIR, suggesting that such techniques would not generalize well enough. We also analyze the behavior on successful retrieval samples, with similar conclusions: positive retrieved samples do not have a significantly different similarity distribution in document tokens than negatives. jina-embeddings-v4 exhibits the same trends (Appendix C), that further hold across all individual datasets. 4. Conclusion And Future Work This work provides insights into Late Interaction model dynamics, highlighting a strict length bias in multi-vector causal architectures that bi-directional models mitigate. Similarly to previous work [5], this suggests that causal models are a poor fit for Late Interaction, encouraging the development of stronger bi-directional models trained for this paradigm. Furthermore, we observed that on standard retrieval benchmarks, no significant similarity trends emerge beyond the top-1 document token, suggesting that current models do not provide information that could be leveraged beyond a MaxSim operator. Interesting future analyses include testing for a length bias in a controlled setting, using synthetic data to precisely adjust text lengths and semantic relevance. Additionally, document token score distributions should be analyzed on a broader set of tasks, to see if task-dependent trends could emerge (e.g., in long-context retrieval). It remains an important analysis to do on newly released models, as training recipes can highly impact these behaviors. Future work could also explore translating these insights to mitigate bias and enhance performance through interventions at training time, during indexing, or by refining the retrieval similarity operator. References [1]O. Khattab, M. Zaharia, Colbert: Efficient and effective passage search via contextualized late interaction over bert, 2020. URL: https://arxiv.org/abs/2004.12832. arXiv:2004.12832. [2] Nanobeir, a collection of smaller versions of beir datasets with 50 queries and up to 10k documents each., 2024. URL: https://huggingface.co/collections/zeta-alpha-ai/nanobeir. [3]M. Günther, S. Sturua, M. K. Akram, I. Mohr, A. Ungureanu, S. Eslami, S. Martens, B. Wang, N. Wang, H. Xiao, jina-embeddings-v4: Universal embeddings for multimodal multilingual retrieval, 2025. URL: https://arxiv.org/abs/2506.18902. arXiv:2506.18902. [4] Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin, F. Huang, J. Zhou, Qwen3 embedding: Advancing text embedding and reranking through foundation models, arXiv preprint arXiv:2506.05176 (2025). [5]P. Teiletche, Q. Macé, M. Conti, A. Loison, G. Viaud, P. Colombo, M. Faysse, Modern- vbert: Towards smaller visual document retrievers, 2025. URL: https://arxiv.org/abs/2510.01149. arXiv:2510.01149. [6]N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, I. Gurevych, Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, 2021. URL: https://arxiv.org/abs/2104.08663. arXiv:2104.08663. [7]S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, J. Lin, Qwen2.5-vl technical report, 2025. URL: https://arxiv.org/abs/2502.13923. arXiv:2502.13923. [8] A.Chaffin,Gte-moderncolbert,2025.URL:https://huggingface.co/lightonai/ GTE-ModernColBERT-v1. [9]A. Chaffin, L. Arnaboldi, A. Chatelain, F. Krzakala, Colbert-zero: To pre-train or not to pre-train colbert models, 2026. URL: https://arxiv.org/abs/2602.16609. arXiv:2602.16609. Appendix A. Experimental Setup A.1. Datasets To empirically validate length bias, we evaluated retrieval performance using NanoBEIR [2], a subset of the BEIR benchmark [6] comprising 13 diverse datasets with 50 queries each. To ensure a wide distribution of chunk lengths, we pooled the 13 datasets into a single unified corpus prior to retrieval. After removing five outlier chunks exceeding 3,000 tokens (which lacked associated queries), the final merged corpus contained 56,718 chunks and 649 queries. 025050075010001250150017502000225025002750 Document length in tokens 0 2000 4000 6000 8000 10000 12000 14000 Frequency NanoBEIR Document Length Distribution Figure 4: Chunk Length Distribution across the merged NanoBEIR corpus. A.2. Setup Chunk sizes were computed using thejina-embeddings-v4[3] tokenizer, a Byte-Pair Encoding tokenizer inherited directly from its base model, Qwen2.5-VL-3B-Instruct [7]. We evaluated four distinct model configurations representing combinations of encoder architectures (causal vs. bi-directional) and pooling strategies (single-vector vs. multi-vector): ModelPooling Strategy Architecture Parameters jina-embeddings-v4 [3]Multi-vectorCausal4B Qwen3-Embedding-4B [4]Single-vectorCausal4B GTE-ModernColBERT-v1 [8]Multi-vectorBi-directional0.15B ColBERT-Zero [9]Multi-vectorBi-directional0.15B Table 1 Summary of the evaluated models, including their pooling strategies, architectures, and sizes. B. Retrieval Errors by Chunk Length Figure 5 illustrates the raw volume of retrieval errors mapped to document chunk lengths. While a general increase in false positives for longer chunks is expected as longer texts naturally contain more semantic coverage, the models exhibit distinct architectural vulnerabilities. The causal multi-vector model (jina-embeddings-v4) is the only configuration to display a strictly monotonic increase in errors starting from zero, corroborating the length bias of causal multi-vector models described in Section 2.2. Conversely, bi-directional models demonstrate non-monotonic error distributions. Instead, distinct peaks emerge at the extreme ends of the length spectrum (for both very short and very long chunks), serving as an additional indicator of the marginal vulnerabilities discussed in Section 2.3. As a side note, both bi-directional models show a high absolute volume of errors despite yielding strong overall nDCG 413183147597183961081211391621902212573033554295802740 Chunk Length Bins (Tokens) 0 1000 2000 3000 4000 5000 Occurrences Ranked Above Ground Truth Chunk counts No Bias Expected Baseline 90% Random Variance Interval (a) jina-embeddings-v4 413183147597183961081211391621902212573033554295802740 Chunk Length Bins (Tokens) 0 250 500 750 1000 1250 1500 1750 Occurrences Ranked Above Ground Truth Chunk counts No Bias Expected Baseline 90% Random Variance Interval (b) Qwen3-Embedding-4B 413183147597183961081211391621902212573033554295802740 Chunk Length Bins (Tokens) 0 2500 5000 7500 10000 12500 15000 17500 Occurrences Ranked Above Ground Truth Chunk counts No Bias Expected Baseline 90% Random Variance Interval (c) GTE-ModernColBERT-v1 413183147597183961081211391621902212573033554295802740 Chunk Length Bins (Tokens) 0 5000 10000 15000 20000 25000 Occurrences Ranked Above Ground Truth Chunk counts No Bias Expected Baseline 90% Random Variance Interval (d) ColBERT-Zero Figure 5: Absolute occurrences of irrelevant chunks ranked above the highest-ranked true positive passage. Bin limits are defined to contain an equal number of chunks. The dashed line plots the no-bias expected baseline, bounded by a 90% variance interval. metrics, indicating that while they rank well on average, their failures are notably severe. This contrast in error severity can largely be explained by the substantial gap in parameter count compared to the two causal models (0.15B vs. 4B). C. Similarity Distribution For jina-embeddings-v4 0.011.122.233.344.455.666.777.888.9100.0 Relative document token rank (%) 0.1 0.0 0.1 0.2 0.3 0.4 Avg similarity Absolute Scores | 229 failed queries | All datasets (13) Positive (mean) Top-1 Neg. (mean) Adjacent Neg. (mean) Worst Neg. (mean) 0.011.122.233.344.455.666.777.888.9100.0 Relative document token rank (%) 0.4 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Relative to max Relative Decline | 229 failed queries | All datasets (13) Positive (mean) Neg-1 (mean) Adjacent neg (mean) Worst neg (mean) 05001000150020002500 Document length (tokens) 0 50 100 150 200 250 Count Pos =319 Neg =233 Doc Length Distribution | All datasets (13) Positive (229) Negative (687) (a)Similarities aggregated across datasets (229 queries). 0.011.122.233.344.455.666.777.888.9100.0 Relative document token rank (%) 0.1 0.0 0.1 0.2 0.3 0.4 0.5 Avg similarity Absolute Scores | 5 failed queries | NanoArguAna Positive (mean) Top-1 Neg. (mean) Adjacent Neg. (mean) Worst Neg. (mean) 0.011.122.233.344.455.666.777.888.9100.0 Relative document token rank (%) 0.4 0.2 0.0 0.2 0.4 0.6 0.8 1.0 Relative to max Relative Decline | 5 failed queries | NanoArguAna Positive (mean) Neg-1 (mean) Adjacent neg (mean) Worst neg (mean) 50100150200250300350400 Document length (tokens) 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Count Pos =120 Neg =183 Doc Length Distribution | NanoArguAna Positive (5) Negative (15) (b) Similarities on NanoArguAna (5 queries). Figure 6:jina-embeddings-v4document token similarities on failed queries. The positive document has a larger distance to the top-1 negative, and remains constantly worse across all document tokens.