Paper deep dive

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

Guanchen Wu, Jiayuan Ding, Subhabrata Mukherjee, Carl Yang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 70

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/18/2026, 4:46:43 AM

Summary

The paper introduces Trident, a framework for long-document multimodal visual question answering (VQA) that addresses the bottleneck of evidence selection in retrieve-then-read pipelines. Trident consists of two components: Trident-R, a reranker that uses multi-aspect page annotations (visual captions, entity tags, structural paths) to improve LLM-based reranking, and Trident-S, a generation module that prompts Vision Language Models (VLMs) under topical, entity, and structural lenses to synthesize answers. The approach significantly improves retrieval F1 and generation accuracy on datasets like MMLongBench-Doc compared to baselines like PageIndex and ColPali.

Entities (12)

MMLongBench-Doc · dataset · 99%Trident · method · 98%Trident-R · component · 95%Trident-S · component · 95%bge-m3 · model · 92%ColPali · model · 92%LongDocURL · dataset · 90%PageIndex · baseline · 90%Visual Caption · feature · 88%Entity Lens · technique · 85%Structural Lens · technique · 85%Topical Lens · technique · 85%

Relation Signals (9)

Trident → consistsof → Trident-R

confidence 95% · We propose Trident, with two complementary components: Trident-R... and Trident-S...

Trident → consistsof → Trident-S

confidence 95% · We propose Trident, with two complementary components: ... and Trident-S...

Trident → evaluatedon → MMLongBench-Doc

confidence 95% · On two long-document datasets... On MMLongBench-Doc...

Trident → outperforms → PageIndex

confidence 90% · every reranked pool exceeding the strongest adaptive-K baseline PageIndex.

Trident-S → uses → Entity Lens

confidence 90% · Trident-S... prompts the VLM under topical, entity, and structural lenses

Trident-R → uses → Visual Caption

confidence 90% · Trident-R... converts each candidate into an LLM-readable semantic record, including a visual caption...

Trident-S → uses → Topical Lens

confidence 90% · Trident-S... prompts the VLM under topical, entity, and structural lenses

Trident-S → uses → Structural Lens

confidence 90% · Trident-S... prompts the VLM under topical, entity, and structural lenses

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Long-document visual question answering (VQA) over documents of tens to hundreds of pages mixing text, tables, charts, and figures typically follows retrieve-then-read pipelines. In our setting, the bottleneck shifts from retrieval recall to reranker-side evidence selection: on MMLongBench-Doc, BGE-M3 reaches Recall@20 = 0.86 but only F1@5 = 0.254, and even the visual retriever ColPali reaches only F1@5 = 0.332; a text-only rerank LLM seeing only raw snippets misses table, chart, and layout evidence even when the upstream retriever encoded images. We propose Trident, with two complementary components: Trident-R, a retriever-agnostic LLM reranker that converts each candidate into an LLM-readable semantic record, including a visual caption, section path, entity tags, multi-axis concept hits, and a text snippet, then performs a single adaptive-K rerank call; and Trident-S, a generation-side module that prompts the VLM under topical, entity, and structural lenses before synthesis. On two long-document datasets, the annotation+rerank protocol substantially improves retrieval F1 across five heterogeneous pools, with every reranked pool exceeding the strongest adaptive-K baseline PageIndex. An LLM rerank without the annotation barely changes first-hit ranking, indicating the lift comes from the structured annotation. Trident-S targets open-ended synthesis questions by design, adding up to 6.6 points in generation accuracy on these questions. The best Trident configuration is the strongest downstream QA pipeline in our evaluation, with rankings consistent across two LLM judges (kappa = 0.913).

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

66,660 characters extracted from source content.

Expand or collapse full text

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering Guanchen Wu Affiliation: Department of Computer Science, Emory University, Atlanta, GA, USA Jiayuan Ding Affiliation: Hippocratic AI, Palo Alto, CA, USA Subhabrata Mukherjee Carl Yang Affiliation: Department of Computer Science, Emory University, Atlanta, GA, USA Affiliation: Hippocratic AI, Palo Alto, CA, USA Abstract Long-document visual question answering (VQA) over documents of tens to hundreds of pages mixing text, tables, charts, and figures typically follows retrieve-then-read pipelines. In our setting, the bottleneck shifts from retrieval recall to reranker-side evidence selection: on MMLongBench-Doc, BGE-M3 reaches Recall@20=0.8620=0.86 but only F1@5=0.2545=0.254, and even the visual retriever ColPali reaches only F1@5=0.3325=0.332—a text-only rerank LLM seeing only raw snippets misses table, chart, and layout evidence even when the upstream retriever encoded images. We propose Trident, with two complementary components: Trident-R, a retriever-agnostic LLM reranker that converts each candidate into an LLM-readable semantic record—visual caption, section path, entity tags, multi-axis concept hits, and a text snippet—then performs a single adaptive-K rerank call; and Trident-S, a generation-side module that prompts the VLM under topical, entity, and structural lenses before synthesis. On two long-document datasets, the annotation+rerank protocol substantially improves retrieval F1 across five heterogeneous pools, with every reranked pool exceeding the strongest adaptive-K baseline PageIndex. An LLM rerank without the annotation barely changes first-hit ranking, indicating the lift comes from the structured annotation. Trident-S targets open-ended synthesis questions by design, adding up to 6.66.6 points in generation accuracy on these questions. The best Trident configuration is the strongest downstream QA pipeline in our evaluation, with rankings consistent across two LLM judges (κ=0.913κ=0.913). 1 Introduction Figure 1: Overview of Trident. Given a query and a long document, a retriever first returns a top-20 candidate pool where evidence pages often already appear. Trident-R improves evidence selection by converting candidates into semantic records for adaptive LLM reranking, while Trident-S synthesizes the final answer through topical, entity, and structural lenses. Long-document visual question answering (VQA) requires answering natural-language questions over documents of tens to hundreds of pages that mix text, tables, charts, and figures. Although frontier Vision Language Models (VLMs) accept very long contexts in principle, feeding entire documents on every query is expensive and can suffer from position-induced degradation (21). Thus, retrieve-then-read pipelines (15) remain practical: a retriever selects a small set of pages, and a VLM answers the question conditioned on them. On MMLongBench-Doc, oracle evidence pages alone lift Large Vision Language Model (LVLM) F1 by 1010–3030 points over full-document inputs (23), showing evidence selection is central to long-document VQA. Where does evidence selection fail? In our setting, BGE-M3 (2) achieves Recall@20=0.86Recall@20=0.86 on MMLongBench-Doc but only F1@5=0.254F1@5=0.254; the multimodal retriever ColPali (9) improves this to F1@5=0.332F1@5=0.332 but remains far from oracle. Evidence pages are typically present in the top-2020 pool—the bottleneck is identifying the small evidence subset within 2020 surface-similar candidates. A natural next step is to insert an LLM reranker between the top-2020 pool and the K≤5K≤5 evidence selection, an approach used by RankGPT (31) and increasingly common in long-document RAG. However, when this reranker only sees raw text snippets of each candidate page, it does not improve first-hit ranking on either pool: on BGE-M3, MRR is essentially unchanged (0.524→0.5230.524→ 0.523); on ColPali, MRR actually drops from 0.6920.692 to 0.5980.598, because the text-only LLM judgment overrides ColPali’s stronger visual ranking signal. The reason is that long-document evidence is often locked in tables, charts, figures, and layout, all poorly represented in short text snippets. A rerank-stage LLM that only sees snippets is partially blind to exactly the evidence that matters most—ColPali’s visual encoding occurs only at retrieval; the rerank LLM still consumes text-only candidate records and cannot access page images directly. This rerank-stage blindness has been recognized: recent work addresses it by replacing the text LLM reranker with a multimodal LLM that directly sees page images (3; 36). This is effective but expensive—every rerank query requires a full multimodal LLM call, and these rerankers typically need instruction-tuning or RL training on multimodal labels. We take an alternative architectural choice—a path used widely in industry RAG systems but underexplored in academic long-document QA: keep a cheap, training-free text-LLM reranker, and pre-compute a structured VLM-generated annotation once per document offline. Concretely, we convert each candidate into an LLM-readable semantic record—a VLM-generated visual caption surfaces table/chart/figure content, and structured fields (section path, entity tags, topic/entity/structure concept hits) expose pre-computed evidence signals. On the ColPali pool, attaching this annotation raises rerank F1 from 0.3320.332 to 0.5810.581. Consistent with the mechanism above, removing the caption hurts both pools but hurts BGE-M3 more than ColPali: a visual retriever partially compensates for the LLM’s blindness at retrieval time, but does not eliminate the need for an explicit visual signal at rerank time. Motivated by these observations, we propose Trident, a long-document multimodal QA pipeline with two components addressing analogous bottlenecks at the rerank and generation stages. Trident-R attaches the annotation above to each top-2020 candidate and uses a single LLM call to select an adaptive set of K≤5K≤5 evidence pages. The same multi-aspect logic carries over to generation: a single VLM call must implicitly trade off attention across text, tables, and figures within each retrieved page, so we additionally propose Trident-S, which prompts the VLM under topical, entity, and structural lenses before synthesizing a final answer. On MMLongBench-Doc, the annotation+rerank protocol applied to five heterogeneous candidate pools (BM25, BGE-M3, RRF, ColPali, and our multi-axis xKG) raises retrieval F1 on all five, with every reranked pool exceeding the strongest adaptive-K baseline PageIndex (39); the improvement also holds on LongDocURL. Trident-S targets open-ended synthesis questions by design, providing consistent gains on that workload while leaving extractive workloads to a standard single-call VLM. End-to-end, the best Trident configuration is the strongest downstream pipeline in our evaluation, with rankings consistent across two LLM judges (κ=0.913κ=0.913). 2 Related Work Long-document VQA benchmarks and multimodal RAG systems. We evaluate on MMLongBench-Doc (23) and LongDocURL (6), two benchmarks targeting VQA over hundred-plus-page documents with multi-modal evidence (text, tables, charts, figures, layout); MMLongBench-Doc reports that GPT-4o-vision with full-document access lags VLMs given oracle evidence pages by ∼30 30 points. Recent benchmarks complement these along different axes: retrieval-focused MMDocIR (7) (313313 long documents, notably finding VLM image descriptions outperform OCR text—consistent with our caption-dominance result), fine-grained evidence-selection MMDocRAG (8) (metrics beyond page-level F1), and difficulty-graded REAL-M-RAG (34) (finance / technical). Multimodal RAG systems combine multi-modal retrievers with multimodal LLMs (M3DocRAG (4)), use hierarchical indexing with intra-page and cross-page chunks (MMRAG-DocQA (11)), build chunk–query graphs (MLDocRAG (41)), triage by document structure (PDFTriage (26)), or use VLM-generated structured JSON descriptions for tables and figures (MultiFinRAG (10)). Trident differs by factoring the pipeline into a swappable Stage 1 retriever and a Stage 2 annotation ++ rerank protocol we empirically isolate (§4.2). Document retrievers and adaptive-K selection. Page-level retrievers span sparse (BM25), dense (BGE-M3 (2); ColBERT-style late interaction (14; 28); RRF fusion (5)), and vision-aware variants (ColPali (9) with PaliGemma plus patch-level MaxSim; DSE (22) with image-level dense embeddings); none condition on document-level structure. Adaptive-K selection has been addressed by section-tree navigation (PageIndex (39)), hierarchical summarization (RAPTOR (29)), reflection-controlled retrieval (Self-RAG (1)), summary-based re-ranking (SimpleDoc (13)), and relevance clustering (AVIR (18)). Recent agentic approaches use iterative refinement (Doc-React (35)), unified MLLM retrieval-generation (URaG (30)), or multi-turn RL (M-Doc-R1 (19)). These either rely on a single structural / semantic / visual signal or require end-to-end MLLM / agent training; Trident is complementary—training-free, retriever-agnostic, and easily bolted onto any pipeline via multi-aspect annotation. LLM rerankers with structured features. Retrieval-augmented generation (15) traditionally couples retriever and generator via end-to-end training; Trident sits at the opposite end of the design space—modular and prompting-based, with multi-aspect annotation serving as a frozen abstraction layer pairable with any pool and any VLM without re-training. RankGPT (31) exposes raw candidate text to an LLM for re-scoring; recent work instead conditions LLM rerankers on structured per-candidate features: CoRank (32) (categories / sections / keywords for scientific retrieval), FinCARDS (42) (schema fields for financial QA), KeyB2 (16) (block selection), and AcuRank (37) (uncertainty-aware adaptive listwise)—all text-only. Trident differs in three ways: (i) we target long-document multimodal QA, using VLM-generated visual captions as an LLM-readable interface for multimodal evidence; (i) we test the interface via cross-pool isolation across five retriever paradigms, and show the gains transfer to the visual ColPali pool via asymmetric drop-caption and pool-invariant caption-only ablations (§4.2, Appendix I); (i) our generation-side analog Trident-S extends the same principle to multi-view VLM prompting. We additionally expose multi-aspect metadata—concept hits across topic / entity / structure axes plus section path, visual caption, and entity tags—enabling annotation-level reasoning when surface text is repetitive among top-20 candidates. Knowledge graphs have been used for retrieval over textual corpora (12) and document QA (27); Trident differs by treating annotation as a page-level reranker-friendly abstraction layer, not as a graph-walk substrate. Caption-based vs MLLM-based multimodal reranking. Two architectural families incorporate visual content into multimodal rerankers. Caption-based approaches convert page images to text via VLM descriptions and feed a text-only LLM reranker; this pattern is common in industry RAG systems (Haystack, NVIDIA RAG Blueprint, IBM Multimodal RAG) but academically underexplored under a “loses visual details” assumption. MLLM-based approaches let a multimodal LLM see page images directly, spanning zero-shot prompt-based rerankers (3; 24), fine-tuned / RL-trained variants (36; 33; 20), and listwise / production rerankers (17); all are effective but costly per query and typically require multimodal instruction-tuning or RL training. Concurrent work also retrieves over VLM-encoded pages directly (38). Trident systematically revisits the caption-based path for long-document multimodal QA and shows that with structured multi-aspect annotation, a text-only LLM reranker is competitive with or better than text-only / visual baselines at a fraction of per-query cost, while remaining drop-in compatible with any upstream retriever—tested via cross-pool isolation across five retriever paradigms (§4.2). VLM-augmented LLM QA and multi-view prompting. SCRA-VQA (40) converts single-image VQA into a caption ++ LLM rerank pipeline, demonstrating that VLM captions can substantially augment LLM-based QA at the single-image scale; MPCAR (25) generates diverse complementary descriptions from multiple analytical perspectives and fuses them in a single-image LVLM reasoning prompt. Trident extends this lineage to long-document multimodal retrieval and generation: visual caption is identified as the strongest universal annotation field on the retrieval side; the three orthogonal lenses on the generation side (Trident-S) are structured around our T / E / S decomposition axes and are followed by a format-aware LLM synthesis call. Crucially, we characterize Trident-S’s effective regime (substantial gains on open-ended synthesis questions only) rather than claim universal lift. 3 Method Figure 2: Overall framework of TRIDENT-R. 3.1 Problem Formulation Let a document D=p1,…,pND=\p_1,…,p_N\ contain N pages of mixed text and visual content (N may exceed 100100). Given a natural-language question q with gold evidence set Eq⊆DE_q D and gold answer aqa_q, the long-document VQA task is to produce an answer a a approximating aqa_q. We decompose the task into retrieval and generation: P^q=ℛ⁡(q,D),|P^q|≤Kmax;a^=⁡(q,P^q), P_q=R(q,D),\ | P_q|≤ K_ ; a=G(q, P_q), (1) where ℛR returns up to KmaxK_ evidence pages and G is a VLM. Following the reranker-interface bottleneck identified in §1, Trident factors ℛR into a candidate-retrieval stage and an annotation ++ rerank stage: P^q=ℛ2(q,ℛ1(q,D),Φ⁡(D)), P_q=R_2(q,R_1(q,D),\ (D)), (2) where ℛ1R_1 returns a top-M=20M=20 pool, Φ⁡(D) (D) is a multi-aspect annotation (§3.2), and ℛ2R_2 is an LLM reranker with Kmax=5K_ =5 (§3.3). G is then realized by either a single VLM call or a multi-view generator Trident-S (§3.5). ℛ1R_1 is treated as a swappable candidate-pool source (§3.4), while ℛ2R_2 and the page annotation Φ are the focus of this work. 3.2 Trident-R-Index: Multi-Aspect Page Annotation Trident-R-Index is a function Φ⁡(D)=(,ℳ) (D)=(C,M) that maps a document D to a set of concepts C and per-page metadata records ℳpp∈D\M_p\_p∈ D. The concept set decomposes into three disjoint axes: =T∪E∪SC\;=\;C_T _E _S (3) where TC_T contains topic concepts, EC_E entity-bundle concepts, and SC_S structural concepts. Each concept c∈Xc _X (X∈T,E,SX∈\T,E,S\) carries a name, an embedding c∈ℝde_c ^d (where applicable), and a weighted membership—a set of (page, weight) pairs (p,wc,p):p∈D,wc,p>0\(p,w_c,p):p∈ D,\ w_c,p>0\ where wc,p∈[0,1]w_c,p∈[0,1] reflects how strongly page p contributes to concept c. Topic concepts TC_T. We segment each page p into overlapping text windows of ≤512≤ 512 tokens (stride 256256), denoted Wp=wp(1),…,wp(np)W_p=\w^(1)_p,…,w^(n_p)_p\. All windows ⋃pWp _pW_p are embedded by a sentence embedding model ℰWE_W and L2-normalized. We cluster the embedding set via mini-batch K-means with KT=min⁡(64,|W|8)K_T= (64, |W|8). Each resulting cluster c∈Tc _T records its centroid ce_c, member windows WcW_c, and per-page weight: wc,p=|Wc∩Wp||Wc|,so that ∑pwc,p=1.w_c,p\;=\; |W_c∩ W_p||W_c|, that _pw_c,p=1. (4) Entity-bundle concepts EC_E. An extraction LLM ℒextL_ext extracts a flat set of typed entities ℰpE_p from each page—named entities, numbers, dates, monetary values, percentages, and table / figure references. We group co-occurring entities across pages using density-based clustering on a graph where entities are connected when they co-occur on at least one page, retaining bundles of size ≥2≥ 2. Each bundle c∈Ec _E stores its member entity set ℰcE_c and the pages on which ℰcE_c surfaces, with weight wc,pw_c,p equal to the fraction of ℰcE_c entities appearing on p. Structural concepts SC_S. We obtain a hierarchical section tree of D from a public PDF structural parser secP_sec (any document parser returning a section hierarchy can be substituted; we ground our choice in §4.1). Each leaf section s becomes a structural concept c with member pages p:p∈span(s)\p:p (s)\ (uniform weights), and embedding c=ℰW(summary(s))e_c=E_W(summary(s)), where summary(s)summary(s) is the section summary returned by secP_sec. Per-page metadata ℳpM_p. In addition to the concepts above, we attach four per-page fields to each p: (a) a structural path string (e.g. “Item 8 / Consolidated Balance Sheets”), read off the section tree; (b) a visual caption vcpvc_p, generated by a captioning VLM capV_cap prompted to describe the page’s visible tables, figures, charts, and layout in ≤60≤ 60 words; (c) a flat entity-tag list etp⊆ℰpet_p _p retaining the top-15 most salient entities/numbers on the page; and (d) the page’s raw text snippet (truncated to 600 characters). Offline cost. Index construction requires per-page LLM/VLM passes once per document and is cached on disk for reuse across all subsequent queries; full token-level accounting is in Appendix H. 3.3 Trident-R: Adaptive-K LLM Rerank Given a query q, we first obtain a candidate page pool Pq⊆DP_q D with |Pq|≤M=20|P_q|≤ M=20 from one of the retrievers in §3.4, then Trident-R performs a single LLM call that returns an adaptive-K ordered subset P^q⊆Pq P_q P_q with |P^q|≤Kmax=5| P_q|≤ K_ =5. Candidate payload. For each candidate page p∈Pqp∈ P_q, we assemble a structured record ℛp=(p,sp,chp,ℳp)R_p=(p,\ s_p,\ ch_p,\ M_p) where sps_p is the underlying retriever score and chpch_p is a concept-hit list chp= _p\;=\; \ (X,c.name):c∈X, (X,c.name):c _X, (5) wc,p>0,X∈T,E,S w_c,p>0,\ X∈\T,E,S\ \ collapsing all concept memberships involving p into a typed list of ≤6≤ 6 tags (e.g. T:Financial Statements, E:Nike). Combined with ℳpM_p, each candidate exposes five discriminating signals to the reranker: concept hits, section path, visual caption, entity tags, and a truncated text excerpt. Rerank step. We serialize ℛpp∈Pq\R_p\_p∈ P_q as a JSON list and prompt a rerank LLM ℛθR_θ (at τ=0τ=0) with the question and candidate list, expecting a JSON response of the form: ℛθ(q,ℛp)=(thinking,P^q),|P^q|≤Kmax.R_θ(q,\R_p\)\;=\; (thinking,\ P_q ), | P_q|≤ K_ . (6) The reranker may return fewer than KmaxK_ pages when the question’s evidence is concentrated, yielding an adaptive |P^q|| P_q|. The verbatim rerank prompt is in Appendix B. Why JSON-conditioned rerank? Compared to a fully-textual rerank prompt (e.g. RankGPT (31)), exposing structured per-candidate fields lets the LLM attend to specific evidence signals—visual caption, entity tags, concept memberships—rather than re-deriving them from the raw page text in its limited context. We empirically validate this design in §4.2 and isolate per-field contributions in §4.3. 3.4 Candidate Pool Sources Trident-R is agnostic to the source of PqP_q. We evaluate four pool choices, each returning the top-M=20M=20 pages by score. Single-signal text pools. BM25 returns the top-20 pages by Okapi BM25 score over page text. BGE-M3 (2) returns the top-20 pages by cosine similarity between the query embedding and page embeddings (dim 10241024). RRF (5) fuses BM25 and BGE-M3 rankings by Reciprocal Rank Fusion (k=60k=60). Vision pool. ColPali (9) encodes each page image with PaliGemma and returns top-20 pages by late-interaction MaxSim. Multi-axis pool (xKG). We compose a multi-channel score from T,E,SC_T,C_E,C_S. A lightweight LLM router (Appendix D) first classifies q into a class κ∈topical,entity-level,structural,mixedκ∈\topical,\ entity-level,\ structural,\ mixed\ and extracts query entities ℰqE_q. The concept-level scores per axis are: scoreT(c,q) _T(c,q) =cos⁡(c,q), =cos(e_c,e_q), c∈T c _T (7) scoreE(c,q) _E(c,q) =|ℰc∩subℰq||ℰc|, = |E_c _subE_q||E_c|, c∈E c _E (8) scoreS(c,q) _S(c,q) =cos⁡(c,q), =cos(e_c,e_q), c∈S c _S (9) where ∩sub _sub denotes loose substring matching to handle entity surface variants. Page-level scores are obtained by aggregating concept contributions weighted by membership and a class-conditional fusion (αTκ,αEκ,αSκ)( _T^κ, _E^κ, _S^κ): spxKG(q)=∑X∈T,E,SαXκ∑c∈XscoreX(c,q)⋅wc,p.s xKG_p(q)\;=\; _X∈\T,E,S\ _X^κ _c _Xscore_X(c,q)· w_c,p. (10) The top-M pages by spxKGs xKG_p form the xKG pool. Specific (αTκ,αEκ,αSκ)( _T^κ, _E^κ, _S^κ) values for each κ, and a router-ablation comparison with uniform weights, are in Appendix D. 3.5 Trident-S: Multi-View Generation Given retrieved pages P^q P_q, Trident-S prompts a VLM V three times in parallel under three lenses applied to the same retrieved image set render(p):p∈P^q\render(p):p∈ P_q\: T: Topical lens. “What is each page about thematically? Which topics relate to the question?” E: Entity / value lens. “Extract the named entities, numerical values, dates, table cells, and figure labels visible on each page. Use verbatim values.” S: Structural lens. “Where does each page sit in the document organization (section title, captions, headers)?” Producing three candidate answers a^T,a^E,a^S a^T, a^E, a^S. A fourth synthesis call S merges them into a final answer: a^=⁡(q,a^T,a^E,a^S), a\;=\;S (q,\ a^T,\ a^E,\ a^S ), (11) with format-aware preferences encoded in the synthesis prompt (preferring the E-view answer for numeric questions, the S-view for location questions, and consensus otherwise). When all three lenses agree the question is unanswerable, S returns “Not answerable”—a calibration property we examine in §4.4. Full prompts are in Appendix C. Why three lenses? Long-document evidence is often heterogeneous within a single retrieved page (e.g., a financial-report page contains both narrative text and a table). A single VLM call must implicitly trade off attention across modalities; by issuing three lens-conditioned prompts we explicitly induce the VLM to surface each evidence type, then defer the trade-off to a smaller textual synthesis call. This is conceptually analogous to multi-aspect annotation at generation time. We treat Trident-S as a question-format-conditioned extension; the empirical characterization of when it helps is in §4.4 and §4.5. 4 Experiments 4.1 Setup Method avg. K Recall Prec. F1 MRR nDCG Fixed-K baselines (no rerank, K=5K=5) BM25 5.00 0.525 0.162 0.231 0.448 0.437 RRF 5.00 0.556 0.172 0.246 0.524 0.496 BGE-M3 5.00 0.572 0.178 0.254 0.524 0.502 ColPali 5.00 0.739 0.236 0.332 0.692 0.675 Adaptive-K, single-aspect signal PageIndex 2.99 0.644 0.442 0.480 0.569 0.583 Adaptive-K, pool ++ Trident annotation ++ rerank (ours) BM25 (sparse) ++ ann. ++ rerank 2.89 0.665 0.471 0.503 0.707 0.681 RRF (hybrid) ++ ann. ++ rerank 3.78 0.748 0.454 0.511 0.751 0.728 xKG (graph) ++ ann. ++ rerank (Trident-R) 2.82 0.692 0.485 0.527 0.714 0.690 BGE-M3 (dense) ++ ann. ++ rerank 2.89 0.726 0.499 0.546 0.744 0.723 ColPali (visual) ++ ann. ++ rerank 2.85 0.766 0.533 0.581 0.788 0.768 Table 1: Cross-pool isolation on MMLongBench-Doc (N=830N=830 aligned QAs; all reranked rows use GPT-4.1 with Kmax=5K_ =5). The same annotation ++ rerank protocol improves sparse, hybrid, graph, dense, and visual candidate pools. Pool Candidate view F1 MRR BGE-M3 raw top-5 retrieval 0.254 0.524 BGE-M3 text snippet ++ LLM rerank 0.374 0.523 BGE-M3 caption-only annotation ++ rerank 0.518 0.692 BGE-M3 full annotation ++ rerank 0.546 0.744 ColPali raw top-5 retrieval 0.332 0.692 ColPali text snippet ++ LLM rerank 0.423 0.598 ColPali caption-only annotation ++ rerank 0.554 0.744 ColPali full annotation ++ rerank 0.581 0.788 Table 2: Mechanism ablations. LLM reranking with only raw text snippets does not improve first-hit ranking on either pool (MRR flat on BGE-M3, and decreasing on ColPali because the LLM judgment overrides ColPali’s visual ranking signal); the ranking lift requires the structured annotation, and the visual caption alone recovers most of it on both text and visual pools. Datasets: MMLongBench-Doc (23) (1,082 QAs / 134 documents) and LongDocURL (6) (1,123 QAs / 200 documents). Retrieval metrics are reported on the N=830N=830 answerable subset of MMLongBench-Doc. End-to-end accuracy is reported on a 300-QA stratified subset; the full sampling details are in Appendix E. Baselines: BM25, BGE-M3 (2), RRF (5), ColPali (9) at K=5K=5 (fixed-K, no rerank), and PageIndex (39) adaptive-K cookbook with GPT-4.1. Our cross-pool isolation pairs each retriever pool with the same Trident-R annotation ++ LLM rerank protocol. Module assignments: We instantiate the abstract modules of §3 as follows—text embeddings (ℰWE_W): text-embedding-3-large; extraction LLM ℒextL_ext, rerank LLM ℛθR_θ, and synthesis LLM S: GPT-4.1 at τ=0τ=0; captioning VLM capV_cap: GPT-4o-mini; answering VLM ansV_ans: GPT-4.1 (vision); structural parser secP_sec: the PageIndex API (39) used purely as a PDF section-tree extractor, independent of the PageIndex retrieval system we benchmark against. The candidate JSON records produced by Φ are consumable by any instruction-tuned LLM; the annotation ++ rerank protocol is therefore not tied to a specific model family. Claude and Gemini additionally serve as LLM-as-judge evaluators (Appendix L). Metrics: Retrieval is evaluated by page-level Recall, Precision, F1, MRR, and nDCG against gold evidence pages. End-to-end QA is evaluated by MMLongBench-Doc Generalized Accuracy and LLM-as-judge accuracy. PageIndex output: PageIndex selects section nodes, which we expand to constituent pages and truncate at top-55 for apples-to-apples comparison. On MMLongBench-Doc this cap is rarely active (18.6%18.6\% of queries); on LongDocURL it is more active (49.7%49.7\%). 4.2 Cross-Pool Retrieval Isolation Table 1 tests whether the proposed interface works only for a particular retriever or transfers across candidate-pool distributions. The answer is consistent: applying the same annotation ++ rerank protocol more than doubles raw retriever F1 for several pools (e.g., BGE-M3 0.254→0.5460.254→ 0.546) and raises all five pools above PageIndex. The visual ColPali pool is the most important case for our story: although ColPali already encodes page images, converting its candidates into LLM-readable annotations still raises F1 from 0.3320.332 to 0.5810.581. This indicates that the interface is not merely supplying missing visual information to text retrievers. 4.3 Mechanism: What the Reranker Sees Table 2 isolates why the interface helps. The text-snippet-only rerank rows show that an LLM with raw page snippets improves F1 by shortening the output, but does not improve first-hit ranking on either pool: on BGE-M3 MRR is essentially unchanged (0.524→0.5230.524→ 0.523), and on ColPali MRR actually drops (0.692→0.5980.692→ 0.598) because the snippet-only LLM judgment overrides ColPali’s stronger visual ranking signal. The ranking lift appears only once the reranker sees structured page annotation, and the visual caption alone is sufficient to recover most of it (BGE-M3 MRR →0.692→ 0.692, ColPali MRR →0.744→ 0.744 with caption-only; →0.744/0.788→ 0.744/0.788 with full annotation). Full per-field drop ablations (Appendix I) show the same pattern: removing visual caption hurts both pools, and the remaining topic/entity/structure axes add a small but nearly identical marginal gain (+0.028+0.028 / +0.027+0.027). Thus the caption is not just modality completion for text retrievers; it is a reranker-facing semantic representation that remains useful even when the upstream retriever is visual. We keep detailed secondary analyses in the appendix to keep the main text focused on the reranker-interface claim. Appendix I reports full field ablations on BGE-M3 and ColPali; Appendix D reports xKG router details; Appendix J reports the adaptive-K distribution; and Appendix F reports bootstrap, Wilcoxon, and McNemar tests. 4.4 End-to-End QA as Supporting Evidence Method GPT J. Claude J. Gen Acc Gen F1 BGE-M3 (Plain) 44.7 44.7 43.5 41.3 ColPali (Plain) 48.3 46.3 46.4 45.9 PageIndex ++Trident-S 49.3 48.3 45.9 42.6 BGE-M3 ++ann. ++rk ++Trident-S 52.0 50.0 48.9 46.4 ColPali ++ann. ++rk, Plain 51.0 49.7 47.1 44.5 ColPali ++ann. ++rk ++Trident-S 53.7 52.7 51.2 48.9 Trident-R (xKG pool, Plain) 50.7 48.0 47.2 42.9 Trident-R (xKG pool) ++Trident-S 52.0 50.7 46.8 44.3 Table 3: End-to-end QA on the 300-QA MMLongBench-Doc subset (selected pipelines; all values in %). Full 14-pipeline results, including answer accuracy and unanswerable abstention, are in Appendix A. Table 3 shows that the retrieval-side gain generally transfers to downstream QA. The ColPali pool with annotation and rerank is the best observed end-to-end configuration in this subset, and adding Trident-S gives the highest Gen Acc and Gen F1 on MMLongBench-Doc. The full 14-pipeline table and inter-judge agreement are in Appendix A and Appendix L; the pipeline ranking is consistent across the two judges (κ=0.913κ=0.913). 4.5 LongDocURL Transfer Check Method avg. K Recall Prec. F1 MRR nDCG Fixed-K baselines (no rerank, K=5K=5) BM25 5.00 0.623 0.191 0.281 0.616 0.568 BGE-M3 5.00 0.649 0.205 0.298 0.622 0.575 RRF 5.00 0.684 0.217 0.315 0.675 0.622 ColPali 5.00 0.750 0.235 0.343 0.763 0.700 Adaptive-K PageIndex 4.03 0.598 0.302 0.366 0.430 0.463 Trident-R 3.19 0.546 0.362 0.398 0.603 0.552 Table 4: LongDocURL retrieval comparison (N=1,122N=1,122 QAs with non-empty gold evidence). The retrieval-side trend transfers, although the full cross-pool isolation study is conducted on MMLongBench-Doc. Table 4 provides an external transfer check on N=1,122N=1,122 QAs. Trident-R achieves the highest retrieval F1, showing that the retrieval-side trend transfers to LongDocURL. End-to-end results in Appendix A show the same pattern: ColPali plus annotation and rerank remains strongest. When does Trident-S help? Trident-S delivers consistent positive gains on MMLongBench-Doc’s open-ended synthesis questions (+1.7+1.7 to +6.6+6.6 Gen Acc across all eight evaluated pipelines, with the largest absolute gain on weaker baselines—PageIndex Plain →⁣+→+Trident-S adds +6.6+6.6). On LongDocURL’s MCQ-style and short-extractive workloads, the three lenses converge on the same extracted answer, and Trident-S is bypassed by design. We treat Trident-S as a question-format-conditioned extension; an a-priori format router that gates Trident-S invocation is a natural deployment refinement. 5 Limitations and Conclusion Limitations. Trident-R requires an offline per-document indexing pass ($0.256\$0.256/100 pages, amortized across queries; Appendix H). The reranker uses GPT-4.1 here, but smaller or open-source LLMs are drop-in substitutes (Appendix M). The five-pool isolation is conducted on MMLongBench-Doc, with LongDocURL as an external transfer check (§4.5). We use PageIndex only as a section-path parser; public PDF parsers such as PyMuPDF or GROBID could replace it. Finally, since most lift comes from visual captions, a caption-first variant is preferable when other fields are costly. Conclusion. We identify a reranker-interface bottleneck in long-document multimodal QA: evidence selection improves when candidate pages are converted into LLM-readable semantic annotations, even with multimodal retrievers. Trident-R implements this as a retriever-agnostic annotation-based LLM reranker. Across five candidate-pool families on MMLongBench-Doc, the same interface substantially improves retrieval F1, and every reranked pool exceeds PageIndex. The gain on ColPali shows that the effect is not merely missing visual information for text retrievers. Mechanism ablations show that snippet-only LLM reranking barely changes first-hit ranking, while visual captions recover most of the full lift on both text and visual pools. Trident-S is a secondary generation extension: useful for open-ended synthesis, but unnecessary for MCQ and short-extractive workloads. Overall, the results shift the focus from what the retriever scores to what the downstream reranker can see about each candidate page. References Asai et al. (2024) A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi Self-rag: learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, Vol. 2024, p. 9112–9141. Cited by: §2. [2] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu Bge m3-embedding: multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Cited by: §1, §2, §3.4, §4.1. Chen et al. (2025) Z. Chen, C. Xu, Y. Qi, X. Jiang, and J. Guo VLM is a strong reranker: advancing multimodal retrieval-augmented generation via knowledge-enhanced reranking and noise-injected training. In Findings of the Association for Computational Linguistics: EMNLP 2025, p. 8140–8158. Cited by: §1, §2. Cho et al. (2024) J. Cho, D. Mahata, O. Irsoy, Y. He, and M. Bansal M3docrag: multi-modal retrieval is what you need for multi-page multi-document understanding. arXiv preprint arXiv:2411.04952. Cited by: §2. Cormack et al. (2009) G. V. Cormack, C. L. Clarke, and S. Buettcher Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, p. 758–759. Cited by: §2, §3.4, §4.1. Deng et al. (2025) C. Deng, J. Yuan, P. Bu, P. Wang, Z. Li, J. Xu, X. Li, Y. Gao, J. Song, B. Zheng, et al. Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1135–1159. Cited by: §2, §4.1. Dong et al. (2025) K. Dong, Y. Chang, D. G. X. Deik, D. Li, R. Tang, and Y. Liu MMDocIR: benchmarking multimodal retrieval for long documents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 30959–30993. Cited by: §2. Dong et al. (2026) K. Dong, C. YUJING, S. Huang, Y. Wang, R. Tang, and Y. Liu Benchmarking retrieval-augmented multimomal generation for document question answering. Advances in Neural Information Processing Systems 38. Cited by: §2. Faysse et al. (2025) M. Faysse, H. Sibille, T. Wu, B. Omrani, G. Viaud, C. Hudelot, and P. Colombo Colpali: efficient document retrieval with vision language models. In International Conference on Learning Representations, Vol. 2025, p. 61424–61449. Cited by: §1, §2, §3.4, §4.1. Gondhalekar et al. (2025) C. Gondhalekar, U. Patel, and F. Yeh MultiFinRAG: an optimized multimodal retrieval-augmented generation (rag) framework for financial question answering. arXiv preprint arXiv:2506.20821. Cited by: §2. Gong et al. (2025) Z. Gong, Y. Huang, and C. Mai Mmrag-docqa: a multi-modal retrieval-augmented generation method for document question-answering with hierarchical index and multi-granularity retrieval. arXiv e-prints, p. arXiv–2508. Cited by: §2. Gutiérrez et al. (2024) B. J. Gutiérrez, Y. Shu, Y. Gu, M. Yasunaga, and Y. Su Hipporag: neurobiologically inspired long-term memory for large language models. Advances in neural information processing systems 37, p. 59532–59569. Cited by: §2. Jain et al. (2025) C. Jain, Y. Wu, Y. Zeng, J. Liu, S. Dai, Z. Shao, Q. Wu, and H. Wang SimpleDoc: multi-modal document understanding with dual-cue page retrieval and iterative refinement. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 28398–28415. Cited by: §2. Khattab and Zaharia (2020) O. Khattab and M. Zaharia Colbert: efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, p. 39–48. Cited by: §2. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, p. 9459–9474. Cited by: §1, §2. Li et al. (2024) M. Li, É. Gaussier, J. Li, and G. Zhou KeyB2: selecting key blocks is also important for long document ranking with large language models. arXiv preprint arXiv:2411.06254. Cited by: §2. Li et al. (2026) M. Li, Y. Zhang, D. Long, K. Chen, S. Song, S. Bai, Z. Yang, P. Xie, A. Yang, D. Liu, et al. Qwen3-vl-embedding and qwen3-vl-reranker: a unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720. Cited by: §2. Li et al. (2025) Z. Li, Y. Li, L. Kang, D. Karatzas, and W. Ma AVIR: adaptive visual in-document retrieval for efficient multi-page document question answering. In Proceedings of the 7th ACM International Conference on Multimedia in Asia, p. 1–7. Cited by: §2. Lin et al. (2026) J. Lin, K. Hu, B. Wang, Y. Zhou, Z. Xi, H. Guo, S. Liu, J. Wang, S. Dou, E. Zhou, et al. M-doc-r1: training agents for long document visual question answering through multi-turn reinforcement learning. arXiv preprint arXiv:2604.13579. Cited by: §2. Lin et al. (2025) S. Lin, C. Lee, M. Shoeybi, J. Lin, B. Catanzaro, and W. Ping Mm-embed: universal multimodal retrieval with multimodal llms. In International Conference on Learning Representations, Vol. 2025, p. 44215–44234. Cited by: §2. Liu et al. (2024) N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang Lost in the middle: how language models use long contexts. Transactions of the association for computational linguistics 12, p. 157–173. Cited by: §1. Ma et al. (2024a) X. Ma, S. Lin, M. Li, W. Chen, and J. Lin Unifying multimodal retrieval via document screenshot embedding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 6492–6505. Cited by: §2. Ma et al. (2024b) Y. Ma, Y. Zang, L. Chen, M. Chen, Y. Jiao, X. Li, X. Lu, Z. Liu, Y. Ma, X. Dong, et al. Mmlongbench-doc: benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems 37, p. 95963–96010. Cited by: §1, §2, §4.1. Mortaheb et al. (2025) M. Mortaheb, M. A. A. Khojastepour, S. T. Chakradhar, and S. Ulukus Re-ranking the context for multimodal retrieval augmented generation. arXiv preprint arXiv:2501.04695. Cited by: §2. Rahman et al. (2025) A. Rahman, Q. Xu, and X. Huang MPCAR: multi-perspective contextual augmentation for enhanced visual reasoning in large vision-language models. arXiv preprint arXiv:2508.12400. Cited by: §2. Saad-Falcon et al. (2024) J. Saad-Falcon, J. Barrow, A. Siu, A. Nenkova, S. Yoon, R. A. Rossi, and F. Dernoncourt Pdftriage: question answering over long, structured documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, p. 153–169. Cited by: §2. Sanmartin (2024) D. Sanmartin Kg-rag: bridging the gap between knowledge and creativity. arXiv preprint arXiv:2405.12035. Cited by: §2. Santhanam et al. (2022) K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia Colbertv2: effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 3715–3734. Cited by: §2. Sarthi et al. (2024) P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, and C. Manning Raptor: recursive abstractive processing for tree-organized retrieval. In International Conference on Learning Representations, Vol. 2024, p. 32628–32649. Cited by: §2. Shi et al. (2026) Y. Shi, J. Wang, Z. Shan, D. Peng, Z. Lin, and L. Jin URaG: unified retrieval and generation in multimodal llms for efficient long document understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 25357–25365. Cited by: §2. Sun et al. (2023) W. Sun, L. Yan, X. Ma, S. Wang, P. Ren, Z. Chen, D. Yin, and Z. Ren Is chatgpt good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 14918–14937. Cited by: §1, §2, §3.3. Tian et al. (2025) R. Tian, X. Xu, B. Jin, S. Kang, and J. Han CoRank: llm-based compact reranking with document features for scientific retrieval. arXiv preprint arXiv:2505.13757. Cited by: §2. Wasserman et al. (2025a) N. Wasserman, O. Heinimann, Y. Golbari, T. Zimbalist, E. Schwartz, and M. Irani DocReRank: single-page hard negative query generation for training multi-modal rag rerankers. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 8651–8669. Cited by: §2. Wasserman et al. (2025b) N. Wasserman, R. Pony, O. Naparstek, A. R. Goldfarb, E. Schwartz, U. Barzelay, and L. Karlinsky Real-m-rag: a real-world multi-modal retrieval benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 31660–31683. Cited by: §2. Wu et al. (2025) J. Wu, Y. Xia, T. Yu, X. Chen, S. S. Harsha, A. V. Maharaj, R. Zhang, V. Bursztyn, S. Kim, R. A. Rossi, et al. Doc-react: multi-page heterogeneous document question-answering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 67–78. Cited by: §2. Xu et al. (2025) M. Xu, J. Dong, J. Hou, Z. Wang, S. Li, Z. Gao, R. Zhong, and H. Cai Mm-r5: multimodal reasoning-enhanced reranker via reinforcement learning for document retrieval. arXiv preprint arXiv:2506.12364. Cited by: §1, §2. Yoon et al. (2026) S. Yoon, G. Kim, G. Cho, et al. AcuRank: uncertainty-aware adaptive computation for listwise reranking. Advances in Neural Information Processing Systems 38, p. 26388–26417. Cited by: §2. Yu et al. (2025) S. Yu, C. Tang, B. Xu, J. Cui, J. Ran, Y. Yan, Z. Liu, S. Wang, X. Han, Z. Liu, et al. Visrag: vision-based retrieval-augmented generation on multi-modality documents. In International Conference on Learning Representations, Vol. 2025, p. 21074–21098. Cited by: §2. Zhang et al. (2025a) M. Zhang, Y. Tang, and P. Team PageIndex: next-generation vectorless, reasoning-based rag. PageIndex Blog. Note: https://pageindex.ai/blog/pageindex-intro Cited by: §1, §2, §4.1, §4.1. Zhang et al. (2025b) Y. Zhang, J. Lin, M. Zhang, K. Xiao, X. Hou, Y. Zhao, and Z. Li SCRA-vqa: summarized caption-rerank for augmented large language models in visual question answering. In International Conference on Database Systems for Advanced Applications, p. 37–52. Cited by: §2. Zhang and Wu (2026) Y. Zhang and Y. Wu MLDocRAG: multimodal long-context document retrieval augmented generation. arXiv preprint arXiv:2602.10271. Cited by: §2. Zhou et al. (2026) Y. Zhou, F. Zhang, Y. Chen, H. Zhang, P. Nakov, and Z. Xie FinCARDS: card-based analyst reranking for financial document question answering. arXiv preprint arXiv:2601.06992. Cited by: §2. Appendix A Additional End-to-End Results This appendix complements §4.4 and §4.5 with the full end-to-end QA tables on both benchmarks. Table 5 reports all 14 evaluated pipelines on the 300-QA MMLongBench-Doc subset, including the answer-accuracy and unanswerable-abstention columns omitted from the main paper. Table 6 reports the corresponding end-to-end results on the 300-QA LongDocURL stratified subset (all questions answerable), comparing Trident-R against the strongest baseline ColPali ++ annotation ++ rerank, each with and without Trident-S. Method GPT-4.1 J. Claude J. Gen Acc Gen F1 Ans. Acc Unans. Abst. BM25 (Plain) 42.3 42.7 41.3 38.2 33.7 65.3 BM25 ++Trident-S 45.0 44.3 44.8 43.8 39.7 61.1 RRF (Plain) 44.3 44.3 42.6 39.6 36.3 62.5 RRF ++Trident-S 44.7 43.3 43.9 41.7 39.3 58.3 BGE-M3 (Plain) 44.7 44.7 43.5 41.3 37.9 61.1 BGE-M3 ++Trident-S 46.0 45.7 43.9 43.4 41.1 52.8 ColPali (Plain) 48.3 46.3 46.4 45.9 44.4 52.8 ColPali ++Trident-S 50.0 49.0 49.2 48.1 47.2 55.6 PageIndex (Plain) 42.7 43.0 41.3 37.2 29.7 77.8 PageIndex ++Trident-S 49.3 48.3 45.9 42.6 40.6 62.5 BGE-M3 ++ann. ++rk, Plain 48.3 46.7 44.6 42.9 43.4 48.6 BGE-M3 ++ann. ++rk, ++Trident-S 52.0 50.0 48.9 46.4 48.1 51.4 ColPali ++ann. ++rk, Plain 51.0 49.7 47.1 44.5 45.3 52.8 ColPali ++ann. ++rk, ++Trident-S 53.7 52.7 51.2 48.9 50.7 52.8 Trident-R (Plain) 50.7 48.0 47.2 42.9 41.9 63.9 Trident-R ++ Trident-S 52.0 50.7 46.8 44.3 45.4 51.4 Table 5: Full end-to-end QA results on the 300-QA MMLongBench-Doc subset (all values in %). J.=LLM-as-judge accuracy; ++ann.=multi-aspect annotation; ++rk=adaptive-K rerank; Plain=single multimodal VLM call; ++Trident-S=three-lens generation with synthesis. Method GPT-4.1 J. Claude J. Gen Acc Gen F1 Trident-R (Plain) 51.3 47.7 42.6 47.3 Trident-R ++Trident-S 50.0 50.0 41.1 45.5 ColPali++ann. ++rk, Plain 68.7 65.7 56.0 58.0 ColPali++ann. ++rk, ++Trident-S 66.3 66.0 56.7 58.5 Table 6: End-to-end QA on the 300-QA LongDocURL stratified subset (all values in %; all questions are answerable). LongDocURL is dominated by MCQ-style and short-extractive answers, so Trident-S gives little average benefit. Appendix B Trident-R Rerank Prompt The full rerank prompt sent to GPT-4.1 is shown below. Variables in angle brackets are filled at runtime. The candidate JSON list (<candidates_json>) is the array of records described in §3.3; <query> is the natural-language question; <k> is the maximum number of pages (K=5K=5 throughout this work). You are picking the top-K pages most likely to answer the question, from a candidate list. Each candidate page comes with the document signals on it AND a list of ‘concept hits’---high-level concepts that pulled it into the candidate pool. Pages hit by multiple distinct concepts are stronger evidence. Question: <query> Candidate pages (<n> total): <candidates_json> Reply as JSON: ‘thinking’: ‘…’, ‘page_list’: [<page_id>, …] Pick the strongest evidence pages first. Return at most <k> pages. Return only JSON. Each candidate record in <candidates_json> has the form: page_id, retriever_score, concept_hits: [‘T:<name>’, ‘E:<name>’, ‘S:<name>’], section, visual, entities, text The retriever_score field holds the underlying Stage-1 pool score (Okapi BM25 for BM25, cosine for BGE-M3, RRF fused rank for RRF, late-interaction MaxSim for ColPali, or our multi-channel xKG fusion of Eq. 10); the field name is uniform across all pools so the rerank prompt template is pool-agnostic. (In our released code the field is currently named graph_score for historical reasons; the value semantics are as described.) Appendix C Trident-S Lens Prompts Trident-S prompts the VLM three times in parallel with the three lenses below, then synthesizes a final answer with the synthesis prompt. All four calls share the same retrieved page images. (T) Topical lens. Question: <query> You are given <n> page images from a long document. Read them through the TOPICAL lens: what is each page about thematically? Which topics on which pages relate to the question? -- Identify the topical content of each page. -- Focus on how the topics connect to the question. Provide a clear, concise answer based on the topical analysis. If none of the provided pages contain information sufficient to answer the question through this topical lens, reply exactly: ‘Not answerable’. (E) Entity / value lens. Question: <query> …Read them through the ENTITY / VALUE lens: focus on the specific named entities, numerical values, dates, currencies, percentages, table cells, and figure labels that appear on each page. -- Extract every named entity, number, date, or specific value. -- Be precise---use verbatim values from the images. Provide a clear, concise answer focused on the specific entities and values surfaced. If none of the provided pages contain the entities or values needed to answer the question, reply exactly: ‘Not answerable’. (S) Structural / section lens. Question: <query> …Read them through the STRUCTURAL / SECTION lens: focus on where each page sits in the document’s organization---section titles visible on the page, headers, chapter / part markers, document structure, captions of figures / tables. -- Identify each page’s structural role. -- Use captions of tables/figures and section paths. Provide a clear, concise answer that leverages structural context. If the structural context does not contain information needed to answer the question, reply exactly: ‘Not answerable’. Synthesis prompt. You are answering a long-document QA question. We elicited three candidate answers from a vision-language model under three orthogonal analytical lenses applied to the same K=<n> retrieved page images: (T) Topical lens (E) Entity / value lens (S) Structural / section lens Question: <query> Candidate answers: [T-view] <ans_T> [E-view] <ans_E> [S-view] <ans_S> Task: produce a single, clear, concise final answer. Synthesis rules: -- If two or three lenses converge, prefer that answer. -- If only one lens has a precise answer while others are vague, prefer the specific one---unless contradicted. -- If question is numeric, prefer E-view’s value. -- If question is ‘where / which section’, prefer S-view. -- If question is ‘what is the topic’, prefer T-view. -- Match the question’s expected answer format. Do not introduce information beyond what the three candidate answers contain. Do not invent values. If all three lenses agree the question is not answerable, reply exactly: ‘Not answerable’. Appendix D Query-Type Router The router prompts an LLM (GPT-4.1, τ=0τ=0, JSON mode) to classify the question into one of four classes and to extract salient entities. The full prompt is: You are routing a question against a multi-view document index that has three concept types: topical (semantic content clusters), entity-bundle (groups of co-occurring named entities/numbers), and section-derived (structural sections). Question: <query> Output JSON with three fields: ‘entities’: list of salient entities / key terms / numbers from the question. Include named entities, technical terms, monetary amounts, percentages, figure / table references. Drop generic stopwords. ‘class’: one of ‘entity-level’ (asks about a specific named entity / number / figure), ‘topical’ (asks for thematic content / overview / explanation), ‘structural’ (asks about location / section / where in the document), or ‘mixed’ if it spans multiple. ‘rationale’: one short sentence explaining the class choice. Reply with JSON only. The class label determines the channel-fusion weights (αT,αE,αS)( _T, _E, _S) applied when aggregating topical / entity / structural concept scores into a candidate-pool score: Class αT _T αE _E αS _S topical 0.60 0.20 0.20 entity-level 0.20 0.60 0.20 structural 0.20 0.20 0.60 mixed 0.40 0.30 0.30 Replacing this class-conditional router with uniform weights (13,13,13)( 13, 13, 13) causes a small but consistent −0.003-0.003 F1 drop on the cross-pool isolation setup (§4.2, measured on the same N=830N=830 aligned subset as Table 1), validating the router’s contribution. Appendix E 300-QA Subset Construction The 300-QA subset is drawn from MMLongBench-Doc with seed 42. Sampling emphasizes the long-document setting that motivates this work: the subset has median document length 72 pages, compared to 28 in the full set, aligning with our framework’s focus on documents that exceed any current VLM’s input window. Table 7 reports the per-axis distribution match. Axis (category) Full % Subset % Δ p Answer format Float 14.8 20.7 +5.9+5.9 Int 26.4 24.0 −2.4-2.4 List 13.3 14.3 +1.0+1.0 Str 23.1 17.0 −6.1-6.1 None (unans.) 22.4 24.0 +1.6+1.6 Evidence source Chart 16.1 14.2 −1.9-1.9 Figure 26.3 28.9 +2.6+2.6 Layout 10.9 8.0 −2.9-2.9 Plain-text 26.9 28.4 +1.5+1.5 Table 19.8 20.4 +0.6+0.6 Evidence pages 0 (unanswerable) 22.6 25.7 +3.1+3.1 1 44.5 44.3 −0.1-0.1 2-3 26.3 24.3 −2.0-2.0 4+ 6.7 5.7 −1.0-1.0 Document length 0-30 pages 52.9 22.0 −30.9-30.9 31-60 pages 22.6 15.3 −7.2-7.2 61-120 pages 19.4 49.0 +29.6+29.6 121-200 pages 4.7 12.7 +8.0+8.0 200+ pages 0.5 1.0 +0.5+0.5 Table 7: Distribution match between the 300-QA subset and the full MMLongBench-Doc dataset. Most axes (Evidence Source, Evidence Pages, Unanswerable rate) match within ±3± 3p. Answer format is moderately biased toward Float (+5.9+5.9p) and away from Str (−6.1-6.1p); document length is intentionally biased toward longer documents (median 72 vs. 28 pages), aligning with our framework’s long-document focus. The LongDocURL subset of 200 QAs (§4.5) is sampled analogously, stratified by document length, answer format, and evidence source. Released QA-id lists for both subsets accompany the code release. Appendix F Bootstrap CIs, Wilcoxon and McNemar tests Retrieval (N=830). We report (i) non-parametric 95% bootstrap CIs on per-QA F1 (B=10,000B=10,000 resamples drawn with replacement from the 830 aligned QAs); and (i) pairwise Wilcoxon signed-rank tests on per-QA F1 against Trident-R (xKG pool). We prefer Wilcoxon on F1 over McNemar on STRICT for retrieval because methods have different effective K (5 for fixed-K baselines; mean K∈[2.7,3.0]K∈[2.7,3.0] for adaptive-K methods), which would structurally favor the larger-K methods. CIs cluster tightly within ±0.024± 0.024. All five raw baselines are significantly worse than Trident-R (p<0.001p<0.001). Among the ++ann. ++ rerank variants, RRF and BGE-M3 swaps are statistically indistinguishable from Trident-R (p=0.077p=0.077 and 0.0550.055), the BM25 swap is borderline (p=0.026p=0.026), and the ColPali swap is significantly higher (p<0.001p<0.001). Taken together, all five ++ann. ++ rerank rows in Table 1 fall within the [0.503,0.581][0.503,0.581] F1 band without pairwise dominance (except ColPali), validating the pool-agnostic claim. End-to-end (N=300). For end-to-end Gen Acc on the 300-QA MMLongBench-Doc subset we use McNemar on a binary correct-score (≥1≥ 1) outcome per QA, with continuity correction. None of the pairwise comparisons between top-tier pipelines (ColPali++ann. ++Trident-S, Trident-R++Trident-S, BGE-M3++ann. ++Trident-S, PageIndex++Trident-S) reach p<0.05p<0.05 at N=300N=300, reflecting the small sample size; however the ranking is consistent across all three LLM judges (Appendix L, inter-judge κ=0.913κ=0.913). Full pairwise results are released with our code. Appendix G Case Study Records We list the full records for the three case-study QAs in §G. All extracted answers are the GPT-4o-extracted short forms after the MMLongBench-Doc official evaluation pipeline. “Score” is the rule-based Generalized Accuracy score (1.0 = correct, 0.0 = wrong). Case 1: Chart axes. Doc 2401.18059v1.pdf; Q: What are the horizontal and vertical axis of Figure 3 respectively? Gold: [Context Length, F1]. • Trident-R: retrieved pages 6,5\6,5\, ext. “[Context Length, F1]”, score 1.0. • PageIndex: retrieved 3\3\, ext. “Not answerable”, score 0.0. • BGE-M3 raw: retrieved 5,2,16,19,22\5,2,16,19,22\, ext. “Not answerable”, score 0.0. Case 2: Activision R&D ratio. Doc ACTIVISIONBLIZZARD_2019_10K.pdf; Q: What is R&D to asset ratio for Activision Blizzard in FY 2019? Gold: 5.03%. • Trident-R: retrieved 70,69\70,69\, ext. “0.0503”, score 1.0. • PageIndex: retrieved 37,62,67,68,69\37,62,67,68,69\, ext. “Not answerable”, score 0.0. • BGE-M3 raw: retrieved 47,29,92,174,175\47,29,92,174,175\, ext. “Not answerable”, score 0.0. Case 3: Calibrated refusal. Doc 2311.16502v3.pdf; Q: In which image type does GPT-4o demonstrate least proficiency …? Gold: Not answerable. • Trident-R: retrieved 21,111,26\21,111,26\, ext. “Not answerable”, score 1.0. • PageIndex: retrieved 110,111,112\110,111,112\, ext. “Chemical”, score 0.0. • BGE-M3 raw: retrieved 112,7,26,110,2\112,7,26,110,2\, ext. “Music”, score 0.0. Appendix H Detailed Cost Breakdown Token-level cost accounting for Trident-R-Index, measured empirically on the 134-document MMLongBench-Doc corpus (6,492 pages, mean 103103 text tokens per page, 4040 caption tokens per page) at public OpenAI 2025 rates ($2.00/$8.00 per 1M input/output tokens for gpt-4.1; $0.15/$0.60 for gpt-4o-mini; $0.13 per 1M tokens for text-embedding-3-large; image tokens ≈1500≈ 1500 per full page at ZOOM=2.0ZOOM=2.0). Component (per 100-page doc) Model Calls USD Visual caption gpt-4o-mini 100 0.028 Entity extraction gpt-4.1 100 0.201 Page embedding emb-3-large 100 0.001 Concept naming gpt-4.1 20 0.026 Offline total 0.256 Online per-query (avg K=2.85K=2.85): Stage 2 LLM rerank gpt-4.1 1 0.010 Plain VLM generation gpt-4.1 (vision) 1 0.012 Trident-S (3 lens ++ synth) gpt-4.1 (vision) 4 0.039 Table 8: Empirical token-level cost breakdown. Entity extraction dominates offline indexing (78%78\%); concept naming and visual captioning contribute ∼10% 10\% each; page embedding is negligible (1%1\%). The reported $0.256\$0.256 per 100-page document is within the ∼$0.30 \$0.30 range stated in the main text and is conservative for dense documents: doubling per-page text tokens (to ∼200 200) raises total cost to approximately $0.36\$0.36. Per-query cost is dominated by the multimodal VLM call: Plain generation is $0.012\$0.012, Trident-S multi-view is $0.039\$0.039 (3 lens calls ++ 1 synthesis). Appendix I Detailed Field Ablation on Two Pools We complement §4.2 with the full per-field ablation on both the BGE-M3 (text-only) and ColPali (visual) pools. Each row removes exactly one annotation field from the rerank candidate JSON. Configuration F1 Δ 1 Recall Prec. BGE-M3 pool (text retriever, F1full=0.546F1_full=0.546) ++ full annotation 0.546 — 0.726 0.499 drop visual_caption 0.416 −0.130-0.130 0.657 0.354 drop concept_hits 0.515 −0.032-0.032 0.688 0.477 drop entities 0.507 −0.039-0.039 0.701 0.455 caption-only (only visual_caption) 0.518 −0.028-0.028 0.669 0.484 ColPali pool (visual retriever, F1full=0.581F1_full=0.581) ++ full annotation 0.581 — 0.766 0.533 drop visual_caption 0.498 −0.083-0.083 0.715 0.444 drop concept_hits 0.593 +0.013+0.013 0.751 0.556 drop entities 0.588 +0.007+0.007 0.773 0.542 caption-only (only visual_caption) 0.554 −0.027-0.027 0.723 0.516 Table 9: Per-field annotation ablation on two retrievers (N=830N=830 aligned, Kmax=5K_ =5). Standard rows: remove one field at a time. Caption-only rows: keep only the visual caption (drop concept_hits, section path, entity tags, text snippet). Caption alone achieves ∼90% 90\% of the full annotation lift on both pools (−0.028-0.028 / −0.027-0.027 from full), with the gap remarkably pool-invariant. Combined with the asymmetric drop-caption result (−0.130-0.130 on BGE-M3 vs. −0.083-0.083 on ColPali; 36%36\% smaller on visual pool), this rules out pure modality completion as the sole mechanism: the lift stems from structuring page evidence as LLM-readable input, with visual caption acting as a semantic interface for the text-only LLM reranker (effective regardless of whether the upstream retriever is visual or textual). Appendix J Adaptive-K Distribution by Question Type We bin the rerank-selected K by MMLongBench-Doc question type (using the evidence_sources field) across four annotated-rerank pools. Question type BM25 BGE-M3 ColPali Trident-R ++ann ++ann ++ann (xKG)++ann Pure-text 2.95 3.08 3.11 2.94 Figure 2.64 2.61 2.60 2.71 Table 2.90 2.90 2.98 2.75 Chart 2.81 2.77 2.55 2.59 Layout 2.84 2.78 2.97 2.75 Multi-source 3.01 2.98 2.95 2.97 Unanswerable 2.69 2.72 2.67 2.62 Table 10: Mean adaptive K per question type, by candidate pool (Kmax=5K_ =5). Two patterns: (1) K is highly stable across pools (range 2.552.55–3.113.11), suggesting that the reranker’s K choice is driven by the question’s evidence requirement, not by which retriever fed the pool. (2) K varies by question type: Pure-text questions request the most pages (K¯≈3.0 K≈ 3.0, K=5K=5 cap rate ∼30% 30\%); Figure / Chart questions request fewer (K¯≈2.6 K≈ 2.6, K=1K=1 rate >30%>30\% on Figure), consistent with visual evidence being concentrated on individual pages. Appendix K Visual Caption QC: Substring and N-gram Overlap To address the concern that VLM-generated captions might leak the gold answer text and thereby trivialize Stage 2 reranking, we audit 200200 randomly sampled answerable QA / candidate-page pairs (N=238N=238 caption-page records after dropping “Text-only page.” captions). • Exact substring (gold answer string contained verbatim in caption): 22/238=9.2%22/238=9.2\%. • Bigram Jaccard ≥0.3≥ 0.3 (significant sentence-level overlap): 0/238=0.0%0/238=0.0\%. • Unigram coverage ≥0.7≥ 0.7 of gold tokens in caption: 32/238=13.4%32/238=13.4\%. The 0%0\% bigram match indicates no caption near-verbatim paraphrases the answer sentence; the 9.2%9.2\% substring rate reflects captions correctly encoding visual content (e.g. chart legend colours such as “Blue”, or diagram labels such as “Elastic Compute Service”) that happen to coincide with the gold answer — the intended behaviour of caption-as-visual-evidence rather than answer leakage. Appendix L Inter-Judge Agreement We use two independent LLM judges (GPT-4.1 and Claude Sonnet 4.5) to score the same 300-QA end-to-end gen outputs from each of the 1414 pipelines in Table 5. Both judges use the identical 33-level rubric (correct / partial / wrong) and the same MMLongBench-Doc gold answers. Table 11 reports per-pipeline pairwise agreement and Cohen’s κ. Pipeline Agree. Cohen κ BM25 (Plain / MAVS) 94.094.0 / 96.796.7 0.890.89 / 0.940.94 BGE-M3 (Plain / MAVS) 94.394.3 / 97.397.3 0.900.90 / 0.950.95 ColPali (Plain / MAVS) 93.093.0 / 95.795.7 0.870.87 / 0.920.92 PageIndex (Plain / MAVS) 97.397.3 / 97.797.7 0.950.95 / 0.960.96 BGE-M3++ann. (Plain / MAVS) 93.793.7 / 96.396.3 0.890.89 / 0.930.93 Trident-R (Plain / MAVS) 92.792.7 / 94.794.7 0.870.87 / 0.910.91 ColPali++ann. (Plain / MAVS) 93.093.0 / 95.795.7 0.880.88 / 0.920.92 Mean (14 pipelines) 95.1% 0.913 Range 92.792.7–97.797.7 0.870.87–0.960.96 Table 11: Inter-judge agreement and Cohen’s κ between GPT-4.1 and Claude on the same 300-QA outputs. Mean κ=0.913κ=0.913 corresponds to “almost perfect agreement” on the Landis-Koch scale. The pipeline-level ranking under the two judges is identical (ColPali ++ ann. ++ rk ++ Trident-S is best under both; Trident-R ++ Trident-S second under Claude, tied second under GPT). This indicates that the observed end-to-end ranking is not dependent on a single judge’s quirks. We do not perform human evaluation due to scale (14×300=4,20014× 300=4,200 prediction-gold pairs); we view the high inter-judge agreement and the agreement of pipeline rankings between two independently developed LLMs as a reasonable proxy. Appendix M Extended Limitations Discussion We elaborate the limitations summarized in §5. Reranker LLM dependence. The rerank stage in this work uses GPT-4.1 at τ=0τ=0. We expect the annotation-conditioned rerank protocol to be reranker-agnostic in principle, because the candidate JSON exposes per-page evidence in a form that any instruction-tuned LLM can consume. A pilot replacement with a smaller reranker would falsify or strengthen this expectation; we leave a full small-reranker / open-source-reranker sweep for follow-up work due to budget constraints. LongDocURL coverage. LongDocURL serves as an external transfer check: the retrieval-side comparison (Table 4) reports adaptive-K methods on the full N=1,122N=1,122 QAs with non-empty gold evidence, but the full five-pool cross-pool isolation conducted on MMLongBench-Doc is not separately replicated on LongDocURL. End-to-end LongDocURL results (Appendix A, Table 6) confirm the broad pattern. Trident-S on extractive workloads. On LongDocURL’s MCQ-style and short-extractive questions, the topical, entity, and structural lenses converge on the same extracted answer; the synthesis call therefore contributes mean Δ≈0 ≈ 0 across four LongDocURL pipelines (Appendix A, Table 6). This motivates the on-demand deployment regime described in §3.5: Trident-S is gated by question format. Parser substitution. We use the PageIndex API as a structural parser for the section-path field while also benchmarking against the PageIndex retrieval system; the two roles are independent. Any public PDF parser that yields a section tree (PyMuPDF, GROBID) could fill this role. A parser-substitution robustness study is left to future work.

Instant research discovery

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

Intelligence

Summary

Entities (12)

Relation Signals (9)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations