Paper deep dive
Document-as-Image Representations Fall Short for Scientific Retrieval
Ghazal Khalighinejad, Raghuveer Thirukovalluru, Alexander H. Oh, Bhuwan Dhingra
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 6/21/2026, 11:37:24 AM
Summary
The paper introduces ArXivDoc, a new benchmark for open-domain scientific document retrieval designed to address the limitations of 'document-as-image' representations. The authors argue that current models, which embed rendered pages as images, struggle with text-rich, multimodal scientific documents where evidence is distributed across structured LaTeX elements like text, tables, and figures. ArXivDoc is constructed from raw LaTeX sources of 8,210 scientific papers, providing direct access to structured content. The study compares various representation paradigmsâtext-only, image-based, and interleaved text+imageâand finds that text-based representations (especially when augmented with VLM captions) and interleaved models outperform document-as-image approaches. The benchmark features 547 manually verified, decontextualized, and naturalized queries grounded in specific evidence types.
Entities (7)
Relation Signals (4)
Text-based representations â aremoreeffectivethan â Document-as-Image
confidence 100% · text-based representations are most effective, even for figure-based queries
ArXivDoc â isconstructedfrom â LaTeX
confidence 100% · ArXivDoc, a new benchmark constructed from the underlying LaTeX sources of scientific papers.
ArXivDoc â outperforms â Document-as-Image
confidence 100% · interleaved textâimage representations outperform document-as-image approaches
ArXivDoc â competeswith â ViDoRe
confidence 90% · The paper compares ArXivDoc against existing benchmarks like ViDoRe and ArXivQA.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Many recent document embedding models are trained on document-as-image representations, embedding rendered pages as images rather than the underlying source. Meanwhile, existing benchmarks for scientific document retrieval, such as ArXivQA and ViDoRe, treat documents as images of pages, implicitly favoring such representations. In this work, we argue that this paradigm is not well-suited for text-rich multimodal scientific documents, where critical evidence is distributed across structured sources, including text, tables, and figures. To study this setting, we introduce ArXivDoc, a new benchmark constructed from the underlying LaTeX sources of scientific papers. Unlike PDF or image-based representations, LaTeX provides direct access to structured elements (e.g., sections, tables, figures, equations), enabling controlled query construction grounded in specific evidence types. We systematically compare text-only, image-based, and multimodal representations across both single-vector and multi-vector retrieval models. Our results show that: (1) document-as-image representations are consistently suboptimal, especially as document length increases; (2) text-based representations are most effective, even for figure-based queries, by leveraging captions and surrounding context; and (3) interleaved text+image representations outperform document-as-image approaches without requiring specialized training.
Tags
Links
- Source: https://arxiv.org/abs/2604.18508v1
- Canonical: https://arxiv.org/abs/2604.18508v1
Trouble viewing inline? Open PDF directly â
Full Text
62,059 characters extracted from source content.
Expand or collapse full text
Preprint. Under review. Document-as-Image Representations Fall Short for Scientific Retrieval Ghazal Khalighinejad, Raghuveer Thirukovalluru, Alexander H. Oh, Bhuwan Dhingra Department of Computer Science Duke University Abstract Many recent document embedding models are trained on document-as-image repre- sentations, embedding rendered pages as images rather than the underlying source. Meanwhile, existing benchmarks for scientific document retrieval, such as ArX- ivQA and ViDoRe, treat documents as images of pages, implicitly favoring such representations. In this work, we argue that this paradigm is not well-suited for text-rich multimodal scientific documents, where critical evidence is distributed across structured sources, including text, tables, and figures. To study this setting, we introduce ArXivDoc, a new benchmark constructed from the underlying LaTeX sources of scientific papers. Unlike PDF or image-based representations, LaTeX provides direct access to structured elements (e.g., sections, tables, figures, equa- tions), enabling controlled query construction grounded in specific evidence types. We systematically compare text-only, image-based, and multimodal representations across both single-vector and multi-vector retrieval models. Our results show that: (1) document-as-image representations are consistently suboptimal, especially as document length increases; (2) text-based representations are most effective, even for figure-based queries, by leveraging captions and surrounding context; and (3) interleaved text+image representations outperform document-as-image approaches without requiring specialized training. Standard Document Retrieval Document as Image LLM Representations Ours (ArXivDoc) Figures Text Her e the r eson anc e ener gy $ epsilon$ is fixe d by en ergy co nser va tion . $ epsilon$ is plo tte d in Fig. ~2(b ) (blac k cur ve ) as a fu nct ion of t he m agnet ic fie ld $B$ t hat cont rols t he in ter actio n str engt h $g$ ( red [gr ay] cur ve) in t he expe rim ent . By in sert ing expr ession s (5) an d (6) in to (2 ) one o bta ins th e imp ort ant r esult gineq uat ion J_ 0, epsilon = t[ phi_ thr mQP ^*( x)\, acd i_ epsilon(x) dx - i_ epsilon(x)\, acd phi_ thr mQP ^*( x) dx ht]_x= x_ thr mba r , g7 dequ atio n with $ phi_ thr mQP (x) $ being t he QP WF of th e at om es capin g from the tra p, def ined a s gineq uat ion phi_ thr mQP (x) = t2 _- ty ty dx'\, i_0 (x')\, i_ thr mCM ^*\! t( acx'+x2 ht)\, i_r^*(x' - x). g8 dequ atio n Latex Source Figures LLM Text Her e the r eson anc e ener gy $ epsilon$ is fixe d by en ergy co nser va tion . $ epsilon$ is plo tte d in Fig. ~2(b ) (blac k cur ve ) as a fu nct ion of t he m agnet ic fie ld $B$ t hat cont rols t he in ter actio n str engt h $g$ (r ed [gr ay] cur ve) in t he ex pe rim ent. By in sert ing ex pr ession s (5) an d (6) in to (2 ) one ob tains t he im por tant resu lt gineq uat ion J_ 0, epsilon = t[ phi_ thr mQP ^*( x)\, acd i_ epsilon(x ) dx - i_ epsilon(x)\, acd phi_ thr mQP ^*( x dequ atio n Representations Scientific Search Figure-based Query Table-based Query Text-based Query Proj. Vision Encoder Scientific Search Figure-based Query Table-based Query Text-based Query Vision Encoder or Captions Figure 1: Comparison of document representation paradigms for scientific retrieval. Standard document-as-image approaches process rendered pages through a vision encoder, which handles figure-based queries but struggles to accurately ground text- and table-based evidence. In contrast, ArXivDoc, leverages underlying LaTeX source files to process text and figures natively. This preserves fine-grained document structure, enabling robust retrieval across all multimodal query types. 1 arXiv:2604.18508v1 [cs.IR] 20 Apr 2026 Preprint. Under review. 1 Introduction Scientific document retrieval requires locating evidence that may appear in text, equations, tables, or figures. For example, a query about a scaling law may depend on an equation and its surrounding discussion, while a query about model architecture may rely on a diagram, and one about experimental results may rely on a specific table row. Yet most existing retrieval systems represent documents either as plain text (e.g., Qwen3-Embedding (Zhang et al., 2025c), Llama-Embed (Babakhin et al., 2025)) or as images (e.g., ColPali (Faysse et al., 2025), Qwen-VL-Embedding (Li et al., 2026)). However, it is not clear what trade-offs are made when representing textual and tabular content as images, nor how these representations scale to longer scientific documents. This raises a fundamental question: which document representations are most appropriate for scientific retrieval, and under what conditions? Recent work has increasingly favored document-as-image representations for embedding, driven by the success of visionâlanguage models (VLMs) that encode rendered page images directly into dense vectors (Yu et al., 2024; Ma et al., 2024a; Faysse et al., 2025; MacĂ© et al., 2025; GĂŒnther et al., 2025; Zhang et al., 2025b). However, this design choice introduces a trade-off. Scientific papers are typically generated from structured markup languages such as LaTeX or XML, which explicitly encode document organization and content types. Representing them as images obscure this structure: boundaries between content types must be inferred visually, cross-references are no longer explicit, and distinctions between prose, equations, and figures are not machine-readable. Consequently, models must encode all contentâincluding text, equations, and tablesâas pixels rather than text tokens. As more text is packed into a page, it is unclear whether such image-based representations remain effective, especially in dense scientific documents where relevant evidence often appears in prose, equations, or tables rather than figures. Empirical evidence supports this concern: PixelWorld shows that pixel-based inputs degrade more on text-centric reasoning tasks, and recent work finds that VLM performance on scientific documents varies with layout and rendering templates, suggesting sensitivity to surface-level presentation choices (Lyu et al., 2025; Cheng et al., 2025). More broadly, there is an inherent trade-off between textual and visual document representations. Text-based representations align well with language model pretraining and preserve explicit semantic content such as terminology, equations, and logical structure, while visual representations capture layout and multimodal cues but require models to infer content boundaries and relationships implicitly from appearance (Faysse et al., 2025; Wei et al., 2025; Lyu et al., 2025; Jiang et al., 2024). Much of the recent shift toward visual representations has been driven by document collections where accurate text extraction or structural markup is unavailable (MacĂ© et al., 2025; Ma et al., 2024b; Cho et al., 2024). As a result, these approaches rely on OCR tools (Smith, 2007) to recover text, which are prone to errors. In contrast, scientific papers provide clean LaTeX (for those on ArXiv) or XML sources that preserve both content and structure, offering a unique testbed for systematically comparing textual and visual document representations under controlled conditions. For papers without available sources, dedicated tools can recover structured LaTeX from PDFs; for example, Mathpix 1 provides high-quality parsing into LaTeX. In this work, we introduce ArXivDoc, a benchmark for analyzing scientific document retrieval across representations. Using raw LaTeX sources, we construct a corpus of8, 210documents containing 144, 653pages, and generate547targeted, evidence-grounded queries that are manually verified. We further build multiple document representationsâtext-only, figure-only, text + VLM captions, document-as-image, and interleaved textâimageâenabling controlled comparison within a single framework. Each query is explicitly grounded in text, tables, or figures, allowing fine-grained analysis of how retrieval performance varies with evidence type. Our main findings are as follows: 1.Document-as-image is consistently suboptimal, even for figure-based queries where visual representations might be expected to excel. 2.Text + VLM captions achieves the strongest overall performance, demonstrating that augmenting text with visionâlanguage model descriptions of figures and tables is more effective than replacing text with images. 1 https://mathpix.com 2 Preprint. Under review. 3.Interleaved textâimage representations outperform document-as-image. We reuse the same embedding models that were originally trained on page-level document images, and apply them to interleaved textâimage inputs without any additional training. Despite this mismatch, they still outperform document-as-image representations, suggesting that combining modalities is more robust than relying on rendered pages alone. 4.As more text is added to a page, document-as-image representations degrade faster than text-based representations. This makes text-based representations more suitable for long scientific documents. 5. Single-vector models outperform late interaction ones despite much smaller index sizes. Single-vector models (e.g., Qwen-Embedding, Qwen-VL-Embedding), which encode each document into a single embedding, consistently achieve better retrieval performance than late interaction models (e.g., ColQwen), which represent documents as multiple vectors. Despite this added complexity, late interaction methods require up toâŒ40Ălarger index sizes while underperforming, indicating a less favorable efficiencyâperformance trade-off. add other retrieval datasets. compare text/page Benchmark# Docs# Pages# Queries# Tokens/PageTarget UnitModalitySource AvailabilityOpen-DomainScientific QASper1,585â5,049âDocumentTextâââ ArXivQA100kâ100kâFigure Imageââ MRMRâ26,2231,435421Webpage Text+Imageâ ViDoRe V1â5,000âPage/Figure Imageâ ViDoRe V2663,2663,000488PageImageââ NL-DIRâ41,000205kâPageImageâ MMDocIR313â1,658700PageText+Imageââ ArXivDoc (Ours)8,210144,653547958DocumentText+Imageâ Table 1: Comparison of document-level multimodal and scientific retrieval benchmarks. Target Unit indicates the expected retrieval granularity (e.g., retrieving a specific page vs. the entire document). Modality indicates the default representation of the corpus. ArXivDoc is the only large-scale, open- domain scientific retrieval benchmark providing raw LaTeX source access alongside rendered images. 2 Related Work In this section, we position ArXivDoc within existing document retrieval and understanding bench- marks, highlighting the key gaps that remain. Table 1 summarizes these differences. ArXivDoc is an open-domain scientific document retrieval task: given a query, the model must retrieve the single relevant paper from a corpus of more than8,000arXiv documents. Queries are context-independent and grounded in specific evidence (text, tables, or figures). This differs from benchmarks such as ArXivQA (Li et al., 2024), which focus on retrieving from a pool of figures rather than complete multimodal documents. Several document benchmarks do not operate in a true retrieval setting. For instance, QASPER (Dasigi et al., 2021) assumes the document is given and evaluates question answering for queries within a paper, while MMLongBench-Doc (Ma et al., 2024b), MMDocIR (Dong et al., 2025), and UniDoc- Bench (Peng et al., 2026) focus on restricted or non-open-domain queries where the relevant context is already provided. As a result, queries do not require identifying the correct document from a large corpus, and models are not challenged to distinguish among many candidatesâsidestepping the central difficulty of scientific retrieval. Other benchmarks evaluate retrieval at the page level using document images. ViDoRe V1 and V2 (Faysse et al., 2025; MacĂ© et al., 2025) retrieve relevant pages from visually rich documents, and NL-DIR (Guo et al., 2025) extends this paradigm to larger-scale image-based retrieval. In these settings, documents are decomposed into pages rather than treated as unified documents. While effective for visually salient content, this formulation is less suitable for scientific papers, where evidence is often distributed across text, equations, tables, and figures spanning multiple pages. Additionally, while OCR can be applied to extract text or isolate elements such as figures and tables, it requires additional preprocessing (Choi et al., 2025; Han et al., 2025) that is often error-prone and computationally expensive, introducing noise and variability in the extracted content. As a result, it 3 Preprint. Under review. In what way can strong short-range repulsion make the escape rate of a paired two-particle state match that of a noninteracting antisymmetric pair...? Step 1: Generate query from paper Generation How can strong short- range repulsive interactions cause the escape (tunneling) rate of a paired two-particle state to equal that of a noninteracting antisymmetric state despite different in-trap spatial densities...? Verification The question is context- independent and uses interpretable physics concepts. How can strong short- range repulsion make the tunneling escape rate of a paired two-particle state match that of a noninteracting antisymmetric two- particle state, and what boundary-coupling approximation at the trap edge explains this despite different in-trap densities? Naturalize Is the query natural- looking? No Rewrite it to make it more natural. How can strong short-range repulsion make the tunneling escape rate of a paired two-particle state match that of a noninteracting antisymmetric two- particle state? Final Query Is it non-ambiguous and answerable by a specific paper? Yes Step 2: Make the query self-contained Step 3: Does BM25 fail to retrieve the source? Step 4: Make more natural Generated Queries Decontext. Queries Verified Queries Naturalized Queries Example: Decontextualize Figure 2: Dataset construction pipeline. Query counts across stages: LLM verification reduces queries from 761â525 (text), 650â159 (figure), and 2648â280 (table); manual verification further reduces them to 229 (text), 100 (figure), and 218 (table). becomes difficult to systematically evaluate how different document representations (e.g., text vs. image) impact retrieval performance. MRMR (Zhang et al., 2025a) is closer in spirit to our setting in that it studies open-domain re- trieval over multimodal documents with access to structured sources. However, the underlying document distribution differs substantially: MRMR operates over webpages, which are typically less information-dense and less structurally explicit than scientific papers, where critical evidence is often embedded in tightly coupled text, equations, tables, and figures. More importantly, the task objectives diverge. ArXivDoc is designed to reflect a realistic scientific search scenario, where a natural, context-independent query must be used to identify the single correct paper from a large corpus of over8,000documents. In contrast, MRMR emphasizes reasoning-intensive retrieval, where queries are often highly specific and tied to localized webpage content, frequently requiring deep multimodal interpretation (e.g., understanding an image) to resolve. ArXivDoc addresses these gaps. It combines: (1) open-domain retrieval, (2) scientific documents, and (3) document-level retrieval. In addition, it provides access to the underlying document sources (e.g., LaTeX when available) alongside rendered pages, which allows us to compare text-based and page-based representations directly. Finally, queries are grounded in specific evidence types (text, tables, figures), enabling controlled evaluation across modalities. 3 Dataset Page distribution RangeCount% 1â5 pages7308.9% 6â10 pages2,17026.4% 11â20 pages3,15238.4% 21â40 pages1,70120.7% 41+ pages4555.5% Token distribution RangeCount% <1k200.2% 1kâ5k3003.7% 5kâ10k2,42929.6% 10kâ20k3,31740.4% 20k+2,14426.1% Table 2: Page and token distribu- tions We introduce ArXivDoc, an open-domain scientific document re- trieval benchmark constructed from raw LaTeX sources. The LaTeX source is used to compute document representations and to generate queries, without relying on OCR-extracted PDF text. This allows document content and structure to be preserved consis- tently across all representations derived from the same underlying source. The benchmark contains 8,210 full-length scientific papers, span- ning substantially more pages than prior multimodal retrieval benchmarks (Table 1). This scale supports evaluation over large document collections in which many papers share similar topics, notation, and experimental structure. Example documents and queries are provided in Appendix C. Queries are generated, filtered, and verified through a multi-stage pipeline usinggpt-5.2, and are then manually rewritten, edited, 4 Preprint. Under review. DatasetExample query ArXivQAWhat does the black sphere with an arrow represent in these diagrams? ViDoRe V1What process is being depicted in the figure? MMDocIRWhat variable is being altered to create the different patterns in each subplot? ArXivDoc Why do UV-based measurements of the low-redshift star formation rate density often come out higher than other estimates? Table 3: Example queries from existing benchmarks and ArXivDoc. Existing benchmarks often include short or context-dependent queries tied to local visual content, whereas ArXivDoc emphasizes decontextualized queries designed to retrieve a single target document from a large corpus. and validated by human annotators. In total, the dataset contains 547queries:100figure-based,218table-based, and229text-based queries. Design Trade-offs in Query Construction. Retrieval benchmark design involves a fundamental trade-off between three competing objectives: (i) naturalness of queries, (i) decontextualization (self-contained queries that do not rely on implicit document references), and (i) unambiguous ground truth. In practice, these objectives are often in tension. Natural queries tend to be short and underspecified, relying on implicit context and permitting multiple valid answers, which increases ambiguity and leads to false negatives. This issue is evident in existing datasets such as ArXivQA (Li et al., 2024), MMDocIR (Dong et al., 2025), and ViDoRe V1 (Faysse et al., 2025), where queries are frequently ambiguous and admit multiple valid matches, making them less suitable for document retrieval (e.g., âWhat is plotted along the x axis?â from ViDoRe can correspond to many documents). We also identified multiple false negatives in ViDoRe V2: despite its framing as blind contextual querying, some queries remain close-domain or admit multiple plausible matches. Other benchmarks avoid this problem by enforcing decontextualization during dataset construction; for example, MultimodalQA (Talmor et al.; Cho et al., 2024) uses formal languages to generate cross- modal queries. However, such design choices can come at the expense of naturalness. ArXivDoc instead prioritizes decontextualization and unambiguous ground truth while keeping queries as natural as possible, enabling controlled evaluation of open-domain scientific document retrieval over a large corpus, where the goal is to identify the correct document among many topically similar papers. In this setting, minimizing ambiguity is essential for reliable evaluation. Table 3 shows examples of queries from different datasets. 3.1 Query Generation and Filtering We construct open-domain retrieval queries from three distinct evidence types present in scientific documents: text, tables, and figures. Each evidence type defines a separate pool of candidate queries, and the same multi-step generation and filtering pipeline is applied independently to each pool. We construct text-based queries from LaTeX (.tex) content, figure-based queries from figures in a LaTeX source, and table-based queries from tableenvironments. For each query type, we apply a multi-step pipeline: synthetic query generation, decontextualization, difficulty-based filtering, verification, and naturalization (see Figure 2). After this process, expert human annotators review, edit, and filter the resulting queries to ensure quality and validity. Prompt templates used at each stage, and annotation instruction are provided in Appendix B. We note that queries in ArXivDoc are intentionally grounded in localized evidence (a specific text span, table, or figure) rather than requiring multi-hop reasoning across multiple document components. This design choice reflects our focus on document retrieval rather than document-level reasoning: the primary challenge is identifying the correct document from a large corpus, not aggregating evidence within a document. Generation For each evidence source (text, table, or figure), we promptgpt-5.2to generate a single query targeting the underlying scientific content. The prompt enforces that the query (i) 5 Preprint. Under review. requires expert-level reasoning (e.g., about implications, trends, limitations, or constraints), (i) avoids direct restatement and minimizes lexical overlap through abstraction and paraphrasing, (i) is answerable from the document without relying on keyword or phrase matching or referencing document-specific elements (e.g., sections, figures, or experiment names), and (iv) consists of exactly one realistic, concise sentence. Full prompt templates are provided in Appendix B. Decontextualization.We observe that many generated queries are context-dependent. For example, the query âCan the top polyhedron be obtained from the cube by a shear?â relies on figure-specific references. After decontextualization, it becomes âCan an oblique parallelepiped be obtained from a cube by an affine shear?â, which removes these references and introduces the required geometric terminology. To make these queries compatible with open-domain retrieval, we rewrite each synthetic query, usinggpt-5.2, into a context-independent form that removes explicit references to figures, tables, or document-local structure. However, this step often produces queries that are only superficially decontextualized. Many rewritten queries remain underspecified, as they do not introduce sufficient scientific context to stand on their own. For example: Original: If the variable on the x-axis represents time, what can be inferred about the rate of change of the parameter over time? Rewritten: If the independent variable represents time, what can be inferred about the rate of change of the parameter over time? Althoughgpt-5.2is supposed to returnnullwhen it cannot generate a valid rewrite, it frequently outputs very slight paraphrases like the one above, which are still unclear without the original context. This makes a final verification step necessary. Difficulty-Based Filtering. We remove trivially easy queries using an automated difficulty filter that leverages retrieval behavior. For each query, we run BM25 (LĂč, 2024) over a chunked document corpus and remove the query if its gold document ranks within the top five results, as this suggests the answer can be found through shallow lexical overlap. This step eliminates roughly40%of all queries. Final Verification. All remaining queries undergo a final verification step usinggpt-5.2to ensure that they constitute valid open-domain retrieval queries. This verification checks that queries are interpretable without document context. After verification, the query counts are reduced from761 to 525 (text), 650 to 159 (figure), and 2648 to 280 (table). Human Annotation Each query is manually evaluated along three dimensions: naturalness, ambiguity, and document answerability. Queries that are unclear, underspecified, or unsupported by evidence in the gold document are rewritten or removed. In total, we involve three annotators: two PhD students and one Masterâs student. A Masterâs student reviews all queries using the following criteria: queries must be clear and plausible (naturalness), uniquely identify a target document (ambiguity), and be directly supported by evidence in the gold document (document answerability). Further annotation details are provided in Appendix 8. 4 Experiments 4.1 Problem Formulation We study open-domain scientific document retrieval. LetD = d 1 , . . . , d N denote a corpus of scientific documents. Each document d i is represented as a collection of embedding unitse i , where e i =e i1 , . . . , e iM i , ande ij denotes thej-th embedding unit of documentd i . Each unit corresponds to a document component such as a text chunk, figure, or page, depending on the chosen representation and model. Letqdenote a context-independent natural language query targeting a specific piece of scientific evidence (text, table, or figure). 6 Preprint. Under review. Given a queryq, a retrieval system computes a similarity score betweenqand each embedding unite ij âe i using a scoring functions(q, e ij ). The document-level score ford i is then defined as S(q, d i ) = max e ij âe i s(q, e ij ), and documents are ranked according to S(q, d i ). 4.2 Experimental Setup We use our dataset, ArXivDoc, to assess how different document representations support open- domain retrieval from scientific papers. Starting from the underlying LaTeX source, we construct multiple representations of the documents and compare their retrieval performance. We also report the corresponding index sizes. Retrieval performance is measured using normalized discounted cumulative gain at rank 10 (nDCG@10), which evaluates whether the relevant document is ranked near the top of the retrieval list. Representations and Models. We consider three classes of document representations: text-only, image-only, and text + image. Within each class, we evaluate multiple representations. Text-only. (i) Text (L A T E X), which indexes raw L A T E X source text. We first flatten the source to a single file to account for projects split across multiple.texinputs (e.g., via ). We then apply lightweight normalization to remove non-semantic markup: comments and common formatting commands (e.g., , , , , and styling macros such as , ), while preserving scientific content such as plain text, math, and structure. (i) Text + VLM Captions, which augments document text with figure descriptions generated by a visionâlanguage model, appended to the end of the document. Image-only. (i) L A T E X Figures, which indexes rendered figures extracted from the L A T E X sources while ignoring document text. We collect all figure assets (e.g.,.png,.jpg,.pdf,.eps) and convert them into a unified format prior to embedding. This representation isolates visual content, but may miss critical information when the evidence required to answer a query resides in the text. Text + Image. (iv) Document-as-Image, which indexes full document pages rendered as images. (v) Interleaved Text + Images, which jointly indexes text and figures while preserving their original order. We parse the L A T E X source to extract textual spans and figure references, resolve each reference to its rendered image, and construct an interleaved sequence reflecting the documentâs narrative flow. Retrieval units are formed by segmenting text into chunks and associating each chunk with nearby figures, producing multimodal units with one or two images. Across these representations, we evaluate several embedding models, depending on modality com- patibility: Qwen3-Embedding-8B (Zhang et al., 2025c) as a text-only embedder, and Qwen3-VL- Embedding-8B (Li et al., 2026), ColQwen2 v1 (Faysse et al., 2025), OpenCLIP ViT-G/14 (Cherti et al., 2023; Radford et al., 2021), and Ops-M-Embedding v1 (Lin et al.) as image embedders. For each representationâmodel pair, we report results using the best-performing configuration; full hyperparameter sweeps are reported separately. ColQwen is a late-interaction model based on the ColBERT framework (Khattab & Zaharia, 2020; Santhanam et al., 2022), built on top of the Qwen2-VL (Wang et al., 2024a), which encodes queries and documents into sets of token-level embeddings and computes document relevance via max- similarity aggregation across embedding units, resulting in substantially larger indices than single- vector models. In contrast, Qwen, OpenCLIP, and Ops-M-Embedding produce a single embedding per input unit and rely on standard vector similarity for retrieval. ColQwen was originally introduced for page-level visual document retrieval and is not trained on text documents. Nevertheless, we apply ColQwen to text chunks by treating each chunk as an embedding unit and using the modelâs language encoder. This allows us to evaluate a late-interaction retrieval model on purely textual representations and to compare its behavior directly with single-vector text embedding models under identical document inputs. Surprisingly, we find that ColQwen is effective on textual inputs, which motivates our experiments with interleaved textâimage representations; as we show, these outperform document-as-image representations when encoded using the same model. We further observe that Qwen-VL-Embeddingâdespite not being trained for text embedding, similar to ColQwenâexhibits the same trend, achieving better performance on interleaved text + image representations than on document-as-image inputs. 7 Preprint. Under review. InputModelIndex (GB)TextTableFigureAvg. Text Only OpenCLIP1.280.180.120.480.21 Ops-M-Embedding2.240.750.520.750.66 ColQwen40.540.770.560.800.69 Qwen3-Embedding2.560.870.600.760.74 Text + VLM Captions OpenCLIP1.650.190.110.500.21 Ops-M-Embedding2.880.740.500.750.65 ColQwen52.100.740.540.900.69 Qwen3-Embedding1.540.870.600.800.75 Figures Only OpenCLIP0.490.030.070.600.15 Ops-M-Embedding0.860.260.250.800.36 ColQwen35.430.180.180.870.30 Qwen3-VL-Embedding1.050.220.250.830.35 Doc-as-Image OpenCLIP0.600.020.050.410.10 Ops-M-Embedding1.040.700.510.750.63 ColQwen70.560.730.520.840.67 Qwen3-VL-Embedding1.300.780.550.780.69 Interleaved (Text + Image) ColQwen49.590.780.560.820.70 Qwen3-VL-Embedding2.980.850.570.750.72 Table 4: Retrieval performance (NDCG@10) across different document representations and models. Index (GB) reflects storage cost. Index Size.Index size is defined as the total storage required for all embedding vectors, excluding model parameters. For text-based representations, index size is controlled by varying the chunk size used to segment documents prior to embedding. Smaller chunks increase the number of units and the total index size, while larger chunks reduce storage at the cost of coarser representations. This mechanism is used for all text indexings. Note that varying chunk size does not substantially change the index size for ColQwen, since text is encoded at the token level and stored as a set of embeddings regardless of chunk boundaries. For vision-based representations, index size is controlled via themax_pixelsparameter, which caps the total number of input pixels processed per image. Images are resized to approximately preserve aspect ratio while satisfying this budget. The number of visual tokens scales with the effective image resolution and can be approximated asT vis â H âČ W âČ P 2 , whereH âČ W âČ â€ max_pixels is the resized image resolution andPis the vision encoderâs patch size. Reducingmax_pixels therefore decreases the number of visual tokens and the resulting index size, trading visual detail for storage efficiency. We tune themax_pixelsparameter for each modelârepresentation pair and report the best-performing configuration. 4.3 Main Results Table 4 reports retrieval performance measured by nDCG@10, using the best configuration for each representationâmodel pair. Results are reported separately for text, table, and figure queries. Main Results.(1) Document-as-image representations are consistently suboptimal. Across all query types (text, table, and figure), document-as-image representations underperform compared to alternatives. Even for figure-based queries, the best-performing model is ColQwen with text + VLM captions, a purely text-based representation. Moreover, the interleaved text + image representation also outperforms document-as-image, indicating that preserving structure and modality alignment is more effective than treating the document as a flat image. (2) Text alone is surprisingly competitive for figure-based queries. Even without access to figures or VLM-generated captions, text-only models achieve strong performance on figure-based queries. For example, Qwen (text-only) is within 0.02 of Qwen-VL (doc-as-image) on figure queries. This suggests that scientific documents often describe and interpret figures in the surrounding text. 8 Preprint. Under review. (3) Interleaved representations outperform document-as-image despite no dedicated training. Even though none of the embedding models are explicitly trained for interleaved text+image inputs, this representation still outperforms document-as-image. This highlights the importance of preserving the documentâs native structure and aligning text with corresponding figures, rather than collapsing the entire document into a single image representation. (4) Single-vector representations outperform multi-vector ones despite smaller index sizes. Across both text and multimodal settings, single-vector models outperform late-interaction models while requiring substantially smaller index sizes. For text inputs, the best-performing configuration of Qwen-Embedding outperforms ColQwen by 0.05. Similarly, for multimodal/image-based inputs, Qwen-VL-Embedding outperforms ColQwen by 0.02. These results suggest that the added complexity and storage cost of multi-vector representations do not translate into improved retrieval performance in this setting. 4.4 Analysis 4.4.1 Scaling Behavior Figure 3: Retrieval performance (NDCG@10) as a function of context length (token length) across document representations. The max-pixels param- eters are tuned (see Appendix A). We study how retrieval performance changes as the amount of context increases. Starting from the flattened LaTeX source, we sample a base window of500tokens and progressively expand it to larger contexts (1000,4000, and8000to- kens), while keeping the query fixed across all variants. For each window, we convert the text into a minimally executable LaTeX snippet and render it to a PDF. Then, we extract a single page image. This enables a controlled compar- ison between text-only and document-as-image representations under matched content. For re- trieval, we use a fixed 100-token query sampled from the original 500-token window, from130 documents. Figure 3, shows that document-as- image representations degrade more sharply as context grows, in contrast to text-based repre- sentations. 4.4.2 Why Do Text Representations Work for Figure-Based Queries? To understand why text representations perform well on figure-based queries, we analyze the highest-scoring retrieved text chunk from the correct document for each query. For each case, we check whether the top chunk (i) lies near the figure, (i) explicitly references it, or (i) contains caption-level information. A chunk is considered near if it lies within a small window (±2 chunks). To detect figure references, we use regex patterns such asFig., fig, and variants of Figure X. To assess caption-level information, we extract informative words from the caption (after removing stopwords) and measure their overlap with the chunk text. MetricCorrect DocIncorrect Doc Count6361 Near Figure (%)41.323.0 References Figure (%)60.342.6 Contains Caption Info (%)68.324.6 Table 5: Analysis of the highest-scoring retrieved text chunk for figure-based queries. As a baseline, we perform the same analysis on top chunks retrieved from incorrect documents. Results are summarized in Table 5. Correct- document retrievals frequently exhibit all three signals, while baseline chunks show lower rates across these indicators. These results suggest that text retrieval does not require access to the figure itself. Instead, it relies on surrounding textual descriptionsâreferences, explanations, and caption contentâthat encode the figureâs information in text form. 9 Preprint. Under review. 4.4.3 Comparison with ViDoRe Benchmark To assess the generalizability of our findings, we evaluate text-only and document-as-image retrieval representations on the ViDoRe benchmark. Since ViDoRe provides only document images, we extract text using PaddleOCR (Cui et al., 2025) to obtain text-only representations. We then compare text-only representation, against document-as-image, across the subsets. Subset#Tokens TextDoc as Image QwenColQwenQwen VL Biomedical1080.640.600.68 Economics7040.500.530.51 ESG582 0.500.560.65 ESG HL5830.610.600.68 Table 6: nDCG@10 on ViDoRe dataset. As shown in Table 6, the results on ViDoRe dif- fer from those on ArxivDoc, with document-as- image representations generally performing bet- ter than text-only representations. We attribute this discrepancy to several differences between the two benchmarks. First, ArxivDoc consists of scientific documents that are inherently text- dense, with an average of 947 tokens per page, structured prose, and technical vocabulary. In contrast, ViDoRe comprises lecture slides, cor- porate ESG reports, and economic reports that are more visually designed, with fewer tokens per page (108â704) and greater reliance on layout, charts, and infographics to convey information. This makes ViDoRe documents better suited for visual representations, whereas the retrieval signal in scientific documents resides predominantly in the text. Second, ViDoReâs page-level evaluation introduces noise due to the combination of shorter, less specific queries (21 tokens on average, compared to 34 in ArxivDoc) and page-level granularity. Because these queries contain less domain-specific terminology, multiple pages from the same document often include relevant information. We sampled100queries where the Qwen text embedder did not retrieve the ground truth in the top5and used an LLM judge (GPT-5.2) to assess the retrieved documents by asking whether the query can be answered given each document. Across the retrieved documents, the judge identified22%of queries containing at least one missed but answerable result, indicating false negatives, an issue also highlighted by recent works (Wang et al., 2024b; Wasserman et al., 2025). In contrast, ArxivDoc operates at the document level with more specific queries that map cleanly to a single ground truth document, and in our evaluation of100queries we observe no false negatives. These findings indicate that the ViDoRe results should be interpreted with caution. These findings highlight a key difference between our setting and ViDoRe: document-as-image representations may be advantageous when queries are broad or when multiple pages contain over- lapping information, whereas text-based representations benefit from more specific queries and clear document-level grounding. Notably, this distinction is also consistent with findings from the MRMR (Zhang et al., 2025a) study, which similarly reports stronger performance from Qwen3-based text embeddings relative to document-as-image approaches on their document-level dataset. 5 Conclusion and Future Work. We present ArXivDoc, a benchmark for scientific document retrieval built from raw LaTeX sources. Using LaTeX allows direct access to structured elements such as sections, tables, figures, and equations, enabling controlled analysis of how different representations behave across text, tables, and figures. Our results show that document-as-image representations are not always well-suited for scientific documents, particularly when evidence is embedded in dense textual and structured content. An important observation is that interleaved text+image representations outperform document-as- image approaches even without explicit training for such inputs, suggesting that training embedding models on interleaved scientific documents is a promising direction. More broadly, this points to a gap between how scientific documents are structured and how current retrieval models are trained. Moreover, extending ArXivDoc to support queries that require combining evidence across multiple parts of a documentâor across documentsâcan further advance retrieval in scientific settings. Ethics Statement We do not believe there are significant ethical issues associated with this research. 10 Preprint. Under review. References Yauhen Babakhin, Radek Osmulski, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer, Bo Liu, and Even Oldridge. Llama-embed-nemotron-8b: A universal text embedding model for multilingual and cross-lingual tasks, 2025. URLhttps://arxiv.org/abs/2511.07025. Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei, Wenyi Hong, Ruiliang Lyu, Weihan Wang, Zhe Su, Xiaotao Gu, Xiao Liu, et al. Glyph: Scaling context windows via visual-text compression. arXiv preprint arXiv:2510.17800, 2025. Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 2818â2829, 2023. Jaemin Cho, Debanjan Mahata, Ozan Ì Irsoy, Yujie He, and Mohit Bansal. M3docrag: Multi-modal retrieval is what you need for multi-page multi-document understanding. 2024. Yejin Choi, Jaewoo Park, Janghan Yoon, Saejin Kim, Jaehyun Jeon, and Youngjae Yu. Zero-shot multimodal document retrieval via cross-modal question generation. In Christos Christodoulopou- los, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, p. 26079â26094, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332- 6. doi: 10.18653/v1/2025.emnlp-main.1324. URLhttps://aclanthology.org/2025. emnlp-main.1324/. Cheng Cui, Ting Sun, Suyin Liang, Tingquan Gao, Zelun Zhang, Jiaxuan Liu, Xueqing Wang, Changda Zhou, Hongen Liu, Manhui Lin, et al. Paddleocr-vl: Boosting multilingual document parsing via a 0.9 b ultra-compact vision-language model. arXiv preprint arXiv:2510.14528, 2025. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 4599â4610, 2021. Kuicai Dong, Yujing Chang, Derrick Goh Xin Deik, Dexun Li, Ruiming Tang, and Yong Liu. MMDocIR: Benchmarking multimodal retrieval for long documents. In Christos Christodoulopou- los, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, p. 30959â30993, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332- 6. doi: 10.18653/v1/2025.emnlp-main.1576. URLhttps://aclanthology.org/2025. emnlp-main.1576/. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, 2025. Michael GĂŒnther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, and Han Xiao. jina- embeddings-v4: Universal embeddings for multimodal multilingual retrieval. In David Ifeoluwa Adelani, Catherine Arnett, Duygu Ataman, Tyler A. Chang, Hila Gonen, Rahul Raja, Fabian Schmidt, David Stap, and Jiayi Wang (eds.), Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), p. 531â550, Suzhuo, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-345-6. doi: 10.18653/v1/2025.mrl-main.36. URL https://aclanthology.org/2025.mrl-main.36/. Hao Guo, Xugong Qin, Jun Jie Ou Yang, Peng Zhang, Gangyan Zeng, Yubo Li, and Hailun Lin. Towards natural language-based document image retrieval: new dataset and benchmark. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 29722â29732, 2025. Siwei Han, Peng Xia, Ruiyi Zhang, Tong Sun, Yun Li, Hongtu Zhu, and Huaxiu Yao. Mdocagent: A multi-modal multi-agent framework for document understanding. arXiv preprint arXiv:2503.13964, 2025. 11 Preprint. Under review. Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580, 2024. Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, p. 39â48, 2020. Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal ArXiv: A dataset for improving scientific comprehension of large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14369â14387, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.775. URL https://aclanthology.org/2024.acl-long.775/. Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking, 2026. URL https://arxiv.org/abs/2601.04720. Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms. In The Thirteenth International Conference on Learning Representations. Zhiheng Lyu, Xueguang Ma, and Wenhu Chen. Pixelworld: Towards perceiving everything as pixels. Transactions on Machine Learning Research, 2025. Xing Han LĂč. Bm25s: Orders of magnitude faster lexical search via eager sparse scoring, 2024. URL https://arxiv.org/abs/2407.03618. Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multi- modal retrieval via document screenshot embedding. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natu- ral Language Processing, p. 6492â6505, Miami, Florida, USA, November 2024a. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.373. URLhttps: //aclanthology.org/2024.emnlp-main.373/. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. Advances in Neural Information Processing Systems, 37:95963â96010, 2024b. Quentin MacĂ©, AntĂłnio Loison, and Manuel Faysse. Vidore benchmark v2: Raising the bar for visual retrieval. arXiv preprint arXiv:2505.17166, 2025. Xiangyu Peng, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, and Chien-Sheng Wu. Unidoc-bench: A unified benchmark for document-centric multimodal rag, 2026. URLhttps://arxiv.org/ abs/2510.03663. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748â8763. PmLR, 2021. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 3715â3734, Seattle, United States, July 2022. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.272. URLhttps: //aclanthology.org/2022.naacl-main.272/. R. Smith. An overview of the tesseract ocr engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 2, p. 629â633, 2007. doi: 10.1109/ICDAR. 2007.4376991. 12 Preprint. Under review. Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: complex question answering over text, tables and images. In International Conference on Learning Representations. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language modelâs perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024a. Shiqi Wang, Yeqin Zhang, and Cam-Tu Nguyen. Mitigating the impact of false negative in dense retrieval with contrastive confidence regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, p. 19171â19179, 2024b. Navve Wasserman, Roi Pony, Oshri Naparstek, Adi Raz Goldfarb, Eli Schwartz, Udi Barzelay, and Leonid Karlinsky. Real-m-rag: A real-world multi-modal retrieval benchmark. arXiv preprint arXiv:2502.12342, 2025. Haoran Wei, Yaofeng Sun, and Yukun Li. Deepseek-ocr: Contexts optical compression, 2025. URL https://arxiv.org/abs/2510.18234. Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594, 2024. Siyue Zhang, Yuan Gao, Xiao Zhou, Yilun Zhao, Tingyu Song, Arman Cohan, Anh Tuan Luu, and Chen Zhao. Mrmr: A realistic and expert-level multidisciplinary benchmark for reasoning-intensive multimodal retrieval, 2025a. URL https://arxiv.org/abs/2510.09510. Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Bridging modalities: Improving universal multimodal retrieval by multimodal large language models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 9274â9285, 2025b. doi: 10.1109/CVPR52734.2025.00866. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025c. A Hyperparameters Table 7 reports retrieval accuracy (nDCG@10) under varying storage budgets, grouped by document representation. Storage is controlled through representation-specific hyperparameters, including text chunk size, visual token budgets (max_pixels), and their combinations for interleaved representa- tions. 13 Preprint. Under review. RepresentationModelIndex Size (GB)TextTableFigure Text Qwen chunk=5122.560.870.600.76 chunk=10241.200.850.540.76 chunk=40960.340.830.470.64 ColQwen chunk=102440.540.770.560.80 chunk=409640.540.770.570.80 L A T E X Figures ColQwen max_pixels = 600K22.090.170.160.86 max_pixels = 1M35.430.180.180.87 max_pixels = 2M65.990.170.160.86 Document-as-Image ColQwen max_pixels = 1M47.230.710.510.83 max_pixels = 2M94.790.730.520.84 Table 7: Retrieval accuracy (NDCG@10) under varying storage budgets, grouped by document representation. 14 Preprint. Under review. B Query Generation The query generation pipeline consists of four stages, three of which involve prompting gpt-5-mini. In this section, we report the system prompts used at each stage. B.1 Synthetic Query Generation Prompt (Text) PROMPT You are given extracted text from a scientific research paper. Your task is to generate a single, high-quality synthetic query that would meaningfully test a document retrieval system. Instructions: 1. The query must require expert-level reasoning over implications, trends, limitations, or constraints discussed in the document, and must not be a direct restatement of any sentence from the input. 2. The query must minimize lexical overlap with the input text by avoiding distinctive phrases or terminology, relying instead on abstraction and paraphrasing rather than keyword matching. 3. The query must be answerable from the document but not trivially retrievable via keyword or phrase matching, and must not reference sections, figures, experiments, or document-specific wording. 4. The query must ask exactly one focused question, without combining multiple sub-questions or enumerating parameters or conditions. 5. The query must be realistic and concise, phrased as a single sentence that a knowledgeable researcher would plausibly ask, without verbose framing or artificial difficulty. 6. If no query satisfying these criteria can be generated, return null. Required Output Format: "query": "<generated question or null>" Here is the document content: paper_text Figure 4: Prompt used for generating synthetic, open-domain retrieval queries from scientific text. 15 Preprint. Under review. B.2 Query Decontextualization Prompt PROMPT You are a scientific question rewriter. You are given an original question that references a specific portion of a research paper. Your task is to rewrite it into a context-independent, open-domain scientific query that targets the same underlying concept, without relying on document-local or visual references. Do not refer to any figure, plot, panel, image, document, or use deictic expressions such as this, that, above, or below. Requirements: 1. Preserve the core scientific intent, variables, and conditions present in the original question. 2. Replace visual or deictic phrasing with concept-level wording (e.g., remove references such as âbased on the graphâ and ask directly about the relationship or effect). 3. If symbols (e.g., f spec ) appear without definition, retain them exactly as written and do not invent meanings. A minimal parenthetical alias may be included only if it appears in the input. 4. Remove all references to figures, plots, tables, panels, or document-local indices. 5. Ensure the rewritten query can be answered by a knowledgeable reader without access to the original document or image. 6. Retain units, ranges, and experimental or observational conditions if present. 7. Avoid unresolved pronouns or placeholders (e.g., âthe parameterâ, âthe systemâ) unless the domain makes them unambiguous. 8. If the original question contains multiple sub-questions, keep only one and discard the rest. 9. The final query must be a single, concise sentence with no superfluous framing or background. Required Output Format: "query": "<single rewritten question or null>", "reasoning": "<one-sentence rationale>" If a valid context-independent query cannot be produced, set "query" to null and briefly explain why in "reasoning". Figure 5: Prompt used for decontextualizing document-dependent scientific questions into open- domain queries. 16 Preprint. Under review. B.3 Query Verification Prompt PROMPT You are a validator that checks whether a decontextualized question is well-formed for open-domain retrieval. Judge only from the provided JSON fields. Do not assume access to the original figure, table, or paper. What âvalid decontextualized questionâ means: A question is valid if and only if all of the following criteria are satisfied: 1. Context-independent: The question contains no references to local context such as âthis figure,â âthe table above,â âthese results,â or any indexical phrasing that requires the original document or image. 2. Answerable in principle: A knowledgeable person or external source could answer the question without access to the original paper or figure. The domain and variables must be sufficiently specified. Crucially, the question must not rely on parameters, symbols, or notations that are defined arbitrarily or only within the source paper (e.g., a tuning parameter with no standard meaning in the field). 3. Intent preserved: The question targets the same underlying information need as the original question, but generalized beyond the local figure or document context. 4. Clarity and unambiguous entities: Any entities, variables, or notations must be interpretable by an expert in the relevant field without requiring the specific paper. Unresolved pronouns or placeholders (e.g., âthe parameter,â âthe systemâ) are not allowed unless they are standard and unambiguous within the domain. Guiding Principle for Ambiguity: Requiring background domain knowledge is acceptable and expected for real search queries. However, ambiguity arising from terms that are defined only within the source document or that depend on the original figure context is not acceptable. Common failure modes (label them if present): âą underspecified_parameter (especially if defined arbitrarily in the source paper) âą still_context_bound âą domain_missing_or_vague âą ambiguity_pronouns_placeholders âą unanswerable_generic Required Output Format (JSON only): "is_valid": boolean, "score": integer, "decision_rationale": string, "confidence": integer Figure 6: Prompt used to verify whether a generated question is a valid, context-independent query suitable for open-domain retrieval. 17 Preprint. Under review. C Example Queries arXiv ID: 1007.4239 How can an optical system rapidly and cheaply switch between four orbital angular momentum (helical-phase) mode indices while keeping polarization independent? arXiv ID: 1011.0302 In the dense phase of the O(n) loop model, how does changing the topology of the graph representation determine if a local perturbation is RG-relevant or RG-irrelevant according to Coulomb gas scaling dimensions? arXiv ID: 1806.03783 How does increasing the normalized accretion rate lead to a turnover in the fraction of IR luminosity reprocessed by circumnuclear dust? arXiv ID: 1205.2806 Which ion-neutral atom pair best enables quantum-threshold scattering while being least sensitive to stray static electric fields, i.e., with a high threshold energy but low static-field strength needed for excess micromotion to reach that energy? arXiv ID: 1304.2695 How do the implicit midpoint and trapezoidal (Crank-Nicolson) rules differ in using a single midpoint evaluation versus averaging endpoint evaluations? arXiv ID: 1411.3004 Which subtype of pulsating star shows the strongest positive correlation between variability amplitude and pulsation period across stars spanning a wide range of distances? arXiv ID: 2006.00262 What is the sequence of processing steps that transforms source and target monolingual corpora into mapped cross-lingual word embeddings (CLWEs)? Figure 7: Representative examples of decontextualized, evidence-grounded queries in TeXODQ. Each query targets a specific piece of scientific evidence. 18 Preprint. Under review. C.1 Human Annotation Protocol ANNOTATION GUIDELINES You are evaluating scientific retrieval queries generated from research documents. Queries may be questions or short search-style phrases. Each query must be assessed and, if necessary, revised based on three criteria: naturalness, ambiguity, and document answerability. Criteria: 1. Naturalness: The query must be clear, understandable, and plausible as a realistic search query. A query is considered unnatural if it (i) combines multiple questions (e.g., âWhat is X, and why Y?â), (i) contains excessive domain-specific terminology (approximately 8 or more specialized terms), (i) is overly verbose (e.g., more than three clauses), or (iv) is difficult to interpret. Such queries should be rewritten to improve clarity while preserving the original scientific intent. 2. Ambiguity: The query must be specific enough to identify a single target document. A query is considered ambiguous if it is overly broad, underspecified, admits multiple interpretations, or could be answered by many documents. Ambiguous queries should be rewritten to resolve underspecification and better target the intended document, and if thatâs not a possibility, they should be removed. 3. Document Answerability: The query must be directly supported by evidence in the document. The annotator must verify that a specific passage, parsed table, or rendered figure from the L A T E X source contains the information needed to answer the query. Queries whose premise is unsupported or contradicted by the document, or for which no identifiable evidence exists, must be removed. Workflow: 1. Perform initial evidence screening using retrieval tools (e.g., NotebookLM or Gemini Flash 2.5) to surface candidate passages, tables, or figures and identify potential issues with naturalness, ambiguity, or answerability. 2. Assign a coarse quality score (1-10) to guide assessment of ambiguity and retrieval specificity; this score is used for calibration and not thresholded directly. 3. Manually inspect the retrieved evidence against the original L A T E X source, including text, parsed tables, and rendered figures, to confirm correctness. 4. Rewrite queries to improve clarity and specificity while preserving intent, or discard queries that cannot be made valid. Figure 8: Human annotation protocol for evaluating and refining scientific retrieval queries. 19