Paper deep dive
MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects
Jason Luo, Saibilila Abudukelimu, Judy Song, Andrew Feng, Shivank Garg, Vasu Sharma, Kevin Zhu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/15/2026, 8:46:33 AM
Summary
The paper introduces MUDDLE, a benchmark designed to measure document understanding under distractor and length effects. It separates the impact of context length from topical similarity by comparing 'hard negatives' (topically similar but irrelevant documents) against 'random distractors' (length-matched but topically unrelated documents). Using 270 human-annotated questions over 85 source documents, the study evaluates models like gpt-5-mini, gemini-3.5-flash, and grok-4.3 across markdown, image, and PDF modalities. Results indicate that hard negatives degrade accuracy more than random distractors for gpt-5-mini, suggesting that topical adjacency is a more significant source of error than length alone.
Entities (10)
Relation Signals (7)
Jason Luo → authored → MUDDLE
confidence 95% · Jason Luo ∗ ... Abstract: ... We present MUDDLE
MUDDLE → evaluates → GPT-5-mini
confidence 95% · In the complete markdown sweep, hard negatives lower accuracy more than length-matched random documents at both context sizes for gpt-5-mini
MUDDLE → evaluates → Gemini-3.5-Flash
confidence 95% · We score answers with an LLM judge across three model families... gemini-3.5-flash
MUDDLE → evaluates → Grok-4.3
confidence 95% · grok-4.3 runs on Azure AI Foundry.
MUDDLE → uses → MMLongBench-Doc
confidence 92% · We built MUDDLE on top of MMLongBench-Doc.
hard negatives → degradesaccuracymorethan → random distractors
confidence 90% · hard negatives lower accuracy more than length-matched random documents at both context sizes for gpt-5-mini
MUDDLE → publishedin → COLM 2026
confidence 90% · Published as a conference paper at COLM 2026
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Document question-answering systems increasingly answer questions over collections of retrieved documents rather than one clean source, so robustness to distracting context matters as much as reading ability. When such systems fail, it is often unclear whether the context was too long or the distractors were too close to the topic, because prior work tends to conflate these two effects. We present MUDDLE, a controlled benchmark that separates them. MUDDLE uses 270 human-annotated questions, each tied to a single source document, and instantiates every question in five conditions: the source alone, the source with two or four topically similar hard negatives, and the source with two or four random distractors. The random distractors are matched to the hard negatives in length and provenance, so an accuracy gap between the two arms reflects topical similarity rather than length. All five conditions are rendered in markdown, page images, and raw PDF, but the distractor sweep reported here is run in markdown, since a source plus its distractors exceeds current image and PDF input limits. We score answers with an LLM judge across three model families. In the complete markdown sweep, hard negatives lower accuracy more than length-matched random documents at both context sizes for gpt-5-mini, while random documents stay near the no-distractor baseline. The effect is small but directionally consistent, and for gpt-5-mini hard negatives significantly underperform length-matched random distractors when pooled across context sizes. We release the data and evaluation code for a reproducible study of context degradation.
Tags
Links
- Source: https://arxiv.org/abs/2608.29477v1
- Canonical: https://arxiv.org/abs/2608.29477v1
Trouble viewing inline? Open PDF directly →
Full Text
42,702 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at COLM 2026 MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects Jason Luo ∗ , Saibilila Abudukelimu ∗ , Judy Song, Andrew Feng, Shivank Garg, Vasu Sharma, Kevin Zhu Abstract Document question-answering systems increasingly answer questions over collections of retrieved documents rather than one clean source, so robust- ness to distracting context matters as much as reading ability. When such systems fail, it is often unclear whether the context was too long or the distractors were too close to the topic, because prior work tends to conflate these two effects. We present MUDDLE, a controlled benchmark that separates them. MUDDLE uses 270 human-annotated questions, each tied to a single source document, and instantiates every question in five conditions: the source alone, the source with two or four topically similar hard negatives, and the source with two or four random distractors. The random distractors are matched to the hard negatives in length and prove- nance, so an accuracy gap between the two arms reflects topical similarity rather than length. All five conditions are rendered in markdown, page images, and raw PDF, but the distractor sweep reported here is run in markdown, since a source plus its distractors exceeds current image and PDF input limits. We score answers with an LLM judge across three model families. In the complete markdown sweep, hard negatives lower accuracy more than length-matched random documents at both context sizes for gpt-5-mini, while random documents stay near the no-distractor baseline. The effect is small but directionally consistent, and for gpt-5-mini hard negatives significantly underperform length-matched random distractors when pooled across context sizes. We release the data and evaluation code for a reproducible study of context degradation. 1 Introduction Large vision-language models are increasingly used not to read a single, clean document but to work over whole collections of documents pulled on demand. An analyst’s question is answered by retrieving reports, papers, and filings; an agent grounds a claim by reading whatever a search turns up. Two trends drive this. First, context windows have grown by orders of magnitude, yet models use that capacity unevenly and degrade as inputs lengthen (Hsieh et al., 2024). This tempts systems to drop an entire retrieved bundle into one prompt rather than curating it first. Second, retrieval-augmented generation is now the standard way to ground outputs in external knowledge (Lewis et al., 2020). The retrieved set is rarely clean: corpora are messy and retrievers are imperfect, so the one document that answers the question often arrives buried among plausible look-alikes that do not. Whether a model can still find that document and reason over it while ignoring the rest is a core property of any document-grounded system, not an edge case. Two lines of work speak to this, but neither isolates the failure we care about. One studies how models use long contexts and finds that they use them unevenly: accuracy depends on where the evidence sits and declines as the context grows (Liu et al., 2024). The other studies retrieval noise and shows that irrelevant or misleading passages pull answers off course, though carefully placed random documents can sometimes help (Shi et al., 2023; Cuconasu et al., 2024). Both rely on short, text-only, open-domain passages, where a distraction is a sentence or a paragraph rather than a full document with its own layout. Multimodal long-document benchmarks close part of the gap. MMLongBench-Doc (Ma et al., 2024), 1 arXiv:2608.29477v1 [cs.CL] 30 Aug 2026 Published as a conference paper at COLM 2026 for instance, shows that reading even one lengthy PDF is hard, with the best-performing model they test (GPT-4o) reaching an F1 of only 44.9%. But these benchmarks judge the source document on its own. They never place it among the related-but-wrong documents a retriever would surface, so a practitioner cannot tell whether a failure comes from the length of the context or from distractors that look like the answer. That distinction decides whether to curate the retrieved set or simply truncate it, and it is exactly what existing benchmarks leave unmeasured. Our starting point is that the failure mode worth isolating is not context length on its own, but semantic adjacency: documents that share a topic and vocabulary with the source, and so look as though they answer the question, yet do not actually contain the answer. We call these documents hard negatives, borrowing the term from dense retrieval, where hard negatives are the topically close but non-relevant passages a retriever is most likely to return (Karpukhin et al., 2020). We expect hard negatives to be more corrosive than the same amount of random text. The form a document takes when it reaches the model, whether a native PDF, text extracted to markdown, or rendered page images, also exposes its layout differently, so we treat input modality as an explicit axis. These considerations lead to four research questions that organize the paper. (RQ1) Topical similarity: at a matched context length, do hard negatives degrade answer accuracy more than random distractors? (RQ2) Length: does adding more distractor documents (k=0→2→4) degrade accuracy, independent of distractor type? (RQ3) Modality, a baseline check rather than a hypothesis test: on the source-only condition, does answer accuracy differ when the same document is presented as markdown, page images, or a raw PDF? (RQ4) Model: do these effects hold across models from different families? We answer RQ1, RQ2, and RQ4 in markdown, where the full distractor sweep is servable, and treat the distractor sweep in image and PDF as an extension that our released code supports (Section 6). Our distractor findings are therefore text-based, and we do not claim the same margin holds for page images or PDF. To answer these questions we built MUDDLE on top of MMLongBench-Doc. We start from 270 human-annotated questions over 85 source documents. For each question we assemble five context conditions that hold the source fixed and change only what surrounds it: a control with the source alone; HN-2 and HN-4, which add two and four hard negatives; and Random-2 and Random-4, which add two and four random distractors. The hard negatives are curated to be on topic yet free of the answer, and the random distractors are drawn so that the two arms differ only in topical relevance (Section 3.5). Every condition is rendered in markdown, page images, and raw PDF and scored with an LLM judge for semantic correctness. The design separates three factors that earlier work tends to mix together: whether extra context is present, how many distractors there are, and what kind of distractor it is. Our contributions are: • A controlled, multi-document evaluation suite that varies distractor type and dis- tractor count over long, real-world documents, holding the source document fixed and matching context length between distractor types. The same conditions are rendered in three input modalities (markdown, page images, and raw PDF); the distractor sweep is evaluated in markdown, and the image and PDF renderings support a source-only modality comparison. 1 •A hard-negative curation pipeline. We retrieve real documents by search, filter them with a model that keeps topical look-alikes while rejecting any that may contain the answer, and verify the result by hand. Each hard negative is matched in length to the random distractor pool, and each question is checked to be answerable only from its source document. •An evaluation protocol and an empirical finding. Under paired, same-question comparison on the complete five-condition markdown sweep, topical adjacency degrades accuracy beyond length alone for gpt-5-mini, significant when pooled across context sizes, while the effect stays small and directional across the three 1 Data and code: https://github.com/luoojason/muddle. 2 Published as a conference paper at COLM 2026 families. Our released code extends the protocol to image and PDF distractor conditions once they can be served at scale. 2 Related Work 2.1 Long context and modality Two properties of modern models motivate our setup. First, models use long contexts unevenly: accuracy is highest when the relevant evidence sits near the start or end of the input and lowest in the middle, and it declines as the context grows (Liu et al., 2024), with stress tests confirming that nominal context windows overstate usable capacity and most models degrade before their advertised limit (Hsieh et al., 2024; Bai et al., 2024). We hold source placement fixed and seeded (Section 3) so these positional effects are controlled rather than measured, isolating distractor type at matched length. Second, the documents a model reads are increasingly multimodal: document question answering is now posed over rendered pages rather than clean text, beginning with single-page settings such as DocVQA (Mathew et al., 2021), and the form a document takes (extracted markdown, page images, or native PDF) exposes layout differently, so we treat modality as an explicit axis rather than a fixed preprocessing choice. 2.2 Hard negatives and retrieval noise Retrieval-augmented generation grounds model outputs in retrieved documents (Lewis et al., 2020), but the retrieved set is rarely clean. Irrelevant or misleading context lowers answer quality: large language models are easily distracted by irrelevant material (Shi et al., 2023), and how much retrieved noise hurts depends on its relationship to the query (Yoran et al., 2024), to the point that some off-topic documents can even help (Cuconasu et al., 2024). This line of work studies short, text-only, open-domain passages; we scale the distractor to a full multimodal document and separate topical from random noise explicitly. In dense retrieval, hard negatives are mined to train the retriever, first from a lexical retriever (Karpukhin et al., 2020) and later globally from the corpus (Xiong et al., 2021). We instead import them as an evaluation condition, testing whether a model can still find and use the one relevant document when it is surrounded by plausible look-alikes. 2.3 Recent benchmarks A growing set of benchmarks evaluates document understanding at length and across modalities, and our design responds to a gap they share. MMLongBench-Doc poses human- annotated questions over long PDFs and shows single-document reading is hard even for strong models, though its renderings can be lossy (Ma et al., 2024). DUDE covers multi-page, visually rich documents and likewise finds a large multi-page gap (Van Landeghem et al., 2023). LongDocURL adds reasoning and locating tasks over long multimodal documents and evaluates parsed-text and page-image inputs as separate modes, but each question is answered over a single document rather than among distractors (Deng et al., 2025). UniDoc-Bench assembles a multimodal retrieval benchmark of 1,600 question-answer pairs comparing text, image, and fused retrieval, but it varies modality and retrieval strategy rather than isolating context length or topical-distractor type (Peng et al., 2025). IRPapers, for scientific retrieval, evaluates image- and text-based retrieval as separate single-modality conditions (Shorten et al., 2026). Across these, none places the target document among the related-but-wrong documents a retriever would surface while holding modality and context length apart. MUDDLE is built for exactly that gap. 3 Benchmark Construction Figure 1 summarizes the construction of MUDDLE, from source questions through distrac- tor curation to multimodal rendering and scoring. 3 Published as a conference paper at COLM 2026 MMLongBench-Doc Source PDFs & QA pairs Distractor Curation Hard negatives Retrieved from Semantic Scholar Claude subagent processing Manual verification of hard negatives Random distractors Broad document pool Randomly sampled PDFs Question Review Check answerability & reduce ambiguity PDF Final Dataset Control: source only HN-2: source + 2 hard negatives HN-4: source + 4 hard negatives Random-2: source + 2 random Random-4: source + 4 random Verified QA pairings Figure 1: Construction of MUDDLE. Human-annotated questions over long PDFs from MMLongBench-Doc (Ma et al., 2024) are paired with curated hard negatives (retrieved by search, filtered by model-assisted review, verified by hand, and held to a 10 to 40 page band) and with random distractors drawn from the hard negatives of other questions (Section 3.5). Each (question, context) cell is rendered in three modalities (markdown, page images, and native PDF) and scored with an LLM judge. 3.1 Source questions MUDDLE is built on MMLongBench-Doc (Ma et al., 2024), a benchmark for question answering over long, multimodal PDFs. We keep questions answerable from text and tables, not from graphs (e.g. a pie chart), since the markdown rendering drops figures that the page images preserve. The resulting set has 270 questions over 85 source documents (1 to 17 per document), with every question and reference answer tracing to the original human annotations rather than model generation. We labeled each question by task type, giving a roughly even split between recall questions (retrieve a stated fact) and reasoning questions (combine information across the document). 3.2 Context conditions For each question we build five context conditions that hold the source document fixed and vary only what surrounds it: a control with the source alone; HN-2 and HN-4, adding two and four hard negatives (documents topically similar to the source); and Random-2 and Random-4, adding two and four random distractors drawn without regard to topic. Crossing 270 questions with five conditions gives 1,350 (question, context) cells per modality. 3.3 Hard-negative curation We assign each source document four hard negatives, mined specifically for that source. Hard negatives are sourced by search over signals from the source document, the question, the answer, and the document topic. A model-assisted review then keeps documents similar to the source while rejecting those likely to contain the answer, and a human checks the result. Every shipped hard negative passes two automatic gates: a token-overlap (Jaccard) below 0.45 against the source, so distractors are topical but not near-duplicates, and a word-boundary answer-leak check. To keep context length from confounding the topical- similarity comparison, every hard negative is constrained to a band of 10 to 40 pages. If a source had more than four mined hard negatives, two were chosen at random for HN-2, then two more were drawn from the rest and added to them for HN-4. Appendix A.1 reports the five conditions in full (Table 2) and details the manual-review axes, length statistics, and provenance. 3.4 Hard-negative retrieval and filtering Candidate retrieval.For each source document we retrieve a pool of distractor candidates by topical search. Academic source documents are matched against the Semantic Scholar Graph and paper-recommendation APIs using a title extracted from the source; we keep 4 Published as a conference paper at COLM 2026 each candidate’s title, abstract, year, and identifiers and take the top eight by retrieval rank. Non-academic sources, such as product manuals, financial reports, and court filings, are matched instead by keyword web search restricted to PDF files, with keywords drawn from a TF-IDF analysis of the source text and its questions; we keep candidates whose TF-IDF cosine similarity to the source falls in a moderate band (about 0.30 to 0.85), so a candidate is related to the source without being a near-duplicate. Candidates already in the benchmark, missing an abstract, or duplicating one another are discarded. Model-assisted filtering.Answer leakage is checked with Claude Haiku 4.5 in two steps: the model answers the question from only the candidate’s title and abstract, replying that it cannot when the text is insufficient; if it instead answers, a second call judges whether that answer matches the reference, and any candidate yielding a match is removed as contaminated. Surviving candidates are then rated by a Claude reviewer on three axes, whether the candidate shares the subject of the source, whether it leaks the answer, and whether its document type fits, and a fixed rule maps these ratings to a good, weak, or bad verdict. Candidates rated weak or bad are replaced from a pool of spare candidates, and the survivors pass to the manual review in Appendix A.1. 3.5 Random distractors Random distractors are the hard negatives belonging to other questions. For a given question we draw from the 339-document pool, excluding that question’s own hard negatives and source, and sample four documents with a per-question seed. Random-2 is the first two of that draw, so it nests inside Random-4. Because each random distractor is itself a hard negative of some other question, the random arm shares the provenance and the 10 to 40 page length distribution of the hard-negative arm. The two arms differ only in topical relatedness to the current source. At matchedktheir page counts are close (median 84 against 86 pages atk=2), so an accuracy gap between hard negatives and random distractors at the samekis attributable mainly to topical similarity rather than length. A random distractor can, by chance, be topically related to the source; we treat such accidental hard negatives as a known source of noise. 3.6 Source placement For conditions with at least one distractor, the source is inserted at a seeded, reproducible position keyed on the cell identifier, identical across all models and modalities. This holds positional effects (Liu et al., 2024) constant across our comparisons. 3.7 Modality renderings The same documents are presented in three forms. Markdown: each PDF is converted to markdown with MarkItDown (Microsoft, 2024a) and documents are concatenated under per-document headers. Image: each PDF page is rendered to a high-resolution PNG by a headless-browser pipeline (Playwright (Microsoft, 2024b) driving pdf.js (Mozilla, 2024) at viewport scale 2.0, one image per page). Because full-resolution PNGs would exceed provider request-size limits, the page images are downgraded at send time, re-encoded as downscaled JPEGs (Appendix A.1 reports the resolutions). Each cell’s rendering is held identical across all models, and we treat the resulting resolution variation as a known confound (Section 6). For the OpenAI model we set the APIdetailparameter tohigh, which the Gemini and Grok APIs do not expose. PDF: the raw PDF files are passed directly to the model as file blocks, except forgrok-4.3, which has no native PDF input and instead receives OpenRouter text extraction or Mistral OCR, so its PDF column is not comparable to the other two. Appendix A.2 reports per-cell size statistics for each modality and condition. 5 Published as a conference paper at COLM 2026 4 Experimental Setup 4.1 Models We evaluate three models from different families:gpt-5-mini,gemini-3.5-flash, and grok-4.3.gpt-5-miniandgemini-3.5-flashrun at a fixed medium reasoning effort. Providers differ in how they expose this control, so it is passed where supported and omitted otherwise, andgrok-4.3runs without an explicit effort parameter. Requests are issued through a single client library (litellm) with a uniform message format. Routing is by provider and request size:gpt-5-miniruns on Azure OpenAI for smaller cells and on Open- Router for cells above the per-request image limit,gemini-3.5-flashruns on OpenRouter, and grok-4.3 runs on Azure AI Foundry. 4.2 Prompt and inference Each query is a single user message containing a short instruction, the question, and the documents in their placed order. The instruction is held fixed up to the phrase naming the document format. For markdown it reads: “You are given one or more documents (converted to markdown). Using ONLY their content, answer the question with a short, direct answer and nothing else. If the documents do not contain the answer, reply ‘Not answerable’.” The image and PDF instructions are identical except for “documents as page images” and “PDF documents”. In the markdown and image conditions, documents are separated by per-document headers; in the PDF condition each document is a separate file block identified by filename. The full prompts are reproduced in Appendix A.3. 4.3 Metrics Our primary metric is an LLM judge (Zheng et al., 2023; Liu et al., 2023) that scores semantic correctness. The judge (gpt-5.4-nano, reasoning effort minimal) receives the question, the reference answer, and the model prediction, and returns a JSON verdict with a boolean correct, a gradedscorein[0, 1]for partial credit, and a one-sentence rationale. The judge prompt specifies equivalence rules for numbers, units, and list order, and marks refusals incorrect; we use the boolean correct as the headline accuracy. The judge implementation is released with our evaluation code. As secondary, fully reproducible metrics we report exact match and token-level F1 over SQuAD-style normalization (lowercasing, article and punctuation removal, and whitespace collapse) (Rajpurkar et al., 2016). We report the judge as primary because answers are short and free-form, so string-overlap metrics understate correctness, especially for models that answer verbosely. 4.4 Reporting protocol All distractor-effect comparisons are computed paired: for each comparison we restrict to the questions answered in both arms and average the per-question difference, rather than comparing raw per-condition averages over mismatched samples. The effects we study are a few points in size, so pairing is necessary to read them at all. 5 Results and Analysis We organize the analysis around the four research questions, reading effects from paired, same-question comparisons (Section 4). All three models are run on all five markdown conditions, plus control runs in image and PDF. We anchor distractor-effect claims on gpt-5-minimarkdown, which has the widest spread across conditions, so a few-point effect is detectable across the 270 questions; we read the cross-model and cross-modality picture as directional given the small effect sizes (Section 6). Effect of Topical Similarity.At matched context length, hard negatives degrade accuracy more than random distractors. Forgpt-5-miniin markdown, hard negatives score below 6 Published as a conference paper at COLM 2026 ModelConditionMarkdownImagePDF gpt-5-mini Control0.7630.8040.804 HN-20.737– Random-20.767– HN-40.707– Random-40.748– gemini-3.5-flash Control0.7330.7890.778 HN-20.711– Random-20.722– HN-40.726– Random-40.711– grok-4.3 Control0.7440.8140.667 HN-20.733– Random-20.726– HN-40.682– Random-40.707– Table 1: Judge accuracy by model, modality, and condition (primary metric, higher is better). Markdown reports all five conditions for all three models. Image and PDF report the control condition only; their distractor conditions are infeasible at scale (Section 6) and are marked –. Per-model F1 and EM are in Appendix A.2. the length-matched random arm by 0.030 atk=2 and by 0.041 atk=4 (both paired), where each gap averages the per-question difference over the questions answered in both arms; the per-condition means are in Table 1. Because hard-negative and random cells are length- matched at eachk(Section 3), this gap is attributable mainly to topical similarity rather than length. The effect is small, a few points, and grows modestly fromk=2 tok=4; pooled across the two context sizes the topical gap is statistically significant (paired p=0.016). Effect of Length. Adding more distractors lowers accuracy within each type. The pure length effect is isolated by the random arm, whose documents add length without deliberate topical overlap (Section 3.5), and it is weaker than the topical effect. Forgpt-5-minimark- down, paired accuracy falls fromk=2 tok=4 by 0.019 in the random arm against 0.030 in the hard-negative arm, and the random conditions stay close to control throughout (Table 1). Unrelated documents thus cost little as context grows, while added context bites mainly when the documents are topically adjacent. That contrast is the central reason to separate distractor type from count. Effect of Modality. As stated in RQ3, modality here is a control baseline rather than a distractor test: the image and PDF distractor conditions cannot be served (Section 6), so the comparison is over the source-only condition. On control, the same questions are read about equally well across modalities, if anything slightly better from images and native PDF than from markdown. Forgpt-5-minithe control scores are 0.763 (markdown), 0.804 (PDF), and 0.804 (image), andgemini-3.5-flashshows the same ordering (Table 1). Whether topical degradation transfers across modalities is left to an extension that our code supports. Effect of Model Family. The picture varies by model family.gpt-5-minihas the clearest ordering, with hard negatives below random at bothk, and its drop from control to HN-4 is significant (pairedp=0.008), the strongest single effect in our data.gemini-3.5-flashis markedly flatter and barely degrades, so it appears robust to topical distractors atk ≤4, consistent with its long-context strength.grok-4.3shows no gap atk=2 and degrades only under the heaviest load (control to HN-4, pairedp=0.026). Where the load is heaviest, the hard-negative effect is significant for the two models that degrade at all (gpt-5-mini andgrok-4.3); the finer hard-negative-versus-random contrast is clearest forgpt-5-mini and directional across the panel. Forgrok-4.3, judge accuracy is the only trustworthy metric, since its F1 and EM are deflated by verbose answers rather than by lower correctness 7 Published as a conference paper at COLM 2026 (Section 6). Establishing whether the topical effect generalizes is the main open question (Section 6). 6 Limitations Our coverage is necessarily uneven. We report the full five-condition sweep only for markdown; for image and PDF we report the control (source-only) condition and do not evaluate distractor settings because combining source and distractor documents routinely exceeds current input-size limits (Appendix A.1). Consequently, RQ1 and RQ2 are answered using markdown alone, and our distractor findings should be read as text-based. The length match between the arms is also approximate: a residual token gap remains between the two pools (Appendix A.1), and because our accuracy differences are small, we cannot fully separate topical adjacency from that gap here; a per-item mixed-effects model with token count as a covariate would settle it. Our accuracies also come from a single evaluation run per cell, so we present the hard-negative penalty as directional rather than precise. Real documents buy realism at some cost in control: unlike synthetic suites such as RULER (Hsieh et al., 2024), they vary in style, complexity, and information density, and our paired design fixes the source but not the distractors. Curation is human-checked for topicality and answer leakage, which makes the comparison trustworthy at 270 questions but bounds its scale; automating that check with a judge ensemble is the most promising route to a larger version. The model panel is also incomplete: evaluatingclaude-sonnet-4.6was beyond the budget of this project and is left for future work. In addition,grok-4.3produces highly verbose responses, which substantially depress lexical F1 and EM even when the judge marks the answer correct, making judge accuracy the most reliable metric for that model (Appendix A.1). Our released codebase is designed to support image and PDF distractor conditions once they become servable, and future work should extend the evaluation to larger distractor loads (e.g.,k=8) and introduce a genuinely unrelated distractor arm to provide a less conservative baseline than the current hard-negative-derived random pool. 7 Conclusion MUDDLE separates two effects that long-context and retrieval-noise studies tend to mix: how far a distractor sits from the source in topic, and how much length it adds. The design holds the source document fixed, matches distractor length across the hard-negative and random arms, and renders every condition in markdown, page images, and PDF. The hard-negative curation pipeline, retrieval followed by model-assisted filtering and human verification, is what makes the matched comparison possible, since it yields distractors that are reliably on topic, free of the answer, and length-matched to the random pool. On the complete markdown sweep, topical adjacency rather than length drives the degrada- tion we observe. Forgpt-5-mini, hard negatives cost more than length-matched random documents at both context sizes, while random documents stay near the no-distractor base- line. The effect is small and model-dependent, and most pronounced at the larger context size. We read it as a directional result that a stronger design should be able to resolve. Read directionally, this suggests that distractor type, not just distractor count, may matter for how much retrieval noise hurts, and that a benchmark aiming to predict deployment behavior should control topical adjacency rather than length alone. Confirming it will take the stronger design we describe (Section 6). Ethics Statement The benchmark is built from an existing, publicly available document QA dataset and from documents retrieved by web search for use as distractors. Questions and answers trace to existing human annotations. We note that two source documents triggered a provider content filter and were handled during curation; the released data and evaluation code are intended for measuring robustness to distracting context. 8 Published as a conference paper at COLM 2026 References Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, p. 3119–3137, 2024. Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campag- nano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for RAG systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), p. 719–729, 2024. Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, and Cheng-Lin Liu. LongDocURL: A comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, p. 1135–1159, 2025. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long- context language models? In Proceedings of the First Conference on Language Modeling (COLM), 2024. Vladimir Karpukhin, Barlas O ̆ guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 6769–6781, 2020. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ̈ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ̈ aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, p. 9459–9474, 2020. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics (TACL), 12:157–173, 2024. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 2511–2522, 2023. Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, and Aixin Sun. MMLongBench-Doc: Benchmarking long-context document understanding with visualizations. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 37, p. 95963–96010, 2024. Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 2199–2208, 2021. Microsoft. MarkItDown: A tool for converting files to markdown.https://github.com/ microsoft/markitdown, 2024a. Microsoft. Playwright: Fast and reliable end-to-end testing for modern web apps.https: //github.com/microsoft/playwright, 2024b. Mozilla. PDF.js: A general-purpose, web standards-based platform for parsing and render- ing pdfs. https://github.com/mozilla/pdf.js, 2024. 9 Published as a conference paper at COLM 2026 Xiangyu Peng, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, and Chien-Sheng Wu. UniDoc-Bench: A unified benchmark for document-centric multimodal RAG. arXiv preprint arXiv:2510.03663, 2025. URL https://arxiv.org/abs/2510.03663. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 2383–2392, 2016. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Sch ̈ arli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, p. 31210–31227, 2023. Connor Shorten, Augustas Skaburskas, Daniel M. Jones, Charles Pierse, Roberto Esposito, John Trengrove, Etienne Dilocker, and Bob van Luijt. IRPAPERS: A visual document bench- mark for scientific retrieval and question answering. arXiv preprint arXiv:2602.17687, 2026. URL https://arxiv.org/abs/2602.17687. Jordy Van Landeghem, Rafal Powalski, Rub ` en Tito, Dawid Jurkiewicz, Matthew B. Blaschko, Lukasz Borchmann, Micka ̈ el Coustaty, Sien Moens, Michal Pietruszka, Bertrand Anckaert, Tomasz Stanislawek, Pawel J ́ oziak, and Ernest Valveny. Document understanding dataset and evaluation (DUDE). In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 19471–19483, 2023. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learn- ing for dense text retrieval. In The Ninth International Conference on Learning Representations (ICLR), 2021. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Representations (ICLR), 2024. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. Author Contributions Authors marked ∗ (Jason Luo and Saibilila Abudukelimu) contributed equally. Jason Luo designed the benchmark, built the shared evaluation pipeline, curated the hard negatives, and ran the GPT and Claude evaluations; Saibilila Abudukelimu built and ran the Gemini evaluations and baselines. Judy Song and Andrew Feng prepared the page-image and markdown tracks. Shivank Garg advised and supervised. Vasu Sharma and Kevin Zhu helped with the project reviews and on framing and writing. A Appendix A.1 Benchmark construction details Hard-negative review. Manual review recorded a verdict per question across explicit axes: whether the answer is recoverable from the source, whether any hard negative leaks the answer, whether hard negatives duplicate one another, whether they remain on topic, whether the question is answerable from general knowledge alone, and whether any hard negative is a dead giveaway. Of 274 reviewed items, 232 passed and 42 were flagged for repair. The answer-leak gate uses word-boundary matching that skips short tokens and bare small numbers to avoid spurious matches. 10 Published as a conference paper at COLM 2026 ConditionDistractor typek (added docs)Purpose Controlnone0Source-only ceiling HN-2hard negative2Topical similarity, short context Random-2random2Length-matched baseline for HN-2 HN-4hard negative4Topical similarity, longer context Random-4random4Length-matched baseline for HN-4 Table 2: The five context conditions. Each is instantiated for all 270 questions and rendered in all three modalities. The source document is identical across conditions; only the sur- rounding distractors change. Length control.Hard negatives fall in a band of 10 to 40 pages (median 23 pages, standard deviation 8.6). Documents outside the band were replaced by a topically matched, in-band substitute rather than truncated; 86 such replacements across 56 source documents are logged with provenance, and we verified that every distractor document falls inside[10, 40] pages. Random-distractor sampling. The per-question draw uses seedrandselv2|qid, ex- cluding that question’s own hard negatives and source; Random-2 is the first two of the same four-document draw, so it nests inside Random-4. Source placement.The insertion position uses a seeded random number generator keyed on the cell identifier (seed 42), so the position is fixed and reproducible; the same key is used across all models and all three modalities, so source placement is identical for a given cell everywhere it appears. Serving constraints. Multi-document image and PDF requests exceed provider limits. Azure OpenAI caps a request at 50 images and routes PDF blocks through the same pipeline (an effective limit of about 50 pages), which a source plus two to four 10 to 40 page distractors routinely exceeds; larger requests must use OpenRouter, whose gateway aborts uploads past roughly 15 MB. Across the suite, 24.7% of all cells exceed 15 MB, concentrated in the distractor conditions (4% of control, 16% ofk=2, 43% ofk=4); payload is driven by visual content rather than page count, so it cannot be bounded by limiting pages. Within image control, the largest cells fit only at a lower resolution (longest side 1024 px at JPEG quality 80) than the rest (1536 px at quality 90), so image resolution is mixed and correlated with document length, and we leave a single-resolution re-render to an extension. A.2 Dataset and run statistics This appendix gives the per-condition size and outcome statistics behind the main results. Table 3 reports the size of each context condition: median markdown characters, median pages or page-images, and the per-document page ranges for sources and distractors. Table 4 reports, for every model and modality we ran, judge accuracy (the primary metric), F1, and exact match. A.3 Prompts Model instruction (markdown). You are given one or more documents (converted to markdown). Using ONLY their content, answer the question with a short, direct answer and nothing else. If the documents do not contain the answer, reply "Not answerable". The image and PDF variants replace “documents (converted to markdown)” with “docu- ments as page images” and “PDF documents” respectively. Judge instruction. 11 Published as a conference paper at COLM 2026 ConditionMarkdown charsPages / imagesPer-doc pages (median)(median)sourcedistractor Control115,94335 9 to 198 (med. 28)10 to 40 (med. 23) HN-2278,31684 Random-2250,67786 HN-4382,850123 Random-4362,197135 Table 3: Per-cell size by condition (medians), and per-document page ranges. Source documents are unbounded in length (9 to 198 pages); hard-negative and random distractors are bounded to[10, 40]pages so that distractor type is not confounded with length at a given k. Pages and page-images are one-to-one. ModelModalityConditionJudgeF1EM Markdown (all five conditions): gpt-5-minimarkdownControl0.7630.6660.448 gpt-5-minimarkdownHN-20.7370.6410.444 gpt-5-minimarkdownRandom-20.7670.6590.444 gpt-5-minimarkdownHN-40.7070.6320.448 gpt-5-minimarkdownRandom-40.7480.6390.437 gemini-3.5-flashmarkdownControl0.7330.5790.374 gemini-3.5-flashmarkdownHN-20.7110.5450.352 gemini-3.5-flashmarkdownRandom-20.7220.5860.415 gemini-3.5-flashmarkdownHN-40.7260.5610.385 gemini-3.5-flashmarkdownRandom-40.7110.5750.393 grok-4.3markdownControl0.7440.3950.248 grok-4.3markdownHN-20.7330.3140.181 grok-4.3markdownRandom-20.7260.4120.270 grok-4.3markdownHN-40.6820.2950.203 grok-4.3markdownRandom-40.7070.3280.208 Image and PDF (control only; distractor conditions infeasible, see §6): gpt-5-minipdfControl0.8040.7010.493 gemini-3.5-flashpdfControl0.7780.6510.448 grok-4.3pdfControl0.6670.2710.152 gpt-5-miniimageControl0.8040.7090.519 gemini-3.5-flashimageControl0.7890.6550.470 grok-4.3imageControl0.8140.2380.123 Table 4: Per-condition judge accuracy (primary), F1, and EM. Markdown reports all five con- ditions for all three models. Image and PDF report control only; their distractor conditions are not run (Section 6).grok-4.3’s F1 and EM are deflated by verbose answers (mean 254 response characters against 34 forgpt-5-mini), not by lower correctness, so judge accuracy is the trustworthy metric for it. You are scoring a model’s answer against a reference answer for a long-context document QA benchmark. [question, reference, prediction inserted] Rules: mark correct: true only if the prediction conveys the same factual content as the reference; numbers and units are matched by value; list order does not matter and missing items earn partial credit; minor paraphrase is acceptable; refusals are incorrect. Return only a JSON object"correct": true|false, "score": 0.0-1.0, "reasoning": "one short sentence". 12