Paper deep dive
HAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generation
Yin Li, Ziyang Hu, Zhiyu Guo, Xiangyu Liu, Wenbin Li, Boo-Ho Yang, Rav Lawana, Ziyue Li, Wei Zeng, Fugee Tsung
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 8/17/2026, 4:52:28 AM
Summary
The paper introduces HAM-RAG, a Hierarchy-Aware Multimodal Retrieval-Augmented Generation framework designed to preserve document structure and local text-image logic during interleaved generation. Unlike existing methods that flatten documents, HAM-RAG uses document hierarchy as a grounding signal to contextualize textual and visual evidence. The authors also propose HAM-Bench, a benchmark covering Wukong, Wiki, arXiv, and Recipe domains to evaluate structure-faithful generation. Experiments show HAM-RAG significantly outperforms non-hierarchical baselines, particularly in local text-image alignment.
Entities (22)
Relation Signals (17)
Fugee Tsung → affiliatedwith → The Hong Kong University of Science and Technology (Guangzhou)
confidence 95% · Fugee Tsung † The Hong Kong University of Science and Technology (Guangzhou)
Yin Li → affiliatedwith → The Hong Kong University of Science and Technology (Guangzhou)
confidence 95% · Yin Li * The Hong Kong University of Science and Technology (Guangzhou)
HAM-Bench → contains → WIKI
confidence 95% · HAM-Bench covers Wukong, Wiki, arXiv, and Recipe
HAM-Bench → contains → Wukong
confidence 95% · HAM-Bench covers Wukong, Wiki, arXiv, and Recipe
HAM-Bench → contains → arXiv
confidence 95% · HAM-Bench covers Wukong, Wiki, arXiv, and Recipe
HAM-Bench → contains → Recipe
confidence 95% · HAM-Bench covers Wukong, Wiki, arXiv, and Recipe
HAM-RAG → uses → document hierarchy
confidence 95% · HAM-RAG uses document hierarchy as a grounding signal across retrieval and generation
Wukong → derivedfrom → Black Myth: Wukong
confidence 90% · Wukong is built from publicly available Black Myth: Wukong walkthroughs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing multimodal RAG methods often flatten structured documents into isolated text and image units, weakening the source organization and local text-image logic needed for faithful evidence selection and placement. We propose HAM-RAG, a Hierarchy-Aware Multimodal RAG framework for structure-faithful interleaved generation. HAM-RAG uses document hierarchy as a grounding signal across retrieval and generation, contextualizing textual and visual evidence and preserving source position and local text-image relations in the prompt. We further introduce HAM-Bench, covering Wukong, Wiki, arXiv, and Recipe across game walkthroughs, web pages, scientific papers, and step-wise recipe documents. Across multiple backbones, HAM-RAG improves the main multimodal average by 17.3% over the strongest non-hierarchical baseline. On Wukong, HAM-RAG improves Img-CBS by 24.2% over the strongest non-hierarchical baseline, demonstrating substantially better local text-image alignment. The main experiments and ablation study together demonstrate that document hierarchy is a key grounding signal for faithful image selection, placement, and local text-image alignment. These findings highlight the value of hierarchy-aware grounding for reliable multimodal assistants that generate answers faithful to the source organization, procedural structure, and local text-image evidence of structured documents, such as technical manuals, maintenance guides, and industrial SOPs. The code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.14032v1
- Canonical: https://arxiv.org/abs/2608.14032v1
Trouble viewing inline? Open PDF directly →
Full Text
39,534 characters extracted from source content.
Expand or collapse full text
Proceedings of the Asia Pacific Industrial Engineering & Management Systems Conference 2026 HAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generation Yin Li * The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China Tel: +86 18689212309, Email: yligt@connect.hkust-gz.edu.cn Ziyang Hu * The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China Tel: +86 18250709675, Email: ziyanghu@hkust-gz.edu.cn Zhiyu Guo * The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China Tel: +86 15557197763, Email: guozhiyu22@mails.ucas.ac.cn Xiangyu Liu The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China Tel: +86 18866254321, Email: t330034034@mail.bnbu.edu.cn Wenbin Li ASCETEX INTERNATIONAL LIMITED, Hong Kong, China Tel: +86 15021995769, Email: arvinlee@ascetex.com Boo-Ho Yang MOVENSYS Inc., Seongnam-si, Republic of Korea Tel: +86 18129872007, Email: byang@movensys.com Rav Lawana Schneider Electric, Shanghai, China Tel: +86 18616886388, Email: rav.lawana@se.com Ziyue Li Technical University of Munich, Munich, Germany Tel: +49 17679780820, Email: ziyue.li@tum.de Wei Zeng The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China Tel: +86 18823423978, Email: weizeng@hkust-gz.edu.cn Fugee Tsung † The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China The Hong Kong University of Science and Technology, Hong Kong SAR, China Tel: +86 14714336811, Email: season@hkust-gz.edu.cn Abstract. Existing multimodal RAG methods often flatten structured documents into isolated text and image units, weakening the source organization and local text-image logic needed for faithful evidence selection and placement.We propose HAM-RAG, a Hierarchy-Aware Multimodal RAG framework for structure-faithful interleaved generation. HAM-RAG uses document hierarchy as a grounding signal across retrieval and generation, contextualizing textual and visual evidence and preserving source position and local text-image relations in the prompt. We further introduce HAM-Bench, covering Wukong, Wiki, arXiv, and Recipe across game walkthroughs, web pages, scientific papers, and step-wise recipe documents. Across multiple backbones, HAM-RAG improves the main multimodal average by 17.3% over the strongest non-hierarchical baseline. On Wukong, HAM-RAG improves Img-CBS by 24.2% over the strongest non-hierarchical baseline, demonstrating substantially better local text-image alignment. The main experiments and ablation study together demonstrate that document hierarchy is a key grounding signal for faithful image selection, placement, and local text-image alignment. These findings highlight the value of hierarchy-aware grounding for reliable multimodal assistants that generate * These authors contributed equally.†: Corresponding Author. arXiv:2608.14032v1 [cs.IR] 14 Aug 2026 answers faithful to the source organization, procedural structure, and local text-image evidence of structured documents, such as technical manuals, maintenance guides, and industrial SOPs. The code is available at https://github.com/MCCodeAI/HAM-RAG.git. Keywords: Multimodal RAG; Structured Documents; Document Hierarchy; Interleaved Text-Image Generation. 1. INTRODUCTION Retrieval-Augmented Generation (RAG) enhances large language models by grounding generation in external knowledge sources (Lewis et al., 2020). Early multimodal RAG systems retrieve textual and visual evidence to support answer generation (Chen et al., 2022). More recent approaches and benchmarks further study multimodal and interleaved text-image responses (Zhu et al., 2025; Ma et al., 2024; Yu et al., 2025a; Dong et al., 2025). Yet many real-world multimodal sources are structured documents, not flat pools of independent passages and images. Scientific papers organize figures under sections and captions; recipes arrange images around procedural steps; game walkthroughs and web pages rely on headings, local descriptions, and ordered visual demonstrations. In these documents, hierarchy is not cosmetic metadata. It defines the semantic scope of a text span, the local context that makes an image relevant, and the order in which multimodal evidence should be understood. The goal of multimodal RAG in this setting is therefore not only to retrieve useful text and images, but to preserve the source document’s contextual organization and text-image logic in the generated answer. This is particularly important for interleaved generation. An image can be topically related to the query but still be wrong if it is taken from a different section, a different step, or a different local explanation. Conversely, a correct image can become misleading when inserted beside unsupported text. As shown in Figure 1, text-only generation lacks visual grounding, while a hierarchy-agnostic multimodal baseline can select plausible visual evidence but arrange it with weak local alignment. A structure-faithful multimodal RAG system should select images and place them where their surrounding text is supported by the original document. This paper studies whether document hierarchy should be treated as a first-class grounding signal for multimodal RAG over structured documents. We focus on three coupled requirements: textual scope, where retrieved text should retain the section or step context that gives it meaning; image grounding, where selected images should be tied to their captions, headings, and neighboring descriptions; and image placement, where generated answers should preserve the local relation between visual evidence and the text it supports. This formulation differs from generic multimodal retrieval because the correctness of an answer depends on source-local organization, not just global semantic relevance. Usesmultimodalevidencebutplacesone broadsceneimagebesidemultipleclaims, r es ul t i nginw eakl oc a lte x t-i m a g e alignment. (a) Text-only baseline (b) HAM-RAG without hierarchical context (c) HAM-RAG with hierarchical context Where is the Pale-Axe Stalwart? The Pale-Axe Stalwart is a boss character you encounter at the beginning of Chapter 5 in Black Myth: Wukong. To find him initially, you must follow the path from the Ashen Pass Keeper's Shrine. This will lead you to an open area where a large fight is taking place, with firebombs being thrown down from above. You will find the Pale-Axe Stalwart in the middle of this fight, taking on Revenants and Bull Soldiers. LLM Where is the Pale-Axe Stalwart? LLM The Pale-Axe Stalwart is a boss character you encounter at the beginning of Chapter 5 in Black Myth: Wukong. To find him initially, you must follow the path from the Ashen Pass Keeper's Shrine. This will lead you to an open area where a large fight is taking place, with firebombs being thrown down from above. You will find the Pale-Axe Stalwart in the middle of this fight, taking on Revenants and Bull Soldiers. After you defeat him, the Pale-Axe Stalwart becomes an NPC with his own side quest. His location will change as you progress through his questline. Text-only response lacks visual grounding and source-local image support. Where is the Pale-Axe Stalwart? LLM The Pale-Axe Stalwart is a character you first encounter as a boss at the start of Chapter 5 in the Woods of Ember region. You will first find the Pale-Axe Stalwart in the Ashen Pass I area. From the first Keeper's Shrine of the chapter, go to the right. After passing through an underpass, you will enter an open area where Revenants and Bull Sergeants are fighting amidst falling firebombs. The Pale-Axe Stalwart is in the middle of this battle. After you defeat him, he will become an NPC who gives you a side quest involving the Five Element Carts. HAM-RAGpreservessourcehierarchy andlocaltext-imagealignment,producing m or efa it hfulandl oc al l ygr oun ded interleavedanswers. Figure 1: Comparison of a text-only baseline, a flat RAG baseline, and HAM-RAG. HAM-RAG better preserves source hierarchy and local text-image alignment in its final answers. To address this problem, we propose HAM-RAG, aHierarchy-AwareMultimodalRetrieval-Augmented Generation framework for structured documents. HAM-RAG represents textual and visual units with hierarchy-aware contextual signals.Specifically, on the text side, each retrievable unit is enriched with source-grounded heading and section context. On the image side, each image is represented with its document position, heading context, nearby text, caption or alt text, and a context-aware visual description. These representations allow retrieval to operate over evidence units that expose their structural role and local cross-modal context.The generator then receives a structured prompt containing ranked text and image evidence, source identifiers, document positions, and image descriptions, encouraging the final answer to use images in locally supported contexts. We further introduce HAM-Bench, a benchmark for evaluating hierarchy-aware multimodal RAG. HAM-Bench covers diverse structured document types, including web pages, scientific papers, step-wise recipe documents, and game walkthroughs.It includes both natively structured documents and documents whose structure is reconstructed from available source materials. Unlike existing multimodal QA and RAG benchmarks (Chang et al., 2022; Li et al., 2022; Yu et al., 2025a; Dong et al., 2025), HAM-Bench focuses on structure-preserving multimodal generation and provides text-image local alignment metadata for evaluating whether generated visual evidence is selected and placed in the correct document context. Based on HAM-Bench, we evaluate hierarchy-aware multimodal RAG from multiple perspectives, including retrieval sufficiency, image selection, image-context alignment, and overall answer quality. Experiments show that HAM-RAG improves the main multimodal average by 17.3% across matched backbones and increases Img-CBS by 24.2% on Wukong over the strongest non-hierarchical baseline. These gains demonstrate that hierarchy-aware modeling strengthens visual grounding and local text-image alignment while preserving source organization. Our contributions are: • We formulate structure-faithful multimodal RAG and propose HAM-RAG, which uses document hierarchy to contextualize text and image evidence. • WeconstructHAM-Bench,coveringnativeand reconstructed structured documents with local text-image alignment metadata. • We design an evaluation protocol and show consistent gains in grounding and structural faithfulness across datasets and backbones. 2. RELATED WORK Multimodalretrieval-augmentedgenerationextends text-only RAG to settings in which textual and visual evidence are retrieved to support answer generation.Early systems such as MuRAG (Chen et al., 2022) retrieve both textual and visual knowledge to support text answer generation. More recent approaches and task formulations such as MuRAR (Zhu et al., 2025) and M2RAG (Ma et al., 2024) extend this setting toward multimodal answers, while VisRAG (Yu et al., 2025b) preserves page appearance and layout through visual document retrieval.However, many existing approaches emphasize retrieval architecture, evidence fusion, or generation without explicitly modeling document hierarchy and local text-image relations. They therefore provide limited support for structured documents, where the meaning of an evidence unit often depends on its hierarchical position and surrounding context. Structure-aware retrieval and multimodal document understanding address some limitations of flat chunking. Wiki-LLaVA uses hierarchical retrieval over multimodal Wikipedia documents (Caffagni et al., 2024); M3DocVQA introduces the M3DocRAG framework, which preserves page-levelvisualinformationduringcross-pageand multi-document retrieval (Cho et al., 2025); and MLDocRAG organizesmultimodalcross-pageevidencethrougha query-centric graph (Zhang and Wu, 2026). These studies show that structural and cross-page context is important for document-level understanding.Nevertheless, their primary focus is document QA, long-document retrieval, or multimodal understanding rather than preserving source organization and local text-image logic during interleaved generation. HAM-RAG instead treats document hierarchy not merely as auxiliary metadata, but as a core grounding signal for both textual and visual evidence. Benchmarks for multimodal RAG and document QA evaluate how systems use multimodal evidence and generate grounded answers.WebQA (Chang et al., 2022) and MMCoQA (Li et al., 2022) evaluate question answering grounded in multimodal evidence,whereas MRAMG-Bench (Yu et al., 2025a) and MMDocRAG (Dong et al., 2025) extend evaluation to multimodal retrieval, evidence selection, and interleaved text-image answer generation. These benchmarks do not explicitly measure whether outputs preserve section hierarchy, procedural order, and local correspondence between text and images. HAM-Bench fills this gap by evaluating hierarchy-aware multimodal RAG over structure-preserving documents, with metrics for grounding, image placement, local text-image alignment, and structural faithfulness to the source. 3. HAM-RAG FRAMEWORK HAM-RAG performs retrieval-augmented generation over hierarchical multimodal documents. As illustrated in Figure 2, it consists of an offline hierarchy-aware indexing stage and an online multimodal retrieval-and-generation stage. Unlike flat RAG pipelines that represent passages and images as isolated chunks, HAM-RAG uses document hierarchy as a shared signal for evidence representation, retrieval, and prompt construction. 3.1 Hierarchy-Aware Document Representation Let D= 퐷 1 ,..., 퐷 푁 denote a corpus of structured multimodal documents.Each document is parsed into a hierarchy tree 푇 퐷 = (푉 퐷 , 퐸 퐷 ),(1) where internal nodes represent the document, sections, subsections, or procedural steps, leaf nodes represent retrievable text and image units, and edges encode parent–child relations. The hierarchy is recovered from available document markup, heading levels, numbering patterns, figure–caption links, layout cues, and source order. For each retrievable leaf unit, HAM-RAG constructs a structured evidence object 푒 푖 =푥 푖 ,푚 푖 , 푝 푖 ,푎 푖 ,푙 푖 ,푟 푖 ,(2) where푥 푖 is the original text or image content, 푚 푖 is its modality, 푝 푖 is the path from the document root to the unit, 푎 푖 contains ancestor headings and section context, 푙 푖 contains nearby text-image context, and 푟 푖 records the unit’s local relation to surrounding evidence. Multimodal Document Multimodal Generation LLM [Instruction] ... [User Query] ... [Retrieved Text] ... [Retrieved Image] ... UserQuery ImageChunks ImageChunks Retrieved Image TextChunks TextChunks RetrievedText Retrieval Topk t Topk v (b) Online Multimodal RAG for Text-Image Generation (a) Offline Hierarchy-Aware Vector StoreConstruction Vector Store Hierarchy-Aware Textual Embedding Hierarchy-Aware Visual Embedding Hierarchy Parsing & Summarization Subsection Section TextChunks TextChunks TextChunks ImageChunks ImageChunks ImageChunks TextChunk: SubsectionSummary: Section Summary: ImageChunk: SubsectionSummary: Section Summary: Enc. VLM Enc. Description Prompt Figure 2: Overview of HAM-RAG. The offline stage constructs and indexes hierarchy-aware text and image evidence, while the online stage retrieves ranked evidence and organizes it for structure-faithful interleaved generation. For example,푟 푖 may indicate that an image illustrates the current procedural step, belongs to a figure caption, or supports the preceding paragraph. This representation makes the semantic scope and local role of each evidence unit explicit before retrieval. 3.2 Offline Evidence Construction and Indexing Figure 2(a) summarizes the offline stage. For a text unit, HAM-RAG serializes the original content together with its document title, heading path, section context, and neighboring evidence into a hierarchy-aware textual representation. This preserves source wording while exposing the structural context needed to distinguish passages that are topically similar but belong to different sections or steps. For an image unit, the framework retains its source position, heading path, caption or alt text, and nearby textual context.It additionally generates a context-aware visual description conditioned on both the image and its surrounding document evidence.This description supplements sparse or missing captions and helps connect visual content to the section or procedure in which it appears. Text and image evidence are then serialized into field-structured embedding prompts, embedded, and stored in a shared vector store together with provenance metadata such as document identifier, source position, modality, and image reference. Consequently, hierarchy affects retrieval through the indexed representation rather than being appended only after retrieval. 3.3 Online Retrieval and Interleaved Generation As shown in Figure 2(b), the online stage receives a user query and retrieves the top-ranked textual and visual evidence under separate budgets 푘 푡 and 푘 푣 .HAM-RAG retains each hit’s relevance score, source identifier, document position, hierarchy-aware content, and local text-image relation. Keeping these fields allows evidence from different documents or sections to remain distinguishable during generation. The selected evidence is assembled into a structured prompt 푃 ham = Assemble(푞,E ∗ ),(3) where E ∗ contains the ranked text and image evidence. Text blocks expose source-grounded headings and neighboring context, while image blocks include the image reference, caption or alt text, source position, and context-aware description. The generator therefore receives not only relevant content but also the local organization needed to decide which image supports which statement and where that image should be placed. In this way, hierarchy participates twice: it shapes the retrieval representation offline and guides evidence organization and interleaved text-image generation online. Table 1: Comparison of HAM-Bench with representative multimodal QA/RAG benchmarks. BenchmarkSettingMM Inter. Struct. Recon. Hier. Eval. WebQA (Chang et al., 2022)M QAYesNoNoNoNo MRAMG-Bench (Yu et al., 2025a) M RAGYesYesPartial PartialNo MMDocRAG (Dong et al., 2025)M DocRAGYesYesPartialNoNo HAM-BenchHier. M RAG YesYesYesYesYes 4. HAM-BENCH In Table 1, M denotes multimodal evidence, Inter. denotes interleaved text-image answer generation, Struct. denotes explicit preservation or use of document structure, Recon.denotes reconstruction of hierarchy from flattened sources, and Hier. Eval. denotes explicit hierarchy-aware evaluation. We assign Yes when a benchmark fully supports a feature, Partial when the support is limited or indirect, and No when the feature is absent. To evaluate whether multimodal RAG can faithfully preserve the contextual organization and text-image logic of structured documents, we introduce HAM-Bench, a benchmark built on hierarchy-preserving multimodal corpora.Unlike prior multimodal QA/RAG benchmarks that mainly focus on evidence usage or multimodal answer quality, HAM-Bench is designed to test whether retrieval and generation can exploit document hierarchy, local context, and text-image associations under a shared retrieval-and-generation setting. As shown in Table 1, HAM-Bench differs from existing benchmarks in three aspects.First, it uses multimodal documents whose hierarchical organization is explicitly preserved.Second, when source documents are originally flattened in prior resources, we reconstruct their document boundaries, section structures, and local text-image alignments. Third, it supports evaluation of hierarchy-aware retrieval and interleaved text-image generation, making it suitable for studying structure-aware multimodal RAG. 4.1 Benchmark Construction and Statistics HAM-Bench contains four diverse types of structured documents.Wukong is built from publicly available Black Myth: Wukong walkthroughs, which naturally contain section-level hierarchy and dense interleaved visual evidence. Wiki is traced back to WikiWeb2M (Burns et al., 2023) pages, arXiv to original PDFs parsed into section-structured Markdown using MinerU (Wang et al., 2024), and Recipe to source recipe documents reorganized into step-wise hierarchical procedures.For all reconstructed subsets, we restore structures already present in the original sources, including headings, section boundaries, figure-caption links, and procedural steps, rather than redesigning the content. Table 2 shows that the four subsets pose complementary structural challenges.arXiv has deep section hierarchies and dense visual content; Recipe and Wukong emphasize procedural order and local text-image alignment, with Wukong additionally requiring multi-document reasoning; and Wiki contains shallower hierarchies and fewer images. Together, they cover scientific, procedural, and web documents. 5. EXPERIMENTS 5.1 Experimental Setup 5.1.1 Compared Methods We adopt and reimplement the three baseline paradigms used in MRAMG-Bench—LLM-Based, MLLM-Based, and Rule-Based—because MRAMG-Bench shares our goal of retrieval-augmented interleaved text-image generation (Yu et al., 2025a). In contrast, related systems such as VisRAG, Wiki-LLaVA, and M3DocRAG primarily target retrieval or document QA rather than interleaved text-image answer generation, making them not directly comparable under our generation metrics. All methods are evaluated under the same retrieval-and-generation setting to isolate the effect of document hierarchy. • HAM-RAG. Our proposed hierarchy-aware framework. It enriches both text and image units with structural context derived from document hierarchy before retrieval, so that retrieved evidence better preserves the source document’s contextual organization and local text-image associations. • LLM-Based.A text-only generation paradigm that represents images through captions and surrounding textual descriptions. The generator receives retrieved text together with image textual proxies, but does not directly process visual inputs. • MLLM-Based.A multimodal generation paradigm that directly feeds retrieved texts and selected images into an MLLM. Since MLLMs are constrained by the number of input images, a CLIP-based filtering module is used to retain the most query-relevant images before generation. • Rule-Based. A post-hoc image insertion paradigm. It first generates a pure textual answer and then inserts images by matching candidate images with answer sentences using lexical and semantic similarity. For each paradigm, we instantiate the generation module with multiple backbone models when applicable, including GPT-5, Gemini-2.5, and Qwen2.5, among others. This allows us to examine whether the effect of hierarchy-aware modeling remains consistent across different model families. 5.1.2 Implementation and Generation Settings All methods use the same retrieval and generation budget. We use BGE-M3 as the shared embedding backbone for initial retrieval, so that the comparison focuses on how evidence units are represented before indexing rather than on differences in retrieval encoders. The baselines encode evidence in a flat manner, whereas HAM-RAG augments each text or image unit with hierarchy-aware contextual signals from the source document. For hybrid retrieval, dense and sparse scores are combined with weights 0.7 and 0.3, respectively.Remote generators are called through the same OpenAI-compatible Table 2: Structural statistics of HAM-Bench. Native/Recon. indicate native/reconstructed document hierarchies. Imgs counts unique source images; Img/Ans, Img/Doc, and QA/Doc are mean images per answer, image occurrences per document, and QA-document associations per document, respectively. #H1/#H2 are mean level-1/2 headings per document. SubsetDomainStruct. Source Multi-Doc Proc. QA Docs QAs Imgs Img/Ans Img/Doc QA/Doc #H1#H2 Wukong Game WalkthroughNativeYesYes43154025944.608.832.111.042.03 WikiWebRecon.NoNo5385005391.000.980.931.003.65 arXivAcademicRecon.NoNo10119131361.3531.051.891.02 12.78 RecipeLifestyleRecon.NoYes1528 2360 85692.155.601.541.004.97 interface with temperature 0.2, streaming disabled, and a maximum generation length of 4,000 tokens. For downstream generation, each method is allowed to use up to 15 retrieved text units and 10 images. This budget balances evidence coverage and inference feasibility: some questions require multiple text units to recover sufficient context, while image-intensive questions require several visual references to produce complete interleaved answers. Due to the relatively large scale and higher inference cost of the Recipe subset, we randomly sample 500 QA pairs from this subset for evaluation using seed 36, while keeping the same retrieval-and-generation settings as the other subsets. Invalid generations are counted as failures, retained in the evaluation, and assigned zero scores on the corresponding automatic metrics, so the final results reflect both answer quality and operational robustness. 5.1.3 Evaluation Metrics We evaluate both retrieval quality and multimodal generation quality. All methods are evaluated with the same prompts, scripts, and retained failed-case policy. For retrieval evaluation, we separately assess textual evidence coverage and visual evidence coverage.For text retrieval, we report Text Context Recall@15 (TCR@15), an LLM-based set-level metric that measures whether the top-15 retrieved text chunks collectively provide sufficient evidence for answering the query. For image retrieval, we report Image Recall (IR), which measures whether the reference images needed by the answer are covered by the retained image set. For generation evaluation, the main table focuses on the metrics that directly reflect interleaved multimodal answer fidelity: image grounding, local text-image alignment, and overall answer quality. We use Image F1 to measure whether the generated answer selects the correct supporting images. To further evaluate image placement without introducing an additional threshold or human annotation, we define Image Context BERTScore (Img-CBS), a placement-sensitive continuous metric that applies BERTScore (Zhang et al., 2020) to compare the text surrounding each correctly selected image in the generated answer with the corresponding local context in the reference answer. Unlike Image F1, which only evaluates image selection, Img-CBS rewards a method only when the selected image is placed near semantically appropriate supporting text. Finally, we use an LLM-based Overall Quality score to assess the correctness, completeness, grounding, and text-image coherence of the generated multimodal answer as a whole. In Table 3, the M Avg. column reports the unweighted mean over the four datasets and three multimodal generation metrics: Image F1, Img-CBS, and Qual. ×20. 5.2 Overall Performance Table 4 summarizes retrieval performance.Since all baselines share the same text retriever, their TCR@15 results are reported jointly; MLLM-Based is separated for image retrieval because it applies CLIP filtering. HAM-RAG achieves higher TCR@15 on all four subsets and higher IR on three. On arXiv, the shared baseline reaches 0.99 IR versus 0.86 for HAM-RAG, likely because flat retrieval retains more figures in this image-dense subset. Since IR measures coverage rather than placement, this exception does not contradict HAM-RAG’s stronger local text-image alignment. Table 3 demonstrates that HAM-RAG’s advantage is consistent across generator families rather than being driven by a single high-capacity model.For all nine matched backbones, HAM-RAG achieves a higher M Avg. than its LLM-Based counterpart. Averaged across these backbones, M Avg.increases from 54.68 to 64.12, corresponding to a 17.3% relative improvement. This consistency across proprietary and open-source models of different scales indicates that the gain primarily comes from hierarchy-aware evidence organization before generation, rather than model-specific generation capability. The improvements are most informative on metrics that evaluate whether visual evidence is selected and used in the correct local context. On Wukong, where answers depend on ordered walkthrough steps and interleaved screenshots, HAM-RAG increases the best Img-CBS from 50.82 to 63.10, a 24.2% improvement over the strongest non-hierarchical baseline.The gains on Recipe and arXiv further indicate that structural context helps associate images with the relevant procedure or section instead of merely retrieving topically related figures. Wiki is less discriminative for image selection because each question contains only one reference image and Img-F1 frequently saturates; its results should be interpreted together with Img-CBS and Qual. Overall, the results show that hierarchy improves not only aggregate performance, but also visual grounding and local text-image alignment. In summary, the results support our central claim that Table 3: Main multimodal generation results on HAM-Bench, covering image selection, local image-context alignment, and holistic multimodal quality. M Avg. is the unweighted mean of Img-F1, Img-CBS, and Qual. ×20 across all subsets. FrameworkModel Wukong↑Wiki↑arXiv↑Recipe↑M Avg. ↑ Img-F1Img-CBSQual.Img-F1Img-CBSQual.Img-F1Img-CBSQual.Img-F1Img-CBSQual. HAM-RAG DeepSeek-V355.6755.493.7410070.483.3950.5750.823.3474.8165.743.7667.35 Gemini-2.5-Flash65.3159.173.7697.5369.713.3954.0149.243.3987.8271.103.8170.07 Gemini-2.5-Pro67.5963.103.8893.8065.933.4745.5241.173.4685.6271.403.8468.93 GPT-4o57.4452.083.7499.6069.593.4244.0947.873.3683.1163.303.7866.92 GPT-5 60.5251.523.8898.3369.073.5058.6953.233.6288.0769.893.9370.66 Llama-3.1-70B-Inst42.9140.323.4684.9762.583.2323.3517.763.1958.7041.743.5553.41 Llama-3.1-8B-Inst44.3046.733.3390.8368.772.9518.6227.652.6763.6058.223.4155.49 Qwen2.5-72B-Inst55.9354.603.5899.5370.263.3252.2247.913.4483.6962.503.6967.27 Qwen2.5-7B-Inst41.1445.693.3396.1367.893.2229.3826.793.1161.2454.093.4156.98 LLM-Based DeepSeek-V343.7945.573.4297.8766.073.3631.1434.283.0755.1352.103.5157.76 Gemini-2.5-Flash52.6550.823.4292.3366.713.2854.2444.333.2372.1361.483.5863.74 Gemini-2.5-Pro 53.0649.613.4582.7358.763.3250.8838.863.3574.1161.443.6061.99 GPT-4o45.9744.553.5298.3369.423.3241.5038.093.0366.9854.113.5760.65 GPT-548.6445.713.7196.7367.733.5349.7444.913.5177.2964.043.8165.50 Llama-3.1-70B-Inst 31.8537.473.1073.9263.183.0212.4712.092.8236.7032.342.9744.85 Llama-3.1-8B-Inst24.0535.613.2922.2922.993.187.9613.532.5429.6834.582.9435.81 Qwen2.5-72B-Inst43.1844.583.4596.6768.563.2636.0630.393.2857.3850.413.4357.97 Qwen2.5-7B-Inst 30.8437.052.9660.9469.862.6712.0012.602.7637.1743.272.7643.89 MLLM-Based Gemini-2.5-Flash39.2740.573.3996.1369.463.3036.0531.423.3358.0949.733.5157.61 Gemini-2.5-Pro 40.2640.823.4586.1361.323.3236.7930.543.3463.1551.473.5757.01 GPT-4o25.8731.143.3898.2070.743.2726.9531.653.1249.9342.833.4953.54 GPT-532.7537.353.6591.4763.053.5234.5732.983.5661.2149.323.8057.77 Qwen2.5-VL-72B-Inst29.8734.233.2793.4769.463.1525.1126.573.1742.3337.243.2051.17 Qwen2.5-VL-7B-Inst 17.0224.282.3058.0139.832.435.174.242.4310.8411.262.2329.87 Rule-Based DeepSeek-V326.0529.053.1630.7022.803.1315.5324.022.8317.8322.392.8335.61 Gemini-2.5-Flash22.5527.393.1040.3329.993.0311.5822.472.8417.5920.112.9735.90 Gemini-2.5-Pro 31.2231.573.1044.4032.453.0314.1025.182.8927.4926.513.1439.68 GPT-4o21.4926.183.1734.8026.183.179.8218.562.8113.9317.602.9734.25 GPT-531.4330.733.3221.2014.983.2315.5922.813.1023.4322.603.3036.81 Llama-3.1-70B-Inst 15.5221.392.8436.2026.842.9114.1616.062.7013.6913.282.8031.85 Llama-3.1-8B-Inst 21.3925.452.6134.4024.682.629.9210.682.4814.5016.512.5630.24 Qwen2.5-72B-Inst25.8927.863.1736.2026.743.1410.2816.922.8816.1219.212.9135.10 Qwen2.5-7B-Inst25.0928.612.8638.4028.332.8511.1714.152.6011.5712.402.6532.41 Table 4: HAM-Bench retrieval results. StageMethod / PipelineWukong↑ Wiki↑ arXiv↑ Recipe↑ Text (TCR@15) HAM-RAG0.990.800.770.97 Shared Baseline (LLM / MLLM / Rule)0.800.700.620.91 Image (IR) HAM-RAG0.831.000.860.97 MLLM-Based (CLIP-filtered)0.500.980.570.73 Shared Baseline (LLM / Rule)0.650.980.990.92 document hierarchy is a critical signal for multimodal RAG over structured documents. Compared with non-hierarchical paradigms, HAM-RAG better preserves source-document context and improves the alignment between retrieved images and their corresponding textual units, which is essential for faithful interleaved text-image generation. 5.3 Ablation Study Using GPT-4o on 540 Wukong questions, we separately ablate the hierarchy encoded in image and text representations. SI and LI denote the context-aware and weaker local image representations, respectively; LI omits context-aware visual descriptions. CT and ST denote hierarchy-aware and isolated text representations. SI+ST weakens text hierarchy, LI+CT weakens image hierarchy, and LI+ST weakens both. Figure 3 reports means with 95% confidence intervals ( ̄푥± 1.96푠/ √ 540); Average Score is the per-question mean of Image F1, Img-CBS, 0.00 0.20 0.40 0.60 Image F1 Score 0.00 0.20 0.40 0.60 Image Context BERTScore 0.30 0.40 0.50 0.60 0.70 Average Score HAM-RAG (Full)HAM-RAG w/ SI+STHAM-RAG w/ LI+CTHAM-RAG w/ LI+ST Figure 3: HAM-RAG ablation results on Wukong for Image F1 Score, Image Context BERTScore, and Average Score. and Qual./5. The full model has the highest mean on all three metrics. Replacing SI with LI sharply reduces both image metrics. SI+ST yields smaller declines, whereas LI+ST has the lowest Average Score, showing complementary image- and text-side contributions. 5.4 Robustness and Cost Across 57,123 main-experiment generation runs, the four evaluated frameworks produced only 38 failures, corresponding to a failure rate of approximately 0.067%. Under the same evaluation setting, HAM-RAG produced only 2 failures, fewer than LLM-Based (29) and MLLM-Based (7), with both failures occurring on the arXiv subset where long section context and dense figures make prompt construction more difficult. The Rule-Based pipeline produced no failures, but it uses a much lighter text-first generation and post-hoc image insertion process rather than generating fully grounded interleaved multimodal answers. Macro-averaged across the four subsets under the GPT-4o pricing model, HAM-RAG has an estimated cost of 3.578 cents per question, lower than LLM-Based and MLLM-Based but higher than Rule-Based. This pattern reflects the intended trade-off of structure-aware multimodal RAG: preserving document organization and local text-image evidence requires richer prompts, but yields stronger grounding and a very low failure rate. 6. CONCLUSION This paper studies faithful interleaved generation over structured multimodal documents,where flat retrieval weakens dependencies among sections,local contexts, and visual evidence.HAM-RAG addresses this issue by incorporating multi-level document context into text and image representations, preserving source organization and local text-image relations across retrieval and generation. HAM- Bench further provides a unified benchmark across diverse structured-document scenarios. Experiments across datasets and backbones show consistent gains in image grounding, local text-image alignment, and holistic answer quality, confirming document hierarchy as a key grounding signal for coherent multimodal generation. These findings support reliable assistants for industrial SOPs, maintenance guides, and operational manuals, where answers must remain faithful to procedural structure and local text-image evidence. REFERENCES Burns, A., Srinivasan, K., Ainslie, J., Brown, G., Plummer, B. A., Saenko, K., Ni, J., and Guo, M. (2023) WikiWeb2M: A page-level multimodal wikipedia dataset. arXiv preprint arXiv:2305.05432. Caffagni, D., Cocchi, F., Moratelli, N., Sarto, S., Cornia, M., Baraldi, L., and Cucchiara, R. (2024) Wiki-LLaVA: Hierarchical retrieval-augmented generation for multimodal LLMs.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1818–1826. Chang, Y., Narang, M., Suzuki, H., Cao, G., Gao, J., and Bisk, Y. (2022) WebQA: Multihop and multimodal QA.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16495–16504, New Orleans, LA, USA. Chen, W., Hu, H., Chen, X., Verga, P., and Cohen, W. (2022) MuRAG: Multimodal retrieval-augmented generator for open question answering over images and text.In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5558–5570, Abu Dhabi, United Arab Emirates. Cho, J., Mahata, D., Irsoy, O., He, Y., and Bansal, M. (2025) M3DocVQA: Multi-modal multi-page multi-document understanding.In 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 6237–6247. Dong, K., Chang, Y., Huang, S., Wang, Y., Tang, R., and Liu, Y. (2025) Benchmarking retrieval-augmented multimodal generation for document question answering. In Advances in Neural Information Processing Systems, volume 38. Datasets and Benchmarks Track. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K ̈uttler, H., Lewis, M., Yih, W.-t., Rockt ̈ aschel, T., Riedel, S., and Kiela, D. (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459–9474. Li, Y., Li, W., and Nie, L. (2022) MMCoQA: Conversational question answering over text, tables, and images.In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4220–4231, Dublin, Ireland. Ma, Z.-A., Lan, T., Tu, R.-C., Hu, Y., Zhu, Y.-S., Zhang, T., Huang, H., Wu, Z., and Mao, X.-L. (2024) Multi-modal retrieval augmented multi-modal generation:Datasets, evaluation metrics and strong baselines. arXiv preprint arXiv:2411.16365. Wang, B., Xu, C., Zhao, X., Ouyang, L., Wu, F., Zhao, Z., Xu, R., Liu, K., Qu, Y., Shang, F., Zhang, B., Wei, L., Sui, Z., Li, W., Shi, B., Qiao, Y., Lin, D., and He, C. (2024) MinerU: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839. Yu, Q., Xiao, Z., Li, B., Wang, Z., Chen, C., and Zhang, W. (2025a) MRAMG-Bench: A comprehensive benchmark for advancing multimodal retrieval-augmented multimodal generation.In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3616–3626. Yu, S., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y., Liu, Z., Wang, S., Han, X., Liu, Z., and Sun, M. (2025b) VisRAG: Vision-based retrieval-augmented generation on multi-modality documents. In The Thirteenth International Conference on Learning Representations (ICLR). Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2020) BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. Zhang, Y. and Wu, Y. (2026) MLDocRAG: Multimodal long-context document retrieval augmented generation. arXiv preprint arXiv:2602.10271. Zhu, Z., Lee, D., Zhang, H., Sree Harsha, S., Feujio, L., Maharaj, A., and Li, Y. (2025) MuRAR: A simple and effective multimodal retrieval and answer refinement framework for multimodal question answering.In Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations, pages 126–135, Abu Dhabi, UAE.