Paper deep dive
PromptDLA: A Domain-aware Prompt Document Layout Analysis Framework with Descriptive Knowledge as a Cue
Zirui Zhang, Yaping Zhang, Lu Xiang, Yang Zhao, Feifei Zhai, Yu Zhou, Chengqing Zong
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/13/2026, 1:02:11 AM
Summary
PromptDLA is a domain-aware framework for Document Layout Analysis (DLA) that addresses performance degradation caused by merging heterogeneous datasets. It utilizes a domain-aware prompter to inject descriptive knowledge as cues into a transformer-based backbone, enabling the model to adapt to varying document types, languages, and labeling styles. The framework achieves state-of-the-art performance across multiple benchmarks and introduces MLDLA, a new multilingual dataset.
Entities (9)
Relation Signals (3)
PromptDLA → introducesdataset → MLDLA
confidence 100% · we have introduced a multilingual DLA dataset—MLDLA
PromptDLA → improvesperformanceon → DocLayNet
confidence 95% · Extensive experiments show that our proposal achieves state-of-the-art performance among DocLayNet
PromptDLA → usesmodel → Llama
confidence 90% · PromptDLA integrates domain priors directly into the analysis process using LLMs like LLAMA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Document Layout Analysis (DLA) is crucial for document artificial intelligence and has recently received increasing attention, resulting in an influx of large-scale public DLA datasets. Existing work often combines data from various domains in recent public DLA datasets to improve the generalization of DLA. However, directly merging these datasets for training often results in suboptimal model performance, as it overlooks the different layout structures inherent to various domains. These variations include different labeling styles, document types, and languages. This paper introduces PromptDLA, a domain-aware Prompter for Document Layout Analysis that effectively leverages descriptive knowledge as cues to integrate domain priors into DLA. The innovative PromptDLA features a unique domain-aware prompter that customizes prompts based on the specific attributes of the data domain. These prompts then serve as cues that direct the DLA toward critical features and structures within the data, enhancing the model's ability to generalize across varied domains. Extensive experiments show that our proposal achieves state-of-the-art performance among DocLayNet, PubLayNet, M6Doc, and D$^4$LA. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.09414v1
- Canonical: https://arxiv.org/abs/2603.09414v1
Trouble viewing inline? Open PDF directly →
Full Text
66,816 characters extracted from source content.
Expand or collapse full text
JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20151 PromptDLA: A Domain-aware Prompt Document Layout Analysis Framework with Descriptive Knowledge as a Cue Zirui Zhang * ,Yaping Zhang * , Lu Xiang, Yang Zhao, Feifei Zhai, Yu Zhou and Chengqing Zong, Fellow, IEEE Abstract—Document Layout Analysis (DLA) is crucial for document artificial intelligence and has recently received in- creasing attention, resulting in an influx of large-scale public DLA datasets. Existing work often combines data from various domains in recent public DLA datasets to improve the generaliza- tion of DLA. However, directly merging these datasets for training often results in suboptimal model performance, as it overlooks the different layout structures inherent to various domains. These variations include different labeling styles, document types, and languages. This paper introduces PromptDLA, a domain-aware Prompter for Document Layout Analysis that effectively leverages descriptive knowledge as cues to integrate domain priors into DLA. The innovative PromptDLA features a unique domain- aware prompter that customizes prompts based on the specific attributes of the data domain. These prompts then serve as cues that direct the DLA toward critical features and structures within the data, enhancing the model’s ability to generalize across varied domains. Extensive experiments show that our proposal achieves state-of-the-art performance among DocLayNet, Pub- LayNet, M6Doc, and D 4 LA. Our code is available on GitHub. 1 Index Terms—Document Layout Analysis, PromptDLA, Doc- ument Understanding I. INTRODUCTION Document Layout Analysis (DLA) aims to distinguish the physical or logical layout structure of documents [1]– [6], identifying areas characterized by elements such as text, image, and table. It is fundamental to modern document artificial intelligence, significantly influencing subsequent doc- ument understanding tasks such as information extraction and digital transformation [7]–[13]. With growing research interest in DLA, large-scale datasets have emerged, such as PubLayNet [14], DocBank [15], Do- cLayNet [16], M6Doc [17] and D 4 LA [18]. To enhance gener- alizability in real-world scenarios, recent DLA datasets such as DocLayNet [16], M6Doc [17], and D 4 LA [18] have increased document diversity by combining data from various domains, Y. Zhang, L. Xiang, Y. Zhao, F. Zhai, Y. Zhou and C. Zong are with the Institute of Automation, Chinese Academy of Sciences, and the University of the Chinese Academy of Sciences. Y. Zhou is also with the Fanyu AI Laboratory, Zhongke Fanyu Technology Co., Ltd. Z. Zhang is with Columbia University; this work was done while at the Fanyu AI Laboratory. Email: z3093@columbia.edu, yaping.zhang, lu.xiang, yang.zhao, yzhou, cqzong@nlpr.ia.ac.cn. Y. Zhang is the corresponding author. Y. Zhang and Z. Zhang contributed equally and are co-first authors. This work is supported by the National Natural Science Foundation of China under Grant 62476275 and 62106265, and by the Young Scientists Fund of the State Key Laboratory of Multimodal Artificial Intelligence Systems. 1 https://github.com/Zirui00/PromptDLA.git (a) Different document type (b) Language variations Financial reports Persian KazakhstanVietnamese ManualsPatents DoclaynetDocbank Text List Publaynet (c) Inconsistent labeling styles in different dataset Fig. 1.Examples of different domain differences across (a) Different document types caused variations in layout structure and element distri- bution (financial report, manual, patent) (b) different language types, and (c)Inconsistent labeling styles. Note that the ”text” and ”list” items in DocLayNet are labeled as smaller individual units while they are integrated as a whole in DocBank. including finance, law, and patents. However, merging data from these diverse domains introduces substantial distribution differences, both across and within datasets. Fig. 1 illustrates three critical yet overlooked domain differences encountered: • Different document types. Document images from dif- ferent types exhibit unique layout structures and label distributions. Fig. 1(a) visually contrasts the typical vi- sual features of three document types: Financial reports (left) are characterized by well-designed color images and text-image overlays; manuals (center) frequently use functional screenshots of software interfaces to present data and controls; while patents (right) are dominated by structured, black-and-white technical line drawings with concise labels. These significant differences in layout highlight the need to incorporate domain priors into DLA. • Different languages. Document images from various countries exhibit unique layout structures influenced by their respective languages. Fig. 1(b) demonstrates how language affects the layout of the document. Persian documents predominantly feature dense blocks of text ar- ranged in continuous paragraphs. In contrast, documents from Kazakhstan integrate numerous small paragraphs interspersed with images, creating a visually diverse page layout. These variations highlight the correlation between language and document layout, as shown in Fig. 1. • Inconsistent labeling styles. Different datasets often adopt disparate annotation guidelines leading to conflict arXiv:2603.09414v1 [cs.CV] 10 Mar 2026 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20152 Document Image x Image Embedding Domain-Aware Prompter Detection Head Transformer Encoder Layer Prompted Transformer Encoder Descriptive Domain information d 퐿 1 Transformer Encoder Layer 퐿 8 Transformer Encoder Layer 퐿 6 Transformer Encoder Layer 퐿 4 Transformer Encoder Layer 퐿 12 ... ... ... ... Domain-Aware Prompter “Financial” 풗 2 1 풗 푖 1 풗 1 1 ... 풗 푀 1 L(V)LM Text Encoder 푃 푣 ... Image Embedding Transformer Encoder Layer Transformer Encoder Layer Pretrained Transformer Encoder 퐿 1 퐿 2 Transformer Encoder Layer 퐿 8 Transformer Encoder Layer 퐿 6 Transformer Encoder Layer 퐿 4 Transformer Encoder Layer 퐿 12 ... ... ... ... Document Image x Domain Prompt Generator Fusion Layer 풑 푣 풗 2 1 풗 푖 1 풗 1 1 ... 풗 M 1 ... 풑 푣 1 푹 풍풐풄 x Descriptive Domain information d 푹 풄풍풔 풗 2 2 풗 푖 2 풗 1 2 풗 푀 2 ... ... Fusion Layer 푃 푣 2 푝 푣 2 풗 2 2 풗 푖 2 풗 1 2 풗 M 2 ... ... 풗 2 2 풗 푖 2 풗 1 2 풗 푚 2 ... ... 풗 2 2 풗 푖 2 풗 1 2 풗 푚 2 ... ... Fig. 2. Overview of the PromptDLA method for domain-aware layout prediction. A Domain-Aware Prompter encodes domain information into a prompt vector, which is prepended to the sequence of image patch embeddings. This augmented input is processed by a vision backbone. Multi-scale features are extracted from the backbone and refined by a FPN before being passed to a detection head for final layout prediction. Image Encoder Pretraining Head Image Encoder Detection Head ImageImage (b) PromptDLA(a) Pretraining without Prompt Image Encoder Pretraining Head Prompted Image Encoder Detection Head Domain Prompter Image Descriptive Domain Knowledge Image weight initiliaztion Fig. 3. Comparison with Pre-training Paradigms in DLA labeling style, even for semantically similar elements. As shown in Fig. 1(c), DocLayNet annotates individual list items, whereas DocBank and PubLayNet group entire lists into single bounding boxes. Similarly, paragraph segmentation varies significantly across datasets. Such inconsistencies create conflicts during joint training and pose obstacles to building scalable, unified models. These overlooked domain-related discrepancies can hinder the learning process and reduce the generalization capability of DLA models trained on combined datasets.A promising direc- tion to mitigate these issues involves endowing DLA models with the capacity to adapt their analysis based on the specific characteristics of the input document’s domain. Recent studies in large vision-language models (LVLMs) like CLIP [19] and large language models (LLMs) like LLaMA [20] have demonstrated the efficacy of prompt engineering for condi- tioning model behavior on domain-specific contexts across various tasks [21]–[26]. Inspired by this, we propose a novel framework for Domain-aware Prompt Document Layout Analysis, named PromptDLA. Unlike traditional pretraining- based DLA methods, PromptDLA integrates domain priors di- rectly into the analysis process using LLMs like LLAMA [20] or LVLMs such as CLIP [19], BLIP2 [27]. Central to our approach is a prompted transformer encoder, fine-tuned with a novel domain-aware prompter (depicted in Fig. 2). This prompter uses descriptive knowledge from the domain infor- mation of corresponding images as cues, guiding the trans- former encoder to recognize and adapt to the variability across different domains effectively. We evaluate our method on the DocLayNet, M6Doc, D 4 LA, and the integration of PubLayNet and DocLayNet datasets. The experimental results have shown that the domain-aware model has outperformed the current state-of-the-art method. In addition, due to the predominance of English in existing datasets, we have introduced a mul- tilingual DLA dataset—MLDLA, which contains document images in seven different languages. Detailed experiments have demonstrated that our method can effectively generalize across scenarios where language serves as domain information. The contributions of this paper are summarized as follows: • Anoveldomain-awareDLAframework,named PromptDLA, is proposed that explicitly introduces domain knowledge to DLA, enabling models to better handle variability across a variety of document domains. • A unique and modular domain-aware prompter is pro- posed to generate customized prompts reflecting specific data attributes. It integrates easily with various backbone architectures, such as CNNs, ViTs, and Swin Transform- ers, using prompts derived from either human knowledge or LLMs generation. • We conduct extensive experiments demonstrating the ef- fectiveness of PromptDLA across multiple domain infor- mation types and datasets, including the newly introduced JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20153 TABLE I THE DETAIL OF DOCUMENT TYPES IN DIFFERENT DLA DATASETS. DatasetDocument TypeA.M.Format#ClassLanguage#Images PubLayNet [14]ArticlesAutomaticPDF5English364232 DocLayNet [16] Financial Reports, Manuals, Scientific Articles, Laws & Regulations, Patents, Government Tenders. ManualPDF11 English, German, French, Japanese 80863 M6Doc [17] Scientific articles, Textbooks, Books, Test papers, Magazines, Newspapers, Notes ManualPDF, Scanned, Photographed74English,Chinese9080 D 4 LA [18] Scientific report, Email, Form, Invoice, Letter, Specification, News article, Presentation, Resume, Scientific publication, Budget, Memo ManualPDF, Scanned,27English11092 MLDLA(ours) Magazine, Newspaper, Government Reports ManualPDF, Scanned, Photographed5 Hindi, Kazakhstan, Vietnam, Turkey, Persia, Laos, Khmer 17505 MLDLA benchmark, achieving state-of-the-art results. I. RELATED WORK A. Document Layout Analysis The current literature on document analysis has redefined document understanding as a broad term that covers various problems and tasks related to document intelligence systems. Based on whether pre-training on large-scale unlabeled doc- ument images can divide the Document Layout Analysis method into two categories: traditional object detection frame- works and document pre-training methods. Traditional object detection frameworks, such as Faster-RCNN [28], Mask- RCNN [29], and YOLO [30], typically train models directly on DLA datasets and may occasionally use ImageNet [31] pre-trained weights. In comparison, another approach involves training the transformer using the self-supervised method on a vast unlabeled document dataset and utilizing the pretrained transformer as the backbone of a two-stage Object Detection Framework. LayoutLMv3 [32] and Structextv2 [33] are critical works in this area. It employs the multi-modal pre-training method, including Mask Image Modeling, Mask Language Modeling, and Word-Patch Alignment, using both the textual information on the image and the image itself as input to the transformer encoder. Other notable works in this area are DiT [34], DocFormer [35], UniDoc [36], and Self-Docseg [37], which relies solely on a vision model, closely aligning with the approach of BEiT [2], and directly applies a general pre- training framework to learn from large-scale document image data. Traditional approaches focus on enhancing the model’s performance. As depicted in Fig. 3, PromptDLA diverges from conventional pre-training methods by directly injecting domain knowledge into the DLA framework. Instead of implicitly learning domain features, our approach leverages descriptive prompts from LLMs, offering a more flexible and resource- efficient paradigm for domain adaptation. B. Prompt Engineering Prompt engineering has significantly improved the adapt- ability of LLMs and LVLMs for specialized tasks. By in- troducing learnable tokens into vision transformers (ViTs), models can effectively focus on task-relevant features with minimal structural changes [38]. This strategy is similar to ` (b)Domain-Heuristic Prompt Generator A doc ument page of [] A page come s from [*] A piece of paper comes from [*] A doc ument page about [*] ... Financial Reports Prompt Sets Image Human knowledge Output Rule-based Selector A document page comes from Financial reports ` (a) LVLM-based Prompt Generator Image Output Thedocumentimageisafinancialstatement, whichisareportthatprovidesinformationa boutacompany’sfinancialperformance,pos ition,andcashflow. Prompt : Please describe the type of this document image ` Image Output LVLM The financial report layout features tables at the top and bar graphs in the middle and bottom, showcasing key financial and operational data across two years. The design employs a clean, industrial-themed visual style with a blue and gray color scheme. Human knowledge (c) Hybrid Knowledge-Augmented Prompt Generator Prompt: Please summar ize the layout of this “financial report” in a few sentences. Financial Reports LVLM Fig. 4. Framework of Prompt Generator. advances in multimodal learning, exemplified by CLIP, which jointly leverages text and image data to enhance robustness and versatility across various visual recognition tasks, particularly in zero-shot or few-shot settings [19]. Furthermore, prompt engineering more generally utilizes carefully designed prompts to effectively elicit contextually appropriate outputs from pre-trained models, greatly reducing the need for extensive retraining [39]. I. METHODOLOGY Document layout analysis significantly depends on the data domain. A model that accurately recognizes domain-specific characteristics can adapt its outputs based on tailored prompts, efficiently handling variations across diverse document do- mains. To facilitate this, we introduce a domain-aware prompter that enables the model to identify the domain of the input data. As illustrated in Fig. 2, our model is composed of four main components: an Image Embedding ModuleF patch , a Domain- aware PrompterF prompter , a Prompted Transformer Encoder F encoder , and a Detection HeadF detect . Given an input document image x ∈R C×H in ×W in , the Image Embedding Module extracts patch embeddings as visual JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20154 tokens: v 1 , v 2 ,..., v M =F patch (x)(1) where v i M i=1 represents the sequence of visual tokens. Currently, the Domain-Aware Prompter generates a domain- specific prompt embedding p v : p v =F prompter (x,d)(2) where d represents descriptive knowledge as explicit domain information, provided by either a human expert or an LLM, as detailed in Fig. 4. Both the visual tokens and the prompt embedding are then processed by the Prompted Transformer Encoder: f 1 , f 2 ,..., f L =F encoder (v 1 , v 2 ,..., v M , p v )(3) yielding multilevel feature maps f l L l=1 .Finally, a subset of these feature maps is passed to the Detection Head. For instance, when using a transformer-based backbone, features from the 4th, 6th, 8th, and 12th layers are selected to predict the layout structure ˆ y: ˆ y =F detect (f 4 , f 6 , f 8 , f 12 )(4) where ˆ y denotes the predicted refined bounding box ˆ b and a class label ˆ l. A. Image Embedding We follow the patch embedding approach used in ViT [40] for image embedding. Specifically, the document image x is resized to H × W , represented as x ∈R C×H×W , where C, H , and W denote the channel, height, and width, respectively. The resized image is then divided into a sequence of non- overlapping patches of size P × P . Each patch is linearly projected into a D-dimensional vector viaF patch , as defined in Eq. 1, resulting in a flattened sequence: v = [v 1 , v 2 ,..., v i ,..., v M ],where v i ∈R D .(5) Here, the sequence length M is determined as M = HW/P 2 . Finally, learnable 1D positional embeddings are added to each vector to encode spatial information. B. Domain-Aware Prompter As shown in Fig. 2, the Domain-aware PrompterF prompter consists of two components: a Text EncoderF t and a Prompt GeneratorF g . 1) Prompt GeneratorF g : The Prompt GeneratorF g is designed to dynamically produce natural language prompts pertinent to the document’s domain. As illustrated in Fig. 4, F g supports three distinct operational modes, ranging from direct utilization of curated human knowledge to automated generation using large models.: • LVLM-based Prompt Generator: This mode leverages the generative capabilities of LVLMs, such as variants based on LLaMA-Adapter [41]. The input consists of the document image x paired with a general instruction (e.g., “Please describe the type of this document” or “Describe the primary use of this document”). The LVLM analyzes the image content and generates a textual description ` ` ` ` ` ... 풗 푖+2 풗 2푖 풗 푖+1 ... ... ... 풗 푗+1 풗 푀 풗 푗 ... ... MLP 푃 푣 Flatten ... 풗 2 풗 푖 풗 1 풊=푾/푷 ... 풗 2 풗 푀 풗 1 푃 푣 1 풗=풗 풊 풊=ퟏ 푴 ,풗 풊 ∈ℝ ퟏ×푫 Window Window W indow W indow 푃 푣 1 풗=푣 푖 푖=1 i+2 , 풗 풊 ∈ℝ ퟏ×푫 ... 풗 푖+2 풗 2푖 풗 푖+1 ... ... ... 풗 푗+1 풗 푀 풗 푗 ... ... ... 풗 2 풗 푖 풗 1 ... 풗 2 풗 i+2 풗 1 W indow 풗 푖+2 풗 푖+1 풗 2 풗 1 Flatten 푃 푣 2) Concat prompt vector by windows 푃 푣 푃 푣 ... 푃 푣 푃 푣 ... MLP 1) Replicate by 퐭퐡퐞 퐧퐮퐦퐛퐞퐫 퐨퐟 퐰퐢퐧퐝퐨퐰퐬 ... 풗 푖+2 풗 2푖 풗 푖+1 ... ... ... 풗 푗+1 풗 푀 풗 푗 ... ... MLP Flatten ... 풗 2 풗 푖 풗 1 풊=푾 풗∈ℝ C×퐻×푊 j = 푯 푃 푣 풑 풗 풑 ∈ℝ ퟏ×푯×푾 padding Fe ature map Fe ature map Fe ature map ∈ℝ ퟏ×풅풊풎 (풅풊풎<푯×푾) (c) Fusion Layer in Swin-Transformer-based backbone (b) Fusion Layer in Vit-based backbone (a) Fusion Layer in CNN-based Backbone j = 푯 / 푷 j = 푯 / 푷 풊=푾/푷 MSA Concat W - MSA Concat Concat Conv 푷 풗 풑 푷 풗 풑 푷 풗 풑 Fig. 5. Framework of Fusion Layer. capturing the inferred document domain and characteris- tics. This approach offers high automation and zero-shot potential but incurs significant computational overhead. • Domain-Heuristic Prompt Generator: This mode re- lies on curated human knowledge. It utilizes predefined ‘Prompt Sets‘ containing various sentence templates de- signed to encapsulate domain information. Given an explicit domain class (e.g., ’invoice’, ’scientific paper’), relevant templates are selected. We employ a rule-based selector, potentially augmented by CLIP’s zero-shot clas- sification capability, to refine template selection. For instance, multiple templates can be instantiated with specific document type names (e.g., from DocLayNet ), embedded using CLIP’s text encoder, and evaluated for their zero-shot classification accuracy on a relevant task. Templates yielding top-k performance are retained.This approach ensures that the generated prompts are inter- pretable and consistent with explicit rules and domain labels. • Hybrid Knowledge-Augmented Prompt Generator: This hybrid approach combines aspects previous two. It uses an LVLM but guides its generation with more specific, human-provided knowledge compared to the general instructions. For example, instead of asking the LVLM to infer the type, a prompt like “Please describe the typical layout elements found in a financial report document” is provided, potentially along with the image x or just the domain label d. This allows the LVLM to generate more precise and contextually relevant de- scriptions tailored to a known document type, balancing automation with targeted knowledge injection. 2) Text Encoder (F t ): The Text EncoderF t converts a natural language prompt describing the document domain into a fixed-dimensional embedding that provides domain-specific guidance to the layout analysis model. This Encoder is instan- tiated using a powerful pre-trained language model, either a JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20155 Algorithm 1: The PromptDLA Algorithm. Require: D(x (n) ,y (n) ) N n=1 , y consists of target bounding boxes ˆ b and class labels l; Ensure: Prediction of layout 1: InitializeF t ← Θ llm or Θ lvlm ,F e ← Θ pretrain e 2: Prompt GeneratorF g 3: while t ≤ max iteration do 4:(v 0 1 ,v 0 2 ,...,v 0 m )←F patch (x) 5: prompt←F g (x) 6: p v ←F t (prompt) 7:for i=1; i ≤ layer nums do 8:f i ← (v i 1 ,v i 2 ,...,v i m ) 9:in i ←F fuse (f i ;F i mlp (p v )) 10:(v i+1 1 ,v i+1 2 ,...,v i+1 m ;p)←F i e (in i ) 11:end for 12: f ←F fpn (f 4 ,f 6 ,f 8 ,f 12 ) 13: Loss← R loc (F detect (f,b), ˆ b) + λR cls (F detect (f ),l) 14:freezeF t 15:updateF fuse ,F e ,F detect 16: t← t + 1 17: end while text encoder from a vision-language model (θ lvlm , e.g., CLIP, BLIP2) or a large language model (θ llm , e.g., LLaMA [41]). To preserve the rich semantic knowledge acquired during pre-training, theF t weights are frozen during training. The resulting embedding, denoted as p v ∈R D prompt , serves as the domain-aware input for subsequent modules. C. Prompted Transformer Encoder The Prompted Transformer EncoderF encoder integrates the visual tokens v i M i=1 with the domain prompt embedding p v . It typically comprises a standard Transformer Encoder backbone and dedicated Fusion Layers (F fuse ) responsible for injecting the prompt information. As shown in Fig. 5, We explore compatibility with CNN, ViT and Swin Transformer [42] architectures, requiring slightly different fusion strategies: • CNN-based Encoder : As illustrated in Fig. 5(a), for a CNN backbone (e.g., ResNet-50 [43]), the prompt embedding p v is first projected to a target dimension (e.g., 512 or 768) using an MLP layer. This projected prompt vector is then spatially padded to match the height H and width W of the feature map F (i) at a specific layer or stage i. The resulting projection tensor p p v , now having dimensions 1 × H × W , is concatenated channel-wise with the feature map F (i) ∈R C×H×W . In our imple- mentation using ResNet-50 [43], this fusion operation is performed at the input of each of the four main residual stages, where the spatially expanded prompt features are concatenated with the stage’s input feature map. • ViT-based Encoder: As depicted in Fig. 5(b), for a ViT backbone, the fusion layerF fuse typically em- ploys an MLP (F mlp ) to project the prompt embedding p v to match the visual token dimension D. The pro- jected prompt embeddings, denoted p 1 v , are then concate- nated with the sequence of position-aware visual tokens ̃ v i M i=1 (where ̃ v i = v i + e i , incorporating patch em- bedding v i and positional embedding e i ). This combined sequence is the input to the standard Transformer Encoder layers (F e ). • Swin Transformer-based Encoder: For the hierarchi- cal Swin Transformer backbone, illustrated in Fig. 5(c), prompt fusion requires adaptation to its windowed at- tention and shifting window mechanisms [42]. Inspired by approaches like VPT [44], the prompt embedding p v is processed by stage-specific MLPs within the fusion moduleF fuse to generate dimension-matched embed- dings p (1) v for each stage. Within a stage, p v is typically replicated N win times (where N win is the number of win- dows in that stage) and combined with the token sequence of each respective window before the windowed self- attention (W-MSA) computation. Appropriate masking or padding is applied to ensure dimensional consistency during attention calculations. The outputs from multiple stages (e.g., corresponding to features at 1/8, 1/16, 1/32 resolutions) are then typically fed into an FPN (F fpn ) to generate multi-scale features for the detection head. The output ofF encoder (potentially viaF fpn ) is a set of feature maps f 1 , f 2 ,..., f L capturing both visual content and domain-specific context. D. Detection Head The Detection HeadF detect takes the contextualized fea- tures from the encoder and performs the final layout element prediction, including bounding boxes and class labels. Our framework is designed to be compatible with two detection head architectures: • RCNN-based Heads: Standard two-stage detection frameworks like Faster R-CNN [28], Mask R-CNN [29], or Cascade R-CNN [1] can be readily employed. In this setup, the R-CNN detection head operates on the generated feature maps by the prompted transformer encoder, optimizing bounding box regression and classi- fication objectives using standard loss functions. Specif- ically, the bounding box regression loss R loc aims to minimize the discrepancy between predicted boxes b i = (b ix ,b iy ,b iw ,b ih ) derived from proposals ˆ b i and features f i , and ground-truth boxes ˆ b i : R loc = N X i=1 L loc (r(f i , b i ), ˆ b i ),(6) where r(·) is the regression function, N is the number of proposals, and L loc is typically the smooth L 1 loss. For Classification, a classifier c(·) assigns feature map patches to one of the classes, the classification loss R cls is defined as: R cls = N X i=1 L cls (c(f i ), l i ),(7) where f i and l i denote the i-th object feature and class label, respectively. And L cls is usually the cross-entropy loss. The total loss is a weighted sum: R total = R loc + λR cls .(8) • DETR-based Heads: Our architecture also supports inte- gration with DETR [45] and its variants. DETR employs an encoder-decoder structure where a set of learnable object queries interacts with the image features (from JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20156 TABLE I COMPARISON WITH STATE-OF-THE-ART METHODS ON DOCLAYNET. OUR METHOD USES DOMAIN PROMPTS FROM HUMAN KNOWLEDGE AND PROMPT ENGINEERING, WITH A VIT BACKBONE, CLIP TEXT ENCODER, AND CASCADE MASK R-CNN DETECTION HEAD. ALL BASELINE RESULTS ARE TAKEN DIRECTLY FROM PREVIOUS PAPERS. MethodPretrainingCaptionFootnoteFormulaList-itemPage-footerPage-headerPictureSection-headerTextTableTitlemAP Mask R-CNN [17]✗71.571.863.480.859.370.072.769.382.985.880.473.5 Faster R-CNN [17]✗70.173.763.581.058.972.072.068.482.285.479.973.4 YOLOv5 [17]✗77.777.266.286.261.167.977.174.686.388.182.776.8 TransDLANet [17]✗68.274.761.681.054.868.268.569.882.483.881.772.3 SwinDocSegmenter [5]✗83.664.862.382.365.166.484.766.587.488.263.376.9 SelfDocSeg [5]✓-----------74.3 LayoutLmV3 [32]✓73.177.569.079.861.361.374.069.086.385.984.475.7 DiT [34]✓75.076.268.183.562.174.074.571.286.486.683.076.4 PromptDLA(ViT, CLIP, Cascade)✓76.683.072.484.963.876.975.273.887.187.984.178.7 PromptDLA(ResNet, CLIP, DETR)✓91.884.154.769.482.939.582.087.370.383.091.377.7 PromptDLA(SwinTran, CLIP, DETR)✓92.585.557.671.384.240.283.188.573.183.892.179.6 F encoder , i.e., f l L l=1 ) via cross-attention mechanisms within the decoder. Feed-forward networks (FFNs) then directly predict the class and bounding box coordinates from the updated object queries. DETR is optimized end-to-end using a set-based bipartite matching loss that jointly considers classification and localization costs (e.g., cross-entropy for class, and a combination of L 1 and GIoU loss for boxes). This flexibility allows leveraging the strengths of different detection paradigms within our PromptDLA framework. IV. EXPERIMENTS This section validates PromptDLA’s effectiveness through three sets of experiments. First, we examine the efficacy of PromptDLA compared to the state-of-art methods. Next, we evaluate our approach’s generalization ability. Lastly, we perform extensive ablation studies and discussions on the model’s design. A. Experimental Settings Datasets. We conduct extensive experiments to validate the proposed PromptDLA on 5 DLA benchmark datasets, includ- ing different document types, different languages, and different layer styles, • PubLayNet [14] consists of 5 typical document layout elements: text, heading, list, graphic, and table. It contains over 364232 page samples, where the annotations were automatically generated by matching PDFs and XML formats of articles from the PubMed Central Open Access subset. • DocLayNet [16] contains 6 document types (Financial Reports, Manuals, Scientific Articles, Laws & Regula- tions, Patents, and Government Tenders) with 11 cate- gories of annotations across four languages (with English documents comprising approximately 95%). The dataset contains about 80,863 manually annotated pages. • M6Doc [17] contains a total of 9, 080 modern document images, which are categorized into 7 document types (Scientific articles, Textbooks, Books, Test papers, Mag- azines, Newspapers, Notes) with 74 detailed categories. • D 4 LA [18] contains a total of 11092 document images, which includes 12 diverse document types (Scientific report, Email, Form, Invoice, Letter, Specification, News article, Presentation, Resume, Scientific publication, Bud- get, Memo) with 27 detailed categories. • MLDLA is a Multi-Language DLA (MLDLA) dataset we constructed to evaluate the model generalization on more different languages. It comprises 175, 000 images, which are manually labeled through a uniform labeling style, including 7 in different languages, such as Persian, Khmer, Kazakh, Lao, Turkish, Hindi, and Vietnamese. Evaluation Metric. Our experiments are evaluated using the category-wise and overall mean average precision (mAP) @IOU[0.50:0.95] of bounding boxes following the litera- ture [1]. This curve describes the relationship between preci- sion and recall and is the most widely used evaluation metric for document layout analysis. Implementation Details. We train our model on 8 3090 GPUs with a batch size 16 using a cosine learning rate schedule and a warm-up strategy with a 0.01 warm-up factor. We set the basic learning rate to 2e-4. Additionally, we adopt the AdamW optimizer. Our study uses DiT [34] pre-trained weights and the Cascade-RCNN [1] detection head as our baseline method. The detailed training process is shown in Algorithm 1. B. Generalization Ability Generalization on Different Document Domains. We evalu- ate the performance of PromptDLA on different DLA datasets with more diverse document domains in Table VIII. The results show that the promptDLA can get consistent improvements on datasets with other domains, such as DocLayNet with 6 domains (2.3% over DiT), M6Doc with 7 domains( 2.0% over DiT), and D4LA (1.4% over DiT), validating the generaliza- tion of PromptDLA on different document types. Generalization on Multi-Language Datasets. We inves- tigate the effects of domain-aware prompts by using dif- ferent language types as domain information based on the MLDLA dataset. Firstly, we validate that CLIP can provide prior knowledge even for documents in different languages, including minority languages, through a zero-shot document classification task. More specifically, we insert the document’s language into a prompt template as text and utilize CLIP to identify similarities between it and the associated image. Our experiments indicate that CLIP achieves a zero-shot classification accuracy of 47.53% across seven languages in MLDLA. Furthermore, we apply PromptDLA to MLDLA JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20157 and present a comparison in Table I. Following the method described in Section 3.2 of the paper to construct prompt sets. Our approach improves precision by +1.0 compared to the DiT model without a prompt. DiT without a prompt achieves higher mAPs in categories like ”List,” which are less domain-relevant. We think that it may lack sufficient domain- specific information for accurate detection. On the contrary, our PromptDLA significantly improves other domain-specific details, such as ”Fig.” (from 54.5% to 57.3%) and ”Table” (from 76.1% to 77.9%). TABLE I EXPERIMENTS ON A REAL MULTI-LANGUAGE DATASET. MethodTextTitleFigureListTablemAP DiT77.958.654.575.676.168.5 PromptDLA78.759.157.374.577.969.5 ∆+0.8+0.5+2.8 −1.1+1.8+1.0 Generalization on Inconsistent Labeling Style. We inves- tigate the impact of inconsistent labeling styles. We perform experiments to jointly train different datasets to enhance the model’s performance in practical applications. However, we encounter challenges due to the inconsistent labeling styles observed in public Document Layout Analysis (DLA) datasets, particularly between DocLayNet and PubLayNet. As depicted in Fig. 6, the labeling styles of PubLayNet and DocLayNet exhibit notable differences. While PubLayNet’s image and table align with DocLayNet’s counterparts, PubLayNet’s text corresponds to the set of DocLayNet’s caption, footnote, and text. Similarly, PubLayNet’s title matches the set of Do- cLayNet’s title and section header. Notably, PubLayNet’s list and DocLayNet’s list items differ, with PubLayNet integrating multiple list items as a whole list, while DocLayNet labels each list item separately. Additionally, PubLayNet omits page- footer, page header, and formula, whereas DocLayNet includes these elements. To address these differences, we perform label mapping, which aligns DocLayNet labels with PubLayNet and retains the page footer, page header, and formula. As shown in Table IV, joint training of the datasets, even after label mapping, does not improve performance on DocLayNet. Instead, there is a decrease in performance attributed to annotation conflicts. To overcome this issue, we introduce domain prompts, observing a consistency improvement on both DocLayNet and PubLayNet. This confirms the model’s adaptive learning capability to handle conflicts and effectively learn models tailored to the target domain. Notably, our method enhances mAP from 76.0 to 77.1 for DocLayNet and from 94.8 to 94.9 for PubLayNet. TABLE IV EXPERIMENTS ON INCONSISTENT LABELING STYLES. DocLayNetPubLayNet BaselinesJointPromptDLABaselinesJointPromptDLA Text86.987.0(+0.1)87.5(+0.6)94.494.5(+0.1)94.5(+0.1) Title71.272.1(+0.9)73.7(+2.5)88.989.4(+0.5)89.5(+0.6) List84.083.5(-0.5)84.4(+0.4)94.895.5(+0.7)95.6(+0.8) Table87.187.2(+0.1)87.7(+0.6)97.697.8(+0.2)97.8(+0.2) Figure 77.275.7(-1.5)76.3(-0.9)96.996.9(+0.0)97.0(+0.1) Formula68.469.1(+0.7)68.9(+0.5)--- Page-footer64.061.7(-2.3)64.1 (+0.1)--- Page-header74.072.1(-1.9)74.3 (+0.3)--- mAP76.676.0(-0.6)77.1 (+0.5)94.594.8(+0.3)94.9(+0.4) Label Mapping Caption Footnote Text List-item Section-header Title Picture Table Page-footer Page-header Formula Text List Title Figure Table Text List Title Figure Table Page-footer Page-header Formula (b) PubLayNet(a) DocLayNet(c)Joint Label Fig. 6. Inconsistent labeling relationship between DocLayNet and PubLayNet. Generalization on Out of Distribution Prompt. We explored the performance of PromptDLA in out-of-distribution (OOD) scenarios by splitting the DocLayNet dataset by document category, using ’manuals’ as the test set, and training on the remaining categories of documents. As shown in Table V, the prompt works effectively in OOD situations. TABLE V OOD RESULT ON MANUALS FROM DOCLAYNET TagMethodmAP∆ (a)DiT62.68 (b) DiT + Human knowledge64.23+1.55 (c)DiT + LVLM63.93+1.25 Generalization Across Different Backbone Architectures. To evaluate the adaptability and robustness of our proposed PromptDLA framework, we integrated and tested it with diverse backbone architectures commonly used in vision tasks. Specifically, we assessed its performance with a standard Vision Transformer (ViT-Base), a hierarchical Vision Trans- former (Swin-Transformer Base), and a widely adopted Con- volutional Neural Network (ResNet-50). For the Transformer- based models (ViT and Swin-Transformer), the domain-aware prompts were incorporated as detailed in *Figure*. For the CNN-based ResNet-50, the prompt embeddings were inte- grated by concatenating them with the pooled features be- fore the final classifier. The performance, measured by mean Average Precision (mAP), was evaluated on the DocLayNet dataset. Table VI compares these backbones with and without the PromptDLA module. PromptDLA consistently enhances the performance across all tested architectures. Notably, it improved +2.3 mAP points for ViT-Base, +1.0 mAP points for Swin-Base, and +0.7 mAP points for ResNet-50 compared to their respective baselines. These consistent gains underscore the versatility of PromptDLA, demonstrating that its effec- tiveness is not confined to a specific architectural paradigm and that it successfully leverages domain cues to benefit both Transformer and CNN models in document layout analysis. Generalization Across Different Detection Heads. We further evaluate the generalizability of PromptDLA by inte- grating it with two representative detection heads: Faster R- CNN and DETR, both using a ResNet-50 backbone. As shown in Table VII, PromptDLA improves mAP by +1.6 on Faster JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20158 TABLE VI PERFORMANCE COMPARISON OF DIFFERENT BACKBONE ARCHITECTURES WITH AND WITHOUT PROMPTDLA ON THE DOCLAYNET DATASET. ∆ INDICATES THE ABSOLUTE MAP IMPROVEMENT ACHIEVED BY ADDING PROMPTDLA COMPARED TO THE RESPECTIVE BASELINE MODEL TagMethodmAP∆ (a)ViT76.4 (b)ViT + PromptDLA78.7+2.3 (c)Swin78.7-0.12 (d)Swin + PromptDLA79.7+1.0 (e)ResNet50 (DETR)77.0 (f)ResNet50 + PromptDLA77.7+0.7 R-CNN and +0.7 on DETR, demonstrating its compatibility with both traditional and Transformer-based detectors. TABLE VII EFFECT OF PROMPTDLA ON DIFFERENT DETECTION HEADS. ∆ DENOTES THE ABSOLUTE MAP IMPROVEMENT. WE RE-IMPLEMENTED BASELINE MODELS FOR FAIR COMPARISON. TagMethodmAP∆ (a)Faster R-CNN (R50)66.8 (b)Faster R-CNN + PromptDLA68.4+1.6 (c)DETR (R50)77.0 (d) DETR + PromptDLA77.7+0.7 C. Comparison with State-of-the-arts We compare the proposed PromptDLA with (1) the DLA frameworks without pretraining: Faster-RCNN [28], Mask- RCNN [29], YOLOV5 [46], SwinDocSegmenter [5], and TransDLANet [17] and (2) the DLA framework with dif- ferent pretraining models: DiT [34], LayoutLMv3 [32], and SelfDocSeg [5]. We evaluate PromptDLA on the challenging DocLayNet dataset [16], which provides diverse document types with significant domain variations. In our experimental setup, we systematically use the document type as the domain prompts, enabling our model to adapt to specific document characteristics. Table I presents that our PromptDLA out- performs the DLA framework with and without pretraining methods. We can observe that the promptDLA outperforms state-of-the-art SwinDocSegmenter [5] with 1.8% mAP. Al- though SwinDoc gets better mAPs in a few rows like ”Table” and ”Picture,” we think these discernible categories are less relevant to the domain. Nevertheless, our PromptDLA exhibits substantial improvement over other domain-related detail cat- egories, such as ”Footnote” (from 64.8 to 83.0) and ”Section- Header” (from 66.4 to 76.9). D. Performance on Different Pretrained Models We assess the performance of PromptDLA upon differ- ent pre-trained DLA models and datasets, validating that PromptDLA is easily plugged to enhance different DLA frameworks. As shown in Table VIII, our method can be applied to different pre-trained frameworks, including Lay- outlmV3 and DiT. The performance of the PromptDLA is a further improvement on the pre-trained model. The stronger the performance of the pre-trained model, the better our method performed based on it. Excitingly, PromptDLA outper- forms state-of-the-art models such as SwinDocSegmenter [5], TABLE VIII PERFORMANCE OF THE PROMPTDLA WITH DIFFERENT PRE-TRAINED MODELS ON DOCLAYNET, M6DOC, AND D 4 LA, AND COMPARISON WITH THE CURRENT STATE-OF-THE-ART METHOD Model mAP@IOU[0.50:0.95] DocLayNetM6DocD 4 LA TransDLA [17]72.363.8- SwinDocSegmenter [5]76.9-- VGT [18]--68.8 LayoutLMv375.760.562.6 +PromptDLA76.4(+0.7)61.3(+0.8)63.1(+0.5) DiT76.467.267.7 +PromptDLA78.7(+2.3)69.2(+2.0)69.1(+1.4) Domain-Aware Prompter 1*d 푃 ! Financial reports Laws and regulations Patents Scientific articles Manuals Government tenders N domain Classes A document page of [*] A page comes from [*] A piece of paper comes from [*] A document page about [*] A document page comes from [*] ... M Prompt Sets (M>=N) A document page comes from Financial reports CLIP Text Encoder BLIP2 Text Encoder LLAMA2 Text Encoder ... Text Encoder Pre-trained Text Encoder Vision- Language model Language only Initialize Fig. 7. Different pre-trained text encoder for PromptDLA. TransDLA [17], and VGT [18] by 1.8%, 5.4%, and 0.3% on DocLayNet, M6Doc, and D 4 LA, respectively. Notably, compared to tailored models like SwinDocSegmenter and VGT, DiT with PromptDLA stands out for its simplicity and effectiveness. E. Different Prompter Different pre-trained text encoder for PromptDLA we delve deeper into the extent of prior knowledge that various large pre-trained models can provide about document images. We evaluate the performance of text encoders derived from different large pre-trained models, which fall into two main categories: those originating from vision-language pre-trained models and those from language-only pre-trained models. Specifically, we explore the capabilities of CLIP [19], BLIP2 [27], and LLAMA2 [20]. Notably, CLIP and BLIP2 are vision- language models, while LLAMA2 exclusively operates as a language model. As illustrated in Table IX, first, compared to the baseline model DiT without any domain-aware prompt, ”w/o Pre- trained Text Encoder” reduces the model’s accuracy by 0.1. This suggests that utilizing domain information but randomly initializing it without a pre-trained text encoder doesn’t effec- tively guide the model to differentiate between various domain documents. Conversely, using weights from pre-trained models improves performance for all three models, emphasizing a JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20159 solid correlation between document images in DLA datasets and their respective domains. It highlights the ability of the Text Encoder from Pre-trained models to provide valuable prior knowledge. TABLE IX EXPERIMENTS ON DIFFERENT PRE-TRAINED MODEL TEXT ENCODERS (W/O PRE-TRAINED TEXT ENCODER MEANS RANDOM INITIALIZING PROMPT WITHOUT USING A PRE-TRAINED TEXT ENCODER). mAP∆ DiT [34]76.4 w/o Pre-trained Text Encoder76.3 −0.1 CLIP Text Encoder [19]78.7+2.3 BLIP2 Text Encoder [27]79.0+2.6 LLAMA2 Text Encoder [20]77.8+1.4 Moreover, CLIP and BLIP2 outperform LLAMA2, indicat- ing that the Text Encoder from vision-language pre-trained models is superior to language-only pre-trained models. A vision-language pre-trained model can offer prior knowledge about both the relationship between document images and their corresponding text descriptions and general text rep- resentation. In contrast, a language-only pre-trained model only possesses knowledge concerning understanding human language. Furthermore, BLIP2 outperforms CLIP by 0.3 when using the same prompt, suggesting that BLIP2 can more accu- rately find the relationship between document images and their text descriptions. Consequently, a visual language pre- training model with superior performance could provide an even greater boost to my approach. In our paper, we uniformly use CLIP to explore the role of other modules, but we can replace it with BLIP2 or other superior visual language pre- training models for more accurate results. Analysis of Prompt Generation Strategies. We investigate different prompt generation strategies as illustrated in Fig. 4. As shown in Table X, we first compare human knowledge versus LVLM generation approaches. For real-world applica- tions where domain labels are unavailable, human-knowledge prompts require a preceding classification step. We trained a ViT-based domain classifier that achieves 90% accuracy on DocLayNet. Integrating this classifier (c) results in a negligible mAP drop of only 0.12 compared to using ground-truth labels (a), confirming the approach’s practical viability. In contrast, the LVLM-based generator (b) offers greater generality by eliminating the need for predefined categories, albeit at a higher computational cost. Despite this difference, its perfor- mance remains nearly identical to the human-knowledge ap- proach. Second, we examine hybrid prompts and their impact on prompt quality. To combine the strengths of both methods, our hybrid strategy (d) leverages human knowledge to guide the LVLM, as shown in Fig. 4(c). This approach performs best, improving the mAP by 0.33 over the baseline, demonstrating that high-level human knowledge can effectively steer the LVLM toward more relevant feature descriptions. Furthermore, we explored concatenating prompts from methods (a) and (b) simultaneously (e). This concatenation resulted in almost no improvement, underscoring that the quality and relevance of a single, well-formed prompt are more critical than prompt quantity. TABLE X ANALYSIS OF DIFFERENT PROMPT GENERATION STRATEGIES ON THE DOCLAYNET DATASET. THE ∆ COLUMN MEASURES THE MAP CHANGE RELATIVE TO THE BASELINE HUMAN-KNOWLEDGE PROMPT (A). TagMethodmAP∆ (a)Human-Knowledge Prompt (Baseline)78.69- (b)LVLM-Generated Prompt78.68-0.01 (c)Human-Knowledge Prompt w/ Classifier78.57-0.12 (d)Hybrid Prompt (Human-Guided LVLM)79.02+0.33 (e)Concatenated Prompts (a) + (b)78.71+0.02 F. Multi-Modalities We finally explore whether the text of a document can improve layout analysis accuracy. We use Optical Character Recognition (OCR) to extract the text information from the document and then use the CLIP text encoder to retrieve text tokens. These tokens are concatenated into the PromptDLA backbone. The results, shown in Table XI, indicate that the text modality does not improve the layout analysis performance. TABLE XI COMPARISON OF VISUAL-ONLY VS TEXTUAL AND VISUAL MODALITIES TagMethodmAP∆ (a)PromptDLA78.69 (b)PromptDLA + OCR78.53-0.16 Transformer Encoder Layer Transformer Encoder Layer Deep Prompted Transformer Encoder 퐿 1 퐿 2 푃 푣 푉 2 2 푉 푖 2 푉 1 2 ... 푉 푚 2 푃 푣 푉 2 1 푉 푖 1 푉 1 1 ... ... ... 푉 푚 1 MLP MLP ... Transformer Encoder Layer Transformer Encoder Layer Shallow Prompted Transformer Encoder 퐿 1 퐿 2 푉 2 2 푉 푖 2 푉 1 2 ... 푉 푚 2 푃 푣 푉 2 1 푉 푖 1 푉 1 1 ... ... ... 푉 푚 1 MLP ... 푃 2 Transformer Encoder Layer 퐿 12 푉 2 12 푉 푖 12 푉 1 12 푉 푚 12 ... ... 푃 12 Transformer Encoder Layer 퐿 12 푃 푣 푉 2 2 푉 푖 2 푉 1 2 푉 푚 2 ... ... MLP Fig. 8. Details of the shallow and deep prompt. G. Ablation Studies This section investigates the impact of different pretraining methods and the model’s design. Effect of Pretraining Methods. We analyze the model’s accuracy in three situations: without pretraining on document images, with self-pretraining on document images using the single vision model, and with the DiT model being self- pretrained using a vision-language approach. Our baseline uses the pre-trained transformer encoder of DiT and the text TABLE XII ABLATION STUDY FOR PRETRAINING METHODS. TagMethodmAP∆ (a)PromptDLA78.69 (b)w/o pretraining70.66-8.03 (c) VLP-pretraining(Layoutlmv3)76.38-2.31 (d) w/o CLIP text encoder76.33-2.36 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 201510 Table Picture Table 75% Picture 20% Section-header 95% Text 91% Picture 55% Table 26% Table 44% Picture 71% Picture 77% Picture 25% Picture 51% Table 44% Table 26% Table 26% Table 90% Picture 70% Text 94% Section-header 96% Table 90% Text Text Text Text Section-header Section-header Text 93% Caption 70% Picture 83% Text 99% Section-header 99% Text 92% Text 99% Text 96% Text Section-header 99% Text 93% Caption 70% Text 92% Text 99% List-item List-item List-item 82% List-item 94% List-item 82% List-item 94% （b）DiT（a）Ground Truth（c）PromptDLA Fig. 9. Qualitative comparison between DiT and PromptDLA on Financial Reports (1st row) and Laws (2nd row) domain from DocLayNet. It is best viewed in color and zooming out. encoder of CLIP. Compared to the baseline, the transformer encoder without pretraining reduces the mAP by 8.03, as illustrated in Table XII. Therefore, self-pretraining on large- scale document data can significantly enhance model accuracy by allowing the model to learn the generative document image representation via self-pretraining. Furthermore, the multi- modal pre-trained model from LayoutLMv3 is less effective than the single-model self-pre-trained transformer from DiT. Therefore, the single vision model’s self-pre-training method is most suitable for dealing with layout analysis problems. The CLIP text encoder is well-pretrained and can provide a prior textual representation. Table XII demonstrates that removing the CLIP text encoder and randomly initializing the prompt vector decreases 2.36 in mAP. The CLIP text encoder is already trained on large-scale image-text pairs. Thus, it can provide domain prior knowledge effectively. Effect of Prompt Location. We explore the impact of prompt location in the transformer encoder, employing two distinct prompt methods illustrated in Fig. 8: Shallow Prompted Encoder (SPE) and Deep Prompted Encoder (DPE). SPE exclusively incorporates a prompt at the first layer of the transformer encoder, whereas DPE integrates a prompt at every layer. As shown in Table XIII, both SPE and DPE models exhibit improvements compared to the baseline model without a prompt. Notably, the DPE method surpasses the baseline by 2.26 mAP. In the case of DPE, the CLIP text encoder generates 512-dimensional vectors, requiring MLP projection to the same 768-dimensional feature space as the image patch. We explore the design of the MLP for each transformer layer, specifically whether to use a shared MLP layer. Our observations indicate that the DPE method with a shared MLP layer results in an improvement of 2.01. Effect of Prompt Design. This study aims to investigate how to design prompts for document domain priors, exploring the correlation between prompts and DLA model performance on DocLayNet. Table XIV presents various prompt templates TABLE XIII ABLATION STUDY FOR PROMPT LOCATION. TagMethodmAP∆ (a)Baselines76.43 (b)SPE78.41+1.98 (c)DPE78.69+2.26 (d)DPE with share MLP Layer78.44+2.01 in the first column, the zero-shot document classification accuracy via CLIP in the second column, and the DLA results in the third column. The experiments demonstrate that prompts achieving higher accuracy in the CLIP classification task also lead to superior performance in our DLA model. This validates the rationale of our approach to creating prompt sets, focusing exclusively on the top-k accuracy rankings for the CLIP zero- shot classification task. TABLE XIV ABLATION STUDY FOR DIFFERENT DOMAIN PROMPTS. PromptAccuracymAP w/o prompt-76.43 A page comes from Domain Class44.0577.49 A document page of Domain Class44.0577.53 A piece of paper concerning with Domain Class 44.4477.55 A piece of paper comes from Domain Class46.9977.97 A document page comes from Domain Class48.4578.12 H. Discussions Results on Each Document Type in DocLayNet. As il- lustrated in Table XV, we trained PromptDLA and DiT using DocLayNet’s entire training dataset and evaluated their performance separately on each document type in the test dataset. Compared to the baseline DiT model without any domain-aware prompt, the model with Prompt outperforms it in terms of mAP, except for ”Laws and Regulations” and ”Patents”. Despite a decrease in our model’s mAP in the JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 201511 TABLE XV EXPERIMENTS RESULT ON EACH DOCUMENT TYPE IN DOCLAYNET (“W/O” AND “W” DENOTE DIT BASELINE AND PROMPTDLA, RESPECTIVELY). Caption Footnote Formula List-item Page-footer Page-header Picture Section-header Table Text TitlemAP Finacial reports w/o55.710.6-69.452.646.164.064.490.4 85.1 54.559.3 w55.423.9-73.860.255.467.767.691.4 86.2 60.564.2 Government tenders w/o15.082.065.390.051.575.675.378.095.4 85.1 39.168.4 w27.188.3100.091.850.980.477.881.795.0 88.3 61.376.6 Laws and regulations w/o28.091.616.781.642.971.849.568.070.4 84.1 84.462.6 w28.596.118.582.042.464.450.469.567.7 84.7 84.662.6 Manuals w/o85.733.8-82.076.187.676.778.370.3 83.5 59.473.3 w89.543.4-82.577.391.977.981.471.7 85.9 69.277.1 Patents w/o78.0-61.690.485.891.189.891.892.9 93.1 88.586.3 w81.1-51.891.787.392.087.791.992.8 93.6 86.685.7 Scientific articles w/o92.970.569.894.687.487.393.990.098.3 91.8 96.288.4 w94.281.274.695.490.289.993.990.898.2 92.8 96.090.7 ”Patents” domain, it still surpassed the baseline model in most categories, such as ”Caption” and ”Page-header.” We attribute the decline in the Formula and Picture categories to the limited correlation between Formula and Picture in the field of patents. Validating CLIP’s Prior Knowledge of Documents. The motivation behind using CLIP as a prompter is its impressive ability to learn visual and textual representations. We validate our choice of CLIP as a prompter by evaluating its zero-shot document classification performance. As shown in Table XVI, CLIP achieves 48.45% accuracy on DocLayNet and 54.55% on M6Doc without any fine-tuning. This indicates that CLIP possesses prior information about the document layout image. Computational Overhead. PromptDLA is computationally efficient, adding negligible overhead. As shown in Table XVII, applying our method to a DiT backbone reduces inference speed by a mere 0.13 FPS (from 6.75 to 6.62 FPS) on an RTX 3090 GPU. This trend holds across various models. TABLE XVI ZERO-SHOT DOCUMENT CLASSIFICATION PERFORMANCE OF CLIP. DocLayNetM6Doc #Document Types67 Accuracy48.45%54.55% TABLE XVII COMPUTATIONAL OVERHEAD OF PROMPTDLA. ModelmAP@IOU[0.50:0.95]FPS DiT-Base76.46.75 +PromptDLA78.7(+2.3)6.62(-0.13) LayoutLMv375.74.44 +PromptDLA76.4(+0.7)4.41(-0.03) Visualization. Fig. 9 presents the visualized results on the Fi- nancial Reports and Laws domain from DocLayNeta compar- ison between ground truth, DiT, and PromptDLA is presented. For the sample of Financial in the 1st row, DiT misidentifies the background as ”Figure” and recognizes the whole ”Table” as two separates. At the same time, the PromptDLA removes the misclassification of ”Figure” and produces a precise box of ”Table.” Moreover, the sample of Laws in 2nd row shows that our method precisely excludes the text box and identifies only the text inside it when processing manuals with text boxes. In contrast, DiT incorrectly identifies the text box as a complete figure. These qualitative results demonstrate the ability of the PromptDLA to recognize ambiguous objects by domain prior. Besides, Fig. 10 in Appendix shows more visualization examples. V. CONCLUSIONS We propose a novel PromptDLA framework, which can explicitly introduce domain prior into the DLA frameworks and steer DLA automatically, distinguishing the variability of different domains. The PromptDLA features a unique domain- aware prompter that could customize prompts according to the specific attributes of the data domain. We underscore the significance of utilizing domain priors in DLA through extensive experiments. The results show a new state-of-the-art performance across multiple datasets, including DocLayNet (78.7), M6Doc (69.2), and D 4 LA (69.1). It’s worth mentioning that the proposed domain-aware prompter is easily plugged into enhance different DLA frameworks. While PromptDLA demonstrates strong performance and adaptability, several av- enues for future work remain. A key direction is efficiency optimization. Integrating large language or vision-language models, particularly in the prompter component, introduces computational overhead compared with baseline DLA models. REFERENCES [1] Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, p. 6154–6162. [2] H. Bao, L. Dong, S. Piao, and F. Wei, “Beit: Bert pre-training of image transformers,” arXiv preprint arXiv:2106.08254, 2021. [3] Z. Zhang, J. Ma, J. Du, L. Wang, and J. Zhang, “Multimodal pre-training based on graph attention network for document understanding,” IEEE Transactions on Multimedia, vol. 25, p. 6743–6755, 2022. [4] H. Bi, C. Xu, C. Shi, G. Liu, Y. Li, H. Zhang, and J. Qu, “Srrv: A novel document object detector based on spatial-related relation and vision,” IEEE Transactions on Multimedia, vol. 25, p. 3788–3798, 2022. [5] A. Banerjee, S. Biswas, J. Llad ́ os, and U. Pal, “Swindocsegmenter: An end-to-end unified domain adaptive transformer for document instance segmentation,” arXiv preprint arXiv:2305.04609, 2023. [6] J. Ma, J. Du, P. Hu, Z. Zhang, J. Zhang, H. Zhu, and C. Liu, “Hrdoc: Dataset and baseline method toward hierarchical reconstruction of document structures,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 2, p. 1870–1877, Jun. 2023. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/25277 [7] C. Ma, Y. Zhang, M. Tu, Y. Zhao, Y. Zhou, and C. Zong, “Multi-teacher knowledge distillation for end-to-end text image machine translation,” in International Conference on Document Analysis and Recognition. Springer, 2023, p. 484–501. JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 201512 [8] J. He, L. Wang, Y. Hu, N. Liu, H. Liu, X. Xu, and H. T. Shen, “Icl-d3ie: In-context learning with diverse demonstrations updating for document information extraction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, p. 19 485–19 494. [9] C. Luo, Y. Shen, Z. Zhu, Q. Zheng, Z. Yu, and C. Yao, “Layoutllm: Layout instruction tuning with large language models for document un- derstanding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, p. 15 630–15 640. [10] Y. Wang, N. Lipka, R. A. Rossi, A. Siu, R. Zhang, and T. Derr, “Knowledge graph prompting for multi-document question answering,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, 2024, p. 19 206–19 214. [11] Y. Liang, Y. Zhang, C. Ma, Z. Zhang, Y. Zhao, L. Xiang, C. Zong, and Y. Zhou, “Document image machine translation with dynamic multi- pre-trained models assembling,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, p. 7077–7088. [12] A. Hu, H. Xu, J. Ye, M. Yan, L. Zhang, B. Zhang, C. Li, J. Zhang, Q. Jin, F. Huang et al., “mplug-docowl 1.5: Unified structure learning for ocr-free document understanding,” arXiv preprint arXiv:2403.12895, 2024. [13] Z. Zhang, Y. Zhang, Y. Liang, L. Xiang, Y. Zhao, Y. Zhou, and C. Zong, “Layoutdit: Layout-aware end-to-end document image translation with multi-step conductive decoder,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, p. 10 043–10 053. [14] X. Zhong, J. Tang, and A. J. Yepes, “Publaynet: largest dataset ever for document layout analysis,” in 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2019, p. 1015– 1022. [15] M. Li, Y. Xu, L. Cui, S. Huang, F. Wei, Z. Li, and M. Zhou, “Docbank: A benchmark dataset for document layout analysis,” arXiv preprint arXiv:2006.01038, 2020. [16] B. Pfitzmann, C. Auer, M. Dolfi, A. S. Nassar, and P. Staar, “Doclaynet: A large human-annotated dataset for document-layout segmentation,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, p. 3743–3751. [17] H. Cheng, P. Zhang, S. Wu, J. Zhang, Q. Zhu, Z. Xie, J. Li, K. Ding, and L. Jin, “M6doc: A large-scale multi-format, multi-type, multi- layout, multi-language, multi-annotation category dataset for modern document layout analysis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, p. 15 138–15 147. [18] C. Da, C. Luo, Q. Zheng, and C. Yao, “Vision grid transformer for doc- ument layout analysis,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, p. 19 462–19 472. [19] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, p. 8748–8763. [20] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi ` ere, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. [21] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Conditional prompt learning for vision-language models,” in Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2022, p. 16 816–16 825. [22] D. Lee, S. Song, J. Suh, J. Choi, S. Lee, and H. J. Kim, “Read- only prompt optimization for vision-language few-shot learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, p. 1401–1411. [23] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, p. 24 824–24 837, 2022. [24] S. Liu, C. Chen, X. Qu, K. Tang, and Y.-S. Ong, “Large language models as evolutionary optimizers,” in 2024 IEEE Congress on Evolutionary Computation (CEC). IEEE, 2024, p. 1–8. [25] B. Chen, Z. Zhang, N. Langren ́ e, and S. Zhu, “Unleashing the potential of prompt engineering in large language models: a comprehensive review,” arXiv preprint arXiv:2310.14735, 2023. [26] Z. Wang, H. Zhang, C.-L. Li, J. M. Eisenschlos, V. Perot, Z. Wang, L. Miculicich, Y. Fujii, J. Shang, C.-Y. Lee et al., “Chain-of-table: Evolving tables in the reasoning chain for table understanding,” arXiv preprint arXiv:2401.04398, 2024. [27] J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597, 2023. [28] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in neural information processing systems, vol. 28, 2015. [29] K. He, G. Gkioxari, P. Doll ́ ar, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, 2017, p. 2961–2969. [30] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, p. 779– 788. [31] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, p. 248–255. [32] Y. Huang, T. Lv, L. Cui, Y. Lu, and F. Wei, “Layoutlmv3: Pre-training for document ai with unified text and image masking,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, p. 4083–4091. [33] Y. Yu, Y. Li, C. Zhang, X. Zhang, Z. Guo, X. Qin, K. Yao, J. Han, E. Ding, and J. Wang, “Structextv2: Masked visual-textual prediction for document image pre-training,” arXiv preprint arXiv:2303.00289, 2023. [34] J. Li, Y. Xu, T. Lv, L. Cui, C. Zhang, and F. Wei, “Dit: Self-supervised pre-training for document image transformer,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, p. 3530–3539. [35] S. Appalaraju, B. Jasani, B. U. Kota, Y. Xie, and R. Manmatha, “Docformer: End-to-end transformer for document understanding,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, p. 993–1003. [36] J. Gu, J. Kuen, V. I. Morariu, H. Zhao, R. Jain, N. Barmpalios, A. Nenkova, and T. Sun, “Unidoc: Unified pretraining framework for document understanding,” Advances in Neural Information Processing Systems, vol. 34, p. 39–50, 2021. [37] S. Maity, S. Biswas, S. Manna, A. Banerjee, J. Llad ́ os, S. Bhattacharya, and U. Pal, “Selfdocseg: A self-supervised vision-based approach to- wards document segmentation,” arXiv preprint arXiv:2305.00795, 2023. [38] R. Rezaei, M. J. Sabet, J. Gu, D. Rueckert, P. Torr, and A. Khakzar, “Learning visual prompts for guiding the attention of vision transform- ers,” arXiv preprint arXiv:2406.03303, 2024. [39] T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020. [40] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020. [41] R. Zhang, J. Han, C. Liu, A. Zhou, P. Lu, Y. Qiao, H. Li, and P. Gao, “Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention,” in The Twelfth International Conference on Learning Representations, 2024. [42] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, p. 10 012–10 022. [43] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, p. 770–778. [44] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in European conference on computer vision. Springer, 2022, p. 709–727. [45] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision, 2020, p. 213–229. [46] X. Zhu, S. Lyu, X. Wang, and Q. Zhao, “Tph-yolov5: Improved yolov5 based on transformer prediction head for object detection on drone- captured scenarios,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, p. 2778–2788. JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 201513 Section-headerTextTable List-item Picture Page-header Page-footer Title Ground Truthwithout PromptDLA PromptDLA Caption Fig. 10. Appendix: More visualization examples on the DocLayNet dataset. From left to right, columns show Ground Truth annotations, predictions without PromptDLA, and predictions with PromptDLA, highlighting document layout segmentation accuracy improvements.