Paper deep dive
Evaluation of Embedding-Based and Generative Methods for LLM-Driven Document Classification: Opportunities and Challenges
Rong Lu, Hao Liu, Song Hou
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/10/2026, 2:35:49 AM
Summary
This paper evaluates embedding-based and generative Vision-Language Models (VLMs) for classifying technical geoscience documents. It finds that generative VLMs, particularly when enhanced with Chain-of-Thought (CoT) prompting, outperform multimodal embedding models in zero-shot accuracy. The study also highlights that while supervised fine-tuning improves VLM performance, it is highly sensitive to training data imbalance.
Entities (5)
Relation Signals (3)
Chain-of-Thought → improves → VLM Performance
confidence 95% · The advanced (“plus”) prompt provided a notable performance uplift over a simple (“base”) prompt
Qwen2.5-VL → outperforms → QQMM
confidence 95% · Qwen2.5-VL-72B achieved the best performance with a macro F1-score of 0.82... compared to state-of-the-art multimodal embedding models like QQMM (63%)
Supervised Fine-Tuning → sensitiveto → Training Data Imbalance
confidence 90% · we demonstrate that while supervised fine-tuning (SFT) can improve VLM performance, it is sensitive to training data imbalance.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This work presents a comparative analysis of embedding-based and generative models for classifying geoscience technical documents. Using a multi-disciplinary benchmark dataset, we evaluated the trade-offs between model accuracy, stability, and computational cost. We find that generative Vision-Language Models (VLMs) like Qwen2.5-VL, enhanced with Chain-of-Thought (CoT) prompting, achieve superior zero-shot accuracy (82%) compared to state-of-the-art multimodal embedding models like QQMM (63%). We also demonstrate that while supervised fine-tuning (SFT) can improve VLM performance, it is sensitive to training data imbalance.
Tags
Links
- Source: https://arxiv.org/abs/2604.04997v1
- Canonical: https://arxiv.org/abs/2604.04997v1
Trouble viewing inline? Open PDF directly →
Full Text
15,501 characters extracted from source content.
Expand or collapse full text
Evaluation of Embedding-Based and Generative Methods for LLM-Driven Document Classification: Opportunities and Challenges Rong Lu ∗ Hao LiuSong Hou Accepted at the IMAGE’25 Workshop (PCW-11), Society of Exploration Geophysicists (SEG). Published version available at: https://doi.org/10.1190/image2025-w11-03.1 Abstract This work presents a comparative analysis of embedding-based and generative models for classifying geoscience technical documents. Using a multi-disciplinary benchmark dataset, we evaluated the trade-offs between model accuracy, stability, and computational cost. We find that generative Vision-Language Models (VLMs) like Qwen2.5-VL, enhanced with Chain-of- Thought (CoT) prompting, achieve superior zero-shot accuracy (82%) compared to state-of-the-art multimodal embedding models like QQMM (63%). We also demonstrate that while supervised fine-tuning (SFT) can improve VLM performance, it is sensitive to training data imbalance. 1 Introduction The oil and gas industry is experiencing an unprecedented data deluge. A vast and growing corpus of technical information resides in unstructured archives of reports, logs, and surveys. Manual classification of these assets is a significant operational bottleneck, making robust, automated systems essential for transforming these archives into actionable intelligence. A key challenge is the multimodal nature of these documents. Critical classification cues often lie in visual elements such as log charts, seismic sections, and specific page layouts, which are missed by text-based models. Furthermore, the quality of Optical Character Recognition (OCR) can be low on legacy scanned documents, diminishing the effectiveness of text-only approaches. This necessitates the use of models that can jointly process visual and textual information. Two primary paradigms have emerged: embedding-based methods, which generate dense vector representations for similarity-based classification, and generative methods, where VLMs directly produce a class label. In this paper, we conducted a comparative study of these two approaches on a benchmark dataset of multi-disciplinary geoscience documents. We investigated the impact of prompting, fine-tuning, and data characteristics, offering insights for practitioners aiming to deploy these technologies effectively. 2 Methodology Our method encompasses a proprietary dataset, a suite of evaluation metrics, and standardized workflows for each modeling paradigm. ∗ Corresponding author. 1 arXiv:2604.04997v1 [cs.IR] 5 Apr 2026 2.1 Dataset We curated a benchmark dataset from an internal collection of technical documents. The dataset comprises eight classes spanning key disciplines in the energy sector: Geology & Geochemistry, Petrophysics, Geophysics, and Petroleum Engineering. The documents are in various formats, including multi-page PDFs and raster images (TIFF/TIF, PNG, JPG). For all experiments, the first page of each document was used to ensure a consistent evaluation basis. 2.2 Metrics We measure classification performance using overall accuracy and macro F1-score. For embedding models, we also evaluate the clustering quality of the ground-truth classes using metrics inspired by Brabandere et al.[2], i.e., an intra-cluster distance (L intra ) to measure cohesion and an inter-cluster distance (L inter ) to measure separation: L intra = 1 C C X c=1 1 N c N c X i=1 d cosine (μ c ,x i )(1) L inter = 1 C(C− 1) X c A ̸=c B d cosine (μ c A ,μ c B )(2) whereμ c represents class centroids andd cosine denotes cosine distance (defined as 1.0 minus the cosine similarity). We also compute the ratio of separation over cohesion, silhouette score, Davies-Bouldin (DB) Index, and Calinski-Harabasz (CH) Index. 2.3 Similarity-Voting using Embedding This approach reframes classification as a similarity-based voting. First, documents are converted to PIL images. Extremely large images are resized to a maximum dimension of 8192 pixels while preserving aspect ratio. Next, both the document images and the class labels are converted to embeddings. For document images, we use a simple prompt that instructs the model to generate a vector representation of the document. For class labels, we found that providing detailed domain- specific definitions boosts performance over using just the class name. Finally, the cosine similarity between the document embedding and each of the class embeddings is calculated. The class with the highest similarity score is chosen as the predicted label. We benchmarked five publicly available multimodal models. 2.4 VLM with Prompt Engineering This approach uses a VLM to directly generate the class label. We designed an advanced prompting strategy that combines CoT reasoning [8] with domain knowledge. The prompt (termed “plus” version) instructs the VLM to follow a multi-step process for more robust and accurate reasoning. It significantly improved performance over simpler prompts (termed “base” version which adds personas in prompts). The prompt and image are sent to a locally deployed endpoint and the model’s generated output is parsed to extract the predicted class. We evaluated four state-of-the-art open-weight VLMs. 2.5 VLM with SFT To evaluate the impact of domain adaptation, we fine-tuned a Qwen2.5-VL-7B model [1] using around 7000 training samples. The dataset is imbalanced: most classes contain hundreds to thousands of 2 samples, while certain minority classes have only dozens. To prevent prompt overfitting, a pool of various templates was used to construct the training samples. The fine-tuned model was then evaluated on the same test set. 3 Results Our experiment results are as follows. 3.1 Embedding Performance The performance of various multimodal embedding models is summarized in Table 1. The QQMM-embedmodel [9] demonstrates the best clustering quality across the board. With enhanced class-definition prompting, it achieved a macro F1-score of 0.64 and an accuracy of 0.63. Without this prompting, its F1-score dropped to 0.55 and accuracy to 0.58. Generally, larger embedding models outperformed smaller ones, though they incur higher computational costs. Table 1: Performance of five multimodal embedding models on the benchmark dataset. Arrows (↓,↑) indicate whether lower or higher values are preferable. QQMM performs better than GME [10], mmE5 [3], vdr [5], and CLIP [7]. ModelIntra (↓) Inter (↑) Ratio (↑) Silh. (↑) DB (↓) CH (↑) F1 (↑) Acc. (↑) QQMM-embed0.0880.1611.8220.2102.180 239.537 0.640.63 gme-Qwen2-VL-7b0.1280.0980.7610.0743.36195.2940.590.62 mmE5-mllama-11b0.1430.1120.7850.0893.28695.3040.510.53 vdr-2b-multi-v10.2080.1670.8040.0684.10789.4970.370.38 clip-ViT-L-140.2050.1100.5360.0025.23165.7320.180.22 3.2 VLM Performance VLMs demonstrated higher zero-shot classification accuracy. As shown in Table 2, Qwen2.5-VL-72B achieved the best performance with a macro F1-score of 0.82 and an accuracy of 0.82. The advanced (“plus”) prompt provided a notable performance uplift over a simple (“base”) prompt for both 7B (10% F1 lift) and 72B (5% F1 lift) models. Qwen models outperformed other tested VLMs like Mistral Small 3.2 [6] and Gemma 3 [4]. Table 2: Zero-shot classification performance of various VLMs. The “plus” prompt includes CoT and domain definitions while the “base” version mainly utilizes persona prompting. Model Base PromptPlus Prompt AccuracyF1AccuracyF1 Qwen2.5-VL-72B0.780.770.820.82 Qwen2.5-VL-7B0.650.650.760.75 Gemma-3-27B0.640.650.700.69 Mistral-3.2-24B0.550.550.580.55 3 3.3 SFT Performance Fine-tuning the Qwen2.5-VL-7B model yielded mixed results, i.e., the performance was dependent on the class distribution in the training data. Classes with thousands of training samples saw significant F1-score improvements (over 20% uplift). Conversely, performance dropped for under-represented ones (which had dozens of training samples), highlighting the model’s sensitivity to data imbalance. The model achieved 0.93 for both macro F1 and accuracy scores on a held-out test set for those classes over 150 training samples. 4 Discussion and Conclusion A classification-focused comparison across all the choices is illustrated in Figure 1. VLMs’ higher accuracy likely stems from their ability to perform deeper, end-to-end reasoning over the entire document image, capturing nuanced relationships between text and layout that are abstracted away into a single vector by embedding models. However, this comes at a cost. VLM inference is computationally expensive and slow, often requiring high-end GPUs to process large document images at scale. Furthermore, their generative nature can lead to nondeterministic outputs, a concern for production systems requiring reproducibility. In contrast, embedding models are lightweight, faster, and produce deterministic outputs, making them more suitable for large-scale batch processing on less powerful hardware. Figure 1: Classification performance for both embedding and VLMs under various configurations. VLMs generally outperform embedding models, especially after fine-tuned with sufficient training samples. Dots are individual models’ results. For both model types, prompt engineering is a highly effective, low-cost method for injecting domain knowledge. For embedding models, providing detailed class definitions boosted F1-score 4 from 0.55 to 0.64. For VLMs, the CoT-style “plus” prompt, which guides the model’s reasoning process, lifted the 7B model’s F1-score by 10 points. This underscores that off-the-shelf models are insufficient; performance is maximized when the models are guided by domain expertise. Successful domain adaptation through fine-tuning is achievable but requires a meticulous focus on creating well-balanced training datasets. References [1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report. arXiv e-prints, art. arXiv:2502.13923, February 2025. doi: 10.48550/arXiv.2502.13923. [2]Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function. CoRR, abs/1708.02551, 2017. URLhttp://arxiv.org/abs/ 1708.02551. [3] Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mmE5: Improving multimodal multilingual embeddings via high-quality synthetic data. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 8254–8275, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. URL https://aclanthology.org/2025.findings-acl.433/. [4]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram ́e, Morgane Rivi`ere, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Ga ̈el Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, Andr ́as Gy ̈orgy, Andr ́e Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Pluci ́nska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju-yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, 5 Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Ptildeoder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, and Oriol Vinyals. Gemma 3 Technical Report. arXiv e-prints, art. arXiv:2503.19786, March 2025. doi: 10.48550/arXiv.2503.19786. [5] LlamaIndex. Model card for vdr-2b-multi-v1. https://huggingface.co/llamaindex/vdr-2b-multi- v1, 2025. Accessed: 2025-07-07. [6]MistralAI.Modelcardformistral-small-3.2-24b-instruct-2506. https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506, 2025.Accessed: 2025-07-07. [7]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. URL https://arxiv.org/abs/2103.00020. [8] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. [9]Youze Xue, Dian Li, and Gang Liu. Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying. arXiv e-prints, art. arXiv:2506.02020, May 2025. doi: 10.48550/arXiv.2506.02020. [10]Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. GME: Improving Universal Multimodal Retrieval by Multimodal LLMs. arXiv e-prints, art. arXiv:2412.16855, December 2024. doi: 10.48550/ arXiv.2412.16855. 6