Paper deep dive

GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models

Yaohan Guan, Pristina Wang, Najim Dehak, Alan Yuille, Jieneng Chen, Daniel Khashabi

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 55

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 4/10/2026, 2:33:45 AM

Summary

GENFIG1 is a new benchmark designed to evaluate the ability of Vision-Language Models (VLMs) to generate 'Figure 1' scientific illustrations from research paper text (title, abstract, introduction, and caption). The dataset includes 9,976 papers from top AI/ML, CV, and NLP conferences, categorized by taxonomy (Overview, Example, Experimental Results). The study evaluates various generation strategies, including zero-shot, Chain-of-Thought (CoT), and Chain-of-Images (CoI), finding that current models struggle with conceptual grounding and visual coherence.

Entities (5)

GENFIG1 · benchmark · 100%Yaohan Guan · researcher · 100%Vision-Language Models · ai-model-class · 98%Chain-of-Thought · generation-strategy · 95%DINOv2 Score · evaluation-metric · 95%

Relation Signals (3)

GENFIG1 → contains → 9,976 research papers

confidence 100% · Our final dataset consists of tuples of (Title, Abstract, Introduction, Figure 1 caption, Figure 1) from 9,976 research papers

GENFIG1 → evaluates → Vision-Language Models

confidence 100% · GENFIG1 evaluates models for their ability to produce figures

DINOv2 Score → measures → Figure Quality

confidence 90% · To better assess the quality of generated figures, we design several automated metrics

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:In many science papers, "Figure 1" serves as the primary visual summary of the core research idea. These figures are visually simple yet conceptually rich, often requiring significant effort and iteration by human authors to get right, highlighting the difficulty of science visual communication. With this intuition, we introduce GENFIG1, a benchmark for generative AI models (e.g., Vision-Language Models). GENFIG1 evaluates models for their ability to produce figures that clearly express and motivate the central idea of a paper (title, abstract, introduction, and figure caption) as input. Solving GENFIG1 requires more than producing visually appealing graphics: the task entails reasoning for text-to-image generation that couples scientific understanding with visual synthesis. Specifically, models must (i) comprehend and grasp the technical concepts of the paper, (ii) identify the most salient ones, and (iii) design a coherent and aesthetically effective graphic that conveys those concepts visually and is faithful to the input. We curate the benchmark from papers published at top deep-learning conferences, apply stringent quality control, and introduce an automatic evaluation metric that correlates well with expert human judgments. We evaluate a suite of representative models on GENFIG1 and demonstrate that the task presents significant challenges, even for the best-performing systems. We hope this benchmark serves as a foundation for future progress in multimodal AI.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

55,094 characters extracted from source content.

Expand or collapse full text

GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models Yaohan Guan, Pristina Wang Najim Dehak, Alan Yuille, Jieneng Chen, Daniel Khashabi Johns Hopkins University Correspondence: yguan19@jhu.edu, pwang71@alumni.jh.edu Abstract In many science papers, “Figure 1” serves as the primary visual summary of the core re- search idea. These figures are visually sim- ple yet conceptually rich, often requiring sig- nificant effort and iteration by human authors to get right, highlighting the difficulty of sci- ence visual communication. With this intu- ition, we introduce GENFIG1, a benchmark for generative AI models (e.g., Vision–Language Models). GENFIG1 evaluates models for their ability to produce figures that clearly express and motivate the central idea of a paper (title, abstract, introduction, and figure caption) as input. Solving GENFIG1 requires more than producing visually appealing graphics: the task entails reasoning for text-to-image generation that couples scientific understanding with vi- sual synthesis. Specifically, models must (i) comprehend and grasp the technical concepts of the paper, (i) identify the most salient ones, and (i) design a coherent and aesthetically ef- fective graphic that conveys those concepts vi- sually and is faithful to the input. We curate the benchmark from papers published at top deep- learning conferences, apply stringent quality control, and introduce an automatic evaluation metric that correlates well with expert human judgments. We evaluate a suite of representa- tive models on GENFIG1 and demonstrate that the task presents significant challenges, even for the best-performing systems. We hope this benchmark serves as a foundation for future progress in multimodal AI. 1 1 Introduction Scientific figures are central to visual communica- tion in research, play a crucial role in how ideas are communicated, understood, and remembered. In many papers—especially in disciplines like AI, 1 We release our data athttps://huggingface.co/ datasets/yaohanguan/GenFig1 NLP and computer vision, “Figure 1” often func- tions as the conceptual centerpiece: a visual distilla- tion of the paper’s core motivation and contribution. These figures are not mere illustrations; they are designed to be both visually simple and conceptu- ally rich, serving as intuitive gateways into complex technical ideas. Compared with later figures, which more often elaborate specific components, imple- mentation details, or experimental analyses, Figure 1 is typically designed to orient the reader and fore- ground the paper’s central contribution. Creating such figures is a non-trivial task, typically requiring significant domain understanding, design iteration, and careful abstraction by authors. This highlights a broader challenge: conveying scientific ideas visu- ally is inherently difficult, especially when abstract concepts must be conveyed succinctly and clearly. This challenge presents a unique opportunity for generative AI systems. Notably, recent advances in multimodal generative AI systems powered by vi- sion–language models (VLMs) (Hurst et al., 2024) have achieved remarkable progress, particularly in generating photorealistic images and even videos from natural language prompts (Singer et al., 2023). Yet it remains unclear whether these models can perform perform the reasoning required for text-to- diagram generation. In particular, can these models digest technical content (e.g., paper title, abstract, introduction, and figure caption) and synthesize visuals that are both aesthetically coherent and con- ceptually faithful to the underlying research idea? To investigate this question, we introduce GEN- FIG1, a benchmark designed to evaluate the ability of generative AI models to produce “Figure 1”- style scientific illustrations. Unlike conventional image generation tasks, GENFIG1 requires models to couple scientific understanding with visual syn- thesis: they must (i) comprehend the content of a paper, (i) identify its key contributions, and (i) design a figure that visually communicates these ideas with clarity and fidelity. 1 arXiv:2604.04172v1 [cs.CV] 5 Apr 2026 Title: Aligning Large Language Models with Human Preferences through Representation Engineering Abstract: Aligning large language models (LLMs) with human preferences is crucial for enhancing their utility in terms of helpfulness, truthfulness, safety, harmlessness, and interestingness. Existing methods for achieving this alignment often involve employing reinforcement learning from human feedback (RLHF) to fine-tune LLMs based on human labels assessing the relative quality of model responses. Nevertheless, RLHF is susceptible to instability during finetuning and presents challenges in implementation... Introduction: While large language models ... language description of paper AI generated figure vs. Human-generated figure Title: Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval Abstract: Training dense passage representations via contrastive learning has been shown effective for Open-Domain Passage Retrieval (ODPR). Existing studies focus on further optimizing by improving negative sampling strategy or extra pretraining. However, these studies keep unknown in capturing passage with internal representation conflicts from improper modeling granularity. Specifically, under our observation that a passage can be organized ... Introduction: Open-Domain Passage Retrieval (ODPR) ... Figure 1: Examples from GENFIG1(the first row from (Liu et al., 2024) and second row from (Wu et al., 2022)). The task is to produce figures that clearly express and motivate the central idea of a paper (title, abstract, introduction, and figure caption) as input. Example responses from models we evaluate are shown in the middle column. Solving GENFIG1 requires more than just visually appealing graphics: the task entails cross-modal reasoning that couples scientific understanding with visual synthesis. We evaluate a range of representative VLMs on GENFIG1. This includes various combinations of VLMs to interpret and to translate reasoning about scientific content into effective visual layouts. We find that even the strongest models struggle with the task. The figures they generate often lack con- ceptual grounding, visual coherence, and faithful alignment with the source material—indicating that current models fall short of the depth of reasoning required for high-quality scientific visualization. Furthermore, we evaluate a range of automatic evaluation metrics to measure different aspects of figure quality, such as clarity, aesthetic quality, etc. We measure the correlation of these metrics with domain expert human judgments to identify the most reliable metrics, enabling scalable and con- sistent benchmarking. Our experiments show that the VLM-as-a-Judge metric, especially on informa- tion density, exhibits the strongest correlation with expert human judgments. We hope that GENFIG1 serves as a foundation for future progress in reasoning-driven multimodal systems, science of AI. At the same time, it sup- ports downstream applications in AI for science, by pushing toward tools that can assist researchers in communicating their ideas more effectively. 2 Related Work We discuss related work in two broad categories: benchmarks and figure generation frameworks. Related datasets: We compare all the previous datasets in Table 1. Several datasets target related but distinct tasks which we briefly highlight. Some datasets are motivated by question answering (QA) on scientific figures. For example, FigureQA (Ka- hou et al., 2018) and DVQA (Kafle et al., 2018) are two datasets with questions and answers about com- mon scientific-style plots. These datasets rely on synthetic figures, which lack the richness of natural researcher-created figures. There are resources that are motivated by other aspects of figure analysis. For instance, few datasets (Hu et al., 2024; Roberts et al., 2024; Chen et al., 2019; Hsu et al., 2021) were developed to facilitate multimodal diagram captioning and interpretation. CharXiv (Wang et al., 2024) is a chart understanding benchmark extracted from arXiv. ArXivCap is a figure-caption dataset extracted from arXiv by Li et al. (2024). FigureSeer dataset is a dataset created by Siegel et al. (2016) extracted from CiteSeerX papers to fa- cilitate figure parsing. FigureSeer dataset contains figures and figure annotations. Rodriguez et al. (2022) created Paper2Fig100k to facilitate figure generation. Paper2Fig100k is a dataset with OCR results, aspect ratios of images, paper figures and 2 Dataset Paper Contents Included Source Excluded Quant. Figures Figure CaptionPaper Text FigureSeer (Siegel et al., 2016)✓✗CiteSeerX✗ DVQA (Kafle et al., 2018)✓✗synthetic✗ FigureQA (Kahou et al., 2018)✓✗synthetic✗ FigCAP (Chen et al., 2019)✓✗synthetic✗ SciCap (Hsu et al., 2021)✓✗arXiv✗ Paper2Fig100k (Rodriguez et al., 2022)✓✗arXiv✗ SciCap+ (Yang et al., 2023)✓text context of figuresarXiv✗ SciFIBench (Roberts et al., 2024)✓✗arXiv✗ CharXiv (Wang et al., 2024)✓✗arXiv✗ ArXivCap (Li et al., 2024)✓✗arXiv✗ M-Paper (Hu et al., 2024)✓text context of figuresPaperWithCode✗ MMSci (Li et al., 2025)✓full textNature Communications✗ SridBench (Chang et al., 2025)✓related sectionarXiv and Nature✗ FigureBench (Zhu et al., 2026b)✓✗text context of figures Research-14K, arXiv, blogs, OpenStax✓ PaperBanana (Zhu et al., 2026a)✓methodology sectionNeurIPS 2025✓ GENFIG1 (ours)✓intro + abstracttop AI/ML venues✓ Table 1: Comparison of our dataset with previous datasets. Unlike previous datasets, ours includes a broad collection of peer-reviewed papers on top AI/ML venues, with a targeted focus on figure with caption, coupled with the introductions and abstracts. their captions extracted from arXiv. More recently, Chang et al. (2025) proposed SridBench for figure generation, which includes images, captions, and related sections. Zhu et al. (2026b) introduced Fig- ureBench, a benchmark of 3,300 long-form scien- tific text–figure pairs drawn from scientific papers, surveys, blogs, and textbooks, where selected fig- ures are paired with source text describing their key visual elements. (Zhu et al., 2026a) introduced Pa- perBananaBench, a benchmark for methodology- diagram generation curated from NeurIPS 2025 publications, containing 584 samples in total, each consisting of a methodology description, a method- ology diagram, and its caption; these are split into a 292-example test set and a 292-example reference set. Our proposed dataset, GENFIG1, introduces aspects not covered by existing datasets (Table 1). Specifically, we focus on generating illustrative fig- ures (i.e., “Figure 1”) of peer-reviewed scientific papers based on their textual content (e.g., abstract, introduction). Scientific figure generation: The broader field of image generation has rapidly advanced over the past decade, evolving from GANs (Goodfellow et al., 2020) to diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Rombach et al., 2022). Diffusion models have emerged as a leading class of generative models, achieving state-of-the-art sample quality in tasks such as text-to-image gen- eration (Nichol et al., 2021; Rombach et al., 2022; Saharia et al., 2022a) and image-to-image transla- tion (Saharia et al., 2022b; Lugmayr et al., 2022; Kawar et al., 2022). ControlNet (Zhang et al., 2023) extended latent diffusion models (Rombach et al., 2022) with dense conditioning inputs (e.g., semantic maps), enabling precise layout-guided image generation (Zheng et al., 2023; Inoue et al., 2023). These advances lay a strong foundation for automating the creation of scientific figures. Within the scientific figure generation community, efforts have expanded beyond diffusion models—though they remain a core component. In particular, code generation approaches using languages like TikZ and SVG have become increasingly prominent. For instance, Belouadi et al. (2024) focused on gen- erating scientific figures via TikZ code, while Ro- driguez et al. (2024) explored both text-to-SVG and image-to-SVG generation. Rodriguez et al. (2023b) introduced FigGen, a diffusion-based method for generating scientific figures from natural language. In our work, we benchmark a range of baselines to evaluate the effectiveness of these figure generation approaches (§4), including standalone diffusion models, hybrid methods that integrate language reasoning, and code generation–based pipelines. 3GENFIG1 The construction pipeline of our dataset contains three stages: data collection, filtering, and analysis. 3.1 Data Curation Initial data collection: We identify top-tier AI venues across multiple domains: Computer Vi- sion (CVPR), Natural Language Processing (ACL, NAACL and EMNLP), and general AI/Machine Learning (ICLR, NeurIPS) from 2020 through 3 2025. Our collection excludes NAACL for years when it did not occur (2020, 2023, and 2025), as well as ACL, EMNLP and NeurIPS 2025, which have not yet taken place when we crawled the data. We crawled the ACL, NAACL, EMNLP long/main track paper titles from ACL Anthology’s website. We crawled older CVPR accepted papers (older than 2022) from open-access and newer accepted papers (newer than 2022) from cvpr.thecvf.com and ICLR accepted (poster/spotlight/oral) papers using OpenReview API. We crawled NeurIPS accepted papers from papers.nips.c. Mapping papers to their L A T E X source: After retrieving paper titles, we used the arXiv API to search for arXiv paper IDs with the titles and cre- ated a database that maps paper titles to arXiv IDs. arXiv API search are not always successful, so we don’t have all of the arXiv IDs for all crawled paper titles. Using the retrieved arXiv IDs, we download the paper L A T E X source from arXiv and save them. Extracting content from L A T E X source:With all the L A T E X source, we useplasTeX(plasTeX Devel- opment Team, 2025) to parse the L A T E X documents into nestedplasTeXdocument objects. With the nestedplasTeXdocument objects and regular ex- pression, we are able to extract abstract section, introduction section, the section of the first figure section in the paper. We then extract the L A T E X text source from these sections and usePyLaTeX(Py- LaTeX Contributors, 2025) to parseLaTeXmacros and math formulas into plain text. Lastly, we ex- tract the image path of the figure image using regu- lar expression. Because parsing is computationally expensive, we usedPebble(Pebble Authors, 2025) to run all extractions in parallel. 3.2 Dataset Cleaning and Filtering Despite a high-quality extraction pipeline, some figures were incorrectly extracted and required fil- tering. We removed icons (images under 100 pix- els) and black images, as well as figures consist- ing of quantitative results (e.g., line charts, his- tograms, box plots, heatmaps) focused on model performance or parameter settings. Since our task inputs lack experimental data, reproducing such figures is not feasible. Unlike prior datasets (see Table 1, last column), we apply this filtering to fo- cus on figures that act as visual blueprints rather than those reporting raw statistics. 3.3 Data Analysis The resulting dataset:Our final dataset consists of tuples of (Title, Abstract, Introduction, Figure 1 caption, Figure 1) from 9,976 research papers that we collected from selected conferences in three do- mains: AI/ML, Computer Vision (CV), and Natural Language Processing (NLP). Within the dataset, AI/ML papers make up the largest proportion (44.7%), followed by CV (31.6%) and NLP (23.7%) (see Table 2). Among the NLP conferences, EMNLP contributes the most papers, while NeurIPS is the largest among the AI/ML venues. Table 5 in the Appendix details the number of papers retained after filtering for each venue by year from 2020 to 2025. Domain Venue# Papers Domain Share (%) Total Share (%) NLP ACL68629.06.9 NAACL1998.42.0 EMNLP1,48462.614.9 Subtotal2,369100.023.7 CVCVPR3,148100.031.6 AI/ML ICLR2,12147.621.3 NeurIPS2,33852.423.4 Subtotal4,459100.044.7 Total9,976100.0 Table 2: Distribution of papers in our GENFIG1 by do- main and venue (2020–2025). It covers six major venues across three research domains, with AI/ML (NeurIPS, ICLR) contributing nearly half of the papers, followed by CV (CVPR) and NLP (EMNLP, ACL, NAACL). Taxonomy: Figure 1 generation is a text- conditioned generation task. We assume with more specific text conditions, the generated image space becomes narrower. Although the provided textual input is already targeted, introducing a taxonomy category label can further restrict the image genera- tion space and enable the model to generate images in a more condition-aware way. To better under- stand the taxonomy of Figure 1s, we sample 50 Figure 1s from the dataset and manually catego- rize them into three main taxonomies—Overview, Example, and Experimental Results—as well as finer-grained subcategories described below. • Overview. The Overview category covers fig- ures that depict the overall structure or workflow of the proposed model or method. It includes two main subcategories: –Model Architecture: providing a detailed visu- alization of the model’s structural components and their interconnections. 4 –Method: illustrating the high-level workflow or procedural framework of the method, includ- ing pipelines or algorithmic steps. These may include frameworks that outline the end-to-end process, often with step-by-step textual exam- ples, as well as algorithmic diagrams—such as those for EM algorithms—that explicate each step of the procedure. •Example. Figures in this category illustrate spe- cific instances for clarity and context. It contains three subcategories: –Background: introducing or contextualizing the research problem or the task being studied. –Method: unlike the high-level method in the Overview category, figures here focus specif- ically on concrete input-output examples or demonstrate comparisons of model outputs against baseline methods. –Dataset: visualizing representative samples from datasets or the structure and composition of datasets. •Experimental Results. This category includes Figure 1s that present or compare experimental findings. We exclude them because they typically do not convey a paper’s core concepts at first glance, making them less effective as Figure 1s. –Model Metrics: numerical performance met- rics specific to a single method (e.g., average number of experts per layer after pruning). – Comparative Results: side-by-side perfor- mance comparisons among multiple methods. –Descriptive Analysis: visualizations of broader trends or phenomena in research(e.g., annual publication trends in text simplification), or the data distribution. In Figure 2, we can see that Overview and Example contribute more. Among them, Exam- ple–Background and Example–Method are the most frequent, followed by Overview–Model Ar- chitecture and Overview–Method. Filtering effect and coverageTable 6 in the Ap- pendix provides a detailed comparison between the raw corpus sizes and the final dataset after filter- ing, as well as the corresponding filtering rates for each venue and year. Papers with only quantitative experiment results are filtered out. Filtering rates generally range from 66% to over 93% depend- ing on the venue and year. CVPR consistently ex- hibits the highest retention rates, exceeding 90% in Figure 2: Resulted taxonomy of Figure 1s. We de- fine three taxonomies(Overview, Example, and Exper- imental Results) and multiple sub-taxonomies, where Overview and Example contribute more. Among them, Example–Background and Example–Method are the most frequent, followed by Overview–Model Architec- ture and Overview–Method. most years. This filtering ensures high data quality and consistency across venues, supporting reliable benchmarking. Textual content analysis Table 7 in the Ap- pendix reports the average lengths of key textual elements—title, abstract, introduction, and Figure 1 caption—for each venue. Titles in average have 9 words, while abstracts average 175 words. In- troductions are much longer, averaging 648 words. Captions in CV and AI/ML venues (CVPR, ICLR, NeurIPS) are approximately 50% longer than those in NLP venues (ACL, NAACL, EMNLP), averag- ing about 60 words versus about 40 in the latter. 4 Evaluation and Analysis With GENFIG1, we assess how effectively current VLMs generate Figure 1 for academic papers. 4.1 Baselines Systems We introduce six baselines spanning both direct and multi-step generation, as well as text-to-image and text-to-SVG paradigms. Zero-shot image generation:For the Zero-shot baseline, we prompt GPT-Image-1 to generate Fig- ure 1 based only on the provided text description. Zero-shotSVGgeneration: Thisvariant prompts GPT-4o mini to generate SVG code of an image, from the text description, offering a direct text-to-SVG alternative. 5 (a) Humans(b) Zero-shot(c) CoT (d) Zero-shot SVG(e) CoT SVG(f) Chain-of-images Figure 3: Figure 1 examples produced by both humans and models for all baselines from the paper (Liu et al., 2023). In the first row are the ones from Humans, Zero-shot, CoT; the second row: Zero-shot SVG, CoT SVG, Chain-of-images. Where we can see that the ones from Zero-shot, CoT and Chain-of-images are relatively decent, although cropped. Chain-of-Thought (CoT) image generation:In- spired by (Wei et al., 2022), this approach decom- poses the generation into stages. GPT-4o mini is first prompted to identify the figure’s taxonomy, then propose a coarse layout (e.g., left-right, top- bottom). It next specifies content for each region of the layout. Finally, we combine the type, layout, region details, and original text description as input to GPT-Image-1, which generates the final figure. Chain-of-Thought SVG generation (CoT SVG): This approach mirrors CoT but produces SVG code instead of an image. GPT-4o mini handles all steps, including SVG generation in the final stage. Chain-of-Images(CoI): Chain-of-Images adopts a multi-step process similar to CoT, but focuses on region-wise generation. GPT-4o mini first determines the figure type and layout, then produces detailed descriptions for each region. Rather than generating the full figure in one step, GPT-Image-1 renders each region individually. The final figure is assembled by combining these region-level images. Other models and failed runs: We attempted to run three additional methods: (1) a diffusion- based approach, (2) a text-to-SVG model, and (3) the text-to-TikZ method AutomaTikZ, proposed by Rodriguez et al.(Rodriguez et al., 2023b, 2024) and Belouadi et al.(Belouadi et al., 2024). However, the first two models are no longer publicly accessi- ble, and AutomaTikZ failed to generate compilable TikZ code for any of the 120 test examples. Example outputs: Figure 3 has examples of human- and AI-generated Figure 1s for the paper by Liu et al. (2023). 4.2 Automatic Metrics To better assess the quality of generated figures, we design several automated metrics that enable repro- ducible evaluation. Broadly, these metrics involve two categories: image-to-image comparison, which uses the reference image as the gold standard, and image-to-text comparison, which uses the accom- panying text (figure caption) as the gold standard. Since these metrics are originally defined on differ- ent scales, we rescale all scores to a common range of [0, 100] for direct comparison. Image-to-Image Comparison (DINOv2 Score): Following the DinoScore proposed in StarVec- tor (Rodriguez et al., 2023a), we adopt the DI- NOv2 Score, which leverages DINOv2 embed- 6 Approaches to construct figures VLM-as-a-Judge Text-Rich Catastrophic Neglect Score DINOv2 Score Human Preference MRR Clarity Faithfulness Info. Density Interestingness Legibility Aesthetic Text-to-Image Zero-shot77.785.275.068.687.676.365.655.545.3 CoT70.179.568.661.783.870.952.451.440.1 CoI68.977.564.363.289.574.258.550.940.5 Text-to-SVG Zero-shot SVG39.865.231.533.287.053.635.940.119.6 CoT SVG42.761.135.429.784.346.032.741.620.8 Humans87.499.586.278.096.682.494.2100.078.7 Table 3: Comparison of the different baselines for generating Figure 1s (see §4.3 and §5). All scores range from 0–100 (higher is better). As shown, image-generation approaches (e.g., CoI) achieve the highest scores, while text-based generation methods perform significantly worse. Overall, all automated methods fall well short of the performance of human-generated figures (final row), highlighting the difficulty of the task. Metrics VLM-as-a-Judge Text-Rich Catastrophic Neglect Score DINOv2 Score Clarity Faithfulness Info. Density Interestingness Legibility Aesthetic Kendall’s τ0.680.590.710.680.300.630.550.36 Spearman’s ρ0.770.670.790.770.350.710.640.44 Table 4: The agreement between automatic metric with human preferences (discussed in §5). As can be seen, VLM-as-a-Judge metrics align most closely with humans on five of six aspects, peaking at Info. Density (τ b =0.71, ρ=0.79), while other evaluation dimension show mild to fair agreement. dings (Oquab et al., 2023) and cosine similarity to assess visual similarity between the generated figure and the golden (reference) figure. Image-to-Text comparison (VLM-as-a-judge): We introduce a VLM-based evaluation metric that compares each generated figure to its correspond- ing caption, excluding the paper’s title, abstract, and introduction. The evaluation prompt defines six dimensions of figure quality: (a) Clarity: Does the figure clearly illustrate the paper’s core tech- nical contribution, including key components and relationships? (b) Faithfulness: Does the figure ac- curately reflect the paper’s content without distor- tion or misrepresentation? (c) Information Density: Is the figure concise, avoiding clutter as well as excessive empty space? (d) Legibility: Is all text readable, with clear labels and no distracting arti- facts? (e) Interestingness: Is the figure engaging and memorable, beyond being merely functional? (f) Aesthetic Score: Does the figure show visual balance, effective color use, and a coherent design? For each aspect, GPT-4.1 assigns a numeric score from 0 to 10, with a brief justification. Scores are then rescaled to the [0,100] range. These dimen- sions collectively capture key qualities of effective figures, including conceptual reasoning (e.g., en- coding core ideas and patterns), visual layout (e.g., spatial organization of elements), and aesthetic ap- peal (e.g., color harmony and visual coherence). Image-to-Text comparison (Text-Rich Catas- trophic Neglect Score): Inspired by the metrics proposed by Grimal et al. (2024), we introduce a text-rich catastrophic neglect score. While Grimal et al. define catastrophic neglect as the omission or misrepresentation of one or more elements from the prompt, we extend this to evaluate whether core ideas from a figure caption are omitted or insuffi- ciently represented in the generated figure. This metric assesses the completeness of alignment be- tween the figure and its caption. To operationalize it, we first extract a list of core ideas from each caption using an VLM. At evaluation time, another VLM is prompted to assess the extent to which each core idea is reflected in the generated figure. Each idea receives a score of 0 (not covered), 0.5 (partially covered), or 1 (fully covered). The final score is computed as the average across all core ideas for a given figure. 4.3 Experimental Analysis Text-to-Image outperform Text-to-SVG base- lines: According to Table 4, even the lowest- performing text-to-image setup (CoI) surpasses the best SVG-based method on Faithfulness, In- 7 formation Density, and Interestingness, suggest- ing image-generation models currently capture scientific-figure complexity better than code-driven SVG pipelines. Chain-of-xdoes not offer gains over Zero-shot baselines: To our surprise, Zero-shot baseline achieves the highest overall performance. This suggests that, despite the simplicity of its prompt strategy, Zero-shot generation benefits from the strong generalization capabilities of models like GPT-Image-1. More structured methods such as CoT and CoI underperform in comparison, indicat- ing that step-wise decomposition does not yet yield clear benefits. VLM-as-a-Judge reveals nuanced distinctions. The VLM-as-a-Judge scores reveal meaningful dis- tinctions across dimensions. For example, CoT figures show modest gains in Information Density and Interestingness over CoI, but lower Faithful- ness. These differences suggest structured prompt- ing may improve layout and content packing, but may also cause drifts from the input content. Human-generated figures consistently outper- form all automated methods: We observe this trend across all metrics. In particular, they achieve near-perfect Faithfulness (99.5) and Legibility scores. This underscores the difficulty of the figure generation and highlights the gap between human and machine capabilities in synthesizing clear, ac- curate, and engaging scientific visuals. Possible Failure Reasons We observe that cur- rent models struggle mainly with spatial constraint satisfaction during generation (also shown in Fig- ure 3). Common failure modes—such as incor- rectly positioned arrows and truncated text—reveal insufficient understanding of element dimensions, spatial relationships, and canvas size limitations, even when models successfully comprehend the underlying scientific concepts. 5 Human Evaluation Study To assess the robustness of our automatic evalua- tion and dataset quality, we conducted a user study involving NLP researchers serving as raters. These participants reviewed a mix of AI-generated and human-created Figure 1s. Setup Nine domain experts (each with at least one NLP or computer vision publication) evaluated a 106-paper test set. For each paper, they were shown the title, abstract, introduction, and the Fig- ure 1 caption, followed by six candidate Figure 1 images generated by different baseline methods. The images were randomly shuffled to avoid posi- tion bias. Annotators ranked the six figures from best (1) to worst (6), with ties allowed. Each paper was evaluated by two or three annotators, yielding 636 ranked judgments. Agreement among human annotators: We measure agreement between human annotators at the rank level using Kendall’sτ(τ b variant which accounts for ties in the rankings). We also measured Spearman’s rank correlation coeffi- cient, which measures the strength and direction of the monotonic relationship between two ranked lists. These metrics are applied to the full 1–6 ranked lists provided by each pair of annotators for a given paper. Results indicate moderate agree- ment (Kendall’sτ = 0.50, Spearman’sρ = 0.60). Disagreements among humans likely stem from a blend of objective and subjective factors: ranking requires both technical judgment (e.g., faithfulness to content) and personal preference (e.g., aesthet- ics). Differences in what annotators prioritize— such as visual design versus information density— naturally reduce alignment. Human evaluation of the baseline approaches: We report the mean reciprocal rank (MRR) of each method in Table 3. MRR is computed by taking the reciprocal of the rank assigned by human anno- tators to the target method’s figure (i.e., reciprocal rank= 1 rank ), and then averaging this value across all annotated papers. When multiple annotators are available for a paper, we average their ranks (Borda count aggregation) before computing the recipro- cal rank for that paper. As shown in the right-most column of Table 3, human-generated figures (last row) are consistently preferred by annotators, with a substantial margin over all automated methods. Agreement between humans and automatic met- rics: To evaluate the reliability of our automatic metrics, we compute their agreement with human rankings using the same correlation measures as above. For papers with multiple human rankings, we apply Borda count aggregation to average ranks before comparison. Table 4 shows the results. Among the metrics, VLM-as-a-Judge scores align most closely with expert judgments, particularly in assessing Information Density—whether a figure is concise and free of clutter. However, it performs 8 less well on Legibility, likely due to limitations in recognizing text readability, spatial relationships, and labeling clarity. Similarly, DINOv2 shows a moderate agreement with human preferences. 6 Conclusion We introduce GENFIG1, a benchmark for evaluat- ing generative AI models on their ability to produce scientific figures that are both visually coherent and conceptually accurate. We also introduce au- tomatic metrics that shows strong agreement with human judgments, suggesting a more reliable path for assessing progress. Despite recent advances in image generation, our evaluation of represen- tative VLMs shows that current models fall short in generating figures with clear visuals and deep conceptual alignment—highlighting the challenges of cross-modal reasoning in scientific contexts. By establishing this benchmark and dataset, we aim to drive progress in multimodal reasoning, visual ab- straction, and scientific communication. We hope GENFIG1 encourages the development of models that help researchers convey complex ideas more clearly and intuitively, advancing the broader mis- sion of AI for science. 7 Limitations One limitation is that we focus on represen- tative pipelines rather than exhaustive model comparisons. We use two publicly accessible representatives—GPT-4o mini and GPT-Image-1 (text-only vs. vision-grounded)—to provide a proof-of-concept benchmark and demonstrate the framework’s generality. Our focus for now is on contrasting figure-generation paradigms rather than exhaustively benchmarking all mainstream VLMs; extending to more recent state-of-the-art models is left for future work. Another limitation is that we currently report only overall scores, without stratifying results by Figure 1 category or domain. We plan to add per- category and per-domain analyses in future work. References Jonas Belouadi, Anne Lauscher, and Steffen Eger. 2024. AutomaTikZ: Text-guided synthesis of scientific vec- tor graphics with TikZ. In The Twelfth International Conference on Learning Representations. Yifan Chang, Yukang Feng, Jianwen Sun, Jiaxin Ai, Chuanhao Li, S Kevin Zhou, and Kaipeng Zhang. 2025. Sridbench: Benchmark of scientific research il- lustration drawing of image generation model. arXiv preprint arXiv:2505.22126. Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, Tong Yu, Ryan Rossi, and Razvan Bunescu. 2019. Figure captioning with reasoning and sequence-level training. Preprint, arXiv:1906.02850. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Communications of the ACM, 63(11):139–144. Paul Grimal, Hervé Le Borgne, Olivier Ferret, and Julien Tourille. 2024. Tiam – a metric for evaluat- ing alignment in text-to-image generation. Preprint, arXiv:2307.05134. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models. Advances in neural information processing systems, 33:6840– 6851. Ting-Yao Hsu, C. Lee Giles, and Ting-Hao ’Kenneth’ Huang. 2021. Scicap: Generating captions for scien- tific figures. Preprint, arXiv:2110.11624. Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan, Chenliang Li, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-paperowl: Scientific dia- gram analysis with the multimodal large language model. Preprint, arXiv:2311.18248. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. 2023. Layoutdm: Dis- crete diffusion model for controllable layout gener- ation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10167–10176. Kushal Kafle, Brian Price, Scott Cohen, and Christo- pher Kanan. 2018.Dvqa: Understanding data visualizations via question answering.Preprint, arXiv:1801.08163. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. 2018. Figureqa: An annotated figure dataset for visual reasoning. Preprint, arXiv:1710.07300. Bahjat Kawar, Michael Elad, Stefano Ermon, and Ji- aming Song. 2022. Denoising diffusion restoration models. In Advances in Neural Information Process- ing Systems. Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024. Multimodal arxiv: A dataset for improving scientific compre- hension of large vision-language models. Preprint, arXiv:2403.00231. 9 Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungy- oung Ji, Byungju Lee, Xifeng Yan, Linda Ruth Petzold, Stephen D. Wilson, Woosang Lim, and William Yang Wang. 2025. Mmsci: A dataset for graduate-level multi-discipline multimodal scientific understanding. Preprint, arXiv:2407.04903. Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. 2023. Aligning large language models with human pref- erences through representation engineering. ArXiv, abs/2312.15997. Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. 2024. Aligning large language models with human prefer- ences through representation engineering. In ACL (1). Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. 2022. Repaint: Inpainting using denoising diffusion prob- abilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 11461–11471. Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021.Glide: To- wards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fer- nandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, and 1 others. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. Pebble Authors. 2025. Pebble: Pebble provides a neat api to manage threads and processes within an appli- cation.https://github.com/noxdafox/pebble. GitHub repository, accessed 2025-08-01. plasTeX Development Team. 2025. plasTeX: A latex compiler written in python.https://github.com/ plastex/plastex.GitHub repository, accessed 2025-08-01. PyLaTeX Contributors. 2025. PyLaTeX: A python library for creating and compiling latex files or snippets.https://github.com/JelteF/PyLaTeX. GitHub repository, accessed 2025-08-01. Jonathan Roberts, Kai Han, Neil Houlsby, and Samuel Albanie. 2024. Scifibench: Benchmarking large mul- timodal models for scientific figure interpretation. Preprint, arXiv:2405.08807. Juan A Rodriguez, Shubham Agarwal, Issam H Laradji, Pau Rodriguez, David Vazquez, Christopher Pal, and Marco Pedersoli. 2023a. Starvector: Generating scalable vector graphics code from images. arXiv preprint arXiv:2312.11556. Juan A. Rodriguez, Abhay Puri, Shubham Agarwal, Is- sam H. Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. 2024. Starvector: Generating scalable vector graphics code from images and text. Preprint, arXiv:2312.11556. Juan A. Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli,and Pau Rodriguez. 2022. Ocr-vqgan: Taming text-within-image generation. Preprint, arXiv:2210.11248. Juan A Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez. 2023b. Figgen: Text to scientific figure generation.Preprint, arXiv:2306.00800. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.High- resolution image synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, and 1 others. 2022a. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487. Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. 2022b. Image super-resolution via iterative refine- ment. IEEE Transactions on Pattern Analysis and Machine Intelligence. Noah Siegel, Zachary Horvitz, Roie Levin, Santosh Ku- mar Divvala, and Ali Farhadi. 2016. Figureseer: Pars- ing result-figures in research papers. In European Conference on Computer Vision. Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, and 1 others. 2023. Make-a- video: Text-to-video generation without text-video data. In The Eleventh International Conference on Learning Representations. Jascha Sohl-Dickstein, Eric Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermo- dynamics. In International conference on machine learning, pages 2256–2265. pmlr. Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Hao- tian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. 2024. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Preprint, arXiv:2406.18521. 10 Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elic- its reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837. Bohong Wu, Zhuosheng Zhang, Jinyuan Wang, and Hai Zhao. 2022. Sentence-aware contrastive learning for open-domain passage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1062–1074. Zhishen Yang, Raj Dabre, Hideki Tanaka, and Naoaki Okazaki. 2023. Scicap+: A knowledge augmented dataset to study the challenges of scientific figure captioning. Preprint, arXiv:2306.03491. Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 3836– 3847. Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhon- gang Qi, Ying Shan, and Xi Li. 2023. Layoutdif- fusion: Controllable diffusion model for layout-to- image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 22490–22499. Dawei Zhu, Rui Meng, Yale Song, Xiyu Wei, Sujian Li, Tomas Pfister, and Jinsung Yoon. 2026a. Pa- perbanana: Automating academic illustration for ai scientists. Preprint, arXiv:2601.23265. Minjun Zhu, Zhen Lin, Yixuan Weng, Panzhong Lu, Qiujie Xie, Yifan Wei, Sifan Liu, Qiyao Sun, and Yue Zhang. 2026b. Autofigure: Generating and refin- ing publication-ready scientific illustrations. arXiv preprint arXiv:2602.03828. 11 Appendix AGENFIG1 Dataset Statistics In this section, we introduce the detailed statis- tics of our GENFIG1 Dataset. Table 5 summa- rizes the number of papers per venue after filtering, from 2020 to 2025. Data gaps (indicated by “—”) reflect years in which conferences were not held. Table 6 reports the raw and filtered paper counts, as well as filtering rates (%) by venue and year. Filtering rates are generally stable across venues (80–90%), though some fluctuations are observed (e.g., NeurIPS 2022: 67.0%). This, in one aspect, demonstrates the consistent application of filtering criteria and supports the overall reliability of the benchmark data. As explained in the Taxonomy, Figure 4 shows representative examples illustrating the taxonomy of Figure 1s in scientific papers. As discussed in §3.3, Table 7 summarizes av- erage section lengths (title, abstract, introduction, caption) by venue. We can see that, CVPR, ICLR, and NeurIPS papers tend to have longer abstracts and captions than NLP venues. For figure analysis, CLIP-based visual embeddings were projected via UMAP; as shown in Figure 5, CVPR figures form a relatively more distinct visual cluster, while fig- ures from other venues and fields are less separable, which suggests a slight style difference across the venues and domains. We also did the embedding space analysis across venues and research fields to try to inspect the dif- ferences between them. We use the CLIP ViT-B/32 model to extract 512-dimensional visual embed- dings for all figures. These embeddings are pro- jected into two dimensions using UMAP for visu- alization, enabling comparison of figure distribu- tions across venues and research domains. Figure 5 shows UMAP plots grouped by venue (a) and by research field (b). As we can see, figures from computer vision venues, such as CVPR, tend to form more coherent and distinct clusters in the em- bedding space. In contrast, figures from NLP and AI/ML venues (ACL, EMNLP, NAACL, NeurIPS, ICLR) exhibit substantial overlap. We could say, in general, Figure 1s from different venues and re- search fields tend to have more in common than not, which suggests that they might share similar visual styles and design conventions. Venue202020212022202320242025 ACL12074183142167— NAACL—5469—76— EMNLP221231263319450— CVPR256308378381949876 ICLR116159199308498841 NeurIPS302345193596902— Table 5: Number of papers per venue by year before and after filtering, covering the most recent five years (2020-2025). B Prompt and Additional Metrics The prompt used for VLM-as-a-judge (see Prompt 6) is outlined as follows. It introduces the evaluation task, specifying that both the figure caption and corresponding image are provided as context. The prompt details the evaluation guide- lines, clarifies the grading criteria across six dimen- sions, and defines the required JSON output format to ensure consistent and objective assessment. The prompts used to evaluate text-rich catastrophic ne- glect score (see Prompt 7) is outlined as following. There are two prompts that are used to evaluate text- rich catastrophic neglect score. The first prompt is used to extract the core concepts of the caption of Fig. 1 into a python list. The second prompt uses the core concept list to evaluate text-rich catas- trophic neglect score based on how thoroughly the concepts are covered in the figure. In addition to the metrics introduced in the pa- per, we also define two supplementary metrics: the Aesthetic Score Predictor and the Legibility Score (LS). Aesthetic Predictor Score We computed aes- thetic score given by a aesthetic score predictor as a reference. We choseAesthetic Predictor V2.5as the predictor model because compared toCLIP+MLP Aesthetic Score Predictor, Aesthetic Predictor V2.5has been improved to evaluate wider range of images such as illus- trations. While still very much different in style, illustrations are closer to scientific paper figures compared to natural images. Aesthetic score com- puted byAesthetic Predictor V2.5is a floating point number between 1 and 10 and scores greater than 5.5 are considered good aesthetic scores ac- cording to the authors. Legibility Score (LS).The Legibility Score (LS) measures the readability of text in a figure. We first apply OCR to localize text regions, then estimate 12 Venue202020212022202320242025 Raw Filt Rate Raw Filt Rate Raw Filt Rate Raw Filt Rate Raw Filt Rate Raw Filt Rate ACL157 120 76.49474 78.7 212 183 86.3 166 142 85.5 194 167 86.1— NAACL—6654 81.88569 81.2—10376 73.8— EMNLP 259 221 85.3 268 231 86.2 312 263 84.3 397 319 80.4 530 450 84.9— CVPR285 256 89.8 333 308 92.5 424 378 89.2 428 381 89.0 1044 949 90.9 969 876 90.4 ICLR141 116 82.3 200 159 79.5 235 199 84.7 358 308 86.0 575 498 86.6 1010 841 83.3 NeurIPS 398 302 75.9 429 345 80.4 294 193 65.6 743 596 80.2 1091 902 82.7— Table 6: Comparison of paper counts before and after filtering, and retention rates (%) by venue and year. Venue n papers TitleAbstractIntroCaption Char./WordsChar. /WordsChar. /WordsChar./Words ACL68668.3 / 9.3963.3 / 157.13325.1/ 564.7219.9/ 38.5 NAACL19969.5 / 9.9937.3 / 154.13223.0/ 551.4223.1/ 39.9 EMNLP148470.3 / 9.7971.1 / 158.23321.2/ 563.3231.9/ 40.8 CVPR314865.7 / 8.91126.4/ 180.23969.5/ 666.2331.3/ 56.5 ICLR212164.5 / 8.81140.6/ 183.53925.4/ 661.0347.3/ 61.3 NeurIPS233964.5 / 8.81118.0/ 181.04098.9/ 699.1342.9/ 61.1 Table 7: Average lengths (characters/words) for titles, abstracts, introductions, and Figure 1 captions, by venue. Across all 9,977 papers, a typical manuscript contains a 66-character (≈9-word) title, a 1.09 k-character (≈175- word) abstract, a 3.80 k-character (≈648-word) introduction, and a 309-character (≈54-word) Figure 1 caption. the effective font size and the contrast between text and background for each region, following estab- lished accessibility standards. The overall LS is computed by aggregating the font size and contrast metrics across all detected text boxes, providing an objective evaluation of text legibility under stan- dard viewing conditions. As shown in Table 8, the aesthetic and leg- ibility scores do not always align with the ex- pected quality of Figure1s. Notably, human-created Figure1s—which serve as the reference upper bound—sometimes receive lower scores than those assigned to figures generated by baseline methods. BaselineLegibility ScoreAesthetic Predictor Score CoT0.803.27 Zero-shot SVG0.343.30 CoT SVG0.423.62 CoI0.873.49 Zero-shot0.863.35 Humans0.753.41 Table 8: Designed image embedding–based metrics 13 Figure 4: Examples for taxonomy of Figure 1s. (a) Embeddings clustered by venues(b) Embeddings clustered by fields Figure 5: UMAP visualizations of the paper representations: (a) clustered by venues and (b) clustered by research fields. Papers from different venues show some clustering tendencies, but there is considerable overlap, indicating shared or transferable representations across domains. 14 Prompt for GPT-4.1 as a judge You are given a scientific paper’s context (specifically the Fig. 1 caption) and the corresponding Fig. 1 image. Figure 1 typically serves important purposes in scientific papers: it may illustrate a key example explaining the paper’s concept, introduce a pipeline or architecture of the method, or present data in other formats. Your task is to evaluate how effectively and concisely the figure describes the underlying idea of the paper. Please provide insightful, objective, and constructive critiques based on established design principles and academic standards. Your evaluation should: 1. Deliver comprehensive and unbiased assessments of the figure design. 2. Identify potential areas for improvement with actionable feedback. 3. Maintain consistent and high evaluation standards. Evaluation guidelines: • Score as objectively as possible. • Apply rigorous grading standards where: – 10 points = flawless figure, – 7 points = mediocre figure, – 4 points = figure with obvious shortcomings, – 1–2 points = very poor figure. • Provide concise reasoning for each rating to avoid response truncation. • Return only a valid JSON object with no additional commentary. Grading criteria (score each from 0–10): • Clarity: Does the figure clearly demonstrate the paper’s core technical idea? Does it effectively show input/output relationships or key components? (10 = core concept is immediately clear; 1 = technical idea remains indecipherable.) • Faithfulness: Does the figure avoid misleading statements or exaggerated claims not supported in the paper? (10 = perfectly faithful; 1 = major misrepresentation.) •Information Density: Does the figure use space efficiently without sparse or wasted areas? (10 = optimally dense; 1 = excessive empty space or clutter.) • Interestingness: Is the figure memorable and engaging? (10 = highly compelling; 1 = generic and unmemo- rable.) •Aesthetic Score: How well does the figure implement design principles (contrast, alignment, hierarchy, balance, modern styling)? (10 = professional, clear design; 1 = poor design that hinders comprehension.) • Legibility: Is all text readable at 100% zoom? Are there typos, unclear annotations, or other impediments? (10 = perfectly legible; 1 = completely unreadable.) Return a JSON object with scores and brief justifications in this format: "Clarity": "score": X, "reason": "Brief explanation", "Faithfulness": "score": X, "reason": "Brief explanation", "Information Density": "score": X, "reason": "Brief explanation", "Interestingness": "score": X, "reason": "Brief explanation", "Aesthetic Score": "score": X, "reason": "Brief explanation", "Legibility": "score": X, "reason": "Brief explanation" Figure 6: Prompt for GPT-4.1 as a judge 15 Prompt for Text-Rich Catastrophic Neglect Score Prompt to extract core concepts in the caption of the paper’s Fig. 1 User prompt is simply the caption and system prompt is as following: The given input is the caption of a teaser figure in a paper. Extract the core ideas mentioned in this caption into a python list. Output the python list and nothing else. Prompt to score the figure for Text-Rich Catastrophic Neglect Score User prompt is core concept list generated earlier and system prompt is as following: The goal of the task is to evaluate how thoroughly and clearly the core concepts are conveyed in the input image. The core concepts are a list of concepts extracted from the teaser figure caption from this paper. The core concepts will be given in list format as string input and all of the concepts must be conveyed in the input image. Use the paper information(paper title, abstract, introduction, teaser figure caption) provided below as reference and evaluate how thoroughly and clearly each concept is conveyed in the input image. paper_information Evaluate each core concept separately and output the score for each concept following the order of the concepts in the input list. To convey a concept thoroughly and clearly means that the visual components and text in the image are both reasonable and necessary. The visual component and text that are conveying such concept need to be clear and meaningful. Score can only take on the values 0, 0.5, 1 0 means that the concept was not covered in the image at all 0.5 means that the concept was partially covered in the image 1 means that the concept was thoroughly and clearly conveyed in the image Provide the scores of each concept in a python list following the order the concepts are given in the input list. Output only the scores list and nothing else. Figure 7: Prompt for Text-Rich Catastrophic Neglect Score 16

Instant research discovery