Paper deep dive
FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation
Yinghao Tang, Tan Zhenwei, Yiyao Wang, Wanli Gu, Xiaolu Zhang, Jun Zhou, Wei Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 87%
Last extracted: 8/9/2026, 3:32:19 AM
Summary
The paper introduces FinReportBench, an expert-grounded benchmark designed to measure and improve the generation of institution-grade financial reports by Large Language Models. It addresses the gap between fluent text generation and professional institutional standards by utilizing a 35-item hierarchical rubric covering deliverability, report identity, and institutional completeness. The benchmark includes a curated dataset of 244 bilingual tasks and demonstrates that while basic deliverability is saturated, report identity and institutional completeness remain bottlenecks. The authors also propose a benchmark-guided skill distillation method that significantly improves model performance across multiple families by converting recurrent failures into reusable constraints.
Entities (16)
Relation Signals (7)
FinReportBench → evaluates → Large Language Models
confidence 95% · We introduce FinReportBench, an expert-grounded benchmark for measuring and improving institution-grade financial report generation.
FinReportBench → uses → 35-item Rubric
confidence 92% · We derive a 35-item rubric through expert partial orders... covering deliverability, report identity, and institutional completeness.
FinReportBench → contains → 244 bilingual tasks
confidence 90% · Starting from 10,000 balanced Chinese and English financial-research source records, we curate 244 bilingual tasks
Skill Distillation → improves → Large Language Models
confidence 88% · Across five model families, the evolved skill improves mean G1 by 33.85 points and mean G2 by 13.83 points
MiniMax-M2.7 → achieveshighestscore → FinReportBench
confidence 85% · The provisional hierarchical-score ordering is MiniMax M2.7 (22.6)... MiniMax M2.7 has the highest G1 score (40.0) and G2 score (46.6).
FinReportBench → curatedfrom → Eastmoney
confidence 80% · We assemble a balanced multilingual reference pool of 5,000 Chinese analyst reports from Eastmoney
FinReportBench → curatedfrom → World Bank
confidence 80% · and 5,000 English institutional publications from the World Bank.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models can produce fluent financial analysis, but fluency alone does not establish whether a report is suitable for institutional delivery. We introduce FinReportBench, an expert-grounded benchmark for measuring and improving institution-grade financial report generation. Expert review reveals recurring gaps in report identity, institutional components, source discipline, and visual delivery. We derive a 35-item rubric through expert partial orders, multimodal evidence, and audits of decision boundaries, covering deliverability, report identity, and institutional completeness. Starting from 10,000 balanced Chinese and English financial-research source records, we curate 244 bilingual tasks across three research objects and two input tiers. Each task separates the public query, reconstructed research trajectory, and hidden source packet. Three independent judge families reproduce the expert partial order at near-ceiling rates, showing that bounded, observable criteria support reliable evaluation. Across nine model families, basic deliverability is nearly saturated, while report identity and institutional completeness remain the primary bottlenecks. The largest cross-model gaps concern generation-trace control, information density, and data discipline rather than basic report framing. We then use benchmark-guided skill distillation to turn recurrent failures into reusable generation and self-review constraints. Across five model families, the evolved skill improves mean G1 by 33.85 points and mean G2 by 13.83 points over paired no-skill runs while preserving G0 for every pair. Code and benchmark artifacts are available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.04374v1
- Canonical: https://arxiv.org/abs/2608.04374v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
42,062 characters extracted from source content.
Expand or collapse full text
FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation Yinghao Tang,1 Tan Zhenwei,2 Yiyao Wang,1 Wanli Gu,2 Xiaolu Zhang,2 JUN ZHOU,2 Wei Chen1 Abstract Large language models can produce fluent financial analysis, but fluency does not establish whether a report is suitable for institutional delivery. Existing evaluations use broad dimensions that do not identify the specific defects that cause professional rejection. We introduce FinReportBench, an expert-grounded benchmark for measuring and improving institution-grade financial report generation. In a pilot review, three experts assign all 75 reports from three models the lowest readiness score despite generally fluent financial prose; the recurring gaps concern report identity, institutional components, source discipline, and visual delivery. Pilot interviews further identify a hierarchy of report criteria. We then derive the rubric through two rounds: expert partial orders over a small output set guide item induction with multimodal evidence, and experts audit the resulting decision boundaries. The final instrument contains 35 observable items across deliverability, report identity, and institutional completeness. Starting from 10,000 balanced Chinese and English financial-research source records, we curate 244 bilingual tasks across three research objects and two input tiers. Each task separates the public query, reconstructed research trajectory, and hidden source packet. Three independent judge families reproduce the expert partial order at near-ceiling rates, showing that the rubric’s bounded, observable criteria support reliable evaluation. Across nine model families, basic deliverability is nearly saturated, while report identity and institutional completeness remain the primary bottlenecks. The largest cross-model gaps arise in generation-trace control, information density, and data discipline rather than basic report framing. Finally, we use benchmark-guided skill distillation to convert these recurrent failures into reusable generation and self-review constraints. Across five model families, the evolved skill improves mean G1 by 33.85 points and mean G2 by 13.83 points over the paired no-skill condition, while preserving G0 for every pair. Code and benchmark artifacts are available at https://github.com/MisterBrookT/finreportbench. Introduction Financial institutions communicate research through professional reports. These reports combine analytical claims, data, charts, source attribution, institutional identity, and a stable document structure (Jin et al. 2026; Lifan et al. 2026). They are reviewed, distributed, and archived as accountable research artifacts. We refer to reports that satisfy this delivery standard as institution-grade financial reports. Language models can produce fluent financial analysis, but their outputs often fail to convince professional readers. In preliminary interviews with three financial experts, all three assigned a satisfaction score of 1, the lowest value on the five-point scale, to all 75 reviewed model-generated artifacts. They found that an output could contain plausible or locally accurate statements while still falling far short of an acceptable research deliverable. The largest deficits concerned professional report conventions and visual presentation, including front-page framing, institutional identity, page systems, compliance components, and source presentation. These shortcomings call for a systematic evaluation that can identify specific problems in generated financial reports. Recent work has begun to benchmark financial report generation, but its evaluation criteria remain broad and coarse-grained. FinSight evaluates factual accuracy, information effectiveness, and presentation quality, while Cogito evaluates data quality, analytical quality, and presentation quality (Jin et al. 2026; Lifan et al. 2026). These dimensions support overall system comparison, but they do not identify specific problems that prevent institutional use. For example, a webpage-like report with polished charts may receive a high presentation score even when it lacks a publisher, analyst identity, and a stable page system. To fill this gap, we present FinReportBench, the first benchmark designed to measure whether generated financial reports meet institutional delivery standards. Building such a benchmark requires addressing three challenges. First, converting expert knowledge into a reliable rubric is difficult. Professionals rely on conventions learned through practice and can often express relative preferences more reliably than an exhaustive scoring rule. Interviews and preference labels must therefore be turned into observable criteria, clear boundaries, and rules that another annotator can reproduce. Full-report review is also costly, which limits the amount of expert annotation available for rubric construction. Second, a published report shows only the final artifact. The original client request and the research steps that produced it are usually unavailable, so realistic benchmark inputs cannot be collected directly. Third, a broad score can rank systems but does not tell a developer what to fix. Actionable evaluation requires specific failure evidence that can guide improvement without exposing test answers or benchmark content. Figure 1: Metadata overview of FinReportBench. (a) A two-dimensional projection of the shared multilingual embedding space places the 244 selected tasks across the density of the 10,000-task reference pool; color denotes research object and marker shape is a language diagnostic. Coverage is computed in the original embedding space, where the suite covers 95.65% of reference tasks at cosine similarity ≥0.60≥ 0.60. (b1) Model inputs are substantially longer than query-only financial-report benchmarks; (b2) component profiles show the anatomy of the 244 scripted research trajectories. (c1) Each task contains multiple retrieved sources and structured evidence facts. (c2) Counts across the diagnostic domain taxonomy show the breadth of selected research topics. We make five contributions: • Expert-grounded gap diagnosis. We provide an expert-grounded diagnosis of why fluent model outputs remain unsuitable as institutional reports. Reviews with three senior financial professionals identify the gap and reveal a hierarchy of professional requirements (§ 2). • Financial-report generation dataset. We introduce a 244-task bilingual dataset curated from a balanced 10,000-task reference space, covering three research objects, two input tiers, and 95.65% of the multilingual semantic space. Each task includes a tool-use research trajectory with separated public queries, retrieved evidence, and hidden source packets (§ 3). • Hierarchical evaluation framework. We propose a hierarchical framework derived from expert reviews and preference-guided rubric induction. It evaluates institutional readiness through 35 observable criteria and an evidence-based multimodal judging protocol (§ 4). • Benchmark-guided skill evolution. We propose a method that converts recurrent item-level failures across models into a compact, reusable production skill. The method develops and selects skill revisions on external tasks before a locked cross-model transfer test (§ 5). • Evaluation and findings. In the completed evaluation, we characterize nine model families through item-level analyses on FinReportBench. The same task set is used for every model, and visible output defects remain part of the measured result. We also test whether the evolved skill transfers across five model families (§ 6). Related Work and Pilot Interviews Related Work Financial NLP benchmarks largely evaluate question answering and multimodal reasoning over tables, filings, and conversations (Chen et al. 2021, 2022; Zhu et al. 2021; Zhao et al. 2022; Islam et al. 2023; Reddy et al. 2024; Liu and others 2025; Xie and others 2024; Xie et al. 2023; Wu et al. 2023), whereas agent benchmarks and training methods emphasize web interaction, retrieval, and multi-step information seeking (Mialon et al. 2024; Zhou et al. 2024; Wu and others 2025; Nakano et al. 2021; Jin and others 2025; Wei and others 2025; Deng et al. 2023; Gou and others 2025; Baek et al. 2024; Li et al. 2026; Bian et al. 2025). Recent financial-report systems add broad factual, analytical, and presentation dimensions (Jin et al. 2026; Lifan et al. 2026; Weng et al. 2025; Tang et al. 2026b; Xie et al. 2024, 2026; Tan and others 2024a; Wu and others 2024; Kim and others 2024; Asai and others 2024), but these scores do not identify the specific defects that cause professional rejection. FinReportBench instead evaluates complete rendered reports through fine-grained observable items organized by an expert-derived hierarchy. Recent work also extracts reusable agent skills from resources or execution failures and selects revisions on held-out tasks (Li and others 2026; Huang and others 2026; Alzubi and others 2026; Shen and others 2026; Zhang and others 2026). We adapt this idea to institutional reporting: recurrent item-level failures become compact production rules, and external validation guards against negative transfer. Pilot Expert Interviews We conducted multi-round pilot interviews with three senior financial professionals, each with more than ten years of experience. They reviewed 75 anonymized reports produced by three models for 25 queries and rated complete-report readiness on a five-point scale. All 75 reports received the lowest score of 1. The interviews yielded three design findings. First, fluent outputs still lacked professional structure, institutional identity, source discipline, risk disclosure, and consistent visual delivery (Qu and others 2025; Wen et al. 2026; Tang et al. 2026c). Second, review was hierarchical: experts checked deliverability, then report identity, then detailed institutional components. Third, research preceded writing; analysts expected a model to continue from an organized evidence state rather than generate directly from a short query. These findings motivate, respectively, fine-grained item evaluation, the G0–G1–G2 hierarchy, and the reconstructed research trajectory used by FinReportBench. Interview questions, review instructions, and additional protocol details appear in the supplementary material. FinReportBench Dataset Dataset Design Objectives FinReportBench targets four properties: authenticity, by deriving tasks from real analyst or institutional sources; diversity, across languages, research objects, domains, and query specificity; efficiency, by selecting a compact suite that preserves the semantic coverage of a much larger pool; and a realistic generation pipeline, in which report writing continues from an evidence-rich research trajectory rather than a query alone. Construction Pipeline Figure 2 summarizes five stages: source collection, task synthesis, query validation, coverage-oriented curation, and trajectory construction. Figure 2: The FinReportBench dataset construction pipeline. We synthesize and validate source-grounded tasks, curate the benchmark for broad coverage, and construct a research trajectory for each selected task. Source-grounded task synthesis. We assemble a balanced multilingual reference pool of 5,000 Chinese analyst reports from Eastmoney and 5,000 English institutional publications from the World Bank. Because original client requests are rarely available, reverse synthesis recovers a plausible request from each source’s research object, context, and analytical angle. An LLM-based audit rejects unnatural requests, taxonomy errors, and publisher, source, or answer leakage. Each accepted task separates a public query card from a hidden source mapping and construction metadata; the source artifact is never provided to the report model. Coverage-oriented curation. We embed all source-derived research tasks in one multilingual semantic space and select a compact suite while balancing language, research object, domain, and query specificity. Language is a diagnostic slice, not a separate benchmark. The resulting 244 tasks contain 122 Chinese and 122 English requests; 61 are company, 92 industry, and 91 macro tasks. Eighty-seven are open-ended T0 requests and 157 are thesis-guided T1 requests. At cosine similarity 0.60, the selected set covers 95.65% of the 10,000-task reference space, including 99.46% of Chinese and 91.84% of English references. Research trajectory construction. For every selected task, we reconstruct a scripted deep-research trajectory containing the public request, search decisions, retrieved evidence, and a final textual research note. The final user message asks for a self-contained HTML report. The trajectory does not replay the hidden source and excludes benchmark identifiers, internal evidence labels, and URL lists from the research note. All 244 trajectories pass role-order, provenance, final-note, and leakage checks. The supplementary material provides source-pool composition, semantic fingerprint fields, embedding and selection details, query-audit criteria, the complete trajectory contract, and additional dataset statistics. Evaluation Design Evaluation Design Objectives Our evaluator targets three properties. Low-cost expert grounding uses sparse within-query preferences instead of exhaustive item annotation. Fine-grained diagnosis returns observable item decisions and supporting evidence rather than only an overall score. Hierarchical validity follows the professional review order found in the pilot study, preventing later strengths from fully compensating for an earlier-stage failure. Evaluation Construction Pipeline Figure 3 summarizes the process. Experts first provide partial orders over small sets of anonymized reports generated for the same query. A multimodal model contrasts preferred and dispreferred reports to mine candidate visual and textual evidence (Tang et al. 2026a; Pan et al. 2025; Masry et al. 2022; Kantharaj et al. 2022; Methani et al. 2020; Masry et al. 2023; Xie et al. 2025; Chen et al. 2025). Recurring, generalizable differences are converted into observable items with pass, partial, fail, and, where appropriate, not-applicable boundaries. Experts then remove, merge, split, or revise invalid and redundant items. Rubric induction and final validation use disjoint report samples. Figure 3: The FinReportBench evaluation pipeline. Sparse expert preferences guide contrastive evidence mining and observable item induction. Expert-audited items form a hierarchical rubric that returns layer scores, item decisions, and supporting evidence from rendered pages and extracted text. This two-round process concentrates scarce expert effort on decision-relevant contrasts: experts supply preference boundaries and audit induced items rather than label every report–criterion pair. The supplementary material contains the annotation protocol, contrastive mining procedure, item-induction prompts, and additional audit examples. Hierarchical Rubric and Scoring The frozen rubric contains 35 items in three ordered layers. G0: deliverability uses four preflight checks for a visible, readable, continuous, and non-broken artifact; any required failure sets the total to zero. G1: report identity uses four items to test whether the artifact is recognizable as institutional research rather than a webpage, dashboard, slide deck, generic summary, or direct task answer. G2: institutional completeness uses 27 items covering front-page framing, institutional identity, compliance, page systems, information density, source and chart discipline, and generation-artifact control. For active G1 and G2 items, pass receives 1, partial 0.5, and fail 0; not applicable is removed from the denominator only when a criterion genuinely does not apply. Let S1S_1 be the weighted G1 percentage, with g1,d1g_1,d_1 the weighted points obtained and available in G1, and define g2,d2g_2,d_2 analogously for G2. For a G0-passing report, S=100g1+(S1/100)g2d1+d2;S=0if G0 fails.S=100 g_1+(S_1/100)g_2d_1+d_2; S=0\ if G0 fails. (1) Thus weak report identity discounts isolated G2 details without a discontinuous score cap. We report G1 and G2 alongside this provisional total. Complete item descriptors, weights, boundaries, and scoring examples appear in the supplementary material. Itemized Multimodal Evaluation We render each HTML report and extract its text. The judge evaluates every applicable item using the appropriate visual, textual, or mixed evidence and returns an item decision with a short evidence citation. This produces both comparable layer scores and an actionable failure profile for model or skill improvement. Held-out validation compares automatic and expert within-query partial orders over model outputs. We also compare judge families while fixing reports, rubric, and scoring code (Zheng et al. 2023; Chiang and others 2024; Tan and others 2024b; Zhuge and others 2024; Zheng et al. 2024). Section Experiments reports the main validation results; complete pair-sampling instructions, prompts, and additional calibration matrices are supplied in the supplementary material. Benchmark-Grounded Skill Evolution Skill Design Objectives Our method follows three objectives. Diagnostic grounding requires every revision to address recurring, observable item-level failures rather than unconstrained self-reflection. Generalizable improvement favors compact rules that transfer across queries and model families instead of fitting one output. Leakage-safe evolution proposes and selects revisions only on external cases; the full FinReportBench test set is used after the skill is frozen. Model weights remain unchanged throughout. Evolution Pipeline Figure 4 shows the complete pipeline. Starting from skill KtK_t, several report models generate outputs for external financial-report cases. FinReportBench converts their rendered reports into item-level decisions and observable evidence. An optimizer turns recurring failures into a candidate Kt+1K_t+1, organized as planning, writing, and review rules. Query-disjoint external validation either accepts the candidate or retains KtK_t. When improvement stops, the best accepted skill is frozen as K⋆K and evaluated once on a locked benchmark subset against a paired no-skill condition. Figure 4: Benchmark-grounded skill evolution. External cases support iterative diagnosis, revision, and validation; the selected skill is frozen before paired evaluation on FinReportBench. The evolution and validation sets cover the same broad financial-report setting but share no query, trajectory, or source document with the benchmark. We use five external discovery cases and five disjoint external validation cases, balanced across Chinese and English and covering company, industry, and macro research. Using multiple report models exposes failures that recur across model families while holding each model’s generation settings fixed. Failure-Grounded Skill Revision For each external report, the evaluator returns a categorical decision and short visual or textual evidence for every applicable item. We aggregate outcomes across cases and models and revise only failures with sufficient cross-model support. The optimizer receives item semantics, aggregate outcomes, and anonymized evidence, but not item identifiers, weights, judge prompts, benchmark queries, or hidden source packets. The optimizer contrasts failed and passed evidence and writes compact constraints with three parts: when a rule applies, what the generator should do, and how the output can be checked. The resulting skill has plan, write, and review blocks. A critic removes task-specific facts, rubric wording, duplication, and conflicting instructions, and rejects rules that require unsupported analyst identities, qualifications, ratings, or target prices. This keeps the skill actionable without copying the rubric into a long prompt. External Validation and Freezing Each candidate is tested on held-out external cases. It is accepted only when targeted items improve without reducing deliverability, report identity, institutional completeness, or materially regressing non-target items; otherwise the previous skill remains active. This failure-driven update follows the general pattern of Alzubi and others (2026), but uses rendered financial reports and hierarchical item evidence as feedback. Evolution stops after a fixed round budget or a patience window without validation improvement. The best accepted revision, rather than the last proposal, becomes K⋆K . Only then does each report model receive the locked FinReportBench evaluation under paired no-skill and K⋆K conditions, with query, trajectory, generation budget, and inference settings held fixed. Test queries and their order are frozen before generation; selection does not use report or judge outcomes. We measure changes in G0, G1, G2, item pass rates, the provisional total, non-target regressions, generation cost, and skill length. Exact external-case composition, support and acceptance thresholds, stopping parameters, skill budget, optimizer and critic prompts, and pseudocode are provided in the supplementary material. Experiments Setup The completed run evaluates nine recent model families from international and Chinese providers: DeepSeek V4 Flash, DeepSeek V4 Pro, Qwen 3.7 Max, GLM-5.2, Kimi K2.6, MiniMax M2.5, and MiniMax M2.7. The run additionally includes MiniMax M3 and Qwen 3.6 27B. Every system is evaluated on FinReportBench in the no_skill condition. We record model identifiers, run date, and inference settings in the supplementary material. We render every output and evaluate extracted text together with up to four rendered pages using GPT-5.6 Luna at medium reasoning effort and the frozen 35-item rubric. All systems share the same judge prompt, rubric, and scoring code. Tasks, trajectories, and the output contract are also fixed before the run. Main Benchmark Results Every system is evaluated on the same benchmark tasks, while visible generation and encoding defects remain system errors. The provisional hierarchical-score ordering is MiniMax M2.7 (22.6), GLM-5.2 (21.8), MiniMax M2.5 (21.1), DeepSeek V4 Flash (20.2), Qwen 3.7 Max (19.1), DeepSeek V4 Pro (18.8), MiniMax M3 (18.6), Kimi K2.6 (17.7), and Qwen 3.6 27B (15.1). MiniMax M2.7 has the highest G1 score (40.0) and G2 score (46.6). Because the total remains provisional, we report G1 and G2 alongside it and do not interpret small score gaps as stable ability differences. Table 1 presents every rubric item and the provisional total. Table 1: Item-level FinReportBench results across all 35 rubric items. Every system is evaluated on the same benchmark tasks, and visible output defects remain included. Cells report mean item credit (pass=100, partial=50, fail=0). Overall is the provisional hierarchical score. G0: Deliverability G1: Report identity G2: Institutional completeness Overall System D01 D02 D03 D04 R01 R02 R03 R04 C01 C02 C03 C04 C05 C06 C07 C08 C09 C10 C11 C12 C13 C14 C15 C16 C17 C18 C19 C20 C21 C22 C23 C24 C25 C26 C27 All MiniMax M2.7 100 99 100 100 100 1 51 22 56 100 86 51 96 7 6 50 0 0 0 47 0 66 0 3 18 100 100 90 82 100 67 50 85 31 18 22.6 GLM-5.2 100 99 100 100 100 0 51 21 2 100 91 51 98 5 8 50 0 0 0 51 0 77 0 4 10 100 100 95 89 100 51 50 88 23 3 21.8 MiniMax M2.5 100 100 100 100 100 0 49 19 54 99 84 51 88 2 3 50 0 0 0 47 0 67 0 0 9 100 100 85 77 100 45 50 97 46 41 21.1 DeepSeek V4 Flash 100 100 100 100 100 0 52 10 78 99 83 50 96 7 2 50 0 0 0 50 0 86 0 0 11 100 100 84 84 100 51 50 92 37 5 20.2 Qwen 3.7 Max 100 100 99 100 99 0 50 12 19 99 87 50 95 5 3 52 0 0 0 49 0 61 0 1 13 100 100 87 83 100 48 49 75 32 1 19.1 DeepSeek V4 Pro 100 100 100 100 100 0 48 9 65 99 82 50 96 1 1 50 0 0 0 49 0 83 0 0 3 100 100 83 88 99 65 50 61 27 6 18.8 MiniMax M3 100 100 97 100 100 1 50 11 2 100 88 50 96 2 3 50 0 0 0 51 0 52 0 0 9 100 100 92 88 100 52 50 79 42 15 18.6 Kimi K2.6 100 100 100 100 100 0 50 4 32 100 89 52 91 1 1 50 0 0 0 48 0 60 0 0 3 100 100 75 89 100 46 50 95 39 25 17.7 Qwen 3.6 27B 94 95 82 100 97 0 47 10 48 98 79 47 87 4 0 50 0 0 0 43 0 47 0 0 3 97 91 82 86 95 52 50 80 47 39 15.1 Human–Evaluator Alignment We conduct a calibration-scale expert validation using relative judgments rather than exhaustive item-level gold labels. Three senior financial experts jointly review 18 anonymized reports from three queries and establish a consensus three-tier partial order: three authentic source reports rank above two generated reports judged to have recognizable report form, which in turn rank above the remaining 13 generations. Expanding only cross-tier relations yields 71 directed pairwise constraints. For each evaluator run, a constraint is satisfied when the hierarchical score of the expert-preferred report is strictly higher. Figure 5: Alignment with the three-expert consensus partial order. GPT5.6-Luna and GPT5.6-Terra satisfy all 71 constraints, while Grok-4.5 satisfies 70. The axis is truncated to distinguish the near-ceiling results. All three evaluator families preserve 98.6–100.0% of the expert partial-order constraints. A separate Terra repeat agrees with the reference run on 97.0% of item decisions, while Luna and Grok agree with the Terra runs on 92.1–93.8%; corresponding score-ranking correlations range from 0.949 to 0.981. Together, the expert alignment and cross-family agreement support the criterion validity and reliability of the automatic evaluator. Hierarchical Diagnostic Profiles The leaderboard establishes ranking; the hierarchical profile explains how systems differ. We aggregate the 35 items into deliverability, report identity, front-page framing, institutional identity, compliance, page systems, source and chart discipline, and generation-trace control. Figure 6 shows five representative models in the main paper; profiles for all models will appear in the supplementary material. Figure 6: Hierarchical component profiles for five representative models. The radial axes aggregate related rubric items while preserving the G0–G1–G2 interpretation. Cross-model gaps are largest for generation-trace control (31.1 points), followed by information density (13.4) and data discipline (13.2). The report-frame gap is only 4.6 points. Performance Across Benchmark Slices We next test whether conclusions depend on one language or research setting. We report Chinese versus English, company versus industry versus macro, and open-ended T0 versus thesis-guided T1 requests. Figure 7 displays each slice as a deviation from the same model’s overall score, preventing globally strong systems from dominating the visual scale. Absolute scores, sample sizes, and confidence intervals remain available in the supplementary tables. Figure 7: Performance variation across language, research object, and query-specificity slices. Cells show deviation from each model’s overall score. We use these slices diagnostically rather than define separate benchmarks. The largest model-relative slice deviation is 4.1 points. Rankings are highly stable across T0 and T1 requests (ρ=0.983ρ=0.983 and 1.0001.000) and remain similar across company, industry, and macro tasks (ρ=0.833ρ=0.833, 0.8330.833, and 0.9000.900). Language is the largest source of rank variation (ρ=0.650ρ=0.650 for Chinese and 0.9170.917 for English). Item Difficulty and Discrimination A useful benchmark should contain neither only trivial checks nor only unattainable requirements. We therefore measure each item’s pass rate and model-discrimination strength, then cluster item failures by their co-occurrence across reports. Figure 8 separates item difficulty from discrimination and reveals whether recurring failures form coherent institutional components. Figure 8: Item analysis. Left: pass rate and model discrimination for all 35 items. Right: failure co-occurrence aggregated by rubric component. Mean item credit spans 0.0–99.9%, confirming that the rubric contains both near-universal delivery checks and consistently absent institutional requirements. The maximum between-model standard deviation is 25.9 points. Table 1 provides the complete auditable item matrix. Skill-Evolution Evaluation Finally, we test whether benchmark diagnostics can improve generation. Five external discovery cases produce the initial skill K0K_0 and candidate revisions; five disjoint external validation cases determine which candidates are retained. After K⋆K is frozen, a locked FinReportBench evaluation compares no skill and K⋆K across Qwen 3.7 Max, DeepSeek V4 Pro, DeepSeek V4 Flash, MiniMax M3, and GLM-5.2 under 100 paired task–model comparisons. Figure 9: Validation performance across skill-evolution stages for five report models. Filled markers are retained versions and open markers are rejected candidates. On the locked FinReportBench evaluation, K⋆K improves mean G1 by 33.85 points (95% CI: 31.17–36.63) and mean G2 by 13.83 points (12.65–15.02), while preserving G0 for every pair. All five models improve: G1 gains range from 26.39 to 41.86 points and G2 gains from 11.22 to 16.98 points. The larger G1 gain shows that the skill primarily repairs report identity and document structure, while the consistent G2 gain shows that it also strengthens institutional completeness. The shared direction across model families demonstrates transfer beyond a single generator. Discussion and Conclusion FinReportBench connects measurement, diagnosis, and improvement for institution-grade financial report generation. Its results show that professional quality is a staged decision: an artifact must first be deliverable, then recognizable as the target document genre, and finally complete in its institutional components. This hierarchy explains why fluent outputs can remain unusable in practice, while the itemized rubric turns recurrent failures into concrete targets for skill distillation. The locked evaluation further shows that these diagnostics can be converted into reusable constraints that improve report identity and institutional completeness across model families. References A. Alzubi et al. (2026) EvoSkill: automated skill discovery for multi-agent systems. arXiv preprint arXiv:2603.02766. External Links: Link Cited by: Related Work, External Validation and Freezing. A. Asai et al. (2024) OpenScholar: synthesizing scientific literature with retrieval-augmented language models. arXiv preprint arXiv:2411.14199. External Links: Link Cited by: Related Work. J. Baek, S. K. Jauhar, S. Cucerzan, and S. J. Hwang (2024) ResearchAgent: iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738. External Links: Link Cited by: Related Work. Y. Bian, X. Lin, Y. Xie, T. Liu, M. Zhuge, S. Lu, H. Tang, J. Wang, J. Zhang, J. Chen, et al. (2025) You don’t know until you click: automated GUI testing for production-ready software evaluation. arXiv preprint arXiv:2508.14104. Cited by: Related Work. Y. Chen, Y. Wu, S. Shen, Y. Xie, L. Shen, H. Xiong, and Y. Luo (2025) ChartMark: a structured grammar for chart annotation. In 2025 IEEE Visualization and Visual Analytics (VIS), p. 311–315. Cited by: Evaluation Construction Pipeline. Z. Chen, W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T. Huang, B. Routledge, and W. Y. Wang (2021) FinQA: a dataset of numerical reasoning over financial data. In Proceedings of EMNLP, External Links: Link Cited by: Related Work. Z. Chen, S. Li, C. Smiley, Z. Ma, S. Shah, and W. Y. Wang (2022) ConvFinQA: exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of EMNLP, External Links: Link Cited by: Related Work. W. Chiang et al. (2024) Chatbot arena: an open platform for evaluating llms by human preference. In Proceedings of ICML, External Links: Link Cited by: Itemized Multimodal Evaluation. X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su (2023) Mind2Web: towards a generalist agent for the web. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Related Work. B. Gou et al. (2025) Mind2Web 2: evaluating agentic search with agent-as-a-judge. arXiv preprint arXiv:2506.21506. External Links: Link Cited by: Related Work. Z. Huang et al. (2026) From raw experience to skill consumption: a systematic study of model-generated agent skills. arXiv preprint arXiv:2605.23899. External Links: Link Cited by: Related Work. P. Islam, A. Kannappan, D. Kiela, R. Qian, N. Scherrer, and B. Vidgen (2023) FinanceBench: a new benchmark for financial question answering. arXiv preprint arXiv:2311.11944. External Links: Link Cited by: Related Work. B. Jin et al. (2025) Search-r1: training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. External Links: Link Cited by: Related Work. J. Jin, Y. Zhang, Y. Xu, H. Qian, Y. Zhu, and Z. Dou (2026) FinSight: towards real-world financial deep research. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), External Links: Link Cited by: Introduction, Introduction, Related Work. S. Kantharaj, X. L. Do, R. T. K. Leong, J. Q. Tan, E. Hoque, and S. Joty (2022) OpenCQA: open-ended question answering with charts. In Proceedings of EMNLP, External Links: Link Cited by: Evaluation Construction Pipeline. S. Kim et al. (2024) The biggen bench: a principled benchmark for fine-grained evaluation of language models with language models. arXiv preprint arXiv:2406.05761. External Links: Link Cited by: Related Work. B. Li, Y. Peng, Y. Xie, S. Lu, Y. Zhu, X. Mu, X. Liu, and Y. Luo (2026) DeepEye: a steerable self-driving data agent system. In Companion of the International Conference on Management of Data, p. 74–77. Cited by: Related Work. X. Li et al. (2026) SkillsBench: benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670. External Links: Link Cited by: Related Work. C. Lifan, W. Ding, J. Yang, X. Zhou, J. Chen, and F. Lin (2026) Cogito: a cognitive agentic framework driven by dynamic graph of thoughts for financial report generation. In Findings of the Association for Computational Linguistics: ACL 2026, p. 8846–8860. External Links: Document, Link Cited by: Introduction, Introduction, Related Work. J. Liu et al. (2025) FinMME: benchmark dataset for financial multi-modal reasoning evaluation. arXiv preprint arXiv:2505.24714. External Links: Link Cited by: Related Work. A. Masry, P. Kavehzadeh, X. L. Do, E. Hoque, and S. Joty (2023) UniChart: a universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761. External Links: Link Cited by: Evaluation Construction Pipeline. A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque (2022) ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In Findings of ACL, External Links: Link Cited by: Evaluation Construction Pipeline. N. Methani, P. Ganguly, M. M. Khapra, and P. Kumar (2020) PlotQA: reasoning over scientific plots. In Proceedings of WACV, External Links: Link Cited by: Evaluation Construction Pipeline. G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom (2024) GAIA: a benchmark for general ai assistants. In Proceedings of ICLR, External Links: Link Cited by: Related Work. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. (2021) WebGPT: browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. External Links: Link Cited by: Related Work. B. Pan, Y. Fu, K. Wang, J. Lu, L. Pan, Z. Qian, Y. Chen, G. Wang, Y. Zhou, L. Zheng, Y. Tang, Z. Wen, Y. Wu, J. Lu, B. Zhu, M. Zhu, B. Zhang, and W. Chen (2025) VIS-Shepherd: constructing critic for LLM-based data visualization generation. arXiv preprint arXiv:2506.13326. External Links: Link Cited by: Evaluation Construction Pipeline. Y. Qu et al. (2025) FinFlier: automating graphical overlays for financial visualizations with knowledge-grounding large language models. IEEE Transactions on Visualization and Computer Graphics (TVCG). External Links: Link Cited by: Pilot Expert Interviews. V. Reddy, R. Koncel-Kedziorski, V. D. Lai, M. Krumdick, C. Lovering, and C. Tanner (2024) DocFinQA: a long-context financial reasoning dataset. In Proceedings of ACL, External Links: Link Cited by: Related Work. S. Shen et al. (2026) SKILLFOUNDRY: building self-evolving agent skill libraries from heterogeneous scientific resources. arXiv preprint arXiv:2604.03964. External Links: Link Cited by: Related Work. H. Tan et al. (2024a) PROXYQA: an alternative framework for evaluating long-form text generation. In Proceedings of ACL, External Links: Link Cited by: Related Work. S. Tan et al. (2024b) JudgeBench: a benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784. External Links: Link Cited by: Itemized Multimodal Evaluation. Y. Tang, X. Liu, B. Zhang, T. Lan, Y. Xie, J. Lao, Y. Wang, H. Li, T. Gao, B. Pan, L. Weng, X. Huang, M. Zhu, Y. Feng, Y. Luo, and W. Chen (2026a) IGenBench: benchmarking the reliability of text-to-infographic generation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, External Links: Link Cited by: Evaluation Construction Pipeline. Y. Tang, Y. Xie, Y. Feng, T. Lan, J. Lao, Y. Cheng, and W. Chen (2026b) ViviDoc: generating interactive documents through human-agent collaboration. arXiv preprint arXiv:2603.27991. External Links: Link Cited by: Related Work. Y. Tang, Y. Xie, Y. Feng, J. Lao, T. Lan, and W. Chen (2026c) Demonstrating chart-plot: closing the last mile of academic chart generation. arXiv preprint arXiv:2606.09174. External Links: Link Cited by: Pilot Expert Interviews. J. Wei et al. (2025) BrowseComp: a simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516. External Links: Link Cited by: Related Work. Z. Wen, L. Weng, Y. Tang, R. Zhang, Y. Liu, B. Pan, M. Zhu, and W. Chen (2026) Exploring multimodal prompt for visualization authoring with large language models. IEEE Transactions on Visualization and Computer Graphics. External Links: Document, Link Cited by: Pilot Expert Interviews. L. Weng, Y. Tang, Y. Feng, Z. Chang, R. Chen, H. Feng, C. Hou, D. Huang, Y. Li, H. Rao, H. Wang, C. Wei, X. Yang, Y. Zhang, Y. Zheng, X. Huang, M. Zhu, Y. Ma, B. Cui, P. Chen, and W. Chen (2025) DataLab: a unified platform for LLM-powered business intelligence. In Proceedings of the IEEE International Conference on Data Engineering, p. 4346–4359. External Links: Link Cited by: Related Work. J. Wu et al. (2025) WebDancer: towards autonomous information seeking agency. arXiv preprint arXiv:2505.22648. External Links: Link Cited by: Related Work. S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann (2023) BloombergGPT: a large language model for finance. arXiv preprint arXiv:2303.17564. External Links: Link Cited by: Related Work. Y. Wu et al. (2024) LongGenBench: benchmarking long-form generation in long-context llms. arXiv preprint arXiv:2409.02076. External Links: Link Cited by: Related Work. Q. Xie, W. Han, X. Zhang, Y. Lai, M. Peng, A. Lopez-Lira, and J. Huang (2023) PIXIU: a large language model, instruction data and evaluation benchmark for finance. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Related Work. Q. Xie et al. (2024) FinBen: a holistic financial benchmark for large language models. arXiv preprint arXiv:2402.12659. External Links: Link Cited by: Related Work. Y. Xie, Y. Luo, G. Li, and N. Tang (2024) HaiChart: human and AI paired visualization system. arXiv preprint arXiv:2406.11033. Cited by: Related Work. Y. Xie, C. Ma, Z. Wang, L. Wang, J. Zhu, C. Zeng, Z. Shen, B. Li, and Y. Luo (2026) DataMagic: transforming tabular data into data insight video. arXiv preprint arXiv:2606.20388. Cited by: Related Work. Y. Xie, Z. Zhang, Y. Wu, S. Lu, J. Zhang, Z. Yu, J. Wang, S. Hong, B. Liu, C. Wu, et al. (2025) VisJudge-Bench: aesthetics and quality assessment of visualizations. arXiv preprint arXiv:2510.22373. Cited by: Evaluation Construction Pipeline. H. Zhang et al. (2026) CoEvoSkills: self-evolving agent skills via co-evolutionary verification. arXiv preprint arXiv:2604.01687. External Links: Link Cited by: Related Work. Y. Zhao, Y. Li, C. Li, and R. Zhang (2022) MultiHiertt: numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of ACL, External Links: Link Cited by: Related Work. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, et al. (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Itemized Multimodal Evaluation. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2024) Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, External Links: Link Cited by: Itemized Multimodal Evaluation. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, et al. (2024) WebArena: a realistic web environment for building autonomous agents. In Proceedings of ICLR, External Links: Link Cited by: Related Work. F. Zhu, W. Lei, Y. Huang, C. Wang, S. Zhang, J. Lv, F. Feng, and T. Chua (2021) TAT-qa: a question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of ACL, External Links: Link Cited by: Related Work. M. Zhuge et al. (2024) Agent-as-a-judge: evaluate agents with agents. arXiv preprint arXiv:2410.10934. External Links: Link Cited by: Itemized Multimodal Evaluation.