Paper deep dive

Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain

Motaz Saad, Anna Borrelli, Ivan Gentile, Kianna Kazemi, Francesco Piccialli, Antonella Longo

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 76

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Environmental, Social, and Governance (ESG) reporting is critical for corporate accountability, with Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) offering strong potential to automate KPI extraction. However, open-source LLM performance in domain-specific ESG tasks remains insufficiently understood. This paper evaluates open-source LLMs in ESG contexts using a structured framework and evaluation resource based on 498 real-world ESG reports from EU-listed companies (2010-2024). We evaluate seven open-source models (2B to 30B parameters) -- glm-4.7-flash, nemotron-3-nano:4b, qwen3:4b-instruct, gemma3:4b, gemma4:e4b, gemma4:e2b, and ministral-3:8b -- using 100 persona-based synthetic QA pairs covering ESG information needs. System performance is assessed via RAGAS metrics, including contextual recall, precision, relevance, faithfulness, answer relevancy, and factual correctness. Results show notable performance variations across architectures. Retrieval performance is strong across models (context recall around 0.58-0.61, context precision around 0.78-0.81, context relevance 0.965-0.985). Generation diverges most on faithfulness (0.607-0.822) and least on answer relevancy (0.760-0.881): glm-4.7-flash leads in faithfulness (0.822), qwen3 in factual correctness (0.449), and ministral-3 in answer relevancy (0.881). Low overall factual correctness (0.387-0.449) highlights the need for domain-specific fine-tuning. This work provides data-driven guidance for deploying open-source models in ESG reporting.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

72,920 characters extracted from source content.

Expand or collapse full text

Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain Motaz Saad Affiliation: University of Salento motazk.saad@unisalento.it ORCID: 0000-0002-1080-7276 Anna Borrelli Affiliation: University of Naples Federico I anna.borrelli3@unina.it Ivan Gentile Affiliation: IFAB Foundation ivan.gentile@ifabfoundation.org Kianna Kazemi Affiliation: IFAB Foundation kianna.kazemi@ifabfoundation.org Francesco Piccialli Affiliation: University of Naples Federico I francesco.piccialli@unina.it Antonella Longo Affiliation: University of Salento antonella.longo@unisalento.it Abstract Environmental, Social, and Governance (ESG) reporting has emerged as a critical component of corporate accountability, with Large Language Models (LLMs) through Retrieval-Augmented Generation (RAG) showing significant potential for automating KPI extraction from reports. However, the performance characteristics of open-source LLMs in ESG domain-specific tasks remain inadequately understood, creating challenges for informed model selection. This paper addresses the performance characteristics of open-source LLM deployment in ESG contexts through a structured evaluation framework and an ESG-RAG evaluation resource grounded in 498 real-world ESG reports. We conduct rigorous performance evaluation of seven open-source models ranging from 2B to 30B parameters using a corpus of 498 ESG reports from EU-listed companies (2010–2024), from which 100 synthetic QA pairs are drawn for evaluation from a larger 284-pair persona-based synthetic dataset covering Environmental, Social, and Governance information needs. Our evaluation employs persona-driven query generation and established RAGAS metrics including contextual recall, precision, relevance, faithfulness, answer relevancy, and factual correctness. Our evaluation of glm-4.7-flash (30B MoE), nemotron-3-nano:4b (4B), qwen3:4b-instruct (4B), gemma3:4b (4B), gemma4:e4b (8B), gemma4:e2b (2B), and ministral-3:8b (8B) reveals notable differences in performance characteristics across different model architectures and sizes. Retrieval is strong but not perfect under LLM-based judging (context recall ≈ 0.58–0.61, context precision ≈ 0.78–0.81, context relevance 0.965–0.985, LLM-judged context precision with reference ≈ 0.83–0.88). Generation diverges most on faithfulness (0.607–0.822) and least on answer relevancy (0.760–0.881): glm-4.7-flash leads faithfulness (0.822), qwen3 leads factual correctness (0.449), and ministral-3 leads answer relevancy (0.881). Factual correctness across all models (0.387–0.449) highlights ongoing opportunities for domain-specific fine-tuning. This evaluation provides organizations with data-driven insights for selecting open-source models tailored to their specific ESG reporting requirements. Index Terms: LLM, open-source models, performance evaluation, RAG, ESG I Introduction ESG reporting refers to the disclosure of data related to a company’s environmental, social, and governance practices. Originally rooted in ethical investing, it has evolved into a key framework for assessing non-financial risks and performance that can influence a company’s long-term sustainability and investor attractiveness [31]. ESG reporting demonstrates a company’s commitment to sustainability, ethical conduct, and regulatory compliance. Investors, policymakers, and other stakeholders increasingly depend on ESG disclosures to assess corporate performance beyond traditional financial indicators, making the accurate and efficient extraction of ESG Key Performance Indicators (KPIs) a pressing priority. However, ESG data is often complex, heterogeneous, and embedded in diverse, unstructured documents, creating significant challenges in achieving consistency and reliability. The environmental component of the ESG evaluates how a company affects and is affected by the natural world, focusing on metrics such as carbon emissions (including Scope 1, 2, and 3), energy use, water consumption, waste management, and climate risk strategies. The social dimension centers on the company’s relationships with key stakeholders, including employees, suppliers, and communities. It addresses labor practices, workplace safety, human rights, diversity and inclusion, community involvement, and consumer protection. The governance aspect pertains to the systems and structures guiding corporate behavior, including board composition and diversity, executive compensation, transparency, shareholder rights, and anti-corruption policies. Together, these three pillars form a comprehensive lens through which organizations are evaluated for long-term sustainability and ethical stewardship [29]. Another dimension of complexity is that there are different ESG Reporting Standards and Frameworks. Table I summarize some of the most common frameworks. TABLE I: Summary of Major ESG Reporting Frameworks Framework Description GRI [19] Global Reporting Initiative: Most widely used ESG framework, focused on a company’s overall sustainability impact across environmental, social, and economic dimensions. SASB [33] Sustainability Accounting Standards Board: Provides industry-specific standards that highlight financially material ESG issues relevant to investors. TCFD [35] Task Force on Climate-related Financial Disclosures: Offers guidance on disclosing climate-related financial risks and opportunities, including governance and scenario analysis. CDP [10] Carbon Disclosure Project: Specializes in environmental disclosures, particularly greenhouse gas emissions, water security, and forest risks. CSRD [14] Corporate Sustainability Reporting Directive: European Union regulation requiring standardized ESG disclosures from large companies with third-party assurance and digital tagging. ESRS [12] European Sustainability Reporting Standards: 12 EFRAG-developed standards (ESRS 1–2 cross-cutting, ESRS E1–E5 environmental, ESRS S1–S4 social, ESRS G1 governance) that operationalise CSRD disclosure requirements under a double-materiality lens. Artificial Intelligence (AI), especially Large Language Models (LLMs), has emerged as a powerful tool for automating ESG KPI extraction, streamlining data processing, and improving reporting accuracy. Retrieval-Augmented Generation (RAG) techniques further enhance this capability by integrating relevant external data sources into LLM outputs, thereby reducing hallucination and increasing the precision of ESG insights. Open-source models offer compelling advantages for ESG applications, including deployment flexibility and customization potential. However, the performance characteristics of open-source models in ESG domain-specific tasks remain inadequately understood, creating challenges for informed model selection. This paper addresses this gap by conducting a comprehensive empirical evaluation of open-source LLMs within RAG frameworks applied to ESG KPI extraction. It provides an assessment framework that evaluates model performance across multiple dimensions including contextual understanding, faithfulness, and factual correctness, enabling organizations to select appropriate open-source solutions tailored to their specific ESG reporting needs. The paper aims to (1) highlight the growing importance of AI-driven ESG KPI extraction in enhancing sustainability reporting, (2) evaluate the performance characteristics of open-source LLMs in ESG domain-specific RAG applications through systematic benchmarking, (3) analyze the practical implications and deployment considerations of employing open-source models for this purpose, and (4) offer comparative insights to guide organizations toward technically effective open-source solutions for their ESG reporting requirements. By focusing on both the performance and practical dimensions of open-source AI adoption in ESG analytics, this paper provides valuable guidance for decision-makers seeking to balance technical capabilities with deployment flexibility and optimize resource allocation in sustainability initiatives. The evaluation framework enables organizations to make informed decisions that account for both the quality of ESG information extraction and the practical advantages of open-source solutions, ensuring sustainable and scalable AI deployment strategies. This work makes two contributions. First, it provides an empirical evaluation of five open-source LLMs for ESG-focused RAG under a controlled retrieval setup. Second, it introduces an ESG-RAG evaluation resource (available at https://github.com/motazsaad/ESG-RAG-benchmark-ragas) consisting of a processed corpus of 498 ESG reports, 284 synthetic QA pairs grounded in source documents with persona-based coverage across ESG pillars, and accompanying evaluation outputs and scripts to support reproducible comparison. I State of the Art Retrieval-Augmented Generation (RAG) is a cutting-edge architecture in NLP that fuses document retrieval with language generation, addressing a key limitation of large language models (LLMs): their inability to access external, up-to-date knowledge at inference time. Originally proposed by Facebook AI Research, RAG systems leverage real-time access to external corpora, enabling models to generate more factually accurate, context-aware answers [23]. Figure 1 shows the key components of a typical RAG system: • Retriever: Selects relevant documents. The retrieval quality directly impacts generation accuracy. • Generator: A pretrained LLM that synthesizes an output based on the retrieved documents plus the query. User QueryRetrieverDocument StoreLLM GeneratorFinal AnswerTop-K Docs Fig. 1: RAG pipeline The Mechanism of Retrieval-Augmented Generation (RAG) is as follows: 1. User Query: Let the input be a natural language query q. 2. Retriever: A dense retriever encodes the query into a vector and retrieves the top k documents from an external corpus D. This can be formalized as: dii=1k=Retriever(q,D)\d_i\_i=1^k=Retriever(q,D) where each did_i is a document retrieved based on semantic similarity. 3. Generator: For each retrieved document did_i, the generator model produces a response conditioned on the pair (q,di)(q,d_i): yi=Generator(q,di)y_i=Generator(q,d_i) This architecture enables dynamic knowledge integration without retraining the generator, as updating the external data is sufficient to refresh the model’s knowledge. RAG has evolved beyond the original retrieve-then-generate paradigm. Gao et al. [17] categorize its development into three generations—Naive, Advanced, and Modular RAG—showing how pre-retrieval query rewriting, post-retrieval re-ranking, and adaptive generation progressively improve factual grounding. These architectural advances make RAG the preferred approach for knowledge-intensive tasks requiring access to specialized corpora such as ESG reports. I-A RAG Evaluation Frameworks Systematic RAG evaluation has been formalized through dedicated frameworks. Es et al. propose RAGAS [15], a reference-free evaluation library measuring faithfulness, answer relevancy, context precision, and context recall using LLM-as-judge scoring—the framework adopted in this study. Saad-Falcon et al. introduce ARES [32], which fine-tunes lightweight LM judges on synthetic training data for automated pipeline assessment. In addition to research-oriented frameworks such as RAGAS and ARES, practitioner-oriented tooling such as DeepEval supports end-to-end and component-level RAG evaluation with metrics for answer relevancy, faithfulness, contextual relevancy, precision, and recall [11]. More broadly, the use of LLMs as evaluators has been studied in general NLG and LLM benchmarking settings through frameworks such as G-Eval [25], JudgeLM [42], and Prometheus [22], while recent survey work highlights both the practical value of LLM-as-a-judge systems and their reliability challenges [20]. Lima et al. [37] provide a taxonomic framework for synthetic benchmark generation in domain-specific RAG, demonstrating that dataset diversity and generation strategy substantially affect evaluation validity—a finding that directly motivates the persona-driven, quality-filtered protocol used in our evaluation dataset construction. I-B LLMs and RAG for ESG Applications The application of LLMs to ESG analysis has attracted growing research attention across extraction, generation, and benchmarking tasks. ESGReveal [43] applies RAG with GPT-4 to extract structured ESG data from 166 Hong Kong company reports, demonstrating strong retrieval-augmented extraction performance but evaluating only a single proprietary model. Climate Finance Bench [26] benchmarks RAG pipelines over corporate climate disclosure PDFs with expert-annotated QA pairs, finding that even the strongest proprietary system achieves only 62% accuracy—highlighting task difficulty and leaving open-source models entirely unevaluated. On the open-source side, SusGen-GPT [39] fine-tunes 7–8B open-source LLMs on a multi-task ESG and financial corpus, showing that small open-source models can approach proprietary performance on sustainability report generation, but without evaluating them within a RAG pipeline. Birti et al. [7] optimize Llama and Gemma models for ESG activity detection and EU taxonomy classification, confirming competitive open-source performance on classification tasks, again without a RAG evaluation setting. Benchmarks like ESGenius [21] and MMESGBench [41] demonstrate that RAG significantly boosts LLM performance on ESG tasks, especially for smaller models, while multimodal approaches outperform text-only systems on visually grounded disclosures. Fine-tuning datasets such as ESG-CID [1] enable domain-specific retrieval optimization. Practical tools like ESG-Consultant [28] show promise in compliance support, while ESGLens [40] extends ESG-focused RAG toward interactive report analysis with source traceability and score prediction. Studies in low-resource languages—such as Turkish ESG report analysis—reveal that traditional IR methods like BM25 can outperform transformer-based retrievers when paired with proprietary generators [4]. I-C Research Gap Despite these advances, there remains limited work comparing multiple open-source LLMs as generators within a shared ESG RAG pipeline under controlled retrieval, prompting, and evaluation conditions. Existing work either benchmarks ESG knowledge more broadly with RAG as one condition rather than as a controlled cross-model pipeline study [21, 41], introduces ESG QA benchmarks or domain resources without a shared open-source generator comparison over one pipeline configuration [1], or presents ESG-focused RAG systems without systematic cross-family generator evaluation [43, 26, 40]. This gap makes it difficult for organizations to make evidence-based decisions about open-source model selection for ESG applications under practical local-deployment constraints. I-D ESG Reports: Structure and Data Extraction Challenges ESG reports follow established frameworks such as GRI, SASB, TCFD, and ESRS [10, 19, 14] to ensure comparability and transparency while focusing on material topics most relevant to stakeholders. Key Structural Elements: ESG reports typically include executive summaries with CEO messages; company profiles and ESG strategy alignment; governance structures and risk management frameworks; materiality assessments identifying priority topics; environmental performance data (emissions, resource use, climate targets); social performance metrics (workforce diversity, human rights, community impact); governance disclosures (board composition, executive compensation tied to ESG); quantitative KPIs with benchmarking; framework alignment tables; and third-party assurance statements. Data Presentation Formats: ESG reports employ diverse layouts including structured tables for KPI comparisons and framework mappings, infographics and visual narratives for stakeholder engagement, interactive dashboards in digital formats, materiality matrices plotting stakeholder concerns against business impact, charts and graphs for trend visualization, and geographic maps showing operational impacts. Data Extraction Challenges: Extracting ESG KPIs from these varied formats presents significant automated processing challenges. Key difficulties include unstructured narrative text with non-standardized phrasing, multi-modal content combining text and visuals that resist traditional parsing methods, inconsistent table formatting across companies and sectors, visual data encoding through charts and color-coding, and variable language expression where identical metrics may be described differently (e.g., ”CO2 footprint reduced by 15%” versus ”Scope 1 emissions fell by 15%”). A typical extraction challenge occurs when emission values appear within pie charts without clear legends, while accompanying narratives provide only vague statements like ”value-chain emissions saw a significant drop,” with actual figures relegated to footnotes. This complexity necessitates advanced techniques combining visual parsing, natural language understanding, and contextual inference for effective ESG KPI retrieval. I Methodology I-A Evaluation framework (RAGAS) We evaluate with RAGAS [15], an open-source library designed specifically for RAG assessment. RAGAS provides metric implementations for: (i) generation quality—faithfulness (does the answer stay grounded in retrieved context?) and answer relevancy (does it answer the user’s question?); and (i) retrieval quality—context precision (signal-to-noise in retrieved chunks) and context recall (were all relevant chunks retrieved). We use the library’s evaluate() API to compute scores over our evaluation set (details below). RAGAS uses an LLM-judge to score most metrics (reference-free), while context recall relies on ground-truth annotations or synthesized gold answers to estimate true positives and false negatives. The evaluation metrics that we use in our evaluation include: • Faithfulness: penalizes hallucinations by checking whether answer claims are supported by the retrieved passages. • Answer relevancy: measures how directly and completely the answer addresses the query. • Context precision: fraction of retrieved content that is actually useful (high precision leads to less distractor text). • Context recall: fraction of all relevant content that was retrieved (high recall leads to fewer misses). Together, these metrics expose common failure modes: low precision (over-stuffed context), low recall (missed evidence), low faithfulness (hallucination), and low answer relevancy (off-topic or incomplete answers). The RAG triad as shown in Figure 2 is composed of three RAG evaluation metrics: answer relevancy, faithfulness, and contextual relevancy. Each metric corresponds to a specific component of the RAG pipeline, making the triad a useful diagnostic tool for identifying which component limits overall performance. Beyond these four core metrics, our evaluation further incorporates LLM Context Precision With Reference, Context Relevance, and Factual Correctness; the complete set of seven metrics is detailed in Section I-E6. InputActualOutputRetrievedContent Answer Relevance how relevant the generated answer is to the input Contextual Relevance how relevant the retrieved context is to the input RAG Triad Faithfulness how faithful the generated answer is to the retrieved content Fig. 2: RAG Triad Figure 3 illustrates that RAG evaluation requires measuring both components separately. Metrics must ensure the retriever finds the right information and that the generator produces accurate, relevant answers based on that retrieved context. This aligns with the synthetic data generation process described in Section I-C1, where question-answer pairs are created along with the relevant source chunks to evaluate both retrieval and generation performance. InputRAG PipelineRetrieverGeneratorLLM Output Retriever Metrics: Contextual Recall, Precision, and Relevancy Generator Metrics: Answer Relevancy and Faithfulness Fig. 3: RAG Evaluation I-B Dataset Selection and Preparation Our evaluation utilizes a dataset of 498 ESG reports collected from publicly available corporate websites, spanning 15 years (2010-2024) of sustainability reporting from EU-listed companies. This corpus provides coverage across different years and document sizes, enabling robust evaluation of model performance across diverse ESG reporting contexts. I-B1 Selection Strategy The document selection process followed a systematic approach intended to provide broad coverage across years and document sizes: • Temporal Coverage: Documents were sampled to ensure balanced representation across all 15 years, with 6-7 documents selected per year depending on corpus availability. • Size Diversity: Documents were selected from different size quartiles to ensure variety in document complexity, with selected documents ranging from 11,137 to 1,482,863 bytes. • Random Selection: Within each stratum (year and size category), random selection was used to reduce manual selection bias. The resulting dataset provides broad coverage of ESG reporting evolution while maintaining manageable computational requirements for multiple model evaluations. I-B2 Dataset Characteristics The selected 498-document dataset exhibits the following characteristics: • Temporal Distribution: Balanced coverage across 2010-2024, enabling evaluation of model performance on ESG reporting evolution. • Size Distribution: Mean size of 282,119 bytes with median of 230,348 bytes, providing variety in document complexity. • Content Diversity: Covers environmental, social, and governance topics across different industries and reporting frameworks. I-B3 PDF-to-Markdown Conversion The source corpus consists of PDF files. Prior to indexing, each document was converted to plain-text Markdown using a custom pipeline built on PyMuPDF (fitz). The pipeline proceeds as follows: 1. Block extraction: Each page is processed via PyMuPDF’s get_text("blocks") API, which returns text blocks with bounding-box coordinates, preserving the spatial layout of the original PDF. 2. Heading detection: Blocks are classified as headings using two heuristics: (a) text length under 50 characters with at most 8 words, or (b) all-uppercase text under 100 characters. Heading level is assigned by length and casing—short all-caps blocks become # H1, blocks under 50 characters become ## H2, and remaining heading-classified blocks become ### H3. 3. Text cleaning: Body text undergoes whitespace normalization (collapsing multiple spaces), insertion of spaces at lowercase-to-uppercase and digit-to-letter transitions to repair common PDF extraction artefacts, and removal of isolated page-number lines. 4. Page markers: A separator is appended after each page to preserve page-boundary information for downstream chunking. This process preserves the logical document structure required for accurate chunk-level retrieval while producing clean plain text suitable for embedding. Tables and figures embedded purely as images are not captured by this text-extraction approach and therefore fall outside the scope of this evaluation. I-B4 Dataset Availability The complete ESG-RAG evaluation resource is publicly available at https://github.com/motazsaad/ESG-RAG-benchmark-ragas. The repository includes all 498 ESG reports in processed Markdown format, the generated query-answer pairs in CSV and JSON formats, complete RAGAS evaluation scores for all seven models, document selection metadata, and evaluation scripts for reproducing the experimental setup. I-C Synthetic Data Generation for RAG Evaluation Evaluating a RAG pipeline requires a labeled dataset containing (i) queries that reflect real user information needs and (i) reference answers grounded in the source documents. Since no such labeled benchmark existed for our ESG corpus described in Section I-B, we constructed one using a custom synthetic generation protocol inspired by prior RAG evaluation workflows [15, 2], illustrated in Figure 4. We used qwen3:4b-instruct as the generator; this model also appears in the evaluated set, which introduces a self-enhancement risk that we quantify in Section V. 1. LoadData2. ChunkData3. GenerateQuestions4. GenerateAnswers5. QualityFilterEvaluationDataset Fig. 4: Five-step synthetic dataset generation protocol for RAG evaluation [15, 2]. Data flows in a snake pattern: documents are loaded and chunked (Steps 1–2); questions and reference answers are derived directly from each sampled chunk (Steps 3–4); a length-based quality filter removes degenerate pairs (Step 5). I-C1 Validity of the Synthetic Evaluation Protocol A legitimate concern with synthetic evaluation datasets is whether machine-generated QA pairs can accurately reflect real-world RAG performance. In our protocol, every question and reference answer is derived solely from the real 498-report ESG corpus, and each retained QA pair records its originating document source. This keeps the benchmark grounded in actual disclosures and preserves document-level provenance for later audit and retrieval analysis. The protocol also applies simple but explicit controls to reduce degenerate samples. Source chunks shorter than 50 characters are discarded before generation, and generated QA pairs are retained only when both the question and the reference answer exceed 10 characters. Combined with persona rotation across the Environmental, Social, and Governance pillars, these design choices are intended to produce a benchmark that captures persona-guided ESG information needs while maintaining broad topical coverage. At the same time, the benchmark should be understood as a custom RAGAS-inspired synthetic resource rather than a direct implementation of the full RAGAS testset-generation pipeline. The dataset was generated with qwen3:4b-instruct; one of the evaluated models shares this generator family, and the same model family also serves as the RAGAS judge. We therefore treat the benchmark as a practical comparative instrument within the scope of this study and explicitly acknowledge the resulting self-enhancement risk in Section V. I-C2 Generation Protocol The pipeline proceeds in five steps: 1. Load Data: Each ESG document in the 498-report corpus is loaded from the document store. 2. Chunk Data: Documents are split into fixed-size chunks of 1000 characters with a 200-character overlap (step size 800 characters). Chunks shorter than 50 characters after stripping are discarded. This produces a knowledge graph of 258,590 chunks across the 498 reports. From this pool we randomly sample 300 chunks (seed 42) for QA generation; personas (Section I-C3) are rotated cyclically across the sample so each ESG pillar receives roughly one third of the questions. 3. Generate Questions: An LLM analyzes each sampled chunk and generates a natural-language question that is (a) answerable solely from the chunk’s content and (b) representative of a genuine ESG information need. Questions are constrained to avoid compound queries that conflate multiple retrieval targets. 4. Generate Answers: Using the same chunk as the sole context, the LLM generates a concise reference answer. This serves as the ground truth against which the RAG system’s output is compared by RAGAS metrics. 5. Quality Filter: Degenerate outputs are removed using minimum length thresholds. Source chunks shorter than 50 characters are discarded before generation. Generated QA pairs are retained only if both the question and the reference answer each exceed 10 characters, filtering out empty or truncated outputs. I-C3 Persona-Driven Query Specialization To ensure domain coverage across all three ESG pillars, we layer the RAGAS persona framework [15] on top of the generation protocol. Each persona directs the question generation LLM toward a specific stakeholder perspective: ⬇ 1 from ragas.testset.persona import Persona 2 persona_environmental_kpi_extractor = Persona( 3 name="Environmental KPI Extractor", 4 role_description="Focuses on extracting key environmental performance 5 indicators (KPIs) from documents, such as carbon emissions, water 6 consumption, energy usage, waste generation, and renewable energy adoption.") 7 persona_social_kpi_extractor = Persona( 8 name="Social KPI Extractor", 9 role_description="Focuses on extracting key social performance indicators 10 (KPIs) from documents, such as employee diversity, labor practices, 11 community engagement, human rights, and health and safety metrics.") 12 persona_governance_kpi_extractor = Persona( 13 name="Governance KPI Extractor", 14 role_description="Focuses on extracting key governance performance 15 indicators (KPIs) from documents, such as board diversity, executive 16 compensation, anti-corruption policies, shareholder rights, and data 17 privacy measures.") Listing 1: ESG Persona Configuration The combination of the structured generation protocol, length-based filtering, and persona-driven specialization produces an evaluation dataset that (i) is fully grounded in real ESG source documents, (i) covers all three ESG pillars, (i) contains traceable evidence links for retrieval validation, and (iv) captures persona-guided ESG information needs—making it a rigorous benchmark for RAG evaluation in the sustainability reporting domain. After length filtering, the protocol yields 284 QA pairs (Environmental: 97, Social: 94, Governance: 93) drawn from 181 unique source documents out of the 498 in the corpus. I-D Open-Source Model Selection Our evaluation focuses on open-source models ranging from 2B to 30B parameters. This band spans the five evaluated checkpoints (plus the Gemma 4 models in the evaluation framework) and remains feasible in our local deployment setting, allowing direct comparison across compact 2B–4B-class models, 8B-class models, and 30B MoE models under the same 48 GB GPU budget. I-D1 Selection Criteria and Rationale The model selection strategy employed a multi-criteria approach to provide practical coverage of several widely used open-source model families under a common local-deployment setup: • Industry Adoption: Models selected based on widespread adoption in enterprise and research settings, as evidenced by leaderboard rankings and community usage statistics • Architectural Diversity: Models from different families (GLM, Qwen, Gemma, Ministral, Nemotron) to evaluate architectural impacts and training methodology differences • Geographic Representation: Models from major AI research hubs (US, Europe, China) to ensure global perspective • Size Range: 2B-30B parameter range covering edge (gemma4:e2b) to workstation-class MoE models (glm-4.7-flash), while remaining feasible on the available hardware • Platform Accessibility: All models accessible through Ollama platform for consistent evaluation environment and reproducibility I-D2 Evaluated Models The evaluation included seven open-source models selected based on their performance in industry benchmarks and research literature: • glm-4.7-flash:q4_K_M: 30 billion parameter Mixture-of-Experts model from Zhipu AI (3B active), demonstrating strong performance in Chinese language tasks and ranking among top models in the Open LLM Leaderboard [5]. The q4_K_M quantization format has been reported to reduce memory footprint substantially while preserving most of the original model performance [16]. • qwen3:4b-instruct: 4 billion parameter model from Alibaba, representing the latest generation of the Qwen series which has shown exceptional performance in multilingual tasks [30]. Selected for its balanced performance across English and Chinese ESG reporting contexts. • gemma3:4b: 4 billion parameter model from Google, representing the latest generation architecture with significant improvements over previous Gemma models [18]. Chosen for its strong performance in reasoning tasks and Google’s extensive research backing in language model development. • gemma4:e4b: 8 billion parameter model from Google DeepMind (4.5B effective), designed for frontier intelligence locally with configurable thinking modes [18]. Selected to represent the latest generation of Gemma architecture with enhanced reasoning and agentic capabilities. • gemma4:e2b: 2 billion parameter model from Google DeepMind (2.3B effective), optimized for edge device deployment [18]. Selected to evaluate efficiency-focused deployment scenarios. • ministral-3:8b: 8 billion parameter model from Mistral AI, the latest generation dense model with state-of-the-art performance in edge deployment scenarios [24]. Selected as the European representative and for its strong performance in multilingual contexts. • nemotron-3-nano:4b: 4 billion parameter model from NVIDIA, featuring a hybrid Mamba-Transformer MoE architecture optimized for enterprise agentic applications [27]. Selected for its enterprise-grade design and NVIDIA’s extensive ecosystem support. I-D3 Exclusion Criteria Several popular models were deliberately excluded to preserve a controlled and computationally feasible comparison. The study design intentionally evaluates representative checkpoints per model family within the practical 2B–30B local-deployment range, under a fixed hardware budget and a single shared pipeline configuration: • Llama Series: Excluded because the study evaluates one representative checkpoint per family rather than multiple checkpoints from the same lineage; broader family coverage, including Llama, is planned as future work [38] • Larger Models (¿30B): Models exceeding 30B total parameters excluded to keep the comparison feasible under the available local hardware budget and to avoid runtime disparities dominating the experimental design • Specialized Models: Domain-specific models (e.g., medical, legal) excluded to keep the study focused on general-purpose generators for ESG RAG rather than task-specific adaptation • Older Generations: Models predating 2023 excluded to keep the comparison centered on recent checkpoints available in the current deployment stack This selection strategy ensures broad coverage of current open-source LLM capabilities while maintaining experimental feasibility and reproducibility standards essential for academic research [8]. I-E Experimental Setup I-E1 Hardware Configuration All experiments were conducted on a system with the following hardware specifications to ensure reproducibility and provide context for computational requirements: • CPU: 64-core Intel QEMU Virtual CPU (x86_64 architecture) • Memory: 125 GB RAM with 118 GB available memory • GPU: NVIDIA RTX 6000 Ada Generation with 48 GB VRAM • GPU Driver: NVIDIA-SMI 580.126.09 with CUDA 13.0 support • Virtualization: KVM full virtualization environment The GPU configuration provided substantial computational resources for model inference, with the RTX 6000 Ada Generation offering 48 GB of VRAM sufficient for loading and evaluating multiple models up to 30B parameters simultaneously. End-to-end runtime was approximately 21 hours and 30 minutes for the original five models: about 9 hours 34 minutes for Step 2 (RAG generation across five models on 100 QA pairs over the 498-document corpus) and about 11 hours 56 minutes for Step 3 (RAGAS LLM-judged scoring of seven metrics over the same five model×100 QA matrix). The two additional Gemma 4 models were evaluated in a separate batch under the same pipeline configuration, extending the total runtime accordingly. I-E2 Evaluation Subset To bound computational cost, RAG evaluation was run on a deterministic 100-item prefix of the 284-pair synthetic testset. The remaining 184 QA pairs were not used in the experiments reported here and are reserved for future sensitivity analyses. Across the seven evaluated models this yields 700 model–query evaluations. I-E3 Prompt Format All five evaluated models receive identical single-turn prompting. The system message instructs the generator to ground its answer in the retrieved context rather than draw on parametric knowledge: “You are a helpful assistant that answers questions based on given documents only.” The human-turn template follows the Python f-string ⬇ 1 "question: query : relevant_doc" where relevant_doc is the single top-1 retrieved chunk wrapped in a one-element Python list. No few-shot exemplars, chain-of-thought scaffolding, or persona conditioning are applied at inference time. Because the system message, user template, retrieved context, and retriever are held constant, the main experimental variable is the generator model itself. However, decoding parameters were not explicitly normalized across models in Step 2, so generation behavior may also reflect model-specific default sampling settings exposed through Ollama. I-E4 Embedding Standardization To ensure consistent evaluation across all models, we standardized the embedding component using qwen3-embedding:4b for both dataset generation and RAG evaluation. This approach eliminates embedding model variability, allowing us to focus evaluation on generation model performance differences. I-E5 Retrieval Configuration Retrieval is configured at top-k=1: for each query the system computes cosine similarity against all 258,590 corpus-chunk embeddings and returns only the single highest-scoring chunk as the entire context window for the generator. There is no reranker, no MMR diversification, no similarity threshold, and no top-k>>1 fusion. This deliberately minimal-context configuration isolates the generator’s behaviour on a fixed evidence window and eliminates cross-chunk fusion as a confound when comparing model families, at the cost of imposing a retrieval ceiling that we quantify in Section IV-B and acknowledge in Section V. Larger top-k, hybrid retrievers (BM25 + dense), and reranker variants are left as future work. I-E6 Evaluation Metrics Our evaluation employs the RAGAS framework with seven key metrics: • LLM Context Precision With Reference: LLM-judged variant of context precision. Given the reference answer and the retrieved contexts, the judge decides which retrieved chunks contain information useful for deriving the reference, then averages the chunk-level verdicts into a precision score. • Context Precision: Proportion of retrieved context that is actually relevant. • Context Recall: Ability to retrieve all relevant context from the document corpus. • Context Relevance: Relevance of retrieved context to the query. • Faithfulness: Consistency of generated answers with retrieved context. • Answer Relevancy: Relevance of generated answers to the original query. • Factual Correctness: F1-style accuracy of information provided in generated answers, scored against the reference. I-E7 Evaluation Process The evaluation process followed these steps: 1. Dataset Generation: Use persona-driven approach to generate evaluation queries from the 498-document dataset. 2. RAG Pipeline Setup: Configure each model with standardized embedding and retrieval components. 3. Query Execution: Process evaluation queries through each RAG system. 4. Metrics Computation: Calculate RAGAS scores for each model-query combination. 5. Performance Analysis: Aggregate and analyze results across models and metrics. I-E8 Implementation Details Table I summarises all hyperparameters and configuration choices used in both the dataset generation and RAG evaluation pipelines, providing the information needed to reproduce the experimental setup. TABLE I: Implementation hyperparameters for dataset generation and RAG evaluation. Parameter Value Dataset Generation Generator LLM qwen3:4b-instruct Generator context window 12,000 tokens Embedding model qwen3-embedding:4b Chunk size 1,000 characters Chunk overlap 200 characters Min. chunk length 50 characters Min. QA pair length 10 characters (question and answer) RAG Evaluation (Step 2: retrieval + generation) Embedding model qwen3-embedding:4b Retrieval method Cosine similarity Top-k 1 Generator temperature Not overridden in Step 2; model-specific Ollama defaults therefore apply unless defined otherwise by the served checkpoint System prompt “You are a helpful assistant that answers questions based on given documents only.” User prompt template "question: query : relevant_doc" (see Section I-E3) RAGAS Scoring (Step 3: LLM-as-judge) Judge LLM qwen3:4b-instruct Judge embedding qwen3-embedding:4b Judge temperature Controlled by the RAGAS LLM wrapper for single-completion scoring (near-deterministic default behavior) Max workers 1 (serial execution) Per-call timeout 3600 s IV Results: Open-Source LLM Performance Analysis in RAG Application for ESG Domain This section presents the evaluation results of seven open-source large language models on our curated ESG dataset, using the methodology described in the previous section. Our evaluation includes models from different families and parameter sizes: glm-4.7-flash (30B MoE, q4_K_M), nemotron-3-nano:4b (4B), qwen3:4b-instruct (4B), gemma3:4b (4B), gemma4:e4b (8B, 4.5B effective), gemma4:e2b (2B, 2.3B effective), and ministral-3:8b (8B). This diverse model selection enables systematic analysis of performance characteristics across different architectures and sizes in the open-source ecosystem. The evaluation encompasses both retrieval and generation performance across environmental, social, and governance domains. To ensure consistency across all experiments, we use qwen3-embedding:4b for both dataset generation and RAG evaluation processes. IV-A Overall Performance Comparison Table I presents the performance evaluation of seven open-source LLMs across seven RAGAS metrics. The results reveal clear differences in model capabilities across different aspects of the RAG pipeline, with notable performance differences emerging between model families and sizes. TABLE I: RAG Performance Evaluation Results Across Seven Open-Source LLMs for ESG Domain. LLMCP(Ref) = LLM Context Precision With Reference, CP = Context Precision, CR = Context Recall, CtxRel = Context Relevance, Faith = Faithfulness, AR = Answer Relevancy, FC = Factual Correctness (F1). Best score per column is in bold; ties highlight all tied models. Model Retriever Metrics Generator Metrics Answer Quality LLMCP(Ref) CP CR CtxRel Faith AR FC (F1) glm-4.7-flash 0.878 0.803 0.600 0.985 0.822 0.818 0.387 nemotron-3-nano 0.878 0.792 0.605 0.985 0.625 0.851 0.402 qwen3 0.860 0.805 0.605 0.985 0.719 0.863 0.449 gemma3 0.875 0.805 0.605 0.985 0.776 0.877 0.427 gemma4:e4b 0.830 0.783 0.578 0.965 0.796 0.762 0.405 gemma4:e2b 0.830 0.783 0.578 0.965 0.784 0.760 0.423 ministral-3 0.875 0.805 0.600 0.985 0.607 0.881 0.405 Fig. 5: RAG performance heatmap across seven models and metrics. Darker green indicates higher scores; lighter/redder indicates lower scores. Retrieval metrics (LLMCP(Ref), CP, CR, CtxRel) cluster tightly, while generation metrics (Faith, AR, FC) show wider variance.0.8780.8030.6000.9850.8220.8180.3870.8780.7920.6050.9850.6250.8510.4020.8600.8050.6050.9850.7190.8630.4490.8750.8050.6050.9850.7760.8770.4270.8750.8050.6000.9850.6070.8810.4050.8300.7830.5780.9650.7960.7620.4050.8300.7830.5780.9650.7840.7600.423glm-4.7nemotron-3qwen3gemma3ministral-3gemma4:e4bgemma4:e2bLLMCP(Ref)CPCRCtxRelFaithARFC0.371.0 IV-B Retrieval Performance Analysis Retrieval performance is strong but no longer perfect once judged with the RAGAS LLM-based metrics. Context relevance reaches 0.965–0.985 across all models, indicating the top-1 retrieved chunk is almost always topically aligned with the query. LLM-judged context precision with reference is also high (0.830–0.878), confirming that the retrieved chunk usually contains the information needed to derive the reference answer. Context precision and context recall sit at 0.783–0.805 and 0.578–0.605 respectively. The shared retriever (qwen3-embedding:4b, top-k=1) means CtxRel would be identical across models in a deterministic retrieval setup; the small differences in LLMCP(Ref), CP, CR, and CtxRel arise from the LLM judge’s per-sample interpretation of whether the retrieved chunk fully supports the reference. The Gemma 4 models (e4b and e2b) show slightly lower retrieval-side scores compared to the other five models, which may reflect differences in how their responses interact with the LLM judge’s scoring criteria. Recall below unity reflects a realistic ceiling for top-k=1 retrieval: when the reference answer spans information that overlaps multiple chunks, a single retrieved chunk cannot cover all of it. These scores are not driven by an embedding-model mismatch between retrieval and judging — both stages use qwen3-embedding:4b — but rather by genuine retrieval gaps that the LLM judge surfaces. IV-C Generation Performance Analysis Generator performance reveals substantial differences between models, particularly in faithfulness. Faithfulness shows the widest spread, with glm-4.7-flash:q4_K_M leading (0.822), followed by gemma4:e4b (0.796), gemma4:e2b (0.784), gemma3:4b (0.776), qwen3:4b-instruct (0.719), nemotron-3-nano:4b (0.625), and ministral-3:8b (0.607). The 0.215 spread between best and worst on this metric is the dominant generation-quality differentiator. Notably, both Gemma 4 models outperform gemma3:4b on faithfulness, with gemma4:e4b (0.796) ranking second overall, despite having lower retrieval-side scores. Since all seven models receive the same anchoring system prompt and the same retrieved context (Section I-E3), this spread is consistent with model-level differences in how strictly each generator follows the “answer from documents only” instruction, though some portion of the variation may also reflect unnormalized default decoding settings in Step 2. Answer relevancy shows a wider spread when Gemma 4 models are included (0.760–0.881). The Gemma 4 models score notably lower on this metric (gemma4:e4b 0.762, gemma4:e2b 0.760), while the remaining five models cluster tightly (0.818–0.881) with ministral-3:8b first (0.881), followed by gemma3:4b (0.877), qwen3:4b-instruct (0.863), nemotron-3-nano:4b (0.851), and glm-4.7-flash:q4_K_M (0.818). The lower AR scores for Gemma 4 models suggest their responses may be less directly aligned with the user query, possibly due to differences in instruction-following behavior or decoding defaults. IV-D Factual Correctness and Domain-Specific Performance Factual correctness scores range from 0.387 to 0.449 across evaluated models. Qwen3:4b-instruct leads (0.449), followed by gemma3:4b (0.427), gemma4:e2b (0.423), ministral-3:8b (0.405), gemma4:e4b (0.405), nemotron-3-nano:4b (0.402), and glm-4.7-flash:q4_K_M (0.387). The 0.062 spread is much narrower than the faithfulness spread, suggesting all seven models face similar headwinds when extracting specific ESG facts from a single retrieved chunk. Even the top performer leaves substantial headroom for improvement. IV-D1 ESG Domain Complexity Challenges The factual correctness challenges in ESG applications stem from several domain-specific complexities that distinguish sustainability reporting from general knowledge domains [6]: • Technical Terminology Evolution: ESG reporting terminology evolves rapidly with new frameworks (GRI, SASB, TCFD, ESRS) and regulatory requirements, creating challenges for models trained on historical data [13]. • Quantitative Data Precision: ESG metrics require high numerical precision (e.g., carbon emissions in tons, percentage reductions, monetary values) where small errors can have significant regulatory implications [9]. • Contextual Dependencies: ESG facts often depend on specific regulatory contexts, industry standards, and geographic variations that general-purpose models may not capture accurately [6, 14, 12]. • Forward-Looking Statements: ESG reports contain numerous forward-looking commitments and targets that require careful distinction between historical facts and projected goals [34]. IV-D2 Model Family Performance Patterns The variation in factual correctness across model families may reflect differences in training emphasis and data composition, though the precise causes cannot be determined from evaluation scores alone: • Qwen3:4b-instruct (0.449): Top factual correctness; given the self-enhancement caveat (Section V), this leadership should be interpreted cautiously, as the dataset generator and RAGAS judge are also instances of qwen3:4b-instruct [30]. • Gemma3:4b (0.427): Strong factual performance; the underlying reasons are unclear from this evaluation alone [18]. • Gemma4:e2b (0.423): Strong factual correctness for a 2B model, nearly matching gemma3:4b despite having fewer parameters [18]. • Gemma4:e4b (0.405): Mid-range factual correctness; performs similarly to ministral-3:8b despite being a comparable effective parameter count [18]. • Ministral-3:8b (0.405): Mid-range factual correctness paired with the highest answer relevancy (0.881) [24]. • Nemotron-3-nano:4b (0.402): Mid-range factual correctness; near-tied with ministral-3 and gemma4:e4b [27]. • GLM-4.7-flash:q4_K_M (0.387): Lowest factual correctness; the 4-bit quantization is a plausible contributor, though the cause cannot be isolated from this evaluation alone [36, 16]. IV-D3 Practical Implications and Improvement Strategies The factual correctness results have important implications for ESG RAG deployment: • Human-in-the-Loop Validation: Current performance levels suggest that critical ESG applications should incorporate human validation mechanisms for high-stakes decisions [3]. • Domain-Specific Fine-Tuning: The performance variation across models indicates significant potential for improvement through ESG-specific fine-tuning using curated sustainability datasets [39, 1]. • Hybrid Retrieval Architectures: Larger top-k, rerankers, and modular retrieval pipelines could leverage complementary retrieval signals to improve overall factual accuracy [17]. • Retrieval Enhancement: Improved retrieval systems with better ESG-specific indexing could provide more accurate context, reducing factual errors [7]. The narrow 0.387–0.449 band across all seven models, all sitting well below 0.5, indicates that factual correctness remains the weakest link of the pipeline in this evaluation. This pattern persists under a fixed retrieval configuration with relatively strong retrieval-side scores (LLMCP(Ref) up to 0.878) and high answer relevancy (≥ 0.760 for every model), indicating that factual correctness remains a major bottleneck in the current pipeline configuration. Because retrieval was fixed at top-k=1 and no retrieval ablation was performed, these results should not be interpreted as ruling out retrieval improvements as a path to better factual accuracy. ESG applications that demand high precision for regulatory compliance and stakeholder trust would therefore benefit substantially from domain-specific fine-tuning or human-in-the-loop validation, regardless of which generator is chosen from this set. IV-E Model-Specific Insights glm-4.7-flash:q4_K_M leads faithfulness (0.822) and ties for the highest LLM-judged context precision with reference (0.878). Its trade-off is the lowest answer relevancy (0.818) and lowest factual correctness (0.387) of the seven, plausibly influenced by 4-bit quantization. Within this evaluation setting, it appears best suited to faithfulness-critical applications where staying anchored to retrieved evidence outweighs phrasing fluency. nemotron-3-nano:4b ties for the highest LLMCP(Ref) (0.878) and ranks mid-pack on AR (0.851) and FC (0.402), but posts the lowest faithfulness (0.625) of the set. Within this evaluation setting, it appears to be a solid generalist for retrieval-side use cases, but weaker when strict context grounding is required. qwen3:4b-instruct leads factual correctness (0.449) and posts competitive faithfulness (0.719) and answer relevancy (0.863). Because the dataset generator and RAGAS judge are also qwen3 family, this leadership is qualified by the self-enhancement caveat (Section V). gemma3:4b is the most balanced model: second on faithfulness (0.776), second on answer relevancy (0.877), and second on factual correctness (0.427) — no clear weakness on any generation metric. Within this evaluation setting, it is a reasonable default when the deployment requirements are not yet known. gemma4:e4b (8B, 4.5B effective) ranks second on faithfulness (0.796) but posts lower answer relevancy (0.762) than the other five models, with mid-range factual correctness (0.405). Its strong faithfulness suggests effective instruction-following for context grounding, but the lower AR indicates room for improvement in query alignment. Within this evaluation setting, it is a strong candidate for faithfulness-prioritizing applications. gemma4:e2b (2B, 2.3B effective) is noteworthy for its efficiency: despite being the smallest model in the set, it achieves competitive faithfulness (0.784) and factual correctness (0.423). It trails only on answer relevancy (0.760), where it is the lowest of the set. Within this evaluation setting, it is a compelling choice for resource-constrained deployments that still require strong context grounding. ministral-3:8b leads answer relevancy (0.881) but posts the lowest faithfulness (0.607) of the set, with mid-range factual correctness (0.405). Within this evaluation setting, it appears best suited to user-facing applications where on-topic phrasing matters most and the consequence of mild context drift is acceptable. Overall, no single model dominates every metric. Model selection should be driven by which generation property the application weighs most heavily, with the self-enhancement caveat in mind when comparing qwen3 against the others. IV-F Implications for ESG RAG Applications Within the reported ESG RAG evaluation setting, glm-4.7-flash:q4_K_M is the strongest choice for faithfulness-critical tasks (0.822), where the system must stay anchored to the retrieved evidence rather than draw on parametric knowledge. For applications prioritizing factual accuracy, qwen3:4b-instruct offers the highest factual correctness (0.449) and competitive answer relevancy (0.863), making it well suited for compliance-focused ESG systems — subject to the self-enhancement caveat in Section V. For applications prioritizing query-response alignment, ministral-3:8b delivers the highest answer relevancy (0.881), making it suitable for user-facing ESG information systems. Gemma3:4b offers the most balanced profile across the three generation metrics in this evaluation and is a reasonable default when deployment requirements are not yet known. Among the Gemma 4 family, gemma4:e4b (8B) is a strong faithfulness-oriented alternative (0.796) with the edge-deployable gemma4:e2b (2B) achieving surprisingly competitive factual correctness (0.423) at minimal computational cost, making it suited for resource-constrained settings. V Limitations Several limitations should be considered when interpreting the findings of this study. Corpus scope: The evaluation corpus comprises 498 ESG reports from EU-listed companies, collected from publicly available corporate websites. Findings may not generalize to ESG reporting in other geographic regions, languages, or regulatory frameworks. Synthetic evaluation set: The 100 QA pairs used for evaluation were generated synthetically by the same LLM family used for dataset construction (qwen3:4b-instruct). Real user queries may differ substantially in phrasing, complexity, and information needs, which could affect the observed performance rankings. Subset selection: The reported results are based on a deterministic prefix of the 284-pair synthetic testset rather than a randomized or stratified evaluation subset. The observed rankings may therefore be sensitive to subset selection. Top-k=1 retrieval ceiling: All retrieval results are bounded by single-chunk top-1 retrieval. Context recall around 0.60 reflects cases where the reference answer’s supporting evidence spans information beyond a single 1000-character chunk. Retrieval performance under higher top-k, larger chunk sizes, or independent (non-synthetic) query sets is unknown. No non-RAG baseline: This study evaluates RAG pipelines only; direct LLM inference without retrieval was not included. The absolute benefit of the retrieval component therefore cannot be quantified from these results alone. No hyperparameter ablation: Chunk size (1000 characters), overlap (200 characters), and top-k (1) were fixed throughout. The sensitivity of results to these choices was not studied. LLM-as-judge variability: RAGAS metrics rely on LLM-based scoring. In our setup, scoring is mediated by the RAGAS LLM wrapper rather than a raw direct call to the judge model, which applies near-deterministic single-completion behavior. Even so, judge-model choice and LLM-as-judge effects may still introduce uncertainty not captured by point estimates alone. Generator decoding control: Step 2 generation used direct ChatOllama(model=...) calls without explicitly normalizing temperature or related sampling parameters across models. As a result, some observed differences may reflect both checkpoint differences and model-specific default decoding behavior. Sampling fraction: Only 300 of 258,590 corpus chunks (0.12%) were drawn for QA generation, covering 181 of the 498 source documents. Results reflect performance on this sample rather than full-corpus coverage. Self-enhancement risk: The dataset generator, one of the evaluated models, and the RAGAS judge are all instances of qwen3:4b-instruct. Although the generator/judge call paths are separate from inference, qwen3 scores on faithfulness, answer relevancy, and factual correctness may be inflated relative to other models due to stylistic and lexical alignment between generation, response, and scoring [2]. Cross-validation with an independent judge model is left to future work. VI Conclusion and Future Work This study presents an empirical evaluation framework for open-source Large Language Models (LLMs) in ESG (Environmental, Social, and Governance) KPI extraction using Retrieval-Augmented Generation (RAG). Our analysis evaluates seven open-source models ranging from 2B to 30B parameters on a dataset of 498 ESG documents, revealing marked differences in performance characteristics across model architectures and sizes. The performance evaluation reveals genuine variation across both retrieval and generation. Retrieval is strong but not perfect (context recall ≈ 0.58–0.61, context precision ≈ 0.78–0.81, context relevance 0.965–0.985, LLM-judged context precision with reference ≈ 0.83–0.88), bounded by the top-k=1 design. Generation diverges most on faithfulness (0.607–0.822) and least on answer relevancy (0.760–0.881). Different models lead different metrics: glm-4.7-flash:q4_K_M leads faithfulness (0.822), qwen3:4b-instruct leads factual correctness (0.449), and ministral-3:8b leads answer relevancy (0.881). Factual correctness across all models (0.387–0.449) indicates ongoing opportunities for domain-specific fine-tuning and further enhancement. Strategically, this evaluation framework offers ESG practitioners directional guidance for open-source model selection based on their specific requirements. Under the reported configuration, organizations can use these results as directional guidance: glm-4.7-flash:q4_K_M for faithfulness-critical applications, qwen3:4b-instruct for factual-accuracy-critical tasks (subject to the self-enhancement caveat), ministral-3:8b for query-focused applications, gemma4:e4b as a faithfulness-oriented alternative, gemma4:e2b for resource-constrained deployments, and gemma3:4b as a balanced default. The narrow factual-correctness band across all seven models indicates that domain-specific fine-tuning and human-in-the-loop validation remain valuable directions regardless of generator choice. The study establishes a transparent and reproducible methodology for evaluating open-source LLMs in sustainability reporting, providing essential guidance for organizations seeking open-source AI solutions for ESG analytics. Beyond the model comparison itself, the study contributes an ESG-RAG evaluation resource that combines a processed report corpus, grounded synthetic QA pairs, and reproducible evaluation artifacts for future benchmarking and ablation studies. VI-A Future Work Several avenues for future research emerge from this study: Extended Open-Source Model Benchmarking: We plan to expand evaluation to include additional open-source models such as Llama 3, Mixtral, and emerging models from different parameter ranges (1B-70B). This broader comparison will provide deeper insights into the evolving landscape of open-source LLM capabilities for ESG applications. Domain-Specific Fine-Tuning: Given the improved but still moderate factual correctness scores across all models, future work will investigate domain-specific fine-tuning approaches for ESG applications. This includes training models on specialized ESG datasets and developing adaptation techniques to further enhance factual accuracy in sustainability reporting contexts, building on the promising progress demonstrated in this evaluation. Broader Corpus Validation: While our 498-document dataset provides coverage of EU companies’ ESG reporting, validating our findings across diverse geographies, industries, and reporting frameworks will enhance the generalizability of the evaluation framework. Cost-Performance Optimization: Future research will explore optimization techniques for open-source model deployment, including quantization strategies, model compression, and efficient serving architectures to maximize performance efficiency for ESG applications. Ablation Studies and Baselines: This study fixed chunk size (1000 characters), overlap (200 characters), and top-k (1) without ablating their individual effects. Future work should vary these hyperparameters systematically and include a non-RAG baseline (direct LLM inference without retrieval) to quantify the retrieval component’s contribution to overall performance. Hybrid Architecture Evaluation: Future work will investigate hybrid approaches that combine different open-source models for different pipeline components, potentially optimizing both accuracy and computational efficiency for ESG RAG systems. Acknowledgements This research was partially supported by the Italian Research Center on High Performance Computing, Big Data and Quantum Computing (ICSC), funded by the European Union through NextGenerationEU (PNRR-HPC, CUP: C83C22000560007). References [1] S. R. Ahmed, A. P. Shah, Q. H. Tran, V. Khetan, S. Kang, A. Mehta, Y. Bao, and W. Wei (2025) Enhancing retrieval for esgllm via esg-cid–a disclosure content index finetuning dataset for mapping gri and esrs. arXiv preprint arXiv:2503.10674. Cited by: §I-B, §I-C, 2nd item. [2] Amazon Web Services (2024) Generate synthetic data for evaluating RAG systems using Amazon Bedrock. Note: https://aws.amazon.com/blogs/machine-learning/generate-synthetic-data-for-evaluating-rag-systems-using-amazon-bedrock/AWS Machine Learning Blog Cited by: Fig. 4, §I-C, §V. [3] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané (2016) Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. Cited by: 1st item. [4] O. Ardic, M. U. Ozturk, I. Demirtas, and S. Arslan (2024) Information extraction from sustainability reports in turkish through rag approach. In 2024 32nd Signal Processing and Communications Applications Conference (SIU), Vol. , p. 1–4. External Links: Document Cited by: §I-B. [5] E. Beeching, C. Fourrier, N. Habib, S. Han, N. Lambert, N. Rajani, O. Sanseviero, L. Tunstall, and T. Wolf (2023) Open LLM leaderboard. Note: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboardHuggingFace Cited by: 1st item. [6] F. Berg, J. F. Kölbel, and R. Rigobon (2022) Aggregate confusion: the divergence of ESG ratings. Review of Finance 26 (6), p. 1315–1344. Cited by: 3rd item, §IV-D1. [7] M. Birti, F. Osborne, and A. Maurino (2025) Optimizing large language models for ESG activity detection in financial texts. In Proceedings of the 6th ACM International Conference on AI in Finance, External Links: Document Cited by: §I-B, 4th item. [8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in Neural Information Processing Systems 33, p. 1877–1901. Cited by: §I-D3. [9] M. Carney (2015) Breaking the tragedy of the horizon: climate change and financial stability. Note: Speech at Lloyd’s of London, Bank of England Cited by: 2nd item. [10] CDP (Carbon Disclosure Project) (2025) CDP: Turning Transparency to Action. Note: https://w.cdp.net/Accessed: 2025-07-08 Cited by: TABLE I, §I-D. [11] Confident AI (2026) DeepEval documentation: RAG evaluation. Note: https://deepeval.com/docs/getting-started-ragAccessed: 2026-05-18 Cited by: §I-A. [12] European Commission (2023) Commission delegated regulation (eu) 2023/2772 supplementing directive 2013/34/eu as regards sustainability reporting standards (ESRS). Note: Official Journal of the European Union, L seriesEuropean Sustainability Reporting Standards, developed by EFRAG Cited by: TABLE I, 3rd item. [13] European Commission (2023) Corporate Sustainability Reporting Directive (CSRD). Note: https://finance.ec.europa.eu/capital-markets-union-and-financial-markets/company-reporting-and-auditing/company-reporting/corporate-sustainability-reporting_en Cited by: 1st item. [14] European Union, European Parliament & Council (2022) Directive (EU) 2022/2464 – Corporate Sustainability Reporting Directive (CSRD). Note: https://eur-lex.europa.eu/eli/dir/2022/2464/ojAdopted 16 December 2022; in force 5 January 2023; phased reporting from January 2024 onwards Cited by: TABLE I, §I-D, 3rd item. [15] ExplodingGradients (2024) Ragas: supercharge your llm application evaluations. Note: https://github.com/explodinggradients/ragas Cited by: §I-A, Fig. 4, §I-A, §I-C3, §I-C. [16] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh (2022) GPTQ: accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. Cited by: 1st item, 7th item. [17] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang (2023) Retrieval-augmented generation for large language models: a survey. arXiv preprint arXiv:2312.10997. Cited by: §I, 3rd item. [18] Gemma Team, Google DeepMind (2024) Gemma: open models based on Gemini research and technology. Note: arXiv preprint arXiv:2403.08295 Cited by: 3rd item, 4th item, 5th item, 2nd item, 3rd item, 4th item. [19] Global Reporting Initiative (GRI) (2021) GRI Standards: Universal, Topic and Sector Standards. Note: https://w.globalreporting.org/Framework first published October 2021, effective for reports in January 2023 Cited by: TABLE I, §I-D. [20] Y. Gu, X. Han, Z. Li, et al. (2024) A survey on LLM-as-a-judge. arXiv preprint arXiv:2411.15594. Cited by: §I-A. [21] C. He, X. Zhou, Y. Wu, X. Yu, Y. Zhang, L. Zhang, D. Wang, S. Lyu, H. Xu, X. Wang, et al. (2025) ESGenius: benchmarking llms on environmental, social, and governance (esg) and sustainability knowledge. arXiv preprint arXiv:2506.01646. Cited by: §I-B, §I-C. [22] S. Kim, D. Lee, S. Kim, H. Park, M. Jeong, S. Yoon, et al. (2024) Prometheus: inducing fine-grained evaluation capability in language models. arXiv preprint arXiv:2310.08491. Cited by: §I-A. [23] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 33, p. 9459–9474. Cited by: §I. [24] A. H. Liu et al. (2026) Ministral 3. arXiv preprint arXiv:2601.08584. Cited by: 6th item, 5th item. [25] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 2511–2522. External Links: Document Cited by: §I-A. [26] R. Mankour, Y. Chafai, H. Saleh, G. Ben Hassine, T. Barreau, and P. Tankov (2025) Climate finance bench: a benchmark for RAG over corporate climate disclosures. arXiv preprint arXiv:2505.22752. Cited by: §I-B, §I-C. [27] NVIDIA (2025) Nemotron 3 Nano: open, efficient mixture-of-experts hybrid Mamba-Transformer model for Agentic reasoning. Note: arXiv preprint arXiv:2512.20848 Cited by: 7th item, 6th item. [28] A. Ontiveros, I. Nikishina, M. Gomm, C. Schmitt, and C. Biemann (2026) ESG-consultant: developing of an esg compliance consulting tool for companies using rag. In Natural Language Processing and Information Systems, R. Ichise (Ed.), Cham, p. 223–229. External Links: ISBN 978-3-031-97144-0 Cited by: §I-B. [29] A. L. Orsolin, L. V. Ávila, M. Trevisan, L. Dal Moro, and D. M. Cavalcante (2024) Analysis of indicators for the integration, implementation, and development of the environmental, social, and governance – esg report in ise/b3 companies. Revista Catarinense da Ciência Contábil. External Links: Document Cited by: §I. [30] Qwen Team (2025) Qwen3 technical report. Note: arXiv preprint arXiv:2505.09388 Cited by: 2nd item, 1st item. [31] T. M. Rusu, A. Odagiu, H. Pop, and L. Paulette (2024) Sustainability performance reporting.. Sustainability (2071-1050) 16 (19). Cited by: §I. [32] J. Saad-Falcon, O. Khattab, C. Potts, and M. Zaharia (2024) ARES: an automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 338–354. External Links: Document Cited by: §I-A. [33] Sustainability Accounting Standards Board (SASB) (2017) Sustainability Accounting Standards Board (SASB) Conceptual Framework. Note: https://w.sasb.org/Standards launched for 77–79 industries; stewardship transferred to IFRS Foundation’s ISSB in August 2022 Cited by: TABLE I. [34] Task Force on Climate-related Financial Disclosures (2017) Recommendations of the Task Force on Climate-related Financial Disclosures. Note: https://w.fsb-tcfd.org/recommendations/ Cited by: 4th item. [35] Task Force on Climate‑related Financial Disclosures, Financial Stability Board (2017) Recommendations of the Task Force on Climate‑related Financial Disclosures. Note: https://w.fsb-tcfd.org/Final recommendations released June 2017; framework active until October 12, 2023; site maintained as resource; monitoring of progress handed to IFRS Foundation as of November 2023 Cited by: TABLE I. [36] Team GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, et al. (2024) ChatGLM: a family of large language models from GLM-130B to GLM-4 all tools. Note: arXiv preprint arXiv:2406.12793 Cited by: 7th item. [37] R. Teixeira de Lima, S. Gupta, C. Berrospi Ramis, L. Mishra, M. Dolfi, P. Staar, and P. Vagenas (2025) Dataset taxonomy and generation strategies for evaluating RAG systems. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, External Links: Link Cited by: §I-A. [38] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: 1st item. [39] Q. Wu, X. Xiang, H. Huang, X. Wang, R. Satapathy, R. S. Filho, and B. Veeravalli (2025) SusGen-GPT: a data-centric LLM for financial NLP and sustainability report generation. In Findings of the Association for Computational Linguistics: NAACL 2025, p. 1184–1203. External Links: Document Cited by: §I-B, 2nd item. [40] T. Yang and M. Chen (2026) ESGLens: an LLM-based RAG framework for interactive ESG report analysis and score prediction. arXiv preprint arXiv:2604.19779. External Links: Link Cited by: §I-B, §I-C. [41] L. Zhang, X. Zhou, C. He, D. Wang, Y. Wu, H. Xu, W. Liu, and C. Miao (2025) Benchmarking multimodal understanding and complex reasoning for esg tasks. arXiv preprint arXiv:2507.18932. Cited by: §I-B, §I-C. [42] C. Zhu, Y. Chen, X. Shen, M. Xu, Q. Zhao, J. Wei, J. Huang, et al. (2025) JudgeLM: fine-tuned large language models are scalable judges. In International Conference on Learning Representations, External Links: Link Cited by: §I-A. [43] Y. Zou, M. Shi, Z. Chen, Y. Liu, J. Chen, S. Xue, P. S. Yu, and F. Zhang (2025) ESGReveal: an LLM-based approach for extracting structured data from ESG reports. Journal of Cleaner Production 489. External Links: Document Cited by: §I-B, §I-C.

Instant research discovery