Paper deep dive
An Industrial-Scale Retrieval-Augmented Generation Framework for Requirements Engineering: Empirical Evaluation with Automotive Manufacturing Data
Muhammad Khalid, Yilmaz Uygun
Intelligence
Status: succeeded | Model: anthropic/claude-sonnet-4.6 | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/24/2026, 3:32:58 AM
Summary
This paper presents a comprehensive empirical evaluation of a production-grade Retrieval-Augmented Generation (RAG) framework for industrial requirements engineering (RE) automation, using authentic automotive manufacturing documentation from a major German OEM. The dataset comprises 669 requirements across four specification standards (MBN 9666-1, MBN 9666-2, BQF 9666-5, MBN 9666-9) spanning 2015-2023, plus 49 supplier qualifications. The framework achieves 98.2% extraction accuracy, MRR of 0.847 via hybrid semantic-lexical retrieval, 83% reduction in manual analysis time, and 47% cost savings through multi-provider LLM orchestration. Longitudinal analysis reveals a 55% reduction in requirement volume, 1,800% increase in IT security focus, and identifies 10 legacy suppliers requiring requalification, representing $2.3M in avoided contract penalties.
Entities (43)
Relation Signals (36)
Hybrid Semantic-Lexical Retrieval → achievesmetric → MRR 0.847
confidence 99% · Hybrid semantic-lexical retrieval achieves MRR of 0.847
RAG Framework for Industrial RE → achievesmetric → 98.2% extraction accuracy
confidence 99% · the framework achieves 98.2% extraction accuracy with complete traceability
Muhammad Khalid → affiliatedwith → Constructor University Bremen
confidence 99% · Constructor University Bremen, Bremen, Germany {m.khalid, y.uygun}@constructor.university
Yilmaz Uygun → affiliatedwith → Constructor University Bremen
confidence 99% · Constructor University Bremen, Bremen, Germany {m.khalid, y.uygun}@constructor.university
RAG Framework for Industrial RE → appliedto → Requirements Engineering (RE)
confidence 99% · comprehensive empirical evaluation of RAG for industrial requirements engineering automation
Muhammad Khalid → authored → RAG Framework for Industrial RE
confidence 99% · Muhammad Khalid and Yilmaz Uygun present the RAG framework for industrial RE
Yilmaz Uygun → authored → RAG Framework for Industrial RE
confidence 99% · Muhammad Khalid and Yilmaz Uygun present the RAG framework for industrial RE
RAG Framework for Industrial RE → evaluatedon →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Requirements engineering in Industry 4.0 faces critical challenges with heterogeneous, unstructured documentation spanning technical specifications, supplier lists, and compliance standards. While retrieval-augmented generation (RAG) shows promise for knowledge-intensive tasks, no prior work has evaluated RAG on authentic industrial RE workflows using comprehensive production-grade performance metrics. This paper presents a comprehensive empirical evaluation of RAG for industrial requirements engineering automation using authentic automotive manufacturing documentation comprising 669 requirements across four specification standards (MBN 9666-1, MBN 9666-2, BQF 9666-5, MBN 9666-9) spanning 2015-2023, plus 49 supplier qualifications with extensive supporting documentation. Through controlled comparisons with BERT-based and ungrounded LLM approaches, the framework achieves 98.2% extraction accuracy with complete traceability, outperforming baselines by 24.4% and 19.6%, respectively. Hybrid semantic-lexical retrieval achieves MRR of 0.847. Expert quality assessment averaged 4.32/5.0 across five dimensions. The evaluation demonstrates 83% reduction in manual analysis time and 47% cost savings through multi-provider LLM orchestration. Ablation studies quantify individual component contributions. Longitudinal analysis reveals a 55% reduction in requirement volume coupled with 1,800% increase in IT security focus, identifying 10 legacy suppliers (20.4%) requiring requalification, representing potential $2.3M in avoided contract penalties.
Tags
Links
- Source: https://arxiv.org/abs/2603.20534v1
- Canonical: https://arxiv.org/abs/2603.20534v1
Trouble viewing inline? Open PDF directly →
Full Text
45,004 characters extracted from source content.
Expand or collapse full text
An Industrial-Scale Retrieval-Augmented Generation Framework for Requirements Engineering: Empirical Evaluation with Automotive Manufacturing Data Muhammad Khalid ∗ and Yilmaz Uygun ∗ ∗ Constructor University Bremen, Bremen, Germany m.khalid, y.uygun@constructor.university Abstract—Requirements engineering in Industry 4.0 faces crit- ical challenges with heterogeneous, unstructured documentation spanning technical specifications, supplier lists, and compliance standards. While retrieval-augmented generation (RAG) shows promise for knowledge-intensive tasks, no prior work evaluates RAG on authentic industrial RE workflows with comprehen- sive production-grade performance metrics. This paper presents a comprehensive empirical evaluation of RAG for industrial requirements engineering automation using authentic automo- tive manufacturing documentation comprising 669 requirements across four specification standards (MBN 9666-1, MBN 9666-2, BQF 9666-5, MBN 9666-9) spanning 2015-2023, plus 49 supplier qualifications with extensive supporting documentation. Through controlled comparisons with BERT-based and ungrounded LLM approaches, the framework achieves 98.2% extraction accuracy with complete traceability, outperforming baselines by 24.4% and 19.6% respectively. Hybrid semantic-lexical retrieval achieves MRR of 0.847. Expert quality assessment averaged 4.32/5.0 across five dimensions. The evaluation demonstrates 83% reduction in manual analysis time and 47% cost savings through multi- provider LLM orchestration. Ablation studies quantify individual component contributions. Longitudinal analysis reveals a 55% reduction in requirement volume coupled with 1,800% increase in IT security focus, identifying 10 legacy suppliers (20.4%) requiring requalification, representing potential $2.3M in avoided contract penalties. IndexTerms—RequirementsEngineering,Retrieval- Augmented Generation, Large Language Models, Industry 4.0, Empirical Software Engineering, Industrial Case Study I. INTRODUCTION The complexity of requirements engineering (RE) in In- dustry 4.0 environments presents unprecedented challenges for industrial organizations. Manufacturing enterprises must navigate vast repositories of heterogeneous documentation including technical specifications, supplier qualifications, com- pliance standards, and evolving security requirements. Manual cross-referencing between legacy and modern standards often leads to requirement drift, compliance gaps, and engineering inefficiency [1], [2]. The automotive industry exemplifies these challenges, where original equipment manufacturers (OEMs) must main- tain specification compliance across thousands of supplier relationships while adapting to evolving cybersecurity and functional safety requirements. A single OEM may manage relationships with hundreds of tier-1 suppliers, each qualified against dozens of specification standards spanning materials handling, robotics, quality assurance, and IT security [3]. Traditional requirements management systems struggle with unstructured natural language documentation, temporal speci- fication evolution, and the need for real-time economic impact assessment [4]. Recent advances in large language models (LLMs) and retrieval-augmented generation (RAG) offer promising so- lutions for automating RE tasks [5], [6]. RAG combines neural retrieval with generative models to ground LLM outputs in retrieved context, addressing hallucination and knowledge staleness [7]. However, existing approaches face significant limitations when applied to industrial contexts: (1) lack of domain-specific evaluation on authentic industrial data, (2) in- sufficient attention to traceability and compliance requirements mandated by regulatory frameworks (ISO 26262, IEC 62443), (3) limited assessment of real-world engineering workflow integration, and (4) absence of cost-benefit analysis for sus- tainable production deployment [8]. A. Research Questions This paper addresses these gaps through a comprehensive empirical evaluation guided by four research questions: RQ1 (Extraction Accuracy): How does RAG-based re- quirements extraction compare to state-of-art approaches (BERT-based classification, ungrounded LLM generation) on authentic industrial documentation spanning heterogeneous formats and eight-year temporal evolution? RQ2 (Retrieval Quality): What is the effectiveness of hy- brid semantic-lexical retrieval versus pure approaches (BM25- only sparse, dense-only vector search) for technical require- ments containing both natural language and precise terminol- ogy? RQ3 (Production Viability): What are the efficiency gains, cost implications, and expert acceptance levels required for industrial deployment sustainability, and how does multi- provider LLM orchestration impact cost-quality tradeoffs? arXiv:2603.20534v1 [cs.SE] 20 Mar 2026 RQ4 (Longitudinal Insights): What fundamental shifts in requirements patterns emerge from multi-year industrial specifications, and what actionable insights enable proactive compliance management? B. Contributions Our contributions are threefold: • Production-Grade RAG Architecture for Industrial RE: We present the first modular RAG framework specif- ically optimized for industrial requirements engineering workflows, incorporating multi-provider LLM orchestra- tion (47% cost reduction), hybrid semantic-lexical re- trieval mechanisms (MRR 0.847, 24.0% improvement over sparse-only), comprehensive traceability features (4.67/5.0 expert rating), and domain-adapted embedding models (+6.7% MRR gain through automotive-specific fine-tuning). • Comprehensive Industrial Evaluation with Authentic Data: We conduct the first large-scale empirical eval- uation using authentic automotive manufacturing docu- mentation from a major German OEM: 669 requirements across four specification standards spanning eight years (2015-2023), 49 supplier qualifications, 127 pages of compliance matrices, comprising an extensive industrial corpus. Evaluation employs multiple complementary ap- proaches: controlled baseline comparisons, multi-expert ground truth annotation (Cohen’s κ=0.89), statistical sig- nificance testing, ablation studies quantifying component contributions, and six-month production deployment in- sights. • Industry 4.0 Requirements Evolution and Economic Impact: We provide longitudinal analysis revealing fun- damental shifts in industrial requirements patterns: 55% volume reduction through specification consolidation and maturation, 1,800% IT security increase reflecting cy- bersecurity imperatives for networked industrial systems, emergence of new requirement categories (network seg- mentation, authentication protocols, vulnerability man- agement), and identification of 10 suppliers (20.4% of approved base) requiring safety requalification, enabling proactive compliance management avoiding potential $2.3M in contract penalties. I. RELATED WORK A. Requirements Engineering Automation Traditional RE relies heavily on manual processes for speci- fication extraction, classification, and traceability maintenance [9]. Early automated approaches employed rule-based natural language processing, using pattern matching and linguistic analysis to identify requirements [10]. Kof demonstrated 72% precision for requirement sentence classification using lin- guistic patterns, but recall remained limited (58%) due to variability in industrial specification writing styles. Machinelearningapproachesbroughtimprovements throughautomatedclassification.Hayesetal.[11] applied information retrieval techniques for traceability link generation, achieving precision 0.45-0.65 on software requirements using vector space models and latent semantic indexing. However, generalization to industrial manufacturing documentation remained limited due to domain-specific terminology(e.g.,“emergencystop,”“loadcapacity,” “network segmentation”) and multi-modal content (text, tables, diagrams, embedded images). B. Deep Learning for Requirements Engineering Recent work explores deep learning for RE automation. Zhao et al. [8] provide systematic mapping of NLP for RE, identifying 58 primary studies applying neural approaches across requirements classification (24 studies), traceability (18 studies), ambiguity detection (9 studies), and quality assess- ment (7 studies). Meta-analysis reveals median F1-score 0.79 across classification tasks, with substantial variance (standard deviation 0.14) indicating dataset dependency. Liu et al. [12] demonstrate BERT-based extraction of prod- uct comparison criteria from user requirements, achieving F1=0.82 on e-commerce reviews from Amazon and Best Buy (12,847 reviews spanning smartphones, laptops, cameras). However, transfer learning to technical specifications yields degraded performance (F1=0.61 on industrial safety documen- tation), highlighting domain gap challenges. Ferrari et al. [13] apply BERT to requirements quality assessment, detecting ambiguity with precision 0.74 and recall 0.69 on PURE dataset (7,246 requirements from 10 industrial projects). Qualitative analysis reveals BERT captures syntactic ambiguity (e.g., modifier attachment, pronoun reference) but struggles with semantic ambiguity requiring domain knowl- edge (e.g., “acceptable performance” without explicit thresh- olds). While promising, these approaches typically: (1) operate on individual requirements in isolation rather than enabling document-level cross-referencing critical for industrial con- texts (e.g., “supplier X must meet safety requirements defined in Section 3.4”), (2) require extensive labeled training data often unavailable for specialized domains (automotive safety, industrial cybersecurity), (3) lack mechanisms for source at- tribution required by regulatory compliance frameworks (ISO 26262, IEC 62443), and (4) demonstrate limited evaluation on authentic industrial data with heterogeneous formats (scanned PDFs, Excel with complex tables, Word with embedded dia- grams) spanning multiple years. Rath et al. [14] address traceability through automated trace link augmentation, achieving precision 0.62-0.89 on open- source software projects (Apache Lucene, ArgoUML, iTrust). Approach combines information retrieval with machine learn- ing classifiers trained on existing trace links. However, focus remains on software artifacts (source code, test cases, design documents) rather than industrial manufacturing specifications with supplier qualification requirements and compliance ma- trices. C. Retrieval-Augmented Generation RAG combines neural retrieval with generative language models to ground LLM outputs in retrieved context [5]. Lewis et al. demonstrate effectiveness for open-domain question answering, achieving exact match scores 44.5% on Natural Questions dataset (307,373 questions from Google search queries) and 56.8% on TriviaQA (78,785 trivia questions). Ap- proach uses dense passage retrieval (DPR) with dual-encoder architecture for query and document encoding, followed by sequence-to-sequence generation conditioned on retrieved pas- sages. The approach addresses key limitations of pure generative models: (1) hallucination—generation of plausible but fac- tually incorrect content (e.g., GPT-3 inventing non-existent academic citations with 21% frequency on scholarly writing tasks [15]), (2) knowledge staleness—inability to incorpo- rate information beyond training cutoff (GPT-4 training data through September 2021, limiting utility for recent events, standards updates, regulatory changes), and (3) lack of source attribution—absence of explicit provenance for generated con- tent, critical for compliance verification [7]. Izacard and Grave [16] extend RAG with fusion-in-decoder (FiD) architecture, achieving state-of-art results on TriviaQA (68.0% exact match) and Natural Questions (54.7%). FiD processes multiple retrieved passages independently through encoder, then fuses representations in decoder for generation. However, computational cost increases linearly with retrieved passage count (10-100 passages typical), limiting real-time applicability. Jiang et al. [17] apply RAG to document summarization with prompt compression, reducing token usage 83% while maintaining ROUGE-L score within 2% of uncompressed baseline. Approach identifies and removes low-information tokens from retrieved context based on perplexity scoring. Demonstrates cost-performance tradeoff optimization for pro- duction deployment. Parvez et al. [18] demonstrate retrieval-augmented code generation, improving BLEU scores 12-15% over pure gen- eration on CoNaLa dataset (2,879 Python code snippets with natural language descriptions). Retrieval from StackOverflow corpus provides relevant code examples conditioning gener- ation. However, evaluation focuses on short code snippets (mean 3.2 lines) rather than industrial-scale systems. Recent surveys [7], [19] identify applications in question answering (38% of surveyed papers), dialogue systems (24%), code generation (18%), and summarization (12%). However, industrial applications remain underexplored (2% of papers). Wang et al. [19] explicitly identify “specialized domains requiring regulatory compliance, traceability, and domain- specific terminology” as open research challenge. D. Semantic Search in Software Engineering Shatila et al. [20] systematically map semantic search in software engineering, identifying 47 primary studies across code search (19 studies), bug localization (12 studies), re- quirements traceability (9 studies), and documentation retrieval (7 studies). However, none evaluate RAG—studies focus on retrieval only without generation component. McBurney and McMillan [21] apply semantic search for requirements traceability, achieving mean average precision (MAP) 0.71 on NASA CM-1 dataset (498 requirements across 10 subsystems) using word embeddings trained on software engineering corpora. Demonstrates domain-specific embed- ding importance—general embeddings (Word2Vec trained on Google News) yield MAP 0.58 (-18% degradation). E. Industrial Requirements Analysis The automotive industry presents particularly stringent RE challenges: (1) complex multi-tier supply chains with hundreds of qualified suppliers across materials, components, assem- blies, and logistics, (2) evolving safety standards (ISO 26262 functional safety introduced 2011, updated 2018) and security requirements (IEC 62443 industrial cybersecurity series 2009- 2021), (3) long product lifecycles (10-15 years for vehi- cle platforms) requiring specification evolution management across model updates, technology insertions, and regulatory changes, and (4) regulatory compliance mandates for com- plete traceability from customer requirements through design, implementation, verification, and validation [1], [2]. Kannenberg and Saiedian [22] document persistent chal- lenges in automotive requirements traceability through sur- vey of 28 automotive organizations (OEMs, tier-1 suppliers, engineering service providers). Results reveal 67% maintain traceability manually or with limited tool support (spread- sheets, word processors), average 2.3 full-time equivalents (FTEs) dedicated to traceability maintenance per project, and 73% report difficulty maintaining trace link consistency during requirement changes. Ferrari and Gervasi [3] discuss RE practices in automotive contexts through case studies at three Italian automotive sup- pliers (anonymized). Qualitative analysis identifies six primary challenges: (1) requirement ambiguity (terminology inconsis- tency across OEMs), (2) incomplete specifications (missing preconditions, unstated assumptions), (3) evolving standards (safety/security updates mid-project), (4) supplier coordination (distributed development across geographic regions, language barriers), (5) legacy system integration (brownfield projects with existing platforms), and (6) audit compliance (prepara- tion effort for external assessments). However, no automated solutions proposed or empirical validation on production data provided. Groen et al. [23] explore crowdsourcing for requirements elicitation in large-scale systems, demonstrating feasibility for gathering diverse stakeholder input (study with 124 partici- pants generating 847 requirement suggestions for Amsterdam smart city project). However, application to technical man- ufacturing specifications with regulatory constraints remains unexplored—crowd workers lack domain expertise for safety- critical automotive requirements. F. Research Gap and Positioning Gap Identification: Synthesizing across related work re- veals critical gap: No prior work evaluates RAG-based au- tomation on authentic industrial requirements engineering workflows with comprehensive metrics spanning extraction accuracy, retrieval quality, production viability (efficiency, cost, expert acceptance), and longitudinal evolution analysis. Existing work either: (1) applies classical ML/deep learn- ing without retrieval augmentation [11]–[13], (2) evaluates RAG on general-domain tasks (QA, summarization) without industrial RE focus [5], [16], or (3) discusses industrial RE challenges without automated solutions [3], [22]. Our Contribution: This paper addresses the identified gap through comprehensive empirical evaluation using 669 authentic automotive requirements spanning 2015-2023 from major German OEM. Unlike prior work that focuses on either classical ML without retrieval [11]–[13] or RAG on general tasks [5], [16], we provide the first end-to- end evaluation of RAG for industrial RE with produc- tion deployment validation, rigorous baseline comparisons, and longitudinal analysis of specification evolution. We demonstrate production-grade performance (98.2% accuracy, 83% time reduction, 4.32/5.0 expert acceptance), quantify component contributions through ablation studies, and reveal actionable Industry 4.0 insights (supplier requalification needs, specification evolution patterns) with economic impact quan- tification ($2.3M penalty avoidance). I. SYSTEM ARCHITECTURE AND DESIGN A. Overview and Design Philosophy The RAG framework implements a modular, production- grade architecture guided by three design principles: (1) Sep- aration of Concerns—independent subsystems for ingestion, retrieval, generation, and traceability enable targeted optimiza- tion and maintenance without system-wide modifications, (2) Extensibility—pluggable interfaces support future integration of alternative retrieval algorithms (e.g., ColBERT late inter- action), embedding models (e.g., domain-specific fine-tuning), and LLM providers (e.g., Claude, Gemini, Llama), and (3) Transparency—comprehensive metadata capture and source attribution support regulatory compliance (ISO 26262, IEC 62443 mandate complete traceability) and build expert trust through verification. Figure 1 illustrates six integrated subsystems processing large-scale industrial documentation through multi-format in- gestion, 512-dimensional embeddings with HNSW index- ing, hybrid semantic-lexical retrieval with cross-encoder re- ranking, intelligent multi-provider LLM orchestration, and comprehensive traceability metadata. B. Document Ingestion Pipeline Design Rationale: Industrial documentation exhibits ex- treme heterogeneity: scanned PDFs from pre-2017 (containing OCR artifacts, skewed images, inconsistent page layouts), Excel spreadsheets with complex multi-table layouts (merged cells, nested headers, embedded charts), Word templates with embedded diagrams and cross-references (section numbering, table of contents, hyperlinks). Generic text extraction (e.g., PyPDF2, pdfminer) loses critical structural metadata (sec- tion hierarchies, table relationships, image captions) essential for traceability and compliance verification. We implement format-specific parsers preserving maximum metadata. Implementation: Three-stage workflow optimized for het- erogeneous industrial content: (1) Format-specific parsing with specialized libraries (PDFPlumber for table extraction preserving cell relationships, openpyxl for Excel maintaining formulas and metadata, python-docx for Word extracting styles and comments), (2) Semantic chunking respecting logical boundaries (section headers via font analysis, paragraph breaks via whitespace, table boundaries via grid detection) rather than fixed-size windows, resulting distribution mean 384±127 tokens optimized for embedding context, (3) Metadata en- richment with document ID, version timestamp, section path, category, compliance level, and supplier tags. Production Performance: Six-month deployment achieves 99.7% successful ingestion rate across 2015-2023 vintages. Comprehensive error handling includes invalid encoding de- tection, corrupted file recovery, and format version compati- bility. C. Hybrid Retrieval Mechanism Design Rationale: Pure dense semantic search excels at paraphrased queries but misses exact technical terms. Sparse BM25 captures terminology but ignores semantics. Hybrid fusion captures both dimensions. Dense Retrieval: Domain-adapted sentence transformers (all-mpnet-base-v2 fine-tuned on 15,127 automotive require- ments using contrastive learning). Fine-tuning improved MRR from 0.741 (base) to 0.791 (+6.7%, validates domain adapta- tion value). Embeddings stored in Pinecone (p2 pods, HNSW indexes, M=16, ef construction=200). Sparse Retrieval: BM25 with k1=1.5, b=0.75. Custom tok- enization preserves multi-word technical terms, part numbers, and acronyms via domain dictionary. Fusion Strategy: Reciprocal Rank Fusion (RRF) with learned weights (α dense =0.7, α sparse =0.3) optimized via grid search. Cross-encoder re-ranking (ms-marco-MiniLM-L-6-v2) on top-20 candidates improves P@5 from 0.801 to 0.824 (+2.9%). D. Multi-Provider LLM Orchestration Design Rationale: Single-provider deployment incurs ex- cessive costs for simple queries while potentially under- utilizing frontier models for complex reasoning. Intelligent routing optimizes cost-quality tradeoff. Task Complexity Classifier: XGBoost with 127 features (query length, entity count, cross-reference patterns, expected verbosity) achieves 91% complexity categorization accuracy. Routing Policy: Three-tier strategy: Simple extractions (complexity <0.4, 35% of queries) → GPT-3.5-turbo ($0.002/1K). Mid-complexity (0.4–0.7, 52%) → GPT-4- turbo ($0.01/1K). Complex reasoning (≥0.7, 13%) → GPT- 4/Claude-3 ($0.03/1K). Reliability: Automatic fallback, exponential backoff (initial 1s, max 30s), circuit breaker pattern prevent cascade failures. Industrial Docs PDF • Excel • Word User Query Natural Language Document Ingestion PDF Parser Excel Handler Word Parser Vector Database Pinecone/Qdrant Query Processor Hybrid Retrieval Semantic + Lexical Dense Vector Sparse BM25 Context Builder LLM Orchestration Multi-Provider Router Prov AProv B Traceability Source Attribution Analysis Results + Source Links Docs Chunks q emb Top-K Response Update 512-dim MRR: 0.847 47% cost↓ Fig. 1. Production-grade RAG framework architecture with six integrated subsystems for industrial requirements engineering. Cost Validation: Blind expert comparison confirms no quality difference between multi-provider and single-provider outputs (χ 2 =1.37, p=0.24), validating 47% cost reduction ($0.07 vs $0.13 per query over 10,000 production queries). E. Traceability and Compliance Layer Design Rationale: Regulatory compliance (ISO 26262, IEC 62443) mandates complete traceability. System captures comprehensive metadata: document provenance (IDs, versions, section paths, page numbers), retrieval scores (dense similarity, BM25, RRF, cross-encoder), generation metadata (provider, model version, prompts, timestamps, token usage), and con- fidence metrics (self-assessed confidence, retrieval coverage, multi-attempt consistency). Expert Feedback: Traceability received highest rating (4.67/5.0, variance 0.49), with qualitative feedback emphasiz- ing “direct source verification builds trust” and “crucial for compliance workflows.” IV. EVALUATION METHODOLOGY A. Dataset Characteristics AuthenticIndustrialData:Productionautomotive manufacturing documentation from major German OEM (anonymized per NDA), collected April–September 2024 with cooperation of RE department (12 senior engineers, 8+ years experience). Dataset Scope: 669 requirements across four specifications: • MBN 9666-1 (Materials Handling, 2015): 412 require- ments for conveyors, AGVs, safety zones • MBN 9666-2 (Robotics, 2018): 178 requirements for collaborative robots, emergency stops, fault detection • BQF 9666-5 (Quality Assurance, 2020): 145 require- ments for inspection, defect classification, SPC • MBN 9666-9 (IT Security, 2023): 95 requirements for network segmentation, authentication, vulnerability man- agement Supporting Documentation: 49 supplier qualifications, 127 pages compliance matrices, comprising an extensive in- dustrial corpus. Temporal Characteristics: Eight-year span enables longi- tudinal analysis. Format evolution: early (2015–2017) scanned PDFs with OCR artifacts, mid-period (2018–2020) mixed digital/scanned, recent (2021–2023) native digital structured templates. B. Ground Truth Annotation Three domain experts (12+ years automotive RE, ISO 26262 certified) independently annotated 669 requirements with metadata. Inter-rater agreement: Cohen’s κ=0.89 (near- perfect). Disagreements (11%) resolved through discussion. For retrieval evaluation, experts provided relevance judgments for 150 queries on 5-point scale, inter-annotator κ=0.84. C. Evaluation Metrics RQ1 (Extraction): Accuracy, precision, recall, F1-score. Statistical significance via paired t-test. Error analysis catego- rizes failure modes. RQ2 (Retrieval): MRR, Precision@k, NDCG@10, latency (ms). MRR measures first relevant result rank. NDCG accounts for graded relevance and position. RQ3 (Viability): Expert quality (5-point Likert across completeness, accuracy, relevance, utility, traceability), time- motion studies (manual vs automated), cost per query (LLM + embedding + database). RQ4 (Evolution): Requirements by category 2015–2023, density trends, supplier compliance assessment. D. Baseline Comparisons BERT: BERT-base-uncased fine-tuned on PROMISE re- quirements (3,000 software requirements), zero-shot transfer to automotive. Ungrounded LLM: GPT-4 zero-shot without retrieval, evaluates RAG value-add. BM25-only: Pure sparse retrieval, evaluates hybrid benefit. Dense-only: Pure vector search, assesses semantic-lexical complementarity. All baselines use identical dataset and ground truth for fair comparison. V. RESULTS A. RQ1: Requirements Extraction Accuracy Table I and Figure 2 present extraction results. TABLE I REQUIREMENTS EXTRACTION PERFORMANCE (RQ1, N=669) ApproachAccuracyPrecisionRecall BERT-based78.8%0.7820.795 Ungrounded LLM82.1%0.8310.812 RAG (Ours)98.2%0.9840.980 Statistical Significance (paired t-test vs RAG) BERTt=28.4, p < 0.001 Ungrounded LLMt=22.1, p < 0.001 Accuracy (%) 0 20 40 60 80 100 78.8% 82.1% 98.2% BERT Baseline Ungrounded LLM RAG (Ours) +24.4% +19.6% Fig. 2. Extraction accuracy comparison (RQ1). RAG achieves +24.4% over BERT, +19.6% over ungrounded LLM (p < 0.001). Answer to RQ1: RAG achieves 98.2% extraction accuracy (F1=0.979), statistically significant improvements of 24.4% over BERT (78.8%, p < 0.001) and 19.6% over ungrounded LLM (82.1%, p < 0.001). RAG also processes 47% faster (47min vs 89min) due to reduced generation burden with relevant context. Error Analysis: Remaining 1.8% errors categorize into: (1) ambiguous requirement boundaries in legacy documents (1.2%), (2) multi-table span requirements (0.4%), (3) OCR ar- tifacts in scanned documents (0.2%). Expert review confirmed these edge cases require human judgment, validating human- in-the-loop deployment. B. RQ2: Retrieval Quality Analysis Table I and Figure 3 demonstrate retrieval performance. Answer to RQ2: Hybrid retrieval achieves MRR 0.847, representing 24.0% improvement over BM25-only (0.683, TABLE I RETRIEVAL PERFORMANCE COMPARISON (RQ2, N=150 QUERIES) MethodMRRP@5NDCGLat. BM25 only0.6830.6410.72989ms Dense only0.7910.7580.812112ms Hybrid+Rerank0.8470.8240.882127ms Improvement over BM25 (Mann-Whitney U test) MRR+24.0% (U =8442, p < 0.001) Metric Score 0.0 0.2 0.4 0.6 0.8 1.0 .683 .791 .847 .641 .758 .824 .729 .812 .882 MRRP@5NDCG@10 BM25 OnlyDense Only Hybrid+Rerank Fig. 3.Retrieval performance across methods (RQ2). Hybrid+reranking consistently outperforms (p < 0.001). p < 0.001) and 6.4% over dense-only (0.791, p < 0.05). This validates synergy between semantic understanding and lexical matching for technical documentation. Latency Analysis: Average 127ms enables real-time use. Decomposition: vector search (89ms), BM25 (23ms), re- ranking (15ms). Re-ranking adds 11.8% overhead while im- proving MRR 2.3%. C. RQ3: Production Viability Assessment Table I presents expert quality assessment. TABLE I EXPERT QUALITY RATINGS (RQ3, N=150 QUERIES, 3 EXPERTS) DimensionMeanStdMinMax Completeness4.410.672.335.00 Accuracy4.530.583.005.00 Relevance4.280.742.675.00 Practical Utility4.070.832.005.00 Traceability4.670.493.335.00 Overall4.320.662.675.00 Table IV quantifies efficiency gains. Answer to RQ3: Production viability confirmed through: (1) Expert acceptance 4.32/5.0 demonstrates practical utility, traceability highest rated (4.67/5.0), (2) Efficiency: 83% aver- age time reduction (5.7hr → 58min), (3) Cost optimization: 47% reduction without quality degradation (validated via blind comparison, χ 2 =1.37, p=0.24). D. RQ4: Longitudinal Requirements Evolution Figure 4 and Table V reveal Industry 4.0 transformation. TABLE IV EFFICIENCY METRICS AND COST ANALYSIS (RQ3) Task CategoryManualRAGReduction Requirements extraction5.2hr47min85% Cross-reference analysis3.8hr38min83% Supplier assessment8.1hr1.8hr78% Average5.7hr58min83% Cost Analysis (per query, 10K production queries) Single-provider (GPT-4)—$0.13— Multi-provider (optimized)—$0.0747% Year Requirements 0 400 800 1200 1600 2015201820202023 1,487 669 5 95 Total (-55%) Security (+1,800%) Industry 4.0 Paradigm Shift Fig. 4.Longitudinal requirements evolution (RQ4, 2015-2023) revealing Industry 4.0 paradigm shift. TABLE V REQUIREMENTS EVOLUTION ANALYSIS (RQ4, 2015-2023) Category20152023Change Total Requirements1,487669-55% IT Security595+1,800% Functional Safety342187-45% Functional (General)1,140387-66% Supplier Compliance (2023 standards) Fully Compliant—3979.6% Require Requalification—1020.4% New Categories (2020-2023) Network Segmentation018New Authentication027New Vulnerability Mgmt031New Answer to RQ4: Longitudinal analysis reveals dual Indus- try 4.0 transformation: (1) Consolidation: 55% volume reduc- tion (1,487 → 669) through maturity, functional requirements decreased 66%, (2) Cybersecurity imperative: IT security increased 1,800% (5→ 95) with new categories emerging, (3) Economic impact: 10 suppliers (20.4%) require requalification, avoiding potential $2.3M penalties ($230K/supplier × 10). E. Ablation Studies Table VI quantifies individual component contributions. Key Findings: Hybrid retrieval provides largest contribution (+16.4% MRR, +3.9% accuracy). Domain fine-tuning adds +6.6% MRR. Multi-provider routing maintains quality while TABLE VI ABLATION STUDY: COMPONENT CONTRIBUTIONS (N=150 QUERIES) ConfigurationMRRAccuracy Full System0.84798.2% Remove Components (measure degradation) - Domain fine-tuning0.79196.1% - Cross-encoder rerank0.82797.8% - Hybrid retrieval0.68394.3% - Multi-provider routing0.84298.0% - Traceability metadata0.84597.9% Component Contribution Domain fine-tuning+6.6%+2.1% Cross-encoder rerank+2.0%+0.4% Hybrid retrieval+16.4%+3.9% reducing cost 47%. Traceability metadata does not impact accuracy but critically affects expert trust (4.67/5.0 rating). VI. DISCUSSION A. Answers to Research Questions RQ1: RAG achieves 98.2% extraction accuracy, signifi- cantly outperforming BERT (+24.4%, p < 0.001) and un- grounded LLM (+19.6%, p < 0.001), demonstrating produc- tion readiness. RQ2: Hybrid semantic-lexical retrieval achieves MRR 0.847, representing 24.0% improvement over sparse-only (p < 0.001), validating synergy for technical documentation. RQ3: Production viability confirmed: 83% time reduction, 47% cost savings, 4.32/5.0 expert acceptance, enabling sus- tainable deployment. RQ4: Eight-year analysis reveals Industry 4.0 shift: 55% consolidation with 1,800% IT security increase, identifying 10 suppliers requiring requalification. B. Implications for Research and Practice For Researchers: (1) Domain adaptation (+6.7% MRR) more impactful than algorithm sophistication, suggesting in- vestment in high-quality domain data yields better ROI, (2) Hybrid retrieval consistently outperforms pure approaches across all metrics, recommending fusion strategies for techni- cal documentation, (3) Traceability essential for expert trust (4.67/5.0 highest rating), suggesting RE automation must prioritize source attribution over pure accuracy metrics. For Practitioners: (1) Human-in-the-loop optimal—RAG handles 98%+ while flagging edge cases, (2) Multi-provider orchestration enables sustainable costs without quality com- promise, (3) Longitudinal analysis reveals actionable insights (supplier requalification) demonstrating business value beyond efficiency. C. Deployment Insights from Six-Month Production Use Six-month deployment revealed critical lessons: (1) Incre- mental adoption—phased rollout (month 1: single team pilot, months 2–3: department-wide, months 4–6: cross-functional) managed change resistance and enabled iterative refinement, (2) Expert validation workflow—mandatory human review for high-stakes decisions (supplier disqualification, compli- ance audit responses) maintained quality while building trust, (3) Continuous monitoring—automated quality tracking (re- trieval scores, generation confidence, expert feedback) enabled early detection of degradation (e.g., new specification for- mats requiring parser updates), (4) Training investment—3- hour workshops for 47 engineers covered system capabilities, limitations, and effective query formulation, reducing misuse and improving adoption (usage increased from 23% week-1 to 87% week-24). D. Threats to Validity Internal Validity: Ground truth established through multi- expert annotation (Cohen’s κ=0.89). Evaluation metrics stan- dard in RE research. Statistical significance confirmed via t- tests and Mann-Whitney U tests. External Validity: Limitation: Automotive manufacturing focus; generalization to aerospace, pharmaceuticals, or medi- cal devices requires validation. Mitigation: Dataset represents authentic industrial complexity (heterogeneous formats, 8-year span, 49 suppliers). Results likely transferable to domains with similar characteristics (complex supply chains, evolving stan- dards, compliance requirements). Future work should validate across additional industrial sectors. Construct Validity: Concern: Expert quality assessment subjective. Mitigation: Three independent experts, structured rubrics, inter-rater reliability measured (κ=0.84–0.89). Quali- tative feedback corroborates quantitative ratings. Conclusion Validity: Concern: Sample size (150 queries for quality assessment, 669 requirements for extraction). Mit- igation: Random sampling ensures representativeness. Statis- tical power analysis confirms sufficient sample for detecting large effects (Cohen’s d > 0.8, power=0.95). Production de- ployment (10,000 queries over 6 months) provides additional validation. E. Limitations and Future Work Uncertainty Quantification: Current system treats all out- puts uniformly. Enhanced confidence scoring could automat- ically flag low-confidence extractions (1.8% error cases) for expert review, further improving human-in-the-loop efficiency. Preliminary experiments with ensemble disagreement and re- trieval score analysis show promise. Continuous Learning: System requires periodic retraining as specifications evolve. Future work should explore online learning mechanisms adapting to new requirement patterns without full retraining cycles, potentially using active learning to identify informative examples for annotation. Cross-Organizational Knowledge Transfer: Current eval- uation single-OEM. Expanding to multi-OEM contexts could reveal opportunities to harmonize and standardize industry- wide requirements, potentially reducing redundancy across automotive supply chains and improving supplier qualification efficiency. Extended Production Deployment: Six-month deploy- ment informed current evaluation. Longer-term studies (12– 24 months) would assess sustained adoption patterns, identify emergent use cases beyond initial design scope, quantify long- term productivity gains accounting for learning curves and workflow integration, and measure organizational impact (RE process transformation, skill requirement shifts). VII. CONCLUSION This paper presents a comprehensive empirical evaluation of RAG-based automation for industrial requirements engineer- ing using authentic automotive manufacturing documentation spanning eight years. We address four research questions using rigorous methodology: 669 requirements, controlled baseline comparisons, multi-expert ground-truth annotation, ablation studies, and a six-month production deployment. Key Findings: (1) Production-grade performance—98.2% extraction accuracy, 83% time reduction, 4.32/5.0 expert ac- ceptance (RQ1, RQ3), (2) Hybrid retrieval provides 24.0% MRR improvement over sparse-only approaches with 127ms real-time latency, validating semantic-lexical synergy (RQ2), (3) Multi-provider orchestration achieves 47% cost reduction without quality degradation, enabling sustainable deployment (RQ3), (4) Longitudinal analysis reveals dual Industry 4.0 transformation: 55% consolidation with 1,800% IT security increase, identifying 10 suppliers requiring requalification with $2.3M economic impact (RQ4), (5) Ablation studies quantify component contributions, with hybrid retrieval largest (+16.4% MRR) and domain adaptation significant (+6.7% MRR). Results provide empirical evidence that RAG approaches have matured for industrial RE deployment, enabling organi- zations to address Industry 4.0 specification complexity while maintaining quality, compliance, and cost-effectiveness. Six- month production deployment demonstrates real-world viabil- ity with sustained expert acceptance and actionable business impact through longitudinal analysis capabilities. DATA AVAILABILITY STATEMENT The industrial documentation used in this study is pro- prietary and subject to non-disclosure agreements with the partnering automotive manufacturer. Raw data cannot be shared due to confidentiality constraints. However, we pro- vide: (1) The RAG framework implementation as open-source Python package at https://re-engineer-app-khalid.replit.app, (2) Anonymized requirement samples (50 requirements with metadata, representative of evaluation dataset structure and complexity), (3) Evaluation protocols and scripts (expert as- sessment rubrics, statistical analysis code, ablation study con- figurations), (4) Supplementary materials (additional figures showing temporal evolution details, extended ablation results, deployment workflow diagrams). The anonymized artifacts enable reproducibility assessment and facilitate adaptation to other industrial domains. Upon acceptance, complete artifacts will be deposited in an open-access repository with a DOI assignment for long-term availability. ACKNOWLEDGMENT This research was funded by the German Research Foun- dation (Deutsche Forschungsgemeinschaft, DFG) under refer- ence number UY-56/5-1. The authors gratefully acknowledge the support of the partnering automotive manufacturer for ac- cess to production documentation and participation by domain experts. REFERENCES [1] W. Maalej and A. K. Thurimella, “Towards a unified requirements en- gineering framework for large-scale automotive systems,” Requirements Engineering, vol. 18, no. 3, p. 257–281, 2013. DOI: 10.1007/s00766- 013-0175-y [2] T. Berger et al., “What is a feature? A qualitative study of features in industrial software product lines,” IEEE Trans. Software Engineering, vol. 46, no. 4, p. 364–385, 2020. DOI: 10.1109/TSE.2018.2856389 [3] A. Ferrari and G. Gervasi, “Requirements engineering in automotive: An empirical study,” in Proc. Int. Conf. Computational Science and Its Applications, Springer, 2019, p. 331–346. DOI: 10.1007/978-3-030- 24296-1 25 [4] F. Dalpiaz and S. Brinkkemper, “Agile requirements engineering with user stories,” in Proc. IEEE 26th Int. Requirements Engineering Conf. (RE), 2018, p. 506–507. DOI: 10.1109/RE.2018.00062 [5] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, p. 9459–9474. https://proceedings.neurips.c/ paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html [6] T. Brown et al., “Language models are few-shot learners,” in Ad- vances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, p. 1877–1901. https://proceedings.neurips.c/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html [7] Y. Gao et al., “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, 2023. https://arxiv. org/abs/2312.10997 [8] L. Zhao et al., “Natural language processing for requirements engineer- ing: A systematic mapping study,” ACM Computing Surveys, vol. 54, no. 3, p. 1–41, 2021. DOI: 10.1145/3460533 [9] J. Cleland-Huang, O. C. Z. Gotel, J. Huffman Hayes, P. M ̈ ader, and A. Zisman, “Software traceability: trends and future directions,” in Proc. Future of Software Engineering (FOSE), ACM, 2014, p. 55–69. DOI: 10.1145/2593882.2593891 [10] L. Kof, “Natural language processing for requirements engineering: Applicability to large requirements documents,” in Proc. 19th Int. Conf. Automated Software Engineering (ASE), 2005, p. 385–388. DOI: 10.1145/1101908.1101971 [11] J. H. Hayes, A. Dekhtyar, and S. K. Sundaram, “Advancing candidate link generation for requirements tracing,” IEEE Trans. Software Engi- neering, vol. 32, no. 1, p. 4–19, 2006. DOI: 10.1109/TSE.2006.3 [12] S. Liu, H. Chen, N. Dilshener, and J. Whittle, “Automated extraction of product comparison criteria from informal user requirements,” IEEE Trans. Software Engineering, vol. 49, no. 4, p. 2331–2347, 2023. DOI: 10.1109/TSE.2023.3243840 [13] A. Ferrari, F. dell’Orletta, A. Esuli, V. Gervasi, and S. Gnesi, “Natural language requirements processing: A 4D vision,” IEEE Software, vol. 37, no. 3, p. 28–35, 2020. DOI: 10.1109/MS.2019.2933481 [14] M. Rath, J. Rendall, J. L. C. Guo, J. Cleland-Huang, and P. M ̈ ader, “Traceability in the wild: Automatically augmenting incomplete trace links,” in Proc. IEEE/ACM 40th Int. Conf. Software Engineering (ICSE), 2018, p. 834–845. DOI: 10.1145/3180155.3180207 [15] H. Alkaissi and S. I. McFarlane, “Artificial hallucinations in ChatGPT: Implications in scientific writing,” Cureus, vol. 15, no. 2, p. e35179, 2023. DOI: 10.7759/cureus.35179 [16] G. Izacard and E. Grave, “Leveraging passage retrieval with generative models for open domain question answering,” in Proc. 16th Conf. European Chapter of the Association for Computational Linguistics (EACL), 2021, p. 874–880. https://aclanthology.org/2021.eacl-main.74/ [17] H. Jiang et al., “LLMLingua: Compressing prompts for accelerated inference of large language models,” in Proc. 2023 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2023, p. 12574– 12588. https://aclanthology.org/2023.emnlp-main.779/ [18] M. R. Parvez, W. Ahmad, S. Chakraborty, B. Ray, and K. Chang, “Retrieval augmented code generation and summarization,” in Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, p. 2719–2734. https://aclanthology.org/2021.findings-emnlp.232/ [19] X. Wang, P. Zhang, Y. Liu, L. Sun, and W. Che, “RAG for knowledge- intensive NLP: A survey of recent advances,” ACM Trans. Information Systems, vol. 42, no. 2, p. 1–35, 2024. DOI: 10.1145/3637528 [20] S. Shatila, J. Zhao, and S. Bhatia, “Semantic search in software engineering: A systematic mapping study,” J. Systems and Software, vol. 193, p. 111449, 2022. DOI: 10.1016/j.jss.2022.111449 [21] P. W. McBurney and C. McMillan, “Automatic documentation genera- tion via source code summarization of method context,” in Proc. 22nd Int. Conf. Program Comprehension (ICPC), 2016, p. 279–290. DOI: 10.1109/ICPC.2016.7503735 [22] A. Kannenberg and H. Saiedian, “Why software requirements traceabil- ity remains a challenge,” CrossTalk: The Journal of Defense Software Engineering, vol. 22, no. 4, p. 14–19, 2012. https://apps.dtic.mil/sti/ citations/ADA613775 [23] E. C. Groen et al., “The crowd in requirements engineering: The landscape and challenges,” IEEE Software, vol. 34, no. 2, p. 44–52, 2017. DOI: 10.1109/MS.2017.33