Paper deep dive
From Regulation to Implementation: A Critical Evaluation of LLM-Assisted Regulatory Compliance in Industry
Adriana Watson, Marco BĂŒcheler, Grant Richards
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The European Union (EU) has emerged as a leading regulatory body in the development of sustainability and privacy regulations. While new regulation requirements vary, many include a documentation artifact to ensure compliance. Notably, the Ecodesign for Sustainable Products Regulation (ESPR) introduces Digital Product Passports (DPPs) for life cycle transparency, while the General Data Protection Regulation (GDPR) mandates Data Protection Impact Assessments (DPIAs) to mitigate privacy risks. Creating these compliance artifacts, however, is challenging. Industrial data, which often exists in heterogeneous formats and is scattered across company and supplier systems, is required for DPPs and can be difficult to extract into compliant DPP formatting. Furthermore, DPIA documents require interdisciplinary expertise and follow no standardized format, making development difficult for novel systems. To address the particular complexity of compliance artifact creation for both regulations, researchers have proposed the use of LLMs in the generation process; however, the impact of the aforementioned problems on the output of these systems is largely unaddressed. This work investigates the existing research gap by exploring how data extraction instructions and regulatory vagueness impact the quality and consistency of LLM-produced compliance artifacts. The resulting artifacts are evaluated by benchmarking different models against manually created ground-truth schemas. The results reveal that less strict guidelines, such as DPIA formatting, require higher context prompts to maintain consistency and completeness. Stricter guidelines, such as formatting for Digital Battery Passports (DBP), result in consistent results regardless of prompt context, but may lead to more hallucinations in the output
Tags
Links
- Source: https://arxiv.org/abs/2608.21317v1
- Canonical: https://arxiv.org/abs/2608.21317v1
Trouble viewing inline? Open PDF directly â
Full Text
50,485 characters extracted from source content.
Expand or collapse full text
1 From Regulation to Implementation: A Critical Evaluation of LLM-Assisted Regulatory Compliance in Industry Adriana Watson, Marco B Ì ucheler, Grant Richards AbstractâThe European Union (EU) has emerged as a lead- ing regulatory body in the development of sustainability and privacy regulations. While new regulation requirements vary, many include a documentation artifact to ensure compliance. Notably, the Ecodesign for Sustainable Products Regulation (ESPR) introduces Digital Product Passports (DPPs) for life cycle transparency, while the General Data Protection Regulation (GDPR) mandates Data Protection Impact Assessments (DPIAs) to mitigate privacy risks. Creating these compliance artifacts, however, is challenging. Industrial data, which often exists in heterogeneous formats and is scattered across company and supplier systems, is required for DPPs and can be difficult to extract into compliant DPP formatting. Furthermore, DPIA documents require interdisciplinary expertise and follow no standardized format, making development difficult for novel systems. To address the particular complexity of compliance artifact creation for both regulations, researchers have proposed the use of LLMs in the generation process; however, the impact of the aforementioned problems on the output of these systems is largely unaddressed. This work investigates the existing research gap by exploring how data extraction instructions and regulatory vagueness impact the quality and consistency of LLM-produced compliance artifacts. The resulting artifacts are evaluated by benchmarking different models against manually created ground- truth schemas. The results reveal that less strict guidelines, such as DPIA formatting, require higher context prompts to maintain consistency and completeness. Stricter guidelines, such as formatting for Digital Battery Passports (DBP), result in consistent results regardless of prompt context, but may lead to more hallucinations in the output. Index TermsâLLM applications, Data privacy, Regulatory compliance, Digital product passport, General Data Protection Regulation, Asset Administration Shell, Prompt engineering, Industry 4.0 I. INTRODUCTION In recent years, the European Union (EU) has led the way in the development and implementation of both environmental and technological regulations. Notably, the Ecodesign for Sustainable Products Regulation (ESPR) and General Data Protection Regulation (GDPR) have distinguished themselves as leading regulatory works that address data rights and product sustainability, respectively [1], [2]. As a by-product of the novelty of these regulations, re- searchers have voiced growing concerns regarding the indus- trial implementation of these regulations [3]â[5]. Notably, a key point of conflict can be found in the challenges associ- ated with producing the required documentation to conform All authors are with the School of Engineering Technology, Purdue Uni- versity, West Lafayette USA (email: watso213@purdue.edu) to regulations; we henceforth refer to these documents as compliance artifacts. Although compliance artifacts such as Digital Product Passports (DPP) for the ESPR and Data Protection Impact Assessments for (DPIA) the GDPR are essential for both compliance and accountability, they each pose unique challenges that makes them difficult to produce manually. The proliferation of large language models (LLM) has brought forth the opportunity to develop novel solutions for the automation of key compliance artifacts. However, current research does not properly address how differences in prompting approaches impact the effectiveness of compliance artifact generation. We address this gap by first identifying the novel ways in which LLMs are being used to produce compliance artifacts and second, reproducing these methods with varying levels of prompt specificity in a case study to identify the extent to which LLM-generated compliance artifacts vary given differences in prompting. Notably, we select two distinct compliance artifacts to examine in the case study from two ends of the compliance spectrumâ one clearly defined but not yet deployed standard and one vaguely defined but fully deployed standard. This serves to represent the extreme ends of existing regulatory requirements as they relate to artifact creation mandates. I. BACKGROUND AND RELATED WORK To understand the landscape of LLM-generated compliance artifacts, we first explore what compliance artifacts are needed for the ESPR and GDPR to identify how LLMs are used in their generation. Next, we present literature on the automatic generation of these artifacts. We then briefly review existing literature discussing the trustworthiness of AI-generated ma- terials, particularly in industrial and regulatory spaces. A. Compliance Artifacts 1) ESPR/EU Battery Regulation: In the EU, the concept of Digital Product Passport (DPPs) was first introduced by the European Green Deal [6] and the Circular Economy Action Plan [7]. These passports are compliance artifacts designed to enable transparency and circularity for products sold in the European market. The ESPR regulation defines requirements for DPPs by outlining the structure and content needed in each artifact. To be considered a valid artifact, DPPs must be structured as a digital product record built on open standards. The passports must be directly linked to a physical product via a data carrier, such as a barcode or QR code. The carrier must arXiv:2608.21317v1 [cs.AI] 21 Aug 2026 2 link to a persistent, unique product identifier to ensure end- to-end life cycle traceability. While the exact data fields will be determined by product-specific delegated acts, the ESPR establishes a baseline for artifact compliance. A compliant DPP must act as a life cycle repository, containing unique operator and facility identifiers to map the supply chain, exact material compositions, and environmental performance indica- tors such as carbon and environmental footprints. Additionally, comprehensive documentation for product stakeholders (e.g., consumers) on how to safely repair, maintain, dismantle, and recycle the product at its end-of-life must be included [1]. While the structure and process involved in generating DPPs is an open and ongoing field of research, the European Commission has made the most progress in defining this process for batteries. The EU Battery Regulation introduces Digital Battery Passports (DBPs) with the primary objective of minimizing the environmental impact of batteries [8], [9]. As a result, DBPs require comprehensive data on a batteryâs identity, material composition (including recycled content and hazardous substances), and carbon footprint. Moreover, perfor- mance and durability metrics, detailed instructions for repair, repurposing guidelines, and end-of-life recycling need to be documented [8], [9]. 2) GDPR: While the ESPR is concerned with the lifecycle of physical products, the General Data Protection Regulation (GDPR) is a comprehensive rulebook on data lifecycles. Notably, it defines the requirements of data controllers for the collection, use, modification, storage, and deletion of data. Compliance with the GDPR generally requires system-wide implementation, where data rectification, removal, and collec- tion justifications must be embedded into the database system itself. However, the GDPR, like the ESPR, also requires explicit documentation as a key compliance stage. Notably, the GDPR introduced the requirement for a Data Protection Impact Assessment (DPIA), in Article 35, intended to ensure data controllers dealing with high-risk data have done their due diligence to evaluate the data life cycle [2]. Specifically, GDPR Article 35(7) outlines the following documentation expectations for the DPIA: 1) A description of the reasons for collecting the data and how it will be used. 2) An analysis of the appropriateness of the data processing as it relates to the previously cited purpose. 3) An analysis of the risks associated with the data pro- cessing operations, specifically as they relate to the data subjectâs data rights. 4) A thorough analysis of the risk mitigation protocols that will be used to maintain data subjectsâ security, privacy, and data rights. B. AI Systems for Regulatory Compliance 1) DPP generation: In industrial environments, data re- quired for regulatory compliance is frequently scattered across company and supplier systems. Additionally, various data formats, including information technology (IT) and operational technology (OT) data, are needed. Building an EU-compliant product life cycle record (such as a DPP), requires data from various sources and heterogeneous formats. As a result, transforming and bundling existing data into a standardized format across supply chain partners is a necessary step in DPP development. One standard that facilitates the generation of structured product life cycle representations is the Asset Administration Shell (AAS), which a variety of sources cite as an ideal candidate to create standardized DPPs [10], [11]. The AAS offers interoperable data exchange possibilities, making it a promising solution for companies to comply with the EUâs ESPR and Battery regulations. Data in the AAS is structured in submodels [12] which describe informational content or func- tional elements of a product (e.g., digital nameplate or carbon footprint submodel). Properties then define the characteristics of each submodel in detail. Creating and populating administration shells is, however, a challenging and labor-intensive process for many companies. As a result, existing literature proposes the use of LLMs to automate AAS creation. In such pipelines, LLMs extract core data attributes from industrial documents, perform semantic searches against established external dictionaries, and syn- thesize the information into structured JSON or XML for- mats compliant with AAS specifications. Empirical evaluations show that LLM-driven architectures can successfully automate the creation of error-free AAS instances with an effective generation rate of 62-79% [13]. 2) DPIA Generation: Industrial systems are also heavily impacted by the GDPR, particularly in how they collect, use, and maintain data. While many solutions for GDPR compliance in data systems relate to more traditional database automation approaches, LLMs have gained popularity in the solution space for a variety of applications [14] [15]. The majority of existing literature focuses primarily on two key aspects of GDPR compliance: regulation interpretation and verification. To enhance interpretation, many researchers have used LLMs as an intermediate interpreter of regulation. Notably, LLMs have been used in research to translate the stated regulation into more compliant data extraction systems [16] [17]. Beyond this, LLMs are also a core component in many novel compliance checking and verification pipelines [18] [19]. Language models have also been proposed as a key tool in generating initial data collection artifacts, notably the DPIA. As the GDPRâs specifications for the DPIA are relatively vague, many researchers have pointed to a great amount of variety in the interpretation and execution when producing this artifact [20]. Although some authorities have developed software to aid in the development of DPIAs, these solutions are not universal across the EU nor required, as the GDPR allows data controllers to use a method of their choice [21]. Korff also notes a key tension in the creation of such a document; notably, companies are used to framing risk assess- ments around risks associated with the companyâ the DPIA is a risk assessment framed around the risk to individuals. Despite this tension, very little literature explores the use of automation techniques or LLMs to assist in the generation of these artifacts [22]. In contrast, the use of LLMs to generate or assist in the creation of risk assessments more broadly is widely explored [23] [24]. Due to the complexity of DPIA 3 generation, it seems worthwhile, then, to explore how LLMs may be used to enhance the uniformity and thoroughness of these artifacts. C. Key Problems with LLM-Generated Artifacts Despite the promising nature of LLM-enhanced artifact generation, similar applications have encountered a variety of concerns that warrant caution. Famously, LLMs are known to hallucinate, or invent material rather than pulling from real sources [25]. In the context of AI-generated artifacts, this may result in the generation of invalid structures, information, or scenarios. Beyond hallucination, the creation of artifacts such as DPPs and DPIAs often requires specified domain knowledge. While LLMs may perform well in some scenarios due to their wide breadth of training data, this advantage may also cause inter- pretation problems, as some researchers have noted that LLMs struggle when it comes to interpreting or using specialized terminology or logic [24]. Finally, LLMs are highly susceptible to data leakage either via direct attacks or unintentional verbatim reproduction [26]â [28]. As many of the data controllers are using either internally protected data (in the case of DPP generation) or externally private data (as is the case for DPIA generation), these privacy risks are a cause for concern. All of these problems, however, do have mitigation strate- gies, including prompting strategies, combined LLM and RAG approaches, and the use of isolated models or federated learning [25], [29], [30]. Thus, it is worth investigating how these artifacts can be generated more effectively using LLMs. I. CASE-STUDY METHODOLOGY To observe how LLMs handle different prompting ap- proaches in the creation of compliance artifacts, we present a case study focused on the automated generation of an ESPR/EU Battery Regulation-compliant DPP and a GDPR- compliant DPIA. A. Experimental Setup Five distinct models, noted in Table I, were selected to represent a variety of developers and accessibility. ModelDeveloperAccess GPT-4oOpenAIAPI Claude 4.6 SonnetAnthropicAPI Meta-Llama-3.1-8B-InstructMetaOpen Mistral-7BMistral AIOpen Qwen2.5-7B-InstructHuggingFaceOpen TABLE I: Overview of Evaluated Language Models The models were all prompted with their default system settings and standard hyperparameters and given no token or word restrictions. Each run used the same structured input, described in Section I-B. Each model was run 3 times per iteration, resulting in a total output of 120 total runs ( 2 tasks (DPP and DPIA)Ă 4 vagueness levels (baseline, high, medium, low)Ă 5 modelsĂ 3 runs each). B. Prompting Strategy The models were prompted with a structured input based on the input task and predefined vagueness levels. 1) DBP Prompting Strategy: To generate the Digital Bat- tery Passport (DBP) outputs, the model was provided with a battery product description and regulatory context, followed by varying levels of contextual guidance to represent different vagueness levels. All four prompts began with the same core task formulation, instructing the model to generate a Digital Battery Passport in structured JSON format using Asset Administration Shell (AAS) principles and battery- related regulatory information. The prompts were designed to evaluate how varying levels of regulatory and structural guidance influence the completeness and compliance quality of the generated DBP outputs. The baseline prompt combines the battery product informa- tion with the regulatory text and instructs the model to generate a DBP aligned with the AAS Technical Data Submodel for batteries, as shown in Listing 1. The low-context prompt simplifies the instructions and only specifies that the model should generate a structured JSON document according to the AAS Technical Data Submodel format for batteries, while allowing the model to determine which fields are necessary for compliance. The prompt is shown in Listing 2. Listing 1: Baseline Prompt dbp_baseline.txt You are a compliance expert. Generate a Digital Batter Passport (DBP) for the following product. PRODUCT: Extract the material composition data for the BoilerCell Battery. PRODUCT_DATA REGULATORY TEXT: REGULATORY_TEXT Using the provided battery documentation and regulation information, build a Digital Battery Passport using the AAS Material Composition Submodel for batteries. Generate the DBP as a structured JSON document. The output must be a valid JSON object only - no prose outside the JSON. The medium-context prompt introduces additional guidance regarding the expected structure and mandatory technical information required for the DBP. It explicitly instructs the model to preserve the AAS template structure, avoid hallu- cinating values, and include several mandatory sustainability and material-related data categories relevant to the Battery Regulation. The prompt is provided in Listing 3. Finally, the high-context prompt extends the previous prompt by incorporating additional domain-specific context related to Asset Administration Shells (AAS), Digital Product Passports (DPP), and industrial interoperability standards. The prompt explicitly constrains the model to preserve the provided AAS submodel structure, extract only explicitly supported values from the technical documentation, and populate detailed 4 Listing 2: Low Context Prompt dbp_low.txt You are a compliance expert. Generate a Digital Battery Passport (DBP) for the following product . PRODUCT: Extract the material composition data for the BoilerCell Battery. PRODUCT_DATA Generate the DBP as a structured JSON document according to the Asset Administration Shell (AAS ) Material Composition Submodel format for batteries. Include all fields you believe are required for a complete and compliant Digital Battery Passport. The output must be a valid JSON object only - no prose outside the JSON. Listing 3: Medium Context Prompt dbp_medium.txt You are a compliance expert. Generate a Digital Battery Passport (DBP) for the following product . Generate the DBP as a structured Asset Administration Shell (AAS) JSON document. The output must be a valid JSON object only - no prose outside the JSON. Use the provided battery material composition specification document and populate the supplied AAS Material Composition submodel as provided in the template by the IDTA. PRODUCT: This document contains data for multiple battery products. Extract all material composition data - battery chemistry, materials with CAS numbers and masses, and hazardous substances - for the product with model name BoilerCell Battery. Do not mix in data from other products. PRODUCT_DATA KEY REQUIREMENTS: - Preserve the JSON structure - Extract only relevant material composition battery information as described in the template - Use metric units - Do not invent values - Leave unknown values empty MANDATORY DATA: - Battery chemistry - Critical raw materials - Materials used in the cathode, anode, and electrolyte - Hazardous substances - Impact of substances on the environment and on human health or safety technical battery properties such as voltage characteristics, efficiency metrics, resistance values, and lifetime indicators. The prompt also includes regulatory annotations and the complete IDTA technical data submodel template. The full prompt is provided in Appendix VII-A. 2) DPIA Prompting Strategy: To generate the DPIA out- puts, the model was provided with a role assignment and scenario, followed by varying levels of context to represent different vagueness levels. All four prompts began with the same role assignment and scenario, provided in Listing 4, which was designed to describe an industrial data usage Listing 4: GDPR Role and Scenario You are a compliance expert. Generate a complete Data Protection Impact Assessment (DPIA) for the following processing scenario. A pan-European parcel logistics operator is deploying an AI-driven workforce management platform across 18 EU distribution and sorting centers. The system continuously processes GPS location tracks, package-handling throughput, biometric entry and exit scans, and task completion patterns for approximately 12,000 warehouse operatives and delivery drivers. The platform uses these data streams to automate shift scheduling, generate real-time performance rankings, and produce automated underperformance flags that feed directly into HR review and employment decision workflows. Listing 5: Baseline Prompt gdpr_baseline.txt Generate the DPIA as a structured JSON document. Your output must contain four top-level sections that map explicitly to Article 35(7)(a), (b), ( c), and (d). Each sectionâs key should reference the relevant sub-paragraph (e.g., " article_35_7_a", "article_35_7_b", etc.). The output must be a valid JSON object only - no prose outside the JSON. scenario that includes high-risk elements (including evaluation components, automated decision making, and the application of new technical solutions), as outlined in existing literature [21]. The prompt then included varying degrees of supplemental information based on the prescribed vagueness level. The baseline prompt includes directly quoted text from GDPR 35(7), along with a clarification from Recital 84 followed by the instructions given in Listing 5. The low-context prompt simply follows the scenario with a simplified version of the instruction above, provided in Listing 6 The medium-context prompt includes a plain language sum- mary of the requirements outlined in GDPR 35(7) followed by a customized instruction statement provided in Listing 7. Finally, the high-context prompt further expands the key requirements list above into individual sections and mandates the model to include all sections in the output. The prompt, provided in Appendix VII-B, uses key information and struc- tural components outlined in existing literature [20] [21]. C. Evaluation To evaluate the outputs of the models, a gold standard schema was written for the DBP and DPIA scenarios, respec- Listing 6: Low Context Prompt gdpr_low.txt Generate the DPIA as a structured JSON document with clearly labeled sections covering all components you believe are required for a thorough impact assessment. The output must be a valid JSON object only - no prose outside the JSON. 5 Listing 7: Medium Context Prompt gdpr_medium.txt KEY REQUIREMENTS (plain-language summary): A DPIA must include four main components: 1. A systematic description of what the processing involves - what data is collected, from whom, for what purposes, how it flows through the organization, and how long it is retained. 2. An assessment of whether the processing is necessary and proportionate - whether the same goals could be achieved with less data or less intrusive means, what the lawful basis for processing is, and how data subject rights are preserved. 3. An assessment of the risks the processing poses to individualsâ rights and freedoms - what could go wrong, how likely each risk is, and how severe the harm would be, including for any vulnerable groups. 4. The measures you will put in place to address those risks - technical security controls, organizational policies, and the residual risk that remains after mitigation. Generate the DPIA as a structured JSON document that explicitly addresses each of these four components. The output must be a valid JSON object only - no prose outside the JSON. tively 1 . These schemas were based on structures and examples present in existing legal, industrial, and academic documenta- tion for each regulation to reflect current best practices. 1) Schemas: The DBP schema was created by [31]. The name of the submodel used for this experiment is âDigital Battery Passport Part 6: Material Composition (IDTA02035- 6)â. The values of an exemplary industrial battery was used and placed into the submodel template as the reference file where the LLM outputs are compared to. The DPIA schema combines two published templates, one with a more narrative structure and one with a more linear structure [20] [32]. It includes key questions from both tem- plates into several sections with both Boolean and written response outputs. The full schema is available in the linked codebase. Each field included a âRequiredâ boolean marker that indicated whether the information was required explicitly by the GDPR, or simply recommended for best practices based on existing templates and documentation. 2) Methodology for Analysis: The outputs were then eval- uated on the basis of two key factors: consistency and compli- ance with regulation. Consistency is important in this context as, once an automatic compliance artifact system is in place, it is expected to yield similar results given similar inputs. Compliance is also clearly a necessary element, as models that produce noncompliant results are not broadly useful in this context. Consistency was determined by assessing the structure and content of each run of the same type (same task, model, and vagueness level) to determine how stable the model outputs were given the same setup. A field stability score was 1 While the gold standard schema is not included here due to length restrictions, it is available in the linked codebase for the experiment. calculated for each field present in the output of each run to determine the presence of a given field across multiple runs. A field was considered stable if it was present in all runs (a stability score of 1.0). To evaluate compliance with the regulation, the gold stan- dard schema was used to compare each consolidated task/- vagueness/model output against regulatory requirements. The components correlating to each section of the appropriate schema were extracted from the model output and combined from outputs of the same task, vagueness level, and model. This information was then compared against the content required by the gold-standard schema, and a regulatory com- pliance score was calculated on the basis of the inclusion and completeness of required components. An assessment of the commonly excluded and included information for each model was then conducted for both regulatory tasks. IV. RESULTS A. ESPR/EU Battery Regulation Compliance 1) Consistency: As indicated by Table I, models per- formed only slightly differently across vagueness levels. Qwen-2.5 and GPT-4o maintained perfect cross-run consis- tency (1.00) at every context level. Llama-3.1 also achieved perfect cross-run consistency for all completed context levels but did not complete the high-context extraction scenario because the input exceeded the available context window. The perfect cross-run consistency observed for these models suggests that they resolve ambiguous inputs through a stable, deterministic extraction strategy. Mistral-7B also follows this trend but demonstrates a slight dip in performance at low context. Claude, by contrast, was the only model to exhibit meaningful drops in consistency, falling to 0.96 at low context and reaching a minimum of 0.77 given medium context. The reduced consistency at medium context indicates that Claude produced a structurally different output in roughly one in four repeated runs for the same moderately vague prompt. Notably, Claude recovered to full consistency (1.00) when given the high-context prompt. ContextQwenClaudeGPTLlamaMistral Baseline1.001.001.001.001.00 Low1.000.961.001.000.96 Medium1.000.771.001.001.00 High1.001.001.00â1.00 TABLE I: Cross-run consistency scores for DBP artifacts by model and vagueness level. 2) Compliance: Figure 1 shows the mean completeness scores based on the given fields in the DBP schema across all models and context levels. At baseline, all four models achieved perfect compliance (1.00), confirming that fully spec- ified inputs pose no challenge for any of the systems evaluated. Qwen-2.5, GPT-4o, Mistral-7B, and Llama-3.1 maintained this perfect score across all context levels (with the already mentioned exception of Llama-3.1, which was not capable of completing the high-context scenario, and Mistral-7B, which dropped slightly at low context). Claude was again the sole 6 exception, with completeness declining to 0.97± 0.06 at low context and 0.87± 0.08 at medium context before recovering to 1.00 at high context. The elevated standard deviation at medium vagueness further reflects the instability already observed in the consistency results. Fig. 1: Mean regulatory compliance (completeness) scores for ESPR artifacts by model and vagueness level. Figure 2 breaks down required field inclusion rates by model, averaged across all vagueness levels. Seven of the ten required fields were included at 100% by every model. The fields that differentiate models are confined exclusively to the hazardous substance sub-schema. Claude included Hazardoussubstanceconcentration in only 75% of outputs while Mistral-7B included this field in 92% of out- puts, Claude also included Hazardoussubstanceclass in 80%, and Hazardoussubstanceidentifier in 85%, while GPT-4o, Llama 3.1, and Qwen-2.5 achieved 100% inclusion across all fields. This pattern indicates that Claudeâs compliance deficit is not spread uniformly across the schema but is concentrated in hazardous substance fields. Fig. 2: Required field inclusion rates (%) for ESPR artifacts, averaged across all vagueness levels and sorted by lowest inclusion rate. Only Claude and Mistral exhibit sub-100% inclusion, confined to hazardous substance fields. B. GDPR Compliance 1) Consistency: As indicated by Table I, models per- formed in vastly different ways given different context lev- els. Claude returned equally inconsistent results across all context levels. GPT, Mistral, and Qwen demonstrated steady improvements in consistency with a large improvement at the highest context level. Finally, Llama demonstrated this same improvement, given the largest amount of context, but observed significantly worse performance at the lowest context level, where many other models observed an improvement at the lowest context level compared to the baseline. Mistral yielded the most consistent DPIA outputs, at 0.80 consistency when given the most context, while GPT-4o yielded the least consistent model, only 0.49 consistency, when given only the regulation text. Surprisingly, though, models behaved fairly consistently with each context level yielding fairly similar consistency scores. ContextClaudeGPTLlamaMistralQwen Baseline0.640.490.500.510.50 Low0.600.560.380.570.52 Medium0.650.580.550.620.64 High0.630.770.760.800.76 TABLE I: Cross-run consistency scores for GDPR artifacts by model and vagueness level. 2) Compliance: Figure 3 demonstrates the mean complete- ness scores calculated based on the inclusion of required fields in the DPIA schema. For most models tested, the highest context prompts yielded the most compliant outputs with the smallest deviation. The baseline produced completeness scores between the low and medium context levels. Similar to the results observed in the consistency evaluation, Claude produced stable but average completeness across all context levels, while all other models improved as more context was included in the prompt. The highest performing models were GPT-4o and Mistral at the highest context levels (93% complete), while the worst performing model was Llama-3.1 at the Lowest context level (49% complete). Fig. 3: Mean regulatory compliance (completeness) scores for GDPR artifacts by model and vagueness level. 7 Fig. 4: Required field inclusion rates (%) for GDPR artifacts, averaged across all vagueness levels. Figure 4 demonstrates the inclusion rate of required fields separated by model and averaged across all context levels. While some fields, such as identified risks, data subject cate- gories, data categories, and technical measures, were always included, others were frequently left out. Notably, compliance mechanisms, residual risks, and data subject rights were almost always excluded. Storage limitations, data minimization, and controllers & processors were included by some models and forgotten in others. In keeping with observations made earlier, Claude almost fully excluded components that other models included at least half of the time, such as residual risk, data minimization, and risk likelihood. Yet, in some cases, Claud outperformed its peers, as the only model to have all runs include controllers & processing and retention period fields. V. DISCUSSION A. Implications for Regulation The DBP results stand in notable contrast to those ob- served for the DPIA. Three of the four models maintained near-perfect compliance and consistency across all vagueness levels, suggesting that the concrete field structure of the AAS submodel compensates for prompt ambiguity in a way that the open-ended GDPR 35(7) requirements cannot. Claudeâs deviation from this pattern carries real compliance risks as fields regarding hazardous substances are legally re- quired under the EU Battery Regulation. Claudeâs outputs had full compliance at both baseline and high context. However, the degradation of performance at medium and low-context prompts suggests that partial context may actively mislead the model. Practitioners should therefore prefer either fully speci- fied prompts or minimal ones as given in the baseline scenario. From a cost perspective, minimal prompts are preferable as fully specified prompts come at a higher token cost. The consistency results for the GDPR highlight a core con- clusion that has been implied in existing literature but rarely tested: regulatory language is too vague to yield consistent results. With the exception of Llama-3.1, all of the models re- turned more consistent results with the lowest provided context compared to models provided with verbatim regulation. This seems to indicate that, in applications that seek to automate the DPIA process, providing the model with the regulation on its own will not yield consistent outputs; more context is needed. This finding is supported by the compliance findings, as the majority of models provided with more context returned results that met a higher number of required pieces of information. This seems to imply that regulation as-is does not provide enough specificity to yield compliant results. Furthermore, Figure 4 highlights the need for more specific language in regulations for identifying and discussing compliance mech- anisms, risks, and their relation to other articles of the reg- ulation. Many of the most frequently missed components (compliance mechanism, residual risk, data subject rights, storage limitations, data minimization) point directly to other GDPR article requirements or verbiage. This seems to indicate that existing templates and recommendations may fixate on the language used purely in GDPR 35 rather than looking at the context of the DPIA in the scope of the GDPR as a whole. Regulation that specify field requirements at a granular level are likely to improve LLM automated compliance further, while regulations that remain at the level of principle will continue to produce variable outputs without additional prompt engineering. B. Limitations The scope of the research, while detailed, only analyzes a small set of theoretical case studies that were designed based on existing scenarios. This was done to limit the scale of the study to verify the evaluation mechanisms and to avoid the use of private or otherwise protected data. Building out the case- study set to include real-world examples with in-use DBP and DPIA documentation would greatly increase the context and insights that could be gained. VI. CONCLUSION As regulations become increasingly complex and ubiqui- tous, it is necessary to carefully examine the tools we use to comply with them. Although LLMs show great promise in closing the gap between regulatory complexity, the work presented in this paper clearly demonstrates that the nature of the prompt matters. While established frameworks, such as those available for DBPs can circumvent the need for de- tailed prompts, practitioners must be more careful in defining instructions for less-defined documents such as the DPIA. 8 Although prompting strategy can have a significant impact on the consistency and completeness of outputs, more research is needed to properly define best practices for compliance artifact generation. Furthermore, to inspire more consistent compliance, the EU should consider formalizing output speci- fications in delegated acts and guidance documents. This work demonstrates that regulations on each end of the compliance artifact creation spectrum, from clearly defined to vaguely de- fined, suffer in unique ways when created in collaboration with LLMs. This highlights the need for further development and research surrounding prompting strategies and best practices before industrial implementation can be considered. REFERENCES [1] European Commission, âRegulation (EU) 2024/1781 of the European Parliament and of the Council of 13 June 2024 establishing a framework for the setting of ecodesign requirements for sustainable products, amending Directive (EU) 2020/1828 and Regulation (EU) 2023/1542 and repealing Directive 2009/125/EC (Text with EEA relevance),â 2024. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2024/1781/oj/eng [2] â, âGeneral Data Protection Regulation,â May 2018. [Online]. Available: https://gdpr-info.eu [3] E. Politou, E. Alepis, and C. Patsakis, âForgetting personal data and revoking consent under the GDPR: Challenges and proposed solutions,â Journal of Cybersecurity, vol. 4, no. 1, p. tyy001, Jan. 2018. [Online]. Available: https://doi.org/10.1093/cybsec/tyy001 [4] A. Huselmann and B. Custers, âThe Right to Rectification and Inferred Personal Data,â Rochester, NY, Dec. 2024. [Online]. Available: https://papers.ssrn.com/abstract=5116221 [5] M. R. N. King, P. D. Timms, and S. Mountney, âA proposed universal definition of a Digital Product Passport Ecosystem (DPPE): Worldviews, discrete capabilities, stakeholder requirements and concerns,â Journal of Cleaner Production, vol. 384, p. 135538, Jan. 2023. [Online]. Available: https://w.sciencedirect.com/science/article/pii/S0959652622051125 [6] European Commission, âThe European Green Deal,â 2019. [Online]. Available: https://eur-lex.europa.eu/resource.html?uri=cellar:b828d165 -1c22-11ea-8c1f-01a75ed71a1.0002.02/DOC 1&format=PDF [7] â, âA new Circular Economy Action Plan For a cleaner and more competitive Europe,â 2020. [Online]. Available: https://eur-lex.europa.eu /legal-content/EN/TXT/?qid=1583933814386&uri=COM:2020:98:FIN [8] â, âRegulation (EU) 2023/1542 of the European Parliament and of the Council of 12 July 2023 concerning batteries and waste batteries, amending Directive 2008/98/EC and Regulation (EU) 2019/1020 and repealing Directive 2006/66/EC (Text with EEA relevance),â 2023, legislative Body: EP, CONSIL. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2023/1542/oj [9] Battery Pass consortium, âBattery Passport Content Guidance,â 2023, version 1.1. [Online]. Available: https://thebatterypass.eu/assets/images/ content-guidance/pdf/2023 BatteryPassportContentGuidance.pdf [10] M. Pourjafarian, C. Plociennik, M. H. Rimaz, P. Stein, M. Vogelge- sang, C. Li, S. Knetsch, S. Bergweiler, and M. Ruskowski, âA Multi- Stakeholder Digital Product Passport Based on the Asset Administration Shell,â in 2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation (ETFA), 2023, p. 1â8. [11] D. Palm, N. Kuenster, F. Dorka, and M. Buecheler, âArchitectures And Systems For Identifying Assets In Circular Supply Chains Using Digital Product Passports And The Asset Administration Shell,â in Proceedings of the CPSL 2024, D. Herberger and M. Hbner, Eds. Hannover: publish- Ing, 2024, p. 898â905. [12] N. Kuenster, F. Dietrich, and D. Palm, âOpportunities And Challenges Of The Asset Administration Shell For Holistic Traceability In Supply Chain Management,â Jul. 2023. [13] Y. Xia, Z. Xiao, N. Jazdi, and M. Weyrich, âGeneration of Asset Administration Shell With Large Language Model Agents: Toward Semantic Interoperability in Digital Twins in the Context of Industry 4.0,â IEEE Access, vol. 12, p. 84 863â84 877, 2024. [14] S. Maguire, J. Friedberg, M.-H. C. Nguyen, and P. Haynes, âA metadata-based architecture for user-centered data accountability,â Electronic Markets, vol. 25, no. 2, p. 155â160, Jun. 2015. [Online]. Available: https://doi.org/10.1007/s12525-015-0184-z [15] F. Hublet, D. Basin, and S. Krsti, âEnforcing theGDPR,â in Computer SecurityESORICS 2023, G. Tsudik, M. Conti, K. Liang, and G. Smaragdakis, Eds.Cham: Springer Nature Switzerland, 2024, p. 400â422. [Online]. Available: https://link.springer.com/chapter/10.100 7/978-3-031-51476-020 [16] S. Hassani, âEnhancing Legal Compliance and Regulation Analysis with Large Language Models,â in 2024 IEEE 32nd International Requirements Engineering Conference (RE), Jun. 2024, p. 507â511, iSSN: 2332-6441. [Online]. Available: https://ieeexplore.ieee.org/abstra ct/document/10628489 [17] I. M. Hasna, G. Moise, and D. Volosevici, âLLM-based Assistant for GDPR Compliance in the Employment Relationships.â [18] M. Alecci, N. Sannier, M. Ceci, S. Abualhaija, J. Samhi, D. Bianculli, T. Bissyand, and J. Klein, âToward LLM-Driven GDPR Compliance Checking for Android Apps,â in Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, ser. FSE Companion â25.New York, NY, USA: Association for Computing Machinery, Jul. 2025, p. 606â610. [Online]. Available: https://dl.acm.org/doi/10.1145/3696630.3728508 [19] L. Garza, L. Elluri, A. Piplai, A. Kotal, D. Gupta, and A. Joshi, âPrivComp-KG: Leveraging KG and LLM for Compliance Verification,â in 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), Oct. 2024, p. 97â106. [Online]. Available: https://ieeexplore.ieee.org/abstract/doc ument/10835639 [20] L. Dalla Corte and R. van Brakel, âData protection impact assessment methods for the urban environment: A report for the Commissie Per- soonsgegevens Amsterdam (CPA),â Tech. Rep., Aug. 2022. [21] D. Korff, âGDPR Requirements on Data Protection Impact Assessments & Methodologies for DPIAs,â Rochester, NY, Jul. 2020. [Online]. Available: https://papers.ssrn.com/abstract=3656234 [22] M. Iacono, M. Mastroianni, F. Palmieri, and A. Saporito, âA Tool to Support Automation of Risk Assessment,â in Computational Science and Its Applications ICCSA 2024 Workshops, O. Gervasi, B. Murgante, C. Garau, D. Taniar, A. M. A. C. Rocha, and M. N. Faginas Lago, Eds. Cham: Springer Nature Switzerland, 2024, p. 126â143. [23] E. Valkama, âUsing Generative AI to Assist in Writing IT Compliance Documentation,â Ph.D. dissertation, Aalto University, 2025. [Online]. Available: https://aaltodoc.aalto.fi/server/api/core/bitstreams/6249fce7-9 3eb-40a1-9885-e5b2acf5a7b5/content [24] Z.A.Collier,R.J.Gruss,andA.S.Abrahams, âHowgoodarelargelanguagemodelsatproductrisk assessment?â Risk Analysis, vol. 45, no. 4, p. 766â789, 2025, eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/risa.14351. [On- line]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/risa.1 4351 [25] Z. Ji, T. Yu, Y. Xu, N. Lee, E. Ishii, and P. Fung, âTowards Mitigating LLM Hallucination via Self Reflection,â in Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali, Eds.Singapore: Association for Computational Linguistics, Dec. 2023, p. 1827â1843. [Online]. Available: https://aclanthology.org/2023.findings-emnlp.123/ [26] N. Carlini, F. Tramr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, . Erlingsson, A. Oprea, and C. Raffel, âExtracting Training Data from Large Language Models,â 2021, p. 2633â2650. [Online]. Available: https://w.usenix.org/conference/us enixsecurity21/presentation/carlini-extracting [27] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, âNot What Youâve Signed Up For: Compromising Real- World LLM-Integrated Applications with Indirect Prompt Injection,â in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, ser. AISec â23.New York, NY, USA: Association for Computing Machinery, Nov. 2023, p. 79â90. [Online]. Available: https://dl.acm.org/doi/10.1145/3605764.3623985 [28] S. Lin, Krishnamurthy, Dvijotham, J. Hayes, C. Shi, I. Shumailov, and S. Song, âLarge Language Models Can Verbatim Reproduce Long Malicious Sequences,â Mar. 2025, arXiv:2503.17578 [cs]. [Online]. Available: http://arxiv.org/abs/2503.17578 [29] S. Abualhaija, M. Ceci, N. Sannier, D. Bianculli, S. Lannier, M. Siclari, O. Voordeckers, and S. Tosza, âLLM-assisted Extraction of Regulatory Requirements: A Case Study on the GDPR,â in 2025 IEEE 33rd International Requirements Engineering Conference (RE), Sep. 2025, p. 142â154, iSSN: 2332-6441. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/11190380 [30] L. Li, Y. Fan, M. Tse, and K.-Y. Lin, âA review of applications in federated learning,â Computers & Industrial Engineering, vol. 149, p. 9 106854, Nov. 2020. [Online]. Available: https://w.sciencedirect.co m/science/article/pii/S0360835220305532 [31] Industrial Digital Twin Association e.V., âDigital Battery Passport - Part 6 Material Composition (Version 1.0),â 2026. [Online]. Available: https://github.com/admin- shell- io/submodel- templates/tree/main/publi shed/Digital%20Battery%20Passport/6Material%20Composition/1/0 [32] âTools for completing a data protection impact assessment (DPIA),â Oct. 2024. [Online]. Available: https://ico.org.uk/for-organisations/uk-g dpr- guidance- and- resources/childrens- information/childrens- code- gui dance-and-resources/dpia-tools/ VII. APPENDIX A. ESPR High Context Prompt The full high-context prompt to produce a DBP is included below, split into two sections for readability. Listing 8: High Context Prompt (1) dbp_high.txt CONTEXT: You are a compliance expert. Generate a Digital Battery Passport (DBP) for the following product . Generate the DBP as a structured JSON document. Your output must map explicitly to the required AAS submodel fields referenced in the regulatory text above. The output must be a valid JSON object only - no prose outside the JSON. You are also an expert in Asset Administration Shells (AAS), and battery material composition documentation. Your task is to generate a valid AAS JSON representation for the DBP AAS Material Composition submodel. Generate the DBP as a structured JSON document. The output must be a valid JSON object only - no prose outside the JSON. The Digital Battery Passport is part of emerging industrial data interoperability standards for battery lifecycle transparency, sustainability, and traceability. The Material Composition submodel contains structured battery properties including: - manufacturer information - voltage characteristics - capacity and energy values - efficiency metrics - resistance metrics - temperature boundaries - lifetime indicators - power capability information Only extract information that is explicitly supported by the provided material composition documentation. Do not hallucinate missing values. Preserve the JSON structure and field names from the provided template. Units must remain metric. If information is not available in the source document, leave the corresponding value empty. You are provided with: 1. A regulatory context document 2. A battery material composition document 3. An AAS Material Composition submodel template Listing 9: High Context Prompt (2) dbp_high.txt Generate the final AAS JSON. PRODUCT: This document contains material composition data for three battery products. Extract only the data for the product with Product Model = "BoilerCell Battery" (Chemistry Short Name: NMC, Total Pack Mass: 450.0 kg). All material masses, hazardous substance concentrations, and CRM flags must come exclusively from Sections 4, 5, and 6 of this document. Ignore Section 7 (Cross-Product Shared Components Register) entirely - those values must not be attributed to this product. PRODUCT_DATA REGULATORY TEXT AND ANNOTATIONS: REGULATORY_ANNOTATIONS SUBMODEL TEMPLATE: IDTA_TEMPLATE B. GDPR High Context Prompt A slightly truncated version of the high-context prompt used to generate a DPIA is provided in Listing 10. Listing 10: High Context Prompt (1)gdpr_high.txt 1. PROCESSING DESCRIPTION - Categories of personal data collected and their sensitivity - Categories of data subjects and their relationship to the controller - Stated purposes of processing and the lawful basis for each under Article 6 - Data flows: how data is collected, stored, accessed, transferred, and deleted - Retention periods for each data category - Recipients or categories of recipients ( internal and external) 2. NECESSITY AND PROPORTIONALITY - Justification that each data category is strictly necessary for the stated purposes - Assessment of whether less privacy-intrusive alternatives could achieve the same outcomes - Data minimization and purpose limitation measures in place - Provisions for data subjects to exercise their rights (access, rectification, erasure, restriction, objection, portability) - Storage limitation justification 3. RISK ASSESSMENT - Enumeration of specific risks to data subjects (e.g., unlawful automated employment decisions, surveillance chilling effects, discriminatory profiling, data breach, function creep) - Likelihood rating for each risk (low / medium / high) - Severity rating for each risk (low / medium / high) - Overall risk level for the processing activity - Any heightened risks for vulnerable subgroups C. Code and Datasets All code, including the gold-standard ground truth data and files, can be found at: this link. The material used for DPP generation can be found at this link. 10 Listing 11: High Context Prompt (2)gdpr_high.txt 4. RISK MITIGATION MEASURES - Technical security measures (e.g., encryption at rest and in transit, pseudonymisation, access controls, audit logging) - Organizational measures (e.g., employee notification procedures, DPO involvement, processor contracts, staff training) - Mechanisms ensuring data subjects can contest automated decisions - Residual risk level after controls are applied - Whether prior consultation with the supervisory authority is required