Paper deep dive
Actionable Meta-Requirements for AI Robustness Auditing
Joshua Owotogbe, Roya Nasiri, Indika Kumara, Damian Andrew Tamburri, Willem-Jan van den Heuvel
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI-enabled systems must remain dependable when the conditions under which they operate change (e.g., due to input perturbations, distribution shifts, or changes in external dependencies). This makes robustness auditing of such systems important for judging whether the available evidence is sufficient to support a defined robustness conclusion. However, designing an AI robustness auditing framework is challenging. A key unresolved issue is determining the requirements such a framework should support; these requirements have not yet been consolidated. We address this gap through meta-requirements: higher-order requirements specifying what a class of auditing frameworks should support. In design science research, literature and domain experts are common sources of meta-requirements; accordingly, we derive the initial set from literature and validate and refine it with practitioners. Our systematic literature review retained 74 of 2,890 records and derived 43 meta-requirements across 16 categories. We evaluated them with 128 practitioners, obtaining 5,504 ratings and 586 substantive comments. Overall, 81.20% of ratings indicated agreement or strong agreement. In particular, 41 of the 43 literature-derived meta-requirements were classified as strongly supported. Practitioner feedback refined the requirements and identified an additional requirement concerning external components and supply chain dependencies. The final set comprises 44 meta-requirements across 16 categories, of which 43 received quantitative validation. These meta-requirements can help organizations implement AI robustness auditing frameworks, assess existing frameworks for completeness, and refine them.
Tags
Links
- Source: https://arxiv.org/abs/2610.14356v1
- Canonical: https://arxiv.org/abs/2610.14356v1
Trouble viewing inline? Open PDF directly โ
Full Text
233,008 characters extracted from source content.
Expand or collapse full text
Actionable Meta-Requirements for AI Robustness Auditing Journal: TOSEMCCS: Software and its engineering Software verification and validationCCS: Computing methodologies Artificial intelligenceCCS: Computing methodologies Machine learning Joshua Owotogbe email: j.s.owotogbe@tilburguniversity.edu Affiliation: Jheronimus Academy of Data Science, โs-Hertogenbosch, North Brabant, Netherlands Affiliation: Tilburg University, Tilburg, North Brabant, Netherlands , Roya Nasiri email: r.nasiri@tilburguniversity.edu Affiliation: Jheronimus Academy of Data Science, โs-Hertogenbosch, North Brabant, Netherlands Affiliation: Tilburg University, Tilburg, North Brabant, Netherlands , Indika Kumara email: i.p.k.weerasinghadewage@tilburguniversity.edu Affiliation: Jheronimus Academy of Data Science, โs-Hertogenbosch, North Brabant, Netherlands Affiliation: Tilburg University, Tilburg, North Brabant, Netherlands , Damian Andrew Tamburri email: datamburri@unisannio.it Affiliation: Jheronimus Academy of Data Science, โs-Hertogenbosch, North Brabant, Netherlands Affiliation: University of Sannio, Benevento, Benevento, Italy and Willem-Jan van den Heuvel email: w.j.a.m.vdnHeuvel@tilburguniversity.edu Affiliation: Jheronimus Academy of Data Science, โs-Hertogenbosch, North Brabant, Netherlands Affiliation: Tilburg University, Tilburg, North Brabant, Netherlands ยฉ , Abstract. AI-enabled systems must remain dependable when the conditions under which they operate change (e.g., due to input perturbations, distribution shifts, or changes in external dependencies). This makes robustness auditing of such systems important for judging whether the available evidence is sufficient to support a defined robustness conclusion. However, designing an AI robustness-auditing framework is challenging. A key unresolved issue is determining the requirements such a framework should support; these requirements have not yet been consolidated. We address this gap through meta-requirements: higher-order requirements specifying what a class of auditing frameworks should support. In design science research, literature and domain experts are common sources of meta-requirements; accordingly, we derive the initial set from literature and validate and refine it with practitioners. Our systematic literature review retained 74 of 2,890 records and derived 43 meta-requirements across 16 categories. We evaluated them with 128 practitioners, obtaining 5,504 ratings and 586 substantive comments. Overall, 81.20% of ratings indicated agreement or strong agreement. In particular, 41 of the 43 literature-derived meta-requirements were classified as strongly supported. Practitioner feedback refined the requirements and identified an additional requirement concerning external components and supply-chain dependencies. The final set comprises 44 meta-requirements across 16 categories, of which 43 received quantitative validation. These meta-requirements can help organizations implement AI robustness-auditing frameworks, assess existing frameworks for completeness, and refine them. Keywords: AI robustness, AI auditing, robustness auditing, meta-requirements, practitioner study, systematic literature review ยฉ Owner/Author 2026. This is the authorโs version of the work. It is posted here for your personal use. Not for redistribution. 1. Introduction AI-enabled systems are increasingly deployed in high-stakes domains such as healthcare, transportation, criminal justice, and financial services, where incorrect predictions, inconsistent decisions, or failures under unexpected operating conditions can affect safety, individual rights, and access to essential services (Rajpurkar et al., 2022; Kuutti et al., 2020; Dressel and Farid, 2018; Lessmann et al., 2015). In this study, we use AI-enabled systems to mean software systems in which at least one AI component contributes to the systemโs functionality (Martรญnez-Fernรกndez et al., 2022). Such systems may also depend on non-AI components, including data pipelines, services, frameworks, APIs, infrastructure, and other software elements that can influence their overall behaviour. For concision, we use AI auditing throughout this paper to refer to the auditing of AI-enabled systems. In these domains, strong predictive performance under ordinary test conditions is insufficient. AI-enabled systems may produce different or degraded outputs when inputs are noisy or perturbed, data distributions shift, external dependencies change, or deployment conditions depart from those assumed during development (Rajpurkar et al., 2022; Dressel and Farid, 2018; Freiesleben and Grote, 2023; Tocchetti et al., 2025). Robustness testing examines whether an AI-enabled system or its components maintain specified acceptable behaviour under defined changes to inputs or operating conditions. Such testing requires specifying the system or component being assessed, the expected behaviour, the disturbances or failure conditions considered, the evaluation procedure, and the acceptance criteria. (Freiesleben and Grote, 2023; Yang et al., 2020; Hendrycks and Dietterich, 2019; Taori et al., 2020; Tocchetti et al., 2025). Tests can produce observations about system behaviour under particular conditions, but an audit has a broader assurance role: it must judge whether the collected evidence is adequate for the stated robustness objective and make explicit the scope and limitations of that judgement. This broader role makes the design of robustness auditing frameworks non-trivial. A useful framework must determine which concerns belong within the audit, what forms of evidence are relevant, how evaluation criteria should be established, how limitations should be documented, and when previously reached conclusions need to be revisited. These questions are increasingly consequential in regulatory and risk-management contexts. Article 15 of the European Union Artificial Intelligence Act (EU AI Act) requires high-risk AI systems to achieve an appropriate level of accuracy, robustness, and cybersecurity and to perform consistently in these respects (European Parliament and Council of the European Union, 2024; Smuha, 2025; Nolte et al., 2025). Article 16 further requires providers to ensure compliance with these requirements; breaches of applicable provider obligations may be subject to administrative fines of up to EUR 15 million or, for undertakings, 3% of their total worldwide annual turnover for the preceding financial year, whichever is higher (Article 99(4)(a)) (European Parliament and Council of the European Union, 2024). The National Institute of Standards and Technology Artificial Intelligence Risk Management Framework (NIST AI RMF), while voluntary, likewise addresses AI trustworthiness through characteristics including validity and reliability, safety, security, and resilience (Tabassi, 2023). Such instruments set important expectations, but they do not, by themselves, provide an operational specification for a robustness auditing framework. Relevant knowledge for such a specification spans several research streams. Technical robustness research develops benchmarks, metrics, perturbation methods, attacks, defences, and evaluation procedures for examining AI-enabled systems under adversarial attacks, data corruptions, data distribution shifts, and other changes in relevant inputs or operating conditions (Hendrycks and Dietterich, 2019; Taori et al., 2020; Yang et al., 2020; Li et al., 2023; Agarwal and Nene, 2025; Tocchetti et al., 2025). Research on AI auditing, in turn, examines audit scoping, evidence collection and interpretation, governance, and documentation (Raji et al., 2020; Mรถkander, 2023; Minkkinen et al., 2022; Thelisson and Verma, 2024). However, to the best of our knowledge, no study has systematically reviewed such literature from the relevant domains and derived a catalog of requirements that can guide the development and evaluation of robustness auditing frameworks for AI-enabled systems (Section 3.3, Table 2). This study addresses this critical research gap. Design science research (DSR) offers a complementary perspective by expressing general design needs for an artifact, independently of a particular implementation (Walls et al., 1992; Zacharias et al., 2022; Hevner et al., 2004). Following the DSR terminology, we use the term meta-requirements to refer to these implementation-independent requirements that capture what a family of auditing frameworks for AI-enabled systems is expected to accommodate. In DSR, the literature provides a basis for identifying such requirements, while practitioner knowledge provides evidence to assess and refine their practical relevance. To derive an actionable set of meta-requirements for an AI robustness auditing framework, inspired by DSR methods, we conducted a mixed-method study (Rabiser et al., 2010; Fahmideh et al., 2021; Waseem et al., 2021) combining a systematic literature review with a practitioner validation survey. The database and supplementary searches identified 2,890 records, of which 74 met the eligibility criteria and quality-assessment threshold and were synthesised into 43 literature-derived meta-requirements across 16 categories and three dimensions. We then evaluated these requirements through a survey of 128 practitioners using Likert ratings and open-text feedback to examine support, refinement, challenge, and additional requirements emerging from professional experience. The findings show substantial convergence between the literature and practitioner perspectives. Overall, 81.20% of the 5,504 ratings indicated agreement or strong agreement; 41 of the 43 literature-derived meta-requirements were classified as strongly supported, and the remaining two as supported. Practitioner feedback primarily refined wording, boundaries, access and evidence conditions, implementation assumptions, and lifecycle considerations, while also identifying an additional concern related to externally controlled dependencies. Integrating both evidence sources resulted in a final practitioner-informed set of 44 meta-requirements organised into 16 categories. This article makes the following contributions: โข We derive a source-traceable set of AI robustness auditing meta-requirements from a systematic review of academic literature. โข We provide empirical evidence from practitioners on the perceived relevance and applicability of the literature-derived meta-requirements, using quantitative ratings and qualitative feedback to identify support, refinements, challenges, and previously underrepresented concerns. โข We discuss the implications of these findings for designing and evaluating future AI robustness auditing processes, methods, tools, and frameworks. โข We provide a replication package containing the review protocol, search strings, included studies, extraction forms, coding records, meta-requirement tables, survey instrument, and analysis outputs. The remainder of this article is organised as follows. Section 2 introduces the key concepts and scope of the study. Section 3 reviews related work. Section 4 describes the research methodology, including the systematic literature review, meta-requirement derivation, and practitioner survey. Sections 5 and 6 present the resulting meta-requirements and practitioner validation findings, respectively. Section 7 discusses the findings and their implications. Section 8 addresses threats to validity, and Section 9 concludes the article. 2. Background This section introduces the concepts needed to interpret the study. Table 1 summarises the terminology used throughout this article. The subsequent subsections show how robustness claims and evidence inform auditing and meta-requirements. Table 1. Key concepts and definitions used in this study. Concept Meaning in this study AI-enabled system A software system in which at least one AI component contributes to the systemโs functionality (Martรญnez-Fernรกndez et al., 2022). AI audit A systematic assessment of an AI-enabled system, its development or use, and relevant evidence against specified technical, organisational, legal, or ethical criteria (Mรถkander, 2023; Minkkinen et al., 2022). AI robustness The ability of an AI-enabled system to maintain specified acceptable behaviour when relevant inputs or operating conditions change (Freiesleben and Grote, 2023; Tocchetti et al., 2025; Wang et al., 2026). Robustness testing Testing whether an AI-enabled system or its components maintain specified acceptable behaviour under defined changes to inputs or operating conditions. (Wang et al., 2026; Yu et al., 2026; Hรฅkansson et al., 2023; Bayram and Ahmed, 2025; Hรฅkansson et al., 2023; Silva and De Franรงa, 2023) Adversarial robustness A form of robustness concerned with adversarial attacks, that is, deliberate attempts to cause incorrect or otherwise undesired system behaviour, for example by manipulating model inputs (Wang et al., 2026; Nolte et al., 2025; Liu et al., 2022). Non-adversarial robustness A form of robustness concerned with naturally occurring changes, such as noise, image blur, lighting changes, sensor degradation, or distribution shift, where the data encountered during evaluation or deployment differ from those represented during model development (Wang et al., 2026; Nolte et al., 2025; Freiesleben and Grote, 2023; Kemmerzell et al., 2025). Threat model The assumptions about what an attacker can know and what actions the attacker can perform against an AI-enabled system or model (Chen and Das, 2023). Robustness metric A quantitative measure used to report robustness, such as performance under specified test conditions, a robustness bound, or the probability that a robustness condition is satisfied (Wang et al., 2026). Acceptance criterion The condition used to determine whether an observed robustness result meets the required level of performance, such as a minimum accuracy, a maximum permitted output change, or a maximum probability of error (Wang et al., 2026). Robustness claim An explicit assertion that an identified system, model, or component will maintain stated acceptable behaviour under specified conditions and according to stated acceptance criteria (Freiesleben and Grote, 2023; Simbeck, 2024). Robustness evidence Information used to assess whether a robustness claim is supported, including test results, monitoring outputs, logs, analyses, and documentation (Raji et al., 2020; Momcilovic et al., 2026; Simbeck, 2024). AI robustness auditing A structured assessment of whether available evidence supports a robustness claim within a defined assessment scope and against stated criteria (Raji et al., 2020; Casper et al., 2024; Simbeck, 2024). Meta-requirement for AI robustness auditing A requirement that states what robustness auditing approaches should support across different AI-enabled systems, while allowing the specific method used to vary with the system and audit context (Walls et al., 1992; Hevner et al., 2004; Zacharias et al., 2022; Mรคntymรคki et al., 2023). Accuracy The extent to which the outputs or predictions produced by an AI-enabled system are correct with respect to its intended task and the evaluation measure being used (Nolte et al., 2025). Computational cost The amount of computational resources consumed or required to develop, evaluate, or operate an AI-enabled system, such as processing time, memory, energy, or computing capacity (Chen and Das, 2023). Explainability The extent to which the outputs, decisions, or behaviour of an AI-enabled system can be understood or explained to people who need to interpret, assess, or act on them (Liu et al., 2022). Fairness The extent to which an AI-enabled system avoids unjustified differences in outcomes for affected people or groups based on their characteristics (Mehrabi et al., 1908; Nanda et al., 2021; Liu et al., 2022; Wei and Liu, 2025). Bias A systematic tendency in data, models, or system behaviour that can produce consistently different or poorer outcomes for particular people, groups, or cases, and may contribute to unfairness (Mehrabi et al., 1908). Safety The extent to which the behaviour or outputs of an AI-enabled system avoid causing or contributing to harm to people, property, or the environment (Liu et al., 2022). Privacy The extent to which an AI-enabled system protects personal or sensitive information from unauthorized access, disclosure, leakage, or inference (Wei and Liu, 2025). 2.1. AI Robustness and Robustness Claims A robustness assessment must specify what is being assessed (e.g., a model or AI-enabled system), the behaviour or outcome to be evaluated together with its acceptance criterion (e.g., prediction accuracy and the minimum accuracy that must be maintained), and the condition under which it is evaluated (e.g., adversarial inputs or distribution shift) (Freiesleben and Grote, 2023; Tocchetti et al., 2025; Wang et al., 2026). These conditions matter because robustness is condition-specific: evidence obtained under one type of change does not establish robustness under every other type of change (Freiesleben and Grote, 2023; Nolte et al., 2025). The robustness requirement being assessed must therefore be expressed as a specific claim before its supporting evidence can be evaluated. For example, an image-classification robustness claim may require a classifier to maintain a specified minimum accuracy when test images are affected by Gaussian noise at a stated severity level; related tests may use blur, fog, or lighting changes to examine performance under other conditions (Oala et al., 2020; Wang et al., 2026). Although this example targets a model, the assessment object may extend to the wider AI-enabled system. While the model represents the trained AI component, an AI-enabled system may also include services, frameworks, toolkits, platforms, infrastructure, and other software components (Yu et al., 2026; Wang et al., 2026). Accordingly, this study considers the robustness of the overall AI-enabled system, while allowing individual robustness claims to focus on a particular model or other system component. 2.2. AI Robustness Auditing Once a robustness claim is specified, an AI robustness audit evaluates the available evidence against that claim, the assessment scope (i.e., the system or components included in the audit), and the criteria used to judge whether the claim is supported. The evidence may include test results, monitoring outputs, logs, data, documentation, and development or deployment records (Raji et al., 2020; Casper et al., 2024; Simbeck, 2024; Momcilovic et al., 2026). Testing the robustness of an AI-enabled system or its components provides one type of audit evidence. For the image-classification claim above, the test could apply the specified noise to the test images and measure whether the classifier maintains the required accuracy (Wang et al., 2026). Beyond such test results, the evidence available to the audit depends on the auditorโs level of access. With black-box access, the auditor can query the system and observe its outputs; with white-box access, the auditor may also inspect internal model information, such as model parameters (weights) and intermediate computations (activations); and with outside-the-box access, development and deployment information, such as code, documentation, data, and internal evaluation results, may also be available (Casper et al., 2024). 2.3. Meta-Requirements for Robustness Auditing The preceding discussion shows why robustness auditing cannot rely on a single fixed procedure. The relevant claim, assessment object, available evidence, and level of auditor access can differ across AI-enabled systems, so a test or evaluation method suitable for one case may not be suitable for another (Freiesleben and Grote, 2023; Tocchetti et al., 2025; Casper et al., 2024; Thelisson and Verma, 2024). Despite these differences, robustness audits share common needs, including defining what is being assessed, obtaining and evaluating relevant evidence, and documenting the basis for conclusions (Raji et al., 2020; Casper et al., 2024; Simbeck, 2024). These differences in how robustness audits are conducted, together with the needs they share, motivate the use of meta-requirements to specify what robustness auditing approaches should address across different AI-enabled systems. Design science research uses meta-requirements to express needs that should be addressed across a class of artefacts, methods, or approaches, while allowing different solutions to address those needs in different ways (Walls et al., 1992; Hevner et al., 2004; Gregor and Jones, 2007; Lee et al., 2011; Zacharias et al., 2022; Mรคntymรคki et al., 2023). In this study, a robustness auditing meta-requirement is actionable when it identifies a concern that can be examined in an audit and can guide the design or assessment of robustness auditing approaches, while the specific way in which that concern is addressed may vary with the AI-enabled system and audit context. Practitioner evaluation assessed the perceived relevance and practical applicability of these meta-requirements, but their effectiveness when applied in real AI robustness audits has not yet been evaluated. 3. Related Work This section positions our study against prior work on challenges in robustness assessment, robustness practices across the AI lifecycle, and existing approaches to AI auditing. 3.1. Challenges Relevant to Auditing AI Robustness Prior research highlights several challenges relevant to auditing AI robustness. First, robustness evaluations can differ in the attacks or input changes considered, their severity, the datasets used, the robustness metrics reported, and the acceptance criteria applied (Freiesleben and Grote, 2023; Wang et al., 2026; Liu et al., 2022). These differences make results difficult to compare and can limit the extent to which evidence from one evaluation setting supports conclusions in another (Freiesleben and Grote, 2023; Javed et al., 2024; Kemmerzell et al., 2025). Second, robustness failures can arise from interactions among multiple system layers and components rather than from the model alone (Yu et al., 2026; Blood et al., 2023). This makes it difficult to determine whether model-level testing is sufficient, which components should be included in the audit scope, and what evidence is needed for a system-level conclusion (Raji et al., 2020; Casper et al., 2024; Mรถkander et al., 2024). Third, thorough robustness assessment can require substantial computation, execution time, and infrastructure, particularly for verification, large-scale testing, and runtime monitoring (Wang et al., 2026; Gupta et al., 2025). These demands can limit the extent of testing and, consequently, the strength of the resulting audit evidence. Fourth, robustness interventions can involve trade-offs with predictive accuracy, efficiency, explainability, fairness, privacy, usability, or deployment cost (Yang et al., 2020; Shafique et al., 2020; Silva and Najafirad, 2020; Nanda et al., 2021; Kaya et al., 2026; Tocchetti et al., 2025; Chander et al., 2025). Such trade-offs can complicate audit conclusions because improvements in robustness may affect other system properties that are also relevant to acceptance. Finally, robustness evidence can become outdated after deployment. Changes in production data, the deployed model, or deployment conditions can make earlier assessment results less representative of the current system and may require monitoring or reassessment (Bayram and Ahmed, 2025; Li and Goel, 2025; Chen and Das, 2023). Together, these challenges motivate audit-oriented requirements addressing evaluation conditions, assessment scope, resource constraints, trade-offs, reassessment, and the basis and limitations of robustness conclusions. 3.2. Robustness Practices Across the AI Lifecycle Prior research places robustness-related activities across the AI lifecycle, including data preparation, model training, robustness evaluation, deployment, and continuous monitoring (Li et al., 2023; Bayram and Ahmed, 2025; Tang et al., 2026). First, at the data-preparation stage, relevant practices include checking and cleaning missing, inconsistent, or noisy data; augmenting the available data with additional examples; and constructing test data that represent relevant conditions, such as noise, blur, fog, lighting changes, or differences between development and deployment data (Silva and De Franรงa, 2023; Oala et al., 2020; Wang et al., 2026; Javed et al., 2024; Tang et al., 2026). Second, during model training, robustness can be improved through adversarial training, which exposes the model to adversarial examples; robust optimisation, which minimises worst-case loss under bounded input changes; regularisation techniques such as dropout; and architecture modifications designed to improve robustness (Yang et al., 2020; Xiong et al., 2022; Agarwal and Nene, 2025; Mohseni et al., 2022; Javed et al., 2024; Tang et al., 2026). Third, robustness evaluation examines how a trained model behaves under specified conditions. It can include adversarial testing, testing with common input changes, and evaluation on data from different distributions (Wang et al., 2026; Javed et al., 2024). Formal or statistical verification can additionally assess whether a specified robustness condition holds within a defined input region or with a stated probability (Wang et al., 2026). These assessment activities examine whether a trained model satisfies specified robustness conditions and should therefore be distinguished from techniques intended to improve robustness (Freiesleben and Grote, 2023). Finally, after deployment, robustness-related activities include estimating prediction confidence or uncertainty, rejecting predictions when confidence is insufficient, detecting inputs that differ from the training distribution or have been deliberately manipulated, and monitoring model behaviour during operation (Mohseni et al., 2022; Gupta et al., 2025). Production processes can also monitor distribution or concept drift, retain relevant documentation and logs, and define when model updates or retraining are required (Bayram and Ahmed, 2025; Li and Goel, 2025). Across the AI-enabled system lifecycle, robustness auditing must relate evidence to the assessed robustness claim, system or component within scope, acceptance criteria, and assessment conditions (Raji et al., 2020; Mรถkander et al., 2024; Thelisson and Verma, 2024). It must also document evidence limitations and determine when changes require reassessment. These needs motivate the consolidation of technical and procedural knowledge into audit-oriented meta-requirements. 3.3. Limitations of Prior Work on AI Robustness Auditing Prior work relevant to AI robustness auditing contributes complementary foundations from different research streams. Robustness-focused studies clarify how robustness can be understood and evaluated. Tocchetti et al. (Tocchetti et al., 2025) systematise robustness terminology and organise prior work around robustness-improvement, application-specific, and evaluation approaches, while Freiesleben and Grote (Freiesleben and Grote, 2023) develop a conceptual account of robustness and distinguish robustness testing within an audit from techniques intended to improve robustness. Together, these studies provide a basis for robustness assessment, but they do not define the requirements of a complete robustness auditing process. Studies on AI auditing provide guidance on how audits are organised and carried out. Raji et al. (Raji et al., 2020) propose a five-stage internal audit process that moves from defining the audit scope and examining the system context to collecting evidence, testing the system, and reflecting on the findings. Mรถkander et al. (Mรถkander et al., 2024) organise LLM auditing across the organisation developing the model, the model itself, and its downstream applications. Simbeck (Simbeck, 2024) structures auditing from defining the scope and assessment criteria through evaluation to subsequent monitoring. Together, these studies show how AI audits can be organised, but they do not define robustness-specific audit requirements. Regulatory and trustworthy-AI studies identify requirements and obligations that AI-enabled systems are expected to satisfy. Nolte et al. (Nolte et al., 2025) analyse the robustness and cybersecurity requirements of the EU AI Act and their implementation implications. Thelisson and Verma (Thelisson and Verma, 2024) examine how high-risk AI systems can be assessed against regulatory requirements before market entry and propose supporting assessment tools. Dรญaz-Rodrรญguez et al. (Dรญaz-Rodrรญguez et al., 2023) analyse seven requirements for trustworthy AI, including technical robustness and safety. Together, these studies clarify what AI-enabled systems should satisfy, but not the meta-requirements needed to design and conduct robustness audits. Requirements-oriented studies provide the closest methodological precedent. Mรคntymรคki et al. (Mรคntymรคki et al., 2023) derive seven meta-requirements for organisational AI governance through design science research, combining literature-based premises with insights from an industry-academia project involving organisational design partners. Zacharias et al. (Zacharias et al., 2022) derive three meta-requirements for explainable feature selection and evaluate a resulting software artefact with practitioners and researchers. Von Zahn et al. (von Zahn et al., 2025) derive four meta-requirements for assessing AI conformity, with their implemented assessment centred on fairness, explainability, and performance. Boufaied et al. (Boufaied et al., 2026) provide practitioner evidence on how fairness requirements are interpreted and managed across the AI/ML lifecycle. These studies provide useful methods for deriving and evaluating requirements, but the reviewed work does not consolidate robustness-specific audit requirements for stating what robustness is being claimed, which parts of the AI-enabled system are covered, which failures or operating changes should be tested, what evidence is needed to support the audit conclusion, and when reassessment is required. We also did not identify a study that derives such meta-requirements from the literature and evaluates them with practitioners (Table 2). Table 2. Comparison of representative studies relevant to AI robustness auditing meta-requirements. Ref. Year Objective Method Robustness Focus Audit/ Assur. Req./MR Output Pract. Eval. Findings / Limitation (Raji et al., 2020) 2020 Develop an end-to-end internal framework for algorithmic auditing for development of AI systems. Framework synthesis informed by auditing practices in adjacent disciplines; illustrative cases โ โ - โ Provides a broad internal AI auditing framework with staged evidence collection and testing, but no robustness-specific meta-requirements. (Zacharias et al., 2022) 2022 Develop a design framework for explainable feature selection and instantiate it as a software artefact. Design Science Research; literature and legal-document review; two expert focus groups โ โ 3 meta-requirements + design principles โ(n=12n=12) Derives and evaluates meta-requirements for explainable feature selection, not robustness auditing. (Mรคntymรคki et al., 2023) 2023 Derive meta-requirements for organisational AI governance frameworks. Design Science Research; literature-derived premises; industry-academia collaboration with organisational design partners โ โ 7 meta-requirements โ Derives meta-requirements for AI governance with organisational design-partner input, but does not develop robustness-specific auditing meta-requirements. (Freiesleben and Grote, 2023) 2023 Develop a unifying conceptual account of ML robustness based on targets and modifiers. Conceptual analysis โ โ - โ Provides a conceptual robustness framework and distinguishes robustness testing from robustness improvement, but no robustness auditing requirements framework. (Dรญaz-Rodrรญguez et al., 2023) 2023 Connect trustworthy-AI principles, ethics, regulation, and technical requirements. Multidisciplinary conceptual analysis โ โ 7 trustworthy-AI requirements analysed โ Analyses seven trustworthy-AI requirements, including robustness and safety, but does not derive robustness auditing meta-requirements. (Mรถkander et al., 2024) 2023 Develop a three-layered blueprint for auditing large language models. Literature review; conceptual synthesis โ โ 7 design claims; no meta-requirements โ Proposes coordinated governance, model, and application audits and seven design claims, but no robustness-specific meta-requirements. (Simbeck, 2024) 2024 Develop an auditing framework for learning analytics systems covering fairness, transparency, and robustness. Conceptual synthesis of audit approaches and domain-specific criteria โ โ - โ Provides a four-phase audit process in which robustness is assessed alongside fairness and transparency, but no robustness-specific meta-requirements. (Tocchetti et al., 2025) 2025 Reconcile terminology concerning AI robustness and organise robustness-improvement and evaluation approaches from a human-centred perspective. Systematic survey; thematic analysis โ โ - โ Systematises robustness concepts, methods, applications, and evaluation approaches, but does not develop an framework of audit requirements. (Nolte et al., 2025) 2025 Analyse robustness and cybersecurity requirements in the EU AI Act and their implementation implications. Legal analysis of AI Act provisions; comparison with ML literature โ โ Legal requirements analysed โ Clarifies legal robustness and cybersecurity requirements and implementation challenges, but does not develop robustness auditing meta-requirements. (von Zahn et al., 2025) 2025 Develop and evaluate a design framework and software artefact for AI conformity assessment. Design Science Research; literature and legal-document review; seven expert interviews; five focus groups โ โ 4 meta-requirements + design principles โ(n=21n=21) Provides four practitioner-evaluated meta-requirements for AI conformity assessment, with the implemented assessment centred on fairness and explainability rather than robustness. (Boufaied et al., 2026) 2026 Investigate how practitioners interpret and manage fairness requirements across the AI/ML software-development lifecycle. Qualitative interview study; 26 semi-structured interviews; thematic analysis โ โ Fairness-requirement findings + challenges framework โ Provides practitioner evidence on fairness requirements, but does not derive or evaluate robustness auditing meta-requirements. This study 2026 Derive literature-grounded meta-requirements for AI robustness auditing and evaluate them with practitioners. Systematic literature review + practitioner survey โ โ 43 literature-derived + 1 practitioner-informed meta-requirement โ(n=128n=128) Derives 43 literature-based meta-requirements, evaluates them with practitioners, and incorporates one additional meta-requirement from qualitative feedback, resulting in a final set of 44 meta-requirements. Note. โ = explicitly addressed; โ= partially addressed; โ = not addressed; - = no requirements or meta-requirements reported. Robustness Focus indicates whether robustness is a central concern of the study. Audit/Assur. indicates explicit audit, assurance, or regulatory-assessment coverage. Req./MR Output distinguishes analysed requirements from explicitly derived meta-requirements and design principles. Pract. Eval. indicates empirical evaluation of a proposed framework, requirements, or artefact with practitioners or domain experts. 4. Methodology This study adopts a mixed-methods design (Rabiser et al., 2010; Fahmideh et al., 2021; Waseem et al., 2021) to derive and validate meta-requirements for AI robustness auditing. It combines a systematic literature review, used to derive the literature-grounded meta-requirements, with a practitioner validation survey, used to assess and refine them based on professional experience. The review component was guided by established systematic review guidelines (Kitchenham et al., 2007; Garousi and Felderer, 2017; Garousi et al., 2017). The research process comprises four phases: (1) literature search and screening, (2) meta-requirement extraction and refinement, (3) practitioner validation survey, and (4) meta-requirement synthesis and reporting (see Figure 1). Figure 1. Overview of the four-phase research method used to derive, refine, validate, and synthesize the meta-requirements. 4.1. Research Questions This study is guided by two research questions. RQ1 identifies the meta-requirements for AI robustness auditing, while RQ2 examines practitionersโ perspectives on those meta-requirements (see Table 3). Table 3. Research questions and motivation. Research Question Motivation RQ1: What meta-requirements should an AI robustness auditing framework satisfy? This question identifies the requirements that should guide the design and conduct of AI robustness auditing. RQ2: What are practitionersโ perspectives on the proposed meta-requirements for AI robustness auditing? This question examines practitionersโ support for, qualifications of, challenges to, and suggested refinements or additions to the proposed meta-requirements. RQ1 is addressed through Phases 1 and 2, which identify, code, and consolidate audit-relevant evidence from the literature into meta-requirements. RQ2 is addressed through Phases 3 and 4, which analyse practitioner ratings and open-text comments and integrate this validation evidence into the final set of meta-requirements. 4.2. Phase 1: Literature Search and Screening Following established systematic-review guidance (Kitchenham et al., 2007), we constructed the search around AI/ML systems, robustness-related concerns, and audit-relevant assessment activities. To focus on studies with practical relevance to robustness auditing, the search strings additionally required terms referring to practitioners (practitioner, developer, engineer, or stakeholder) and terms referring to empirical studies (interview or survey). Pilot searches indicated that removing these restrictions substantially increased the number of irrelevant records. We therefore retained the restrictions to improve search precision, while recognising that they could exclude relevant studies that did not use this terminology. Candidate studies were retrieved from three academic databases: IEEE Xplore (N=1,292N=1,292), ACM Digital Library (N=925N=925), and Scopus (N=473N=473) (see Phase 1 in Figure 1). We selected these databases because they provide broad coverage of software engineering, artificial intelligence, and information systems research (Chen et al., 2010). The search was restricted to studies published between January 2019 and February 2026, and all results were exported in BibTeX and CSV formats for reference management and data processing. Table 4 reports the final search strings and per-database retrieval counts. Table 4. Search Strings and Results Digital Library Search String Retrieved Articles Numbers ACM [[Abstract: "robustness"] OR [Abstract: resilien*] OR [Abstract: reliab*] OR [Abstract: dependab*]] AND [[Abstract: "artificial intelligence"] OR [Abstract: "machine learning"] OR [Abstract: "algorithmic"] OR [Abstract: "ai"] OR [Abstract: "ml"]] AND [[Abstract: audit*] OR [Abstract: assess*] OR [Abstract: test*] OR [Abstract: evaluat*]] AND [[Abstract: practitioner*] OR [Abstract: developer*] OR [Abstract: engineer*] OR [Abstract: stakeholder*]] AND [[Abstract: interview*] OR [Abstract: survey*]] 925 Scopus TITLE-ABS-KEY (("robustness" OR "resilien*" OR "reliab*" OR "dependab*") AND ("artificial intelligence" OR "machine learning" OR "algorithmic" OR "AI" OR "ML") AND ("audit*" OR "assess*" OR "test*" OR "evaluat*") AND ("practitioner*" OR "developer*" OR "engineer*" OR "stakeholder*") AND ("interview*" OR "survey*")) 473 IEEE Xplore ("All Metadata":"robustness" OR "All Metadata":resilien* OR "All Metadata":reliab* OR "All Metadata":dependab*) AND ("All Metadata":"artificial intelligence" OR "All Metadata":"machine learning" OR "All Metadata":"algorithmic" OR "All Metadata":"AI" OR "All Metadata":"ML") AND ("All Metadata":audit* OR "All Metadata":assess* OR "All Metadata":test* OR "All Metadata":evaluat*) AND ("All Metadata":practitioner* OR "All Metadata":developer* OR "All Metadata":engineer* OR "All Metadata":stakeholder*) AND ("All Metadata":interview* OR "All Metadata":survey*) 1292 Total - 2690 To complement the database searches, we conducted a supplementary Google Scholar search, following prior systematic reviews that used it to extend database-based retrieval (Gaber et al., 2024; Ajaykumar et al., 2021; Ihle et al., 2023). We examined the first 20 pages using terms reflecting the same AI/ML, robustness, audit, practitioner, and empirical-study scope as the database searches, and applied the same eligibility criteria. The database searches yielded 2,690 records, and the supplementary Google Scholar search added 200, resulting in 2,890 records for title/abstract screening. Following established SLR guidance (Islam et al., 2019; Butijn et al., 2020; Garousi et al., 2019), Table 5 presents the inclusion and exclusion criteria. Table 5. Inclusion and exclusion criteria. Inclusion Criteria Exclusion Criteria IC1: Studies must be written in English. EC1: Studies not written in English. IC2: Studies must be published between January 2019 and February 2026. EC2: Studies published outside the selected time window. IC3: Studies must be available in full text. EC3: Studies whose full text is unavailable or inaccessible. IC4: Studies must be academic research publications, including journal articles, conference papers, or workshop papers of at least six pages. EC4: Duplicate records, papers shorter than six pages, books, editorials, abstracts, posters, proceedings summaries, presentation-only papers, or other publication types lacking sufficient research content. IC5: Studies must examine AI, machine learning, deep learning, large language models, algorithmic systems, or AI-enabled software systems as the object of study. EC5: Studies outside the AI/ML domain, or studies that use AI/ML only as an analytical tool for prediction, classification, diagnosis, forecasting, optimisation, sensing, or domain analysis without examining the AI/ML system itself. IC6: Studies must address robustness or related dependability concerns of AI/ML systems, such as reliability, resilience, safety, fault tolerance, failure behaviour, distribution shift, out-of-distribution behaviour, adversarial robustness, or failure analysis. EC6: Studies that use robustness, reliability, or resilience only in a non-AI sense, or only to describe general predictive performance, efficiency, usability, adoption, trust, sustainability, educational outcomes, physical-system performance, or organisational resilience. IC7: Studies must connect AI/ML robustness or dependability to audit-relevant activities, including assessment, testing, evaluation, verification, validation, monitoring, assurance, compliance, governance, documentation, or evidence generation. EC7: Studies focused only on adjacent topics such as fairness, privacy, explainability, cybersecurity, traceability, forensics, ethics, policy, adoption, education, usability, or general trust without a direct connection to AI/ML robustness or dependability assessment. The selection process proceeded in five steps (see Phase 1 in Figure 1). Two researchers independently screened all 2,890 records at title/abstract level using IC1โIC5 and EC1โEC5, achieving 96.23% agreement (Cohenโs ฮบ=0.882ฮบ=0.882). The 109 disagreements were resolved by consensus, leaving 606 studies for full-text review. The same researchers independently assessed the 606 full texts using IC6โIC7 and EC6โEC7, achieving 94.22% agreement (ฮบ=0.741ฮบ=0.741); we resolved 35 disagreements by consensus. After reconciliation, 88 studies were retained for quality assessment. 4.2.1. Quality Assessment The 88 eligible studies were assessed using six quality criteria adapted from prior SLR practices (Bajaj and Sangwan, 2019; Browning and Bustard, 2024). The criteria examined whether: (QA1) the study objective or research question was clearly stated; (QA2) the study design or methodology was suitable for that objective; (QA3) the AI/ML system context was sufficiently described; (QA4) the data or evidence sources and analysis procedures were sufficiently described; (QA5) the findings and conclusions were supported by the reported evidence and analysis; and (QA6) relevant assumptions, limitations, or threats to validity were reported. Each criterion was scored as Yes (1), Partly (0.5), or No (0). Studies scoring at least 4 out of 6 were retained for the final synthesis. The first researcher scored all 88 studies against QA1-QA6, and the second researcher reviewed the completed quality assessments for all studies. Any differences identified during this review were discussed and resolved by consensus. After quality assessment, 74 studies were retained. The QA1-QA6 scores, total quality score, and final inclusion decision for each of the 88 assessed studies are provided in the replication package (see the Data Availability statement). Among the 74 studies retained after quality assessment, publication activity was limited in 2020 (N=4N=4) and 2021 (N=2N=2), followed by 11 studies in 2022, 14 in 2023, 13 in 2024, and a peak of 22 in 2025 (Figure 3). The 2026 count (N=8N=8) reflects the February 2026 search cut-off. Journal articles accounted for 50 studies, followed by conference papers (N=23N=23) and one workshop paper. Figure 3 shows that 33 studies appeared in Q1 venues and 28 in unranked venues, followed by Q2 (N=6N=6), A* (N=5N=5), and A (N=2N=2) venues. Venue ranking was reported descriptively and was not used as a proxy for study quality or as an exclusion criterion. Review rigour was instead supported through predefined eligibility criteria, independent screening, consensus resolution, and traceable evidence extraction. Following Galster et al. (Galster et al., 2013), lower-ranked and unranked studies were retained when they met the eligibility criteria and contributed relevant evidence. Figure 2. Annual distribution of the reviewed studies by publication venue type. Figure 3. Distribution of the reviewed studies by venue ranking. 4.3. Phase 2: Meta-Requirement Extraction and Refinement The 74 retained studies were analysed using a structured qualitative coding procedure to derive meta-requirements for AI robustness auditing (Nekrasov et al., 2026; Charmaz, 2006). The coding focused on how studies define robustness claims, specify audit scope, evaluate robustness, document evidence, assign audit responsibilities, involve stakeholders, monitor deployed systems, report limitations, and support reassessment. For each study, we used a standard extraction form to record the bibliographic details, AI/ML context, robustness concern, audit or evaluation activity, supporting quotation, extracted pattern, candidate category, and associated meta-requirement. The completed extraction records are provided in the replication package (see the Data Availability statement) and link each supporting quotation to its extracted pattern, candidate category, and literature-derived meta-requirement. The first stage applied open coding (Charmaz, 2006) to identify audit-relevant passages and summarise each as an extracted pattern. In the second stage, axial coding grouped related patterns by underlying audit concern, including patterns that spanned different domains and system contexts. For example, evidence concerning the effects of robustness interventions on accuracy, fairness, and explainability was progressively grouped into the broader concern of trade-off measurement (see Figure 4). The third stage converted the candidate categories into meta-requirements using explicit merge, separation, and retention criteria, following prior studies that consolidated lower-level evidence into higher-level tasks, categories, or requirements (Fahmideh et al., 2021; Nekrasov et al., 2026; Rabiser et al., 2010). Candidate categories were merged when they represented the same auditing requirement despite differences in terminology, technique, or application context. They were kept separate when merging them would remove a different audit responsibility, evidence requirement, or audit decision. A candidate category was retained as a meta-requirement when it was within the study scope, supported by traceable source evidence, applicable beyond a single technique or system, and not already represented by another meta-requirement. The resulting categories were assigned to three higher-level dimensions according to their primary purpose, following the use of explicit classification criteria in prior empirical studies (Sรกnchez et al., 2022; Ustunboyacioglu et al., 2024). Categories were classified as Human when they primarily concerned people, roles, governance, competence; as Functional when they primarily concerned audit activities or capabilities; and as Quality when they primarily concerned properties of the auditing approach itself. Where a category related to more than one dimension, its dominant purpose determined the assignment. Recurrence across studies increased the supporting evidence for a concern but was not used as a fixed inclusion threshold. A concern could therefore be retained when it met the criteria above even if it appeared in relatively few studies, while repeated evidence for the same higher-level concern did not result in additional meta-requirements. Following established principles for writing quality requirements (Heck et al., 2010; Heck and Zaidman, 2014), we refined the resulting meta-requirements for clarity, lack of ambiguity, atomicity, unique identification, linkage to supporting source evidence, audit assessability, and consistency across the set. We formulated each meta-requirement as an abstract, technology-agnostic statement specifying what a robustness auditing framework should support, require, or make explicit, and assigned each an identifier, category, and requirement statement. One researcher conducted the extraction, coding, and initial meta-requirement formulation. A second researcher independently reviewed the extracted patterns, category assignments, and resulting meta-requirements. The team resolved disagreements and ambiguities through discussion and consensus. This process produced 43 literature-derived meta-requirements across 16 categories, which formed the basis of the practitioner validation survey in Phase 3. Table 7 reports these together with MR-8.4, which was added after the survey and was therefore not rated by respondents. The replication package contains the corresponding extraction records, including the source passage, extracted pattern, category assignment, and resulting meta-requirement, allowing readers to inspect how the source evidence was consolidated. Figure 4. Example of how extracted literature evidence is coded and consolidated into MR-3.2, Robustness Trade-off Assumptions. 4.4. Phase 3: Practitioner Validation Survey The 43 meta-requirements derived in Phase 2 were evaluated through a practitioner survey addressing RQ2. Following established survey guidance in software engineering, we combined Likert-scale ratings with optional open-text feedback to assess practitioner support, interpretation, and suggested refinements, challenges, and additional concerns (Kitchenham and Pfleeger, 2002a; Kitchenham and Pfleeger, 2002b; Linรฅker et al., 2015). Comparable studies have similarly used practitioner surveys to assess and refine previously identified requirements and framework elements (van der Linden et al., 2019; Zellmer et al., 2026). 4.4.1. Survey Instrument Preparation We developed a Google Forms questionnaire comprising 44 numbered questions. First, the survey opened with a study-information and consent section describing the research purpose, expected completion time, voluntary participation, confidentiality arrangements, and researcher contact information. Second, Q2-Q5 collected demographic information, including age, gender, country or region, and education, while Q6-Q11 captured participantsโ professional profile, including AI/ML experience, robustness-assessment or auditing experience, academic and professional experience levels, AI/ML-related role, and involvement in robustness-related activities. Third, Q12-Q44 formed the validation section. The 43 meta-requirements were distributed across 16 matrix questions, one for each validation category, with each meta-requirement presented as a separate rating row. Respondents rated each meta-requirement using a five-point Likert scale from 1 = strongly disagree to 5 = strongly agree. Each matrix question was followed by an optional open-text question for comments or suggested changes, and the final question invited feedback on missing requirements, duplication, unclear wording, implementation barriers, or other improvements. The complete questionnaire is provided in the replication package (see the Data Availability statement). 4.4.2. Pilot Testing and Refinement Following prior practitioner-survey practice (van der Linden et al., 2019; Waseem et al., 2021), we piloted the survey from 24 June to 5 July 2026 with eight participants with relevant AI/ML, robustness, or auditing experience. The pilot assessed completion time, clarity of instructions and terminology, interpretation of the category descriptions and meta-requirements, suitability of the five-point scale, possible overlap, and usability across devices. Reported completion times ranged from approximately 15 to 30 minutes, with some reviewers finding completion more difficult on mobile devices. Based on the feedback, we shortened the introductory text, clarified category descriptions and meta-requirement labels, simplified wording, and retained the five-point scale and optional definitions. The eight pilot participants were not included in the main survey sample, and their responses were excluded from the main dataset and analysis. 4.4.3. Target Population, Recruitment, and Survey Administration Eligible participants were researchers and practitioners worldwide with academic, professional, or applied experience in AI/ML systems and related activities, including robustness assessment, model evaluation, adversarial evaluation, AI auditing, AI safety, model reliability, or assurance. We used purposive non-probability sampling because the study required participants with relevant experience in areas related to AI robustness auditing, while the specialised target population could not be comprehensively enumerated for probability sampling (Kitchenham and Pfleeger, 2008; Fahmideh et al., 2021; Waseem et al., 2021; Canedo et al., 2023). Potential participants were identified through GitHub and LinkedIn, channels that have also been used to recruit specialised practitioners in prior software engineering surveys (Fahmideh et al., 2021; Waseem et al., 2021; van der Linden et al., 2019). On GitHub, we searched relevant repositories and contributors using robustness-related queries and GitHubโs Best match ranking. Representative queries included "AI robustness", "robustness auditing", "adversarial robustness", and "ML robustness". Approximately 1,200 potential participants were identified through GitHub and 520 through LinkedIn. Survey invitations were sent to all 1,720 identified candidates. We received 143 responses, corresponding to a response rate of 8.31%. After applying the predefined response-screening criteria, 15 responses were excluded, leaving 128 valid responses for analysis. Survey invitations briefly introduced the study, stated the eligibility criteria, and directed potential participants to the study-information and consent page of the questionnaire (see Section 4.4.1). The study received ethical clearance from the Research Ethics and Data Management Committee of the Tilburg School of Humanities and Digital Sciences, Tilburg University (REDC 2026.04). The main survey was administered through Google Forms from 8 July to 15 August 2026, and informed consent was obtained before participation. After completing the survey, respondents could optionally enter a draw for a 100 gift card (De Stefano et al., 2022). Contact information submitted for the draw was collected separately from the survey responses. 4.4.4. Data Screening and Preparation Following prior practitioner-survey research (Waseem et al., 2021), we defined inclusion and exclusion criteria for screening the survey responses, as shown in Table 6. The inclusion criterion required respondents to demonstrate relevant academic, professional, or practical experience in at least one of the studyโs target areas, as assessed from their professional-profile responses. Responses containing inconsistent, apparently random, or meaningless answers were considered for exclusion, as were responses without informed consent. We received 143 responses to the main survey. After applying the predefined screening criteria, 15 responses were excluded under E1, leaving 128 valid responses for quantitative analysis. The responses were exported from Google Forms and processed in Python by identifying the 43 meta-requirement rating columns, standardising response labels, converting the five-point Likert responses to values from 1 to 5, checking missing values, assigning anonymous respondent identifiers, and transforming the ratings into long format. The resulting dataset contained 5,504 ratings, with no missing meta-requirement values. Open-text responses from the 128 valid respondents were screened separately because completion of these fields was optional. Across the 16 category-level comment fields and the final overall-comment field, 670 entries were extracted. Empty responses, uninterpretable statements, and comments without analytically usable feedback were removed, leaving 586 substantive comments for qualitative coding and synthesis. Each retained comment was linked to its anonymous respondent identifier, source category, and position within the corresponding response batch to preserve traceability. Table 6. Inclusion and exclusion criteria for selecting valid responses. Inclusion I1: The respondent must demonstrate relevant academic, professional, or practical experience in AI/ML systems, AI robustness assessment or auditing, robustness testing, model evaluation, adversarial evaluation, AI safety, model reliability, or related assurance activities. Exclusion E1: A response is excluded if it contains inconsistent, apparently random, or meaningless answers. E2: A response without informed consent is excluded. 4.4.5. Quantitative Analysis of Likert Ratings The 5,504 Likert ratings were analysed descriptively at three levels: the overall response distribution, the individual meta-requirement level, and the category level. This type of item-level and aggregated analysis has been used in prior practitioner surveys (Vogelsang et al., 2020). At the overall level, we calculated the frequencies and percentages of all five response options. At the meta-requirement level, each of the 43 literature-derived meta-requirements was analysed using the 128 corresponding participant ratings. At the category level, ratings were aggregated across the meta-requirements belonging to each of the 16 survey categories; these percentages summarise all ratings within a category and are not respondent-level agreement rates. For each meta-requirement and category, we reported response frequencies and percentages for agreement, neutral, and disagreement, as well as the median and interquartile range (IQR). Because individual Likert items are ordinal, the median and IQR were used as the primary descriptive summaries (Agresti, 2010; van der Linden et al., 2019). Mean, standard deviation (SD), and mode were reported only as supplementary descriptive measures. Consistent with prior practitioner-survey studies, the quantitative ratings were interpreted alongside open-text feedback to assess support for the proposed requirements and identify refinements or missing concerns (Fahmideh et al., 2021; van der Linden et al., 2019). Ratings of 4 (agree) and 5 (strongly agree) were grouped as agreement-level responses, a rating of 3 as neutral, and ratings of 1 (strongly disagree) and 2 (disagree) as disagreement-level responses. To provide a consistent descriptive interpretation of practitioner support, we defined study-specific quantitative-status rules for each meta-requirement. Prior consensus studies have used percentage-agreement thresholds to distinguish levels of agreement, including 60% as a minimum consensus threshold and 75% as a commonly used consensus threshold (Olivero et al., 2022; Diamond et al., 2014). A meta-requirement was classified as strongly supported when its median was at least 4 and at least 75% of ratings were agreement-level responses, and as supported when its median was at least 4 and agreement was at least 60% but below 75%. A meta-requirement was classified as mixed or unclear when its median was 3 or at least 40% of ratings were neutral, and as challenged when its median was at most 3 and at least 30% of ratings were disagreement-level responses. Cases not meeting these criteria were designated as requiring qualitative interpretation. These thresholds were analytic decision rules defined for this study, not inferential significance criteria. The quantitative results were interpreted together with the open-text comments to determine the extent to which the literature-derived meta-requirements were supported, refined, challenged, or extended (Fahmideh et al., 2021; van der Linden et al., 2019). No inferential comparisons between demographic subgroups were performed because the survey was designed to evaluate the meta-requirements rather than test differences between respondent groups. Table 7. Meta-requirements for AI robustness auditing: 43 literature-derived requirements and MR-8.4, added after the practitioner survey. ID Sub-Category Requirement Source(s) Human Dimension Category 1: Standards and Governance MR-1.1 Standardized Metrics Robustness auditing should use standardised robustness metrics and benchmark definitions where available, so that reported robustness results are comparable across systems, audits, and regulatory contexts. (Nolte et al., 2025; Khinvasara et al., 2024; Oala et al., 2020; Blood et al., 2023; Yu et al., 2026; Tocchetti et al., 2025; Yang et al., 2020; Agarwal and Nene, 2025; Chen and Das, 2023; Madaio et al., 2022; Freiesleben and Grote, 2023; Casper et al., 2024; Raji et al., 2020; Thelisson and Verma, 2024; Nanda et al., 2021; Hornischer, 2026; Javed et al., 2024; Liu et al., 2022; Li et al., 2023; Wei and Liu, 2025; Costanza-Chock et al., 2022) MR-1.2 Standardized Lexicon and Terminology Robustness audits should use a shared vocabulary, taxonomy, and terminology for robustness and auditability concepts. (Mohseni et al., 2022; Vadlapati, 2026; Yu et al., 2026; Ge et al., 2024; Tocchetti et al., 2025; Yang et al., 2020; Chen and Das, 2023; Freiesleben and Grote, 2023; Raji et al., 2020; Thelisson and Verma, 2024; Nanda et al., 2021; Nolte et al., 2025; Khinvasara et al., 2024; Hornischer, 2026; Liu et al., 2022; Li et al., 2023; Wei and Liu, 2025; Costanza-Chock et al., 2022) MR-1.3 Standardized Tools Robustness audits should use common or comparable tools, checklists, templates, scorecards, libraries, or toolkits where they support consistent assessment. (Raji et al., 2020; Momcilovic et al., 2026; Ojewale et al., 2025; Oala et al., 2020; Tocchetti et al., 2025; Dev et al., 2025; Yu et al., 2026; Ge et al., 2024; Yang et al., 2020; Agarwal and Nene, 2025; Casper et al., 2024; Kokina et al., 2025; Thelisson and Verma, 2024; Nolte et al., 2025; Khinvasara et al., 2024; Javed et al., 2024; Li et al., 2023; Costanza-Chock et al., 2022) MR-1.4 Actionable Guidance Practitioners should receive practical guidance for selecting tests, thresholds, evidence, mitigations, and audit steps. (Kokina et al., 2025; Khinvasara et al., 2024; Narula et al., 2025; Blood et al., 2023; Habbal et al., 2024; Thelisson and Verma, 2024; Cooper et al., 2022; Laux et al., 2024; Chen and Das, 2023; Saeed and Alsharidah, 2024; Xiong et al., 2022; McCormack et al., 2025; Koshiyama et al., 2024; Momcilovic et al., 2026; Chander et al., 2025) MR-1.5 Robustness-aware Integration Guidance Robustness guidance should support embedding robustness into development, deployment, monitoring, and assurance workflows. (Li et al., 2023; Yu et al., 2026; Tocchetti et al., 2025; Chen and Das, 2023; Madaio et al., 2022; Freiesleben and Grote, 2023; Casper et al., 2024; Kokina et al., 2025; Raji et al., 2020; Thelisson and Verma, 2024; Oala et al., 2020; Nolte et al., 2025; Khinvasara et al., 2024; Hornischer, 2026; Javed et al., 2024; Liu et al., 2022; Wei and Liu, 2025; Costanza-Chock et al., 2022) MR-1.6 Governance Protocols Organisational or regulatory protocols should define responsibility, escalation, remediation, accountability, and risk decisions for robustness assurance. (Habbal et al., 2024; Liu et al., 2022; Li and Goel, 2025; Cooper et al., 2022; Raji et al., 2020; Costanza-Chock et al., 2022; Mรถkander et al., 2024; Laux et al., 2024; McCormack et al., 2025; Chamola et al., 2023; Thelisson and Verma, 2024; Nolte et al., 2025) MR-1.7 Audit Access Policy Audit access policy should specify the artefacts and access level required for robustness assessment, including relevant datasets, documentation, source code, model or system information, deployment details, logs, internal evaluation results, APIs, and conditions for black-box, white-box, or outside-the-box access. (Li and Goel, 2025; Costanza-Chock et al., 2022; Raji et al., 2020; Koshiyama et al., 2024; Simbeck, 2024; Thelisson and Verma, 2024; Casper et al., 2024; Ojewale et al., 2025; Mรผller et al., 2025; Chander et al., 2025; Gupta et al., 2025) MR-1.8 Auditor Protocols Auditor protocols should specify how the audit is conducted, including auditor conduct, independence, method transparency, evidence handling, reporting format, and conditions under which audit results may be interpreted. (Raji et al., 2020; Costanza-Chock et al., 2022; Ojewale et al., 2025; Casper et al., 2024; Thelisson and Verma, 2024; Oala et al., 2020; Mรถkander et al., 2024; Ge et al., 2024; Tocchetti et al., 2025; Chen and Das, 2023; Madaio et al., 2022; Freiesleben and Grote, 2023; Kokina et al., 2025; Nanda et al., 2021; Nolte et al., 2025; Khinvasara et al., 2024; Liu et al., 2022; Li et al., 2023) Category 2: Engagement MR-2.1 Diverse Stakeholders Robustness audits should identify and involve relevant technical, domain, governance, user, and affected-party perspectives, including users or groups exposed to robustness failures, according to the robustness risks and impacts covered by the audit scope. (Tocchetti et al., 2025; Mรผller et al., 2025; Chander et al., 2025; Gupta et al., 2025; Habbal et al., 2024; Costanza-Chock et al., 2022; Li and Goel, 2025; Balendran et al., 2025; Mรถkander et al., 2024; Zahid et al., 2025; Raji et al., 2020; Silva and De Franรงa, 2023; Laux et al., 2024; Momcilovic et al., 2026; Ojewale et al., 2025; Madaio et al., 2022; Lee et al., 2023; McCormack et al., 2025; Robertson et al., 2023; Dev et al., 2025; Hancox-Li, 2020; Nanda et al., 2021; Nolte et al., 2025; Hornischer, 2026; Liu et al., 2022; Kaur et al., 2022) MR-2.2 Stakeholder Involvement Prioritization Stakeholder involvement should be prioritised according to each groupโs exposure to robustness failures, domain knowledge, decision authority, and relevance to deployment risks. (Agarwal and Nene, 2025; Madaio et al., 2022; Kokina et al., 2025; Raji et al., 2020; Thelisson and Verma, 2024; Oala et al., 2020; Nolte et al., 2025; Li et al., 2023; Costanza-Chock et al., 2022) MR-2.3 Independent Auditor Robustness audits should include independent review with conflict control, methodological transparency, and sufficient access for the intended assurance claim. (Ojewale et al., 2025; Casper et al., 2024; Li and Goel, 2025; Liu et al., 2022; Thelisson and Verma, 2024; Costanza-Chock et al., 2022; McCormack et al., 2025; Kokina et al., 2025; Raji et al., 2020; Habbal et al., 2024; Laux et al., 2024; Mรถkander et al., 2024) MR-2.4 Early-stage Involvement Domain and technical experts should be involved during problem framing, design, and assessment planning when early expert judgement is needed to define robustness-relevant assumptions, risks, or evidence. (Raji et al., 2020; Li and Goel, 2025; McCormack et al., 2025; Cooper et al., 2022; Madaio et al., 2022; Hornischer, 2026; Kaur et al., 2022) Category 3: Robustness Definition MR-3.1 Scoping Robustness Concerns The robustness scope should define the audit claim, system boundary, disturbance types, failure modes, deployment contexts, assumptions, and justified exclusions. (Dev et al., 2025; Xiong et al., 2022; Chander et al., 2025; Ge et al., 2024; Narula et al., 2025; Balendran et al., 2025; Hornischer, 2026; Liu et al., 2022; Blood et al., 2023; Gupta et al., 2025; Mohseni et al., 2022; Saqib et al., 2024; Tocchetti et al., 2025; Pan et al., 2023; Saeed and Alsharidah, 2024; Javed et al., 2024; Habbal et al., 2024; McCormack et al., 2025; Freiesleben and Grote, 2023; Wei and Liu, 2025; Vadlapati, 2026; Li et al., 2023; Casper et al., 2024; Wang et al., 2026; Palazzolo et al., 2024; Kemmerzell et al., 2025; Zahid et al., 2025; Tsingenopoulos et al., 2025; Chen and Das, 2023; Kaya et al., 2026; Silva and De Franรงa, 2023; Ouyang et al., 2021; Chamola et al., 2023; Bayram and Ahmed, 2025; Petersen et al., 2022; Hรฅkansson et al., 2023; Roy, 2024; Tang et al., 2026; Raji et al., 2020; Koshiyama et al., 2024; Mรถkander et al., 2024; Nolte et al., 2025; Khinvasara et al., 2024; Lee et al., 2023; Yang et al., 2020; Costanza-Chock et al., 2022; Taesiri et al., 2022; Simbeck, 2024; Robertson et al., 2023; Oala et al., 2020; Chen et al., 2025; Agarwal and Nene, 2025) MR-3.2 Robustness Trade-off Assumptions The audit should identify assumptions about how the scoped robustness claim may affect, depend on, or conflict with other system properties, including accuracy, fairness, privacy, explainability, safety, cost, and affected-user outcomes, and should document how these assumptions shape measurement, interpretation, and decision-making. (Li and Goel, 2025; Hardy et al., 2025; Liu et al., 2022; Nolte et al., 2025; Chander et al., 2025; Balendran et al., 2025; Nanda et al., 2021; Yang et al., 2020; Simbeck, 2024; Koshiyama et al., 2024; Javed et al., 2024; Chen et al., 2025; Ge et al., 2024; Bayram and Ahmed, 2025; Tocchetti et al., 2025; Li et al., 2023; Habbal et al., 2024; Vadlapati, 2026; Tsingenopoulos et al., 2025; Saeed and Alsharidah, 2024; Palazzolo et al., 2024; Mohseni et al., 2022; Kemmerzell et al., 2025; Freiesleben and Grote, 2023; Thelisson and Verma, 2024; Hรฅkansson et al., 2023; Dev et al., 2025; Oala et al., 2020; McCormack et al., 2025; Chen and Das, 2023; Costanza-Chock et al., 2022; Pan et al., 2023; Lefcourt et al., 2022; Lee et al., 2023; Madaio et al., 2022; Ouyang et al., 2021; Kaya et al., 2026; Tang et al., 2026; Mรถkander et al., 2024; Kaur et al., 2022) Category 4: Knowledge and Skills MR-4.1 Multidisciplinary Skills Robustness audits should involve combined AI, machine-learning, software, security, data, domain, social, legal, risk, and audit expertise where required by the assessment. (Costanza-Chock et al., 2022; Li and Goel, 2025; Vadlapati, 2026; Ala-Luopa et al., 2024; Mรผller et al., 2025; Kokina et al., 2025; Laux et al., 2024; Robertson et al., 2023; Tocchetti et al., 2025; Madaio et al., 2022; McCormack et al., 2025; Nolte et al., 2025; Mรถkander et al., 2024; Khinvasara et al., 2024; Xiong et al., 2022; Li et al., 2023; Silva and De Franรงa, 2023; Kaur et al., 2022) MR-4.2 Robustness-awareness Educational Programmes and Training Personnel should receive education or training on robustness risks, methods, monitoring, mitigation, communication, and tool use relevant to their roles. (Zahid et al., 2025; Mรผller et al., 2025; Chander et al., 2025; Koshiyama et al., 2024; Ala-Luopa et al., 2024; Habbal et al., 2024; Geleta et al., 2023; Kokina et al., 2025) Functional Dimension Category 5: Data Auditability MR-5.1 Data Coverage and Sufficiency Robustness assessment data should sufficiently cover relevant populations, data types, modalities, subgroups, scenarios, domains, edge cases, and operating conditions. (Javed et al., 2024; Tocchetti et al., 2025; Yang et al., 2020; Agarwal and Nene, 2025; Chen and Das, 2023; Madaio et al., 2022; Yu et al., 2026; Freiesleben and Grote, 2023; Kokina et al., 2025; Raji et al., 2020; Nanda et al., 2021; Oala et al., 2020; Nolte et al., 2025; Hornischer, 2026; Li et al., 2023; Wei and Liu, 2025; Costanza-Chock et al., 2022) MR-5.2 Data Description, Provenance, and Lineage Robustness audits should require dataset-level descriptions and provenance records that identify data sources, acquisition conditions, processing steps, annotation basis, known limitations, and links between the data and the robustness claim. (Javed et al., 2024; Lefcourt et al., 2022; Bayram and Ahmed, 2025; Ge et al., 2024; Casper et al., 2024; Kokina et al., 2025; Raji et al., 2020; Thelisson and Verma, 2024; Oala et al., 2020; Nolte et al., 2025; Khinvasara et al., 2024; Li et al., 2023; Wei and Liu, 2025; Costanza-Chock et al., 2022) MR-5.3 Data Quality Data used for robustness assessment should be checked for noise, missingness, errors, corruption, leakage, imbalance, representativeness, and other data defects. (Ouyang et al., 2021; Blood et al., 2023; Lefcourt et al., 2022; Bayram and Ahmed, 2025; Narula et al., 2025; Oala et al., 2020; McCormack et al., 2025; Martรญnez-Fernรกndez et al., 2022; Yang et al., 2020; Hardy et al., 2025; Silva and De Franรงa, 2023; Mohseni et al., 2022) MR-5.4 Data Subgroup and Bias Source Identification Data subgroups, target attributes, representation gaps, and data-level bias sources should be identified where they may affect the credibility of robustness findings across relevant populations, contexts, or operating conditions. (Roy, 2024; Li et al., 2023; Oala et al., 2020; Kokina et al., 2025; McCormack et al., 2025; Kemmerzell et al., 2025; Khinvasara et al., 2024; Alzubaidi et al., 2023; Mรถkander et al., 2024; Kaur et al., 2022) MR-5.5 Robustness Pre-processing Mitigation The audit should assess whether data-level mitigation methods, such as cleaning, filtering, transformation, augmentation, or recovery of contaminated training data, address identified robustness risks and improve robustness under the assessed conditions. (Blood et al., 2023; Dev et al., 2025; Palazzolo et al., 2024; Tocchetti et al., 2025; Chen and Das, 2023; Freiesleben and Grote, 2023; Javed et al., 2024; Li et al., 2023; Wei and Liu, 2025) Category 6: Model Auditability MR-6.1 Model Robustness Probing Auditors should probe model behaviour with defined robustness stressors, including perturbations, adversarial examples, stress cases, distribution shifts, benchmark scenarios, and complementary test methods where one probing method is insufficient. (Ouyang et al., 2021; Kou et al., 2024; Martรญnez-Fernรกndez et al., 2022; Mรถkander et al., 2024; Oala et al., 2020; Yang et al., 2020; Robertson et al., 2023; Vadlapati, 2026; Lee et al., 2023; Nolte et al., 2025; Hardy et al., 2025; Kaya et al., 2026; Petersen et al., 2022; Agarwal and Nene, 2025; Wang et al., 2026; Tocchetti et al., 2025; Costanza-Chock et al., 2022; Silva and De Franรงa, 2023; Tsingenopoulos et al., 2025; Saqib et al., 2024; Nanda et al., 2021; Dev et al., 2025; Ge et al., 2024; Hancox-Li, 2020) MR-6.2 Model Intended Use The modelโs intended application, users, operating conditions, automation level, limitations, and unsuitable uses should be specified for the audited model. (Madaio et al., 2022; Raji et al., 2020; Thelisson and Verma, 2024; Oala et al., 2020; Nolte et al., 2025) MR-6.3 Model Explainability Model behaviour, decision logic, and robustness failures should be interpretable or otherwise explainable enough to support the audit. (Ouyang et al., 2021; Javed et al., 2024; Methnani et al., 2024; Hancox-Li, 2020; Kaya et al., 2026; Tocchetti et al., 2025; Khinvasara et al., 2024; Habbal et al., 2024; Casper et al., 2024; Tang et al., 2026; Saqib et al., 2024; McCormack et al., 2025; Zahid et al., 2025; Kemmerzell et al., 2025; Liu et al., 2022; Chander et al., 2025; Saeed and Alsharidah, 2024; Chen et al., 2025; Petersen et al., 2022; Mohseni et al., 2022; Roy, 2024; Oala et al., 2020; Taesiri et al., 2022; Ala-Luopa et al., 2024; Li et al., 2023; Ge et al., 2024; Chamola et al., 2023; Kokina et al., 2025; Koshiyama et al., 2024; Pan et al., 2023; Cooper et al., 2022; Li and Goel, 2025; Kaur et al., 2022) MR-6.4 Model Bias and Failure Source Identification Model-level causes of robustness failure, including shortcut learning, overfitting, spurious correlations, and unstable features, should be identified where relevant. (Lee et al., 2023; Petersen et al., 2022; Robertson et al., 2023; Hancox-Li, 2020; Freiesleben and Grote, 2023; Casper et al., 2024; Kemmerzell et al., 2025) MR-6.5 Robustness In-processing Mitigation The audit should assess whether training-time mitigation methods, such as adversarial training, robust optimisation, regularisation, or adapted loss functions, address identified robustness risks and improve robustness under the assessed conditions. (Yang et al., 2020; Agarwal and Nene, 2025; Javed et al., 2024; Wei and Liu, 2025; Tocchetti et al., 2025) Category 7: Output Auditability MR-7.1 Robustness Post-processing Mitigation Robustness audits should identify output-level unstable, unsafe, biased, harmful, or inconsistent behaviour under robustness stress, trace the likely source of the failure, and assess whether post-processing mitigations reduce the identified robustness risk before or during deployment. (Yu et al., 2026; Chen and Das, 2023; Javed et al., 2024; Liu et al., 2022; Tocchetti et al., 2025) Category 8: Workflow Integration MR-8.1 Software Development Process Integration Robustness checks should be integrated into requirements, design, implementation, testing, deployment, maintenance, and review activities. (Silva and De Franรงa, 2023; Mohseni et al., 2022; Chen and Das, 2023; Li et al., 2023; Laux et al., 2024; Martรญnez-Fernรกndez et al., 2022; Alzubaidi et al., 2023; Tang et al., 2026; Blood et al., 2023; Kaur et al., 2022) MR-8.2 MLOps Frameworks Integration Robustness tests, alerts, retraining triggers, monitoring, and reassessment steps should be embedded into ML or MLOps pipelines where applicable. (Koshiyama et al., 2024; Narula et al., 2025; Dev et al., 2025; Wei and Liu, 2025; Bayram and Ahmed, 2025; Ge et al., 2024) MR-8.3 Robustness-aware Integration Compatibility Robustness tools and procedures should be compatible with relevant software, data, model, deployment, monitoring, and incident-management infrastructure. (Koshiyama et al., 2024; Tocchetti et al., 2025; Martรญnez-Fernรกndez et al., 2022; Dev et al., 2025; Kokina et al., 2025; Thelisson and Verma, 2024; Khinvasara et al., 2024) MR-8.4โ External Component and Supply Chain Assurance Robustness audits should identify external models, data, software, services, and infrastructure on which the audited AI-enabled system depends; assess how their failure or change could affect system behaviour; and state how limited access to those dependencies restricts the robustness conclusion. Practitioner feedback (P079, P102); supported by prior studies (Dev et al., 2025; Yu et al., 2026; Martรญnez-Fernรกndez et al., 2022; Blood et al., 2023; Casper et al., 2024; Mรถkander et al., 2024) Category 9: Human-in-the-Loop MR-9.1 Robustness-aware Human Participation Levels Robustness audits should define human participation levels for robustness-relevant decisions, including review, validation, intervention, override, or approval, and should state how available access to inputs, outputs, procedures, transparency information, model internals, or other evidence limits the resulting robustness judgement. (Tocchetti et al., 2025; Kokina et al., 2025; Chamola et al., 2023; McCormack et al., 2025; Kou et al., 2024; Oala et al., 2020; Habbal et al., 2024; Chen and Das, 2023; Casper et al., 2024; Thelisson and Verma, 2024; Kaur et al., 2022) Category 10: Evaluation MR-10.1 Continuous Robustness Assessment Robustness should be formally reassessed repeatedly or when triggered by changes in data, users, attacks, models, or deployment conditions. (Thelisson and Verma, 2024; Blood et al., 2023; Momcilovic et al., 2026; Laux et al., 2024; Chen and Das, 2023; Raji et al., 2020; Li and Goel, 2025; Mรถkander et al., 2024; Simbeck, 2024; Khinvasara et al., 2024; McCormack et al., 2025; Costanza-Chock et al., 2022; Wang et al., 2026; Madaio et al., 2022; Koshiyama et al., 2024) MR-10.2 Robustness-aware Progress Tracking and Prioritisation Robustness evaluations should maintain a prioritised record of completed checks, unresolved robustness concerns, follow-up tests, mitigation needs, and reassessment actions, so that remaining audit work is visible and sequenced. (Costanza-Chock et al., 2022; Mรถkander et al., 2024; Dev et al., 2025; Tocchetti et al., 2025; Chen and Das, 2023) MR-10.3 Cross-Layer Failure Validation Selected failures should be validated across relevant data, model, software, service, infrastructure, and human-interaction layers. (Raji et al., 2020; Blood et al., 2023; Mohseni et al., 2022; Yu et al., 2026; Dev et al., 2025) Category 11: Monitoring MR-11.1 Continuous, Automated, and Real-time Robustness Monitoring Deployed AI-enabled systems should maintain a defined post-deployment robustness monitoring process that tracks robustness indicators, applies predefined thresholds, issues automated or real-time alerts for specified robustness violations, and triggers reassessment when system behaviour, data, deployment conditions, or threat conditions change. (Gupta et al., 2025; Blood et al., 2023; Nolte et al., 2025; Habbal et al., 2024; Chen and Das, 2023; Bayram and Ahmed, 2025; Li and Goel, 2025; Mohseni et al., 2022; Thelisson and Verma, 2024; Robertson et al., 2023; Koshiyama et al., 2024; Khinvasara et al., 2024; Narula et al., 2025; Vadlapati, 2026; Momcilovic et al., 2026; McCormack et al., 2025; Martรญnez-Fernรกndez et al., 2022; Chen et al., 2025; Roy, 2024; Schmelczer and Visser, 2023; Raji et al., 2020; Ala-Luopa et al., 2024; Methnani et al., 2024; Chamola et al., 2023; Hรฅkansson et al., 2023; Kaya et al., 2026; Ojewale et al., 2025) MR-11.2 Robustness-aware Logging Inputs, outputs, model behaviour, errors, alerts, updates, and context should be logged where needed to diagnose robustness failures. (Thelisson and Verma, 2024; Schmelczer and Visser, 2023; Methnani et al., 2024; Raji et al., 2020; Li and Goel, 2025) Category 12: Communication MR-12.1 Robustness Dashboards and Visualisation Robustness dashboards and visualisations should present audit-relevant metrics, model behaviour, monitoring trends, subgroup outcomes, failures, alerts, and supporting evidence in a form that helps intended users inspect and interpret robustness evidence. (Xiong et al., 2022; Robertson et al., 2023; Tocchetti et al., 2025; Methnani et al., 2024; Ala-Luopa et al., 2024; Li and Goel, 2025; Mรผller et al., 2025; Nolte et al., 2025; Agarwal and Nene, 2025; Kokina et al., 2025; Raji et al., 2020; Oala et al., 2020) MR-12.2 Robustness Documentation and Risk Report Robustness audits should document and report the audit objective, evidence base, evaluated system and datasets, robustness findings, limitations, mitigations, residual risks, reproducibility conditions, traceability information, and audience-appropriate explanations needed to interpret the audit conclusion. (Koshiyama et al., 2024; Mรถkander et al., 2024; Laux et al., 2024; Costanza-Chock et al., 2022; Ojewale et al., 2025; McCormack et al., 2025; Raji et al., 2020; Momcilovic et al., 2026; Li et al., 2023; Chander et al., 2025; Li and Goel, 2025; Vadlapati, 2026; Chen and Das, 2023; Petersen et al., 2022; Oala et al., 2020; Kemmerzell et al., 2025; Palazzolo et al., 2024; Tsingenopoulos et al., 2025; Ouyang et al., 2021; Alzubaidi et al., 2023; Bayram and Ahmed, 2025; Yu et al., 2026; Cooper et al., 2022; Simbeck, 2024; Gupta et al., 2025; Blood et al., 2023; Zahid et al., 2025; Kaya et al., 2026; Methnani et al., 2024; Hancox-Li, 2020; Freiesleben and Grote, 2023; Taesiri et al., 2022; Casper et al., 2024; Dev et al., 2025; Ala-Luopa et al., 2024) Quality Dimension Category 13: Efficiency MR-13.1 Resource Efficiency and Scalability A robustness auditing process should be cost-effective, time-efficient, and scalable, so that organisations with different resource capacities can assess, monitor, mitigate, and report robustness risks within practical audit-cycle constraints and across the models, datasets, perturbations, users, and deployment settings covered by the audit scope. (Kou et al., 2024; Kokina et al., 2025; Tocchetti et al., 2025; Madaio et al., 2022; Nanda et al., 2021; Khinvasara et al., 2024; Javed et al., 2024; Wei and Liu, 2025; Kaya et al., 2026; Saqib et al., 2024; Thelisson and Verma, 2024; Martรญnez-Fernรกndez et al., 2022; Gupta et al., 2025; McCormack et al., 2025; Chen et al., 2025; Mohseni et al., 2022; Wang et al., 2026; Mรถkander et al., 2024; Hornischer, 2026; Dev et al., 2025) Category 14: Adaptability MR-14.1 Domain-specific Adaptability Robustness definitions, perturbations, thresholds, tests, and interpretations should be tailored to risks and evidence standards specific to the deployment domain. (Ouyang et al., 2021; Mohseni et al., 2022; Tocchetti et al., 2025; Lefcourt et al., 2022; Gupta et al., 2025; Roy, 2024; Oala et al., 2020; Mรผller et al., 2025; Javed et al., 2024; Mรถkander et al., 2024; Freiesleben and Grote, 2023; Khinvasara et al., 2024; Balendran et al., 2025; Hardy et al., 2025; Casper et al., 2024; Wang et al., 2026; Chen et al., 2025; Petersen et al., 2022; Kaur et al., 2022) MR-14.2 Task and Model Versatility Robustness assessment should support the task types, model classes, modalities, datasets, and deployment settings covered by the audit scope. (Balendran et al., 2025; Kemmerzell et al., 2025; Agarwal and Nene, 2025; Thelisson and Verma, 2024; Javed et al., 2024; Li et al., 2023) Category 15: Accessibility MR-15.1 Accessible and User-Friendly Robustness Tools, Visualisations, and Documents Robustness tools, visualisations, documents, and outputs should be understandable, usable, and available at an appropriate level of technical detail for their intended technical and non-technical audiences. (Hancox-Li, 2020; Methnani et al., 2024; Kemmerzell et al., 2025; Chander et al., 2025; Li and Goel, 2025; Ojewale et al., 2025; Agarwal and Nene, 2025; McCormack et al., 2025; Chen and Das, 2023; Madaio et al., 2022; Yu et al., 2026; Freiesleben and Grote, 2023; Kokina et al., 2025; Nolte et al., 2025; Javed et al., 2024; Li et al., 2023; Costanza-Chock et al., 2022) Category 16: Configurability MR-16.1 Configurable Robustness Assessment Tools Robustness assessment tools should allow configuration of perturbations, thresholds, metrics, dashboards, reports, tools, and deployment-specific scenarios. (Agarwal and Nene, 2025; Vadlapati, 2026; Palazzolo et al., 2024; Yang et al., 2020; Freiesleben and Grote, 2023; Kokina et al., 2025; Javed et al., 2024; Liu et al., 2022) Note. โMR-8.4 was not identified during the initial synthesis of the 74 retained studies; it emerged from practitioner feedback and was subsequently supported by evidence found on re-examining those studies. 4.4.6. Qualitative Coding of Open-Text Comments The 586 substantive open-text comments were organised as separate records and linked to an anonymous respondent identifier (P001-P128), source category, and relevant meta-requirement. Each comment was manually coded for its theme, function, and related meta-requirement, with supporting quotations and coder notes retained for traceability. Comment functions captured confirmations, clarifications, implementation concerns, challenges, and proposed additions, while coding decisions distinguished retention, refinement, addition, merging or splitting, and no change. Examples of anonymised comments and their Retain, Refine, Add, and Not usable coding decisions are provided in the replication package (see the Data Availability statement). To reduce dependence on a single interpretation, one researcher coded the comments, while a second researcher independently reviewed the coding and proposed synthesis decisions, following multi-researcher coding practices used in related practitioner studies (van der Linden et al., 2019; von Zahn et al., 2025). We then reconciled the qualitative feedback with the quantitative results in Phase 4. For each meta-requirement, the quantitative support status was considered together with the related practitioner comments. Proposed changes were compared against the complete set of meta-requirements to determine whether they supported retention, required refinement, or identified a concern not already represented in the existing set. This integration informed the final practitioner-informed set of meta-requirements. 4.5. Phase 4: Meta-Requirement Synthesis We integrated the quantitative ratings and qualitative comments to produce the final practitioner-informed set of meta-requirements. For each literature-derived meta-requirement, we considered the quantitative support status alongside the associated qualitative feedback. Practitioner suggestions were then compared with the existing meta-requirements and relevant literature to determine whether they reinforced, refined, challenged, or extended the literature-derived set of meta-requirements. Qualitative synthesis decisions were reconciled as Retain, Refine, Add, No change, or Not usable. We retained suggested additions only when the underlying concern was not sufficiently represented by an existing meta-requirement and was supported by relevant evidence; otherwise, we incorporated them as refinements or did not adopt them. Merge and split suggestions were similarly reviewed against neighbouring meta-requirements and supporting evidence to clarify requirement boundaries where appropriate. The synthesis retained the 43 literature-derived meta-requirements, refined their wording and scope where supported by practitioner feedback and the evidence base, and added one practitioner-informed requirement. The final set therefore comprises 44 meta-requirements across the original 16 categories. MR-8.4, External Component and Supply Chain Assurance, was added to Category 8, Workflow Integration, because two overall survey comments (P079, P102) identified a concern not covered by the existing meta-requirements. Comparison against the complete set showed that MR-8.1-MR-8.3 were the closest matches, but they did not address limited auditor access to external dependencies, the use of third-party validation evidence, or how these limitations affect the conclusions that can be drawn. MR-8.4 was therefore retained as a separate meta-requirement and placed in Workflow Integration because these external dependencies form part of the AI-enabled systemโs operational workflow. As MR-8.4 was introduced after the survey, it was not rated in the original Likert-scale validation and therefore does not share the quantitative validation status of the other 43 meta-requirements. 5. Meta-Requirements for AI Robustness Auditing This section presents the final practitioner-informed meta-requirements for AI robustness auditing (RQ1). As shown in Figure 5, the final set consists of 44 meta-requirements across 16 categories and three higher-level dimensions: Human (16 meta-requirements), Functional (23), and Quality (5). We derived the initial set of 43 meta-requirements from the 74 retained studies and refined it with practitioner feedback, including the addition of MR-8.4, External Component and Supply-Chain Assurance. The replication package (see the Data Availability statement) reports the original 43 literature-derived meta-requirements used in the survey. The human dimension covers the organisational, stakeholder, conceptual, and competence conditions needed to make robustness auditing governable and interpretable. The quality dimension covers the qualities that make robustness auditing feasible, context-sensitive, understandable, usable, and configurable across different systems and deployment settings. Finally, the functional dimension covers the audit activities and evidence needed across the AI-enabled system lifecycle, including data, model, output, workflow, human review, evaluation, monitoring, and communication. The rest of this section presents each meta-requirement and category. Figure 5. Final practitioner-informed meta-requirements for AI robustness auditing, comprising 44 meta-requirements across 16 categories and three dimensions. 5.1. Category 1: Standards and Governance This category defines the measurement conventions, terminology, tools, guidance, governance responsibilities, access conditions, and audit procedures needed to make AI robustness audit evidence comparable, interpretable, and governable. 5.1.1. MR-1.1: Standardised Metrics Standardisation means using the same measures and the same rules for deciding whether a result is acceptable when similar robustness claims are assessed. Audit results are difficult to compare when similar claims are tested using different measures, test conditions, or acceptance rules (Nolte et al., 2025; Blood et al., 2023; Tocchetti et al., 2025). Prior studies measure robustness through performance under corrupted data, adversarial attacks, and changes in the data distribution (Oala et al., 2020; Yang et al., 2020; Nanda et al., 2021; Chen and Das, 2023; Khinvasara et al., 2024; Casper et al., 2024; Thelisson and Verma, 2024; Raji et al., 2020; Costanza-Chock et al., 2022). Each audit should therefore state what is measured, how it is measured, the conditions under which it is tested, and what result is considered acceptable (Madaio et al., 2022; Yu et al., 2026; Nolte et al., 2025; Blood et al., 2023; Agarwal and Nene, 2025). 5.1.2. MR-1.2: Standardised Lexicon and Terminology A common set of definitions helps everyone involved in the audit use key robustness terms consistently. This is necessary because terms such as robustness, reliability, failure, and auditability can have different meanings in engineering, legal, and organisational work (Tocchetti et al., 2025; Mohseni et al., 2022; Vadlapati, 2026; Freiesleben and Grote, 2023; Hornischer, 2026; Yu et al., 2026; Wei and Liu, 2025). If these terms are not defined clearly, auditors, developers, regulators, and system users may interpret the same robustness claim differently (Raji et al., 2020; Thelisson and Verma, 2024; Costanza-Chock et al., 2022). The audit should therefore define the terms needed to understand the claim, the assessment, and the conclusion, and should distinguish robustness from related properties such as safety, fairness, privacy, explainability, and accountability (Chen and Das, 2023; Yang et al., 2020; Nolte et al., 2025; Mohseni et al., 2022; Vadlapati, 2026; Ge et al., 2024; Liu et al., 2022; Li et al., 2023; Khinvasara et al., 2024; Nanda et al., 2021). 5.1.3. MR-1.3: Standardised Tools This MR emphasizes using software tools or test instruments in a consistent way to assess the same type of robustness problem. Different tools may be used for the same type of assessment, provided that they conform to a common standard that supports consistent and comparable results (Raji et al., 2020; Ojewale et al., 2025; Momcilovic et al., 2026; Oala et al., 2020; Dev et al., 2025; Yu et al., 2026; Agarwal and Nene, 2025). Tools differ in what they test, the conditions they can reproduce, and how they are configured, so a tool should be selected to match the robustness claim, the system or component being assessed, and the type of input data (Casper et al., 2024; Ge et al., 2024; Yang et al., 2020; Tocchetti et al., 2025; Javed et al., 2024; Khinvasara et al., 2024). The audit should state which tool and version were used, the main settings applied, and any known limitations that affect the result (Casper et al., 2024; Kokina et al., 2025; Thelisson and Verma, 2024). 5.1.4. MR-1.4: Actionable Guidance This MR explains how a robustness requirement is turned into specific tasks that an auditor can carry out for the system being assessed (Kokina et al., 2025; Khinvasara et al., 2024). The guidance should translate applicable technical, legal, and organisational requirements, including requirements related to properties such as safety, security, or reliability where they are relevant, together with conditions that could cause the system to fail or be deliberately manipulated, into concrete audit actions (Cooper et al., 2022; Habbal et al., 2024; Thelisson and Verma, 2024; Xiong et al., 2022; Narula et al., 2025). It should state which checks or tests to perform, what evidence to examine, and when further review is required (Blood et al., 2023; McCormack et al., 2025). 5.1.5. MR-1.5: Robustness-aware Integration Guidance The guidance on when and how robustness checks should be included as an AI-enabled system is developed, deployed, used, changed, and reviewed, rather than being performed only as a separate test (Chen and Das, 2023; Raji et al., 2020; Thelisson and Verma, 2024; Li et al., 2023; Yu et al., 2026; Tocchetti et al., 2025; Madaio et al., 2022). The guidance should state at which stages the checks are required and when they must be repeated after changes to the model, data, operating conditions, or intended use of the AI-enabled system (Oala et al., 2020; Nolte et al., 2025; Khinvasara et al., 2024; Kokina et al., 2025; Wei and Liu, 2025; Hornischer, 2026; Javed et al., 2024; Liu et al., 2022). MR-1.5 concerns when and where robustness checks should be required across the AI lifecycle, whereas MR-8.1 concerns whether those checks are integrated into development workflows. 5.1.6. MR-1.6: Governance Protocols These protocols define who is responsible for investigating robustness problems, deciding what action is needed, and deciding whether the AI-enabled system may continue to be used when a problem cannot be fully resolved (Li and Goel, 2025; Cooper et al., 2022; McCormack et al., 2025). Prior work assigns these responsibilities to organisations that develop, operate, or oversee AI-enabled systems (Habbal et al., 2024; Liu et al., 2022; Raji et al., 2020; Mรถkander et al., 2024). The protocols should state who investigates the problem, who decides on corrective action, who approves continued use of the system, and who must be informed when the problem could affect important properties of the AI-enabled system or its outcomes, such as safety, security, fairness, reliability, or privacy, or could result in non-compliance with applicable law (Laux et al., 2024; Nolte et al., 2025; Thelisson and Verma, 2024; Costanza-Chock et al., 2022; Mรถkander et al., 2024; Chamola et al., 2023; Liu et al., 2022). 5.1.7. MR-1.7: Audit Access Policy This policy defines what information and system components the auditor is allowed to inspect and what remains unavailable. Audit conclusions may depend on access to data, source code, model information, system documentation, logs, test results, and deployment information (Casper et al., 2024; Ojewale et al., 2025; Koshiyama et al., 2024; Raji et al., 2020; Li and Goel, 2025; Chander et al., 2025; Mรผller et al., 2025). The policy should state what can and cannot be inspected, how long that access is available, and how any restrictions limit the conclusions that can be drawn (Koshiyama et al., 2024; Simbeck, 2024; Thelisson and Verma, 2024; Costanza-Chock et al., 2022; Mรผller et al., 2025; Ojewale et al., 2025). When privacy, security, legal, or confidentiality requirements prevent access to some information, the policy should identify the restriction and any permitted alternative source of information (Simbeck, 2024; Thelisson and Verma, 2024; Chander et al., 2025). 5.1.8. MR-1.8: Auditor Protocols These protocols define how a robustness audit is carried out, including who performs it, the steps followed, the methods used, and how the results from those methods are used to reach the final conclusion. Audits may be conducted by internal teams or independent external assessors and may include technical testing, document review, or examination of how the organisation develops and operates the system (Casper et al., 2024; Thelisson and Verma, 2024; Ge et al., 2024; Kokina et al., 2025; Chen and Das, 2023). The protocols should state the audit objective, the order of the audit activities, the methods and criteria used, and what each method contributes to the final conclusion (Ojewale et al., 2025; Oala et al., 2020; Costanza-Chock et al., 2022; Casper et al., 2024). When more than one method is used, the protocol should explain how their results are combined to support that conclusion (Madaio et al., 2022; Nanda et al., 2021; Liu et al., 2022). MR-1.8 concerns how the audit is conducted, whereas MR-2.3 concerns the role, independence, and access of an independent auditor. 5.2. Category 2: Engagement This category establishes which stakeholder, user, expert, and independent-auditor perspectives must inform the framing, assessment, and use of AI robustness audit evidence. 5.2.1. MR-2.1: Diverse Stakeholders The stakeholders include individuals and organisations that develop, operate, use, oversee, or experience consequences from decisions made with an AI-enabled system or from its failures. An audit that relies only on technical developers may miss robustness problems, harms, or practical constraints recognised by other stakeholders (Chander et al., 2025; Balendran et al., 2025; Costanza-Chock et al., 2022; Robertson et al., 2023; Li and Goel, 2025; Raji et al., 2020; McCormack et al., 2025). The audit should therefore include perspectives from the stakeholder types connected to the robustness claim, especially where the same failure may have different consequences for different people (Nanda et al., 2021; Nolte et al., 2025; Liu et al., 2022; Hancox-Li, 2020; Tocchetti et al., 2025; Mรถkander et al., 2024; Zahid et al., 2025). 5.2.2. MR-2.2: Stakeholder Involvement Prioritisation This MR determines which stakeholders need more involvement in the audit, when their input is needed, and how much participation is required. Priority should depend on how directly they may experience the consequences of a system failure, the knowledge they can contribute, and the likelihood and seriousness of the failure being assessed (Costanza-Chock et al., 2022; Nolte et al., 2025; Madaio et al., 2022; Raji et al., 2020; Thelisson and Verma, 2024; Oala et al., 2020; Li et al., 2023). The audit should therefore state which stakeholders need one-time, repeated, or focused involvement and why, while considering available resources and any restrictions on access to the system or information (Raji et al., 2020; Thelisson and Verma, 2024; Oala et al., 2020; Li et al., 2023). 5.2.3. MR-2.3: Independent Auditor Auditor independence means that the person or organisation conducting the audit can reach a conclusion without pressure from parties that develop, operate, fund, or manage the system being assessed (Ojewale et al., 2025; McCormack et al., 2025). External auditors may be more separate from the organisation, while internal auditors may have more detailed knowledge of the system and its development (Casper et al., 2024; Liu et al., 2022; Raji et al., 2020). The audit should therefore state who selected and funds the auditor, whether the auditor previously helped develop or operate the system, and whether any organisational relationship could influence the audit conclusion (Costanza-Chock et al., 2022; Laux et al., 2024; Thelisson and Verma, 2024; Mรถkander et al., 2024; McCormack et al., 2025; Li and Goel, 2025). Any relationship that could influence the auditorโs judgement should be disclosed and managed (McCormack et al., 2025; Li and Goel, 2025). 5.2.4. MR-2.4: Early-stage Involvement This MR implies bringing the needed expertise early in planning the robustness audit, such as technical, domain, legal or compliance, and organisational expertise, before key audit decisions are made. These decisions may include which robustness problems will be examined, which parts of the AI-enabled system will be included, which tests will be performed, what evidence will be collected, and what results will be considered acceptable. This is important because robustness problems can be harder to address when the needed expertise is introduced only after the audit has already been planned or testing has begun (Raji et al., 2020). Experts should therefore contribute while the audit scope, test plan, evidence needs, and acceptance criteria are being decided (Li and Goel, 2025; McCormack et al., 2025; Madaio et al., 2022; Hornischer, 2026; Cooper et al., 2022). Their early input should help ensure that the audit examines the robustness problems and system conditions that matter for the case being audited (McCormack et al., 2025; Cooper et al., 2022; Madaio et al., 2022; Li and Goel, 2025). 5.3. Category 3: Robustness Definition This category defines the robustness claim an audit will assess, the assumptions that bound that claim, and the trade-offs that affect how robustness evidence is interpreted. 5.3.1. MR-3.1: Scoping Robustness Concerns This MR considers which robustness problems the audit will assess, the conditions under which they will be tested, and which conditions are excluded. The audit should state whether it covers adversarial robustness, non-adversarial robustness, or both (see Table 1 for definitions) (Freiesleben and Grote, 2023; Tocchetti et al., 2025; Bayram and Ahmed, 2025; Chen and Das, 2023; Hornischer, 2026; Li et al., 2023; Liu et al., 2022; Wang et al., 2026; Zahid et al., 2025). It should then identify the specific conditions to be tested, such as corrupted or missing data, distribution changes, poisoning, or evasion attacks (Blood et al., 2023; Narula et al., 2025; Wei and Liu, 2025; Balendran et al., 2025; Vadlapati, 2026; Saqib et al., 2024; Javed et al., 2024; Koshiyama et al., 2024; Saeed and Alsharidah, 2024; Costanza-Chock et al., 2022; Chander et al., 2025; Mรถkander et al., 2024; Pan et al., 2023; Tang et al., 2026). 5.3.2. MR-3.2: Robustness Trade-off Assumptions A robustness trade-off occurs when improving robustness changes another important property of the system, such as accuracy, computational cost, explainability, fairness, safety, and privacy. Ignoring such changes can make a robustness improvement appear beneficial even when another property becomes worse (Koshiyama et al., 2024; Liu et al., 2022; Ge et al., 2024; Nanda et al., 2021). The audit should therefore identify which properties may change, compare them before and after the method used to improve robustness, and state how any observed gain or loss affects the robustness conclusion (Yang et al., 2020; Simbeck, 2024; Javed et al., 2024; Chen et al., 2025; Bayram and Ahmed, 2025; Tocchetti et al., 2025; Li et al., 2023; Vadlapati, 2026; Palazzolo et al., 2024). 5.4. Category 4: Knowledge and Skills This category defines the competence and role-specific training needed to conduct, interpret, and maintain AI robustness audits. 5.4.1. MR-4.1: Multidisciplinary Skills These mean having expertise from more than one field when a robustness audit requires different kinds of knowledge. This is important because assessing robustness may require knowledge of AI models, the software system in which they operate, the domain in which the system is used, and relevant security, legal, or organisational requirements (Li and Goel, 2025; Mรถkander et al., 2024; Xiong et al., 2022). The audit should therefore involve people with the skills needed for the system and robustness problems being assessed, which may include machine learning, software engineering, cybersecurity, statistics, law or policy, auditing, and domain expertise (Khinvasara et al., 2024; Nolte et al., 2025; Laux et al., 2024; Costanza-Chock et al., 2022). If required expertise is missing, the audit should state how that limitation affects the conclusions that can be drawn (Ala-Luopa et al., 2024; Mรผller et al., 2025; Vadlapati, 2026; Kokina et al., 2025; McCormack et al., 2025; Silva and De Franรงa, 2023). 5.4.2. MR-4.2: Robustness-awareness Educational Programmes and Training Training should give each person the knowledge needed for their role in robustness auditing. Auditors, developers, system administrators, users, and domain experts may need different training because they perform different tasks and make different decisions (Zahid et al., 2025; Chander et al., 2025). The training should cover the robustness problems, methods, tools, and responsibilities relevant to each role, including adversarial testing or secure system operation where needed (Mรผller et al., 2025; Kokina et al., 2025; Geleta et al., 2023; Ala-Luopa et al., 2024; Koshiyama et al., 2024; Habbal et al., 2024). Because methods, tools, attacks, and legal requirements change over time, the training should be updated periodically and may include practical exercises, simulated failures, or audit examples (Mรผller et al., 2025; Zahid et al., 2025). 5.5. Category 5: Data Auditability This category establishes whether data used in robustness audits are sufficiently covered, documented, reliable, bias-aware, and prepared for data-level mitigation before interpreting robustness claims. 5.5.1. MR-5.1: Data Coverage and Sufficiency Data coverage means that the data used in the audit include the people, cases, and operating conditions covered by the robustness claim. Beyond including these cases and conditions, the audit also needs enough data to assess them reliably; this is what we mean by data sufficiency. Robustness evidence can be misleading when important groups, rare cases, distribution changes, or failure conditions are missing, or when too little data are available to assess them reliably (Javed et al., 2024; Madaio et al., 2022; Nanda et al., 2021). The audit should therefore check whether the available data represent the intended users, important variations in the data, rare but important cases, and the conditions under which the AI-enabled system is expected to operate, and whether enough data are available to evaluate these reliably (Yang et al., 2020; Tocchetti et al., 2025; Freiesleben and Grote, 2023; Chen and Das, 2023; Nolte et al., 2025; Hornischer, 2026; Yu et al., 2026; Oala et al., 2020; Costanza-Chock et al., 2022; Agarwal and Nene, 2025; Li et al., 2023; Wei and Liu, 2025). If important cases or conditions are missing, or are represented by too little data, the audit should state how these limitations affect the robustness conclusion (Raji et al., 2020; Kokina et al., 2025). 5.5.2. MR-5.2: Data Description, Provenance, and Lineage This MR describes what the data contain, where they came from, and how they were collected or changed before being used in the audit. Without this information, auditors may not know whether the history of the data influenced the robustness result (Raji et al., 2020; Thelisson and Verma, 2024). The audit should identify the data source, collection conditions, labelling process, preprocessing steps, dataset versions, and any synthetic data used (Oala et al., 2020; Bayram and Ahmed, 2025; Lefcourt et al., 2022). It should also trace major changes made to the data used in the audit, such as filtering, merging, relabelling, or transformation, so that their origin and processing history can be checked (Oala et al., 2020; Costanza-Chock et al., 2022; Ge et al., 2024; Kokina et al., 2025; Raji et al., 2020; Thelisson and Verma, 2024). 5.5.3. MR-5.3: Data Quality This concerns whether the data used in the audit are accurate, consistent, and suitable for producing reliable robustness results. Errors such as incorrect labels, duplicated records, corrupted inputs, missing values, data leakage, or contamination can distort the assessment (Bayram and Ahmed, 2025; Hardy et al., 2025; Silva and De Franรงa, 2023). The audit should therefore check for these problems and explain how any detected defects affect the robustness conclusion (Martรญnez-Fernรกndez et al., 2022; Blood et al., 2023; Mohseni et al., 2022; McCormack et al., 2025; Yang et al., 2020; Hardy et al., 2025; Narula et al., 2025; Ouyang et al., 2021). Where reference labels are used, their accuracy and completeness should also be checked. 5.5.4. MR-5.4: Data Subgroup and Bias Source Identification This MR examines whether robustness differs across defined groups in the data and whether those differences can be linked to how the data were collected, labelled, or processed. Overall results can hide weaker performance for particular demographic, geographic, device, usage, or domain groups (Roy, 2024; Oala et al., 2020; Li et al., 2023; McCormack et al., 2025; Mรถkander et al., 2024; Kemmerzell et al., 2025; Alzubaidi et al., 2023). The audit should therefore compare robustness across the groups relevant to the claim and investigate whether observed differences are associated with sampling imbalance, annotation errors, preprocessing choices, missing group information, or changes in the data distribution (Kemmerzell et al., 2025; Kokina et al., 2025; Khinvasara et al., 2024). These data-related causes should be considered when interpreting differences in robustness across groups (Roy, 2024; Alzubaidi et al., 2023). 5.5.5. MR-5.5: Robustness Pre-processing Mitigation Pre-processing robustness mitigation refers to data-level changes made before model training or evaluation to address an identified robustness problem. Examples include cleaning corrupted data, filtering harmful inputs, transforming features, balancing data, or generating additional test or training examples (Dev et al., 2025; Palazzolo et al., 2024; Wei and Liu, 2025; Chen and Das, 2023; Tocchetti et al., 2025; Freiesleben and Grote, 2023). The audit should assess whether the chosen data changes resolve the identified problem and whether they actually improve robustness under the conditions being tested (Dev et al., 2025; Javed et al., 2024; Tocchetti et al., 2025). It should also check whether those changes hide existing weaknesses or introduce new ones (Wei and Liu, 2025). 5.6. Category 6: Model Auditability This category establishes how model behaviour, intended use, explanations, failure sources, and training-time mitigation should be made auditable for robustness claims. 5.6.1. MR-6.1: Model Robustness Probing This MR means deliberately testing the model under the robustness conditions selected in MR-3.1 to see how its behaviour changes. This is needed because a model may appear to perform well overall while still failing under specific conditions (Vadlapati, 2026; Robertson et al., 2023; Martรญnez-Fernรกndez et al., 2022; Hardy et al., 2025; Kaya et al., 2026; Petersen et al., 2022). The audit should use tests that match the robustness problem being assessed, such as corrupted inputs, distribution changes, specially constructed test cases, or adversarial examples (Oala et al., 2020; Yang et al., 2020; Lee et al., 2023; Nolte et al., 2025; Wang et al., 2026; Tocchetti et al., 2025; Agarwal and Nene, 2025; Ge et al., 2024; Dev et al., 2025; Tsingenopoulos et al., 2025). For each test, the audit should state the test conditions, the measure used, and how the result is interpreted (Ouyang et al., 2021; Mรถkander et al., 2024). 5.6.2. MR-6.2: Model Intended Use This defines what the model is designed to do, who is expected to use it, and the operating conditions under which its robustness is assessed. Robustness results can be misleading if they are applied to uses or conditions that were not examined (Madaio et al., 2022; Oala et al., 2020; Raji et al., 2020; Thelisson and Verma, 2024). The audit should therefore state the modelโs intended task, expected users, role in decision-making, operating conditions, known limitations, and uses that fall outside the assessment (Nolte et al., 2025; Madaio et al., 2022; Raji et al., 2020). This keeps the robustness conclusion limited to the uses and conditions that were actually evaluated (Oala et al., 2020; Thelisson and Verma, 2024). 5.6.3. MR-6.3: Model Explainability This MR encompasses using methods that help auditors understand why a model produced a particular result or behaved differently under a robustness test. This is useful when the modelโs behaviour cannot be understood from performance results alone (Khinvasara et al., 2024; McCormack et al., 2025; Mohseni et al., 2022). Such methods may show which inputs or model features influenced the result and may help reveal errors, bias, or unusual behaviour (Javed et al., 2024; Methnani et al., 2024; Tang et al., 2026; Ouyang et al., 2021; Tocchetti et al., 2025; Liu et al., 2022; Chander et al., 2025; Zahid et al., 2025; Habbal et al., 2024). The audit should use methods that show why the model produced a result only when they help answer the robustness question being assessed, and should state what those methods can and cannot show (Kemmerzell et al., 2025; Li and Goel, 2025; Kaya et al., 2026; Li et al., 2023; Oala et al., 2020; Pan et al., 2023; Casper et al., 2024; Chamola et al., 2023; Cooper et al., 2022; Ala-Luopa et al., 2024; Kokina et al., 2025; Koshiyama et al., 2024). Because such explanations can be unstable or misleading, they should not be treated as proof of the true cause of a model failure (Hancox-Li, 2020; Kaya et al., 2026; Saqib et al., 2024; Ge et al., 2024; Taesiri et al., 2022; Casper et al., 2024). 5.6.4. MR-6.4: Model Bias and Failure Source Identification This examines which parts of the modelโs learned behaviour may be responsible for an observed robustness weakness. A model may perform poorly because it relies on misleading patterns, unstable features, memorised examples, or rules that do not hold outside the data used during development (Freiesleben and Grote, 2023; Robertson et al., 2023; Petersen et al., 2022; Casper et al., 2024; Hancox-Li, 2020). The audit should therefore investigate whether the weakness can be linked to the features the model relies on, the patterns it has learned, or differences in behaviour across tested cases, rather than reporting only the performance loss (Lee et al., 2023; Kemmerzell et al., 2025; Robertson et al., 2023; Hancox-Li, 2020). Any claimed cause should be supported by evidence from the tests performed (Kemmerzell et al., 2025). 5.6.5. MR-6.5: Robustness In-processing Mitigation This mitigation method changes how the model is trained so that it performs more reliably under the failure condition being assessed. Such changes may include training with adversarial examples, adjusting the training objective, or using methods that reduce reliance on patterns that fail when the input or operating conditions change (Agarwal and Nene, 2025; Javed et al., 2024; Wei and Liu, 2025; Tocchetti et al., 2025). The audit should assess whether the chosen training change addresses the robustness problem being tested and whether the model performs more reliably after training (Agarwal and Nene, 2025; Javed et al., 2024). Any improvement should be verified through testing rather than assumed from the use of the training method alone (Yang et al., 2020). 5.7. Category 7: Output Auditability This category examines how robustness audits connect observed outputs and inference behaviour to the failure sources that produced them. 5.7.1. MR-7.1: Robustness Post-processing Mitigation Post-processing robustness mitigation refers to measures applied after model inference to reduce the effect of an identified robustness problem. Examples include rejecting uncertain outputs, checking outputs against additional rules, routing selected cases for human review, or using a safer alternative when an output cannot be trusted (Yu et al., 2026; Chen and Das, 2023). The audit should assess whether the selected mitigation reduces the identified robustness problem under the conditions being tested and should compare system behaviour before and after the mitigation is applied (Yu et al., 2026; Chen and Das, 2023; Liu et al., 2022; Tocchetti et al., 2025). If the same failures remain, the post-processing mitigation should not be treated as sufficient evidence that the robustness problem has been resolved (Javed et al., 2024; Liu et al., 2022; Tocchetti et al., 2025). 5.8. Category 8: Workflow Integration This category establishes how robustness auditing requirements are embedded into development workflows, machine-learning operations, and existing organisational systems rather than treated as isolated assessment tasks (Silva and De Franรงa, 2023; Bayram and Ahmed, 2025; Tocchetti et al., 2025). 5.8.1. MR-8.1: Software Development Process Integration This MR means building robustness checks into the activities used to develop and change the AI-enabled system, rather than performing them only at the end of development. Robustness problems can be introduced when requirements, data, models, code, or system components are changed (Martรญnez-Fernรกndez et al., 2022; Silva and De Franรงa, 2023; Mohseni et al., 2022; Alzubaidi et al., 2023; Laux et al., 2024). The audit should therefore check whether robustness testing is included at the development stages where such changes occur and whether testing is repeated after important changes to the model, data, code, or supporting infrastructure (Chen and Das, 2023; Tang et al., 2026; Martรญnez-Fernรกndez et al., 2022; Silva and De Franรงa, 2023; Li et al., 2023; Blood et al., 2023). 5.8.2. MR-8.2: MLOps Frameworks Integration This MR empathises including robustness checks in the processes used to release, deploy, update, and operate ML models. Robustness problems can appear when models or data change after development, so these processes should not rely only on tests performed before release (Bayram and Ahmed, 2025). The audit should check whether the operational workflow includes robustness testing before release, checks on production data, verification that protective measures still work, and clear actions when required conditions are not met (Dev et al., 2025; Wei and Liu, 2025; Narula et al., 2025). Those actions may include stopping a release, pausing operation, or returning to an earlier model version (Ge et al., 2024). 5.8.3. MR-8.3: Robustness-aware Integration Compatibility Integration compatibility means that the tools used for robustness auditing can work with the system being assessed and its existing technical environment. Audit tools are less useful when they cannot connect to the system, read the required data, or run under its operating constraints (Thelisson and Verma, 2024; Tocchetti et al., 2025; Martรญnez-Fernรกndez et al., 2022; Kokina et al., 2025). The audit should therefore check whether the selected tools work with the systemโs interfaces, data formats, software environment, and available computing resources (Martรญnez-Fernรกndez et al., 2022; Dev et al., 2025; Koshiyama et al., 2024; Thelisson and Verma, 2024). Using audit tools should not require converting data into unsupported formats, disrupt communication with the system, cause unacceptable delays, or prevent required robustness tests from being completed (Khinvasara et al., 2024). 5.8.4. MR-8.4: External Component and Supply Chain Assurance This MR concerns models, data, software, services, or infrastructure supplied by another organisation on which the audited AI-enabled system depends. Practitioner feedback identified this as a missing MR in the initial literature-based MRs: P079 highlighted "third-party and supply chain risks" and restricted auditor access to external components, while P102 called for third-party validation and guidance for auditing foundation models and third-party APIs. Prior studies likewise show that third-party components, APIs, software dependencies, data pipelines, and infrastructure can introduce failures or changes that affect the behaviour of the wider AI-enabled system (Dev et al., 2025; Yu et al., 2026; Martรญnez-Fernรกndez et al., 2022; Blood et al., 2023). They also show that the strength of an audit conclusion depends on the information and system access available to the auditor (Casper et al., 2024; Mรถkander et al., 2024). The audit should therefore identify external dependencies that can affect the robustness claim, assess how their failure or change could alter system behaviour, and state how any inability to examine them limits the conclusion that can be drawn. To support this assessment, the audit may use Software Bills of Materials (SBOMs), which document software components and dependencies, and Artificial Intelligence Bills of Materials (AIBOMs), which extend this documentation to AI-specific assets such as models, datasets, software dependencies, configurations, and provenance information (Nocera et al., 2025; Radanliev et al., 2026). 5.9. Category 9: Human-in-the-Loop This category defines how human judgement, review, and oversight are assigned when robustness auditing decisions cannot be supported by automated procedures alone. 5.9.1. MR-9.1: Robustness-aware Human Participation Levels Human participation defines when a person must review a robustness-related result, what action that person may take, and whether the system can continue operating without human approval. Human review may be needed when a robustness test fails, the system repeatedly produces unreliable results, or continued operation could create serious consequences (Tocchetti et al., 2025; Kou et al., 2024; McCormack et al., 2025; Chamola et al., 2023; Chen and Das, 2023; Casper et al., 2024; Kokina et al., 2025). The audit should state when human review is required and whether the reviewer may correct a result, reject a system output, approve continued operation, or stop the system from acting (Oala et al., 2020; Chamola et al., 2023; Thelisson and Verma, 2024; Chen and Das, 2023; Casper et al., 2024). Different forms of human involvement should be distinguished by whether the person only reviews the system, can intervene in its operation, or has final authority over its actions (Habbal et al., 2024). 5.10. Category 10: Evaluation This category establishes how robustness conclusions are reassessed over time, how unresolved robustness work is prioritised, and how failures are validated across system layers. 5.10.1. MR-10.1: Continuous Robustness Assessment This MR means checking again whether an earlier robustness conclusion still holds after the system or its operating conditions change. A conclusion reached before deployment may no longer be valid after changes to the model, data, deployment environment, protective measures, or attack methods considered in the earlier assessment (Blood et al., 2023; Li and Goel, 2025; Khinvasara et al., 2024). The audit should therefore require a new assessment at planned intervals and after important changes, such as model updates, retraining, deployment changes, robustness failures, attack methods discovered or reported after the earlier assessment, or changes to measures used to reduce those failures (Momcilovic et al., 2026; Laux et al., 2024; Chen and Das, 2023; Raji et al., 2020; Thelisson and Verma, 2024; Simbeck, 2024). Each new assessment should identify which earlier robustness conclusions may have been affected and whether they are still supported by the available evidence (Mรถkander et al., 2024; Madaio et al., 2022; Wang et al., 2026; McCormack et al., 2025; Costanza-Chock et al., 2022; Koshiyama et al., 2024). 5.10.2. MR-10.2: Robustness-aware Progress Tracking and Prioritisation Progress tracking means keeping a clear record of each robustness problem until it has been resolved or formally accepted. Without such tracking, problems may be forgotten, repeated, or left unresolved (Costanza-Chock et al., 2022; Mรถkander et al., 2024; Chen and Das, 2023; Momcilovic et al., 2026). The audit should record the problem, its current status, the action required, and the evidence used to judge whether it has been resolved (Momcilovic et al., 2026; Chen and Das, 2023; Costanza-Chock et al., 2022). Problems should be addressed first when they are likely to cause more serious or repeated consequences, or affect more users (Yu et al., 2026; Robertson et al., 2023; Petersen et al., 2022). As a follow-up, the audit should test the problem again after corrective action to determine whether it still remains (Chen and Das, 2023; Koshiyama et al., 2024; Bayram and Ahmed, 2025). 5.10.3. MR-10.3: Cross-Layer Failure Validation Cross-layer failure validation means checking whether a robustness problem stems from interactions between different parts of the AI-enabled system rather than the model alone. Such problems may involve the model, software framework, external service, infrastructure, runtime process, or human interaction (Yu et al., 2026). The audit should identify the parts involved, reproduce the suspected failure, observe what happens, and determine whether the evidence supports the proposed cause (Raji et al., 2020; Blood et al., 2023; Dev et al., 2025; Mohseni et al., 2022; Yu et al., 2026). If some interactions cannot be tested, the audit should state which ones remain unchecked (Yu et al., 2026). 5.11. Category 11: Monitoring This category establishes how deployed AI-enabled systems are observed for robustness-relevant change, runtime violations, and auditable operational records after the original assessment. 5.11.1. MR-11.1: Continuous, Automated, and Real-time Robustness Monitoring Robustness monitoring means observing a deployed AI-enabled system for changes that may reduce its ability to behave reliably. Such changes may include worsening performance, changes in incoming data, unusual outputs, signs that the system is being deliberately manipulated, or changes in operating conditions (Blood et al., 2023; Chen and Das, 2023; Li and Goel, 2025; Gupta et al., 2025; Mohseni et al., 2022; Nolte et al., 2025; Thelisson and Verma, 2024; Chen et al., 2025; Vadlapati, 2026; Narula et al., 2025; Methnani et al., 2024; Bayram and Ahmed, 2025; Robertson et al., 2023; Khinvasara et al., 2024). Monitoring may be continuous, meaning that the system is checked throughout operation, and automated where software can perform those checks without repeated manual inspection. Where rapid response is required, changes should be detected and reported as they occur or soon afterwards. The audit should state what is monitored, how often it is checked, and what change should trigger further investigation or reassessment (see MR-10.1 and MR-10.2) (Chen and Das, 2023; Bayram and Ahmed, 2025; Koshiyama et al., 2024; Robertson et al., 2023; Martรญnez-Fernรกndez et al., 2022; Chamola et al., 2023; Kaya et al., 2026; Ojewale et al., 2025). Automated alerts should be checked before being treated as evidence that robustness has degraded (Blood et al., 2023; Raji et al., 2020; Narula et al., 2025; Schmelczer and Visser, 2023; Ala-Luopa et al., 2024; McCormack et al., 2025). 5.11.2. MR-11.2: Robustness-aware Logging This means keeping records that allow auditors to reconstruct what happened when a robustness-related problem occurs. Without such records, it may be difficult to determine what the system received, what it produced, which model version was running, or what changes had been made (Methnani et al., 2024; Li and Goel, 2025). The audit should therefore check whether logs capture the information needed to investigate the event, including relevant inputs, outputs, model versions, system settings, alerts, and recorded failures (Thelisson and Verma, 2024; Schmelczer and Visser, 2023; Raji et al., 2020; Li and Goel, 2025). The logs should be kept for a defined period, protected from unauthorised access or alteration, and available when later investigation is required (Thelisson and Verma, 2024; Schmelczer and Visser, 2023; Li and Goel, 2025). 5.12. Category 12: Communication This category establishes how evidence from robustness audits is communicated through visual interfaces, risk reports, and user-facing documentation so the intended audience can inspect, interpret, and act on audit results. 5.12.1. MR-12.1: Robustness Dashboards and Visualisation This MR entails presenting robustness audit results in a visual form so that users can quickly see important robustness changes, problems, and trends. This is useful when relevant information is spread across different measures, AI-enabled system components, or periods of system operation (Nolte et al., 2025; Li and Goel, 2025). The audit should check whether the visualisation clearly shows the robustness measures being tracked, important changes over time, detected problems, and any uncertainty that affects interpretation (Xiong et al., 2022; Robertson et al., 2023; Tocchetti et al., 2025; Methnani et al., 2024; Mรผller et al., 2025; Raji et al., 2020; Ala-Luopa et al., 2024). It should also identify who the visualisation is intended for and present enough detail for that audience to understand the result without hiding important problems or limitations (Robertson et al., 2023; Oala et al., 2020; Nolte et al., 2025; Agarwal and Nene, 2025; Kokina et al., 2025). 5.12.2. MR-12.2: Robustness Documentation and Risk Report The written audit report provides a clear record of what was assessed, how the assessment was performed, what was found, and what limits the conclusion. Without this information, readers may not know which evidence supports the robustness conclusion or where uncertainty remains (Mรถkander et al., 2024; Koshiyama et al., 2024; Laux et al., 2024; Costanza-Chock et al., 2022; Ojewale et al., 2025; McCormack et al., 2025; Petersen et al., 2022; Kaya et al., 2026; Raji et al., 2020; Momcilovic et al., 2026). The report should state the audit objective, robustness claim, methods used, evidence examined, main findings, known limitations, unresolved problems, and any required follow-up actions. It should also identify the system, model, data, and test conditions to which the conclusion applies, drawing on retained operational records where needed (see MR-11.2) (Li et al., 2023; Kemmerzell et al., 2025; Oala et al., 2020; Koshiyama et al., 2024; Palazzolo et al., 2024; Vadlapati, 2026; Bayram and Ahmed, 2025; Cooper et al., 2022; Simbeck, 2024; Tsingenopoulos et al., 2025; Ouyang et al., 2021; Gupta et al., 2025; Blood et al., 2023; Yu et al., 2026). 5.13. Category 13: Efficiency This category examines whether robustness auditing activities can be executed within financial, temporal, computational, and operational constraints without overstating what constrained evidence can support. 5.13.1. MR-13.1: Resource Efficiency and Scalability This MR concerns whether a robustness audit can be carried out with the available time, computing resources, infrastructure, and people with the required expertise as the size or complexity of the system increases. Audits may become difficult to perform when models, datasets, or system environments require substantial computation, specialist expertise, or long execution times (Kokina et al., 2025; Tocchetti et al., 2025; Kou et al., 2024; Saqib et al., 2024; Madaio et al., 2022; Nanda et al., 2021; Khinvasara et al., 2024; Kaya et al., 2026; Thelisson and Verma, 2024; Martรญnez-Fernรกndez et al., 2022; Chen et al., 2025; Dev et al., 2025; Wang et al., 2026; Gupta et al., 2025; McCormack et al., 2025). The audit should therefore state the resources and time required and whether those limits reduce the amount of testing that can be performed (Javed et al., 2024; Wei and Liu, 2025; Tocchetti et al., 2025; Saqib et al., 2024; Mรถkander et al., 2024). If resource limits require fewer tests or smaller-scale evaluation, the robustness conclusion should be limited to what was actually assessed (Mohseni et al., 2022; Hornischer, 2026; Kokina et al., 2025; Nanda et al., 2021; Javed et al., 2024; Wei and Liu, 2025). 5.14. Category 14: Adaptability This category establishes whether robustness audits are adapted to the domain, task, model type, data modality, and deployment setting being assessed. 5.14.1. MR-14.1: Domain-specific Adaptability This MR means adjusting the robustness audit to the characteristics of the application area in which the AI-enabled system is used. A test that is suitable in one domain may be misleading in another because operating conditions, acceptable performance, and the consequences of failure can differ (Freiesleben and Grote, 2023; Hardy et al., 2025). The audit should therefore adapt the test conditions, measures, and acceptance criteria to the domain, including the type of data, expected operating conditions, and robustness problems that matter for that application (Oala et al., 2020; Mรถkander et al., 2024; Javed et al., 2024; Ouyang et al., 2021; Petersen et al., 2022; Chen et al., 2025; Wang et al., 2026; Balendran et al., 2025; Mohseni et al., 2022; Gupta et al., 2025). The robustness conclusion should apply only to the domain and conditions that were actually assessed (Roy, 2024; Ouyang et al., 2021; Lefcourt et al., 2022; Mรผller et al., 2025; Freiesleben and Grote, 2023; Tocchetti et al., 2025; Khinvasara et al., 2024; Javed et al., 2024; Casper et al., 2024; Mรถkander et al., 2024). 5.14.2. MR-14.2: Task and Model Versatility This MR means checking whether an audit method remains suitable when the AI task, model type, or input data format changes. A method designed for one setting may give misleading results when transferred to another without adjustment (Kemmerzell et al., 2025). The audit should therefore assess whether the chosen method fits the systemโs task, model type, input data, and intended use, and should state any limits on using that method in other settings (Balendran et al., 2025; Kemmerzell et al., 2025; Li et al., 2023). For example, a method developed for a classification task, a neural-network model, or a single input type may not work in the same way for other tasks, model types, or systems that use text, images, audio, or combinations of these inputs (Balendran et al., 2025). Where no well-established assessment method exists, the audit should state that limitation rather than assume that one method is suitable for every model or task (Thelisson and Verma, 2024). 5.15. Category 15: Accessibility This category concerns whether tools for robustness auditing, explanations, visualisations, documents, and outputs are understandable and usable by their intended audiences (Hancox-Li, 2020; Ojewale et al., 2025; Zahid et al., 2025). 5.15.1. MR-15.1: Accessible and User-Friendly Robustness Tools, Visualisations, and Documents Accessibility means that tools for robustness auditing can be operated, visualisations can be understood, and documents can be read and interpreted by their intended users. User-friendly means that these materials are easy to navigate and use without unnecessary effort. Different users may have different levels of technical knowledge, language needs, or accessibility requirements (Hancox-Li, 2020; Kemmerzell et al., 2025; Chander et al., 2025; Li and Goel, 2025; Ojewale et al., 2025). The audit should therefore check whether tool interfaces are easy to use, visualisations have clear labels and explanations, and documents use language and detail suited to the intended audience (Chander et al., 2025; Ojewale et al., 2025). Non-technical users should receive clear summaries, while qualified users should be able to access the technical detail needed to examine the robustness evidence (Methnani et al., 2024; Kemmerzell et al., 2025; McCormack et al., 2025; Agarwal and Nene, 2025; Chen and Das, 2023). 5.16. Category 16: Configurability This category establishes whether robustness-assessment tools can be configured to match the scoped system, domain, thresholds, perturbations, attack assumptions, and deployment conditions. 5.16.1. MR-16.1: Configurable Robustness Assessment Tools Configurability means that the settings of a robustness-assessment tool can be changed to match the system and the robustness condition being tested. Fixed default settings may give misleading results when the audit requires different levels of input or operating-condition change, different acceptance criteria, or different assumptions about how inputs or system conditions may be changed (Agarwal and Nene, 2025; Kokina et al., 2025; Freiesleben and Grote, 2023; Yang et al., 2020; Javed et al., 2024; Palazzolo et al., 2024; Vadlapati, 2026; Liu et al., 2022). The audit should therefore check whether the required settings can be changed and should state which settings were used and why. Any change from the default settings should be explained, and the selected settings should still match the conditions covered by the robustness claim (Freiesleben and Grote, 2023; Yang et al., 2020; Javed et al., 2024; Liu et al., 2022; Kokina et al., 2025; Palazzolo et al., 2024). Key Findings of RQ1 Finding 1. The literature review produced 43 meta-requirements organised into 16 categories and three dimensions: Human, Functional, and Quality. Finding 2. Most literature-derived meta-requirements concern audit activities and evidence (22 Functional), compared with organisational and human aspects (16 Human) and audit-approach qualities (5 Quality). 6. Practitionersโ Perspectives on the Meta-Requirements RQ2 examines practitionersโ perspectives on the 43 literature-derived meta-requirements evaluated in the survey. The analysis draws on 5,504 Likert ratings from 128 valid respondents and 586 substantive comments. We report respondent characteristics, quantitative support, and qualitative feedback, including proposed refinements, challenges, and additional concerns. These findings informed the final set of meta-requirements presented in Section 5. 6.1. Respondent Characteristics After screening, we retained 128 valid responses. Respondent characteristics describe the validation sampleโs experience and demographic composition, including AI/ML roles, AI robustness assessment or AI robustness auditing experience, robustness-related activities, AI/ML experience, location, age, gender, and education. Professional Background and Experience Respondents represented diverse AI/ML-related backgrounds, including machine-learning engineers, data scientists, AI/ML auditors, software engineers, data engineers, and other related roles. Figure 7 shows these roles together with respondentsโ AI robustness assessment or auditing experience. The two largest experience groups had less than one year (40%) and one to three years (41%) of robustness assessment or auditing experience, respectively, while smaller groups reported longer experience or no prior experience. The sample therefore represents respondents with varying levels of AI/ML and robustness-assessment experience, rather than exclusively experienced AI robustness auditors. The findings should be interpreted in light of this participant composition. Robustness-related activities Respondents had participated in a range of activities directly relevant to the validation task. The most frequently reported were robustness testing (73/128, 57.03%), failure or error analysis (71/128, 55.47%), robustness-focused model evaluation (59/128, 46.09%), robustness metric selection or definition (54/128, 42.19%), and adversarial or non-adversarial evaluation (54/128, 42.19%). Other reported activities included domain-specific robustness evaluation (46/128, 35.94%), development or use of robustness-assessment tools (44/128, 34.38%), AI robustness auditing or assessment (42/128, 32.81%), and post-deployment robustness monitoring (29/128, 22.66%). The complete distribution of respondentsโ robustness-related activities is provided in the replication package (see the Data Availability statement). Geographic distribution Figure 7 shows that respondents came from multiple countries across six continents. Asia accounted for the largest share of the sample (66/128, 51.56%), followed by Africa (24/128, 18.75%), Europe (22/128, 17.19%), North America (11/128, 8.59%), South America (3/128, 2.34%), and Oceania (2/128, 1.56%). The largest country-level contributions came from India, Nigeria, and the United States, followed by Pakistan, Kenya, and Tรผrkiye. Demographic characteristics: The sample also covered different age and educational backgrounds. Most respondents were aged 18-24 years (73/128, 57.03%), followed by 25-34 years (29/128, 22.66%) and 35-44 years (16/128, 12.50%). The remaining respondents were distributed across older age groups. Full distributions for age, gender, education, AI/ML experience, and robustness-related experience are provided in the replication package (see the Data Availability statement). Figure 6. Geographic distribution of respondents by country and continent. Figure 7. Distribution of participants by robustness auditing experience and primary AI/ML-related role. 6.2. Overall Quantitative Validation The 128 valid responses yielded 5,504 ratings across the 43 literature-derived meta-requirements. Ratings of 4-5 were classified as agreement, 3 as neutral, and 1-2 as disagreement. Overall, 4,469 ratings (81.20%) indicated agreement, 709 (12.88%) were neutral, and 326 (5.92%) indicated disagreement. By response option, 2,624 ratings (47.67%) were strongly agree, 1,845 (33.52%) agree, 235 (4.27%) disagree, and 91 (1.65%) strongly disagree. Table 8. Overall practitioner validation results across all MR ratings. Response group No. ratings % Agree / Strongly agree 4,469 81.20 Neutral 709 12.88 Disagree / Strongly disagree 326 5.92 Total 5,504 100.00 Table 9. Distribution of individual Likert responses across all MR ratings. Likert response No. ratings % Strongly agree 2,624 47.67 Agree 1,845 33.52 Neutral 709 12.88 Disagree 235 4.27 Strongly disagree 91 1.65 Total 5,504 100.00 Table 10. Support classification of rated meta-requirements. Support status No. of MRs Strongly supported 41 Supported 2 Mixed or unclear 0 Challenged 0 Total rated MRs 43 These results indicate broad practitioner support for the literature-derived meta-requirements for AI robustness auditing. However, the neutral and disagreement-level responses show that validation should not be interpreted as unqualified acceptance. Instead, the quantitative results establish the overall level of practitioner support, while the qualitative comments explain where clarification, refinement, contextualisation, or extension was needed. The absence of a quantitatively challenged MR does not mean that practitioners raised no concerns. It means no MR reached the predefined aggregate threshold to be classified as challenged. The 326 disagreement-level ratings and the open-text comments therefore remained important for interpreting conditional acceptance, feasibility concerns, scope concerns, and overlap concerns. 6.3. Meta-Requirement-Level Validation Table 12 summarises the practitioner ratings for the 43 literature-derived meta-requirements. Overall, support was consistently high: agreement or strong agreement ranged from 71.88% to 88.28% across the individual meta-requirements. Of the 43 meta-requirements, 41 were classified as strongly supported and two as supported; none was quantitatively classified as challenged. MR-8.3, concerning compatibility with AI-enabled system infrastructure, received the highest agreement (88.28%), followed by MR-5.3 on robustness-focused data quality checking (87.50%) and MR-12.2 on documenting robustness risks, evidence, and conclusions (86.72%). The two meta-requirements classified as supported rather than strongly supported were MR-2.2 on risk-based stakeholder involvement (71.88%) and MR-6.5 on robustness-focused training-time mitigation (73.44%). Despite their comparatively lower ratings, both retained clear majority agreement. 6.3.1. Category-Level Validation The category-level analysis assessed whether practitioner support extended across the original survey structure. All 16 categories received broad agreement-level support (Table 11). Table 11. Category-level agreement ranking across the original 16 survey categories. Rank Category Agree (%) Rank Category Agree (%) 1 Workflow Integration 86.20 9 Communication 81.64 2 Configurability 83.59 10 Robustness Definition 81.25 3 Data Auditability 83.12 11 Human-in-the-Loop 81.25 4 Knowledge and Skills 82.81 12 Accessibility 81.25 5 Output Auditability 82.81 13 Efficiency 79.69 6 Monitoring 82.81 14 Standards and Governance 79.49 7 Adaptability 82.81 15 Model Auditability 79.38 8 Evaluation 82.55 16 Engagement 76.37 Practitioner support extended across all 16 survey categories. Qualitative feedback identified needs for clearer wording and sharper boundaries between some meta-requirements. We therefore interpreted category-level ratings alongside the open-text feedback. 6.4. Qualitative Practitioner Validation and Meta-Requirement Refinement Open-text responses complemented the quantitative ratings by showing how practitioners proposed retaining, refining, challenging, or extending the literature-derived meta-requirements. Comments were coded (open coding) as confirmations, refinements, scope clarifications, implementation concerns, additions, merge/split suggestions, or challenges. During reconciliation, we compared each proposed change with the existing meta-requirements and other comments; we treated proposals already covered as refinements rather than new additions, merges, or splits. The final decisions comprised 162 retentions, 394 refinements, 13 no-change cases, and 15 comments not usable for meta-requirement synthesis. Only two comments retained an Add decision, both identifying the same previously uncovered concern. Retention of the literature-derived meta-requirements. Practitioner comments reinforced the broad quantitative support for the literature-derived meta-requirements by explaining why specific requirements remained important. P003 described governance and lifecycle requirements as "essential and foundational for credible audits," while P011 and P034 emphasised continuous and proactive lifecycle integration. P117 similarly highlighted standardised metrics, terminology, governance, and audit procedures as important for consistent, reproducible, and comparable assessments. Comparable confirmations across data, model, output, monitoring, adaptability, and configurability concerns (e.g., P019, P031, P047, P072, P091, P110, P117) supported retention of the underlying requirements while allowing further refinement of their scope and implementation. Refinement of existing meta-requirements Refinement was the dominant qualitative outcome. Practitioners generally accepted the literature-derived requirements but requested clearer boundaries, applicability conditions, evidence expectations, and implementation guidance. A recurring concern was balancing standardisation with contextual flexibility (P006, P010, P012, P016, P019, P040, P096, P121, P123). Respondents supported common metrics, terminology, governance, and reporting expectations, while cautioning against requiring identical tools or methods across contexts. P016 explicitly called for a "balance [between] standardization and flexibility," and P123 argued that standardisation should support comparability without constraining new evaluation methods, while also noting an "absence of actionable, tool-agnostic guidance." These comments helped distinguish common expectations for comparability from justified, context-specific choices of tools, metrics, and evaluation methods. Practitioners also refined audit scope, access conditions, and assurance claims (P002, P006, P017, P024, P092, P112, P123). P002 called for explicit "out-of-scope" boundaries, while P017 and P112 distinguished white-, grey-, and black-box contexts and argued that audit requirements should reflect available access. P123 likewise identified "unclear access scope" as a practical barrier, and P024 and P092 requested clearer assumptions, thresholds, limitations, and evidence conditions. These comments led to making access, exclusions, assumptions, and limitations explicit and aligning assurance claims with available evidence. Risk proportionality and lifecycle reassessment formed a second cross-cutting pattern (P005, P016, P019, P024, P047, P048, P072, P077, P081, P092, P106, P108, P121). Practitioners argued that stakeholder involvement, review, testing, monitoring, and reassessment should vary with system risk, deployment context, and material change. P121 cautioned against "over-burdening low risk systems," while P024 proposed risk-tiered requirements. P047 stressed that robustness is valid only for a specific context and time, and P058 described it as a "living process." P077 and P081 further supported change-sensitive reassessment through scheduled, event-driven, and incident-driven triggers. These comments refined continuous assessment into a risk-sensitive, change-triggered process rather than repeated testing after every change. Stakeholder participation, expertise, independence, and human oversight were also refined (P002, P005, P006, P010, P016, P017, P022, P047, P057, P091, P117, P122, P125). P006 called for documenting stakeholder input and resolving conflicting viewpoints, while P057 distinguished stakeholder identification from risk-based prioritisation. P047 argued that domain expertise should shape the test basis, assumptions, risks, and evidence needs. P002 and P057 further stressed sufficient technical access and independence from development or commercial pressures. For human oversight, P002 warned against "rubber-stamping," while P005, P016, and P091 linked involvement to risk, reviewer expertise, defined intervention conditions, and auditable decisions. These comments clarified who should participate, when their input is required, and how to exercise independence and oversight. A second refinement concerned the distinction between auditing and mitigation and the evidence needed to support conclusions (P034, P037, P047, P058, P072, P091, P112, P123). Practitioners stressed that the presence of a mitigation does not establish robustness; audits should evaluate its effectiveness, conditions, limitations, and trade-offs. P058 warned against "a conflation of auditing with engineering," while P112 called for assessment of the "actual impact and trade-offs" of mitigations. P034, P072, and P091 further emphasised testing mitigations across failure types, stress conditions, users, and deployment scenarios. Related comments requested clearer thresholds, versioning, reproducibility, reporting, scalable testing, and configuration records (e.g., P024, P047, P072, P091, P112, P123). We incorporated these points as refinements where the underlying requirement already existed. Challenges and qualifications. No literature-derived meta-requirement was removed; however, some practitioner comments questioned whether specific implementation mechanisms should be universal or mandatory. The clearest qualifications concerned workflow integration and communication. P121 described workflow and MLOps integration as "important but not foundational," while P123 questioned treating mature MLOps integration as a prerequisite for valid auditing. P108 and P125 likewise argued that integration depth should depend on system complexity, deployment context, risk, resources, and MLOps maturity. P077 viewed dashboards as optional, whereas P123 and P113 stressed that dashboards should not replace a documented and authoritative evidence base. These comments challenged implementation-specific mechanisms rather than the underlying needs for integration, communication, monitoring, and evidence retention. Other comments qualified how evidence should be interpreted. P058 supported stakeholder involvement but argued that audit conclusions should remain evidence-driven. P077 also questioned whether accessibility should have the same mandatory status as other meta-requirements, although other respondents emphasised accessible, audience-appropriate communication (e.g., P037, P047, P054, P072, P081, P091, P097, P099). These comments were retained as qualifying evidence and used to introduce proportionality, conditionality, and clearer distinctions between required audit outcomes and optional implementation mechanisms. Practitioner-identified extension. Several comments initially proposed additional requirements, including reproducibility and re-audit, auditor independence, stakeholder-conflict management, competency assessment, risk-tiered assurance, incident response, evidence sufficiency, configuration governance, and regulatory or reporting controls (e.g., P006, P022, P024, P058, P081, P091, P122, P125). Reconciliation against the literature-derived meta-requirements showed that most clarified or operationalised existing requirements and were therefore incorporated as refinements. One concern remained insufficiently represented: assurance for externally controlled components and dependencies. P079 called for addressing "third-party and supply chain risks," including external models, foundation-model APIs, pretrained components, datasets, libraries, plugins, and cloud services, while noting that restricted visibility limits defensible assurance claims. P102 independently highlighted third-party validation and auditing of foundation models and APIs, with related concerns also raised by P111 and P123 (P079, P102, P111, P123). Because P079 and P102 identified the same uncovered concern, their convergent evidence supported one additional practitioner-informed meta-requirement, MR-8.4, External Component and Supply Chain Assurance. Merge and split proposals. Practitioners identified overlap or uneven granularity among some literature-derived requirements (P034, P057, P081, P091, P092, P113). P034 noted overlap among MR-1.1, MR-1.2, and MR-1.3, while P057 distinguished stakeholder identification (MR-2.1) from risk-based prioritisation (MR-2.2). P113 similarly observed overlap between MR-6.1 and MR-6.5 but supported separating stress-test outcomes from the rationale and effectiveness of training-time mitigation. P081, supported by P037 and P091, proposed distinguishing output-failure identification from post-training mitigation assessment. Broader comments from P091 and P092 also noted overlap across evaluation, monitoring, workflow integration, documentation, accessibility, adaptability, and configurability. No MR-level merge or split was adopted; these comments were used to sharpen requirement boundaries, while the original 16-category structure was retained. Integrated effect of practitioner validation. Practitioner validation was primarily refinement-oriented rather than replacement-oriented. The literature-derived concerns were broadly retained, while practitioners clarified their scope, applicability, evidence expectations, and implementation. Challenges mainly targeted universal or implementation-specific interpretations, and proposed merges and splits were addressed through boundary clarification or refinement of existing requirements. The main exception was the gap concerning external components and supply chain dependencies, which resulted in one new meta-requirement. Overall, the 43 literature-derived meta-requirements were refined and extended to a practitioner-informed set of 44 meta-requirements while retaining the 16-category structure. Key Findings of RQ2 Finding 3. Practitioner ratings showed broad support for the literature-derived meta-requirements: 41 of the 43 meta-requirements were strongly supported and two were supported, with MR-level agreement ranging from 71.88% to 88.28%. Finding 4. Practitioner feedback primarily refined the existing meta-requirements and identified one additional standalone requirement on external-dependency robustness, resulting in a final set of 44 meta-requirements across 16 categories. Table 12. Practitioner validation results for the 43 literature-derived meta-requirements (n=128n=128 respondents per meta-requirement). Cat. MR Meta-requirement n Mean SD Median Mode IQR Agree (%) Neutral (%) Disagree (%) Quantitative status 1 MR-1.1 Should use standardized robustness metrics 128 4.23 1.12 5.0 5 1.0 82.81 6.25 10.94 Strongly supported 1 MR-1.2 Should use shared robustness terminology 128 4.15 1.04 4.0 5 1.0 76.56 16.41 7.03 Strongly supported 1 MR-1.3 Should use standardized robustness tools 128 3.90 0.92 4.0 4 0.0 78.12 12.50 9.38 Strongly supported 1 MR-1.4 Should provide actionable audit guidance 128 4.16 1.00 4.0 5 1.0 75.78 17.19 7.03 Strongly supported 1 MR-1.5 Should integrate robustness into AI lifecycle workflows 128 4.30 0.93 5.0 5 1.0 81.25 14.06 4.69 Strongly supported 1 MR-1.6 Should establish robustness governance protocols 128 4.31 0.97 5.0 5 1.0 83.59 10.16 6.25 Strongly supported 1 MR-1.7 Should define audit access requirements 128 4.22 0.96 4.5 5 1.0 79.69 14.06 6.25 Strongly supported 1 MR-1.8 Should define auditor procedures 128 4.17 0.97 4.0 5 1.0 78.12 14.06 7.81 Strongly supported 2 MR-2.1 Should identify diverse stakeholder perspectives, including those affected by robustness failures based on robustness risk 128 4.10 0.96 4.0 5 1.0 76.56 15.62 7.81 Strongly supported 2 MR-2.2 Should prioritize stakeholder involvement by robustness risk 128 4.00 1.00 4.0 5 2.0 71.88 19.53 8.59 Supported 2 MR-2.3 Should include independent robustness review 128 4.18 0.94 4.0 5 1.0 75.78 20.31 3.91 Strongly supported 2 MR-2.4 Should involve experts early in robustness framing 128 4.27 0.92 5.0 5 1.0 81.25 13.28 5.47 Strongly supported 3 MR-3.1 Should scope robustness concerns and audit claims 128 4.27 0.90 5.0 5 1.0 81.25 14.06 4.69 Strongly supported 3 MR-3.2 Should identify assumptions about robustness trade-offs 128 4.26 0.97 5.0 5 1.0 81.25 11.72 7.03 Strongly supported 4 MR-4.1 Should involve relevant multidisciplinary robustness expertise 128 4.32 0.96 5.0 5 1.0 81.25 14.06 4.69 Strongly supported 4 MR-4.2 Should provide role-specific robustness training 128 4.24 0.86 4.0 5 1.0 84.38 10.94 4.69 Strongly supported 5 MR-5.1 Should ensure robustness assessment data coverage 128 4.33 0.89 5.0 5 1.0 85.94 9.38 4.69 Strongly supported 5 MR-5.2 Should document data description, provenance, and lineage 128 4.23 0.88 4.0 5 1.0 81.25 13.28 5.47 Strongly supported 5 MR-5.3 Should check data quality for robustness assessment 128 4.48 0.89 5.0 5 1.0 87.50 7.81 4.69 Strongly supported 5 MR-5.4 Should identify subgroups and data-level bias sources 128 4.20 0.89 4.0 5 1.0 83.59 12.50 3.91 Strongly supported 5 MR-5.5 Should apply robustness-focused data preprocessing mitigations 128 4.02 0.96 4.0 4 1.0 77.34 14.84 7.81 Strongly supported 6 MR-6.1 Should perform model robustness stress testing 128 4.29 0.99 5.0 5 1.0 81.25 12.50 6.25 Strongly supported 6 MR-6.2 Should specify model intended use and limits 128 4.16 0.99 4.0 5 1.0 77.34 17.97 4.69 Strongly supported 6 MR-6.3 Should support explainability for robustness assessment 128 4.12 0.89 4.0 4 1.0 80.47 13.28 6.25 Strongly supported 6 MR-6.4 Should identify model-level robustness failure sources 128 4.31 0.99 5.0 5 1.0 84.38 7.03 8.59 Strongly supported 6 MR-6.5 Should apply robustness-focused training-time mitigations 128 3.94 0.94 4.0 4 2.0 73.44 21.09 5.47 Supported 7 MR-7.1 Should identify output-level bias or failure sources and assess whether post-training mitigations effectively address them 128 4.27 0.95 5.0 5 1.0 82.81 11.72 5.47 Strongly supported 8 MR-8.1 Should integrate robustness checks into AI development workflows 128 4.34 0.85 5.0 5 1.0 85.94 10.16 3.91 Strongly supported 8 MR-8.2 Should embed robustness checks in ML/MLOps pipelines 128 4.27 0.84 4.0 5 1.0 84.38 13.28 2.34 Strongly supported 8 MR-8.3 Should ensure compatibility with AI-enabled system infrastructure 128 4.27 0.77 4.0 4 1.0 88.28 7.81 3.91 Strongly supported 9 MR-9.1 Should define human participation levels in robustness decisions 128 4.18 0.96 4.0 5 1.0 81.25 12.50 6.25 Strongly supported 10 MR-10.1 Should reassess robustness continuously or when triggered 128 4.25 1.01 5.0 5 1.0 83.59 8.59 7.81 Strongly supported 10 MR-10.2 Should track and prioritize robustness assessment progress 128 4.20 0.88 4.0 5 1.0 79.69 16.41 3.91 Strongly supported 10 MR-10.3 Should validate failures across relevant system layers 128 4.27 0.86 4.0 5 1.0 84.38 12.50 3.12 Strongly supported 11 MR-11.1 Should maintain continuous, automated robustness monitoring 128 4.22 0.93 4.0 5 1.0 81.25 13.28 5.47 Strongly supported 11 MR-11.2 Should log evidence needed to diagnose robustness failures 128 4.33 0.85 5.0 5 1.0 84.38 10.94 4.69 Strongly supported 12 MR-12.1 Should provide robustness dashboards and visualizations 128 3.99 1.04 4.0 4 1.0 76.56 14.06 9.38 Strongly supported 12 MR-12.2 Should document robustness risks, evidence, and conclusions 128 4.38 0.87 5.0 5 1.0 86.72 7.03 6.25 Strongly supported 13 MR-13.1 Should ensure robustness auditing is cost-effective, time-efficient, and scalable 128 4.18 0.93 4.0 5 1.0 79.69 14.84 5.47 Strongly supported 14 MR-14.1 Should adapt robustness assessment to the deployment domain 128 4.26 0.85 4.0 5 1.0 85.94 10.16 3.91 Strongly supported 14 MR-14.2 Should support relevant tasks, models, and modalities 128 4.18 0.90 4.0 5 1.0 79.69 14.06 6.25 Strongly supported 15 MR-15.1 Should make robustness tools, visualizations, and related outputs accessible to intended users 128 4.21 0.99 4.0 5 1.0 81.25 11.72 7.03 Strongly supported 16 MR-16.1 Should allow configuration of robustness assessment tools 128 4.20 0.87 4.0 5 1.0 83.59 10.94 5.47 Strongly supported Note. Agree (%) combines Agree and Strongly agree; Disagree (%) combines Disagree and Strongly disagree. 7. Discussion This section interprets the main insights arising from the literature synthesis and practitioner validation, relates them to prior work, and considers their implications for AI robustness auditing research and practice. 7.1. A Robustness Test Is Necessary but Not Sufficient for a Robustness Audit The derived meta-requirements reinforce the distinction between robustness testing and auditing: tests provide behavioural evidence, whereas an audit evaluates whether that evidence supports a defined robustness claim within specified scope, criteria, assumptions, and limitations (see Sections 5.3 and 5.6). Prior work reflects this division: robustness research develops benchmarks, metrics, and evaluation methods for generating such evidence (Hendrycks and Dietterich, 2019; Taori et al., 2020; Freiesleben and Grote, 2023), whereas AI auditing research addresses how evidence is scoped, documented, and interpreted for assurance decisions (Raji et al., 2020; Mรถkander, 2023; Mรถkander et al., 2024). Practitioner feedback reinforces this interpretation; P123 emphasised documented evidence, limitations, and reproducibility conditions (see Section 6.4). Thus, a successful robustness test supports an audit conclusion only when its conditions and evidence are traceable to the robustness claim being assessed. 7.2. From Model-Level Robustness to System-Level Assurance Our findings indicate that model-level robustness cannot, by itself, establish the robustness of the deployed AI-enabled system, because failures may arise from data, preprocessing, software components, output controls, or deployment workflows (see Sections 5.5-5.8). Model-level evidence must therefore be interpreted within the wider system boundary and the interactions that can affect the robustness claim. This interpretation aligns with prior work distinguishing model- from system-level assessment (Mรถkander et al., 2024; Nolte et al., 2025) and showing that failures can originate outside the model and propagate across components (Yu et al., 2026). Practitioner feedback supports this view. For example, P031 noted that failures are "rarely isolated to the model" and should be assessed across the system (see Section 6.4). Accordingly, robustness auditing should treat model-level evidence as one component of system-level assurance and bound audit conclusions to the deployed system and relevant component interactions. 7.3. Strong Literature-Practice Convergence with Contextual Qualifications The RQ2 findings indicate substantial convergence between the literature-derived meta-requirements and practitioner judgement, but not unconditional acceptance. Practitioner feedback largely retained the underlying concerns while refining their scope, access assumptions, feasibility, and contextual applicability (see Section 6.4). P016โs call to balance "standardization with flexibility" captures this qualification. Prior work similarly suggests that common assurance requirements can provide a shared baseline, while their operationalisation depends on system and deployment context (Costanza-Chock et al., 2022; Freiesleben and Grote, 2023; Thelisson and Verma, 2024). The implication is therefore not a fixed audit procedure, but a stable set of robustness auditing concerns whose implementation should remain context-sensitive. Within the studied practitioner sample, literature provides a useful foundation for what should be addressed, while practitioner input helps determine how those requirements should be applied. 7.4. Robustness Is Conditional, Not Universal Our findings indicate that robustness depends on the claim being assessed rather than being a context-free property of an AI-enabled system. The relevant system boundary, protected behaviour, operating conditions, disturbances or threats, and acceptance criteria determine what the available evidence can support (see Sections 5.3 and 5.14). Practitioner feedback reinforces this interpretation: P002 called for explicit "out-of-scope" boundaries, while other respondents emphasised context-appropriate threat models, thresholds, operating conditions, and evaluation methods (see Section 6.4). Prior work similarly treats robustness as dependent on assessment conditions and context-specific assurance criteria (Freiesleben and Grote, 2023; Mรถkander et al., 2024; Thelisson and Verma, 2024). Accordingly, audit conclusions should state what is claimed to be robust, under which conditions, and against which criteria, rather than generalising beyond the assessed context. 7.5. Robustness Assurance Is Time-Bounded and Requires Reassessment Our findings indicate that robustness assurance remains valid only while the conditions supporting the original conclusion remain applicable. Changes to models, data, deployment conditions, threats, or other relevant factors can therefore weaken earlier evidence (see Sections 5.10 and 5.11). Prior work similarly treats AI assurance as an ongoing activity in which material system changes may require renewed evaluation (Mรถkander et al., 2024; Thelisson and Verma, 2024; Simbeck, 2024; Madaio et al., 2022). Practitioner feedback adds an important qualification: P047 described robustness as valid only for a specific context and time, while P077 cautioned against reassessment after every minor change (see Section 6.4). Reassessment should therefore be triggered by changes that can materially affect the robustness claim, with its scope proportionate to their significance and risk. Robustness assurance is thus better treated as a maintained conclusion than as a one-time judgement. 7.6. An Auditable AI-Enabled System Requires an Audit-Ready Organisation Our findings indicate that technical auditability alone is insufficient for credible robustness auditing. Governance, engagement, competence, and human-oversight requirements show that robustness evidence must be interpreted and acted on by people with appropriate expertise, independence, authority, and defined responsibilities (see Sections 5.1, 5.2, 5.4, and 5.9). Prior work similarly frames AI auditing as an organisational and interdisciplinary activity rather than a purely technical assessment (Raji et al., 2020; Costanza-Chock et al., 2022; Chander et al., 2025; Li and Goel, 2025). Practitioner feedback reinforces this interpretation: P003 emphasised early expert involvement and independence, while P006 highlighted documented stakeholder input and resolution of conflicting viewpoints (see Section 6.4). Accordingly, audit readiness should cover both the AI-enabled system and the organisation by defining required expertise, independence, responsibilities, decision authority, and procedures for handling disagreements and follow-up actions. 7.7. External Components Extend the Scope of Robustness Audits Our findings indicate that robustness assurance may depend on components outside the audited organisationโs direct control. External models, APIs, datasets, libraries, plugins, and cloud services can affect the robustness claim while limiting auditor visibility and access to evidence. Practitioner feedback exposed this gap: P079 highlighted "third-party and supply chain risks," while P102 identified foundation models and third-party APIs as requiring explicit audit attention (see Section 6.4). Prior work similarly shows that assurance evidence may be distributed across providers, deployers, and auditors (Mรถkander et al., 2024; Thelisson and Verma, 2024; Laux et al., 2024). MR-8.4 therefore extends the audit boundary to relevant external dependencies by requiring the audit to identify them, assess how their failure or change could affect the robustness claim, and state how limited access constrains the robustness conclusion (Dev et al., 2025; Yu et al., 2026; Martรญnez-Fernรกndez et al., 2022; Blood et al., 2023; Casper et al., 2024; Mรถkander et al., 2024). Relevant records for identifying and tracing these external dependencies may include SBOMs and AIBOMs (Nocera et al., 2025; Radanliev et al., 2026). Because MR-8.4 emerged after the original requirements were rated, its practical value requires further empirical evaluation. 7.8. Implications for Practice The meta-requirements are intended to support the completeness and design of AI robustness audits by specifying what should be defined, tested, documented, assigned, monitored, and reassessed. They may also complement instruments such as the EU AI Act and NIST AI RMF (Thelisson and Verma, 2024; Li and Goel, 2025), although no systematic mapping to their provisions was performed. 7.8.1. For AI Robustness Auditors Our findings imply that robustness audits should begin with a clearly defined robustness claim rather than an available test. Auditors should establish the system boundary, operating conditions, relevant threats or disturbances, access conditions, metrics, and decision criteria before selecting evaluation methods (Raji et al., 2020; Mรถkander et al., 2024). For example, these requirements may translate into distribution-shift tests and reassessment triggers for an image classifier, or jailbreak conditions, tool interactions, and dependency evidence for an LLM-based system. Accordingly, audit findings should remain traceable to the supporting evidence and evaluated conditions, while limitations in evidence or access should be stated explicitly (see Sections 7.1 and 7.4). 7.8.2. For Providers and Deployers of AI-Enabled Systems Our findings indicate that providers and deployers should maintain audit-ready evidence throughout the AI-enabled system lifecycle. This includes preserving relevant system versions, data provenance, configurations, evaluation settings, monitoring records, incidents, external dependencies, and robustness decisions (Raji et al., 2020; Simbeck, 2024; Li and Goel, 2025). Accordingly, changes that may affect a robustness claim should be traceable to the reassessment undertaken and any resulting follow-up actions (see Sections 7.2 and 7.5). 7.8.3. For Builders of Robustness Auditing Tools Our findings indicate that robustness testing tools are insufficient on their own for AI robustness auditing. Existing toolkits can generate adversarial examples, run perturbation tests, measure robustness, or identify vulnerabilities (Agarwal and Nene, 2025). However, tools that support robustness auditing should also link the robustness claim and scope to the test configuration, system versions, supporting evidence, findings, mitigation status, retesting, and reassessment (Raji et al., 2020). Accordingly, different user views should remain linked to the same audit evidence record to support consistent, traceable reporting. 7.8.4. For Standards and Certification Bodies Our findings suggest that standards and certification schemes should define common requirements for substantiating robustness claims rather than mandate a single test, metric, or threshold. These requirements should cover terminology, claim definition, minimum evidence, access disclosure, auditor competence, reporting, reassessment, and closure of audit findings (Thelisson and Verma, 2024; Nolte et al., 2025). However, assessment methods and thresholds should remain adaptable to the domain, system type, risk level, and deployment context (see Section 7.4). Accordingly, standards should require such choices and deviations to be justified and should state how restricted access or external dependencies limit the assurance that can be claimed. 7.9. Implications for Research on Robustness Auditing The findings also identify several directions for robustness auditing research. In particular, practitioner validation establishes the relevance and interpretation of the meta-requirements, but further work is needed to determine how they should be operationalised and whether their use improves audit practice. 7.9.1. Operationalising the Meta-Requirements through a Robustness Auditing Artefact This study identifies and validates meta-requirements for AI robustness auditing but does not specify how to implement them. Future design-science research should therefore translate them into a concrete audit artefact that defines procedures, links robustness claims to evidence, specifies access and decision rules, assigns responsibilities, and supports reassessment. Building on structured AI auditing processes (Raji et al., 2020; Mรถkander et al., 2024), such an artefact should be evaluated for usability, evidence completeness, traceability, and integration with existing development, MLOps, governance, and assurance processes. This would establish how the meta-requirements can be applied consistently while allowing adaptation to different AI-enabled systems and audit contexts. 7.9.2. Evaluating the Meta-Requirements in Real Audit Engagements Practitioner validation establishes perceived relevance, but it does not show that applying the meta-requirements improves the quality of an actual robustness audit. Future case studies and action-research engagements should therefore apply the meta-requirements to AI-enabled systems that vary in domain, architecture, risk, lifecycle stage, and audit-access level. Such studies could examine audit coverage, evidence completeness, traceability, effort, finding quality, follow-up actions, and decision usefulness. This would extend prior work calling for structured AI auditing processes to be evaluated within organisational practice (Raji et al., 2020; Mรถkander et al., 2024). The objective is to determine which meta-requirements are practically effective, which remain difficult to operationalise, and where contextual adaptation is necessary without weakening the evidential basis of the audit. 7.9.3. Assessing Inter-Auditor Consistency and Repeatability A further open question is whether the meta-requirements help different auditors reach sufficiently consistent and explainable conclusions. The present validation examined practitioner support for the meta-requirements, not their consistency of application. Future studies could therefore ask multiple auditors to assess the same AI-enabled system and compare the scopes they define, the evidence they request, the tests they select, the findings they identify, and the conclusions they reach. Where feasible, comparison with audits conducted without guidance from the meta-requirements could test whether their use improves traceability, transparency, and repeatability. The goal should not be identical audit reports, since professional judgement may legitimately differ across contexts, but rather to determine whether the meta-requirements makes such differences explicit and defensible through documented scope, assumptions, evidence, access limitations, and decision criteria. 8. Threats to Validity We considered potential threats to external, construct, internal, and conclusion validity (Wohlin et al., 2012) arising from the literature synthesis and practitioner validation. 8.1. External Validity External validity concerns the extent to which the findings generalise beyond the reviewed literature and practitioner sample (Wohlin et al., 2012). Relevant studies may have been missed because AI robustness auditing spans multiple research communities and terminology, and relevant evidence may exist outside academic literature. In particular, the practitioner-oriented and interview or survey search terms may have underrepresented technically relevant robustness studies that did not use such terminology. We mitigated this through multiple database searches, pilot-refined search strings, supplementary Google Scholar screening, and explicit eligibility criteria (see Section 4.2). Nevertheless, the 74 retained studies represent a systematically selected academic evidence base rather than an exhaustive account; proprietary, unpublished, non-English, and other non-academic practices may be underrepresented. We recruited participants for the main survey through GitHub and LinkedIn using purposive non-probability sampling (see Section 4.4.3), introducing possible self-selection and coverage bias. Although we targeted participants with relevant AI/ML, software engineering, robustness, governance, monitoring, and auditing backgrounds and collected profile information to characterise the sample, it is not statistically representative of the wider practitioner population. The findings may therefore transfer differently across stakeholder groups, domains, system types, and organisational settings. 8.2. Construct Validity Construct validity concerns whether the study adequately captures the concepts it investigates (Wohlin et al., 2012). Concepts such as robustness, auditing, monitoring, and assurance may be interpreted differently across research and professional communities, and the technology-agnostic formulation of the meta-requirements may have led respondents to interpret their operational meaning differently. Moreover, a five-point Likert rating captures agreement but does not establish shared interpretation, implementation feasibility, or practical effectiveness. We mitigated these risks by grounding the meta-requirements in source-traceable literature evidence (see Section 4.3), organising them into explicit audit categories, and piloting the survey for terminology, wording, presentation, and completion effort (see Section 4.4). Likert ratings were complemented by open-text feedback on ambiguity, overlap, feasibility, and missing issues. Nevertheless, some interpretive variation remains across heterogeneous professional and system contexts. 8.3. Internal Validity Internal validity concerns potential bias introduced by the research process (Wohlin et al., 2012). Study screening and the derivation of meta-requirements required researcher judgement, particularly for borderline studies, overlapping concepts, and abstraction decisions. We mitigated this through explicit eligibility criteria, independent screening by two researchers, Cohenโs ฮบ assessment before reconciliation (ฮบ=0.882ฮบ=0.882 for title/abstract and ฮบ=0.741ฮบ=0.741 for full text), consensus resolution of screening disagreements, source-to-requirement traceability, and review by a second researcher of extracted patterns, category assignments, and meta-requirement formulations, with remaining ambiguities resolved through discussion (see Sections 4.2 and 4.3). Nevertheless, other researchers may make different decisions for some studies or formulate particular meta-requirements differently. The practitioner phase may also be affected by response and interpretation bias. Rating 43 meta-requirements could cause fatigue or satisficing, while the desirable nature of robustness, governance, and auditability may encourage favourable responses. Pilot testing addressed clarity and completion effort, and open-text feedback on qualifications, implementation concerns, and challenges complemented the agreement ratings. Qualitative coding still involved researcher interpretation; despite structured coding and traceable coding and synthesis decisions (see Section 4.4.6), some comments may reasonably be interpreted or mapped differently. 8.4. Conclusion Validity Conclusion validity concerns whether the conclusions are supported by the analysed evidence (Wohlin et al., 2012). High practitioner agreement supports the literature-derived meta-requirements but does not establish that applying the meta-requirements improves audit quality, robustness outcomes, or inter-auditor consistency. Likewise, open-text comments capture how respondents interpreted and refined the meta-requirements, rather than prevalence in the wider practitioner population. We therefore analysed the ratings descriptively and integrated them with qualitative evidence rather than treating quantitative agreement as sufficient validation. We compared candidate additions against the existing meta-requirements to avoid isolated or overlapping requirements. This synthesis produced one new requirement, MR-8.4, which was not quantitatively rated because it emerged after the original 43 meta-requirements had been evaluated. Thus, the study provides content-oriented practitioner validation rather than evidence of operational effectiveness; further evaluation through real audit engagements and inter-auditor studies is required. 9. Conclusion and Future Work AI robustness auditing knowledge is distributed across multiple research domains, including AI, software engineering, and auditing. This study addressed this gap by deriving literature-grounded meta-requirements for AI robustness auditing and evaluating them with practitioners. The systematic literature review retained 74 studies and produced 43 literature-derived meta-requirements across 16 categories. The findings show that robustness auditing extends beyond model-level testing to scoping robustness claims, assessing available evidence and system components, documenting limitations, and continuing monitoring and reassessment. The practitioner survey with 128 participants showed broad support, with 81.20% of ratings indicating agreement or strong agreement and 41 of the 43 literature-derived requirements classified as strongly supported. Practitioner feedback nevertheless refined requirement definitions, scope, implementation conditions, stakeholder responsibilities, workflow integration, monitoring, and reporting. Integrating both evidence sources resulted in a final practitioner-informed set of 44 meta-requirements across 16 categories, including one additional requirement addressing external components and supply chain dependencies. The final set of meta-requirements organises the identified auditing concerns into 44 meta-requirements covering areas such as audit scope and responsibilities, data and model assessment, evidence, workflow integration, monitoring, reporting, and reassessment. Future work will operationalise the meta-requirements into a decision model for selecting appropriate requirements, evidence, tests, metrics, and assessment conditions for a given AI-enabled system and audit context, and subsequently integrate this model into a design support system for practical AI robustness auditing. Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. This research received funding from Deloitte, as disclosed in the Acknowledgments. Deloitte had no role in the study design, data collection, analysis, interpretation of results, or decision to submit the manuscript for publication. Acknowledgments This research has been supported by the Safeguard project (grant No. 20506020). It has also received funding from Deloitte. The authors thank all practitioners who participated in the validation survey and provided feedback on the proposed AI robustness auditing meta-requirements. We also thank colleagues and professional networks who helped distribute the pilot survey, as well as those who provided constructive comments on earlier versions of the manuscript. Ethics Approval The practitioner survey received ethical clearance from the Research Ethics and Data Management Committee of the Tilburg School of Humanities and Digital Sciences, Tilburg University (REDC 2026.04). We obtained informed consent from all survey participants before participation. Data Availability Statement For transparency and reproducibility, we provide a replication package11 1 https://doi.org/10.5281/zenodo.23267777 containing the review protocol, search strings, included-study and quality-assessment records, extraction and coding records, meta-requirement tables, survey instrument, practitioner-validation data, and analysis outputs. References Agarwal and Nene (2025) A. Agarwal and M. J. Nene Advancing trustworthy ai: a comparative evaluation of ai robustness toolboxes. SN Computer Science 6 (3), p. 234. Cited by: ยง1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.3, ยง5.12.1, ยง5.15.1, ยง5.16.1, ยง5.5.1, ยง5.6.1, ยง5.6.5, ยง7.8.3. Agresti (2010) A. Agresti Analysis of ordinal categorical data. 2 edition, John Wiley & Sons, Hoboken, NJ. Cited by: ยง4.4.5. Ajaykumar et al. (2021) G. Ajaykumar, M. Steele, and C. Huang A survey on end-user robot programming. ACM Computing Surveys (CSUR) 54 (8), p. 1โ36. Cited by: ยง4.2. Ala-Luopa et al. (2024) S. Ala-Luopa, T. Olsson, K. Vรคnรคnen, M. Hartikainen, and J. Makkonen Trusting intelligent automation in expert work: accounting practitionersโ experiences and perceptions. Computer Supported Cooperative Work (CSCW) 33 (4), p. 1343โ1371. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.11.1, ยง5.12.1, ยง5.4.1, ยง5.4.2, ยง5.6.3. Alzubaidi et al. (2023) L. Alzubaidi, A. Al-Sabaawi, J. Bai, A. Dukhan, A. H. Alkenani, A. Al-Asadi, H. A. Alwzwazy, M. Manoufali, M. A. Fadhel, A. S. Albahri, et al. Towards risk-free trustworthy artificial intelligence: significance and requirements. International Journal of Intelligent Systems 2023 (1), p. 4459198. Cited by: Table 7, Table 7, Table 7, ยง5.5.4, ยง5.8.1. Bajaj and Sangwan (2019) A. Bajaj and O. P. Sangwan A systematic literature review of test case prioritization using genetic algorithms. Ieee Access 7, p. 126355โ126375. Cited by: ยง4.2.1. Balendran et al. (2025) A. Balendran, C. Beji, F. Bouvier, O. Khalifa, T. Evgeniou, P. Ravaud, and R. Porcher A scoping review of robustness concepts for machine learning in healthcare. npj Digital Medicine 8 (1), p. 38. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.14.1, ยง5.14.2, ยง5.2.1, ยง5.3.1. Bayram and Ahmed (2025) F. Bayram and B. S. Ahmed Towards trustworthy machine learning in production: an overview of the robustness in mlops approach. ACM Computing Surveys 57 (5), p. 1โ35. Cited by: Table 1, ยง3.1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.10.2, ยง5.11.1, ยง5.12.2, ยง5.3.1, ยง5.3.2, ยง5.5.2, ยง5.5.3, ยง5.8.2, ยง5.8. Blood et al. (2023) J. C. Blood, N. W. Herbert, and M. R. Wayne Reliability assurance for ai systems. In 2023 Annual Reliability and Maintainability Symposium (RAMS), Orlando, FL, USA, p. 1โ6. External Links: Document Cited by: ยง3.1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.4, ยง5.10.1, ยง5.10.3, ยง5.11.1, ยง5.12.2, ยง5.3.1, ยง5.5.3, ยง5.8.1, ยง5.8.4, ยง7.7. Boufaied et al. (2026) C. Boufaied, T. Nguyen, and R. d. S. Santos Practitioner insights on fairness requirements in the ai development life cycle: an interview study. ACM Transactions on Software Engineering and Methodology. External Links: Document Cited by: ยง3.3, Table 2. Browning and Bustard (2024) J. W. Browning and J. Bustard A systematic literature review of entrepreneurial education in electrical, electronic, and computer engineering curricula. IEEE access 12, p. 7927โ7941. Cited by: ยง4.2.1. Butijn et al. (2020) B. Butijn, D. A. Tamburri, and W. van den Heuvel Blockchains: a systematic multivocal literature review. ACM Computing Surveys (CSUR) 53 (3), p. 1โ37. External Links: Document, Link Cited by: ยง4.2. Canedo et al. (2023) E. D. Canedo, I. N. Bandeira, A. T. S. Calazans, P. H. T. Costa, E. C. R. Canรงado, and R. Bonifรกcio Privacy requirements elicitation: a systematic literature review and perception analysis of it practitioners. Requirements Engineering 28 (2), p. 177โ194. Cited by: ยง4.4.3. Casper et al. (2024) S. Casper, C. Ezell, C. Siegmann, N. Kolt, T. L. Curtis, B. Bucknall, A. Haupt, K. Wei, J. Scheurer, M. Hobbhahn, et al. Black-box access is insufficient for rigorous ai audits. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA, p. 2254โ2272. External Links: Document Cited by: ยง2.2, ยง2.3, Table 1, ยง3.1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.3, ยง5.1.7, ยง5.1.8, ยง5.14.1, ยง5.2.3, ยง5.6.3, ยง5.6.4, ยง5.8.4, ยง5.9.1, ยง7.7. Chamola et al. (2023) V. Chamola, V. Hassija, A. R. Sulthana, D. Ghosh, D. Dhingra, and B. Sikdar A review of trustworthy and explainable artificial intelligence (xai). IEEe Access 11, p. 78994โ79015. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.6, ยง5.11.1, ยง5.6.3, ยง5.9.1. Chander et al. (2025) B. Chander, C. John, L. Warrier, and K. Gopalakrishnan Toward trustworthy artificial intelligence (tai) in the context of explainability and robustness. ACM Computing Surveys 57 (6), p. 1โ49. Cited by: ยง3.1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.7, ยง5.15.1, ยง5.2.1, ยง5.3.1, ยง5.4.2, ยง5.6.3, ยง7.6. Charmaz (2006) K. Charmaz Constructing grounded theory: a practical guide through qualitative analysis. sage. Cited by: ยง4.3, ยง4.3. Chen et al. (2010) L. Chen, M. A. Babar, and H. Zhang Towards an evidence-based understanding of electronic data sources. In 14th Int. Conf. on Evaluation and Assessment in Software Engineering (EASE), Limerick, Ireland, p. 1โ4. External Links: Document, Link Cited by: ยง4.2. Chen and Das (2023) P. Chen and P. Das AI maintenance: a robustness perspective. Computer 56 (2), p. 48โ56. Cited by: Table 1, Table 1, ยง3.1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.5, ยง5.1.8, ยง5.10.1, ยง5.10.2, ยง5.11.1, ยง5.15.1, ยง5.3.1, ยง5.5.1, ยง5.5.5, ยง5.7.1, ยง5.8.1, ยง5.9.1. Chen et al. (2025) Q. Chen, L. Tan, J. Tang, and X. Qu AI-enabled iot security: a survey on advances, challenges, and cross-domain collaborative frameworks. In Proceedings of the 2025 8th Int. Conf. on Computer Information Science and Artificial Intelligence, New York, NY, USA, p. 1615โ1621. External Links: Document Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.11.1, ยง5.13.1, ยง5.14.1, ยง5.3.2. Cooper et al. (2022) A. F. Cooper, E. Moss, B. Laufer, and H. Nissenbaum Accountability in an algorithmic society: relationality, responsibility, and robustness in machine learning. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA, p. 864โ876. External Links: Document Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.4, ยง5.1.6, ยง5.12.2, ยง5.2.4, ยง5.6.3. Costanza-Chock et al. (2022) S. Costanza-Chock, I. D. Raji, and J. Buolamwini Who audits the auditors? recommendations from a field scan of the algorithmic auditing ecosystem. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA, p. 1571โ1583. External Links: Document Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.6, ยง5.1.7, ยง5.1.8, ยง5.10.1, ยง5.10.2, ยง5.12.2, ยง5.2.1, ยง5.2.2, ยง5.2.3, ยง5.3.1, ยง5.4.1, ยง5.5.1, ยง5.5.2, ยง7.3, ยง7.6. De Stefano et al. (2022) M. De Stefano, F. Pecorelli, D. Di Nucci, F. Palomba, and A. De Lucia Software engineering for quantum programming: how far are we?. Journal of Systems and Software 190, p. 111326. Cited by: ยง4.4.3. Dev et al. (2025) J. Dev, N. B. Akhuseyinoglu, G. Kayas, B. Rashidi, and V. Garg Building guardrails in ai systems with threat modeling. Digital Government: Research and Practice 6 (1), p. 1โ18. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.3, ยง5.10.3, ยง5.13.1, ยง5.5.5, ยง5.6.1, ยง5.8.2, ยง5.8.3, ยง5.8.4, ยง7.7. Diamond et al. (2014) I. R. Diamond, R. C. Grant, B. M. Feldman, P. B. Pencharz, S. C. Ling, A. M. Moore, and P. W. Wales Defining consensus: a systematic review recommends methodologic criteria for reporting of delphi studies. Journal of clinical epidemiology 67 (4), p. 401โ409. Cited by: ยง4.4.5. Dรญaz-Rodrรญguez et al. (2023) N. Dรญaz-Rodrรญguez, J. Del Ser, M. Coeckelbergh, M. L. De Prado, E. Herrera-Viedma, and F. Herrera Connecting the dots in trustworthy artificial intelligence: from ai principles, ethics, and key requirements to responsible ai systems and regulation. Information Fusion 99, p. 101896. Cited by: ยง3.3, Table 2. Dressel and Farid (2018) J. Dressel and H. Farid The accuracy, fairness, and limits of predicting recidivism. Science advances 4 (1), p. eaao5580. Cited by: ยง1. European Parliament and Council of the European Union (2024) European Parliament and Council of the European Union Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence and amending regulations (ec). Note: Official Journal of the European Union, L 2024/1689Published 12 July 2024 External Links: Link Cited by: ยง1. Fahmideh et al. (2021) M. Fahmideh, A. Ahmad, A. Behnaz, J. Grundy, and W. Susilo Software engineering for internet of things: the practitionersโ perspective. IEEE Transactions on Software Engineering 48 (8), p. 2857โ2878. Cited by: ยง1, ยง4.3, ยง4.4.3, ยง4.4.5, ยง4.4.5, ยง4. Freiesleben and Grote (2023) T. Freiesleben and T. Grote Beyond generalization: a theory of robustness in machine learning. Synthese 202 (4), p. 109. Cited by: ยง1, ยง2.1, ยง2.3, Table 1, Table 1, Table 1, ยง3.1, ยง3.2, ยง3.3, Table 2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.2, ยง5.14.1, ยง5.16.1, ยง5.3.1, ยง5.5.1, ยง5.5.5, ยง5.6.4, ยง7.1, ยง7.3, ยง7.4. Gaber et al. (2024) M. G. Gaber, M. Ahmed, and H. Janicke Malware detection with artificial intelligence: a systematic literature review. ACM Computing Surveys 56 (6), p. 1โ33. Cited by: ยง4.2. Galster et al. (2013) M. Galster, D. Weyns, D. Tofan, B. Michalik, and P. Avgeriou Variability in software systemsโa systematic literature review. IEEE transactions on software engineering 40 (3), p. 282โ306. Cited by: ยง4.2.1. Garousi et al. (2017) V. Garousi, M. Felderer, and T. Hacaloฤlu Software test maturity assessment and test process improvement: a multivocal literature review. Information and Software Technology 85, p. 16โ42. External Links: Document, Link Cited by: ยง4. Garousi et al. (2019) V. Garousi, M. Felderer, and M. V. Mรคntylรค Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology 106, p. 101โ121. External Links: Document, Link Cited by: ยง4.2. Garousi and Felderer (2017) V. Garousi and M. Felderer Experience-based guidelines for data extraction in systematic reviews in software engineering. In Proceedings of the 21st Int. Conf. on Evaluation and Assessment in Software Engineering, Ankara, Turkey, p. 170โ179. External Links: Document, Link Cited by: ยง4. Ge et al. (2024) Y. Ge, S. Liu, Z. Fu, J. Tan, Z. Li, S. Xu, Y. Li, Y. Xian, and Y. Zhang A survey on trustworthy recommender systems. ACM Transactions on Recommender Systems 3 (2), p. 1โ68. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.2, ยง5.1.3, ยง5.1.8, ยง5.3.2, ยง5.5.2, ยง5.6.1, ยง5.6.3, ยง5.8.2. Geleta et al. (2023) M. Geleta, J. Xu, M. Loya, and J. Wang Maestro: a gamified platform for teaching ai robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, Washington, DC, USA, p. 15816โ15824. External Links: Document Cited by: Table 7, ยง5.4.2. Gregor and Jones (2007) S. Gregor and D. Jones The anatomy of a design theory. Journal of the Association for Information Systems 8 (5), p. 312โ335. External Links: Document Cited by: ยง2.3. Gupta et al. (2025) A. Gupta, T. A. Henzinger, K. Kueffner, K. Mallik, and D. Pape Monitoring robustness and individual fairness. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, p. 790โ801. Cited by: ยง3.1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.11.1, ยง5.12.2, ยง5.13.1, ยง5.14.1. Habbal et al. (2024) A. Habbal, M. K. Ali, and M. A. Abuzaraida Artificial intelligence trust, risk and security management (ai trism): frameworks, applications, challenges and future research directions. Expert Systems with Applications 240, p. 122442. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.4, ยง5.1.6, ยง5.4.2, ยง5.6.3, ยง5.9.1. Hรฅkansson et al. (2023) A. Hรฅkansson, Y. C. Dรผndar, and R. L. Hartung Towards robustness analysis for adaptive artificial intelligence in multi-autonomous agent systems. Procedia Computer Science 225, p. 4657โ4666. Cited by: Table 1, Table 7, Table 7, Table 7. Hancox-Li (2020) L. Hancox-Li Robustness in machine learning explanations: does it matter?. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, New York, NY, USA, p. 640โ647. External Links: Document Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.15.1, ยง5.15, ยง5.2.1, ยง5.6.3, ยง5.6.4. Hardy et al. (2025) A. F. Hardy, A. Reuel, K. Jafari Meimandi, L. Soder, A. Griffith, D. M. Asmar, S. Koyejo, M. S. Bernstein, and M. J. Kochenderfer More than marketing? on the information value of ai benchmarks for practitioners. In Proceedings of the 30th International Conference on Intelligent User Interfaces, New York, NY, USA, p. 1032โ1047. External Links: Document Cited by: Table 7, Table 7, Table 7, Table 7, ยง5.14.1, ยง5.5.3, ยง5.6.1. Heck et al. (2010) P. Heck, M. Klabbers, and M. van Eekelen A software product certification model. Software Quality Journal 18 (1), p. 37โ55. Cited by: ยง4.3. Heck and Zaidman (2014) P. Heck and A. Zaidman A quality framework for agile requirements: a practitionerโs perspective. Technical Report, Software Engineering Research Group, Delft University of Technology, Delft, The Netherlands. External Links: 1406.4692, Link Cited by: ยง4.3. Hendrycks and Dietterich (2019) D. Hendrycks and T. Dietterich Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261. Cited by: ยง1, ยง1, ยง7.1. Hevner et al. (2004) A. R. Hevner, S. T. March, J. Park, and S. Ram Design science in information systems research. MIS Quarterly 28 (1), p. 75โ106. Cited by: ยง1, ยง2.3, Table 1. Hornischer (2026) L. Hornischer Robustness and trustworthiness in ai: a no-go result from formal epistemology. Synthese 207 (1), p. 22. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.2, ยง5.1.5, ยง5.13.1, ยง5.2.4, ยง5.3.1, ยง5.5.1. Ihle et al. (2023) C. Ihle, D. Trautwein, M. Schubotz, N. Meuschke, and B. Gipp Incentive mechanisms in peer-to-peer networksโa systematic literature review. ACM Computing Surveys 55 (14s), p. 1โ69. Cited by: ยง4.2. Islam et al. (2019) C. Islam, M. A. Babar, and S. Nepal A multi-vocal review of security orchestration. ACM Computing Surveys 52 (2), p. 1โ45. External Links: Document, Link Cited by: ยง4.2. Javed et al. (2024) H. Javed, S. El-Sappagh, and T. Abuhmed Robustness in deep learning models for medical diagnostics: security and adversarial challenges towards robust ai applications. Artificial Intelligence Review 58 (1), p. 12. Cited by: ยง3.1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.3, ยง5.1.5, ยง5.13.1, ยง5.14.1, ยง5.16.1, ยง5.3.1, ยง5.3.2, ยง5.5.1, ยง5.5.5, ยง5.6.3, ยง5.6.5, ยง5.7.1. Kaur et al. (2022) D. Kaur, S. Uslu, K. J. Rittichier, and A. Durresi Trustworthy artificial intelligence: a review. ACM computing surveys (CSUR) 55 (2), p. 1โ38. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7. Kaya et al. (2026) O. Kaya, A. S. Shah, M. A. Karabulut, S. N. Karahan, M. S. Osmanca, and N. Acฤฑr Explainable artificial intelligence (xai): concepts, applications, challenges, and future perspectives. IEEE Access. Cited by: ยง3.1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.11.1, ยง5.12.2, ยง5.13.1, ยง5.6.1, ยง5.6.3. Kemmerzell et al. (2025) N. Kemmerzell, A. Schreiner, H. Khalid, M. Schalk, and L. Bordoli Towards a better understanding of evaluating trustworthiness in ai systems. ACM Computing Surveys 57 (9), p. 1โ38. Cited by: Table 1, ยง3.1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.12.2, ยง5.14.2, ยง5.15.1, ยง5.5.4, ยง5.6.3, ยง5.6.4. Khinvasara et al. (2024) T. Khinvasara, A. Shankar, and C. Wong Robustness and reliability testing in healthcare using artificial intelligence. Asian Journal of Research in Computer Science 17 (7), p. 103โ118. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.3, ยง5.1.4, ยง5.1.5, ยง5.10.1, ยง5.11.1, ยง5.13.1, ยง5.14.1, ยง5.4.1, ยง5.5.4, ยง5.6.3, ยง5.8.3. Kitchenham and Pfleeger (2002a) B. A. Kitchenham and S. L. Pfleeger Principles of survey research: part 2: designing a survey. ACM SIGSOFT Software Engineering Notes 27 (1), p. 18โ20. Cited by: ยง4.4. Kitchenham and Pfleeger (2002b) B. A. Kitchenham and S. L. Pfleeger Principles of survey research: part 3: constructing a survey instrument. ACM SIGSOFT Software Engineering Notes 27 (2), p. 20โ24. Cited by: ยง4.4. Kitchenham and Pfleeger (2008) B. A. Kitchenham and S. L. Pfleeger Personal opinion surveys. In Guide to Advanced Empirical Software Engineering, F. Shull, J. Singer, and D. I. K. Sjรธberg (Eds.), p. 63โ92. Cited by: ยง4.4.3. Kitchenham et al. (2007) B. Kitchenham S. Charters et al. Guidelines for performing systematic literature reviews in software engineering. UK. Cited by: ยง4.2, ยง4. Kokina et al. (2025) J. Kokina, S. Blanchette, T. H. Davenport, and D. Pachamanova Challenges and opportunities for artificial intelligence in auditing: evidence from the field. International Journal of Accounting Information Systems 56, p. 100734. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.3, ยง5.1.4, ยง5.1.5, ยง5.1.8, ยง5.12.1, ยง5.13.1, ยง5.16.1, ยง5.4.1, ยง5.4.2, ยง5.5.1, ยง5.5.2, ยง5.5.4, ยง5.6.3, ยง5.8.3, ยง5.9.1. Koshiyama et al. (2024) A. Koshiyama, E. Kazim, P. Treleaven, P. Rai, and L. Szpruch Towards algorithm auditing: managing legal, ethical and technological risks of ai, ml and associated algorithms. Royal Society Open Science 11 (5). Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.7, ยง5.10.1, ยง5.10.2, ยง5.11.1, ยง5.12.2, ยง5.3.1, ยง5.3.2, ยง5.4.2, ยง5.6.3, ยง5.8.3. Kou et al. (2024) J. Kou, X. Lu, A. Yin, W. Zhang, X. Fang, Y. Li, T. Pan, L. Zhang, G. Cheng, H. Yuan, et al. Beyond isolated fixes: a comprehensive survey on hallucination mitigation with a three-dimensional taxonomy and integrative framework. In 2024 7th International Conference on Universal Village (UV), p. 1โ60. Cited by: Table 7, Table 7, Table 7, ยง5.13.1, ยง5.9.1. Kuutti et al. (2020) S. Kuutti, R. Bowden, Y. Jin, P. Barber, and S. Fallah A survey of deep learning applications to autonomous vehicle control. IEEE Transactions on Intelligent Transportation Systems 22 (2), p. 712โ733. Cited by: ยง1. Laux et al. (2024) J. Laux, S. Wachter, and B. Mittelstadt Trustworthy artificial intelligence and the european union ai act: on the conflation of trustworthiness and acceptability of risk. Regulation & Governance 18 (1), p. 3โ32. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.6, ยง5.10.1, ยง5.12.2, ยง5.2.3, ยง5.4.1, ยง5.8.1, ยง7.7. Lee et al. (2023) E. Lee, Y. Lee, and T. Lee Adversarial attack-based robustness evaluation for trustworthy ai.. Computer Systems Science & Engineering 47 (2). Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.6.1, ยง5.6.4. Lee et al. (2011) J. S. Lee, J. Pries-Heje, and R. Baskerville Theorizing in design science research. In International conference on design science research in information systems, p. 1โ16. Cited by: ยง2.3. Lefcourt et al. (2022) S. Lefcourt, N. Gordon, H. Wong, and G. Falco Robustness assurance quotient: demonstrating context matters for ai performance and ml security. In 2022 IEEE International Conference on Assured Autonomy (ICAA), p. 21โ27. Cited by: Table 7, Table 7, Table 7, Table 7, ยง5.14.1, ยง5.5.2. Lessmann et al. (2015) S. Lessmann, B. Baesens, H. Seow, and L. C. Thomas Benchmarking state-of-the-art classification algorithms for credit scoring: an update of research. European journal of operational research 247 (1), p. 124โ136. Cited by: ยง1. Li et al. (2023) B. Li, P. Qi, B. Liu, S. Di, J. Liu, J. Pei, J. Yi, and B. Zhou Trustworthy ai: from principles to practices. ACM Computing Surveys 55 (9), p. 1โ46. Cited by: ยง1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.2, ยง5.1.5, ยง5.12.2, ยง5.14.2, ยง5.2.2, ยง5.3.1, ยง5.3.2, ยง5.5.1, ยง5.5.4, ยง5.6.3, ยง5.8.1. Li and Goel (2025) Y. Li and S. Goel Artificial intelligence auditability and auditor readiness for auditing artificial intelligence systems. International Journal of Accounting Information Systems 56, p. 100739. Cited by: ยง3.1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.6, ยง5.1.7, ยง5.10.1, ยง5.11.1, ยง5.11.2, ยง5.12.1, ยง5.15.1, ยง5.2.1, ยง5.2.3, ยง5.2.4, ยง5.4.1, ยง5.6.3, ยง7.6, ยง7.8.2, ยง7.8. Linรฅker et al. (2015) J. Linรฅker, S. M. Sulaman, R. Maiani de Mello, and M. Hรถst Guidelines for conducting surveys in software engineering. Technical report Department of Computer Science, Lund University. Cited by: ยง4.4. Liu et al. (2022) H. Liu, Y. Wang, W. Fan, X. Liu, Y. Li, S. Jain, Y. Liu, A. Jain, and J. Tang Trustworthy ai: a computational perspective. ACM Transactions on Intelligent Systems and Technology 14 (1), p. 1โ59. Cited by: Table 1, Table 1, Table 1, Table 1, ยง3.1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.2, ยง5.1.5, ยง5.1.6, ยง5.1.8, ยง5.16.1, ยง5.2.1, ยง5.2.3, ยง5.3.1, ยง5.3.2, ยง5.6.3, ยง5.7.1. Madaio et al. (2022) M. Madaio, L. Egede, H. Subramonyam, J. Wortman Vaughan, and H. Wallach Assessing the fairness of ai systems: ai practitionersโ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6 (CSCW1), p. 1โ26. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.5, ยง5.1.8, ยง5.10.1, ยง5.13.1, ยง5.2.2, ยง5.2.4, ยง5.5.1, ยง5.6.2, ยง7.5. Mรคntymรคki et al. (2023) M. Mรคntymรคki, M. Minkkinen, M. P. Zimmer, T. Birkstedt, and M. Viljanen Designing an ai governance framework: from research-based premises to meta-requirements. Cited by: ยง2.3, Table 1, ยง3.3, Table 2. Martรญnez-Fernรกndez et al. (2022) S. Martรญnez-Fernรกndez, J. Bogner, X. Franch, M. Oriol, J. Siebert, and A. Trendowicz Software engineering for ai-based systems: a survey. ACM Transactions on Software Engineering and Methodology (TOSEM) 31 (2), p. 1โ59. Cited by: ยง1, Table 1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.11.1, ยง5.13.1, ยง5.5.3, ยง5.6.1, ยง5.8.1, ยง5.8.3, ยง5.8.4, ยง7.7. McCormack et al. (2025) L. McCormack, M. Bendechache, D. Lewis, and D. Huyskes Trust and transparency in ai: industry voices on data, ethics, and compliance. AI & SOCIETY, p. 1โ29. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.4, ยง5.1.6, ยง5.10.1, ยง5.11.1, ยง5.12.2, ยง5.13.1, ยง5.15.1, ยง5.2.1, ยง5.2.3, ยง5.2.4, ยง5.4.1, ยง5.5.3, ยง5.5.4, ยง5.6.3, ยง5.9.1. Mehrabi et al. (1908) N. Mehrabi, F. Morstatter, K. Lerman, and A. Galstyan A survey on bias and fairness in machine learning. arxiv. Ithaca, NY: arXiv (Cornell University). Retrieved from https://arxiv. org/abs. Cited by: Table 1, Table 1. Methnani et al. (2024) L. Methnani, M. Chiou, V. Dignum, and A. Theodorou Whoโs in charge here? a survey on trustworthy ai in variable autonomy robotic systems. ACM computing surveys 56 (7), p. 1โ32. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.11.1, ยง5.11.2, ยง5.12.1, ยง5.15.1, ยง5.6.3. Minkkinen et al. (2022) M. Minkkinen, J. Laine, and M. Mรคntymรคki Continuous auditing of artificial intelligence: a conceptualization and assessment of tools and frameworks. Digital Society 1 (3), p. 21. Cited by: ยง1, Table 1. Mohseni et al. (2022) S. Mohseni, H. Wang, C. Xiao, Z. Yu, Z. Wang, and J. Yadawa Taxonomy of machine learning safety: a survey and primer. ACM Computing Surveys 55 (8), p. 1โ38. Cited by: ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.2, ยง5.10.3, ยง5.11.1, ยง5.13.1, ยง5.14.1, ยง5.5.3, ยง5.6.3, ยง5.8.1. Mรถkander et al. (2024) J. Mรถkander, J. Schuett, H. R. Kirk, and L. Floridi Auditing large language models: a three-layered approach. AI and Ethics 4 (4), p. 1085โ1115. Cited by: ยง3.1, ยง3.2, ยง3.3, Table 2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.6, ยง5.10.1, ยง5.10.2, ยง5.12.2, ยง5.13.1, ยง5.14.1, ยง5.2.1, ยง5.2.3, ยง5.3.1, ยง5.4.1, ยง5.5.4, ยง5.6.1, ยง5.8.4, ยง7.1, ยง7.2, ยง7.4, ยง7.5, ยง7.7, ยง7.8.1, ยง7.9.1, ยง7.9.2. Mรถkander (2023) J. Mรถkander Auditing of ai: legal, ethical and technical approaches. Digital Society 2 (3), p. 49. Cited by: ยง1, Table 1, ยง7.1. Momcilovic et al. (2026) T. B. Momcilovic, B. Buesser, G. Zizzo, M. Purcell, and D. Balta Assuring eu ai act compliance and adversarial robustness of llms. Shaping the Digital Future Through Innovation and Practice, p. 355. Cited by: ยง2.2, Table 1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.3, ยง5.10.1, ยง5.10.2, ยง5.12.2. Mรผller et al. (2025) I. Mรผller, K. Dill, and H. Mihaljeviฤ Practical perspectives on fairness in technology-aided personnel selection. Proceedings of the ACM on Human-Computer Interaction 9 (7), p. 1โ34. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.7, ยง5.12.1, ยง5.14.1, ยง5.4.1, ยง5.4.2. Nanda et al. (2021) V. Nanda, S. Dooley, S. Singla, S. Feizi, and J. P. Dickerson Fairness through robustness: investigating robustness disparity in deep learning. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, New York, NY, USA, p. 466โ477. External Links: Document Cited by: Table 1, ยง3.1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.8, ยง5.13.1, ยง5.2.1, ยง5.3.2, ยง5.5.1. Narula et al. (2025) S. Narula, M. Ghasemigol, J. Carnerero-Cano, A. Minnich, E. Lupu, and D. Takabi Exploring research and tools in ai security: a systematic mapping study. IEEE Access 13, p. 84057โ84080. External Links: Document Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.4, ยง5.11.1, ยง5.3.1, ยง5.5.3, ยง5.8.2. Nekrasov et al. (2026) R. Nekrasov, S. Fossati, I. Kumara, D. A. Tamburri, and W. van den Heuvel IaC generation with LLMs: an error taxonomy on configuration knowledge injection. ACM Transactions on Software Engineering and Methodology. External Links: Document Cited by: ยง4.3, ยง4.3. Nocera et al. (2025) S. Nocera, M. D. Penta, F. Ahmed, S. Romano, and G. Scanniello What we know about aiboms: results from a multivocal literature review on artificial intelligence bill of materials. ACM Transactions on Software Engineering and Methodology. Cited by: ยง5.8.4, ยง7.7. Nolte et al. (2025) H. Nolte, M. Rateike, and M. Finck Robustness and cybersecurity in the eu artificial intelligence act. In Proc. of the 2025 ACM Conf. on Fairness, Accountability, and Transparency, New York, NY, USA, p. 283โ295. External Links: Document Cited by: ยง1, ยง2.1, Table 1, Table 1, Table 1, ยง3.3, Table 2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.5, ยง5.1.6, ยง5.11.1, ยง5.12.1, ยง5.2.1, ยง5.2.2, ยง5.4.1, ยง5.5.1, ยง5.6.1, ยง5.6.2, ยง7.2, ยง7.8.4. Oala et al. (2020) L. Oala, J. Fehr, L. Gilli, P. Balachandran, A. W. Leite, S. Calderon-Ramirez, D. X. Li, C. Matek, A. Shroff, F. Kherif, B. Sanguinetti, and T. Wiegand ML4H auditing: from paper to practice. In Proceedings of the Machine Learning for Health NeurIPS Workshop, Proceedings of Machine Learning Research, Vol. 136, p. 280โ317. Cited by: ยง2.1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.3, ยง5.1.5, ยง5.1.8, ยง5.12.1, ยง5.12.2, ยง5.14.1, ยง5.2.2, ยง5.5.1, ยง5.5.2, ยง5.5.4, ยง5.6.1, ยง5.6.2, ยง5.6.3, ยง5.9.1. Ojewale et al. (2025) V. Ojewale, R. Steed, B. Vecchione, A. Birhane, and I. D. Raji Towards ai accountability infrastructure: gaps and opportunities in ai audit tooling. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, p. 1โ29. External Links: Document Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.3, ยง5.1.7, ยง5.1.8, ยง5.11.1, ยง5.12.2, ยง5.15.1, ยง5.15, ยง5.2.3. Olivero et al. (2022) M. A. Olivero, A. Bertolino, F. J. Dominguez-Mayo, I. Matteucci, and M. J. Escalona A delphi study to recognize and assess systems of systems vulnerabilities. Information and Software Technology 146, p. 106874. Cited by: ยง4.4.5. Ouyang et al. (2021) T. Ouyang, Y. Isobe, V. S. Marco, J. Ogata, Y. Seo, and Y. Oiwa AI robustness analysis with consideration of corner cases. In 2021 IEEE International Conference on Artificial Intelligence Testing (AITest), p. 29โ36. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.12.2, ยง5.14.1, ยง5.5.3, ยง5.6.1, ยง5.6.3. Palazzolo et al. (2024) P. Palazzolo, B. Stahl, and H. Webb Measurable trust: the key to unlocking user confidence in black-box ai. In Proceedings of the Second International Symposium on Trustworthy Autonomous Systems, p. 1โ7. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.12.2, ยง5.16.1, ยง5.3.2, ยง5.5.5. Pan et al. (2023) C. Pan, J. You, and Y. Gao AI software reliability: concepts and related domains. In 2023 2nd International Conference on Artificial Intelligence and Intelligent Information Processing (AIIIP), p. 287โ292. Cited by: Table 7, Table 7, Table 7, ยง5.3.1, ยง5.6.3. Petersen et al. (2022) E. Petersen, Y. Potdevin, P. Mohammadi, et al. Responsible and regulatory conform machine learning for medicine: a survey of challenges and solutions. IEEE access 10, p. 58375โ58418. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.10.2, ยง5.12.2, ยง5.14.1, ยง5.6.1, ยง5.6.4. Rabiser et al. (2010) R. Rabiser, P. Grรผnbacher, and D. Dhungana Requirements for product derivation support: results from a systematic literature review and an expert survey. Information and Software Technology 52 (3), p. 324โ346. Cited by: ยง1, ยง4.3, ยง4. Radanliev et al. (2026) P. Radanliev, O. Santos, C. Maple, and K. Atefi Operationalising artificial intelligence bills of materials for verifiable ai provenance and lifecycle assurance. Frontiers in Computer Science 8, p. 1735919. Cited by: ยง5.8.4, ยง7.7. Raji et al. (2020) I. D. Raji, A. Smart, R. N. White, M. Mitchell, T. Gebru, B. Hutchinson, J. Smith-Loud, D. Theron, and P. Barnes Closing the ai accountability gap: defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency, p. 33โ44. Cited by: ยง1, ยง2.2, ยง2.3, Table 1, Table 1, ยง3.1, ยง3.2, ยง3.3, Table 2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.3, ยง5.1.5, ยง5.1.6, ยง5.1.7, ยง5.10.1, ยง5.10.3, ยง5.11.1, ยง5.11.2, ยง5.12.1, ยง5.12.2, ยง5.2.1, ยง5.2.2, ยง5.2.3, ยง5.2.4, ยง5.5.1, ยง5.5.2, ยง5.6.2, ยง7.1, ยง7.6, ยง7.8.1, ยง7.8.2, ยง7.8.3, ยง7.9.1, ยง7.9.2. Rajpurkar et al. (2022) P. Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol AI in health and medicine. Nature medicine 28 (1), p. 31โ38. Cited by: ยง1. Robertson et al. (2023) S. Robertson, Z. J. Wang, D. Moritz, M. B. Kery, and F. Hohman Angler: helping machine translation practitioners prioritize model improvements. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, p. 1โ20. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.10.2, ยง5.11.1, ยง5.12.1, ยง5.2.1, ยง5.6.1, ยง5.6.4. Roy (2024) P. Roy Enhancing real-world robustness in ai: challenges and solutions. J. Recent Trends Comput. Sci. Eng 12 (1), p. 34โ49. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.14.1, ยง5.5.4. Saeed and Alsharidah (2024) M. M. Saeed and M. Alsharidah Security, privacy, and robustness for trustworthy ai systems: a review. Computers and Electrical Engineering 119, p. 109643. Cited by: Table 7, Table 7, Table 7, Table 7, ยง5.3.1. Sรกnchez et al. (2022) A. B. Sรกnchez, P. Delgado-Pรฉrez, I. Medina-Bulo, and S. Segura Mutation testing in the wild: findings from github. Empirical Software Engineering 27 (6), p. 132. Cited by: ยง4.3. Saqib et al. (2024) M. Saqib, S. Mahdavifar, B. C. Fung, and P. Charland A comprehensive analysis of explainable ai for malware hunting. ACM Computing Surveys 56 (12), p. 1โ40. Cited by: Table 7, Table 7, Table 7, Table 7, ยง5.13.1, ยง5.3.1, ยง5.6.3. Schmelczer and Visser (2023) A. Schmelczer and J. Visser Trustworthy and robust ai deployment by design: a framework to inject best practice support into ai deployment pipelines. In 2023 IEEE/ACM 2nd Int. Conf. on AI Engineering-Software Engineering for AI (CAIN), p. 127โ138. Cited by: Table 7, Table 7, ยง5.11.1, ยง5.11.2. Shafique et al. (2020) M. Shafique, M. Naseer, T. Theocharides, C. Kyrkou, O. Mutlu, L. Orosa, and J. Choi Robust machine learning systems: challenges, current trends, perspectives, and the road ahead. IEEE Design & Test 37 (2), p. 30โ57. Cited by: ยง3.1. Silva and Najafirad (2020) S. H. Silva and P. Najafirad Opportunities and challenges in deep learning adversarial robustness: a survey. arXiv preprint arXiv:2007.00753. Cited by: ยง3.1. Silva and De Franรงa (2023) S. Silva and B. B. N. De Franรงa A case study on data science processes in an academia-industry collaboration. In Proceedings of the XXII Brazilian Symposium on Software Quality, p. 1โ10. Cited by: Table 1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.4.1, ยง5.5.3, ยง5.8.1, ยง5.8. Simbeck (2024) K. Simbeck They shall be fair, transparent, and robust: auditing learning analytics systems. AI and Ethics 4 (2), p. 555โ571. Cited by: ยง2.2, ยง2.3, Table 1, Table 1, Table 1, ยง3.3, Table 2, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.7, ยง5.10.1, ยง5.12.2, ยง5.3.2, ยง7.5, ยง7.8.2. Smuha (2025) N. A. Smuha Regulation 2024/1689 of the eur. parl. & council of june 13, 2024 (eu artificial intelligence act). International Legal Materials 64 (5), p. 1234โ1381. Cited by: ยง1. Tabassi (2023) E. Tabassi Artificial intelligence risk management framework (AI RMF 1.0). Technical report Technical Report NIST AI 100-1, National Institute of Standards and Technology, Gaithersburg, MD. External Links: Document, Link Cited by: ยง1. Taesiri et al. (2022) M. R. Taesiri, G. Nguyen, and A. Nguyen Visual correspondence-based explanations improve ai robustness and human-ai team accuracy. Advances in Neural Information Processing Systems 35, p. 34287โ34301. Cited by: Table 7, Table 7, Table 7, ยง5.6.3. Tang et al. (2026) Q. Tang, J. Qian, X. Du, S. Wang, and H. Liu Advancing adversarial and llm robustness in trustworthy ai: a comprehensive survey. Artificial Intelligence Review. Cited by: ยง3.2, Table 7, Table 7, Table 7, Table 7, ยง5.3.1, ยง5.6.3, ยง5.8.1. Taori et al. (2020) R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems 33, p. 18583โ18599. Cited by: ยง1, ยง1, ยง7.1. Thelisson and Verma (2024) E. Thelisson and H. Verma Conformity assessment under the eu ai act general approach. AI and Ethics 4 (1), p. 113โ121. Cited by: ยง1, ยง2.3, ยง3.2, ยง3.3, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.3, ยง5.1.4, ยง5.1.5, ยง5.1.6, ยง5.1.7, ยง5.1.8, ยง5.10.1, ยง5.11.1, ยง5.11.2, ยง5.13.1, ยง5.14.2, ยง5.2.2, ยง5.2.3, ยง5.5.2, ยง5.6.2, ยง5.8.3, ยง5.9.1, ยง7.3, ยง7.4, ยง7.5, ยง7.7, ยง7.8.4, ยง7.8. Tocchetti et al. (2025) A. Tocchetti, L. Corti, A. Balayn, M. Yurrita, P. Lippmann, M. Brambilla, and J. Yang AI robustness: a human-centered perspective on technological challenges and opportunities. ACM Computing Surveys 57 (6), p. 1โ38. Cited by: ยง1, ยง1, ยง2.1, ยง2.3, Table 1, ยง3.1, ยง3.3, Table 2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.3, ยง5.1.5, ยง5.12.1, ยง5.13.1, ยง5.14.1, ยง5.2.1, ยง5.3.1, ยง5.3.2, ยง5.5.1, ยง5.5.5, ยง5.6.1, ยง5.6.3, ยง5.6.5, ยง5.7.1, ยง5.8.3, ยง5.8, ยง5.9.1. Tsingenopoulos et al. (2025) I. Tsingenopoulos, V. Rimmer, D. Preuveneers, F. Pierazzi, L. Cavallaro, and W. Joosen The adaptive arms race: redefining robustness in ai security. In 2025 28th International Symposium on Research in Attacks, Intrusions and Defenses (RAID), p. 611โ628. Cited by: Table 7, Table 7, Table 7, Table 7, ยง5.12.2, ยง5.6.1. Ustunboyacioglu et al. (2024) I. Ustunboyacioglu, I. Kumara, D. Di Nucci, D. A. Tamburri, and W. Van Den Heuvel Data quality assessment in the wild: findings from github. In Proc. of the 28th Int. Conf. on Evaluation and Assessment in Software Engineering, p. 120โ129. Cited by: ยง4.3. Vadlapati (2026) P. Vadlapati AI reliability gap: why large language models fail in safety-critical systems. Available at SSRN 6349038. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.2, ยง5.11.1, ยง5.12.2, ยง5.16.1, ยง5.3.1, ยง5.3.2, ยง5.4.1, ยง5.6.1. van der Linden et al. (2019) D. van der Linden, I. Hadar, and A. Zamansky What practitioners really want: requirements for visual notations in conceptual modeling. Software & Systems Modeling 18 (3), p. 1813โ1831. Cited by: ยง4.4.2, ยง4.4.3, ยง4.4.5, ยง4.4.5, ยง4.4.6, ยง4.4. Vogelsang et al. (2020) A. Vogelsang, J. Eckhardt, D. Mendez, and M. Berger Views on quality requirements in academia and practice: commonalities, differences, and context-dependent grey areas. Information and Software Technology 121, p. 106253. Cited by: ยง4.4.5. von Zahn et al. (2025) M. von Zahn, J. Zacharias, M. Lowin, J. Chen, and O. Hinz Navigating ai conformity: a design framework to assess fairness, explainability, and performance. Electronic markets 35 (1), p. 24. Cited by: ยง3.3, Table 2, ยง4.4.6. Walls et al. (1992) J. G. Walls, G. R. Widmeyer, and O. A. El Sawy Building an information system design theory for vigilant EIS. Information Systems Research 3 (1), p. 36โ59. Cited by: ยง1, ยง2.3, Table 1. Wang et al. (2026) J. Wang, J. Ai, M. Lu, J. Zhao, H. Su, D. Yu, Y. Zhang, J. Zhu, and J. Liu A survey of neural network robustness assessment in image recognition. ACM Computing Surveys 58 (12). External Links: Document Cited by: ยง2.1, ยง2.2, Table 1, Table 1, Table 1, Table 1, Table 1, Table 1, ยง3.1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.10.1, ยง5.13.1, ยง5.14.1, ยง5.3.1, ยง5.6.1. Waseem et al. (2021) M. Waseem, P. Liang, M. Shahin, A. Di Salle, and G. Mรกrquez Design, monitoring, and testing of microservices systems: the practitionersโ perspective. Journal of Systems and Software 182, p. 111061. Cited by: ยง1, ยง4.4.2, ยง4.4.3, ยง4.4.4, ยง4. Wei and Liu (2025) W. Wei and L. Liu Trustworthy distributed ai systems: robustness, privacy, and governance. ACM Computing Surveys 57 (6), p. 1โ42. Cited by: Table 1, Table 1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.2, ยง5.1.5, ยง5.13.1, ยง5.3.1, ยง5.5.1, ยง5.5.5, ยง5.6.5, ยง5.8.2. Wohlin et al. (2012) C. Wohlin, P. Runeson, M. Hรถst, M. C. Ohlsson, B. Regnell, A. Wesslรฉn, et al. Experimentation in software engineering. Vol. 236, Springer, Karlskrona, Sweden. Cited by: ยง8.1, ยง8.2, ยง8.3, ยง8.4, ยง8. Xiong et al. (2022) P. Xiong, S. Buffett, S. Iqbal, P. Lamontagne, M. Mamun, and H. Molyneaux Towards a robust and trustworthy machine learning system development: an engineering perspective. Journal of Information Security and Applications 65, p. 103121. Cited by: ยง3.2, Table 7, Table 7, Table 7, Table 7, ยง5.1.4, ยง5.12.1, ยง5.4.1. Yang et al. (2020) Y. Yang, C. Rashtchian, H. Zhang, R. R. Salakhutdinov, and K. Chaudhuri A closer look at accuracy vs. robustness. Advances in neural information processing systems 33, p. 8588โ8601. Cited by: ยง1, ยง1, ยง3.1, ยง3.2, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.3, ยง5.16.1, ยง5.3.2, ยง5.5.1, ยง5.5.3, ยง5.6.1, ยง5.6.5. Yu et al. (2026) G. Yu, G. Tan, H. Huang, Z. Zhang, P. Chen, R. Natella, Z. Zheng, and M. R. Lyu A survey on failure analysis and fault injection in ai systems. ACM Transactions on Software Engineering and Methodology 35 (1), p. 1โ42. Cited by: ยง2.1, Table 1, ยง3.1, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.1.1, ยง5.1.2, ยง5.1.3, ยง5.1.5, ยง5.10.2, ยง5.10.3, ยง5.12.2, ยง5.5.1, ยง5.7.1, ยง5.8.4, ยง7.2, ยง7.7. Zacharias et al. (2022) J. Zacharias, M. von Zahn, J. Chen, and O. Hinz Designing a feature selection method based on explainable artificial intelligence. Electronic Markets 32 (4), p. 2159โ2184. Cited by: ยง1, ยง2.3, Table 1, ยง3.3, Table 2. Zahid et al. (2025) I. A. Zahid, S. Garfan, M. Chyad, A. Albahri, O. Albahri, A. Alamoodi, M. Deveci, R. Z. Homod, and L. Alzubaidi Explainability, robustness, and fairness in user-centric intelligent systems: a systematic review. IEEE Transactions on Emerging Topics in Computational Intelligence. Cited by: Table 7, Table 7, Table 7, Table 7, Table 7, ยง5.15, ยง5.2.1, ยง5.3.1, ยง5.4.2, ยง5.6.3. Zellmer et al. (2026) P. Zellmer, J. Krรผger, and T. Leich Decision making for automotive-platform engineering: a mixed-methods study on practitionersโ requirements and criteria. Journal of Systems and Software 233, p. 112670. Cited by: ยง4.4.