Paper deep dive

Coverage-Driven Verification for Safety-by-Design in AI-Based Collision Avoidance Systems

Thomas Stefani, Johann Maximilian Christensen, Elena Hoemann, Frank Köster, Sven Hallerbach

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 66

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/24/2026, 5:31:31 AM

Summary

This paper presents a coverage-driven verification method for AI-based collision avoidance systems (HCAS and VCAS) to ensure compliance with EASA safety standards. The authors propose a two-step process assessing Operational Design Domain (ODD) representativeness and completeness. They evaluate statistical metrics, finding the chi-squared test unsuitable for large datasets, and recommend Kullback-Leibler (KL) divergence and Cramér's V for assessing representativeness against target distributions derived from real-world, simulation, or expert data.

Entities (10)

EASA · organization · 98%ODD · concept · 98%Cramér's V · metric · 97%Kullback-Leibler Divergence · metric · 97%Chi-squared test · metric · 96%DLR · organization · 95%HCAS · system · 95%VCAS · system · 95%FlightGear · software · 90%pyCASX · software · 90%

Relation Signals (8)

Chi-squared test → foundunsuitablefor → large data sets

confidence 95% · the chi-squared goodness-of-fit test is examined and found unsuitable for the large data sets arising in this setting

EASA → requires → ODD representativeness

confidence 95% · EASA emphasizes the need to demonstrate the representativeness and completeness of the Operational Design Domain (ODD)

Cramér's V → usedfor → representativeness assessment

confidence 95% · leading to the adoption of the Kullback--Leibler divergence and Cramér's V for the representativeness assessment

Kullback-Leibler Divergence → usedfor → representativeness assessment

confidence 95% · leading to the adoption of the Kullback--Leibler divergence and Cramér's V for the representativeness assessment

HCAS → evaluatedin → FlightGear

confidence 90% · both vertical and horizontal advisories can be evaluated within the flight simulator FlightGear

VCAS → evaluatedin → FlightGear

confidence 90% · both vertical and horizontal advisories can be evaluated within the flight simulator FlightGear

VCAS → uses → pyCASX

confidence 90% · integrating HCAS and VCAS into the generic open-source Python library pyCASX

HCAS → uses →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Artificial Intelligence (AI) offers significant potential for future aviation systems; however, its integration into safety-critical applications requires compliance with the aviation sector's stringent safety standards. For AI and Machine Learning (ML)-based systems, the European Union Aviation Safety Agency (EASA) emphasizes the need to demonstrate the representativeness and completeness of the Operational Design Domain (ODD) and the associated data distributions used during development and verification. Despite this requirement, a structured engineering process for defining target distributions and evaluating representativeness within ODDs remains largely unexplored. This work presents a method for representativeness assessment of AI/ML constituent ODDs in the context of aviation safety assurance. Starting from the methodical identification of suitable target distributions, a process flow is proposed that guides developers from ODD definition and parameter distribution modeling to the quantitative assessment and interpretation of coverage results with respect to EASA's learning assurance objectives. As quantitative measures, the chi-squared goodness-of-fit test is examined and found unsuitable for the large data sets arising in this setting, leading to the adoption of the Kullback--Leibler divergence and Cramér's $V$ for the representativeness assessment. The method is demonstrated using the example of AI-based airborne collision avoidance, employing experimental data from previous Horizontal Collision Avoidance System (HCAS) and Vertical Collision Avoidance System (VCAS) simulations. The results illustrate how statistical distribution comparison methods can support the assessment of representativeness for safety-critical AI applications and contribute toward a systematic Safety-by-Design AI engineering process aligned with emerging EASA guidance.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

62,212 characters extracted from source content.

Expand or collapse full text

COVERAGE-DRIVEN VERIFICATION FOR SAFETY-BY-DESIGN IN AI-BASED COLLISION AVOIDANCE SYSTEMS Thomas Stefani 1 , Johann Maximilian Christensen 2 , Elena Hoemann 2 , Frank Köster 2 & Sven Hallerbach 2 1 Institute for AI Safety and Security, German Aerospace Center (DLR), Ulm, Germany 2 Institute for AI Safety and Security, German Aerospace Center (DLR), Sankt Augustin, Germany Abstract Artificial Intelligence (AI) offers significant potential for future aviation systems; however, its integration into safety-critical applications requires compliance with the aviation sector’s stringent safety standards. For AI and Machine Learning (ML)-based systems, the European Union Aviation Safety Agency (EASA) emphasizes the need to demonstrate the representativeness and completeness of the Operational Design Domain (ODD) and the associated data distributions used during development and verification. Despite this requirement, a structured engineering process for defining target distributions and evaluating representativeness within ODDs remains largely unexplored. This work presents a method for representativeness assessment of AI/ML constituent ODDs in the context of aviation safety assurance. Starting from the methodical identification of suitable target distributions, a process flow is proposed that guides developers from ODD definition and parameter distribution modeling to the quantitative assessment and interpretation of coverage results with respect to EASA’s learning assurance objectives. As quantitative measures, the chi-squared goodness-of-fit test is examined and found unsuitable for the large data sets arising in this setting, leading to the adoption of the Kullback–Leibler divergence and Cramér’sVfor the representativeness assessment. The method is demonstrated using the example of AI-based airborne collision avoidance, employing experimental data from previous Horizontal Collision Avoidance System (HCAS) and Vertical Collision Avoidance System (VCAS) simulations. The results illustrate how statistical distribution comparison methods can support the assessment of representativeness for safety-critical AI applications and contribute toward a systematic Safety-by-Design AI engineering process aligned with emerging EASA guidance. Keywords: AI Engineering, Safety-by-Design, Operational Design Domain, Coverage, AI Certification 1. Introduction Current developments in Artificial Intelligence (AI) are transforming industries, promising exponential gains in efficiency, while simultaneously presenting engineers with novel challenges. Here, aviation is no exception. Analysts expect annual growth rates of approximately35 %for AI-based applications in aviation [1]. This is driving ongoing research into various AI-based safety-critical applications, such as collision avoidance, particularly simplified versions of ACAS X called HCAS and VCAS [2]. The increasing complexity of AI challenges regulators in safety assurance and certification [3]. However, the use of any AI-based system in aviation requires that the development process comply with European Union Aviation Safety Agency (EASA) guidelines. Within the learning assurance framework, EASA emphasizes the need to demonstrate the “completeness and representativeness of data sets” used for the development and verification of AI/ML constituents [4]. In particular, the AI/ML constituent Operational Design Domain (ODD) defines the operational conditions and corresponding input space under which the system is intended to operate safely [4]. Consequently, the verification data used throughout the assurance process must adequately represent the distributions within the defined ODD. https://doi.org/10.71945/icas2026_0536 arXiv:2608.20864v1 [cs.AI] 21 Aug 2026 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS One step in this direction was taken in previous work [5,6,3], which focused on deriving scenarios from ODD descriptions using the collision-avoidance use cases HCAS and VCAS. By integrating HCAS and VCAS into the generic open-source Python library pyCASX [3], both vertical and horizontal advisories can be evaluated within the flight simulator FlightGear. The generated experimental data now provide a foundation for investigating representativeness assessment methods for AI-based collision avoidance systems. Despite EASA’s explicit emphasis on representativeness, it remains unclear how developers should define suitable target distributions for ODD parameters and quantitatively assess whether available verification data adequately represent these distributions. Existing approaches for coverage measures often rely on geometric coverage metrics or statistical measures for comparing probability distributions. However, many of these methods suffer from scalability issues in higher-dimensional spaces, or do not directly address the statistical representativeness of the data distribution. Therefore, this paper presents a method for representativeness assessment of AI/ML constituent ODDs within the context of aviation safety assurance. A process flow is proposed that guides developers from defining target parameter distributions to evaluating representativeness using statistical distribution comparison methods. For this purpose, the chi-squared goodness-of-fit test, the Kullback–Leibler (KL) divergence, and Cramér’sVare investigated regarding their applicability to ODD representativeness assessment. Using experimental HCAS and VCAS data, the proposed method demonstrates how representativeness analysis can support a Safety-by-Design AI engineering process while aligning with EASA’s emerging AI safety requirements. The remainder of this paper is structured as follows. Section 2 reviews the state of the art in ODD coverage assessment for AI-based systems and identifies the gap addressed in this work. The proposed method is then introduced in Section 3, presenting the two-step coverage assessment loop and detailing the representativeness analysis based on statistical distribution comparison. Building on this, Section 4 applies the method to the HCAS and VCAS collision avoidance use cases, describing the experimental setup, the definition of target distributions, and the resulting representativeness assessment. The findings are discussed in Section 5, with particular focus on the limited suitability of the chi-squared test for large-scale assessment and the complementary roles of the KL divergence and Cramér’sV. Finally, Section 6 concludes the paper and outlines directions for future work toward a comprehensive coverage-driven verification process. 2. State of the Art The recent progress in AI prompted the EASA to launch an AI Roadmap [7] and to publish a concept paper [4] addressing Level 1 and Level 2 AI applications. Central to these publications are the notion of the ODD and the learning-assurance framework built on the W-shaped process, which prescribes dedicated steps for the trustworthy development of AI-based systems, including the demonstration of full ODD coverage. How to translate these objectives into a usable AI engineering process, however, remains only loosely defined [8]. Prior work [6,9] approached this by combining the W- shaped process with the DevOps cycle, yielding a first AI engineering framework for AI certification in aviation, while Werner [10] established how to methodically derive the AI/ML constituent Operational Domain (OD) and ODD. Because the AI/ML constituent ODD defines the input space of the neural network, demonstrating sufficient coverage of this space is essential to the safety argument [11]. In safety-critical systems, coverage has traditionally referred to structural code metrics, but AI-based functions shift the focus toward verifying the ODD itself [12]. In line with the EASA objectives, demonstrating ODD coverage rests on two complementary properties: completeness, whether the data span the ODD combinatorially, and representativeness, whether the data follow the distribution it prescribes. Most existing work addresses completeness. Weissensteiner [13] defined a high- level ODD coverage process for validating automated driving systems and proposed a sampling method for n-dimensional scenario-parameter distributions, in which an initial k-means clustering is adapted under predefined boundary conditions to require substantially fewer scenarios; the approach strengthens the overall safety argument but does not account for attribute interactions and the resulting high dimensionality. Using combinatorial testing, Diemert [14] introduced the CACTus framework to systematically model the ODD input space and derive a manageable scenario suite, showing that combinatorial testing together with expert judgment can reduce the parameter space, though without quantifying how representatively that space is populated. Geometry-based metrics form a 2 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS further line of work. Hirschle [12] examined a convex hull enclosing all samples, which does not credit new samples falling inside the hull and scales poorly in higher dimensions, as well as a per-sample hypersphere metric, which can overestimate coverage and yield values above one depending on the chosen radius. As a solution, Hirschle proposed two new metrics, of which a kernel-density-estimation metric showed promising scalability and the ability to localize low-coverage regions. Common to all these approaches is that they quantify how much of the input space is covered, rather than whether the data within it follow the distribution prescribed by the ODD. The latter facet is addressed most directly by the joint EASA and Collins Aerospace ForMuLA report [15], which demonstrated a representativeness assessment of an ML constituent ODD within a learning-assurance toolchain. For a remaining-useful- life prediction use case, the durations of individual flight regimes were compared against the Gaussian target distributions prescribed in the ODD using goodness-of-fit tests, with the chi-squared test as the primary example alongside the Kolmogorov–Smirnov, Lilliefors, and Anderson–Darling tests. There, representativeness was reduced to a binary pass/fail verdict at a fixed significance level, and the authors themselves reported that data may be rejected even when they visually match the target, that raising the significance level alters the verdict but demands further justification, and that small or empty expected bin counts together with a limited dataset size reduce the reliability of the test. While this confirms the feasibility of statistical representativeness assessment in an EASA context, it was demonstrated on a single low-dimensional feature and relied on a single significance-test outcome, leaving open how representativeness can be quantified robustly across a high-dimensional ODD. Despite these advances in quantifying ODD sub-spaces, the literature still lacks a comprehensive coverage process that satisfies EASA’s coupled requirements for data representativeness and com- pleteness, both of which are essential for the formal certification of AI-based systems. 3. Method for ODD Coverage Assessment The assurance of safety-critical AI-based systems requires methods that enable a systematic and trustworthy verification of their behavior. In aviation, where AI/ML applications must comply with stringent safety requirements, regulatory authorities have introduced guidance to support the certifica- tion and assurance of such systems. In particular, the EASA defines several anticipated means of compliance (MOC) addressing the coverage of the ODD, including requirements for demonstrating completeness and representativeness of the data and parameter space [4]. However, despite these objectives and their corresponding MOCs being formulated at a high level, the practical demonstration of sufficient ODD coverage remains largely unresolved. Existing guidance does not specify how completeness and representativeness should be quantified, nor which metrics and processes are adequate for demonstrating coverage in high-dimensional ODDs. In other works [16], a method for demonstrating completeness is proposed. Still, there is currently a lack of a generic and scalable method that accounts for the representativeness of the data and supports the verification of AI/ML constituents while aligning with EASA’s Safety-by-Design and learning assurance principles. 3.1 Method Overview The method proposed in this work operationalizes EASA’s coverage assessment objectives through a structured two-step process, as illustrated in Figure 1. The method takes two inputs: the formally defined AI/ML Constituent ODD and a dataset, which in this case are simulated near mid-air collision scenarios. The use case is introduced in more detail in Section 4. Together, these inputs drive an iter- ative assessment loop that evaluates whether the dataset sufficiently covers the operational parameter space. The assessment proceeds sequentially. First, a representativeness analysis examines whether the dataset adequately reflects the distribution and density of the ODD parameter space, identifying gaps and underrepresented regions. If representativeness is found to be insufficient, the loop feeds back, requiring dataset augmentation before proceeding further. Only once representativeness is established does the process advance to the second step: a completeness analysis, which verifies that all relevant combinations of ODD parameters are covered to the required degree [16]. If both criteria are satisfied, coverage is deemed sufficient, and the assessment concludes. If completeness is found lacking despite adequate representativeness, the loop again iterates, targeting the specific parameter combinations that remain uncovered. This sequential structure follows a logical dependency: param- eter distribution must be evaluated before parameter combinations. Because addressing any gaps 3 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS found in the underlying distribution requires adding new data, doing so directly alters the dataset’s combinatorial mix, meaning any completeness check performed beforehand would be invalidated. Figure 1 – Activity Diagram for the Coverage Assessment loop aligning with EASA’s MOC DM-08 through a two-stage process consisting of demonstrating data representativeness and completeness. 3.2 Data Representativeness Demonstrating data representativeness is a central aspect across several of EASA’s anticipated objectives [4]. At the data-management requirements stage, Objective DA-04 requires the applicant to capture the data quality requirements for all training, validation, and test data, explicitly including the completeness and representativeness of the data sets and the traceability of their origin. The dedicated representativeness means of compliance, Anticipated MOC DM-07-2, characterizes repre- sentativeness as the property that the data are sampled according to the distribution prescribed for the input space, and anticipates statistical goodness-of-fit methods for assessing operating parame- ters [4]. Finally, the data-verification step of Objective DM-08 confirms that the test data set covers the whole ODD with the necessary level of completeness and representativeness. The method proposed here operationalizes this chain for the representativeness property: starting from the requirements captured under DA-04, it provides the quantitative assessment anticipated by DM-07-2 and produces the evidence consolidated at the DM-08 verification step [4]. This work addresses the distributional aspect of DM-07-2 at the level of the individual ODD parameters: for each parameter, the observed data are compared against the univariate distribution prescribed for it, so that representativeness is established parameter by parameter. This parameter-level representativeness is one of the two coupled properties on which EASA’s coverage demonstration rests; its counterpart, the pairwise combinatorial coverage of the ODD parameters anticipated by Objective LM-16, is the object of the complementary completeness pillar and is treated separately [16]. Two aspects therefore lie outside the scope of the present contribution: the sampling independence between data sets (Anticipated MOC DM-07-5), which is assumed rather than assessed, and any characterization of the joint ODD density beyond these two coupled properties, which is not treated in this work. Consequently, the process flow for demonstrating representativeness is defined as follows. The AI/ML constituent ODD and the generated simulation data are assumed as given inputs. Based on the ODD definition, the first step consists of identifying the target distribution for each relevant ODD parameter. Once the target distributions are defined, a representativeness assessment is performed to quantify how well the available dataset represents the expected operational parameter space. To this end, different metrics and divergences are investigated, including the KL divergence, the chi-squared test, and Cramér’s V. The objective of the representativeness step is therefore to identify underrepresented regions and distribution mismatches within the ODD before proceeding to the completeness assessment. Subsequent coverage analyses yield meaningful assurance regarding AI/ML constituent behavior only if the dataset adequately represents the intended ODD. 4 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS 3.2.1 Target Distribution To assess representativeness, a target distribution must first be defined for each parameter within the AI/ML constituent ODD. Figure 2 illustrates a structured process flow to guide engineers in systematically deriving suitable target distributions from available data sources, statistical assumptions, or expert knowledge. Starting from a formally defined ODD parameter, the method first evaluates whether representative real-world data are available. If such data exist, they are directly used to derive the target distribution. Otherwise, the process investigates whether simulation data are available from representative reference simulations. If neither real-world nor simulation data are available, the method evaluates whether a physically motivated statistical assumption can reasonably describe the parameter distribution, for example, using uniform or normal distributions. If this is not feasible, expert knowledge or existing domain standards are considered as alternative sources for defining the target distribution. In cases where none of these information sources are available, the parameter is flagged as undefined and requires further investigation. Regardless of the selected source, the rationale and origin of the chosen target distributions are documented to ensure traceability in accordance with EASA MOC DM-07-2. The resulting target distributions are subsequently used within the coverage assessment process to evaluate representativeness and identify potential coverage gaps requiring mitigation. 3.2.2 Representativeness via Kullback–Leibler Divergence The KL divergence, or relative entropy, is an asymmetric statistical measure used to quantify the difference between two probability distributions. In the context of data representativeness, it can be used to evaluate how well the observed test data distribution, typically defined asQ(x), matches a target or expected distributionP(x). For a continuous parameter space, the divergenceD KL is defined via integrating over the parameterspace [17]: D KL (P∥ Q) = Z ∞ −∞ p(x) log p(x) q(x) dx.(1) To compute a continuous KL divergence, the exact probability density functions (PDFs) must be known. However, since data or simulation outputs are collections of discrete samples, it is necessary to estimate these continuous curves using techniques such as kernel density estimation (KDE). Further, if multiple parameters are involved, calculating this integral numerically becomes computationally expensive. Therefore, we focus explicitly on the discrete formulation: D KL (P∥ Q) = ∑ x∈X P(x) log P(x) Q(x) .(2) Because the discrete divergence is unbounded whenever a bin carries target mass but contains no observed samples (P(x)> 0,Q(x) = 0), the observed distribution is regularized prior to evaluation: each empty bin is assigned a floor probabilityε = 10 −6 , andQis subsequently renormalized. This keepsD KL finite while confining the divergence of unpopulated bins to a defined, bounded contribution. As the magnitude of these empty-bin contributions depends on the choice ofε, a single value is fixed across all parameters and documented here to ensure reproducibility in accordance with EASA MOC DM-07-2. A lower divergence indicates that the generated or collected data is more representative of the intended operational environment. The KL divergence is governed by three key properties, namely its non-negativity (D KL (P∥ Q)≥ 0), its identity (D KL (P∥ Q) = 0 ⇐⇒ P = Q), and its asymmetry (D KL (P∥ Q)̸= D KL (Q∥ P)), making it an adequate choice to assess the representativeness because it quantifies information loss when approximating a true distribution while prioritizing a specific direction of error. Since EASA’s anticipated means of compliance, such as MOC DM-07-2, require evidence for the representativeness of data sets without prescribing concrete quantitative thresholds, heuristic interpretation ranges are proposed in Table 1. These ranges are intended to support the coverage argumentation process and provide a practical guideline for evaluating whether the sampled data adequately represent the AI/ML constituent ODD. The proposed thresholds are not regulatory limits, but engineering-oriented guidance values derived from common interpretations of statistical divergence measures in distribution comparison and dataset shift analysis [18, 19, 20]. 5 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS Real-world data available? ODD parameter Use real data Simulation data available? Use simulation data Physically motivated assumption possible (e.g., uniform, normal)? Use statistical assumption Expert knowledge or domain standards available? Use expert knowledge Flag parameter: distribution undefined—next steps tbd. Document source and rationale for traceability (EASA Objective DA-04) Apply coverage assessment Data Representativeness verified or gap flagged for mitigation No No No No Yes Yes Yes Yes Figure 2 – Generic process flow for identifying the target distributions for the ODD parameters that are subsequently used for the coverage assessment. Starting from real-world data, over simulation data, physically motivated assumptions, to expert knowledge. 6 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS Table 1 – Interpretation of KL divergence values demonstrating parameter representativeness (EASA MOC DM-07-2) [18, 19, 20]. Lower D KL indicates a better fit between the two distributions. D KL Ranges Interpretation Coverage Assessment Implication 0≤ D KL < 0.05 ExcellentStrong evidence of representativeness 0.05≤ D KL < 0.15 GoodGenerally acceptable with rationale 0.15≤ D KL < 0.30 ModerateMitigation or targeted sampling recommended 0.30≤ D KL < 0.50 PoorCoverage argument weakened D KL ≥ 0.50 FailRepresentativeness claim at risk 3.2.3 Representativeness via Chi-Squared Anticipated MOC DM-07-2 names the chi-squared test, alongside the Z- and Kolmogorov–Smirnov tests, among the goodness-of-fit methods anticipated for assessing the representativeness of operating parameters [4]. The chi-squared goodness-of-fit test is a non-parametric hypothesis test used to assess whether an observed sample distribution is consistent with a fitted target distribution [21]. In the context of ODD coverage verification, it operationalizes the representativeness requirement: given a target distribution specified for an ODD parameter, the test quantifies whether the collected scenario data reflect that distribution sufficiently well. The test statistic partitions the parameter range intok non-overlapping bins and compares the observed counts against the expected counts under the target distribution: χ 2 = k ∑ i=1 (Q i − n· p i ) 2 n· p i ,(3) whereQ i is the observed count in biniandn· p i the corresponding expected count, withnthe total number of samples andp i the probability mass assigned to biniby the target distribution. Under the null hypothesisH 0 that the observed data are drawn from the target distribution, the statistic follows a chi-squared distribution withν = k− 1degrees of freedom.H 0 is rejected at significance levelαifχ 2 > χ 2 α,ν ; a failure to reject, that is, ap-value exceedingα, is taken as evidence that the data are consistent with the target and thereby supports the representativeness claim. Following common practice, a conventional fixed level ofα = 0.05is used as the baseline criterion [22]. A practical prerequisite of the test is that the expected counts be sufficiently large, commonlyn· p i ≥ 5 for each bin, to ensure the validity of the chi-squared approximation [23]. This couples the choice of bin granularity to the available dataset size: finer bins increase the sensitivity to local distributional deviations but require proportionally more data. Its applicability to any target distribution—uniform, normal, or empirically defined—further makes the test well suited to the heterogeneous nature of ODD parameters. A decisive limitation, however, is that theχ 2 statistic grows with the sample sizen, so that for large data sets even negligible deviations are flagged as significant. This motivates the sample-size-normalized effect-size formulation introduced in the following subsection. 3.2.4 Representativeness via Cramér’s V The chi-squared goodness-of-fit test of Section 3.2.3 yields a binary verdict on the existence of a distributional difference, but its statistic scales with the sample sizen: asngrows, even practically negligible deviations become statistically significant, so the test tends to rejectH 0 for any sufficiently large data set [22]. To assess representativeness in terms of the magnitude of the deviation, theχ 2 statistic from Equation (3) is converted into an effect-size measure. Cramér’sV[24] normalizes aχ 2 statistic by the sample size and the table dimensionality, V = s χ 2 n(q− 1) ,q = min(r, c).(4) whererandcdenote the number of rows and columns of the underlying contingency table. For the goodness-of-fit comparison, the smaller dimension isq = 2(the observed against the target distribution 7 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS over k bins), so that the measure applied to the statistic of Equation (3) is V = r χ 2 n .(5) This quantity is equivalent to Cohen’s effect-size measurewfor a chi-squared goodness-of-fit test [25]. By normalizingχ 2 by the sample size,Vremoves the large-sample inflation that drives the bareχ 2 test toward rejection, while reusing the identical binning and target-distribution inputs already required for theχ 2 computation. It satisfiesV ≥ 0, withV = 0if and only if the observed and target distribution coincide, and larger values indicating progressively stronger deviation. Unlike the association measure of a full contingency table, the goodness-of-fit effect size is not bounded above by one: for equiprobable binning, its maximum is √ k− 1, soVmust be interpreted relative to a fixed bin resolution. Being constructed from a different principle than the information-theoretic KL divergence,Vprovides a construction-independent second indicator of representativeness, allowing the two measures to be used as a mutual cross-check. As with the KL divergence, EASA’s anticipated means of compliance do not prescribe quantitative thresholds forV. The interpretation ranges in Table 2; therefore, follow Cohen’s conventional effect-size magnitudes fordf = 1[25], adapted as engineering-oriented guidance rather than regulatory limits. A stricter pass threshold may furthermore be assigned to parameters of higher safety relevance [26]. Table 2 – Interpretation of Cramér’sVvalues for the2× kgoodness-of-fit framing, demonstrating parameter representativeness (EASA MOC DM-07-2). The thresholds are adopted from Cohen [25] and translated into finer granularity. V Ranges Interpretation Coverage Assessment Implication 0≤ V< 0.10 ExcellentStrong evidence of representativeness 0.10≤ V< 0.20 GoodGenerally acceptable with rationale 0.20≤ V< 0.30 ModerateMitigation or targeted sampling recommended 0.30≤ V< 0.40 PoorCoverage argument weakened V ≥ 0.40 FailRepresentativeness claim at risk 4. Case Study: Horizontal and Vertical AI-Based Collision Avoidance The following case study applies the representativeness method from Section 3 to a concrete aviation AI application: an AI-based collision avoidance system comprising a HCAS and a VCAS [2]. This use case was selected for two reasons. First, it represents a well-defined, safety-critical AI/ML constituent with a clearly bounded ODD, making it well-suited for demonstrating the proposed coverage assessment. Second, both systems have been tested in prior work [5,6,3], providing the simulation infrastructure and scenario data necessary to populate the ODD. VCAS and HCAS are neural-network- based proof-of-concept implementations of the ACAS X family of collision avoidance standards, addressing the vertical and horizontal separation dimensions, respectively [2]. 4.1 ODD Definition HCAS issues horizontal advisories based on seven state variables: the range to intruderρ, the bearing angleθ, the relative headingψ, the ownship speedv own , the intruder speedv int , the time to closest point of approach (CPA)τ, and the previous advisorys adv . VCAS extends this logic to the vertical plane, issuing vertical advisories—such as climb or descend commands—based on five state variables which are listed in [2]: the relative altitude between the ownship and the intruderh, the own climb/descent rate ̇ h own , the intruder’s climb/descent rate ̇ h int , the time to CPAτ, and the previously issued advisorys adv . While a traditional ODD typically contains a broader set of environmental attributes, such as scenery and environmental conditions, the focus here is on the immediate input space of the neural networks. Consequently, for the purpose of this analysis, these state variables define the AI/ML constituent ODD for each system, as summarized in Table 3 and Table 4. While our parameter values are adapted from Kochenderfer et al. [27], they vary slightly from those used in other prior studies [3,28,5,6,16]. 8 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS This adjustment is due to differences in scenario settings, as the primary objective of this work is to demonstrate the underlying process flow rather than to focus strictly on numerical precision. For reasons of simplification, for both HCAS and VCAS, the previous advisory is not considered in the ODD. Table 3 – Derived AI/ML constituentODD HCAS from the HCAS State Variables [27]. The values are oriented on the GitHub release athttps://github.com/sisl/HorizontalCASbut differ based on previous work [3,28,5,6] wherev own was kept the same throughout the executed simulations and therefore remained at a constant value. Variable DescriptionRange ρRange to intruder [0 ft, 56 000 ft] θBearing angle[−180°, 180°] ψRelative heading[−180°, 180°] v own Ownship speed865 ft s −1 v int Intruder speed[845 ft s −1 , 868 ft s −1 ] τTime to CPA[0 s, 55 s] Table 4 – Derived AI/ML constituentODD VCAS from the VCAS State Variables [27]. The values are oriented on the GitHub release athttps://github.com/sisl/VerticalCASbut differ based on previous work [3, 28, 5, 6] where only the ownship avoided the collision, therefore ̇ h int is set to 0. Variable DescriptionRange hRelative altitude[−2800 ft, 2800 ft] ̇ h own Ownship vert. rate [−3200 ft min −1 , 3200 ft min −1 ] ̇ h int Intruder vert. rate[0 ft min −1 ] τTime to CPA[0 s, 40 s] 4.2 Simulation Setup and Data Collection The data used for the coverage assessment was obtained from previously executed HCAS and VCAS simulation experiments presented in earlier work [5,6,3]. The scenarios were executed in a simulation environment using a highly automated framework to generate randomized trajectories beginning approximately55 sbefore CPA. The resolution advisories provided by HCAS and VCAS were automatically translated into ownship avoidance maneuvers via an auto-avoid function, at approximately30 sto CPA. The simulation outputs, containing the corresponding state variables for either VCAS or HCAS, were collected for future analysis. Figure 3 shows an example excerpt from the VCAS dataset, illustrating the ownship vertical rate ̇ h own plotted over time to CPA (τ). The blue dots indicate the ownship’s climb and descent maneuver issued by VCAS to avoid a potential near mid-air collision between an intruder aircraft and the ownship. The different reached rates reflect either a strong climb, a climb, a clear of conflict, a descent, or a strong descent. The original experiments were not conducted with the objective of demonstrating representativeness or ODD coverage. Instead, existing data are used in this work as an exemplary basis to illustrate how a representativeness assessment process for AI/ML constituent ODD coverage can be performed. For both HCAS and VCAS, a CSV file was used to store 1.97 million rows of state variables extracted from simulations. 4.3 Coverage Results for Representativeness As described in Figure 2, the proposed process flow for identifying the target distribution was followed. However, obtaining representative real-world operational data or generating dedicated simulation data is often time-intensive and may not be feasible during early methodological investigations. Since the focus of this paper is the exemplary demonstration of the proposed representativeness assessment method, the target distributions for the selected ODD parameters were defined based on expert 9 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS Figure 3 – Excerpt from the executed collision avoidance scenarios. The blue dots indicate the available data for the ownship’s vertical rate over the time to CPAτ. At approximately30 sto CPA the ownship performs an avoidance maneuver through either climbing or descending. knowledge and engineering assumptions. To ensure traceability in accordance with EASA MOC DM- 07-2 objectives, the rationale and source for each assumption are documented. An excerpt for two parameters is given in Table 5. Before presenting the coverage results, we justify why only the KL Table 5 – Exemplary documentation of target distribution assumptions for ODD parameters in accor- dance with EASA MOC DM-07-2 traceability objectives. ParameterTargetRationale / Source Distribution ODD VCAS : ̇ h own Normal dis- tribution The ownship vertical rate is assumed to follow a normal distribution based on expert knowledge of typical aircraft flight behavior. During nominal operations, aircraft predom- inantly operate around stable climb or descent rates, while extreme vertical rates occur less frequently. The normal distribution, therefore, reflects the expected concentration around standard operational conditions while still account- ing for rare deviations. ODD HCAS : τ Uniform dis- tribution In approximately unaccelerated encounters, time to CPA (τ) decays at a constant ratio with elapsed time. Recording potential encounters at a fixed sampling rate over a set window (e.g., from 55 to 0 seconds) therefore yields an approximately uniform distribution. divergence and Cramér’sVare considered in the coverage report. The Pearsonχ 2 test, although one of the candidate metrics for demonstrating representativeness introduced in Section 3, did not yield interpretable results. To isolate the cause, we conducted a bin-count sensitivity analysis on the VCAS parameterτ. This parameter was chosen deliberately since the distribution was approximately uniform based on the scenario setting. Therefore, a representative sample should not be rejected (p> 0.05) and should exhibit negligible association. Figure 4 shows the outcome of the sensitivity analysis. As shown in Figure 4a, theχ 2 p-value remains many orders of magnitude below the0.05significance level for every bin count and deteriorates further as the number of bins increases. This is the expected consequence of the large-sample inflation of theχ 2 statistic: at the encounter-set sample sizes 10 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS used here, statistical significance is driven by sample size and granularity, which renders thep-value unusable as a representativeness criterion. Cramér ’sV, which normalizes the sameχ 2 statistic into a bounded effect size, behaves as required (see Figure 4b). It remains below the negligible-association threshold (V< 0.1) across the entire sweep and stabilizes with increasing bin counts. The magnitude 2 100200300400500600700800900 1000 Number of Bins 10 51 10 44 10 37 10 30 10 23 10 16 10 9 10 2 P-value p = 0.05 (a)χ 2 sensitivity analysis for theODD VCAS parameterτ regarding number of bins. The red dashed line highlights the minimum p-value to be reached. Lower p-values indicate stronger deviation. 2 100200300400500600700800900 1000 Number of Bins 10 2 10 1 Cramer's V Excellent (V < 0.1) (b) Cramér’s V sensitivity analysis for the ODD VCAS parameterτregarding bin count. The green line shows the maximum value for excellent fit. Consequently, lower V -values indicate better alignment. Figure 4 – Sensitivity of the χ 2 test and Cramér’s V to the number of bins for the ODD VCAS parameter τ which has an approximately uniform distribution. of the distributional difference is thus recovered robustly and independently of the binning choice. We therefore retainχ 2 solely as the documented basis for this decision and report its normalized effect size, Cramér’sV, alongside the KL divergence in Table 6 and Table 7, providing the traceable rationale required by MOC DM-07-2. Table 6 and Table 7 report the ODD coverage results for the VCAS and HCAS input parameters, evaluated with the KL divergenceD KL and Cramér’sV, the two measures retained following the sensitivity analysis in Figure 4. Range utilization denotes the ratio between the interval spanned by the collected data and the defined ODD range of the respective parameter, while bin coverage quantifies how completely the bins within that range are populated; a bin coverage of 100 %therefore indicates that every bin contains at least one data point. Forv int , a coarser partition of 20bins was chosen rather than the100used for the other parameters: since the intruder speed varies only over the narrow interval, a100-bin partition would resolve this range too finely, leaving individual bins sparsely populated and in conflict with the expected-count prerequisite (n· p i ≥ 5) established in Section 3.2.3. For both measures, the value column is accompanied by its categorical interpretation in accordance with Table 1. For VCAS,τshows a range utilization of374.85 %, substantially exceeding Table 6 – Coverage report for the ODD VCAS with bin coverage, KL divergence and Cramér’s V . ODD VCAS Nr.TargetRangeBin Cov.D KL D KL Cramér’s VCramér’s V Bins Dist.Util. (%)(%)ValueResultValueResult h100Normal92.9689.651.0382Fail0.1682Good ̇ h own 100Normal87.8064.351.5507Fail0.2357Moderate τ100Uniform374.85100.000.0002 Excellent0.0013Excellent its baseline range and indicating that the collected data extend well beyond the originally specified ODD boundaries. The parametershand ̇ h own reach92.96 %and87.80 %, reflecting adequate but incomplete coverage of their defined ranges. A comparable pattern holds for HCAS:ρ,v int , andτ all exceed their baseline ranges (173.95 %,469.36 %, and272.62 %), whereasθandψeach attain exactly100 %. In terms of bin coverage,τachieves100 %andhreaches89.65 %for VCAS, while ̇ h own yields the lowest value at64.35 %, indicating that a notable portion of its input space remains 11 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS Table 7 – Coverage report for the ODD HCAS with bin coverage, KL divergence and Cramér’s V . ODD HCAS Nr.TargetRangeBin Cov.D KL D KL Cramér’s VCramér’s V Bins Dist.Util. (%)(%)ValueResultValueResult ρ100Uniform173.95100.000.0236 Excellent0.0212Excellent θ100Uniform100.0099.411.1164Fail0.2782Moderate ψ100Uniform100.0095.291.1599Fail0.5563Fail v int 20Uniform469.36100.000.0444 Excellent0.0202Excellent τ100Uniform272.62100.000.0019 Excellent0.0060Excellent sparsely populated, as also visible in Figure 3. For HCAS,ρ,v int , andτare fully covered, whileθ reaches99.41 %andψ 95.29 %. The distributional measures provide a more differentiated picture than bin coverage alone. The parameters with the broadest data support—τfor VCAS as well asρ, v int , andτfor HCAS—are rated Excellent by bothD KL and Cramér’sV, with negligible divergences (below0.05) and correspondingly small association values, confirming that their empirical distributions closely match the assumed uniform baselines. In contrast,hand ̇ h own (VCAS) together withθandψ (HCAS) obtain a Fail KL rating (1.0382,1.5507,1.1164, and1.1599, respectively). Under Cramér’sV, h, ̇ h own , andθare assessed more favorably, withhrated Good (0.1682) and both ̇ h own (0.2357) and θ(0.2782) rated Moderate. The relative headingψ, by contrast, is the only parameter rated Fail by both measures (Cramér’sV = 0.5563), so that the two indicators concur on a substantial distributional mismatch. Figure 5 illustrates the KL divergence assessment for the relative altitudehofODD VCAS against the assumed normal target distribution. The upper panel contrasts the observed data with the target density: the collected data are sharply concentrated aroundh = 0 ftand largely confined to the interval [−500 ft, 500 ft], whereas the target normal spreads its mass broadly across the full ODD range delimited by the boundaries at≈±2800 ft. The discrepancy is therefore one of concentration: the empirical distribution is far more peaked than the reference, despite both sharing the same central location. The lower chart decomposes the total divergence into per-bin contributions and localizes the source of the mismatch. The dominant contributions arise in the flanking regions, from a few hundred feet out to roughly2000 fton either side of the center, where the target assigns appreciable probability mass that the observed data do not realize. The densely populated central bins, in turn, form a narrow region of opposite-signed contribution. The accumulation of these flank terms yields the overall divergence ofD KL = 1.0382, rated with Fail. This Fail rating coexists with a high range utilization of92.96 %and a bin coverage of89.65 %. The data thus populate nearly the entire defined input space, yet in proportions that deviate strongly from the target. This case demonstrates concretely that coverage and representativeness are distinct properties: complete coverage does not imply that the data follow the intended distribution—precisely the gap that the distributional measures are introduced to detect. Figure 6 shows the Cramér’sVassessment for the time to CPAτofODD VCAS against the assumed uniform target distribution. Within the ODD range delimited by the boundaries atτ = 0 sandτ = 40 s, the observed density (upper panel) is essentially flat at≈ 0.011and closely tracks the uniform reference, in marked contrast to the strongly peaked altitude distribution of Figure 5. A substantial share of the collected data lies beyond the ODD boundaries—most visibly the negative-τsamples and the elevated bins nearτ≈−30 s—consistent with the range utilization of374.85 %; the representativeness assessment is nonetheless evaluated only over the defined range, where the empirical and target distributions agree closely. Beneath, the chart reports the per-bin contributions to theχ 2 statistic underlying Cramér’sV, which are non-negative by construction. The largest contributions concentrate at short times to CPA (roughlyτ ∈ [0 s, 8 s]), with a secondary cluster between15 sand30 s, indicating that the mild residual deviations from perfect uniformity are localized rather than systematic. Relative to the large in-range sample size, these deviations remain negligible, and the resulting Cramér’s V = 0.0013is rated Excellent. Here, the strong coverage indicators are corroborated rather than contradicted by the distributional measure: the data fill the ODD range completely and also approximate 12 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS 3000200010000100020003000 0.0000 0.0005 0.0010 0.0015 0.0020 0.0025 Probability Density Utilization: 92.96% Bin Coverage: 89.65% KL Divergence: 1.0382 Observed Data ODD Boundary Target Distribution 3000200010000100020003000 h (ft) 0.04 0.02 0.00 0.02 0.04 0.06 Contribution Score Figure 5 – Kullback–Leibler divergence of relative altitudehfromODD VCAS against normal distribution. the intended uniform distribution, so coverage and representativeness are jointly satisfied. This contrasts with the altitude case in Figure 5, where high coverage masked a substantial distributional mismatch. Representativeness of a dataset with respect to an ODD parameter is demonstrated when the following two conditions hold, evaluated in order: 1.Bin-Coverage-Gate. Every bin of the parameter’s ODD partition contains at least one sample, i.e. the bin coverage equals100 %. The number of bins is fixed by the sensitivity analysis in Figure 4 and is not adjusted to satisfy this criterion. 2.Divergence-Condition. Provided condition 1 holds, the divergence of the empirical distribution from the target distribution remains within the tolerance assigned to the parameter’s safety- relevance tier under both the KL divergenceD KL and Cramér’sV, in accordance with the interpretation thresholds (e.g. Good or better) listed in Table 1 and Table 2. A parameter satisfies the representativeness criterion only if both conditions are met. Failure of condition 1 precludes the evaluation of condition 2, as the distributional measures are reliable only over a fully populated partition. Finally, since representativeness is anchored to a chain of EASA’s anticipated objectives, the corresponding mapping is summarized in Table 8 [4]. 5. Discussion This work introduces a novel, quantitative method for demonstrating ODD coverage in the learning assurance of safety-critical AI-based systems, developed in alignment with EASA’s anticipated objec- tives [4]. ODD coverage rests on two complementary properties of the data set: representativeness, that the data follow the distribution prescribed by the ODD, required by Objective DA-04 and An- ticipated MOC DM-07-2 and confirmed at the DM-08 verification step; and completeness, that the data span the ODD combinatorially, underlying the pairwise ODD-parameter coverage anticipated by Objective LM-16. The present work formalizes the representativeness property. Consequently, 13 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS 40200204060 0.000 0.005 0.010 0.015 0.020 Probability Density Utilization: 374.85% Bin Coverage: 100.00% Cramer's V: 0.0013 Observed Data ODD Boundary Target Distribution 0510152025303540 (s) 0 2 4 6 8 10 12 Contribution Score Figure 6 – Cramér’s V of time to CPA τ from ODD VCAS against uniform distribution. the combinatorial completeness of the joint parameter space is not in the scope of this work and developed separately [16]. However, only the two pillars together operationalize the coupled repre- sentativeness and completeness expectations of EASA’s learning-assurance building block. Since demonstrating representativeness requires more than confirming that the data populate the ODD range, a parameter may span its entire range and fill nearly every bin while its empirical distribution still deviates substantially from the target. Populating the input space, therefore, does not by itself establish representativeness; the proportions in which that space is sampled must also be assessed. This motivates the two-stage structure of the proposed criterion, in which a population gate precedes the distributional evaluation. The distributional assessment was initially based on theχ 2 goodness-of-fit test, following its use for learning-assurance data analysis in the Collins/EASA study [15]. In our setting, this test proved unsuitable as a decision criterion. Because theχ 2 statistic scales with the sample size, the large data sets generated from the simulation drive itsp-value toward zero for even negligible deviations, leading to rejection of the null hypothesis for every parameter—includingτ, whose empirical distribution is in excellent agreement with its uniform target under every effect-size measure. A test that cannot pass even a well-matched parameter offers no discriminatory power and cannot serve as a representativeness gate. We therefore adopted Cramér’sV, which normalizes the sameχ 2 statistic by the sample size and the degrees of freedom to yield a bounded effect size. We retain the KL divergence alongside Cramér’sVbecause the two measures capture complementary aspects of distributional agreement. The KL divergence is an information-theoretic measure that is particularly sensitive to regions where the target assigns probability mass the data do not realize, making it responsive to shape and tail mismatches, whereas Cramér’sVsummarizes the overall magnitude of deviation on a bounded, sample-size-independent scale. Requiring both to fall within tolerance guards against failure modes that either measure alone could miss. Notably, the two measures concur for every parameter that satisfies the bin-coverage gate. Among the parameters with unpopulated 14 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS Table 8 – Mapping of the proposed coverage method to EASA’s anticipated objectives and means of compliance (Issue 02) for the data representativeness properties [4]. EASA itemData-quality requirementAddressed by DA-04Capture of data quality requirements, incl. com- pleteness and representativeness, per ODD pa- rameter, with documented origin and traceability Target-distribution definition and documentation (Figure 2, Table 5) DM-07-2Verification that data are sampled per the pre- scribed distribution;χ 2 /KS goodness-of-fit an- ticipated for low-dimensional operating parame- ters Representativeness assessment viaD KL and Cramér’s V (Section 4.3) DM-08Data-verification step confirming ODD coverage with the necessary completeness and represen- tativeness Two-stage coverage loop (Figure 1) bins,ψis rated Fail by both measures, whereas forh, ̇ h own , andθ, the KL divergence is inflated by the empty-bin contributions to which it is inherently sensitive, so that it fails these parameters while the bounded Cramér’sVrates them only Good to Moderate. This pattern both supports the joint use of the two measures and reinforces the condition ordering: each statistic is trusted only within the regime in which it is reliable—χ 2 away from very large samples, the KL divergence away from sparse partitions. Several limitations remain within the representativeness pillar addressed here. The bin-coverage-gate in its current form requires only that each bin be populated, which guarantees that the ODD range is sampled but not that the per-bin frequencies are estimated with sufficient statistical reliability. Consequently, determining the adequate number of bins is of great importance to account for the required level of granularity. The tolerance bands for Cramér’sVare currently mapped to qualitative interpretation ranges and are not yet anchored to the safety-relevance tier of each parameter, which is required before the criterion can support a formal certification argument. However, the assessment is only as meaningful as the assumed target distributions: the Fail ratings obtained forhand ̇ h own partly reflect the broad normal reference imposed on parameters whose realistic encounters concentrate near small relative altitudes, and replacing these assumed references with empirical distributions derived from operational surveillance data would strengthen the representativeness claim. Finally, this work provides a generic and reusable coverage assessment loop that is aligned with EASA’s guidelines for AI-based systems in aviation, in particular with respect to the data representativeness objectives outlined in the learning assurance building block. The results confirm the practical usability of the proposed method and highlight that no single metric is sufficient for a comprehensive ODD coverage assessment. A combination of bin coverage and distributional similarity measures, such as KL divergence and Cramér’sV, provides a more complete and reliable picture of data set quality for safety-critical AI/ML applications. 6. Conclusion and Outlook This work addressed the assessment of data representativeness for the ODD of safety-critical AI- based systems—the property required by Objective DA-04 and Anticipated MOC DM-07-2 and verified, together with completeness, at the DM-08 data-verification step. We proposed a two- stage representativeness criterion that couples a data population gate over the ODD partition with a divergence evaluation based on the KL divergence and Cramér’sV, and demonstrated it on the neural- network-based HCAS and VCAS. The case study supports three conclusions. First, theχ 2 goodness- of-fit test loses its usability on the large simulation-derived data sets considered here and cannot serve as a decision criterion; sample-size-robust effect-size measures are required instead. Second, no single metric is sufficient: bin coverage establishes that the ODD range is populated, whereas the KL divergence and Cramér’sVcapture complementary aspects of distributional agreement, and their joint use yields a more reliable picture of data set quality than any measure alone. Third, the resulting assessment loop is generic and reusable, and maps directly onto the data-representativeness expectations of the EASA learning-assurance building block. 15 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS Several directions extend this work. The most immediate is the formalization of the second pillar, completeness, which evaluates the combinatorial coverage of the joint ODD space, and which the present criterion does not yet address. The bin-coverage-gate can be strengthened from a minimal occupancy requirement to a statistically grounded one: a per-bin sample-size requirement calibrated by a power analysis is a direction for future work, ensuring not only that each bin is populated but that the per-bin frequencies are estimated reliably enough for the distributional measures to be trustworthy. A complementary refinement concerns the acceptance thresholds, where the qualitative interpretation ranges currently used for the KL divergence and Cramér’sVwould be replaced by application-specific acceptance ranges that map value interpretations to the safety criticality of each parameter, holding safety-relevant parameters to tighter tolerances; such a tier-anchored mapping is a precondition for supporting a formal certification argument. Finally, the assumed target distributions can be replaced by empirical references derived from operational data, and metrics better suited to ordered continuous variables can be investigated as complements to the present measures. 7. Contact Author Email Address thomas.stefani@dlr.de 8. Copyright Statement The authors confirm that they, and/or their company or organization, hold copyright on all of the original material included in this paper. The authors also confirm that they have obtained permission, from the copyright holder of any third party material included in this paper, to publish it as part of their paper. The authors confirm that they give permission, or have obtained permission from the copyright holder of this paper, for the publication and distribution of this paper as part of the ICAS proceedings or as individual off-prints from the proceedings. References [1]Precedence Research. Artificial Intelligence in Aviation Market Size, Share, and Trends 2024 to 2034. Precedence Research; 2022. 1748. Available from:https://w.precedenceresearch.com/ artificial-intelligence-in-aviation-market. [2] Julian KD, Kochenderfer MJ, Owen MP. Deep Neural Network Compression for Aircraft Collision Avoidance Systems. Journal of Guidance, Control, and Dynamics. 2019 03;42(3):598-608. [3]Christensen JM, Anilkumar Girija A, Stefani T, Durak U, Hoemann E, Köster F, et al. Advancing the AI-Based Realization of ACAS X Towards Real-World Application. In: 36th IEEE International Conference on Tools with Artificial Intelligence (ICTAI). Herdon, VA, USA; 2024. p. 57-64. [4]European Union Aviation Safety Agency (EASA). EASA Concept Paper: Guidance for Level 1 & 2 Machine Learning Applications. Postfach 10 12 53, 50452 Cologne, Germany: European Union Aviation Safety Agency (EASA); 2024. Available from:https://w.easa.europa.eu/en/document-library/ general-publications/easa-artificial-intelligence-concept-paper-issue-2. [5] Stefani T, Christensen JM, Anilkumar Girija A, Gupta S, Durak U, Köster F, et al. Automated Scenario Generation from Operational Design Domain Model for Testing AI-based Systems in Aviation. CEAS Aeronautical Journal. 2024 11:1-19. [6]Stefani T, Christensen JM, Hoemann E, Anilkumar Girija A, Köster F, Krüger T, et al. Applying Model-Based System Engineering and DevOps on the Implementation of an AI-based Collision Avoidance System. In: 34th Congress of the International Councilof the Aeronautical Sciences (ICAS). Florence, Italy: CEAS; 2024. p. 1-12. Available from:https://w.icas.org/icas_archive/icas2024/data/papers/ icas2024_0869_paper.pdf. [7]European Union Aviation Safety Agency (EASA). Artificial Intelligence Roadmap 1.0. Postfach 10 12 53, 50452 Cologne, Germany: European Union Aviation Safety Agency (EASA); 2020. Avail- able from:https://w.easa.europa.eu/en/document-library/general-publications/ easa-artificial-intelligence-roadmap-10. [8] MLEAP Consortium. EASA Research – Machine Learning Application Approval (MLEAP) Final Report. European Union Aviation Safety Agency (EASA); 2024. 1. Available from:https://w.easa.europa. eu/en/research-projects/machine-learning-application-approval. [9]Christensen JM, Stefani T, Anilkumar Girija A, Hoemann E, Vogt A, Werbilo V, et al. Formulating an Engineering Framework for Future AI Certification in Aviation. Aerospace. 2025 03;12(6):482. [10] Werner F, Christensen JM, Stefani T, Köster F, Hoemann E, Hallerbach S. Formulating a Learning Assurance-Based Framework for AI-Based Systems in Aviation. Aerospace. 2025 11. 16 A COVERAGE-DRIVEN VERIFICATION FOR AI-BASED SYSTEMS [11]Namiot D, Ilyushin E. On Certification of Artificial Intelligence Systems. Physics of Particles and Nuclei. 2024 06;55(3):343-6. [12]Hirschle M, Kirov D, Aievola R, Adamy J. Coverage-Driven Synthetic Data Generation for Machine Learning Assurance. In: 2024 AIAA DATC/IEEE 43rd Digital Avionics Systems Conference (DASC). IEEE; 2024. p. 1-10. [13] Weissensteiner P, Stettinger G, Khastgir S, Watzenig D. Operational Design Domain-Driven Coverage for the Safety Argumentation of Automated Vehicles. IEEE Access. 2023;11:12263-84. Available from: https://ieeexplore.ieee.org/document/10036064. [14]Diemert S, Casey A, Robertson J. Challenging Autonomy with Combinatorial Testing. In: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE; 2023. p. 214-22. [15] EASA and Collins Aerospace. Formal Methods use for Learning Assurance (ForMuLA); 2023. [16] Stefani T, Christensen JM, Hoemann E, Köster F, Hallerbach S. From High-Dimensional Spaces to Verifiable ODD Coverage for Safety-Critical AI-based Systems. In: Troubitsyna E, de Andrés D, Bitsch F, editors. Computer Safety, Reliability, and Security. Lecture Notes in Computer Science. Valencia, Spain: Springer Nature Switzerland; 2026. p. 146-60. [17]Kullback S, Leibler RA. On Information and Sufficiency. The Annals of Mathematical Statistics. 1951 03;22(1):79-86. [18]Belov DI, Armstrong RD. Distributions of the Kullback–Leibler divergence with applications. British Journal of Mathematical and Statistical Psychology. 2011 04;64(2):291-309. [19]Bonnici V. A Maximum Value for the Kullback–Leibler Divergence between Quantized Distributions. Information. 2024;15(9). Available from: https://w.mdpi.com/2078-2489/15/9/547. [20]Kvalseth T. On Normalized Mutual Information: Measure Derivations and Properties. Entropy. 2017 11;19:631. [21]Pearson K. X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science. 1900 07;50(302):157-75. [22]Kennedy-Shaffer L. Before p<0.05 to Beyond p<0.05: Using History to Contextualize p -Values and Significance Testing. The American Statistician. 2019 03;73(sup1):82-90. [23] Fisher RA. In: Statistical Methods for Research Workers. Springer New York; 1992. p. 66-70. [24]Cramér H. Mathematical Methods of Statistics (PMS-9). No. 9 in Princeton Mathematical Series. Princeton, NJ: Princeton University Press; 1946. [25] Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Routledge; 1988. [26] Gibbs AL, Su FE. On choosing and bounding probability metrics; 2002. Available from:https://arxiv. org/abs/math/0209021. [27]Julian KD, Kochenderfer MJ. Guaranteeing Safety for Neural Network-Based Aircraft Collision Avoidance Systems. In: 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC). San Diego, CA, USA: IEEE; 2019. p. 1-10. [28]Christensen JM, Zaeske W, Beck J, Friedrich S, Stefani T, Girija A, et al. Towards Certifiable AI in Aviation: A Framework for Neural Network Assurance Using Advanced Visualization and Safety Nets. In: 2024 AIAA DATC/IEEE 43rd Digital Avionics Systems Conference (DASC). San Diego, CA, USA: IEEE; 2024. p. 1-9. Available from: https://ieeexplore.ieee.org/document/10749321. 17

Instant research discovery