Paper deep dive
CUBICS: Situation-aware performance estimation for safety-relevant ML components
Benjamin Herd, Jessica Kelly, Mario Trapp
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/23/2026, 2:22:22 AM
Summary
The paper introduces CUBICS, a context-modular framework for estimating the performance and safety of Machine Learning Components (MLCs) in safety-relevant applications. Unlike traditional Bayesian approaches that assume a single global failure probability, CUBICS partitions the Operational Design Domain (ODD) into discrete situations based on context dimensions (e.g., weather, lighting). It uses Subjective Logic (SL) to represent and update situation-specific assumptions and probabilistic guarantees (contracts) based on field data. This allows for modular, per-component safety assurance that accounts for context-dependent performance variations and uncertainty, avoiding the need for monolithic system-level statistical models.
Entities (9)
Relation Signals (9)
Benjamin Herd → affiliatedwith → Fraunhofer Institute for Cognitive Systems IKS
confidence 99% · Benjamin Herd1... Affiliation: 1Fraunhofer Institute for Cognitive Systems IKS
Mario Trapp → affiliatedwith → Technical University of Munich
confidence 99% · Mario Trapp2... Affiliation: 2Technical University of Munich
Jessica Kelly → affiliatedwith → Fraunhofer Institute for Cognitive Systems IKS
confidence 99% · Jessica Kelly1... Affiliation: 1Fraunhofer Institute for Cognitive Systems IKS
CUBICS → uses → Subjective Logic
confidence 97% · CUBICS... represented and updated in a Bayesian manner using Subjective Logic (SL)
CUBICS → partitions → Operational Design Domain
confidence 95% · CUBICS partitions the operational design domain (ODD) into discrete situations
CUBICS → targets → Machine Learning Components
confidence 95% · situation-aware performance estimation of safety-relevant ML components
CUBICS → evaluatedon → YOLOv12
confidence 93% · We then apply CUBICS to a YOLOv12-based object detector
YOLOv12 → trainedon → BDD100K
confidence 93% · YOLOv12-based object detector trained on BDD100K
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Machine learning (ML) is a key technology driving innovation today, but ensuring ML safety remains a major challenge for safety-related applications. A promising idea is to build proven-in-use arguments from field data, e.g. by running ML components (MLCs) in shadow mode or within safety envelopes so that their outputs can be monitored as 'safe probes' without affecting safety. These probes can then be used to build a statistical argument about field performance in a Bayesian way. However, many Bayesian field-data approaches in safety engineering model failures as a simple Bernoulli (or binomial) process with a single global failure probability and i.i.d. trials, which is rarely adequate for MLCs whose performance depends strongly on context. Statistical evidence is also about coverage of relevant situations, including edge cases, and building a single integrated statistical model for the entire system is usually not feasible. To address these challenges, this paper introduces CUBICS, a context-modular framework for per-component, situation-aware performance estimation of safety-relevant ML components. CUBICS partitions the operational design domain into situations and, for each safety-relevant component, defines a set of situation-specific assumptions and probabilistic guarantees that are represented and updated in a Bayesian manner using Subjective Logic (SL). By combining these guarantees with beliefs about how often each situation occurs, CUBICS derives an overall risk estimate for each component without requiring a monolithic system-level statistical model, and thus provides a building block for modular, field-data based safety assurance.
Tags
Links
- Source: https://arxiv.org/abs/2608.16564v1
- Canonical: https://arxiv.org/abs/2608.16564v1
Trouble viewing inline? Open PDF directly →
Full Text
72,777 characters extracted from source content.
Expand or collapse full text
CUBICS: Situation-aware performance estimation for safety-relevant ML components Benjamin Herd1, Jessica Kelly1, and Mario Trapp2 1 Affiliation: 1Fraunhofer Institute for Cognitive Systems IKS Garching, Germany benjamin.herd, jessica.kelly@iks.fraunhofer.de Affiliation: 2Technical University of Munich Garching, Germany mario.trapp@tum.de Abstract Machine learning (ML) is a key technology driving innovation today, but ensuring ML safety remains a major challenge for safety-related applications. A promising idea is to build proven-in-use arguments from field data, e.g. by running ML components (MLCs) in shadow mode or within safety envelopes so that their outputs can be monitored as ‘safe probes’ without affecting safety. These probes can then be used to build a statistical argument about field performance in a Bayesian way. However, many Bayesian field-data approaches in safety engineering model failures as a simple Bernoulli (or binomial) process with a single global failure probability and i.i.d. trials, which is rarely adequate for MLCs whose performance depends strongly on context. Statistical evidence is also about coverage of relevant situations, including edge cases, and building a single integrated statistical model for the entire system is usually not feasible. To address these challenges, this paper introduces CUBICS, a context-modular framework for per-component, situation-aware performance estimation of safety-relevant ML components. CUBICS partitions the operational design domain into situations and, for each safety-relevant component, defines a set of situation-specific assumptions and probabilistic guarantees that are represented and updated in a Bayesian manner using Subjective Logic (SL). By combining these guarantees with beliefs about how often each situation occurs, CUBICS derives an overall risk estimate for each component without requiring a monolithic system-level statistical model, and thus provides a building block for modular, field-data-based safety assurance. Index Terms: continuous safety assurance, machine learning, safety contracts I Introduction The safety assurance of systems based on Machine Learning (ML) remains a paramount challenge that necessitates the development of innovative paradigms, such as continuous safety assurance. This often involves operating ML components (MLCs) in shadow mode, where they function without any safety-relevant impact on the system to collect in-field data and establish a foundation of statistical evidence. Bayesian statistics provide a useful approach here: by treating each probe as a success or failure, a prior over the failure probability can be updated as data accumulates, yielding both an estimated failure probability and a measure of uncertainty. In this paper, we use the term ‘performance’ to refer to task-level ML metrics such as recall, and ‘reliability’ to refer to the probability that a safety-relevant component behaves correctly (i.e. achieves adequate performance) in a given situation. In practice, this is often instantiated as a simple Beta-Bernoulli model with a single global failure probability. For ML components whose performance is strongly context-dependent, this global i.i.d. assumption may mask situation-specific insufficiencies. For instance, a vision-based system is more likely to fail in heavy rain than in clear conditions. To address this, we introduce CUBICS, a framework for per-component, situation-aware performance estimation of safety-relevant MLCs that can serve as a building block for modular, field-data-based safety assurance. CUBICS partitions the operational design domain (ODD) into discrete situations defined by context dimensions such as weather or lighting conditions. For each safety-relevant component and situation, CUBICS defines a set of situation-specific assumptions and probabilistic guarantees about relevant failure modes (e.g. false-negative detection). Within each situation we assume approximately stationary failure behaviour, so simple Bernoulli/Beta-style updates remain valid; at operation time, new evidence only updates the guarantees of the situations in which it may have occurred. This makes explicit which situations are well covered by evidence and where uncertainty about the component’s behaviour remains high. Even with situation-based models, it can be argued that compiling sound, fully integrated statistical evidence for system-level claims (such as the positive risk balance of an automated driving system) is infeasible and prone to modelling errors. Building and maintaining a system-wide Bayesian network (BN) that captures all relevant dependencies is particularly difficult in practice. In current industrial practice, complete statistical evidence for an entire system is typically neither available nor required; instead, system safety cases combine quantitative arguments for selected components with qualitative reasoning for the rest. CUBICS therefore uses Subjective Logic (SL) as the primary calculus for representing and updating component-level, situation-specific statistical evidence using subjective opinions with explicit belief and uncertainty. Prior work [10] has formalised safety contracts in SL and derived a corresponding assurance argument pattern that separates assumption sufficiency from system resilience. That work focused on a single, context-agnostic binary contract and static evidence. In this paper, we extend this approach towards explicit ODD-based situation modelling, per-component context-modular contracts, and runtime updates. For each safety-relevant component, CUBICS defines an assume–guarantee contract. The assumptions capture beliefs about the context in which the component operates (e.g. distributions over weather or lighting conditions), while the guarantees express probabilistic statements about the component’s safety-relevant behaviour, conditioned on these situations. Both assumptions and guarantees are represented as SL opinions that can be updated over time. By combining assumptions about situations with their conditional guarantees, CUBICS derives (i) situation-specific guarantees and (i) a marginal, context-weighted overall risk contribution for the component. In principle, contracts of upstream components (e.g. perception) can be used as evidence for assumptions of downstream components (e.g. planning), but each contract can rely on different types of arguments (SL-based models, traditional safety analyses, or qualitative reasoning). In this paper, we instantiate and evaluate CUBICS for an individual component and the composition of contracts across multiple components is left as future work. This preserves component-level modularity over ODD situations, uses Bayesian statistical methods where they are most needed for ML components, and avoids a single, fragile system-wide statistical model in favour of feasible, component-specific models that can feed into the overall safety case. In particular, we make the following contributions: • We introduce CUBICS, a contract-based methodology for safety-relevant ML components that structures assumptions as SL opinions over context dimensions and guarantees as SL opinions over situation-conditional failure behaviour, and derives both per-situation guarantees and a marginal, context-weighted risk contribution for an individual component. • We develop a context-aware update mechanism for CUBICS contracts that allows for the distribution of (positive and negative) runtime evidence across situations under context uncertainty. To investigate the effectiveness of our approach, we assess the following three research questions: 1. RQ1 (Internal validity): Can CUBICS recover known, situation-specific reliability patterns in a controlled scenario? 2. RQ2 (Benefit): Does CUBICS yield more informative and situation-specific reliability assessments compared to a pooled Bernoulli model, and how is this advantage impacted by varying data sizes? 3. RQ3 (Sensitivity): How sensitive are the guarantees to prior choices, misclassification of context, and data scarcity? In the evaluation, we first instantiate CUBICS in a synthetic case study to assess its internal validity, i.e., whether it can recover known situation-specific reliability patterns and the corresponding marginal guarantee under controlled conditions (RQ1). We then apply CUBICS to a YOLOv12-based object detector trained on BDD100K, evaluating its behaviour on several safety-critical object classes (person, bicycle, car, etc.) and, due to space, report detailed results for the person class. We compare its situation-specific SL guarantees with a single global Bernoulli model to show that CUBICS exposes localized performance deficits and data gaps that the global model masks (RQ2). Finally, we analyse how the resulting guarantees change under different priors, imperfect context information, and varying evidence strength to assess the sensitivity of the approach to these factors (RQ3). The paper is structured as follows. Section I introduces Subjective Logic; Section I presents the CUBICS methodology, including the contract model, situation-based ODD decomposition, and the derivation of conditional and marginal guarantees with continuous updates; Sections IV–V provide experimental results on the application of CUBICS to an ML-based safety-relevant component and demonstrate the resulting context-aware risk assessment; Section VI discusses threats to validity; Section VII reviews related work; Section VIII summarises findings and outlines avenues for future work. I Background I-A Subjective Logic Subjective Logic (SL) [13] is a framework for reasoning under uncertainty that combines ideas from probability theory and Dempster–Shafer evidence theory. Its core data structures are subjective opinions representing an agent’s belief, disbelief, and uncertainty about the truth of a proposition. SL provides algebraic operators for combining and transforming opinions. Depending on whether the underlying domain X is binary (i.e., =x,x¯X=\x, x\) or n-ary (i.e., =x1,…,xKX=\x_1,…,x_K\), opinions are binomial or multinomial. We focus here on multinomial opinions. Definition 1 (Multinomial opinion) Let =x1,…,xKX=\x_1,…,x_K\ be a finite domain of mutually exclusive and collectively exhaustive states. A multinomial opinion over X is a tuple ωX=(X,uX,X) _X=(b_X,u_X,a_X) where: • X=(bx1,…,bxK)b_X=(b_x_1,…,b_x_K) (belief masses) is a distribution of belief over the states, with bxib_x_i the belief mass supporting xix_i being the true state; • uXu_X (uncertainty) is the remaining, uncommitted belief mass and the complement of confidence (1−u1-u); • X=(ax1,…,axK)a_X=(a_x_1,…,a_x_K) (base rates) is an a priori probability distribution over X in the absence of committed belief; and • bxi,u,axi∈[0,1]b_x_i,u,a_x_i∈[0,1] for all i, ∑i=1Kbxi+u=1 _i=1^Kb_x_i+u=1, and ∑i=1Kaxi=1 _i=1^Ka_x_i=1. A vacuous opinion (full uncertainty) is denoted ωV=(,1,) _V=(0,1,a) and an absolute opinion focusing all belief on a single state xix_i is denoted ωxi⊤=(,0,) _x_i =(b,0,a) with bxi=1b_x_i=1 and bxj=0b_x_j=0 for all j≠ij≠ i, for any choice of base rate vector a. I-A1 Constructing Multinomial opinions Given a finite domain =x1,…,xKX=\x_1,…,x_K\, evidence counts =(rx1,…,rxK)r=(r_x_1,…,r_x_K) with rxi≥0r_x_i≥ 0 for each state xix_i, and a non-informative prior weight11 1 W ensures that, as evidence accumulates (i.e. ∑irxi _ir_x_i grows), the uncertainty u decreases accordingly. W is typically set to the cardinality of the domain (W=KW=K), which is equivalent to adding one pseudo-observation to each state. Larger values of W require more evidence for uncertainty to decrease. W, a multinomial opinion can be computed as follows: bxi b_x_i =rxi∑j=1Krxj+W,i=1,…,K = r_x_i _j=1^Kr_x_j+W, i=1,…,K (1) u u =W∑j=1Krxj+W = W _j=1^Kr_x_j+W (2) with base rate vector =(ax1,…,axK)a=(a_x_1,…,a_x_K), where axi∈[0,1]a_x_i∈[0,1] and ∑i=1Kaxi=1 _i=1^Ka_x_i=1. Multinomial opinions correspond to Dirichlet distributions over the categorical probabilities on X. Given (,,W)(r,a,W), the corresponding parameters are αxi _x_i =rxi+axiW,i=1,…,K, =r_x_i+a_x_iW, i=1,…,K, (3) and the expectation value of xix_i is E(xi)=bxi+axi⋅u E(x_i)=b_x_i+a_x_i· u (4) I-A2 Combining opinions SL provides a wide range of combination operators [13]. Combining opinions provides an elegant and intuitive way to combine the underlying distributions, a direct manipulation of which would be significantly more complex. In this paper, we use the following operators (see [13] for full definitions): Multinomial multiplication given independent opinions ωX _X and ωY _Y about variables X and Y which take their values from distinct domains X and Y, the joint opinion on the Cartesian product ×X×Y is computed using multinomial multiplication ωX∧Y=ωX⋅ωY _X Y= _X· _Y. We denote the corresponding joint belief as bxyb_xy, uncertainty as uxyu_xy, and product base rate as axya_xy. Further details on computing multinomial multiplication can be found in [13]. Multinomial deduction given a conditional relationship where conclusion variable Y depends on premise variable X, the deduction operator derives the marginal opinion on Y from an opinion on X as ωY|X=ωX⊚ωY|X _Y\|X= _X _Y X,where: • ωX _X is the multinomial opinion on the premise X. • ωY|X _Y X represents the set of conditional opinions on Y given the mutually exclusive states of X. • ωY|X _Y\|X is the resulting deduced marginal opinion on Y. • ⊚ denotes the deduction operator. Full details for computing the marginal opinion ωY|X _Y\|X are provided in [13]. I The CUBICS methodology CUBICS structures the safety assurance of ML-based components (MLCs) around three central ideas: 1. a situation-based decomposition of the operational design domain (ODD) is performed; 2. modular safety contracts are defined per component and situation; 3. an SL-based approach is used (1) to update beliefs about the current situation and the safety of the component based on operation-time evidence, and (2) to obtain conditional and marginal component safety guarantees. This section explains how these ideas jointly yield a context-aware assessment of a component’s safety guarantees as well as the overall risk contribution that can be embedded into a broader safety case. Throughout this section, we refer to the simplified scenario in Fig. 1 as a running example. Context (C) Situations (S) Expansion of s7s_7 ContractAssume-Guarantee Contract Boundary Rain = Yes, No Wind = Low, High Time = Day, Night 01ωcrain _c_rain 01ωcwind _c_wind 01ωctime _c_time s0s_0: Yes, Low, Day s1s_1: Yes, Low, Night s2s_2: Yes, High, Day s3s_3: Yes, High, Night s4s_4: No, Low, Day s5s_5: No, Low, Night s6s_6: No, High, Day s7s_7: No, High, Night G ∣ s0s_0 G ∣ s1s_1 G ∣ s2s_2 G ∣ s3s_3 G ∣ s4s_4 G ∣ s5s_5 G ∣ s6s_6 G ∣ s7s_7 01G ∥ S Context-Specific Design-Time Argumentation Opinion on Situation ωS7 _S_7 Probabilistic Guarantee ωG|s7 _G s_7 Fractional Evidential Update Mechanism Operation-Time Evidence of G Expected value E(s7)E(s_7)Update Fig. 1: Overview of the Unified CUBICS methodology over a simplified operational design domain. Subjective context opinions over Rain, Wind, and Time of Day make up eight situations (from s0s_0 to s7s_7). Each situation establishes a unique safety contract (G|siG s_i), which are ultimately aggregated into the overarching global guarantee G|SG S. Fractional Update Mechanism Situation Opinion ωs7 _s_7 Field Evidence Stream x∈0,1x∈\0,1\ Weighting Function E(s7)=bs7+as7us7E(s_7)=b_s_7+a_s_7u_s_7 Fractional Evidence Accumulation Δr=E(s7)⋅(x=1) r=E(s_7)·I(x=1)Δs=E(s7)⋅(x=0) s=E(s_7)·I(x=0) Refined Guarantees Updated Conditional Guarantee ωG|s7 _G s_7 Weight P(s7)P(s_7) Fig. 2: The fractional evidential update mechanism. SL situational opinions ωsi _s_i provide a probabilistic weighting signal E(si)E(s_i) to incoming field observations. Binary evidence is accumulated fractionally into Dirichlet parameters (α,β)(α,β), thereby refining the conditional guarantee ωG|si _G s_i as more data is gathered. I-A Situation-based decomposition of the ODD We assume that the relevant ODD of the system can be characterised by a set of discrete context dimensions C1,…,CkC_1,…,C_k (e.g. weather, lighting). Each dimension CiC_i is a finite set of possible values, e.g. C1==,C_1\!=\!Rain\!=\!\Yes,No\ and C2==,ℎC_2\!=\!Wind\!=\!\Low,High\. CUBICS partitions the ODD into situations s∈Ss∈ S, where S=C1×C2×⋯×CkS\!=\!C_1× C_2×·s× C_k, and each situation s=(c1,…,ck)∈Ss=(c_1,…,c_k)∈ S corresponds to a particular combination of context values (e.g. =Rain\!=\!Yes,=ℎWind\!=\!High). Each such situation represents a subspace of the ODD that is assumed to have a distinct performance or risk profile for the component under consideration. For example, with =,Rain\!=\!\Yes,No\ and =,ℎWind\!=\!\Low,High\ we obtain four situations, and adding a third dimension =,ℎTime\!=\!\Day,Night\ yields the eight situations shown in Fig. 1. The key assumption is that, in each situation sis_i, the component’s failure behaviour can be treated as approximately stationary, so that a Bernoulli model of success and failure is more representative. More precisely, individual outcomes of the considered failure mode (e.g. misclassifications of an object) are exchangeable Bernoulli trials with an (approximately) constant failure probability pfail(si)p_fail(s_i). In the contract, the resulting context model (assumptions over CiC_i and S) is linked to per-situation guarantees ωG|si _G s_i as introduced below. I-B Contracts for safety-relevant components For each MLC and situation sis_i as defined above, CUBICS defines a safety contract with two central elements: • Assumptions capture probabilistic beliefs about the current context, i.e. which values the context dimensions take, e.g. Rain=,Rain=\Yes,No\, Wind=,ℎWind=\Low,High\, and thus which situation sis_i the component is believed to be operating in. These beliefs result from environment perception, operational profiles, or scenario-based analyses. As described below, beliefs are represented by multinomial opinions in SL which are then combined into a joint situational assessment. • Guarantees express beliefs about the safety of the component or function, conditional on the current situation. They are underpinned by a safety case with appropriate evidence, such as design-time analyses, field or shadow-mode data, and expert judgement. For each situation sis_i, belief in the guarantee is represented as a probabilistic assessment of the relevant safety claim G, encoded as a conditional opinion ωG|si _G s_i. Assumptions and guarantees are the building blocks of a conditional model that links beliefs about operating in a certain situation sis_i with beliefs about the safety of the component in sis_i based on available evidence. Conceptually, this conditional model resembles a Bayesian Network (BN) but replaces conditional probability tables with conditional probability density functions. More precisely, CUBICS uses SL to represent assumptions and guarantees as follows: • Assumptions are modelled by first forming multinomial opinions ωCi _C_i about each context dimension CiC_i; these opinions are then combined into a joint situational opinion ωS=∏i=1kωCi _S= _i=1^k _C_i. • Guarantees are situation-specific and are thus represented as a set G|S=ωG|si∣si∈S ω_G S=\\, _G s_i s_i \,\ of conditional opinions. Conceptually, G|S ω_G S forms a context-indexed family of opinions, one per situation sis_i. Using the example from Fig. 1, a belief such as “it is likely clear, not windy, and daytime” is first encoded in the individual context opinions (e.g. ω _Rain, ω _Wind, ω _Time) and then combined into the joint situational opinion ωS _S, which in turn links directly to the corresponding per-situation guarantee, e.g. ωG|s4 _G s_4 for (=,=,=)(Rain=No,Wind=Low,Time=Day). The approach is component-level and context-modular: in a system with multiple interacting components, each component is equipped with its own CUBICS contract over ODD situations. In principle, the contracts of ‘upstream’ components (e.g. perception) can provide SL opinions over their guarantees that can be used as evidence for the assumptions of ‘downstream’ contracts (e.g. planning or actuation). In this way, guarantees do not need to be recomputed within a single monolithic probabilistic model: each component maintains its own context-dependent guarantees in the SL space, and the system-level safety case connects these contracts by treating upstream guarantees as inputs to downstream assumptions. This avoids a single, fragile system-wide statistical model in favour of component-specific models that can feed into the overall safety case. Based on this conditional model, both situation-specific guarantees and an overall risk contribution can be derived, as described below. Note that, in this paper, we instantiate and evaluate CUBICS at the level of individual components; the use of composed contracts across multiple components is left for future work. I-C Conditional and marginal contract guarantees A component contract gives rise to two closely related types of guarantees: the conditional view maintains one guarantee per situation – explicitly conditioned on si∈Ss_i∈ S – while the marginal view aggregates these per-situation guarantees into a single overall guarantee under the modelled operational profile. We describe both views below. I-C1 Conditional view (one guarantee per situation) As defined in Section I-B, for each situation si∈Ss_i∈ S we maintain a conditional opinion ωG|si _G s_i over a binary guarantee domain G (e.g. Safe,UnsafeSafe,Unsafe). The collection of these opinions, G|S=ωG|si∣si∈S ω_G S=\\, _G s_i s_i \,\, constitutes the conditional view of the contract. This view supports detailed, context-specific reasoning about the component in each situation. I-C2 Marginal view (one aggregated guarantee over all situations) In contrast, the marginal view considers the contract from the perspective of the actual operation of the system, where different situations occur with different probabilities according to the operational profile and context model. Here we are interested in a single overall opinion about the guarantee G that already takes into account how likely each situation is. CUBICS obtains this overall opinion by combining: (i) the SL opinions over the context dimensions (assumptions), which induce a belief over situations sis_i, and (i) the conditional per-situation opinions ωG|si _G s_i. This combination is performed by multinomial deduction in SL, which yields a single marginal opinion ωG||S _G||S which, for simplicity, we abbreviate with ωG _G: ωG≔ωG||S=ωS⊚G|S _G _G||S= _S ω_G|S where ωS=∏i=0kωCi _S= _i=0^k _C_i denotes the joint opinion over all possible context dimensions CiC_i, computed via multinomial conjunction.22 2 In practice, this corresponds to combining the SL opinions over the context variables into a joint belief over situations. Intuitively, multinomial deduction weighs each per-situation guarantee ωG|si _G s_i by the belief that the corresponding situation sis_i actually occurs, and aggregates the results into one overall conclusion. The marginal opinion ωG _G can be understood as the component’s overall risk contribution under the modelled operational profile, i.e. as a probabilistic assessment (with uncertainty) of whether its guarantee will be satisfied during operation when all situations and their probability of occurrence are taken into account. In summary, the conditional view provides a set G|S ω_G S of per-situation guarantees ωG|si _G s_i, while the marginal view provides one aggregated opinion ωG _G. Both are derived from the same contract: engineers can inspect per-situation guarantees for detailed, context-specific reasoning, and use the marginal guarantee as a compact summary of the component’s overall risk contribution. I-D Continuous update of situational beliefs and guarantees CUBICS follows a Bayesian perspective: an important design principle is that neither the situational opinions ωCi _C_i about the individual context dimensions nor the conditional guarantee opinions ωG|si _G s_i stated in a contract are static. As additional evidence becomes available, both sets of opinions are updated, which, in turn, influences both the situational joint opinion set S ω_S and the inferred marginal risk opinion ωG _G. CUBICS focuses on runtime updates from statistical evidence obtained during operation or testing. Here, operation-time data from field or shadow-mode deployments, as well as additional test datasets, are mapped into SL opinions and fused with the existing contextual and conditional per-situation opinions ωCi _C_i and ωG|si _G s_i. In Bayesian terms, this corresponds to an update of the underlying Dirichlet parameters. Over time, this process refines beliefs and reduces uncertainty about both the occurrence probability of each situation sis_i as well as the guarantees that can be given in sis_i. CUBICS uses a fractional update mechanism to account for uncertainty in the assumptions about which situation we are in. Given opinion ωS=∏i=1kωCi _S= _i=1^k _C_i derived from opinions ωCi _C_i on context parameters, we compute the expected probability E(si)E(s_i) of being in situation sis_i according to Eq. 4 and perform an update of the conditional guarantees ωG∣si _G s_i by weighing observed evidence x by the corresponding expectation value: ri(t+1) r_i^(t+1) =ri(t)+E(si)[xt=1], =r_i^(t)+E(s_i)\,I[x_t=1], (5) si(t+1) s_i^(t+1) =si(t)+E(si)[xt=0]. =s_i^(t)+E(s_i)\,I[x_t=0]. (6) where rir_i and sis_i are the accumulated success and failure counts for situation sis_i, and [⋅]I[·] is the indicator function. For example, if ωS _S projects E(s0)=0.8E(s_0)=0.8 and E(s1)=0.2E(s_1)=0.2 at time t and we observe a failure (xt=0x_t=0), then the failure counts are updated by s0(t+1)=s0(t)+0.8s_0^(t+1)=s_0^(t)+0.8 and s1(t+1)=s1(t)+0.2s_1^(t+1)=s_1^(t)+0.2. An overview of the fractional update mechanism is provided in Figure 2. By operating in the SL space, CUBICS allows practitioners to transparently integrate heterogeneous statistical evidence sources and domain knowledge, and to reflect both supporting and adverse observations in the evolving contract. The resulting per-situation guarantees and marginal risk assessments are continuously updated in a Bayesian way and may serve as an input to higher-level assurance reasoning without requiring a monolithic, system-wide statistical model. IV Experimental Set-Up In this section, we describe our experimental setup, including the Python implementation of CUBICS, architectural decisions, training pipeline details, and used datasets. The source code is available at https://doi.org/10.5281/zenodo.21932463. IV-1 CUBICS Python implementation The methodology has been implemented as a Python framework that provides executable support for defining context-dependent safety contracts, representing them in SL, and updating them continuously as new evidence becomes available. Conceptually, it mirrors the structure described in Section I. Context dimensions (e.g. weather, lighting, time of day) are represented as multinomial SL opinions and combined into a joint situational opinion over the Cartesian product of all context dimensions. For each resulting situation, the framework maintains a conditional opinion that captures the corresponding contract guarantees. These per-situation guarantees are linked to the situational opinion via the SL deduction operator, yielding a single marginal guarantee that summarises the component’s overall risk contribution under its operational profile. The python code also implements the fractional evidential update mechanism introduced in Section I-D. Incoming operation-time evidence is weighted by the expected value of the situations in which it may have occurred and accumulated into the corresponding per-situation guarantees. The implementation builds on standard Python scientific computing tools, facilitating integration into simulation environments and safety-assurance workflows. IV-2 Datasets Experiments were conducted using the Berkeley DeepDrive (BDD100K) dataset [28]. It was selected due to its high diversity in driving scenarios and the inclusion of frame-level environmental attributes. The dataset consists of 100,000 annotated images, divided into a 70k/10k/20k split for training, validation, and testing, respectively. To support our analysis, the standard 10k validation set was stratified into mutually exclusive sub-datasets based on the weather attribute present in the original JSON annotations. The evaluated conditions include: Clear, Overcast, Rainy, Snowy, and Foggy. We pragmatically restrict our analysis to binary context dimensions, given the constraints of the existing BDD100K annotations. However, CUBICS itself is not limited to binary variables; because the SL opinions over context dimensions are multinomial, they can naturally represent multiple values per dimension. The models were trained to detect 10 standard classes: person, rider, car, truck, bus, train, motorcycle, bicycle, traffic light, and traffic sign. IV-3 Model Architecture and Training We used the YOLOv12-Large (YOLOv12l) architecture [23] as our primary baseline, initialized with pre-trained COCO weights to accelerate training. To preserve the fidelity of small, distant objects (such as pedestrians in low-visibility conditions), the input image resolution was scaled to 1024×10241024× 1024 pixels. Training was conducted over 100 epochs. IV-4 Hardware All experiments were executed on a single NVIDIA GeForce RTX 4090 GPU with 24GB of VRAM. V Evaluation In this section we provide quantitative results addressing RQ1, RQ2, and RQ3. To address RQ1, we provide a synthetic case study to highlight the CUBICS methodology. We then evaluate RQ2 and RQ3 on the experimental setup outlined in Section IV. V-A RQ1 (Internal validity): Can CUBICS recover known, situation-specific reliability patterns in a controlled scenario? To address RQ1, we use a synthetic scenario that mimics a perception component operating in an environment with known situation-specific failure rates. This controlled setup allows us to verify whether CUBICS can reconstruct the underlying per-situation reliability patterns33 3 Here, by reliability patterns we mean the mapping from each situation sis_i to its ground-truth probability of correct behaviour (e.g., classification accuracy) and the corresponding differences between situations (e.g., some situations being systematically harder than others). and the resulting marginal guarantee from observed success/failure outcomes. We consider an abstract perception-based function (e.g. a pedestrian detector) in the context of automated driving whose failure behaviour is assumed to depend on three discrete context dimensions: Rain=Yes,NoRain=\Yes,No\, Wind=High,LowWind=\High,Low\, and ToD=Day,NightToD=\Day,Night\. The Cartesian product of these dimensions induces eight distinct situations, each of which is assumed to have an approximately stationary failure probability. For each situation, we define a conditional guarantee about the component’s safety-relevant behaviour and combine them with beliefs about the occurrence of the situations to obtain a marginal, context-weighted risk contribution. All steps of this process are implemented and executed in Python. Design-time Following the CUBICS methodology, the assessment of the ML component is represented as a safety contract that is evaluated against the situations induced by the context dimensions RainRain, WindWind, and ToDToD. The contract assumptions capture beliefs about the current context and are represented as multinomial SL opinions ωR _R, ωW _W, and ωT _T, which are then combined into a joint situational opinion ωS _S over the eight situations S=Rain×Wind×ToDS=Rain× Wind× ToD using multinomial multiplication. The contract guarantees are concerned with a single safety-relevant property G of the component (e.g. no safety-relevant miss occurs in a critical detection scenario) and are made explicit on a per-situation basis. At design time, engineers construct a safety case that argues, for each situation si∈Ss_i∈ S, for bounds on the probability that G is satisfied when the component operates in sis_i. The detailed argumentation (tests, analyses, expert judgement) is not modelled explicitly here; instead, its outcome is summarised as an SL opinion ωG|si _G s_i over the binary domain G=Safe,UnsafeG=\Safe,Unsafe\. The collection of these eight conditional opinions ωG|S=ωG|si∣si∈S _G S=\ _G s_i s_i∈ S\ constitutes the conditional part of the contract and forms the design-time initialisation of the case study. Fig. 3: Progression of beta distributions across situational guarantees. Dotted red line represents true failure probability across situations. The shaded beta distributions represent the final conditional guarantees. Operation-time. At operation-time, we initialize both the context opinions ωR _R, ωW _W, ωT _T and the conditional guarantees ωG|S _G S with near-complete uncertainty (vacuous priors). This represents a system deployed in shadow mode with minimal prior empirical data, relying instead on operation-time evidence to learn the behaviour across all conditions. We simulate a deployment over N=5,000N=5,000 discrete operational cycles. Each situation sis_i is assigned a ground-truth physical failure rate pfail(si)p_fail(s_i). These rates reflect intuitive physical constraints; for instance, the failure probability is highest in the most adverse conditions (Rain=YesRain=Yes, Wind=HighWind=High, ToD=NightToD=Night) and lowest in ideal conditions. During each simulated cycle, a situation is sampled from the ground-truth context distribution. The agent records this occurrence and accumulates environmental evidence to continuously update the base context opinions ωR,ωW _R, _W, and ωT _T. Simultaneously, the component’s performance w.r.t. its guarantees is simulated against the target situation’s pfail(si)p_fail(s_i), yielding a binary success (r) or failure (s) outcome. This specific evidence is mapped to the corresponding conditional opinion ωG|si _G s_i. In this synthetic setting, the true situation is observed without ambiguity, so the situational opinion ωS _S collapses to a point mass (i.e. P(si)=1P(s_i)=1 for the realised situation and 00 otherwise). Consequently, the fractional evidential update mechanism described in Section I-D reduces to a standard per-situation Bernoulli/Beta update; in scenarios with uncertain context, the same mechanism would distribute each observation fractionally across multiple situations according to E(si)E(s_i). Results Figure 3 illustrates the evolution of the conditional safety guarantees (ωG|si _G s_i) for all eight context situations over N=5,000N=5,000 operational cycles. At iteration 0, the beliefs are initialized with vacuous priors, represented by the flat, uniform distribution (grey line). As the system accumulates evidence, epistemic uncertainty decreases, and the Beta distributions progressively sharpen (iterations 50 and 500). By iteration 5,000, the expected value of every conditional opinion has converged towards its respective ground-truth success probability (red dashed line). The final distributions exhibit varying degrees of uncertainty (represented by the width and peak density of the curves), accurately reflecting the underlying occurrence probabilities E(si)E(s_i) of the respective situation sis_i. Frequently encountered situations (e.g., s1s_1 and s6s_6) accumulate substantial evidence, resulting in highly confident, narrow density peaks. Conversely, rare situations (e.g., s2s_2 or s5s_5) accumulate less operational evidence, naturally resulting in wider distributions. This demonstrates that the CUBICS framework correctly preserves uncertainty where data is scarce, preventing the system from making overconfident safety claims about rare edge cases. RQ1 Summary: In this controlled scenario, CUBICS converges towards the known situation-specific reliability patterns: per-situation guarantees ωG|si _G s_i recover the predefined failure rates within their uncertainty bounds. This indicates that, when its assumptions hold, CUBICS correctly captures situation-dependent reliability from observed success/failure outcomes. V-B RQ2 (Benefit): Does CUBICS yield more informative and situation-specific reliability assessments compared to a global pooled Bernoulli estimate, and how is this advantage impacted by varying available data sizes? To answer RQ2, we analyse the YOLOv12l object detector on the BDD100K validation set as described in Section IV, and compare two different strategies for deriving a system-level recall guarantee from the same body of evidence for the person class. We repeated the analysis for other object classes and observed qualitatively similar behaviour; we therefore focus the following description primarily on the person class for brevity. Each ground-truth person instance detection is treated as a Bernoulli trial, so we use true positive (TP) and false negative (FN) counts as evidential successes and failures for the true-positive rate (recall): 1. Global pooled guarantee. We consider a global pooled estimate for comparison, where all TP and FN counts are pooled into a single opinion ωgp _gp that yields one context-agnostic overall guarantee. 2. SL marginal guarantee. The ODD is decomposed into k situations. A conditional opinion ωG|si _G|s_i is formed from the per-situation evidence, a multinomial context opinion ωS _S captures the situation distribution, and the marginal ωG|S _G\|S is obtained via SL deduction [13]. This also yields one global guarantee, but it preserves both evidential and contextual uncertainty. Situational guarantees expose masked performance insufficiencies. Table I reports the per-situation opinions for the person class. The global pooled estimate concentrates at E=0.653E=0.653 with negligible uncertainty (u≈10−4u≈ 10^-4), suggesting high confidence in a moderate recall. CUBICS reveals that this figure conceals situation-specific performance insufficiencies: Clear + Daytime achieves a comparable expected probability (E=0.650E=0.650), whereas Foggy + Dawn/dusk collapses to E=0.250E=0.250 with uncertainty elevated by two orders of magnitude (u=0.083u=0.083). This adverse scenario contains only 22 observations, making the wide posterior an honest reflection of evidential scarcity. TABLE I: CUBICS opinions for the person class. The global pooled estimate masks the performance collapse and high uncertainty in adverse scenarios such as Foggy + Dawn/dusk. Scenario TP (r) FN (s) Bel. (b) Disbel. (d) Unc. (u) Exp. Prob (E) Global Pooled 16 101 8 549 0.6531 0.3468 0.0001 0.6532 Clear + Daytime 2 758 1 484 0.6499 0.3497 0.0005 0.6501 Rainy + Night 218 157 0.5782 0.4164 0.0053 0.5809 Foggy + Dawn/Dusk 5 17 0.2083 0.7083 0.0833 0.2500 Fig. 4: Beta PDFs for the person class. The global pooled estimate’s narrow peak (black) conceals the performance gap between a benign scenario (green) and an adverse one (red). Figure 4 visualizes this contrast. The global pooled estimate and Clear + Daytime distributions overlap as narrow peaks near p≈0.65p≈ 0.65, while the Foggy + Dawn/dusk distribution is wide and shifted leftward – both lower in expectation and less certain. A single pooled recall figure is therefore neither representative of benign conditions (where the system performs adequately) nor of adverse ones (where it does not). Marginal vs. global pooled guarantees When the situational opinions are aggregated via SL deduction into a single marginal guarantee ωG|S _G\|S and compared with the global pooled opinion (Figure 5, Table I), the expected guarantee E(Safe)E(Safe) is identical (0.65320.6532 for the person class). The critical difference is the uncertainty mass: the marginal carries u=1.74×10−3u=1.74× 10^-3, roughly 21×21× the global pooled value of u=8.10×10−5u=8.10× 10^-5. This gap arises because the pooled view compresses two additional sources of uncertainty into a single narrow posterior: (i) conditional uncertainty from data-scarce situations, whose wide per-situation posteriors are averaged away by the pooled count, and (i) context uncertainty from the finite-sample estimate of how often each situation occurs in operation. The global pooled opinion still reflects finite-sample uncertainty via its u, but only at this aggregated level and without distinguishing where the evidence comes from. This behaviour is consistent across classes (with results for Bike, Car, and Truck shown in Table I). Fig. 5: Marginal (top, blue) vs. context-agnostic pooled guarantee (bottom, red). The marginal distribution is wider and reflects the compositional uncertainty that pooling conceals. TABLE I: Marginal vs. global pooled guarantee across classes. Both share the same expected value; the SL marginal exposes ∼15–25× 15--25× more uncertainty depending on the class. Uncertainty Mass (u) Class []E[x] Marginal Pooled Uncertainty Ratio Person 0.6532 1.74×10−31.74× 10^-3 8.10×10−58.10× 10^-5 21.4× Bike 0.5695 1.52×10−21.52× 10^-2 1.00×10−31.00× 10^-3 15.2× Car 0.7775 2.40×10−42.40× 10^-4 1.00×10−51.00× 10^-5 24.6× Truck 0.6575 4.82×10−34.82× 10^-3 2.30×10−42.30× 10^-4 21.0× Depending on the situation at hand, the global pooled guarantee may therefore be overconfident: its narrow credible interval implicitly assumes constant operating conditions. The SL marginal provides a calibrated account of what is known (and what remains uncertain) about system-level reliability across a heterogeneous ODD. CUBICS thus enables more targeted decisions than a global model. An engineer can (i) identify situations where the expected recall is below a required threshold and uncertainty is low, indicating systematic weakness and the need for model improvement, and (i) identify situations where recall appears adequate but uncertainty remains high, indicating that additional situation-targeted testing is required before deployment. In our example, the Foggy + Dawn/dusk situation falls into the latter category, which would justify either restricting deployment in such conditions or prioritising data collection and testing there. From a system safety perspective, these per-situation and marginal guarantees can be used to derive concrete ODD restrictions, prioritise additional testing and data collection in high-uncertainty situations, and allocate the detector’s quantitative contribution to system-level risk in the safety case. RQ2 Summary. CUBICS identifies localized performance insufficiencies (e.g. Foggy + Dawn/dusk: E=0.25E=0.25, u=0.083u=0.083) that the global pooled guarantee masks behind a narrow, overconfident estimate. The SL marginal guarantee retains substantially more uncertainty than the pooled context-agnostic view while recovering the same expected value, replacing false precision with transparent, actionable safety insights. V-C RQ3 (Sensitivity): How sensitive are the guarantees to prior choices, misclassification of context, and data scarcity? We assess the sensitivity of the CUBICS framework to the choice of base-rate prior, misclassification of context, and data scarcity as follows: Prior Choice To investigate the effect of the choice of prior on the results, we vary the prior and observe the effect on the situation-specific guarantees. Table I demonstrates that in a data-rich ODD (sCDs_CD = Clear Day, N=4,242N=4,242), the prior has very little impact on the safety guarantee. Because uncertainty u is so low (0.00050.0005), the evidence (r,sr,s) overwhelms the prior and anchors the expected probability E(Safe∣scd)E(Safe s_cd) at 0.650.65. In contrast, the data-scarce ODD (sfds_fd = Foggy Dawn, N=22N=22) exhibits significant uncertainty (u=0.0833u=0.0833). Here, the prior becomes a stronger lever: shifting from a pessimistic (a=0.1a=0.1) to an optimistic (a=0.9a=0.9) prior moves the guarantee by roughly 3%3\%. This behavior is safety-desirable; it ensures that in situations where empirical data is lacking, the assessment is forced to rely on explicit prior assumptions rather than making overconfident claims from insufficient evidence. TABLE I: RQ3: Sensitivity to the Base Rate Prior (a) in Data-Rich vs. Data-Scarce ODDs Prior (a) Clear Day (N=4,242N=4,242) E(Safe∣scd)E(Safe s_cd) Foggy Dawn (N=22N=22) E(Safe∣sfd)E(Safe s_fd) 0.10.1 0.6499(u=0.0005)0.6499\ (u=0.0005) 0.2167(u=0.0833)0.2167\ (u=0.0833) 0.50.5 0.6501(u=0.0005)0.6501\ (u=0.0005) 0.2500(u=0.0833)0.2500\ (u=0.0833) 0.90.9 0.6503(u=0.0005)0.6503\ (u=0.0005) 0.2833(u=0.0833)0.2833\ (u=0.0833) Context Misclassifications We simulated an imperfect context classifier by transferring varying percentages of detection evidence (from 0% to 50%) out of a benign, data-rich source environment (Clear Day) and falsely attributing it to two distinct adverse Operational Design Domains (ODDs): • Data-rich target: Snowy Night (substantial evidence). • Data-scarce target: Foggy Dawn (minimal evidence). By incrementally “poisoning” the target ODDs with evidence from the easier, higher-performing Clear Day scenario scds_cd, we tracked the resulting shift in the expected probability E(Safe∣s[sn/fd])E(Safe s_[sn/fd]). The results are shown in Figure 6. As the proportion of misclassified evidence increases, the per-situation guarantees for the target ODDs become more optimistic and less distinct from the original source ODD. For the data-rich target, the effect on E(Safe∣ssn)E(Safe s_sn) remains moderate because misclassified samples are diluted by existing evidence; for the data-scarce target, even small amounts of misclassification have a visible impact on E(Safe∣sfd)E(Safe s_fd). In both cases, however, the associated uncertainty u remains higher than in the original source ODD and increases with poisoning, and the overall behaviour gradually approaches that of a global Bernoulli model. This indicates that context misclassification mainly erodes some of the benefit of situation-specific guarantees rather than leading to overconfident, falsely precise assessments. Fig. 6: Sensitivity of situation-specific expected guarantee E(Safe∣s[sn/fd])E(Safe s_[sn/fd]) to context misclassification. Increasing amounts of Clear Day evidence are incorrectly assigned to (a) a data-rich target (Snowy Night) and (b) a data-scarce target (Foggy Dawn), gradually inflating guarantees and reducing the distinction between situations. Data Scarcity We describe the advantage of CUBICS in scenarios where data is scarce by examining the edge case of detecting a truck during a foggy night. In our test set, this scenario occurred only once and resulted in a missed detection (TP=0TP=0, FN=1FN=1). A traditional frequentist evaluation yields a recall of exactly 0.0%0.0\% – a brittle and overconfident penalty derived from a single sample. CUBICS avoids this deterministic trap. Recognizing the severe data scarcity (N=1N=1), it assigns a dominant epistemic uncertainty mass (u=0.6667u=0.6667). Consequently, the expected guarantee is not anchored to 0%0\%, but is mathematically pulled toward the non-informative base rate prior (a=0.5a=0.5), resulting in E(Safe∣sfn)=0.3333E(Safe s_fn)=0.3333. This demonstrates CUBIC’s capacity for graceful degradation: rather than outputting an absolute (and statistically insignificant) safety claim, it formally bounds the localized risk with uncertainty and signals to the safety monitor that this specific ODD requires further targeted testing. Overall, across all three experiments, realistic variations in priors, context classification quality, and sample size affect the numerical values of E(Safe∣sx)E(Safe s_x) and u, but did not change the qualitative risk ordering between situations. RQ3 Summary: CUBICS behaves robustly with respect to prior choices and moderate context misclassification: in data-rich situations, different reasonable priors are quickly overridden by evidence and lead to similar guarantees, whereas in data-scarce situations prior assumptions visibly influence E(Safe∣sx)E(Safe s_x) but are accompanied by high uncertainty u. Context misclassification gradually reduces the benefit of situation-specific guarantees and makes the model resemble a global Bernoulli view, but for realistic misclassification levels the qualitative conclusions and risk ordering between situations remain stable. Overall, CUBICS is sensitive in the intended way: it exposes uncertainty where evidence is weak, without being overly brittle to plausible modelling and sensing imperfections. VI Threats to Validity VI-A Internal Situational decomposition relies on correct per-image metadata; as a consequence, mislabelled metadata routes evidence to the wrong conditional opinion and distorts the corresponding situational and marginal guarantee. All opinions use the standard SL non-informative prior weight W=2W=2; while reasonable for uninformed initialisation, it does not reflect domain-specific prior knowledge (e.g. from expert judgement), so posteriors for situations with sparse evidence remain sensitive to this choice. Finally, the update function treats every detection outcome within a situation as an exchangeable Bernoulli trial. In practice, FNs differ in safety relevance — missing a nearby, fully visible pedestrian is more critical than missing a distant, heavily occluded one — and failures may cluster in sub-conditions (e.g. dark clothing at night) that the current situation granularity does not distinguish. Incorporating severity-weighted evidence or finer situation partitions could address this, at the cost of increased data requirements. VI-B External Validity The evaluation uses TP/FN counts for a single object class, which naturally map to a binomial model. A threat to external validity is that many perception failures are continuous; extending CUBICS to such metrics would require replacing the binomial with an appropriate continuous-outcome model. Scalability is also an issue: with k context dimensions of cardinality nin_i, the number of situations grows as ∏ini _in_i, and a realistic ODD with dozens of dimensions would cause a combinatorial explosion, with many cells having sparse evidence and guarantees dominated by the prior. Hierarchical or factored representations of the situation space are a natural mitigation. It is important to note that SL itself does not pose a computational bottleneck: the SL operators we use scale linearly with the number of situations. Furthermore, the framework assumes a fixed failure rate within each situation, although ML model performance can drift over time. Because cumulative fusion weights all historical evidence equally, outdated observations are never discounted, which may yield overconfident guarantees that no longer reflect current system behaviour. Finally, the marginal guarantee assumes that the defined situations are exhaustive. Novel or adversarial conditions not anticipated in the ODD specification fall outside the framework, and the marginal, which aggregates over defined situations weighted by observed frequencies, is blind to scenarios that have never been encountered or conceived. This limitation is shared by all scenario-based approaches but remains important: the CUBICS guarantee is only as complete as the underlying ODD partition. VII Related work VII-A Continuous safety assurance and runtime monitoring Continuous safety assurance [20] aims to keep a system’s safety argument valid throughout its lifecycle by continuously monitoring operational behaviour, detecting assurance deficits, and updating the safety case as the system or environment changes [5]. Runtime monitoring and runtime assurance architectures operationalise this by observing system and environment variables, checking conformance to safety constraints, and triggering mitigation or enforcement actions when constraints might be violated [3, 12]. While such approaches can detect violations quickly and support dynamic safety cases, they typically provide qualitative signals (“constraint violated/not violated”) rather than quantified confidence in failure probabilities, and they usually lack mechanisms to (a) incorporate context distributions and (b) propagate monitoring evidence compositionally through modular arguments. Moreover, membership in scenarios or contexts (rain, night, etc.) may itself be uncertain, so updates must reflect contextual ambiguity and convert runtime evidence into quantitative, uncertainty-aware guarantees that evolve over time. VII-B Assurance confidence and reliability modelling Quantitative assurance confidence estimation aims to turn heterogeneous evidence (e.g., tests, field data, near misses) into explicit confidence levels in safety claims. In autonomous driving, naïve mileage-based demonstrations for rare catastrophic events are impractical in terms of miles required to obtain meaningful statistical bounds [14], which motivates statistical and especially Bayesian approaches that explicitly model uncertainty and allow incremental updating of confidence as evidence accumulates. Standards such as ISO 26262 recognise proven-in-use style arguments but do not prescribe how to construct them for ML components whose performance is context-dependent and evolves over time. In this setting, quantitative assurance methods must cope with rare events, nonstationary failure rates, and evolving uncertainties. Bayesian inference provides a probabilistic framework for updating hypotheses based on evidence. In safety contexts, it has been used to model component or system failure probabilities and refine them as test or field data accumulate [7, 4, 11]. For perception systems and autonomous vehicles, Bayesian reliability assessments explicitly consider that failure rates may depend on environmental conditions, moving beyond a single global Bernoulli parameter [1, 19]. This is crucial in the rare-event regime, where context-sensitive models are needed to make efficient use of limited evidence and pure mileage-based demonstration is infeasible. Bayesian approaches also underpin assurance-oriented test planning, where sample sizes are chosen to achieve specified posterior confidence targets [26]. However, prior probabilities can be subjective, evidence may be non-representative or correlated, and building and maintaining a single integrated probabilistic model for an entire system is often impractical and fragile, especially for ML-heavy systems. These limitations motivate more modular and context-structured approaches. Subjective Logic (SL) [13] is an evidential reasoning framework that extends classical probability theory with explicit representation of uncertainty and base rates, and provides operators to combine and transform probabilistic opinions. It offers a convenient bridge between statistical models (e.g., Beta/Dirichlet distributions) and argumentation-level reasoning about confidence and defeaters. In assurance confidence estimation, SL has been used to reinterpret Baconian-style confidence as Beta distributions and visualise it in an opinion triangle [6], to make formal inferences within assurance arguments [29], to model relationships and dependencies between evidential artefacts [8, 2], to model the behaviour of defeaters [9], and to formalise safety contracts and derive argument structures [10]. Building on this work, CUBICS can be seen as a concrete instantiation of the SL contract formalism for ML components: a single, context-agnostic binary contract is refined into a set of situation-indexed guarantees over context-dependent failure behaviour and equipped with an explicit Bayesian update scheme driven by field and test data. This positions SL as a suitable intermediate calculus between Bayesian statistical models and safety-case style reasoning. In the context of continuous safety assurance, these properties are particularly valuable. SL enables assurance confidence to be updated as operational data and concerns evolve, without requiring a single monolithic system-level probabilistic model. Statistical evidence (e.g., Beta–binomial updates over pass/fail data) can be mapped into SL opinions, modified by defeaters or external information, and mapped back to probabilistic representations that feed into modular safety contracts. This interplay between Bayesian statistics and SL underpins CUBICS: situation-specific probabilistic guarantees and SL-based update rules yield quantitative, uncertainty-aware assurance confidence in a modular, context-aware manner. Prior work already showed that reliability claims should be weighted by how systems are actually used. Musa defines reliability assessment around a quantitative characterisation of expected use, while later Bayesian approaches model reliability over input/context-space partitions and explicitly treat uncertainty and drift in the operational profile [17, 18]. In parallel, covariate-dependent reliability models in classical reliability engineering use environmental factors, degradation trends, or time-varying stresses to estimate failure risk as a function of operating conditions rather than a single global rate [21, 30]. At the probabilistic level, CUBICS instantiates these ideas as a context-stratified reliability model over situation-specific failure probabilities; the next section connects this to ODD-/scenario-based safety analysis for autonomous systems. VII-C Context-aware and scenario-based safety analysis Context-aware safety analysis treats system behaviour as explicitly dependent on operating conditions represented by an ODD specification. In automated driving, the ODD is the set of operating conditions under which a driving automation system is intended to function. Scenario-based safety evaluation builds on this by structuring verification and validation around scenarios derived from or constraining the ODD. The PEGASUS methodology [27] promotes systematic scenario catalogues and links scenario-based testing to safety argumentation. Complementary work on logical scenarios formalises abstraction levels (functional, logical, concrete) and parameterisation for safety validation within the ODD [24]. In the SOTIF context, scenario-based approaches are used to identify triggering conditions for functional insufficiencies and to construct accelerated tests that expose SOTIF-related hazards. For autonomous systems, scenario- and ODD-based safety research argues that pure mileage is infeasible, and that safety evidence should be organised around scenario coverage, environmental conditions, and risk exposure frequencies [15, 25, 22]. Recent reviews [31] distinguish between ordinary and safety-critical scenarios, discuss bidirectional interaction modelling and critical scenario generation (including corner cases), and emphasise the need to quantify exposure frequencies and residual risks per scenario. A survey on risk assessment for autonomous driving by Lu et al. [16] shows that many methods ultimately evaluate risk at the scenario level, but still struggle with (i) uncertain or evolving operational profiles, (i) long-tail and unknown scenarios, and (i) aggregating heterogeneous scenario evidence into modular, uncertainty-aware guarantees suitable for safety cases. CUBICS directly addresses these gaps by (i) decomposing the ODD into situations that play the role of logical scenarios, (i) associating each situation with a continuously updated Bayesian guarantee over safety-relevant behaviour, and (i) maintaining explicit beliefs over situation occurrence. This turns the scenario/ODD partition into a set of probabilistic “cells” whose contributions to component-level risk can be aggregated using contracts, while preserving uncertainty and supporting through-life updates from operational evidence. VIII Conclusions and future work This paper introduced CUBICS, a methodology for continuous, situation-aware performance estimation of safety-relevant ML components. Building on earlier work on SL-based safety contracts [10], we extend the focus to ML components whose behaviour depends on an explicit ODD. Assumptions are represented as SL opinions over context dimensions, guarantees as SL opinions over situation-conditional failure behaviour, and SL deduction is used to derive both per-situation guarantees and a marginal, context-weighted risk contribution for each component. In this work, SL is used to represent and update component-level statistical evidence in the opinion space. At the probabilistic level, CUBICS instantiates a context-stratified hierarchical model over situation-specific failure probabilities. The contribution is not a novel theory, but the way this structure is embedded into situation-specific SL contracts is: (i) contracts are maintained per component and per situation rather than in a monolithic system model, (i) runtime evidence is incorporated via a context-aware fractional update mechanism that respects uncertainty in ODD labelling, and (i) the resulting per-situation and marginal opinions can be used as building blocks in wider safety assurance arguments. Our synthetic case study showed that, under its modelling assumptions, CUBICS recovers known situation-specific reliability patterns (RQ1). The YOLO/BDD100K study demonstrated that it exposes localised performance deficits and data gaps that a global Bernoulli model hides, and yields a more accurate component-level assessment via its marginal guarantee (RQ2). Sensitivity analyses indicated that results behave robustly under reasonable changes in priors, context misclassification, and data volume, while correctly reflecting increased uncertainty where evidence is scarce (RQ3). Several limitations suggest directions for future work. First, the current instantiation focuses on binomial failure modes for a single component; we plan to extend CUBICS to continuous-valued metrics and to small multi-component chains to demonstrate contract composition in practice. Second, the combinatorial growth of situations and the assumption of stationarity within each cell call for hierarchical or factored situation models and temporal discounting schemes compatible with SL. Finally, we have only partially exploited the resilience branch of the contract structure in [10]; integrating explicit resilience goals and runtime monitors is an important step towards end-to-end, through-life safety assurance. Acknowledgment The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Energy within the project “Safe AI Engineering – Sicherheitsargumentation befähigendes AI Engineering über den gesamten Lebenszyklus einer KI-Funktion”. The authors would like to thank the consortium for the successful cooperation. We also thank the anonymous reviewers for their helpful feedback. References [1] M. Berk, H. Kroll, O. Schubert, B. Buschardt, and D. Straub (2017) Bayesian test design for reliability assessments of safety-relevant environment sensors considering dependent failures. In WCX™ 17: SAE World Congress Experience, Cited by: §VII-B. [2] S. Burton, B. Herd, and J. Zacchi (2024) Uncertainty-aware evaluation of quantitative ML safety requirements. In SAFECOMP (Workshops), Lecture Notes in Computer Science, Vol. 14989, p. 391–404. Cited by: §VII-B. [3] D. Cofer, I. Amundson, R. Sattigeri, A. Passi, C. Boggs, E. Smith, L. Gilham, T. Byun, and S. Rayadurgam (2020) Run-time assurance for learning-enabled systems. In NASA Formal Methods Symposium, p. 361–368. Cited by: §VII-A. [4] E. Denney, G. Pai, and I. Habli (2011) Towards measurement of confidence in safety cases. In ESEM, p. 380–383. Cited by: §VII-B. [5] E. Denney, G. Pai, and I. Habli (2015) Dynamic safety cases for through-life safety assurance. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2, p. 587–590. Cited by: §VII-A. [6] L. Duan, S. Rayadurgam, M. P. E. Heimdahl, O. Sokolsky, and I. Lee (2016) Representation of confidence in assurance cases using the beta distribution. In HASE, p. 86–93. Cited by: §VII-B. [7] B. Guo (2003) Knowledge representation and uncertainty management: applying Bayesian Belief Networks to a safety assessment expert system. In International Conference on Natural Language Processing and Knowledge Engineering, Vol. , p. 114–119. Cited by: §VII-B. [8] B. Herd and S. Burton (2024) Can you trust your ML metrics? Using Subjective Logic to determine the true contribution of ML metrics for safety. In SAC, p. 1579–1586. Cited by: §VII-B. [9] B. Herd, J. Kelly, J. Zacchi, C. Heinemann, and S. Diemert (2025) Integrating defeaters into Subjective Logic-based quantitative assurance arguments. In 2025 20th European Dependable Computing Conference (EDCC), Vol. , p. 141–149. External Links: Document Cited by: §VII-B. [10] B. Herd, J. Zacchi, and S. Burton (2024) A deductive approach to safety assurance: formalising safety contracts with Subjective Logic. In SAFECOMP (Workshops), Lecture Notes in Computer Science, Vol. 14989, p. 213–226. Cited by: §I, §VII-B, §VIII, §VIII. [11] C. Hobbs and M. Lloyd (2012) The application of Bayesian Belief Networks to assurance case preparation. In S, p. 159–176. Cited by: §VII-B. [12] K. L. Hobbs, M. L. Mote, M. C. Abate, S. D. Coogan, and E. M. Feron (2023) Runtime assurance for safety-critical systems: an introduction to safety filtering approaches for complex control systems. IEEE Control Systems Magazine 43 (2), p. 28–65. Cited by: §VII-A. [13] A. Jøsang (2016) Subjective logic. Vol. 3, Springer. Cited by: §I-A2, §I-A2, §I-A2, §I-A, item 2, §VII-B. [14] N. Kalra and S. M. Paddock (2016) Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability?. Transportation research part A: policy and practice 94, p. 182–193. Cited by: §VII-B. [15] C. W. Lee, N. Nayeer, D. E. Garcia, A. Agrawal, and B. Liu (2020) Identifying the operational design domain for an automated driving system through assessed risk. In 2020 IEEE Intelligent Vehicles Symposium (IV), p. 1317–1322. Cited by: §VII-C. [16] D. Lu, H. Du, Z. Wu, and S. Yang (2025) Risk assessment in autonomous driving: a comprehensive survey of risk sources, methodologies, and system architectures. Autonomous Intelligent Systems 5 (1), p. 24. Cited by: §VII-C. [17] J. D. Musa (2002) Operational profiles in software-reliability engineering. IEEE software 10 (2), p. 14–32. Cited by: §VII-B. [18] R. Pietrantuono, P. Popov, and S. Russo (2020) Reliability assessment of service-based software under operational profile uncertainty. Reliability Engineering & System Safety 204, p. 107193. Cited by: §VII-B. [19] P. Popov (2025) Dynamic safety assessment of autonomous vehicle based on multivariate Bayesian inference (DyAVSA). Journal of Reliable Intelligent Environments 11 (3), p. 14. Cited by: §VII-B. [20] P. Schleiss, F. Carella, and I. Kurzidem (2022) Towards continuous safety assurance for autonomous systems. In 2022 6th International Conference on System Reliability and Safety (ICSRS), p. 457–462. Cited by: §VII-A. [21] H. Shyur, E. Elsayed, and J. T. Luxhøj (1999) A general model for accelerated life testing with time-dependent covariates. Naval Research Logistics (NRL) 46 (3), p. 303–321. Cited by: §VII-B. [22] L. Tang, R. Wang, Z. Liu, Y. Liang, Y. Niu, W. Zhu, and Z. Duan (2024) Scenario-based accelerated testing for sotif in autonomous driving: a review. IEEE Internet of Things Journal 12 (2), p. 1453–1470. Cited by: §VII-C. [23] Y. Tian, Q. Ye, and D. Doermann (2025) Yolov12: attention-centric real-time object detectors. arXiv preprint arXiv:2502.12524. Cited by: §IV-3. [24] H. Weber, J. Bock, J. Klimke, C. Roesener, J. Hiller, R. Krajewski, A. Zlocki, and L. Eckstein (2019) A framework for definition of logical scenarios for safety assurance of automated driving. Traffic injury prevention 20 (sup1), p. S65–S70. Cited by: §VII-C. [25] P. Weissensteiner, G. Stettinger, S. Khastgir, and D. Watzenig (2023) Operational design domain-driven coverage for the safety argumentation of automated vehicles. IEEE Access 11, p. 12263–12284. Cited by: §VII-C. [26] K. J. Wilson and M. Farrow (2021) Assurance for sample size determination in reliability demonstration testing. Technometrics 63 (4), p. 523–535. Cited by: §VII-B. [27] H. Winner, K. Lemmer, T. Form, and J. Mazzega (2018) PEGASUS – first steps for the safe introduction of automated driving. In Road Vehicle Automation 5, p. 185–195. Cited by: §VII-C. [28] F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, and T. Darrell (2018) BDD100K: A diverse driving video database with scalable annotation tooling. CoRR abs/1805.04687. Cited by: §IV-2. [29] C. Yuan, J. Wu, C. Liu, and H. Yang (2017) A subjective logic-based approach for assessing confidence in assurance case. International Journal of Performability Engineering 13 (6), p. 807. Cited by: §VII-B. [30] H. Zheng, X. Kong, H. Xu, and J. Yang (2021) Reliability analysis of products based on proportional hazard model with degradation trend and environmental factor. Reliability Engineering & System Safety 216, p. 107964. Cited by: §VII-B. [31] Z. Zhong, Y. Tang, Y. Zhou, V. d. O. Neves, Y. Liu, and B. Ray (2021) A survey on scenario-based testing for automated driving systems in high-fidelity simulation. arXiv preprint arXiv:2112.00964. Cited by: §VII-C.