Paper deep dive
Reliable Remediation Impact Prediction for Black-Box Security Ratings
Nada Hanad, Mehdi Acheli, Ali NourEldin, Mohamed Sellami, Walid Gaaloul
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/21/2026, 3:07:58 AM
Summary
The paper proposes a surrogate-based approach to predict the impact of remediation actions on black-box security ratings without exposing the proprietary scoring engine. It explicitly models checkpoint applicability to improve prediction reliability and introduces a reliability layer using Conformal Risk Control to identify unstable predictions. Evaluated on 5,188 organization configurations, the method improves score prediction accuracy and provides cautious interpretation guidelines for remediation impacts.
Entities (8)
Relation Signals (6)
Security Rating Platforms → provides → Numerical Score
confidence 95% · aggregate them into a numerical score that is then used in procurement, cyber insurance, and third party risk management
Surrogate Model → uses → Checkpoint Applicability
confidence 95% · The surrogate predicts scores from organization configurations while explicitly representing checkpoint (i.e., a security check) applicability
BitSight → isvendorof → Security Rating Platforms
confidence 94% · Vendors such as BitSight [4], SecurityScorecard [25], and Board of Cyber [5] collect signals from organizations’ digital footprint
SecurityScorecard → isvendorof → Security Rating Platforms
confidence 94% · Vendors such as BitSight [4], SecurityScorecard [25], and Board of Cyber [5] collect signals from organizations’ digital footprint
Reliability Layer → uses → Conformal Risk Control
confidence 92% · we build a reliability layer that predicts instability from checkpoint applicability features and attaches selective guarantees using Conformal Risk Control
Surrogate Model → predicts → Remediation Score Impact
confidence 90% · the surrogate predicts the score impact of supported actions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Security rating platforms summarize externally observable cyber exposure and are expected to help organizations prioritize remediation. A platform may want to tell an organization how a candidate remediation action would affect its score, but repeatedly exposing exact score responses can reveal information about the hidden scoring engine. We propose a surrogate based approach for remediation score impact prediction that is designed to respect this opacity constraint. The surrogate predicts scores from organization configurations while explicitly representing checkpoint (i.e., a security check) applicability and the observed checkpoint set. A main challenge is that such predictions are not uniformly reliable: they depend on the amount and structure of the observable checkpoint evidence available for a given configuration. To address this, the approach combines applicability-aware surrogate construction, sensitivity analysis under controlled checkpoint restriction, a reliability layer for identifying unstable predictions, and score-impact prediction for supported remediation actions. Explicit modeling of checkpoint applicability is central throughout: it improves score prediction and provides the feature basis used by the reliability layer to identify unstable cases. We evaluate the approach on a real-world dataset of 5,188 organization configurations from a commercial security rating platform. The results show that the applicability-aware surrogate improves score prediction over simpler feature representations. For remediation, the surrogate predicts the score impact of supported actions, while the reliability layer helps identify cases in which these predicted impacts should be interpreted cautiously.
Tags
Links
- Source: https://arxiv.org/abs/2607.16357v1
- Canonical: https://arxiv.org/abs/2607.16357v1
Trouble viewing inline? Open PDF directly →
Full Text
59,864 characters extracted from source content.
Expand or collapse full text
11institutetext: SAMOVAR, Telecom SudParis, Institut Polytechnique de Paris, Palaiseau, France 11email: nada.hanad@telecom-sudparis.eu Reliable Remediation Impact Prediction for Black-Box Security Ratings Nada HANAD Mehdi Acheli Ali Nour Eldin Mohamed Sellami Walid GAALOUL Abstract Security rating platforms summarize externally observable cyber exposure and are expected to help organizations prioritize remediation. A platform may want to tell an organization how a candidate remediation action would affect its score, but repeatedly exposing exact score responses can reveal information about the hidden scoring engine. We propose a surrogate based approach for remediation score impact prediction that is designed to respect this opacity constraint. The surrogate predicts scores from organization configurations while explicitly representing checkpoint (i.e., a security check) applicability and the observed checkpoint set. A main challenge is that such predictions are not uniformly reliable: they depend on the amount and structure of the observable checkpoint evidence available for a given configuration. To address this, the approach combines applicability-aware surrogate construction, sensitivity analysis under controlled checkpoint restriction, a reliability layer for identifying unstable predictions, and score-impact prediction for supported remediation actions. Explicit modeling of checkpoint applicability is central throughout: it improves score prediction and provides the feature basis used by the reliability layer to identify unstable cases. We evaluate the approach on a real-world dataset of 5,188 organization configurations from a commercial security rating platform. The results show that the applicability-aware surrogate improves score prediction over simpler feature representations. For remediation, the surrogate predicts the score impact of supported actions, while the reliability layer helps identify cases in which these predicted impacts should be interpreted cautiously. 1 Introduction Commercial security rating platforms are increasingly used to summarize cyber exposure from publicly observable evidence such as exposed services, DNS and mail configurations, TLS settings, and publicly reachable web assets. Vendors such as BitSight [4], SecurityScorecard [25], and Board of Cyber [5] collect signals from organizations’ digital footprint and aggregate them into a numerical score that is then used in procurement, cyber insurance, and third party risk management [16, 19]. In practice, such scores are derived from the outcomes of multiple security checks, which we refer to here as checkpoints. Statistics suggest that these scores are operationally meaningful: BitSight states that companies rated at or below 400 on its 250 to 900 scale are five times more likely to experience a publicly disclosed breach than those rated at or above 700 [3], while SecurityScorecard reports that organizations receiving an F grade on its A to F scale are 13.8 times more likely to suffer a breach than those with an A grade [27]. This growing operational role of security rating platforms is also reinforced by regulations. In the European Union, for example, the Digital Operational Resilience Act (DORA) requires financial entities to continuously monitor the information and communication technology risk posed by third party providers [12]. Beyond assigning a score, these platforms are also expected to provide useful remediation guidance. In practice, organizations want to know not only their current rating, but also which concrete actions could improve it and by how much. Providing such guidance, however, may leak information about the underlying scoring logic when different remediation options are explored and their predicted score effects can be compared. This is particularly problematic because security rating platforms are designed to remain black boxes, with providers keeping their exact scoring logic proprietary and undisclosed [19, 30]. In this paper, we address the problem of predicting remediation related score impacts without exposing the underlying scoring engine. Instead of providing organizations with the exact scores corresponding to different candidate post remediation configurations, we propose a machine learning surrogate model that predicts approximate scores, trained on the scoring engine’s historical data, and can therefore be used to predict the score impact of candidate remediation actions without exposing exact score responses of the proprietary scoring engine. Because the surrogate only approximates the original scoring engine, its outputs remain predictions rather than direct evaluations of the proprietary function. However, using a surrogate model to approximate a proprietary scoring engine introduces non uniform deployment-time prediction quality across different organization configurations, meaning that the resulting score predictions are not equally reliable for all configurations. This pattern is particularly visible for lower scoring configurations, whose associated scores are harder to predict. These lower scoring configurations also tend to have fewer applicable checkpoints for evaluation. This suggests that the cases on which the surrogate performs less accurately are not random, and that part of this variation in prediction quality may be driven by the size and structure of the applicable checkpoint set associated with a given organization configuration. This links our problem both to work on informative missingness, where the pattern of what is observed can itself carry predictive information, and to work on predictive reliability, where the goal is to identify when model outputs should be trusted. More broadly, this can be viewed as a machine learning problem of prediction reliability under heterogeneous measurement conditions, in a setting where surrogate score predictions are used to assess candidate remediation actions [1, 20, 21, 26]. This observation leads to four research questions. First, if prediction quality varies with the number of applicable checkpoints, how sensitive are deployment-time surrogate score predictions to variations in the number of applicable checkpoints, that is, security checks that are relevant and observable for a given organization configuration? (RQ1RQ1) Second, what is the impact of explicitly modeling checkpoint applicability on surrogate score prediction? (RQ2RQ2) Third, can checkpoint applicability identify which organization configurations are prone to unstable predictions, and can formal reliability guarantees be attached to accepted predictions? (RQ3RQ3) Fourth, can the surrogate reliably predict the score impact of user simulated candidate remediation actions for a given organization configuration? (RQ4RQ4) To answer RQ1RQ1, we study how surrogate predictions change when the observed checkpoint set is deliberately restricted at deployment-time. For RQ2RQ2, we compare predictions from surrogate representations with and without explicit checkpoint applicability information. For RQ3RQ3, we build a reliability layer that predicts instability from checkpoint applicability features and attaches selective guarantees using Conformal Risk Control [1]. For RQ4RQ4, we use the surrogate to predict the score impact of user simulated candidate remediation actions and compare these predictions against trusted reference scores, while also testing whether predicted instability helps identify cases that should be interpreted more cautiously. Specifically, we put forward the following contributions: (i) We propose a surrogate based score-prediction approach for black box security rating platforms, designed to assess candidate remediation changes without directly exposing the proprietary scoring engine. As part of this approach, we enrich the surrogate features with checkpoint applicability and related measurement structure. (i) We introduce a reliability layer that uses checkpoint applicability information to identify organization configurations that are prone to unstable surrogate predictions, and we study how formal reliability guarantees can be attached to accepted predictions. (i) We evaluate the proposed approach along three dimensions: surrogate score prediction and the impact of applicability aware features through an ablation study; prediction sensitivity and reliability under controlled restriction of the observed checkpoint set; and remediation support, by validating predicted score impacts for user simulated candidate remediation actions against trusted reference scores. The rest of the paper is organized as follows. Section 2 introduces the motivating example and the problem setting. Section 3 reviews related work. Section 4 presents the proposed approach. Section 5 describes the study setting, evaluation protocol, and experimental results. Section 6 discusses the findings and their limitations. Section 7 concludes. 2 Motivating Example To motivate our work, we consider a realistic scenario in which an organization monitors and improves its security posture through a commercial security rating platform illustrated in Fig. 1. Such platforms typically cover multiple analysis domains (e.g., Attack Surface and Mail) derived from externally observable evidence. For simplicity, however, we restrict the motivating example to the Mail domain. The platform inspects a set of mail-related security checks (see Fig. 1, (1)), for example checks associated with mail authentication records, domain-level mail configuration, and exposed mail services, and aggregates the resulting observations into an analysis domain score (see Fig. 1, (2)). We call such security checks checkpoints. The outcomes of these checkpoints are then processed by the platform’s internal scoring engine to produce a numerical score on a 0–100 scale, which is returned to the user alongside a description of the issues uncovered by the checkpoints. As shown in Fig. 1, these scores and identified issues are then stored in the platform repository (3). From the organization’s perspective, the returned Mail score is useful but not sufficient. Knowing this score is only part of the problem. The organization also wants to answer a natural what-if question: “what would happen to the score if I implemented a candidate remediation action, such as strengthening a mail authentication setting or correcting a misconfigured mail-related service?” This corresponds to the lower-left query shown in Fig. 1, which is later addressed through the surrogate interaction in step (6). The practical need is therefore not merely to observe the current score, but to evaluate the score impact of possible remediation actions before deployment. Figure 1: Motivating example for the Mail analysis domain On one hand, estimating the score impact of candidate remediation actions is useful for remediation assessment. On the other hand, directly exposing exact scores for many candidate remediation actions would reveal too much about the proprietary scoring logic. This practical dilemma motivates the use of a surrogate model, i.e., a machine learning model trained to approximate the platform’s internal scoring behavior, that predicts scores from an organization configuration while avoiding direct exposure of the underlying scoring engine. In Fig. 1, this transition is represented by using the historical scores and issues stored in the repository to learn the surrogate scoring behavior (4), and then using the surrogate to predict score impacts for candidate remediation actions (5). This setting is complicated by two factors. First, not all checkpoints apply to all organizations. Applicability depends on the technical conditions that make a checkpoint relevant, so two organizations in the same analysis domain may be evaluated through different sets of applicable checkpoints. For example, if an organization does not deploy a given mail security mechanism, then checkpoints that assess properties of that mechanism are inapplicable. Missing checkpoint outcomes are therefore often structural rather than accidental, because a checkpoint may be absent simply because the technical conditions that make it relevant are not satisfied for the assessed organization [17, 20, 21, 24, 26]. Second, the scoring engine that maps checkpoint outcomes to a score is proprietary and undisclosed [19, 30]. A platform may display the current score, but it does not reveal the exact function that would map every hypothetical post-remediation configuration to its corresponding score. The surrogate therefore addresses the original objective shown in Fig. 1, namely providing remediation impact feedback despite engine opacity. However, introducing the surrogate does not by itself solve the full problem. It creates a second layer of challenges, also shown in Fig. 1, that concern how accurately the surrogate predicts scores, how to detect when its predictions are unstable, and whether its predicted score impacts are reliable enough to support remediation assessment. Problem setting. We now abstract the motivating example into a more general problem. We consider a black box security rating system that evaluates organizations through analysis domain specific checkpoints and assigns a numerical score from the resulting observations [19, 30]. For a given organization at a given observation time, only a subset of the full checkpoint set may be applicable. We refer to the resulting analysis domain specific observation, consisting of the applicable checkpoints and their observed outcomes, as an organization configuration. The first challenge is predictive. If a surrogate model is used to predict scores from organization configurations, average predictive accuracy alone is not sufficient. Prediction quality may vary across configurations because different configurations are associated with different sets of applicable checkpoints. This is the predictive challenge highlighted in Fig. 1, and is the basis of RQ1RQ1 and RQ2RQ2. This motivates our analysis of sensitivity to checkpoint availability and whether explicit modeling of checkpoint applicability improves surrogate score prediction. The second challenge is reliability. If prediction quality is not uniform across configurations, then a system needs a way to identify configurations that are more likely to yield unstable surrogate predictions. This corresponds to the reliability challenge in Fig. 1, which motivates RQ3RQ3. This motivates the design of a reliability layer based on checkpoint applicability information, together with formal reliability guarantees attached to accepted predictions [1]. The third challenge is actionability. The goal is not for the model to invent remediation actions on its own. Rather, a user specifies a candidate remediation action, the corresponding post-change configuration is constructed, and the surrogate is used to predict the resulting score and thus the impact associated with that action. In this setting, the key empirical question is whether these predicted score impacts are reliable enough to support remediation assessment in practice. This is the actionability challenge in Fig. 1, and it motivates RQ4RQ4. 3 Related Work This section reviews related work on commercial security rating systems and external cyber measurement (Section 3.1), on missing data and informative missingness in predictive models (Section 3.2), and on reliability in predictive models (Section 3.3). 3.1 Security Rating Systems Commercial security rating systems are widely used in practice, but their meaning and validity remain debated [19, 30]. Keskin et al. [19] compared several non intrusive scoring platforms and found substantial disagreement across vendors evaluating the same organizations. Their results raise a basic interpretability question: when two platforms observe the same public facing surface and disagree, it becomes unclear what exactly each score is measuring. Woods and Böhme [30] reached a related conclusion at a broader level in their systematization of cyber risk quantification. They showed that observable signals can be useful, but that the link between such signals and the broader notion of cyber risk is indirect and depends strongly on the design of the measurement process. Earlier measurement work also showed that organization level security assessment is feasible, but highly sensitive to modeling choices. Noroozian et al. [22] combined heterogeneous abuse datasets and used Item Response Theory to estimate provider security performance as a latent trait. Tajalizadehkhoob et al. [28] similarly showed that conclusions about responsibility for weak security depend strongly on how the measurement problem is structured. Taken together, these studies support the idea that external cyber measurement can provide useful signals, while showing that its interpretation depends on the scoring design itself. In the broader literature on black-box model auditing, surrogate models have been used to analyze opaque predictive systems. Tan et al. [29], for example, use transparent student models to mimic black-box risk scores for auditing. However, their focus is model auditing rather than surrogate reliability under varying measurement conditions, and they do not study score changes under hypothetical interventions. Overall, these works focus on validity, interpretation, or disagreement across external cyber scores, or on auditing opaque risk scores through surrogate distillation. They do not address how surrogate prediction behavior varies with measurement structure, such as checkpoint applicability and the size of the observed checkpoint set associated with a given organization configuration, nor whether such a surrogate can predict score changes under hypothetical interventions such as user simulated candidate remediation changes. 3.2 Missing Data and Informative Missingness in Predictive Models Missingness refers to the absence of values that could in principle have been observed in the data. Rubin [24] introduced the classical taxonomy of missingness mechanisms, and Heckman [17] showed that the process determining what is observed can itself affect inference. In predictive settings, Lipton et al. [20] showed that modeling missingness patterns explicitly can improve clinical time series classification. Sharafoddini et al. [26] similarly highlighted that missingness in intensive care data can themselves be informative rather than reflecting random noise. Van Ness et al. [21] further showed that missing indicators can remain useful predictive signals even in higher dimensional settings. These works are directly relevant to our problem because they show that the pattern of what is observed can itself carry information. In our case, missing checkpoint outcomes are structural rather than random: a value is absent because its checkpoint is not applicable. Checkpoint applicability therefore forms part of the observed measurement basis itself. Existing work on missingness supports the idea that such structural information may carry predictive signal, but it has not been studied in the context of black-box security rating systems. A narrower technical connection can be found in work on feature decrement and missingness shift. Cheng et al. [7] study settings where information available during training becomes unavailable at test time, and Rockenschaub et al. [23] study prediction under changes in the missingness process. These settings are not the main framing of our work, since low checkpoint number is already present in the training data rather than arising only at deployment time. However, they remain relevant to the controlled checkpoint restriction analysis that we use later as a diagnostic sensitivity test. 3.3 Reliability in Predictive Models Work on reliability in predictive models has developed along several closely related lines. Chow [8] introduced the classical reject option formulation, where a model may abstain rather than make a low confidence prediction. El-Yaniv and Wiener [11] studied the foundations of selective classification, and Geifman and El-Yaniv [14, 15] extended this perspective to modern neural predictors. Taken together, these works study how predictive systems can identify cases on which they are less likely to be reliable. A related line of work studies reliability guarantees through conformal methods. Angelopoulos et al. [1] introduced Conformal Risk Control, which allows guarantees to be attached to accepted predictions with respect to a user defined risk criterion rather than accuracy alone. These works are directly relevant to our reliability layer. Nevertheless, prior work has not connected selective prediction or conformal-style reliability guarantees to the reliability of surrogate predictions under varying measurement conditions, nor to the identification of configurations that are prone to instability when the available evidence is limited. Discussion. Existing work addresses important parts of our problem, but in separate settings and for different objectives. We draw on the security rating literature for the broader context of black-box scoring systems, on the missingness literature to motivate checkpoint applicability as informative signal, and on the reliability literature, especially selective prediction and conformal risk control, to design a mechanism for identifying and selectively accepting predictions that are less likely to be unstable. Our contribution is to connect these strands in a single security rating setting through a surrogate-based approach that models checkpoint applicability, studies prediction sensitivity to available checkpoint evidence, adds a reliability layer with formal guarantees, and evaluates remediation score impact prediction without exposing the proprietary scoring function. 4 Applicability-Aware and Reliable Surrogate Prediction Our approach consists of four linked phases as illustrated in Fig. 2. We first construct a surrogate model that predicts analysis domain scores from organization configurations while explicitly encoding checkpoint applicability. We then apply controlled restriction of the applicable checkpoint information as a diagnostic sensitivity test in order to study when surrogate predictions become unstable. On top of this diagnostic, we build a reliability layer that uses checkpoint applicability information to identify configurations whose surrogate predictions are more likely to be unstable, and to define a selective acceptance rule with finite sample risk guarantees. Finally, we use the surrogate to predict the score impact of user simulated candidate remediation actions. Figure 2 summarizes how these four phases are connected. Starting from an organization configuration, represented by outcome features oio_i and applicability features sis_i, the surrogate model produces a baseline score prediction. A restricted version of the same representation is then used in the sensitivity phase to measure score drift under reduced applicable checkpoint information. This drift defines the instability labels used by the reliability phase, while the original representation is also transformed under a candidate remediation action to estimate the corresponding score impact. Together, the four phases define the path from an observed organization configuration to a score prediction, an instability assessment, and a remediation score impact prediction. Figure 2: Overview of the proposed approach. In the following sections we detail these four phases: surrogate construction in Section 4.1, sensitivity analysis in Section 4.2, reliability assessment in Section 4.3, remediation score impact prediction in Section 4.4. 4.1 Surrogate Construction In this phase, we take as input an organization configuration and construct both its feature representation and the surrogate model used throughout the rest of the approach. Let =1,…,MC=\1,…,M\ denote the full set of checkpoints defined by a security rating platform provider for an analysis domain D. For an organization’s configuration i, each checkpoint j∈j is either applicable or not. We denote checkpoint applicability by aij∈0,1a_ij∈\0,1\, where aij=1a_ij=1 means that checkpoint j applies to configuration i and can therefore contribute an observed outcome, while aij=0a_ij=0 means that the checkpoint does not apply. When a checkpoint j is applicable, we denote its observed outcome by zijz_ij. The observed checkpoint set for a configuration i is therefore defined as i=j∈:aij=1, _i=\j :a_ij=1\, (1) and the corresponding number of applicable checkpoints is Ki=∑j=1Maij. K_i= _j=1^Ma_ij. (2) Next, we construct a feature representation xi=(oi,si)x_i=(o_i,s_i) for each organization configuration i. The block oio_i summarizes checkpoint outcomes observed on applicable checkpoints, while sis_i summarizes checkpoint applicability and the structure of the applicable checkpoint set. This design lets the surrogate distinguish weak security evidence from limited observed evidence: missing or sparse values in oio_i are interpreted together with explicit information in sis_i about which checkpoints apply and how much checkpoint evidence is available. The surrogate model is defined as a predictor f^:xi↦y^i, f:x_i y_i, (3) where y^i y_i is the predicted analysis domain score for organization configuration i. This predictor underlies the rest of the approach: it provides the baseline predictions for the sensitivity analysis, defines the predictions whose reliability is assessed later, and is used to estimate the score impact of user-simulated candidate remediation actions. In later sections, specifically in Section 5.2, we evaluate the applicability-aware representation against simpler alternatives that rely only on checkpoint outcome information, in order to determine whether explicit modeling of checkpoint applicability improves surrogate score prediction. This phase therefore provides the model construction used to study the predictive challenge associated with checkpoint applicability, which is evaluated in the context of RQ2RQ2. We now turn to the second phase, which studies how this surrogate behaves when part of the applicable checkpoint information is removed. 4.2 Sensitivity Analysis We use controlled restriction of applicable checkpoints as a diagnostic sensitivity test. The purpose of this test is to examine whether the score predictions produced by the surrogate model constructed in Section 4.1 are sensitive to changes in the checkpoint information available to the model. Starting from the feature representation xix_i of an original organization configuration, we define a restricted representation xi(r)=Rr(xi), x_i^(r)=R_r(x_i), (4) where RrR_r uses the subscript r to index the restriction policy being applied. It is a restriction operator that hides part of the applicable checkpoint information according to a predefined restriction policy while keeping the underlying organization configuration fixed. Intuitively, xi(r)x_i^(r) represents the same organization configuration evaluated from a reduced set of available checkpoint information, not a different underlying configuration. In our setting, the restriction operator therefore masks part of the applicable checkpoint information and recomputes any derived features consistently. The surrogate prediction for this restricted representation is y^i(r)=f^(xi(r)). y_i^(r)= f(x_i^(r)). (5) We quantify prediction sensitivity under restriction through the induced score change: Δi(r)=y^i(r)−y^i. _i^(r)= y_i^(r)- y_i. (6) This quantity provides a direct diagnostic of fragility under restriction and the basis for defining instability in the next phase. For a given threshold τ>0τ>0, we say that organization configuration i yields an unstable prediction under restriction if |Δi(r)|>τ. | _i^(r)|>τ. (7) For later use for the reliability assessment, we define the corresponding unstable-versus-stable binary instability label: ui(r,τ)=(|Δi(r)|>τ). u_i^(r,τ)=1\! (| _i^(r)|>τ ). (8) This label is obtained by thresholding the score drift induced by the restriction procedure and provides the supervision target for the instability predictor introduced in Section 4.3. This diagnostic allows us to study, in a controlled way, whether surrogate predictions become fragile when the available applicable checkpoint information is reduced. It also gives an operational definition of instability that can later be used in the reliability assessment phase. This phase therefore addresses the predictive challenge identified in Section 2 and provides the protocol for RQ1RQ1, which asks whether surrogate score predictions are sensitive to variation in checkpoint number. We next use this definition of instability to train a reliability model that predicts when surrogate outputs should be trusted. 4.3 Reliability Assessment This phase assesses whether a given organization configuration is likely to yield an unstable surrogate prediction and whether predictions can be selectively accepted under a controlled instability risk. Here, instability risk is defined with respect to the diagnostic restriction in Section 4.2. Let sis_i denote the applicability and measurement-structure feature block used by the reliability layer. The layer includes a reliability model defined by g:si↦piunstable g:s_i p_i^unstable (9) where piunstablep_i^unstable is the predicted probability that configuration i yields an unstable surrogate prediction under the diagnostic restriction. Operationally, this phase has three parts. First, the sensitivity analysis defines the binary instability label ui(r,τ)u_i^(r,τ) in Eq. 4.2. Second, the model g is trained on sis_i to predict this label and output piunstablep_i^unstable. Third, this probability is converted into an acceptance rule that retains only predictions whose instability risk is sufficiently low for subsequent score prediction and remediation analysis. The reliability layer serves two purposes. First, it uses sis_i to identify organization configurations whose surrogate predictions are more likely to be unstable. Second, it provides a basis for selectively accepting predictions under finite-sample risk guarantees. To do so, we apply Conformal Risk Control (CRC) [1]. We use a held-out calibration set to define the acceptance rule. The calibration and future configurations to which this rule is applied are assumed exchangeable, which is the standard assumption used by conformal methods [2]. Concretely, let λ∈[0,1]λ∈[0,1] be a threshold on the predicted instability probability. A configuration is accepted when its predicted instability probability is no larger than λ: Aλ=i:piunstable≤λ, A_λ=\i:p_i^unstable≤λ\, (10) The corresponding selective instability risk is ℛ(λ)=[ui(r,τ)∣i∈Aλ]. (λ)=E\! [u_i^(r,τ) i∈ A_λ ]. (11) Given a target risk level α, CRC selects an acceptance threshold λ such that the instability risk among accepted predictions remains below α. We denote the retained threshold by λ⋆λ and the corresponding accepted set by Aλ⋆A_λ . If no feasible threshold is found, then Aλ⋆=∅A_λ = . Under the exchangeability assumption stated above, this yields a finite-sample guarantee with confidence at least 1−δ1-δ that the instability risk among accepted predictions does not exceed α. The practical role of this phase is to determine when the surrogate’s score predictions should be trusted, thereby addressing the reliability challenge identified in Section 2 and providing the method used for RQ3RQ3. 4.4 Remediation Score Impact Prediction The final phase focuses on remediation score impact prediction. Rather than generating remediation actions automatically, the surrogate model takes a user-defined candidate action for a given organization configuration and predicts the resulting score and score impact. We also use the reliability layer to accept such score impacts as dependable. In our experiments, these actions correspond to structured interventions such as deploying SPF or deploying DMARC (Mail analysis domain). For a configuration i considered for remediation, the input to this phase is the observed representation xix_i together with a user-specified candidate action a. The output is a post-action score prediction, the corresponding predicted score impact relative to the current score and whether this impact is accepted. Let ℐI denote the set of organization configurations considered for remediation. For an organization’s configuration i∈ℐi , yiy_i denotes its current score. The score impact is defined relative to yiy_i. Let TaT_a denote the transformation induced by candidate remediation action a. Applying TaT_a produces a post-action organization configuration ia′i _a, with the representation: xia′=Ta(xi). x_i _a=T_a(x_i). (12) The surrogate then predicts the score of ia′i _a as y^ia′=f^(xia′), y_i _a= f(x_i _a), (13) and the predicted score impact of a is Δ^i,arem=y^ia′−yi. _i,a^rem= y_i _a-y_i. (14) We rely on the reliability layer (Section 4.3) to score the post-action configuration. Given ia′i _a, we extract its checkpoint applicability features sia′s_i _a and compute pia′unstable=g(sia′). p_i _a^unstable=g(s_i _a). (15) Exploiting the calibrated acceptance threshold λ⋆λ from Section 4.3, we accept the score as dependable if pia′unstable≤λ⋆. p_i _a^unstable≤λ . (16) Thus, a remediation prediction is retained only when the predicted instability probability of its post-action configuration does not exceed λ⋆λ . An important requirement is that TaT_a must produce a semantically valid post-action representation in surrogate feature space. In particular, when a candidate action changes the technical conditions that determine checkpoint applicability, the relevant checkpoint outcomes, the applicable checkpoint set, and all derived features must be updated consistently. Under this formulation, the surrogate does not recommend actions and does not optimize over possible remediations. Its role is narrower: given an observed organization configuration and a user-simulated candidate action, it predicts the resulting score and the corresponding score impact. The reliability layer then filters non dependable actions. This phase therefore addresses the actionability challenge identified in Section 2 and provides the method used for RQ4RQ4. 5 Experimental Setup and Evaluation This section evaluates the proposed approach on the Mail analysis domain. We first describe the study setting and protocols in Section 5.1, then report results for surrogate construction, sensitivity analysis, reliability assessment, and remediation score impact prediction. 5.1 Study Setting, Data, and Evaluation Protocol We instantiate the problem on the Mail analysis domain of one commercial security rating platform. The archive contains timestamped records that combine checkpoint outcomes with externally observable assets such as domains, URLs, IP addresses, and mail related services. Checkpoints are evaluated only when relevant to those assets, and each detected issue is assigned one of five severity levels, e.g., "Low" or "High". Because the same checkpoint configuration can appear repeatedly, either over time for one organization or across different organizations, we deduplicate observations across checkpoint related fields while ignoring organization identifier and timestamp. This yields 5,188 distinct organization configurations from 375,408 original records. For each retained configuration, the dataset includes the platform assigned Mail score and 40 Mail checkpoints. Each checkpoint contributes outcome features describing the number of measured assets, the worst observed severity, and the count and proportion of each severity level, for a total of 12 outcome fields per checkpoint. We then add one applicability indicator per checkpoint together with structural features summarizing the observed checkpoint set and its measurement pattern, including the number and fraction of applicable checkpoints, summaries of applicability distribution, and configuration level missingness and severity summaries. The final dataset contains 533 columns in total. Our evaluation follows the four phases of Section 4. Table 1 summarizes the protocols. Whenever a split (train/val/test or train/test) is used, it is grouped by organization identifier so that configurations from the same identifier never appear in different partitions of the same experiment. The Repetitions column indicates how many times each split and subsequent experiment is performed. Table 1: Evaluation protocols used in the paper. Evaluation phase Research questions Protocol Repetitions Surrogate Construction RQ2RQ2 80/10/10 20 seeds Sensitivity Analysis RQ1RQ1 80/20 20 seeds Reliability Assessment RQ3RQ3 60/20/20 one seed Remediation Score Impact Prediction RQ4RQ4 No split – For the sensitivity analysis, the source dataset on which the split is applied is a 10% held-out test partition from the previous phase. For reliability assessment too, we consider a held out 20% partition test from the previous experiment and apply the split on it. Since there is no training notion for the fourth phase, no split is applied. 5.2 Surrogate Construction and Score Prediction The enriched applicability aware surrogate is a LightGBM regressor [18] trained to predict the platform assigned Mail score. It uses 500 trees, learning rate 0.03, 63 leaves, row subsampling 0.8, and feature subsampling 0.8, and is reused in the later sensitivity, reliability, and remediation phases. For each grouped seed, the model is trained on the training partition, validation MAE is measured on the 10% validation split, and comparison is based on the held out 10% test split. We compare this model with two simpler baselines: an outcome only representation without explicit applicability or structural features, and an applicability only ablation. Table 2 summarizes the three representations. Table 2: Feature representations used for the first experiment. Representation Contents Input fields Outcome Outcome features 480 Applicability 40 checkpoint applicability indicators plus engineered structural fields 53 Full Outcome features, checkpoint applicability indicators, and engineered structural fields 533 The applicability features encode which checkpoints apply and how much checkpoint evidence is available. Organization identifier and timestamp are excluded from the inputs. Table 3 shows a clear separation between the full model and the two simpler baselines. The full representation achieves the lowest test error at 0.7816±0.03270.7816± 0.0327, versus 1.4554±0.15651.4554± 0.1565 for the outcome only baseline and 4.6388±0.19724.6388± 0.1972 for the applicability only baseline. These results show that explicit applicability and measurement structure features materially improve score prediction when combined with checkpoint outcome features. The applicability only ablation also confirms that applicability information alone is not sufficient. Table 3: Results of the ablation study. Representation Validation MAE Test MAE Full 0.8154 ± 0.0745 0.7816 ± 0.0327 Outcome-only 1.4935 ± 0.1836 1.4554 ± 0.1565 Applicability-only 4.6872 ± 0.1258 4.6388 ± 0.1972 5.3 Sensitivity Analysis To answer RQ1RQ1, we apply controlled checkpoint restriction to the full surrogate. For each seed, checkpoints are ranked by applicability on the 80% training partition, and only the top K checkpoints are retained when evaluating restriction statistics on the held out 20% test partition: K∈40,35,30,25,22,18,14,10.K∈\40,35,30,25,22,18,14,10\. For checkpoints not retained at a given K, the applicability indicator is set to 0 and all associated outcome fields are set to missing. Engineered structural fields are then recomputed to keep the restricted representation internally consistent. For each configuration i, we compare unrestricted and restricted surrogate predictions. We report Δi(K) _i^(K), the mean induced score change, the 90th percentile of |Δi(K)|| _i^(K)|, and the fraction of cases with |Δi(K)|>2| _i^(K)|>2. Table 4 shows that the surrogate remains unchanged at K=40K=40 and K=35K=35, and still essentially unchanged at K=30K=30. Drift stays limited at K=25K=25 and K=22K=22, where the 90th percentile reaches only 0.51450.5145 and 0.78620.7862, and the fraction of cases with |Δi(K)|>2| _i^(K)|>2 remains below 2.3%2.3\%. The main transition occurs between K=22K=22 and K=18K=18. The 90th percentile of the absolute score change rises from 0.78620.7862 to 10.661310.6613, and the fraction of unstable predictions rises from 2.27%2.27\% to 69.38%69.38\%. Below this regime, instability remains high, reaching 83.50%83.50\% at K=14K=14 and 97.43%97.43\% at K=10K=10. We therefore use the transition between K=22K=22 and K=18K=18 as the main diagnostic regime for the rest of the evaluation. Table 4: Surrogate sensitivity under controlled checkpoint restriction. Values are means across seeds with standard deviations in parentheses. K Mean Δi(K) _i^(K) p90(|Δi(K)|)p90(| _i^(K)|) %(|Δi(K)|>2)\%(| _i^(K)|>2) 40 0.0000 (0.0000) 0.0000 (0.0000) 0.00 (0.00) 35 0.0000 (0.0000) 0.0000 (0.0000) 0.00 (0.00) 30 0.0084 (0.0041) 0.0000 (0.0002) 0.00 (0.00) 25 -0.0931 (0.0267) 0.5145 (0.0738) 1.43 (0.56) 22 -0.1355 (0.0251) 0.7862 (0.0860) 2.27 (0.62) 18 4.4723 (0.6027) 10.6613 (0.9142) 69.38 (6.63) 14 8.0524 (0.9394) 15.8331 (1.2710) 83.50 (4.17) 10 20.2486 (1.2704) 35.2518 (2.3442) 97.43 (0.50) 5.4 Reliability Assessment and CRC In the third experiment, instability labels are derived from the K=18K=18 diagnostic restriction. A configuration is labeled unstable when the absolute score change exceeds τ; the main setting uses τ=2τ=2, with additional sweeps over τ∈1,2,3τ∈\1,2,3\. The reliability model from Section 4.3 is a LightGBM classifier using checkpoint applicability features. It is trained on the derived training split, calibrated on the validation split for CRC threshold selection, and evaluated on the held out test split. For selective reliability control, we use CRC [1] on the validation partition. We search over thresholds on predicted instability probability and retain the largest accepted set whose one sided Clopper–Pearson upper bound on accepted instability rate is at most α [9]. We evaluate α∈0.05,0.10,0.15,0.20,0.25,0.30α∈\0.05,0.10,0.15,0.20,0.25,0.30\ with confidence parameter 0.05, then apply feasible thresholds to the test partition. We report ROC AUC [13], average precision [10], Brier score [6], accepted coverage, and observed instability rate among accepted test cases. We now report results for instability prediction and CRC selective acceptance. Table 5 reports instability prediction results across the three thresholds. At τ=1τ=1, the model reaches ROC AUC 0.88720.8872 and Brier score 0.14060.1406. At the main threshold τ=2τ=2, performance improves to ROC AUC 0.92550.9255 and Brier score 0.10010.1001. At τ=3τ=3, ranking remains strong, with ROC AUC 0.92400.9240 and Brier score 0.07230.0723. These results show that checkpoint applicability and measurement structure features are informative for identifying unstable predictions under the diagnostic restriction regime, with the strongest ranking performance at τ∈2,3τ∈\2,3\. Table 5: Instability prediction results for the reliability model across thresholds. τ ROC AUC AP Brier AURC 1 0.8872 0.8995 0.1406 0.2299 2 0.9255 0.8639 0.1001 0.0721 3 0.9240 0.6432 0.0723 0.0210 CRC shows that selective guarantees are feasible, with achievable coverage depending on τ and α. For τ=1τ=1, no feasible threshold is found below α=0.15α=0.15, and accepted test coverage rises from 23.1%23.1\% at α=0.15α=0.15 to 38.5%38.5\% at α=0.30α=0.30. For the main threshold τ=2τ=2, feasibility also begins at α=0.15α=0.15, with coverage increasing from 16.3%16.3\% to 89.4%89.4\% as α is relaxed to 0.300.30. For τ=3τ=3, feasibility starts at α=0.10α=0.10, with 78.8%78.8\% accepted coverage, and reaches 98.1%98.1\% for α≥0.15α≥ 0.15. Overall, the pipeline supports practical selective guarantees at substantial coverage for moderate and larger instability thresholds. 5.5 Remediation Score Impact Prediction To answer RQ4RQ4, we evaluate two candidate remediation action families in the Mail analysis domain: deploying SPF and deploying DMARC. For each action family, source pools are built from observed configurations that already exhibit the corresponding post-action pattern and serve as a reference for the transformation. Post-action configurations are then generated for each eligible configuration (configurations for which the action is plausible in real-world settings) and applicability summaries and structural fields are recomputed for each post-action configuration. Practically, we evaluate predicted score impacts against trusted reference gains. We also examine whether predicted instability distinguishes more dependable remediation predictions from less dependable ones. Table 6 shows that, on eligible cases, both targeted actions produce positive average predicted gains. This is realistic since the actions are remediative in nature and contain fixes that should improve the score. SPF deployment applies to 8 eligible configurations while DMARC deployment applies to 51. The MAE column indicates a moderate mean absolute error between the predicted gains and reference ones with a stronger performance for DMARC than for SPF. We notice a divergence in these MAE compared with the initial test MAE reported for the enriched surrogate model in subsection 5.2. Table 6: Remediation score impact summaries and validation against reference gains. Action family Eligible Mean Predicted Gain MAE Deploy SPF 8 10.11 15.00 Deploy DMARC 51 9.70 8.20 As a solution, we use our reliability layer to help interpret these remediation predictions. We perform a reliability analysis on DMARC post-action cases, split by predicted instability probability. Using a threshold of λ⋆=0.5λ =0.5, 22.6%22.6\% of the cases fall into the higher instability non-accepted group and 77.4%77.4\% into the lower instability accepted group. The non-accepted group has mean predicted gain −1.58-1.58, with 71.4%71.4\% negative score impacts and 9.359.35 MAE. By contrast, the accepted group has a mean predicted gain of 11.2611.26, no negative impact scores and 6.946.94 MAE. This suggests that the reliability layer adds value by separating cases where predicted score gains are more dependable from cases that should be interpreted more cautiously. Importantly, cases where the predicted gain is far from the trusted reference gain tend also to be flagged as unstable. Large remediation errors therefore do not appear as unexplained failures of the surrogate, but as part of a pattern already captured by the instability signal. This provides a useful warning signal for cases in which the predicted gain should not be trusted at face value. Table 7 illustrates this pattern with four anonymized DMARC cases. Cases A–C have high predicted instability and predicted gains near or below zero even though the reference gains are highly positive. Case D has low predicted instability, a large positive predicted gain close to the reference gain. These examples show at the case level that mismatched remediation estimates tend to coincide with high predicted instability, whereas the most accurate and clearly positive estimates appear in low instability cases. Table 7: Illustrative DMARC cases for remediation oriented reliability analysis. The reference gain is reported for comparison. Illustrative case Instab. prob. Predicted Impact Reference gain MAE A 0.823 -1.82 25.00 26.82 B 0.766 -2.65 25.00 27.65 C 0.766 -1.45 25.00 26.45 D 0.154 22.83 30.00 7.17 Overall, the RQ4RQ4 results show that surrogate based remediation score impact prediction is feasible for the evaluated action families. SPF and DMARC deployment produce positive predicted gains on eligible cases, with lower error for DMARC than for SPF. The remediation oriented reliability analysis further shows that cases with large discrepancies from trusted reference gains are typically the same cases flagged as unstable, which makes the reliability layer useful for identifying predictions that should be interpreted cautiously. 6 Discussion and Future Work The results show that checkpoint applicability matters for both score prediction and reliability. When modeled jointly with checkpoint outcomes, it improves surrogate performance over both the outcome only and applicability only baselines. It is also informative for reliability assessment: under controlled checkpoint restriction, the surrogate remains stable when enough evidence is retained, but becomes sharply unstable once the retained set drops from 22 to 18 checkpoints. This should be interpreted as a diagnostic property of the surrogate under restricted checkpoint information, not as a direct statement about the proprietary scoring engine. The remediation results applies the surrogate to user specified actions. For supported actions such as SPF and DMARC deployment, the surrogate predicts positive average score impacts, with stronger agreement to trusted reference gains for DMARC than for SPF. At the same time, the remediation reliability analysis shows that these gains are not equally dependable across configurations. The reliability layer plays therefore the role of a screening mechanism. Several scope limitations remain and point to directions for future work. The empirical study is restricted to the Mail analysis domain of one commercial security rating platform. The remediation analysis currently covers only action families for which we have a reference post-action configuration in our historical dataset. Finally, the CRC guarantees rely on the standard exchangeability assumption between calibration and future cases. In real‑world deployments, however, this assumption may not always hold, particularly under distribution shift or temporal drift.. Ethical and data-handling considerations. Our study uses historical security rating records derived from externally observable organization configurations in the Mail analysis domain. We report only aggregated and anonymized results, do not disclose organization identities or sensitive raw records, and present remediation impact predictions as decision support estimates rather than authoritative security recommendations. 7 Conclusion In this work, we proposed a surrogate-based approach for remediation score impact prediction in black-box security rating platforms. The approach predicts analysis domain scores without exposing the proprietary scoring engine, studies prediction instability through a controlled checkpoint-restriction diagnostic, and adds a reliability layer calibrated with Conformal Risk Control. Evaluation on 5,188 real-world Mail organization configurations shows that checkpoint applicability matters for both prediction and reliability. The applicability aware surrogate improves score prediction over simpler baselines, while the sensitivity analysis reveals a sharp instability transition once the retained checkpoint set becomes too small. The reliability layer then uses applicability features to identify unstable cases and to attach selective guarantees to accepted predictions at practical operating points. For remediation, the surrogate predicts positive score impacts for supported candidate actions with action-dependent accuracy against trusted reference gains. The remediation oriented reliability analysis shows that predicted instability is useful for identifying cases where predicted gains should be interpreted cautiously. Overall, the results show that surrogate-based score feedback in opaque security rating settings should not rely on average predictive accuracy alone. Applicability-aware modeling, combined with reliability-aware assessment, enables remediation score impact feedback while preserving scoring engine opacity. credits 7.0.1 The authors have no competing interests to declare that are relevant to the content of this article. References [1] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster (2024) Conformal risk control. In Proceedings of the 12th International Conference on Learning Representations (ICLR), Cited by: §1, §1, §2, §3.3, §4.3, §5.4. [2] R. F. Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani (2023) Conformal prediction beyond exchangeability. The Annals of Statistics 51 (2), p. 816–845. External Links: Document Cited by: §4.3. [3] BitSight and Marsh McLennan (2022) Study finds significant correlation between BitSight analytics and cybersecurity incidents. Note: https://w.bitsight.com/press-releases/study-finds-significant-correlation-between-bitsight-analytics-and-cybersecurityAccessed April 2026 Cited by: §1. [4] BitSight (2026) BitSight cyber risk intelligence platform. Note: https://w.bitsight.com/platformAccessed: 2026-04-12 Cited by: §1. [5] Board of Cyber (2026) Security rating. Note: https://w.boardofcyber.io/en/nos-solutions/notation-cyberAccessed: 2026-04-12 Cited by: §1. [6] G. W. Brier (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 (1), p. 1–3. External Links: Document Cited by: §5.4. [7] Z. Cheng, Z. Jia, K. Yu, Z. Zhou, and L. Guo (2025) Fully test-time adaptation for feature decrement in tabular data. In Proceedings of the 34th International Joint Conference on Artificial Intelligence (IJCAI), p. 4940–4948. Cited by: §3.2. [8] C. K. Chow (1970) On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory 16 (1), p. 41–46. External Links: Document Cited by: §3.3. [9] C. J. Clopper and E. S. Pearson (1934) The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26 (4), p. 404–413. External Links: Document Cited by: §5.4. [10] J. Davis and M. Goadrich (2006) The relationship between precision-recall and roc curves. In Proceedings of the 23rd International Conference on Machine Learning, p. 233–240. External Links: Document Cited by: §5.4. [11] R. El-Yaniv and Y. Wiener (2010) On the foundations of noise-free selective classification. Journal of Machine Learning Research 11, p. 1605–1641. Cited by: §3.3. [12] European Parliament and Council of the European Union (2022) Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). Note: Official Journal of the European Union, L 333, 27 December 2022Applicable from 17 January 2025 Cited by: §1. [13] T. Fawcett (2006) An introduction to roc analysis. Pattern Recognition Letters 27 (8), p. 861–874. External Links: Document Cited by: §5.4. [14] Y. Geifman and R. El-Yaniv (2017) Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), p. 4878–4887. Cited by: §3.3. [15] Y. Geifman and R. El-Yaniv (2019) SelectiveNet: a deep neural network with an integrated reject option. In Proceedings of the 36th International Conference on Machine Learning (ICML), PMLR, Vol. 97, p. 2151–2159. Cited by: §3.3. [16] R. He, Z. Jin, and J. S. Li (2024) Modeling and management of cyber risk: A cross-disciplinary review. Annals of Actuarial Science 18 (2), p. 270–309. External Links: Document Cited by: §1. [17] J. J. Heckman (1979) Sample selection bias as a specification error. Econometrica 47 (1), p. 153–161. External Links: Document Cited by: §2, §3.2. [18] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T. Liu (2017) LightGBM: a highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30, p. 3146–3154. Cited by: §5.2. [19] O. F. Keskin, K. M. Caramancion, I. Tatar, O. Raza, and U. Tatar (2021) Cyber third-party risk management: a comparison of non-intrusive risk scoring reports. Electronics 10 (10), p. 1168. External Links: Document Cited by: §1, §1, §2, §2, §3.1. [20] Z. C. Lipton, D. C. Kale, and R. Wetzel (2016) Directly modeling missing data in sequences with RNNs: improved classification of clinical time series. In Proceedings of the 1st Machine Learning for Healthcare Conference, Proceedings of Machine Learning Research, Vol. 56, p. 253–270. External Links: Link Cited by: §1, §2, §3.2. [21] M. V. Ness, T. M. Bosschieter, R. Halpin-Gregorio, and M. Udell (2023) The missing indicator method: from low to high dimensions. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), p. 2497–2508. External Links: Document Cited by: §1, §2, §3.2. [22] A. Noroozian, M. Ciere, M. Korczyński, S. Tajalizadehkhoob, and M. van Eeten (2017) Inferring security performance of providers from noisy and heterogenous abuse datasets. In Proceedings of the 16th Workshop on the Economics of Information Security (WEIS), Cited by: §3.1. [23] P. Rockenschaub, Z. Xian, A. Zamanian, M. Piperno, O. Ciora, E. Pachl, and N. Ahmidi (2024) Robust prediction under missingness shifts. CoRR abs/2406.16484. Cited by: §3.2. [24] D. B. Rubin (1976) Inference and missing data. Biometrika 63 (3), p. 581–592. External Links: Document Cited by: §2, §3.2. [25] SecurityScorecard (2026) SecurityScorecard. Note: https://securityscorecard.com/Accessed: 2026-04-12 Cited by: §1. [26] A. Sharafoddini, J. A. Dubin, D. M. Maslove, and J. Lee (2019) A new insight into missing data in intensive care unit patient profiles: observational study. JMIR Medical Informatics 7 (1), p. e11605. External Links: Document Cited by: §1, §2, §3.2. [27] B. Sohval (2024) Applying machine learning to optimize the correlation of SecurityScorecard scores with relative likelihood of breach. Technical report SecurityScorecard. Note: Accessed April 2026 External Links: Link Cited by: §1. [28] S. Tajalizadehkhoob, T. van Goethem, M. Korczyński, A. Noroozian, R. Böhme, T. Moore, W. Joosen, and M. van Eeten (2017) Herding vulnerable cats: a statistical approach to disentangle joint responsibility for web security in shared hosting. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS), p. 553–567. External Links: Document Cited by: §3.1. [29] S. Tan, R. Caruana, G. Hooker, and Y. Lou (2018) Distill-and-compare: auditing black-box models using transparent model distillation. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, p. 303–310. External Links: Document Cited by: §3.1. [30] D. W. Woods and R. Böhme (2021) SoK: quantifying cyber risk. In 2021 IEEE Symposium on Security and Privacy (SP), p. 211–228. External Links: Document Cited by: §1, §2, §2, §3.1.