Paper deep dive

Conformal Risk Control for Safety-Critical Wildfire Evacuation Mapping: A Comparative Study of Tabular, Spatial, and Graph-Based Models

Baljinnyam Dayan

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 42

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 99%

Last extracted: 3/26/2026, 1:30:38 AM

Summary

This paper introduces Conformal Risk Control (CRC) to wildfire spread prediction to provide formal, distribution-free safety guarantees on the false negative rate (FNR). By evaluating three model architectures (LightGBM, Tiny U-Net, and Hybrid ResGNN-UNet) on the Next Day Wildfire Spread (NDWS) dataset, the authors demonstrate that standard probability thresholds are inadequate for safety-critical applications. The study finds that while model architecture determines evacuation efficiency, CRC ensures safety, with spatial models achieving 4.2x greater efficiency than tabular baselines under a 95% fire coverage guarantee.

Entities (5)

Conformal Risk Control · methodology · 100%Hybrid ResGNN-UNet · model-architecture · 100%LightGBM · model-architecture · 100%Next Day Wildfire Spread dataset · dataset · 100%Tiny U-Net · model-architecture · 100%

Relation Signals (4)

Conformal Risk Control → appliedto → wildfire spread prediction

confidence 100% · We address this gap by presenting, to our knowledge, the first application of conformal risk control (CRC) to wildfire spread prediction

LightGBM → evaluatedon → Next Day Wildfire Spread dataset

confidence 100% · We train and evaluate three architectures of increasing complexity on the Next Day Wildfire Spread (NDWS) dataset

Tiny U-Net → evaluatedon → Next Day Wildfire Spread dataset

confidence 100% · We train and evaluate three architectures of increasing complexity on the Next Day Wildfire Spread (NDWS) dataset

Hybrid ResGNN-UNet → evaluatedon → Next Day Wildfire Spread dataset

confidence 100% · We train and evaluate three architectures of increasing complexity on the Next Day Wildfire Spread (NDWS) dataset

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Every wildfire prediction model deployed today shares a dangerous property: none of these methods provides formal guarantees on how much fire spread is missed. Despite extensive work on wildfire spread prediction using deep learning, no prior study has applied distribution-free safety guarantees to this domain, leaving evacuation planners reliant on probability thresholds with no formal assurance. We address this gap by presenting, to our knowledge, the first application of conformal risk control (CRC) to wildfire spread prediction, providing finite-sample guarantees on false negative rate (FNR <= 0.05). We expose a stark failure: across three model families of increasing complexity (tabular: LightGBM, AUROC 0.854; convolutional: Tiny U-Net, AUROC 0.969; and graph-based: Hybrid ResGNN-UNet, AUROC 0.964), standard thresholds capture only 7-72% of true fire spread. CRC eliminates this failure uniformly. Our central finding is that model architecture determines evacuation efficiency, while CRC determines safety: both spatial models with CRC achieve approximately 95% fire coverage while flagging only approximately 15% of total pixels, making them 4.2x more efficient than LightGBM, while the graph model's additional complexity over a simple U-Net yields no meaningful efficiency gain. We propose a shift-aware three-way CRC framework that assigns SAFE/MONITOR/EVACUATE zones for operational triage, and characterize a fundamental limitation of prevalence-weighted bounds under extreme class imbalance (approximately 5% fire prevalence). All models, calibration code, and evaluation pipelines are released for reproducibility.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

41,837 characters extracted from source content.

Expand or collapse full text

Conformal Risk Control for Safety-Critical Wildfire Evacuation Mapping: A Comparative Study of Tabular, Spatial, and Graph-Based Models Baljinnyam Dayan 1 Abstract Every wildfire prediction model deployed today shares a dangerous property: none of these meth- ods provides formal guarantees on how much fire spread is missed. Despite extensive work on wildfire spread prediction using deep learn- ing, no prior study has applied distribution-free safety guarantees to this domain, leaving evacu- ation planners reliant on probability thresholds with no formal assurance. We address this gap by presenting, to our knowledge, the first application of conformal risk control (CRC) (Angelopoulos et al., 2022) to wildfire spread prediction, pro- viding finite-sample guarantees on false negative rate (FNR ≤ 0.05). We expose a stark fail- ure: across three model families of increasing complexity (tabular: LightGBM, AUROC 0.854; convolutional: Tiny U-Net, AUROC 0.969; and graph-based: Hybrid ResGNN-UNet, AUROC 0.964), standard thresholds capture only 7–72% of true fire spread. CRC eliminates this failure uniformly. Our central finding is that model ar- chitecture determines evacuation efficiency, while CRC determines safety: both spatial models with CRC achieve∼95%fire coverage while flagging only∼15%of pixels of total map, making them 4.2×more efficient than LightGBM, while the graph model’s additional complexity over a sim- ple U-Net yields no meaningful efficiency gain. We propose a shift-aware three-way CRC frame- work that assigns SAFE/MONITOR/EVACUATE zones for operational triage, and characterize a fundamental limitation of prevalence-weighted bounds under extreme class imbalance (∼5%fire prevalence). All models, calibration code, and evaluation pipelines are released for reproducibil- ity. 1 Imperial College London,London,United King- dom.Correspondence to:Baljinnyam Dayan<baljin- nyam.dayan25@imperial.ac.uk>. Preprint. March 25, 2026. 1. Introduction Wildfire evacuation planning requires prediction systems that prioritize safety over accuracy. In this domain, a false negative (a fire pixel classified as safe) poses direct risk to human life, while false positives carry their own operational cost: overly broad evacuation perimeters trigger shadow evacuations, where residents outside the official zone self- evacuate, congesting road networks and potentially trapping those in genuine danger (Zhao et al., 2022). This dual asymmetry renders standard accuracy-based evaluation in- adequate: a system reporting 90% pixel-level accuracy may simultaneously miss the majority of fire pixels due to ex- treme class imbalance, as we demonstrate empirically in this work. Controlling pixel-level FNR is a necessary con- dition for safe evacuation mapping: a system that misses individual fire pixels cannot produce reliable fire perimeters, even if downstream spatial post-processing is applied. Despite decades of wildfire prediction research (Jain et al., 2020) and the recent availability of distribution-free safety tools such as conformal risk control (CRC) (Angelopoulos et al., 2022), no prior work has applied CRC, or any con- formal method, to wildfire spread prediction. Wildfire ML continues to evaluate models on accuracy-based metrics (F1, IoU, AUROC) that are blind to the safety-critical nature of the task, while the conformal prediction community has fo- cused on medical imaging, autonomous driving, and natural language processing. This leaves a clear gap: the domain where false negatives are most dangerous has no formal safety guarantees. We bridge this gap. CRC selects a decision threshold from held-out calibration data such thatE[FNR]≤ αon test data, regardless of the underlying model’s quality or calibration (Angelopoulos & Bates, 2021). A natural question then mo- tivates our study: can a sufficiently powerful model eliminate the need for post-hoc safety calibration? To answer this, we train and evaluate three architectures of increasing complex- ity on the Next Day Wildfire Spread (NDWS) dataset (Huot et al., 2022): (i) a tabular LightGBM baseline (Ke et al., 2017) treating each pixel independently, (i) a Tiny U-Net (Ronneberger et al., 2015) capturing local spatial structure, and (i) a Hybrid ResGNN-UNet combining convolutional encoders with graph attention layers (Veli ˇ ckovi ́ c et al., 2018) 1 arXiv:2603.22331v1 [cs.LG] 20 Mar 2026 Conformal Risk Control for Safety-Critical Wildfire Prediction to model fire propagation on pixel graphs. All models are trained from scratch on identical data splits. Our experiments yield a clear answer: no model is safe without CRC. At their respective optimal thresholds, Light- GBM misses 92.8% of fires, U-Net misses 61.0%, and the ResGNN-UNet misses 28.5%. With CRC, all three models achieve≥94%coverage, but spatial models flag only∼15% of pixels compared to LightGBM’s 62.6%, confirming that spatial inductive bias drives efficiency under the same safety guarantee. Contributions. •We demonstrate that standard probability thresholding is fundamentally inadequate for wildfire evacuation across three model families, with fire coverage ranging from 7% to 72%. • We show that CRC provides guaranteed≥95%fire coverage regardless of model quality, and that archi- tecture determines efficiency (evacuation zone size), with spatial models achieving 4.2× tighter zones than tabular baselines under the same safety guarantee. •We propose shift-aware three-way CRC with SAFE/- MONITOR/EVACUATE zones and characterize its be- havior under extreme class imbalance, identifying a fundamental limitation of prevalence-weighted bounds when π 0 ≫ π 1 . •We provide a complete, reproducible pipeline with deterministic splits and open-source code for all exper- imental phases. 1 2. Related Work Machine learning for wildfire prediction. Machine learning has been widely applied to wildfire risk assessment and spread prediction (Jain et al., 2020; Radke et al., 2019). The NDWS dataset (Huot et al., 2022) established a stan- dardized pixel-level benchmark, enabling systematic com- parison across architectures including graph neural networks for spatial propagation (Kipf & Welling, 2017; Veli ˇ ckovi ́ c et al., 2018). However, every existing work evaluates exclu- sively on accuracy-based metrics (F1, IoU, AUROC); none provides formal safety guarantees on the false negative rate. Conformal prediction and distribution-free inference. Conformal prediction (Vovk et al., 2005; Shafer & Vovk, 2008) provides distribution-free prediction sets with finite- sample coverage guarantees under exchangeability. Romano et al. (2020) extended these ideas to adaptive classification, 1 https://github.com/baljinnyamday/wildfire-evacuation-crc and Angelopoulos et al. (2022) generalized them to con- formal risk control (CRC), which controls arbitrary mono- tone loss functions—including the FNR loss central to our work. CRC has been applied in medical imaging and au- tonomous driving, yet geospatial hazard prediction remains unexplored. To our knowledge, this is the first application of CRC to wildfire spread prediction, where the asymmet- ric cost structure—missed fires endanger lives; over-broad zones cause shadow evacuations (Zhao et al., 2022)—makes formal FNR guarantees uniquely valuable. Uncertainty quantification and calibration. Modern neural networks are poorly calibrated (Guo et al., 2017), and standard recalibration methods such as Platt scaling (Platt, 1999) and temperature scaling improve confidence estimates but provide no finite-sample coverage guarantees. We show that even well-calibrated models remain unsafe under stan- dard thresholding due to the extreme class imbalance in wildfire data (∼5%fire pixels), motivating distribution-free bounds rather than approximate confidence. Cost-sensitive learning and spatial extremes.Focal loss (Lin et al., 2017) and cost-sensitive learning (Elkan, 2001) improve empirical recall under imbalance but provide no finite-sample FNR bound. Neyman–Pearson classification (Scott & Nowak, 2005) targets a type-I error constraint with only asymptotic guarantees. Spatial extreme-value theory (Coles, 2001; Davison et al., 2012) models tail behavior of spatial processes but requires parametric assumptions unsuited to pixel-level spread prediction. CRC is comple- mentary: distribution-free, finite-sample, and applicable post-hoc to any model—including those already trained with cost-sensitive losses. Safety-critical machine learning. In safety-critical do- mains, the cost of false negatives far exceeds that of false positives, motivating formal guarantees over empirical per- formance alone (Amodei et al., 2016). Wildfire evacuation is a stark example—yet it is precisely the domain where such guarantees have been absent. Our work closes this gap, demonstrating a 23–87 percentage-point improvement in fire detection rate when replacing standard thresholds with CRC-calibrated ones. 3. Preliminaries 3.1. Problem Formulation We consider pixel-level binary classification for wildfire spread prediction. Given an input samplex ∈ R H×W×C (in our setting,H=W=64,C=12) and binary fire mask y ∈ 0, 1 H×W , a modelf θ outputs per-pixel probability estimatesˆp ij = f θ (x) ij ∈ [0, 1]. Binary predictions are 2 Conformal Risk Control for Safety-Critical Wildfire Prediction U-Net Probability Map (fire risk per pixel) Cross-section (row 31) 0102030405060 Pixel position along cross-section 0.0 0.2 0.4 0.6 0.8 1.0 p (fire) CRC catches fire pixels that standard threshold misses Probability Cross-Section (row 31) Fire probability decays with distance from fire center Fire probability Actual fire CRC threshold =0.0023 (guarantees 95% fire detection) Standard threshold (p0.5) CRC evacuation zone 0.0 0.2 0.4 0.6 0.8 1.0 p (fire) Figure 1. Why standard thresholds fail. A cross-section through a U-Net fire prediction shows probability peaking at∼0.8and decaying smoothly. The standard threshold (ˆp≥ 0.5, gray) misses the majority of the fire region; the CRC threshold ( ˆ λ = 0.002, green) captures it entirely. obtained by thresholding: ˆy ij (λ) = ⊮ˆp ij ≥ λ.(1) Our primary safety objective is to control the false negative rate: FNR(λ) = P ij y ij (1− ˆy ij (λ)) P ij y ij ≤ α, α = 0.05, (2) equivalently requiring empirical fire coverage1− FNR≥ 95%. 3.2. Conformal Risk Control CRC (Angelopoulos et al., 2022) provides a principled method for selectingλwith finite-sample guarantees. Given calibration examples(x i ,y i ) n i=1 drawn exchangeably with the test data, letmdenote the number of positive (fire) calibration pixels andp (1) ≤ · ≤ p (m) their sorted pre- dicted probabilities. CRC selects: ˆ λ = p (⌈α (m+1)⌉) ,(3) theα-quantile of positive-pixel probabilities with finite- sample correction. At threshold ˆ λ , at most⌈α(m+1)⌉− 1 positive pixels fall below it, controlling the empirical FNR ≤ α. The resulting guarantee is: E[FNR( ˆ λ)]≤ α,(4) which holds regardless of the model’s quality, calibration, or architecture, requiring only exchangeability of calibration and test data. Exchangeability assumption. The guarantee in Equa- tion (4) requires that calibration and test samples are drawn exchangeably from the same distribution. Our pixel-pooled formulation treats pixels across different images as ex- changeable units; in practice, pixels within the same64×64 patch exhibit spatial correlation, which is a standard simpli- fication in conformal segmentation tasks. We discuss the implications of this assumption, including potential viola- tions under temporal and geographic shift, in Section 6. Safety–efficiency decomposition.A key insight motivat- ing our study is that CRC decouples safety from efficiency. Equation (4) holds for any model; what varies is the predic- tion set size, the fraction of pixels flagged as fire. A model with strong fire/no-fire separation allows CRC to set a higher ˆ λ, flagging fewer pixels while maintaining the same safety guarantee. A poorly separating model forces ˆ λ toward zero, expanding the evacuation zone. Thus, the model determines efficiency; CRC determines safety. Figure 2 illustrates this: both spatial models maintain safe FNR over a much wider threshold range than LightGBM, explaining the large differ- ence in evacuation zone size when all three are wrapped in CRC. 4. Methods We train three architectures of increasing complexity on the same dataset and splits, then apply CRC uniformly. 4.1. Data We use the NDWS dataset (Huot et al., 2022; Haddad, 2024): 18,545 samples of64×64patches with 12 input channels and a binary fire target (Table 1). We enforce a determinis- tic split (seed 42): train 12,981 (70%), calibration 2,781 (15%), test 2,783 (15%). The calibration set is never used during model training. Fire targets include no-data pixels (−1); we construct valid-pixel masks and apply masked loss and evaluation throughout. Only∼5%of valid pixels are 3 Conformal Risk Control for Safety-Critical Wildfire Prediction 0.00.20.40.60.81.0 Threshold 0.0 0.2 0.4 0.6 0.8 1.0 False Negative Rate (FNR) LightGBM U-Net ResGNN-UNet = 0.05 LightGBM CRC =0.0031 U-Net CRC =0.0023 ResGNN-UNet CRC =0.0221 Standard (p0.5) Figure 2. FNR as a function of decision threshold for all three models. The red dashed line marksα = 0.05. Both spatial models maintain safe FNR over a much wider threshold range than Light- GBM, explaining their 4.2×smaller evacuation zones under CRC. The near-overlapping U-Net and ResGNN-UNet curves illustrate diminishing returns from architectural complexity. Markers show each model’s CRC ˆ λ. Table 1. NDWS input features (12 channels per64×64pixel patch). All features are from remote-sensing and reanalysis sources (Huot et al., 2022). CategoryFeatureVariableUnit TopographyelevationTerrain heightm WeatherthWind direction ◦ vsWind speedm/s tmmnMin temperatureK tmmxMax temperatureK sphSpecific humiditykg/kg prPrecipitationmm DroughtpdsiPalmer Drought Index — Fire weatherercEnergy Release Comp. — VegetationNDVIVegetation index[−1, 1] Demographics populationPopulation densityppl/km 2 Prior statePrevFireMask Previous fire mask 0, 1 labeled fire, creating extreme class imbalance with profound implications for threshold selection (Section 5). 4.2. Model 1: Tabular Baseline (LightGBM) We flatten each64×64patch into pixel-level rows with 12 features and train an LGBMClassifier (Ke et al., 2017) with binary cross-entropy objective. To control memory, we subsample 5% of training pixels. This model treats each pixel independently and serves as a baseline to isolate the contribution of spatial inductive biases. 4.3. Model 2: Tiny U-Net We implement a shallow U-Net (Ronneberger et al., 2015) with two encoder blocks (12→32→64channels), a bottle- neck (64→128), and two decoder blocks with skip connec- tions (∼470K parameters). A final1×1convolution with sigmoid activation produces per-pixel probabilities. Train- ing uses AdamW (lr= 10 −3 , weight decay= 10 −4 ) with cosine annealing over 50 epochs, masked BCE loss, and best-model checkpointing on validation loss. 4.4. Model 3: Hybrid ResGNN-UNet To test whether more expressive architectures improve probability separation, we implement a Hybrid ResGNN- UNet (∼229K parameters): (i) a three-block CNN encoder (12→32→64→64) downsampling to8×8; (i) a graph at- tention bottleneck with three GATConv layers (Veli ˇ ckovi ́ c et al., 2018) (4 heads each) on an 8-connected pixel grid, with a residual connection; (i) a U-Net decoder with three upsampling blocks, skip connections, and Dropout2d. Train- ing uses AdamW (lr= 3×10 −4 , weight decay= 10 −3 ) with BCEWithLogitsLoss (posweight= 8.0), cosine an- nealing over 25 epochs, gradient clipping, and early stop- ping (patience=7). The GNN bottleneck treats the downsampled feature map as a graph where each pixel attends to its 8 spatial neighbors, designed to capture fire propagation patterns. 4.5. CRC Calibration For each model, we compute per-pixel probabilities on the calibration set, then select ˆ λvia Equation (3) withα = 0.05. This threshold is applied unchanged to the test set. 4.6. Three-Way CRC Extension We adapt shift-aware CRC to wildfire evacuation, routing each pixel to one of three actions: SAFE : ˆp < λ min , MONITOR : λ min ≤ ˆp < λ max , EVACUATE : ˆp≥ λ max . (5) The MONITOR zone routes uncertain pixels to human review rather than forcing a binary automated decision. We use cost-weighted CRC withc fn =5,c fp =1(Zhao et al., 2022) and cost-weighted risk levelα cw =0.50(cost- weighting with∼95%non-fire pixels shifts the risk scale relative to the binary FNR target). To account for seasonal prevalence variation, we specify a shift interval[ρ lo ,ρ hi ] = [0.9, 1.1]on the prevalence ratio ρ = π dep 1 /π cal 1 . The adjusted calibration level is: ε max = B pw · max ∥δ ρ lo ∥ 1 , ∥δ ρ hi ∥ 1 + B pw N+1 ,(6) α safe = α cw − ε max ,(7) whereB pw = max(c fn π 1 , c fp π 0 )is the prevalence- weighted bound andδ ρ the importance weight mismatch at shiftρ. A base threshold ˆ λis found via cost-weighted CRC at levelα safe , then shifted proportionally to the end- 4 Conformal Risk Control for Safety-Critical Wildfire Prediction Table 2. Test-set results across three models and three decision frameworks. CRC targetsFNR ≤ 0.05. Three-way CRC uses cost-weightedα cw =0.50,c fn =5,c fp =1,ρ∈ [0.9, 1.1]. All mod- els trained from scratch on identical 70/15/15 splits. Values show ± 95% bootstrap CI (10K image resamples). MethodParams ˆ λCov.↑FNR↓Set↓AUC↑ LGBM (p≥.5)—.500 .072±.02 .928±.02.001.854 LGBM+CRC—.003 .941±.01 .059±.01 .626±.01.854 LGBM+3way—[0,.03]1.00.000.040.854 UNet (p≥.5)470K.500 .390±.02 .610±.02.007.969 UNet+CRC470K.002 .947±.01 .053±.01 .149±.01.969 UNet+3way470K[0,.02]1.00.000.051.969 ResGNN (p≥.5) 229K.500 .715±.02 .285±.02 .031±.00.964 ResGNN+CRC229K.022 .948±.01 .052±.01 .151±.01.964 ResGNN+3way229K[0,.02]1.00.000.149.964 point weight mismatches to produceλ min andλ max (Algo- rithm B). On the decided set,E[ℓ| decided]≤ α cw /(1−d), wheredis the MONITOR fraction. Ifα safe ≤ 0, the interval is too wide for safe calibration. 5. Experiments 5.1. Setup We evaluate three model families under three decision frame- works: (i) standard thresholding atˆp≥ 0.5, (i) CRC with α = 0.05, and (i) three-way CRC with cost-weighted calibration. All thresholds are computed on the held-out calibration set and applied unchanged to the test set. Metrics. Our primary metrics are: coverage(1−FNR), the fraction of true fire pixels correctly flagged (target: ≥95%); and set size, the fraction of all valid pixels flagged as fire (smaller is better, given safety is met). Secondary metrics include AUROC, precision, F1, IoU, AUPRC, and three-way zone statistics. 5.2. Main Results Table 2 reveals four key findings. Standard thresholds are unsafe. Atˆp ≥ 0.5, Light- GBM captures only 7.2% of fire pixels, U-Net 39.0%, and ResGNN-UNet 71.5%. Even the best-performing model at a standard threshold misses nearly a third of all fires. The ∼95%non-fire class dominance concentrates probability mass near zero, making any conventional threshold unsafe (Figure 1). The FNR sweep (Figure 2) confirms this: safe FNR is achievable only at thresholds far below 0.5. CRC delivers safety; model quality delivers efficiency. CRC raises all three models to≥94%coverage, meeting theFNR≤ 0.05target. However, the cost differs substan- tially (Figure 4): LightGBM must flag 62.6% of all pixels, while both spatial models flag only∼15%, a 4.2×reduc- tion in evacuation zone size. This gap reflects the AUROC difference between tabular and spatial models (0.854 vs. 0.964–0.969): better fire/non-fire separation allows CRC to set a higher ˆ λwhile maintaining the same safety guaran- tee. Figure 8 (Appendix) visualizes this safety–efficiency decomposition across all nine configurations. Spatial inductive bias, not graph complexity, drives efficiency. The ResGNN-UNet (AUROC 0.964) nearly matches the Tiny U-Net (0.969), and both achieve near- identical CRC efficiency: 15.1% vs. 14.9% set size under the same safety guarantee. On64×64patches, the U-Net’s re- ceptive field already covers the full spatial extent; the GNN bottleneck adds engineering complexity without meaningful improvement. This reinforces our central claim: CRC de- termines safety, spatial inductive bias determines efficiency, and additional architectural complexity yields diminishing returns. Per-pixel risk gradients. The U-Net produces smooth probability gradients where fire risk decays with distance from fire centers (Figure 6). The CRC threshold ( ˆ λ = 0.002) captures the entire “tail” of low-probability but real fire pixels that conventional thresholds miss, yielding an interpretable risk map where the CRC boundary defines the guaranteed-safe evacuation zone (Figure 3). 5.3. Before and After CRC Figure 5 presents a multi-sample comparison of fire detec- tion with and without CRC. Without CRC, the standard threshold produces predominantly red error maps with vast regions of missed fire. With CRC ( ˆ λ = 0.002), the maps turn green (correctly detected) with some blue (over-alerts), and the three-way CRC provides an interpretable MONI- TOR/EVACUATE policy. 5.4. Risk Gradient Analysis Figure 6 visualizes the per-pixel probability landscape. Rather than binary predictions, the U-Net produces a contin- uous risk gradient where each pixel carries a calibrated fire probability. The CRC threshold contour and the EVACU- ATE boundary provide principled decision surfaces overlaid on this risk field. 5.5. Three-Way Zone Analysis Table 3 shows the zone breakdown. Three-way CRC achieves 100% coverage with compact EVACUATE zones (4–15%), but the SAFE zone collapses entirely (λ min = 0). With∼5% fire prevalence: B pw = max(5× 0.05, 1× 0.95) = 0.95, 5 Conformal Risk Control for Safety-Critical Wildfire Prediction Ground Truth (3.6% fire) Standard Model (p0.5) Caught: 56% of fires With CRC ( =0.0023) Caught: 100% | Zone size: 16.3% Probability Map (green = CRC, gray = standard) 0.0 0.2 0.4 0.6 0.8 1.0 p (fire) 0102030405060 Pixel position along cross-section 0.0 0.2 0.4 0.6 0.8 1.0 p (fire) These fire pixels have low probability but ARE real fires. Standard model misses them. CRC catches them. Cross-Section (row 33): Fire probability decays with distance CRC captures the tail Actual fire pixels Model probability p(fire) Standard threshold (p0.5) CRC threshold = 0.0023 (guarantees 95% fire detection) Extra pixels CRC catches (standard model would miss these!) Pixels standard model catches Fire pixels saved by CRC Why Conformal Risk Control Matters: The Same Model, Dramatically Different Safety Error Map Colors True PositiveFalse Negative (missed!)False Positive Figure 3. Same U-Net, different thresholds. Standard thresholding (ˆp ≥ 0.5) catches 56% of fires; CRC ( ˆ λ = 0.002) catches 100%. Bottom: probability cross-section showing how CRC captures low-probability fire pixels. Green shading marks pixels saved by CRC. LightGBMU-NetResGNN-UNet 0.70 0.75 0.80 0.85 0.90 0.95 1.00 AUROC 0.853 0.969 0.964 Model Quality (ranking ability) LightGBMU-NetResGNN-UNet 0.0 0.2 0.4 0.6 0.8 1.0 Coverage (fire detection rate) 7.2% 39.0% 71.5% UNSAFEUNSAFEUNSAFE Without CRC (standard threshold) 95% safety target LightGBM + CRC U-Net + CRC ResGNN-UNet + CRC 0.0 0.2 0.4 0.6 0.8 1.0 Fraction 94.1% 94.7% 94.8% 62.6% 14.9% 15.1% 4.2× smaller evacuation zone With CRC (guaranteed safe) 95% target Coverage Set Size Figure 4. Three-model comparison. Left: AUROC. Center: with- out CRC, no model reaches 95% coverage. Right: with CRC, all three meet the safety target; both spatial models achieve∼4× smaller evacuation zones than LightGBM. dominated by the majority-class termc fp π 0 . The resulting shift correction exceeds the base CRC threshold (∼0.003), pushingλ min below zero and producing a maximally con- servative policy: all non-evacuated pixels are routed to hu- man review. This reveals a fundamental limitation of the prevalence-weighted bound in rare-event regimes where π 0 ≫ π 1 . 5.6. Training Details Table 4 summarizes the training investment. The Tiny U-Net achieves the best AUROC (0.969) with a standard convolu- tional architecture and no graph-processing dependencies. The ResGNN-UNet early-stopped at epoch 23 (best epoch 16, val loss 0.131), training substantially longer than in Table 3. Three-way zone fractions on the test set. The SAFE zone collapses becauseB pw = max(c fn π 1 ,c fp π 0 ) = 0.95is dominated by c fp π 0 under∼5% fire prevalence. ModelSAFEMONITOREVACUATE B pw LightGBM0.0%95.96%4.04%0.988 U-Net0.0%94.90%5.10%0.987 ResGNN-UNet0.0%85.15%14.85%0.987 Table 4. Model training summary. All models trained from scratch on identical NDWS splits. ModelParams EpochsOptim Val Loss AUC LightGBM—100 iterGBM—.854 Tiny U-Net470K50AdamW0.048.969 ResGNN-UNet 229K23 † AdamW0.131.964 †Early-stopped from 30 (patience=7, best epoch 16). preliminary experiments and achieving an AUROC (0.964) close to the U-Net’s (0.969), suggesting that with proper training configuration the graph architecture can nearly match convolutional performance. Figure 7 shows the U- Net’s smooth convergence with minimal overfitting. 6. Discussion Separation of safety and efficiency.Our central result is that CRC cleanly separates safety from efficiency. All three 6 Conformal Risk Control for Safety-Critical Wildfire Prediction Ground Truth (0.5% fire) Before CRC (p0.5) Missed: 100% of fires After CRC ( =0.0023) Missed: 24% of fires Three-Way CRC (SAFE / MONITOR / EVACUATE) Ground Truth (2.7%) p0.5 Missed: 73% CRC =0.0023 Missed: 0% Three-Way CRC Ground Truth (7.8%) p0.5 Missed: 34% CRC =0.0023 Missed: 0% Three-Way CRC The Safety Gap: Standard Thresholds vs Conformal Risk Control True Positive (caught fire)False Negative (MISSED fire)False Positive (over-alert)True NegativeSAFEMONITOREVACUATE Figure 5. The safety gap across three test samples (0.5%, 2.7%, 7.8% fire). Col. 1: ground truth. Col. 2: standard threshold, with missed fire in red (100%, 73%, 34% missed). Col. 3: CRC threshold, with improved detection (green). Col. 4: three-way zones (SAFE/MONITOR/EVACUATE). models achieve the same FNR guarantee (≤0.05); what varies is the evacuation zone size: 62.6% for LightGBM versus∼15%for both spatial models. This decomposition has practical significance: teams can invest in spatial models to reduce evacuation costs without compromising or re- validating safety properties. Diminishing returns from complexity. The ResGNN- UNet nearly matches the Tiny U-Net in AUROC (0.964 vs. 0.969) and achieves virtually identical CRC efficiency (15.1% vs. 14.9% set size). While the graph attention bottle- neck does not degrade performance, it provides no meaning- ful improvement: on64×64patches, the U-Net’s receptive field already spans the full tile, and GNN message-passing over an 8-connected grid captures no additional information. This confirms that spatial inductive bias is sufficient for this task, and further architectural complexity adds engineering cost without practical benefit. Limitations of three-way CRC under imbalance. The three-way framework is conceptually appealing, routing uncertain pixels to human review rather than binary de- cisions. However, under extreme imbalance (∼5%fire), B pw is dominated by the majority-class cost, collapsing the SAFE zone. Future work could explore: (i) tighter bounds for rare-event settings, (i) hybrid approaches using sepa- rate precision-based thresholds for SAFE, or (i) regional calibration where local prevalence is higher. Limitations and assumptions. As noted in Section 3.2, CRC’s guarantee rests on exchangeability between cali- 7 Conformal Risk Control for Safety-Critical Wildfire Prediction Ground Truth (2.7% fire) Per-Pixel Fire Probability p(firex) Risk Zones (probability bands + CRC boundary) Calibrated Risk Gradient (CRC-informed evacuation map) Ground Truth (5.7%) Fire Probability Risk ZonesCalibrated Risk Gradient Ground Truth (11.3%) Fire Probability Risk ZonesCalibrated Risk Gradient 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p (fire) Per-Pixel Fire Risk: From Raw Probability to Calibrated Evacuation Zones < CRC threshold (0.0023) CRC5% 5 20% 2050% 5080% > 80% CRC boundary ( =0.0023) EVACUATE boundary (0.0207) Figure 6. Per-pixel fire risk gradient. Col. 1: ground truth. Col. 2: continuous probability heatmap. Col. 3: probability bands with CRC contour (green dashed) and EVACUATE boundary (red). Col. 4: calibrated risk gradient for operational mapping. 01020304050 Epoch 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10 BCE Loss Training & Validation Loss Train BCE Val BCE Best (epoch 48, 0.0300) 01020304050 Epoch 0.0000 0.0002 0.0004 0.0006 0.0008 0.0010 Learning Rate Cosine Annealing Schedule Figure 7. Tiny U-Net training curves: BCE loss and cosine an- nealing LR over 50 epochs. Smooth convergence with minimal train–val gap. bration and test data. Distributional shift from novel fire regimes, climate change, or geographic transfer may vio- late this assumption; our evaluation uses a single dataset (NDWS) with US wildfire events. The three models rep- resent a limited architectural sweep; transformer-based ar- chitectures may yield different efficiency–complexity trade- offs. Future work.Our results open several directions that we leave to future investigation. Class-conditional three-way CRC. The three-way zone col- lapse under extreme imbalance (Section 5.5) stems from cou- pling both error types through a single prevalence-weighted bound. A natural fix is to calibrate each zone boundary independently:λ max via FNR-CRC on positive pixels and λ min via FPR-CRC on negative pixels. Because the nega- tive class has abundant calibration data (∼95%of pixels), this approach should produce a non-degenerate SAFE zone regardless of class imbalance, resolving the primary limita- tion of our current framework. We did not pursue this here because the theoretical guarantees require a careful joint analysis of two simultaneous conformal procedures, which 8 Conformal Risk Control for Safety-Critical Wildfire Prediction is beyond the scope of this comparative study. Image-level CRC. Our current calibration pools all pixels across calibration images, treating each pixel as an inde- pendent sample. Pixels within the same64×64patch are spatially correlated, which formally violates the exchange- ability assumption underlying CRC. An image-level formu- lation, which defines the loss as per-image FNR and applies CRC to then=2,781image-level losses, would restore the- oretical validity at the cost of a smaller effective sample size. Comparing pixel-pooled and image-level CRC would quantify whether the practical convenience of pooling de- grades the guarantee, and would produce per-image FNR distributions that are more operationally meaningful than aggregate statistics. Formalizing the AUROC–efficiency relationship. Our central finding, that model discrimination determines evacuation zone size under a fixed safety guarantee, is currently em- pirical. Under a bi-normal score model, the CRC set size decomposes asS(α) = π 1 (1−α) + π 0 (1− F 0 (F −1 1 (α))), whereF 0 ,F 1 are the class-conditional score CDFs. This would yield a closed-form prediction of evacuation cost from AUROC alone, enabling practitioners to answer: what discrimination level is needed for a target zone size? We leave the formal statement and empirical validation of this relationship to future work. Temporal and geographic shift. CRC assumes exchange- ability between calibration and test data. Wildfire behavior varies across seasons, years, and geographies due to cli- mate change, drought cycles, and regional fuel loads, cre- ating natural label shift that may violate this assumption. The shift-aware recalibration framework of Angelopoulos et al. (2022) could address this, but validating it requires temporally stratified splits and multi-region datasets that the current NDWS benchmark does not cleanly support. Evaluating CRC robustness under realistic temporal shift is critical for any operational deployment. 7. Conclusion We have demonstrated that conformal risk control trans- forms wildfire spread prediction from an accuracy- optimization problem into a safety-guaranteed decision sys- tem. Across three model families, our experiments establish that: 1.No model is safe without CRC. Standard thresholding captures only 7–72% of fires. 2.CRC guarantees safety regardless of model. All three calibrated models achieve≥94%coverage on held-out data. 3.Model quality determines efficiency, not safety. The Tiny U-Net achieves 4.2×tighter evacuation zones than LightGBM under the same CRC guarantee. 4. Complexity without separation is waste. A graph- augmented model underperforms a standard U-Net in AUROC despite its more complex architecture. For practitioners: a well-trained simple model wrapped in CRC provides both stronger safety guarantees and tighter evacuation zones than a complex model evaluated on F1 alone. The few lines of conformal calibration code deliver what no amount of architectural sophistication can: a math- ematical guarantee that≥95% of fires will be detected. Impact Statement This work targets safer wildfire evacuation decision support by providing, for the first time, fire spread predictions with formal guarantees on missed detections (≥95%coverage, and 100% under the three-way framework). If deployed operationally, such guarantees could transform how lim- ited emergency resources are allocated: EVACUATE zones give firefighters, first responders, and military personnel a trusted map for immediate civilian evacuation and safe operational corridors, while MONITOR zones enable pre- cautionary self-evacuation and resource pre-positioning in areas the model cannot yet confirm as safe. Rather than treating the entire landscape as equally dangerous, respon- ders could concentrate assets where the model is certain and warn communities in uncertain regions to prepare— replacing blanket alerts with targeted, risk-proportionate action. Potential negative impact includes over-reliance on automated recommendations; we mitigate this through the three-way framework that explicitly routes uncertain pixels to human review rather than issuing a binary safe/unsafe judgment. Software and Data All code, trained models, and evaluation pipelines areavailableathttps://github.com/baljinnyamday/ wildfire-evacuation-crc, with deterministic data splits (seed 42, 70/15/15) and CLI entry points for each phase. The dataset is NDWS (Haddad, 2024), subject to its original terms of use. References Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Man ́ e, D. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021. 9 Conformal Risk Control for Safety-Critical Wildfire Prediction Angelopoulos, A. N., Bates, S., Cand ` es, E. J., and Jor- dan, M. I. Conformal risk control. arXiv preprint arXiv:2208.02814, 2022. Coles, S. An Introduction to Statistical Modeling of Extreme Values. Springer, 2001. Davison, A. C., Padoan, S. A., and Ribatet, M. Statistical modeling of spatial extremes. Statistical Science, 27(2): 161–186, 2012. Elkan, C. The foundations of cost-sensitive learning. In International Joint Conference on Artificial Intelligence, 2001. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 2017. Haddad, F. Next day wildfire spread dataset. Kaggle, 2024. Accessed 2026-03-17; dataset identifier: fantineh/next- day-wildfire-spread. Huot, F., Hu, R. L., Goyal, N., Sankaranarayanan, T., An- gelopoulos, A. N., et al. Next day wildfire spread: A machine learning dataset for predicting wildfire spread- ing from remote-sensing data. IEEE Transactions on Geoscience and Remote Sensing, 60:1–13, 2022. Jain, P., Coogan, S. C., Subramanian, S. G., Crowley, M., Taylor, S., and Flannigan, M. D. A review of machine learning applications in wildfire science and management. Environmental Reviews, 28:478–505, 2020. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, 2017. Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. In International Conference on Learning Representations, 2017. Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Doll ́ ar, P. Fo- cal loss for dense object detection. In IEEE International Conference on Computer Vision, 2017. Platt, J. C. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers. MIT Press, 1999. Radke, D., Hessler, A., and Ellsworth, D. Firecast: Leverag- ing deep learning to predict wildfire spread. Proceedings of the International Joint Conference on Artificial Intelli- gence, 2019. Romano, Y., Sesia, M., and Cand ` es, E. Classification with valid and adaptive coverage. Advances in Neural Infor- mation Processing Systems, 2020. Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Inter- vention (MICCAI), 2015. Scott, C. and Nowak, R. A Neyman–Pearson approach to statistical learning. IEEE Transactions on Information Theory, 51(11):3806–3819, 2005. Shafer, G. and Vovk, V. A tutorial on conformal prediction. Journal of Machine Learning Research, 9:371–421, 2008. Veli ˇ ckovi ́ c, P., Cucurull, G., Casanova, A., Romero, A., Li ` o, P., and Bengio, Y. Graph attention networks. In International Conference on Learning Representations, 2018. Vovk, V., Gammerman, A., and Shafer, G. Algorithmic Learning in a Random World. Springer, 2005. Zhao, X., Xu, Y., Lovreglio, R., Kuligowski, E., Nilsson, D., Cova, T. J., Wu, A., and Yan, X. Estimating wildfire evac- uation decision and departure timing using large-scale GPS data. Transportation Research Part D: Transport and Environment, 107:103277, 2022. 10 Conformal Risk Control for Safety-Critical Wildfire Prediction A. Additional Figures LGBM (p0.5) LGBM + CRC LGBM + 3-way U-Net (p0.5) U-Net + CRC U-Net + 3-way ResGNN (p0.5) ResGNN + CRC ResGNN + 3-way 0.0 0.2 0.4 0.6 0.8 1.0 Coverage (1 FNR) 7.2% 94.1% 100.0% 39.0% 94.7% 100.0% 71.5% 94.8% 100.0% Safety: Fire Detection Rate 95% target LGBM (p0.5) LGBM + CRC LGBM + 3-way U-Net (p0.5) U-Net + CRC U-Net + 3-way ResGNN (p0.5) ResGNN + CRC ResGNN + 3-way 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Set Size (fraction flagged) 0.1% 62.6% 4.0% 0.7% 14.9% 5.1% 3.1% 15.1% 14.8% Efficiency: Evacuation Zone Size Figure 8. Safety vs. efficiency for all nine configurations. Top: coverage (safety), where only CRC and three-way methods achieve≥95%. Bottom: set size (efficiency), where both spatial models with CRC achieve the best trade-off. B. Three-Way CRC Procedure Algorithm 1 Three-Way CRC for Wildfire Evacuation 1 def threewaycrc(scores, labels, α cw , c fn , c fp , ρ lo , ρ hi ): 2 # Prevalence and bound 3 π 1 = mean(labels) 4 π 0 = 1− π 1 5 B pw = max(c fn · π 1 , c fp · π 0 ) 6 7 # Importance weight mismatch at each shift endpoint 8 for ρ in ρ lo , ρ hi : 9w 1 (ρ) = (ρπ 1 ) / (ρπ 1 + π 0 ) / π 1 10w 0 (ρ) = π 0 / (ρπ 1 + π 0 ) / π 0 11∥δ ρ ∥ 1 = |w 1 (ρ)− 1| + |w 0 (ρ)− 1| 12 13 # Shift−aware recalibration 14 ε max = B pw · max(∥δ ρ lo ∥ 1 , ∥δ ρ hi ∥ 1 ) + B pw /(N+1) 15 α safe = α cw − ε max 16 if α safe ≤ 0: 17 return INFEASIBLE # shift interval too wide 18 19 # Base threshold and zone boundaries 20 ˆ λ = costweightedcrc(scores, labels, α safe ) 21 s = B pw / max(c fn , c fp ) # shift scale 22 λ min = max(0, ˆ λ− s · ∥δ ρ lo ∥ 1 ) 23 λ max = min(1, ˆ λ + s · ∥δ ρ hi ∥ 1 ) 24 25 # Decision rule 26 return λ min , λ max 27 # SAFE if ˆp < λ min 28 # MONITOR if λ min ≤ ˆp < λ max 29 # EVACUATE if ˆp≥ λ max Under∼5%fire prevalence withc fn =5,c fp =1:B pw = max(0.25, 0.95) = 0.95ands = 0.95/5 = 0.19. With ρ ∈ [0.9, 1.1],∥δ∥ 1 ≈ 0.105, giving a shift of∼0.02. Since the base threshold ˆ λis itself∼0.02(driven by extreme class imbalance),λ min is pushed to zero—collapsing the SAFE zone entirely and routing all non-evacuated pixels to 11 Conformal Risk Control for Safety-Critical Wildfire Prediction 0 10 20 30 40 50 60 Sample #2 (0.5% fire) Ground Truth Fire Probability p(firex) Caught: 0% Standard Model (p0.5, no CRC) Caught: 76% With CRC (p, guaranteed safe) 0102030405060 0.0 0.2 0.4 0.6 0.8 1.0 Probability Cross-Section (risk decay from fire center) p(fire) Standard (p0.5) CRC ( =0.0023) Pixels CRC saves 0 10 20 30 40 50 60 Sample #4 (1.8% fire) Caught: 0%Caught: 43% 0102030405060 0.0 0.2 0.4 0.6 0.8 1.0 0 10 20 30 40 50 60 Sample #24 (3.6% fire) Caught: 56%Caught: 100% 0102030405060 0.0 0.2 0.4 0.6 0.8 1.0 0 10 20 30 40 50 60 Sample #3 (7.8% fire) Caught: 66%Caught: 100% 0102030405060 Pixel position 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p (fire) Standard Model vs CRC: Per-Pixel Fire Detection Comparison True Positive (caught fire)False Negative (MISSED fire!)False Positive (over-alert)True Negative Figure 9. Bare model vs. CRC across four test samples. Fire catch rate improves from 8%→98%, 0%→83%, 56%→100%, 48%→100%. MONITOR (Table 3). 12 Conformal Risk Control for Safety-Critical Wildfire Prediction Ground TruthU-Net Probabilities U-Net Standard (p0.5) U-Net + CRC ( =0.0023) U-Net + Three-Way CRCCRC Error Map (TP/FP/FN/TN) 0.0 0.2 0.4 0.6 0.8 1.0 Test Sample #4: Qualitative Comparison Figure 10. Qualitative prediction maps: ground truth, probabilities, standard prediction, CRC prediction, three-way zones, and error analysis. 13

Instant research discovery

Conformal Risk Control for Safety-Critical Wildfire Evacuation Mapping: A Comparative Study of Tabular, Spatial, and Graph-Based Models

Intelligence

Summary

Entities (5)

Relation Signals (4)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations