Paper deep dive

Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias

Aditya Parikh, Eike Petersen, Stella Frank, Enzo Ferrante, Melanie Ganz, Aasa Feragen

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 111

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 9/29/2026, 12:32:21 PM

Summary

This paper challenges the assumption that demographic representation invariance is sufficient for algorithmic fairness. It mathematically and empirically demonstrates that enforcing marginal representation invariance (statistical independence of representations from sensitive attributes) implies demographic parity, which is incompatible with equalized odds and can harm predictive performance when group prevalences differ. Conversely, class-conditional representation invariance implies equalized odds but is incompatible with calibration and sufficiency. The authors argue that removing demographic information can create new biases or fail to mitigate existing ones, as some encoding is necessary when demographics correlate with target labels.

Entities (12)

Aasa Feragen · person · 99%Aditya Parikh · person · 99%Class-Conditional Representation Invariance · concept · 98%Marginal Representation Invariance · concept · 98%Demographic Parity · fairness-metric · 97%Equalized Odds · fairness-metric · 97%Adult · dataset · 95%CheXpert · dataset · 95%COMPAS · dataset · 95%Demographic Parity Gap · metric · 92%Calibration by Groups · fairness-concept · 90%Fair Normalizing Flows · method · 90%

Relation Signals (10)

Aasa Feragen → isauthorof → Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias

confidence 99% · Aasa Feragen Affiliation: Technical University of Denmark... title: Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias

Aditya Parikh → isauthorof → Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias

confidence 99% · Aditya Parikh Affiliation: Technical University of Denmark... title: Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias

Marginal Representation Invariance → implies → Demographic Parity

confidence 98% · Marginal representation invariance implies demographic parity/independence.

Class-Conditional Representation Invariance → implies → Equalized Odds

confidence 98% · Class-conditional representation invariance implies the separation/equalized odds fairness criterion.

Marginal Representation Invariance → degradesperformancewhen → Unequal Group Prevalences

confidence 96% · Under unequal group prevalences, marginal invariance creates an upper bound on predictive performance.

Class-Conditional Representation Invariance → isincompatiblewith → Demographic Parity

confidence 95% · Class-conditional representation invariance is incompatible with the independence fairness criterion.

Marginal Representation Invariance → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal state. However, while demographic shortcut learning is a genuine threat, some degree of encoding is necessary when demographics correlate with target labels. Here, we show, mathematically and empirically, that enforcing demographic invariance can actually hamper bias mitigation and even create new biases. We distinguish marginal from class-conditional representation invariance, and show that they imply the standard group fairness notions of demographic parity and equalized odds, respectively. We evaluate the effects on predictive performance and fairness of enforcing both invariance types, both theoretically and empirically across five tabular and two chest X-ray imaging datasets. Our findings support our mathematical argument that demographic representation invariance is neither desirable nor sufficient for fairness.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

108,609 characters extracted from source content.

Expand or collapse full text

Is invariance all you need for algorithmic fairness? Removing Demographic Information Can Create New Bias Aditya Parikh Affiliation: Technical University of Denmark, Kgs. Lyngby, Denmark Eike Petersen Affiliation: Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany Stella Frank Affiliation: Technical University of Denmark, Kgs. Lyngby, Denmark Enzo Ferrante Affiliation: CONICET – Universidad de Buenos Aires, ICC, Buenos Aires, Argentina Melanie Ganz Affiliation: University of Copenhagen, Copenhagen, Denmark†Equal contribution. Corresponding author: adipa@dtu.dk Aasa Feragen Affiliation: Technical University of Denmark, Kgs. Lyngby, Denmark Abstract Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal state. However, while demographic shortcut learning is a genuine threat, some degree of encoding is necessary when demographics correlate with target labels. Here, we show, mathematically and empirically, that enforcing demographic invariance can actually hamper bias mitigation and even create new biases. We distinguish marginal from class-conditional representation invariance, and show that they imply the standard group fairness notions of demographic parity and equalized odds, respectively. We evaluate the effects on predictive performance and fairness of enforcing both invariance types, both theoretically and empirically across five tabular and two chest X-ray imaging datasets. Our findings support our mathematical argument that demographic representation invariance is neither desirable nor sufficient for fairness. 11 1 An earlier version is available as a non-peer reviewed preprint (Petersen et al., 2023a). This submission supersedes it with consent of all authors; all experiments are new. Full code, configs, and random seeds are made available anonymously.22 2 Available at: https://anonymous.4open.science/r/invariance-project-931C/README.md 1 Introduction Demographic information can be recovered from latent representations produced by medical imaging models, even when these models are neither explicitly trained to predict demographic attributes, nor trained using them (Glocker et al., 2023; Yang et al., 2024; Duffy et al., 2022; Gichoya et al., 2022; Yi et al., 2021; Lotter, 2024; Quarta et al., 2026). This raises the valid concern that a patient’s group membership, once encoded in the representation, may be exploited by the model, e.g. as shortcuts used instead of task-relevant features (Stanley et al., 2025b; Lin et al., 2024; Brown et al., 2023; Yang et al., 2024), or to reproduce historical biases encoded in systematically biased labels (Akintande et al., 2025; Parikh et al., 2026). In this paper, we argue that this intuitive concern is a misunderstanding. We show, mathematically and empirically, that demographic decodability: the ability to recover group membership from representation, by itself, does not explain unfairness or performance disparities between groups (Jones et al., 2023; Stanley et al., 2025a; Jones et al., 2025). It is entirely possible for a representation to only encode task-specific features, yet still have high demographic decodability, simply because the task-specific features correlate with group membership, e.g., when disease symptoms or prevalence differ between groups. Removing all demographic information therefore risks removing task-specific features associated with group membership and thus harming performance. In the context of different true prevalences, enforcing demographic invariance results in systematic differences in error rates between the groups, harming equalized odds fairness. These mathematical arguments stand in contrast to previous studies reporting positive empirical effects of applying alignment methods such as adversarial learning (Edwards and Storkey, 2016; Zhang et al., 2018) or distribution matching (Louizos et al., 2016) to minimize internal encoding of demographic information (Zemel et al., 2013; Creager et al., 2019; Zhao and Gordon, 2022; Wang et al., 2024; Rashid et al., 2026; Sadri et al., 2025; Sarridis et al., 2024). The theoretical guarantees and fundamental limitations of these methods have, however, received little attention (Zhao et al., 2022). Our experiments support Cerrato et al. (2024) showing that removing demographic information is surprisingly hard, and that most well-known methods fail at accurately removing demographic information from internal representations. This suggests that the regularizing effect of demographic decoding found in previous work is likely caused by failure to actually remove demographic information as intended. We show that marginal representation invariance (i.e., the representation having the same distribution in every demographic group), which is the intuitive and common target of such studies, implies demographic parity, also known as the independence fairness criterion. This is important, as demographic parity is well known to be incompatible with other fairness criteria often deemed more appropriate in real-life applications (Barocas et al., 2023). We distinguish marginal representation invariance, in practice the most commonly considered form of invariance, from class-conditional representation invariance (i.e., the representation is required to have the same distribution only among individuals with the same label), which we show implies the well-known equalized odds, or separation, fairness criterion (Barocas et al., 2023). Our experiments compare a range of different methods for enforcing representation invariance, and we use the one that actually does work. We clearly demonstrate how marginal invariance leads to failure in satisfying equalized odds fairness. Our experiments also show that while class-conditional invariance enforces equalized odds well, it also comes with the known risk of leveling performance down (Mittelstadt et al., 2023) for the best groups in order to obtain fairness. Figure 1: Two forms of representation invariance and the effects of enforcing those in a case study predicting disease for two groups (a1a_1 and a2a_2) with different prevalence. Top: Z⟂AZ A gives identical marginal risk distributions across groups. Under unequal prevalence, the class-conditional risk distributions must differ, giving different error rates across groups. Bottom: Z⟂A|YZ A Y gives equal error rates, but still has drawbacks: it is incompatible with calibration by groups and requires equalizing away potentially important differences between unknown disease subtypes. 2 Problem Setting We denote random variables by uppercase letters (X,Y,Z)(X,Y,Z) and their realizations by lowercase letters (x,y,z)(x,y,z). Consider a supervised prediction task with input X∈X , outcome R∈ℛR sometimes thresholded to a categorical Y∈Y , and sensitive attribute A∈=a1,…,amA =\a_1,…,a_m\ with P⁡(A=a)>0P(A=a)>0 for all a. The input X can be tabular features, an image, or any other data type. The outcome can be binary (Y∈0,1Y∈\0,1\), multiclass (Y∈0,1…,K−1Y∈\0,1…,K-1\), a regressed score R∈ℛR , or even a more complex output such as a segmentation mask. The attributes A identify groups such as sex, ethnicity, or age group, and may be multi-valued or intersectional. A deterministic feature extractor g:→g produces a representation Z=g⁡(X)Z=g(X), which is mapped to predictions R∈ℛR by a second deterministic component (e.g., a classifier head) h:→ℛh , producing a risk/score vector R=h⁡(Z)R=h(Z). If required, R may be discretized to obtain binary or categorical outcome labels Y Y. Neither g, h, nor τ take A as input. We consider a model to encode33 3 We use the term encoding for consistency with the literature. Above-chance prediction of A from Z does not necessarily mean the model explicitly represents demographic identity or uses it as a shortcut. group membership A in its latent representation Z if group membership is decodable from Z above chance (i.e. Z⟂̸AZ A). We use decodability as a measurement (or a proxy) of invariance throughout. Notice that decodability depends on the evaluation dataset D and is not a property of the model alone. 3 What does Representation Invariance Guarantee? While the propositions in this section are not new individually, the way we connect them with two forms of representation invariance (illustrated in Fig. 1) and derive their consequences for fairness and performance is new. This connection is important to avoid common misunderstandings that may increase bias and decrease performance. We cite prior work where proofs are available. Sec. 4 then introduces an experimental setup that allows us to empirically test these propositions in Sec. 5. 3.1 Marginal Representation Invariance Marginal representation invariance asks that the distribution of Z be identical across groups: p(z∣a)=p(z)∀a⟺Z⟂Ap(z a)=p(z) ∀ a Z A (1) Under this condition, the learned representation Z is statistically independent of A, denoted by Z⟂AZ A: the model does not encode group membership on this dataset. Proposition 1. Marginal representation invariance implies demographic parity/independence. Proof. If the risk prediction R and the decision Y^=[R≥τ] Y= 1[R≥τ] depend only on Z, then both predictions inherit this independence of A from Z: Z⟂A⟺p(z∣a)=p(z)∀a⟹p(r∣a)=p(r),p(y^∣a)=p(y^),∀a.Z A\; \;p(z a)=p(z)\ ∀ a\; \;p(r a)=p(r), p( y a)=p( y), ∀ a. (2) That is, if Z⟂AZ A, then we get R⟂AR A and Y^⟂A Y A: the distributions of predicted risk scores and predicted decisions are identical across sensitive groups, which implies the fairness criterion known as demographic parity, or independence. ∎ For a binary attribute A∈0,1A∈\0,1\ this gives P⁡(Y^=1∣A=0)=P⁡(Y^=1∣A=1)P( Y=1 A=0)=P( Y=1 A=1), whose violation is measured by the demographic parity gap ΔDP=|P⁡(Y^=1∣A=0)−P⁡(Y^=1∣A=1)|. _DP= |P( Y=1 A=0)-P( Y=1 A=1) |. (3) Therefore, exact marginal representation invariance implies ΔDP=0 _DP=0 if the classification head and the decision threshold do not use A (Zemel et al., 2013; Zhao et al., 2022; Madras et al., 2018). Its finite-sample estimate may differ from zero (ΔDP≈0)( _DP≈ 0) due to sampling variability. Unequal prevalence across groups causes problems for marginal invariance Outcomes Y often vary across demographic groups: Income differs across age groups or sex; crime rates differ across populations due to historical and institutional factors; disease prevalence depends on e.g. sex and age, with breast cancer, which affects some but very few men, as an extreme example. Proposition 2. Under unequal group prevalences, marginal invariance creates an upper bound on predictive performance. Proof. While marginal invariance requires p⁡(y^∣A=0)=p⁡(y^∣A=1)p( y A=0)=p( y A=1), an accurate classifier should approximately reproduce the group-specific outcome distributions p⁡(y^∣a)≈p⁡(y∣a)p( y a)≈ p(y a). However, when p⁡(Y∣A=0)≠p⁡(Y∣A=1)p(Y A=0)≠ p(Y A=1), these requirements, i.e., demographic parity and perfect (desired) prediction, cannot both hold. Marginal representation invariance must thus introduce prediction errors in at least one group. ∎ For a binary outcome, the group-conditional representation distribution is p⁡(z∣a)=∑y∈0,1p⁡(z∣y,a)p(y∣a)=πap(z∣Y=1,A=a)+(1−πa)p(z∣Y=0,A=a), splitp(z a)&= _y∈\0,1\p(z y,a)\,p(y a)\\ &= _a\,p(z Y=1,A=a)+(1- _a)\,p(z Y=0,A=a), split (4) where πa=P⁡(Y=1∣A=a) _a=P(Y=1 A=a) denotes the positive-label prevalence in group a. If the within-class distributions are shared across groups, p⁡(z∣y,a)=p⁡(z∣y)p(z y,a)=p(z y), and Z⟂̸YZ Y: the difference in group-wise p⁡(z∣a)p(z a) originates in πa _a. For a binary prediction task, let ea=P⁡(Y^≠Y∣A=a)e_a=P( Y≠ Y A=a) denote the prediction error for group a. Any predictor satisfying demographic parity obeys e0+e1≥|π0−π1|e_0+e_1≥| _0- _1| (McNamara et al., 2019; Zhao et al., 2022). For multi-class tasks, an iterative division into binary one-versus-rest prediction problems can be used to derive a similar bound. The bound constrains the combined error across groups; the entire cost may fall on one group. 3.2 Class-Conditional Representation Invariance Figure 2: Effects of enforcing Class-conditional representation invariance in the presence of intra-class variation. When the within-class distribution of a clinically relevant factor (here disease severity among Y=1Y=1 patients) differs between groups, enforcing Z⟂A|YZ A Y requires mapping patients with differing disease severities to the same latent representation, depending on which group a patient belongs to (center). The aligned representation then hides a real difference in disease presentation, and the same z no longer means the same severity in both groups (see also Lohaus et al. (2022)). Class-conditional representation invariance instead aligns the representation distributions separately within each outcome class: p(z∣y,a)=p(z∣y)∀a,y⟺Z⟂A∣Y.p(z y,a)=p(z y) ∀ a,y Z A Y. (5) This condition allows the marginal representation distributions p⁡(z∣a)p(z a) to differ when the groups have different outcome prevalences. Thus, class-conditional representation invariance does not imply marginal representation invariance (Z⟂A|Y⟹̸Z⟂AZ A Y\; \;Z A). Proposition 3. Class-conditional representation invariance implies the separation/equalized odds fairness criterion. Proof. If the risk prediction R and thresholded prediction Y Y depend only on Z, then they inherit the conditional independence of Z implying R⟂A|YandY^⟂A|Y.R A Y~and~ Y A Y. This property is known as separation (Hardt et al., 2016; Madras et al., 2018); separation on the classification level (Y Y) in the case of binary classification is also known as equalized odds. ∎ We note that when Y=1Y=1 we get equal TPRsTPRs (equal opportunity fairness criterion), whereas with the additional Y=0Y=0 we also get equal FPRsFPRs. Therefore, class-conditional representation invariance implies ΔEOpp=0andΔEO=0 _EOpp=0~and~ _EO=0. 3.3 Invariance-fairness incompatibilities Combining Props. 1 and 3 with known fairness incompatibility theorems, we obtain the following invariance-fairness incomparability results for classification: Corollary 1. Assume that R and Y are not independent of A. In this case: (i) Assume that Y is binary; then marginal representation invariance is incompatible with the separation fairness criterion. If group-wise prevalences differ, representation invariance is incompatible with separation even for multiple classes. (i) Marginal representation invariance is incompatible with the sufficiency fairness criterion, as well as with calibration by groups. (i) Class-conditional representation invariance is incompatible with the independence fairness criterion. (iv) Assume that all events in the joint distribution of (A,R,Y)(A,R,Y) have positive probability. Then, class-conditional representation invariance is incompatible with the sufficiency fairness criterion, as well as with calibration by groups. Proof. (i) If marginal invariance holds, the independence criterion holds by Prop. 1. For binary Y, (i) follows from (Barocas et al., 2023, Ch. 3, Prop. 3). If prevalences depend on A and marginal independence holds, then constant predictive rates lead to differences in either TPR, FPR, or both, across those groups where prevalences differ. As a consequence, separation does not hold. (i) If marginal invariance holds, the independence criterion holds by Prop. 1, and thus sufficiency cannot hold by (Barocas et al., 2023, Ch. 3, Prop. 2). If calibration by groups were to hold, then sufficiency would hold as well, which it cannot; this proves the claim for calibration by groups. (i) If class-conditional invariance holds, then equalized odds holds by Prop. 3. Thus, independence cannot hold by (Barocas et al., 2023, Ch. 3, Prop. 3). (iv) If class-conditional invariance holds, then equalized odds holds by Prop. 3. Thus, sufficiency cannot hold by (Barocas et al., 2023, Ch. 3, Prop. 4). If calibration by groups were to hold, then sufficiency would hold as well, which it cannot; this proves the claim for calibration by groups. ∎ These results all follow from known incompatibility results as stated in (Barocas et al., 2023); we note that even stronger or more detailed fairness incompatibility results from the literature will generalize into invariance-fairness incompatibility results following the same line of reasoning. We also note that groupwise prevalence differences, which are used as a running challenging example in this paper, are a special case of A and Y not being independent, which is an assumption for all these incompatibility results. Fig. 2 shows a further concern about class-conditional invariance conflicting with within-class differences in disease severity across groups. 3.4 What happens under prevalence shifts? Consider a fixed model evaluated on two populations P and Q, with unchanged within-class distribution (i.e., P⁡(X∣Y,A)=Q⁡(X∣Y,A)P(X Y,A)=Q(X Y,A)) but a change in group-specific prevalence (P⁡(Y∣A)≠Q⁡(Y∣A)P(Y A)≠ Q(Y A)). This could, e.g., happen if a disease over time becomes more prevalent in one group due to, e.g., updated diagnostic criteria that fit the group better, or behavioral changes that affect one group but not the other (e.g., a new contraceptive pill entering the market affecting women, but not men). It also commonly occurs when intentionally up- or downsampling certain classes in training and evaluation datasets. In such settings, we show: Proposition 4. Class-conditional representation invariance is preserved under group-specific prevalence shifts, while marginal representation invariance is not. Proof. As the model is fixed, its conditional representation distribution remains unchanged (P⁡(Z∣Y,A)=Q⁡(Z∣Y,A)P(Z Y,A)=Q(Z Y,A)). Therefore, if representations across groups for each outcome Y are aligned for P, they also remain aligned for Q (Asiaee and Aryan, 2026): Z⟂PA∣Y⟹Z⟂QA∣Y.Z _PA Y Z _QA Y. For binary outcomes, let πaQ=Q⁡(Y=1∣A=a) _a^Q=Q(Y=1 A=a). The representation distribution for Q becomes Q⁡(Z∣A=a)=πaQP(Z∣Y=1,A=a)+(1−πaQ)P(Z∣Y=0,A=a).Q(Z A=a)= _a^QP(Z Y=1,A=a)+(1- _a^Q)P(Z Y=0,A=a). Changing π also changes the proportions of positive and negative representations in each group. Their overall distributions can therefore become different, even when they were aligned under P: Z⟂PA ／⟹Z⟂QA.Z _PA to0.0pt 3.75pt -5.27776pt$ $ to0.0pt 3.75pt -5.27776pt$ $ to0.0pt 2.625pt -4.45831pt$ $ to0.0pt 1.875pt -3.95834pt$ $ Z _QA. This finalizes our proof of both claims. ∎ These results show that marginal representation invariance depends on the population it is evaluated on (Zhao and Gordon, 2022). Class-conditional representation alignment, on the other hand, is preserved under group-specific prevalence shift. Figure 3: Fig. 1 estimated from data (Dutch, shift-50, seed 42). Rows: ERM, Z⟂AZ A, Z⟂A|YZ A Y. Left: marginal risk densities p⁡(r∣a)p(r a) A∈0,1A∈\0,1\, with ΔDP _DP and DAD_A. Middle and right: class-conditional densities p⁡(r∣y,a)p(r y,a) within each group, with ΔEO _EO and DA|YD_A Y. Dotted: Youden threshold. Marginal alignment makes the left column coincide by pulling the within-class curves apart across groups (P1P_1); class-conditional alignment makes the middle and right columns coincide and leaves the left column at the base-rate gap (P3P_3). 4 Experimental Design Datasets and prevalence conditions: We examine five tabular benchmarks (Adult, Dutch, Communities & Crime, Taiwan credit, COMPAS) (Kohavi, 1996; Calders and Verwer, 2010; Redmond and Baveja, 2002; Yeh and Lien, 2009) and two CheXpert tasks (edema, cardiomegaly) (Irvin et al., 2019); depending on the dataset, sensitive attributes are sex, race, or racial composition. Tabular datasets are used to experiment with different group prevalence disparities: natural, equalized, and removal of 50% or 75% of positive examples from the lower-prevalence group (shift-50, shift-75), resulting in substantial prevalence disparities. In CheXpert, we simply remove 50% of female-positive cases and compare with natural prevalence. Each condition is a fixed subsample, so every train/test split shares the same prevalence under the given condition (See App. 6). Methods: All methods are compared against empirical risk minimization (ERMERM) baselines. On tabular data, we use categorical Fair Normalizing Flows (FNFs) (Balunović et al., 2022) sweeping over γ∈0,.02,.1,.2,.5,.9,1γ∈\0,.02,.1,.2,.5,.9,1\, two MADE hidden widths 50,100\50,100\, and two feature sets, across four prevalence conditions, and three seeds 42,123,456\42,123,456\ to yield 5×4×3×(7×2×2+2)=1,8005× 4× 3×(7× 2× 2+2)=1,800 runs. For CheXpert, we compare the invariant methods (FNFs, MMD regularizer, adversarial) across two tasks, with the same three seeds, γ∈.05,0.50,0.95γ∈\.05,0.50,0.95\, yielding 2×3×(3×3+1)=602× 3×(3× 3+1)=60 runs. Data splits are 60/10/10/20% (tabular) and 60/5/5/10/20% (CheXpert) for training, validation, probe-validation, and test partitions; thresholds are selected on the validation set. Evaluation: Decodability DAD_A is measured with independent probes (logistic regression, gradient-boosted tree, 3 MLPs) trained on held-out probe-validation representations. We evaluate predictive performance using the AUROC score. For fairness evaluation, we report group differences: ΔDP _DP, ΔEO _EO and ΔEOpp _EOpp, along with ΔPPV and ΔNPV . Calibration is measured by calibration bias (Δbias=|(y¯1−p¯1)−(y¯0−p¯0)| _bias=|( y_1- p_1)-( y_0- p_0)|) and group-wise recalibration intercepts Δcal=|a1−a0| _cal=|a_1-a_0| (Cox, 1958) (unlike ECEECE, both stay unbiased under unequal group sizes (Petersen et al., 2023b)). All results are reported as mean ± standard deviation over training seeds. Paired bootstrapping uses the same samples for each method and is matched to the ERM baseline: 1,000 resamples for tabular, and 2,000 resamples for CheXpert. Additional results for robustness, including 95% percentile intervals, Bonferroni-adjusted confidence intervals for probe-AUROC, sensitivity analysis over γ, and calibration-bin tests, are in Appendixes A and B. 5 Results Utility Decodability of A Indep. Separation Sufficiency Group calibration Method AUROC DAD_A DA|YD_A Y ΔDP _DP ΔEO _EO ΔPPV/ΔNPV _PPV\,/\, _NPV Δbias/Δcal _bias\,/\, _cal Adult ERM 0.890.89 0.930.93 0.990.99 0.400.40 0.280.28 0.18/ 0.050.18\,/\,0.05 0.02/ 0.320.02\,/\,0.32 Z⟂AZ A 0.790.79 0.580.58 0.850.85 0.04✓0.04\, 0.400.40 0.49/ 0.040.49\,/\,0.04 0.23/ 2.120.23\,/\,2.12 Z⟂A|YZ A Y 0.860.86 0.620.62 0.550.55 0.170.17 0.08✓0.08\, 0.40/ 0.070.40\,/\,0.07 0.15/ 1.800.15\,/\,1.80 Dutch ERM 0.900.90 0.770.77 0.770.77 0.300.30 0.050.05 0.27/ 0.280.27\,/\,0.28 0.22/ 2.000.22\,/\,2.00 Z⟂AZ A 0.710.71 0.520.52 0.790.79 0.01✓0.01\, 0.310.31 0.63/ 0.210.63\,/\,0.21 0.44/ 2.320.44\,/\,2.32 Z⟂A|YZ A Y 0.890.89 0.680.68 0.560.56 0.270.27 0.03✓0.03\, 0.34/ 0.270.34\,/\,0.27 0.23/ 2.050.23\,/\,2.05 C & C ERM 0.850.85 0.790.79 0.660.66 0.430.43 0.200.20 0.50/ 0.300.50\,/\,0.30 0.23/ 2.100.23\,/\,2.10 Z⟂AZ A 0.670.67 0.550.55 0.730.73 0.05✓0.05\, 0.280.28 0.71/ 0.310.71\,/\,0.31 0.55/ 2.870.55\,/\,2.87 Z⟂A|YZ A Y 0.850.85 0.740.74 0.570.57 0.400.40 0.280.28 0.53/ 0.370.53\,/\,0.37 0.29/ 2.340.29\,/\,2.34 Taiwan ERM 0.750.75 0.630.63 0.620.62 0.070.07 0.080.08 0.19/ 0.060.19\,/\,0.06 0.08/ 0.680.08\,/\,0.68 Z⟂AZ A 0.680.68 0.520.52 0.690.69 0.01✓0.01\, 0.140.14 0.28/ 0.040.28\,/\,0.04 0.11/ 0.860.11\,/\,0.86 Z⟂A|YZ A Y 0.750.75 0.540.54 0.530.53 0.040.04 0.01✓0.01\, 0.18/ 0.070.18\,/\,0.07 0.09/ 0.790.09\,/\,0.79 COMP. ERM 0.710.71 0.680.68 0.650.65 0.210.21 0.150.15 0.23/ 0.150.23\,/\,0.15 0.13/ 0.780.13\,/\,0.78 Z⟂AZ A 0.640.64 0.560.56 0.630.63 0.02✓0.02\, 0.110.11 0.33/ 0.180.33\,/\,0.18 0.24/ 1.190.24\,/\,1.19 Z⟂A|YZ A Y 0.690.69 0.560.56 0.540.54 0.100.10 0.04✓0.04\, 0.27/ 0.180.27\,/\,0.18 0.20/ 1.090.20\,/\,1.09 Edema ERM 0.840.84 0.980.98 0.980.98 0.150.15 0.100.10 0.08/ 0.020.08\,/\,0.02 0.01/ 0.090.01\,/\,0.09 Z⟂AZ A 0.830.83 0.620.62 0.630.63 0.01✓0.01\, 0.050.05 0.16/ 0.060.16\,/\,0.06 0.10/ 0.800.10\,/\,0.80 Z⟂A|YZ A Y 0.830.83 0.600.60 0.590.59 0.050.05 0.02✓0.02\, 0.13/ 0.050.13\,/\,0.05 0.07/ 0.580.07\,/\,0.58 Cardio. ERM 0.850.85 0.980.98 0.980.98 0.130.13 0.090.09 0.09/ 0.020.09\,/\,0.02 0.01/ 0.190.01\,/\,0.19 Z⟂AZ A 0.840.84 0.640.64 0.650.65 0.01✓0.01\, 0.050.05 0.17/ 0.030.17\,/\,0.03 0.07/ 0.960.07\,/\,0.96 Z⟂A|YZ A Y 0.840.84 0.620.62 0.620.62 0.040.04 0.01✓0.01\, 0.15/ 0.030.15\,/\,0.03 0.05/ 0.740.05\,/\,0.74 Table 1: Fairness criteria under the shift-50 on test split, mean over 33 seeds. Z⟂AZ A is marginal FNF at γ=1γ=1 (CheXpert: γ=0.95γ=0.95; strongest setting), Z⟂A|YZ A Y class-conditional FNF at the same γ, both with the feature set and width selected by the lowest probe AUROC. DAD_A and DA|YD_A Y are the maximum probe AUROC for recovering A from Z marginally and within class. ✓ denotes the criterion each invariance target guarantees (P1,3P_1,3); every other Δ is not-stable relative to ERM. ± are omitted for space and reported, together with the full γ and feature-set sweep and the remaining CheXpert configurations, in Appendix A.4 and A.3. Table 1 summarizes the shift-50 case; the rest, including full-hyperparameter sweeps and per-seed scores, are in the Appendix A.4. We find that DAD_A drops for every alignment method, relative to ERMERM, but no method achieves exact statistical invariance: the best methods, i.e., with the lowest DAD_A values, reach AUROCAUROC 0.520.52–0.580.58, while the within-class probes reach 0.530.53–0.620.62. The results reflect approximate invariance, not exact invariance; therefore, we advise reading fairness conclusions with residual dependence in mind. In the following, we refer to propositions from Section 3 using PnP_n. Marginal alignment achieves independence but not separation or sufficiency (P1,2P_1,2). Marginal alignment methods reduce ΔDP DP to 0.01−0.050.01-0.05 across all datasets (P1P_1). However, due to inherent incompatibility, it degrades separation (ΔEO EO). On the Dutch dataset, probe-AUC reduces 0.77→0.520.77→ 0.52 and the ΔDP DP 0.30→0.010.30→ 0.01, while ΔEO EO increases 0.05→0.310.05→ 0.31. The predictive performance cost reflects the e0+e1≥|π0−π1|e_0+e_1≥| _0- _1| bound, resulting in task-AUROCAUROC drop of 0.070.07–0.190.19 points across datasets (P2P_2) (Detailed in Appendix A.1). Finally, forcing identical score distributions across groups breaks sufficiency and calibration:ΔPPV PPV increases in four of five datasets (Dutch 0.27→0.630.27→ 0.63; Adult 0.18→0.490.18→ 0.49; C&C 0.50→0.710.50→ 0.71; Credit 0.19→0.280.19→ 0.28), and the group calibration-bias gap Δbias _bias nearly doubles (Dutch 0.22→0.440.22→ 0.44; C&C 0.23→0.550.23→ 0.55). Class-conditional alignment implies separation (P3P_3). Class-condition representation invariance produces the smallest ΔEO EO (Dutch →0.030.05\!→\!0.03, Credit →0.010.08\!→\!0.01, COMPAS →0.040.15\!→\!0.04, Adult →0.080.28\!→\!0.08; the exception is C&C whose stratum for A=0A=0 has <80<80 rows), implying separation (P3P_3). Unlike marginal alignment, task-AUROCAUROC is maintained within 0.01−0.030.01-0.03 of ERMERM. Calibration and sufficiency metrics slightly degrade (Dutch ΔPPV PPV: 0.27→0.340.27→ 0.34; Δbias _bias: 0.22→0.230.22→ 0.23), showing conflict between score-level separation and calibration. Notice this only applies to risk score separation (R⟂A|YR A Y); calibrated scores can still yield equal error rates under thresholded decisions (Lazar Reich and Vijaykumar, 2021). Prevalence shifts break marginal but not conditional alignment (P4P_4). Fig. demonstrates evaluation stability under change in population shifts. When base-rate P⁡(Y∣A)P(Y A) changes while P⁡(X∣A,Y)P(X A,Y) remains fixed: marginal alignment methods see increase in ΔDP _ DP (0.02(equalized)→0.19(shift-75)0.02~(equalized)→ 0.19~(shift-75)), hence independence is lost. Under the same shift, conditional alignment maintains a stable ΔEO _ EO (0.05→0.030.05→ 0.03), but ΔDP _ DP increases (0.01→0.01→ 0.32); confirming P4P_4. Figure 4: Guarantees under a change of evaluation population (P4P_4). Trained on the equalized condition (shaded) and evaluated on increasing base-rate gap: P⁡(Y∣A)P(Y A) changes, P⁡(X∣Y,A)P(X Y,A) fixed. Z⟂AZ A loses parity as the gap grows; Z⟂A|YZ A Y keeps ΔEO _EO flat and loses parity by exactly the base-rate gap. Δbias _bias grows. Mean ± std over three seeds. 6 Discussion Removing demographic information as a fairness objective? The motivating question for this work is whether models should be trained to not encode group membership in their latent space. Our analysis shows that representation invariance is neither desirable nor sufficient for fairness. Enforcing strict marginal representation invariance while ensuring demographic parity also removes genuinely useful information and hurts model performance when outcome prevalence differs (Barocas et al., 2023; Dwork et al., 2012; Zhao and Gordon, 2022; Zhao et al., 2022). This goes beyond the well-known leveling down effect (Mittelstadt et al., 2023; Zietlow et al., 2022), as we do not just harm performance for the advantaged group; marginal invariance is incompatible with commonly desired fairness criteria. In short, removing information via marginal alignment methods is harmful to the fairness objective: enforcing parity at the cost of performance, separation, and sufficiency. Removing unnecessarily strong demographic encoding is desirable. As illustrated in Table 1, class-conditional representation invariance reduces decodability without costing AUROCAUROC and improves EOEO relative to ERMERM. This suggests that some (excess) demographic information learned by ERMERM can be erased without costing fairness and performance. This presents a different trade-off, removing information about intra-class variations (e.g. comorbidities or disease phenotypes, see Castro et al. (2020); Li et al. (2020); Mahajan et al. (2021); Oakden-Rayner et al. (2020)), and compromises calibration by groups (Barocas et al., 2023; Hedden, 2021), although recent work questions whether this is truly problematic (Kina and Petersen, 2026). Every trade-off we discuss stays aligned with the well-established incompatibility between independence, separation and sufficiency (Kleinberg et al., 2017; Barocas et al., 2023). Even with the reported cost of class-conditional invariance, there is no reason to prefer marginal representation invariance over it. The strength of constraints matters. Representations, scores, and thresholded decisions impose different requirements. Marginal representation invariance makes the score distribution identical across all groups. However, at a chosen threshold, achieving parity only requires the same positive prediction rate. Similarly, aligning Z within each outcome class aligns the class-conditional score distribution, whereas achieving separation at a threshold requires equal TPRsTPRs and FPRsFPRs. This shows that representation invariance is sufficient but not necessary for satisfying decision-level criteria (Madras et al., 2018). This difference also matters for calibration: under unequal prevalences, class-conditional alignment of risk scores generally conflicts with calibration by groups, even though equality of error rates at a chosen threshold does not impose the same restrictions (Lazar Reich and Vijaykumar, 2021; Petersen et al., 2023b). Further relaxations are conceivable and potentially desirable, such as distribution support matching (Tong et al., 2022), or matching only the first moment of the conditional score distributions (Kina and Petersen (2026); also see Appendix A.6). When can representation invariance help? Our results do not imply that invariance-based methods are never useful, and the drawbacks discussed depend on what form is applied. Marginal representation invariance should generally be avoided when group prevalences (πa _a) are not near equal. Invariance might also be used as a less restrictive mitigation technique (Wu et al., 2022; Chiu et al., 2023) by using weaker regularization strength (in our experiments, γ<1γ<1 achieves a relatively better performance-parity tradeoff) or in unsupervised domain adaptation where labels y are unavailable for the target domain (Moyer et al., 2020; Li et al., 2018). Here, representation invariance should be applied strategically, acknowledging the associated risks, and favoring conditional approaches for supervised tasks or optimizing post-hoc thresholds if strict latent alignment is unnecessary (Hardt et al., 2016; Lazar Reich and Vijaykumar, 2021). Statistical invariance vs. model invariance. We discuss a case where a model treats two patients with similar physiology differently due to their group membership. A model is invariant to A if any change in A while keeping everything else fixed leaves y y unchanged. This still allows Y Y and A to be correlated when the labels Y correlate with A. By contrast, a marginally invariant representation requires Z,Y^⟂AZ, Y A even when Y and A are correlated. Enforcing this can require disparate treatment: if disease presentations differ between groups, the model still maps them to identical distributions by treating similar patients from different groups differently (Lohaus et al., 2022). To address this, an alternative conceptual goal is to learn invariant mechanisms that are stable across all environments (Arjovsky et al., 2019; Makar et al., 2022; Puli et al., 2022; Schölkopf et al., 2021; Subbaswamy and Saria, 2020). One approach is counterfactual invariance, where the attribute A or a specific intervention should not change the model’s prediction (Veitch et al., 2021). When A is a demographic attribute, this corresponds to counterfactual fairness (Kusner et al., 2017). However, defining these counterfactuals is challenging in medical imaging. Changing A may also change anatomy, physiology, overlapping diseases, or even disease presentation. Therefore, specifying which factors change and which remain fixed can affect the predictions. This makes any fairness guarantees dependent on these modeling choices, their (unknown or undiscovered) confounding, and also the validity of the causal model (Chiappa and Gillam, 2018; Kilbertus et al., 2020). Limitations of our work. We advise reading our findings with caution. Our models and methods achieved approximate rather than exact statistical invariance, leaving demographic traces in representations. However, this does not change our primary conclusion. Our empirical analysis is limited to classification tasks and simulated prevalence differences between groups, which may not reflect real-world extremes. Fairness gaps and calibration costs are sensitive to operating thresholds, compression, or model selection. As future work, we plan to study why decodability behaves differently across data domains and methods, and expand our analysis to complex tasks like segmentation. 7 Conclusion We have shown that representation invariance, in spite of its intuitive appeal for fair AI, is incompatible with most popular fairness criteria when A and Y are not independent. We show, moreover, that previous work showing the benefit of representation independence might have been a result of failure to completely remove demographic information, suggesting that while invariance is often not desirable, reducing the encoding of demographic information can have a regularizing effect on both performance and fairness. In particular, we implore the community to avoid the often strongly highlighted concerns regarding encoded demographic information. While encoded demographics may sound deeply alarming to users and even developers not familiar with the invariance-fairness-performance tradeoff, these encoded demographics are, in many practical applications, necessary to obtain the most equitable and high-performing models. There is a real risk of demographic shortcut learning in the case where models encode demographic information more strongly than is necessary for accurate prediction, as was the case for all ERMERM models in our experiments. In such cases, the impetus must not be to aim for no demographic encoding, however, but rather to limit encoding to the smallest level that is sufficient for maintaining accurate, robust, and fair prediction. Generative AI Disclosure Following fair and responsible use of AI for research, we list the use of generative AI for the following sub-tasks: • Major use: Writing boilerplate code for the experimental pipeline, automated logging and tracking runs across thousands of experiments, formatting LaTeX tables for the main paper and appendix (entries were verified). • Minor use: Writing scripts to visualize metrics from raw logs, and mapping different notations across the literature (all mathematical proofs and steps are verified), minor checks for grammar, phrasing, flow, and typo fixes. • Not used for: Ideation, core methodology, deriving theoretical proofs, main manuscript, literature referencing and the bibliography, or structuring the paper. All AI-assisted outputs were manually verified by the authors. The authors take full responsibility for the final output and claims in this work. References Akintande et al. (2025) O. J. Akintande, S. A. Bigdeli, and A. Feragen Medicine after death: XAI and algorithmic fairness under label bias. In Fourth European Workshop on Algorithmic Fairness, p. 171–186. Cited by: §1. Arjovsky et al. (2019) M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz Invariant risk minimization. arXiv preprint arXiv:1907.02893. Cited by: §6. Asiaee and Aryan (2026) A. Asiaee and K. Aryan Fairness under group-conditional prior probability shift: invariance, drift, and target-aware post-processing. External Links: 2602.05144, Link Cited by: §3.4. Balunović et al. (2022) M. Balunović, A. Ruoss, and M. Vechev Fair Normalizing Flows. In International Conference on Learning Representations, Cited by: §B.1, §4. Barocas et al. (2023) S. Barocas, M. Hardt, and A. Narayanan Fairness and machine learning: limitations and opportunities. MIT press. Cited by: §1, §3.3, §3.3, §3.3, §3.3, §3.3, §6, §6. Belrose et al. (2023) N. Belrose, D. Schneider-Joseph, S. Ravfogel, R. Cotterell, E. Raff, and S. Biderman Leace: perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems 36, p. 66044–66063. Cited by: Appendix C. Brown et al. (2023) A. Brown, N. Tomasev, J. Freyberg, Y. Liu, A. Karthikesalingam, and J. Schrouff Detecting shortcut learning for fair medical ai using shortcut testing. Nature communications 14 (1), p. 4314. Cited by: §1. Calders and Verwer (2010) T. Calders and S. Verwer Three naive Bayes approaches for discrimination-free classification. Data mining and knowledge discovery 21 (2), p. 277–292. Cited by: §4. Castro et al. (2020) D. C. Castro, I. Walker, and B. Glocker Causality matters in medical imaging. Nature Communications 11 (1), p. 3673. Cited by: §6. Cerrato et al. (2024) M. Cerrato, M. Köppel, P. Wolf, and S. Kramer 10 years of fair representations: challenges and opportunities. External Links: 2407.03834, Link Cited by: §1. Chiappa and Gillam (2018) S. Chiappa and T. P. S. Gillam Path-specific counterfactual fairness. External Links: 1802.08139, Link Cited by: §6. Chiu et al. (2023) C. Chiu, H. Chung, Y. Chen, Y. Shi, and T. Ho Toward fairness through fair multi-exit framework for dermatological disease diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 97–107. Cited by: §6. Cox (1958) D. R. Cox Two further applications of a model for binary regression. Biometrika 45 (3/4), p. 562–565. Cited by: Table 8, §4. Creager et al. (2019) E. Creager, D. Madras, J. Jacobsen, M. Weis, K. Swersky, T. Pitassi, and R. Zemel Flexibly fair representation learning by disentanglement. In International conference on machine learning, p. 1436–1445. Cited by: §1. Duffy et al. (2022) G. Duffy, S. L. Clarke, M. Christensen, B. He, N. Yuan, S. Cheng, and D. Ouyang Confounders mediate AI prediction of demographics in medical imaging. NPJ digital medicine 5 (1), p. 188. Cited by: §1. Dwork et al. (2012) C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, p. 214–226. Cited by: §6. Edwards and Storkey (2016) H. Edwards and A. J. Storkey Censoring Representations with an Adversary. In International Conference on Learning Representations, Cited by: §1. Ganin et al. (2016) Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky Domain-adversarial training of neural networks. Journal of machine learning research 17 (59), p. 1–35. Cited by: Appendix C. Gichoya et al. (2022) J. W. Gichoya, I. Banerjee, A. R. Bhimireddy, J. L. Burns, L. A. Celi, L. Chen, R. Correa, N. Dullerud, M. Ghassemi, S. Huang, et al. AI recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health 4 (6), p. e406–e414. Cited by: §1. Glocker et al. (2023) B. Glocker, C. Jones, M. Bernhardt, and S. Winzeck Algorithmic encoding of protected characteristics in chest x-ray disease detection models. EBioMedicine 89. Cited by: §1. Gretton et al. (2006) A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola A kernel method for the two-sample-problem. Advances in neural information processing systems 19. Cited by: Appendix C. Hardt et al. (2016) M. Hardt, E. Price, and N. Srebro Equality of opportunity in supervised learning. Advances in neural information processing systems 29. Cited by: §3.2, §6. Hedden (2021) B. Hedden On statistical criteria of algorithmic fairness. Philosophy and Public Affairs 49, p. 209. Cited by: §6. Irvin et al. (2019) J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. L. Ball, K. S. Shpanskaya, J. Seekins, D. A. Mong, S. S. Halabi, J. K. Sandberg, R. Jones, D. B. Larson, C. P. Langlotz, B. N. Patel, M. P. Lungren, and A. Y. Ng Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33, p. 590–597. Cited by: §4. Jones et al. (2025) C. Jones, F. de Sousa Ribeiro, M. Roschewitz, D. Castro, and B. Glocker Rethinking fair representation learning for performance-sensitive tasks. In International Conference on Learning Representations, Vol. 2025, p. 29269–29288. Cited by: §1. Jones et al. (2023) C. Jones, M. Roschewitz, and B. Glocker The role of subgroup separability in group-fair medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 179–188. Cited by: §1. Kilbertus et al. (2020) N. Kilbertus, P. J. Ball, M. J. Kusner, A. Weller, and R. Silva The sensitivity of counterfactual fairness to unmeasured confounding. In Uncertainty in artificial intelligence, p. 616–626. Cited by: §6. Kina and Petersen (2026) M. A. Kina and E. Petersen Prevalence calibration as shortcut mitigation. In 17th International Workshop on Machine Learning in Medical Imaging (MLMI 2026) at MICCAI 2026, Cited by: §A.5, §6, §6. Kleinberg et al. (2017) J. Kleinberg, S. Mullainathan, and M. Raghavan Inherent Trade-Offs in the Fair Determination of Risk Scores. In 8th Innovations in Theoretical Computer Science Conference (ITCS 2017), Vol. 67, p. 43:1–43:23. Note: Keywords: algorithmic fairness, risk tools, calibration External Links: Document Cited by: §6. Kohavi (1996) R. Kohavi Scaling up the accuracy of Naive-Bayes classifiers: a decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD’96, p. 202–207. Cited by: §4. Kusner et al. (2017) M. J. Kusner, J. Loftus, C. Russell, and R. Silva Counterfactual fairness. Advances in neural information processing systems 30. Cited by: §6. Lazar Reich and Vijaykumar (2021) C. Lazar Reich and S. Vijaykumar A Possibility in Algorithmic Fairness: Can Calibration and Equal Error Rates Be Reconciled?. In 2nd Symposium on Foundations of Responsible Computing (FORC 2021), Leibniz International Proceedings in Informatics (LIPIcs), Vol. 192, p. 4:1–4:21. Note: Keywords: fair prediction, impossibility results, screening decisions, classification, calibration, equalized odds, optimal transport, risk scores External Links: Document Cited by: §5, §6, §6. Li et al. (2020) B. Li, Y. Wang, T. Che, S. Zhang, S. Zhao, P. Xu, W. Zhou, Y. Bengio, and K. Keutzer Rethinking distributional matching based domain adaptation. arXiv preprint arXiv:2006.13352. Cited by: §6. Li et al. (2018) Y. Li, M. Gong, X. Tian, T. Liu, and D. Tao Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §6. Lin et al. (2024) M. Lin, N. Weng, K. Mikolaj, Z. Bashir, M. B. Svendsen, M. G. Tolsgaard, A. N. Christensen, and A. Feragen Shortcut learning in medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 623–633. Cited by: §1. Lohaus et al. (2022) M. Lohaus, M. Kleindessner, K. Kenthapadi, F. Locatello, and C. Russell Are two heads the same as one? identifying disparate treatment in fair neural networks. Advances in Neural Information Processing Systems 35, p. 16548–16562. Cited by: Figure 2, §6. Long et al. (2018) M. Long, Z. Cao, J. Wang, and M. Jordan Conditional adversarial domain adaptation. Advances in neural information processing systems 31. Cited by: Appendix C. Lotter (2024) W. Lotter Acquisition parameters influence ai recognition of race in chest x-rays and mitigating these factors reduces underdiagnosis bias. Nature Communications 15 (1), p. 7465. Cited by: §1. Louizos et al. (2016) C. Louizos, K. Swersky, Y. Li, M. Welling, and R. S. Zemel The variational fair autoencoder. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Y. Bengio and Y. LeCun (Eds.), External Links: Link Cited by: §1. Madras et al. (2018) D. Madras, E. Creager, T. Pitassi, and R. Zemel Learning adversarially fair and transferable representations. External Links: 1802.06309, Link Cited by: §3.1, §3.2, §6. Mahajan et al. (2021) D. Mahajan, S. Tople, and A. Sharma Domain generalization using causal matching. In International conference on machine learning, p. 7313–7324. Cited by: §6. Makar et al. (2022) M. Makar, B. Packer, D. Moldovan, D. Blalock, Y. Halpern, and A. D’Amour Causally motivated shortcut removal using auxiliary labels. In International Conference on Artificial Intelligence and Statistics, p. 739–766. Cited by: §6. McNamara et al. (2019) D. McNamara, C. S. Ong, and R. C. Williamson Costs and benefits of fair representation learning. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, p. 263–270. Cited by: §A.1, §3.1. Mittelstadt et al. (2023) B. Mittelstadt, S. Wachter, and C. Russell The unfairness of fair machine learning: leveling down and strict egalitarianism by default. Mich. Tech. L. Rev. 30, p. 1. Cited by: §1, §6. Moyer et al. (2020) D. Moyer, G. Ver Steeg, C. M. Tax, and P. M. Thompson Scanner invariant representations for diffusion mri harmonization. Magnetic resonance in medicine 84 (4), p. 2174–2189. Cited by: §6. Oakden-Rayner et al. (2020) L. Oakden-Rayner, J. Dunnmon, G. Carneiro, and C. Ré Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, p. 151–159. Cited by: §6. Parikh et al. (2026) A. Parikh, S. Frank, S. Das, and A. Feragen Towards fairness under label bias in image segmentation: impact, measurement and mitigation. arXiv preprint arXiv:2605.06891. Cited by: §1. Pérez-Suay et al. (2017) A. Pérez-Suay, V. Laparra, G. Mateo-García, J. Muñoz-Marí, L. Gómez-Chova, and G. Camps-Valls Fair kernel learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, p. 339–355. Cited by: Appendix C. Petersen et al. (2023a) E. Petersen, E. Ferrante, M. Ganz, and A. Feragen Are demographically invariant models and representations in medical imaging fair?. arXiv preprint arXiv:2305.01397. Cited by: footnote 1. Petersen et al. (2023b) E. Petersen, M. Ganz, S. H. Holm, and A. Feragen On (assessing) the fairness of risk score models. External Links: 2302.08851, Link Cited by: §4, §6. Puli et al. (2022) A. M. Puli, L. H. Zhang, E. K. Oermann, and R. Ranganath Out-of-distribution generalization in the presence of nuisance-induced spurious correlations. In International Conference on Learning Representations, Cited by: §6. Quarta et al. (2026) A. Quarta, F. Santos, A. Marzullo, J. M. Sousa, S. M. Vieira, L. A. Celi, L. Seyyed-Kalantari, and F. Calimeri Hidden in plain sight: vector embeddings give away demographic information. IEEE Journal of Biomedical and Health Informatics. Cited by: §1. Rahimi and Recht (2007) A. Rahimi and B. Recht Random features for large-scale kernel machines. Advances in neural information processing systems 20. Cited by: Appendix C. Rashid et al. (2026) D. Rashid, R. Imam, D. Mahapatra, and B. Lall Stride-net: fairness-aware disentangled representation learning for chest x-ray diagnosis. arXiv preprint arXiv:2602.10875. Cited by: §1. Ravfogel et al. (2022) S. Ravfogel, F. Vargas, Y. Goldberg, and R. Cotterell Adversarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 6034–6055. Cited by: Appendix C. Redmond and Baveja (2002) M. Redmond and A. Baveja A data-driven software tool for enabling cooperative information sharing among police departments. European Journal of Operational Research 141 (3), p. 660–678. Cited by: §4. Sadri et al. (2025) A. R. Sadri, T. DeSilvio, and S. E. Viswanath Mutual information regularization for fairness-aware deep imaging representations. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 399–409. Cited by: §1. Sarridis et al. (2024) I. Sarridis, C. Koutlis, S. Papadopoulos, and C. Diou Flac: fairness-aware representation learning by suppressing attribute-class associations. IEEE Transactions on Pattern Analysis and Machine Intelligence 47 (2), p. 1148–1160. Cited by: §1. Schölkopf et al. (2021) B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio Toward causal representation learning. Proceedings of the IEEE 109 (5), p. 612–634. Cited by: §6. Shakibania et al. (2026) H. Shakibania, J. H. Grebe, T. Braun, E. Aktemur, S. Aslani, M. G. Yiğit, and M. Rohrbach Obliviate: erasing concepts from autoregressive image generation models. In European Conference on Computer Vision, p. 665–683. Cited by: Appendix C. Stanley et al. (2025a) E. A. Stanley, R. Mehta, and M. Roschewitz Exploring the interplay of label bias with subgroup size and separability: a case study in mammographic density. In Fairness of AI in Medical Imaging: Third International Workshop, FAIMI 2025, Held in Conjunction with MICCAI 2025, Daejeon, South Korea, September 23, 2025, Proceedings, p. 74. Cited by: §1. Stanley et al. (2025b) E. A. Stanley, R. Souza, M. Wilms, and N. D. Forkert Where, why, and how is bias learned in medical image analysis models? a study of bias encoding within convolutional networks using synthetic data. EBioMedicine 111. Cited by: §1. Subbaswamy and Saria (2020) A. Subbaswamy and S. Saria From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics 21 (2), p. 345–352. Cited by: §6. Tong et al. (2022) S. Tong, T. Garipov, Y. Zhang, S. Chang, and T. S. Jaakkola Adversarial support alignment. In International Conference on Learning Representations, External Links: Link Cited by: §6. Veitch et al. (2021) V. Veitch, A. D’Amour, S. Yadlowsky, and J. Eisenstein Counterfactual invariance to spurious correlations: why and how to pass stress tests. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21. Cited by: §6. Wang et al. (2024) Z. Wang, Z. Chu, R. Blanco, Z. Chen, S. Chen, and W. Zhang Advancing graph counterfactual fairness through fair representation learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, p. 40–58. Cited by: §1. Wu et al. (2022) Y. Wu, D. Zeng, X. Xu, Y. Shi, and J. Hu Fairprune: achieving fairness through pruning for dermatological disease diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 743–753. Cited by: §6. Yang et al. (2024) Y. Yang, H. Zhang, J. W. Gichoya, D. Katabi, and M. Ghassemi The limits of fair medical imaging AI in real-world generalization. Nature medicine 30 (10), p. 2838–2848. Cited by: §1. Yeh and Lien (2009) I. Yeh and C. Lien The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications 36 (2), p. 2473–2480. Cited by: §4. Yi et al. (2021) P. H. Yi, J. Wei, T. K. Kim, J. Shin, H. I. Sair, F. K. Hui, G. D. Hager, and C. T. Lin Radiology “forensics”: determination of age and sex from chest radiographs using deep learning. Emergency Radiology 28 (5), p. 949–954. Cited by: §1. Zemel et al. (2013) R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork Learning fair representations. In International conference on machine learning, p. 325–333. Cited by: Appendix C, §1, §3.1. Zhang et al. (2018) B. H. Zhang, B. Lemoine, and M. Mitchell Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, p. 335–340. Cited by: §1. Zhao et al. (2022) H. Zhao, C. Dan, B. Aragam, T. S. Jaakkola, G. J. Gordon, and P. Ravikumar Fundamental limits and tradeoffs in invariant representation learning. Journal of machine learning research 23 (340), p. 1–49. Cited by: §1, §3.1, §3.1, §6. Zhao and Gordon (2022) H. Zhao and G. J. Gordon Inherent tradeoffs in learning fair representations. Journal of Machine Learning Research 23 (57), p. 1–26. Cited by: §1, §3.4, §6. Zietlow et al. (2022) D. Zietlow, M. Lohaus, G. Balakrishnan, M. Kleindessner, F. Locatello, B. Schölkopf, and C. Russell Leveling down in computer vision: pareto inefficiencies in fair deep classifiers. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10400–10411. Cited by: §6. Appendix A Appendix 1: Tabular Datasets A.1 Additional Details: Fairness under increasing base-rate gap and prevalence shift conditions Refer to Fig. 6, Fig. 5 and Table 2 for results of all evaluation metrics against the training base-rate gap for the four prevalence conditions. Class-conditional alignment is similar to ERMERM’s AUROCAUROC at every gap, while the marginal flow manages to reduce decodability, and its summed error e0+e1e_0+e_1 rises with the gap. DAD_A for ERMERM follows the gap where A becomes decodable from Y. ΔDP _DP for ERMERM and class-conditional alignment follows the default base-rate; ΔEO _EO for the marginal method increases with the gap; ΔPPV _PPV and Δbias _bias grow for every method, as anticipated from our propositions. Fig. 7 plots the summed error of all 2,0402,040 evaluations against the bound (McNamara et al., 2019); none falls below it. Figure 5: Fairness gaps increase with the base-rate gap. Top: ΔDP _DP; Middle: ΔEO _EO; Bottom: Δbias _bias against the training base-rate gap |π1−π0|| _1- _0| over the four prevalence conditions, per dataset and method. Marginal representation invariance harms separation for parity at a rate set by the gap (P1P_1); class-conditional representation invariance holds ΔEO _EO (P3P_3) but not Δbias _bias (P3P_3). Mean ± std over three seeds reported. Figure 6: Every metric against the training base-rate gap |π1−π0|| _1- _0| (equalized, natural, shift-50, shift-75; matched test). Dotted line in the ΔDP _DP row: ΔDP=|π1−π0| _DP=| _1- _0|. Mean ± sd over seeds. Figure 7: Summed group error e0+e1e_0+e_1 against the bound max⁡0,|π0−π1|−ΔDP \0,| _0- _1|- _DP\ (mentioned in P2P_2) for every model×prevalence condition. Table 2: Evaluation metrics by training condition across datasets (reported as mean ± std over all seeds). Δπ=|π1−π0| π=| _1- _0| is the base-rate gap. Conditions: Eq (equalized), Nat (natural), S50 (shift-50), S75 (shift-75). Methods: ERM, M (marginal alignment, Z⟂AZ A), and C (class-conditional alignment, Z⟂A|YZ A Y). Datasets: Adult, Dutch census (Dutch), Communities & Crime (Crime), Taiwan credit (Credit), and COMPAS. Dataset Cond. Method π AUROC D_A | D_A Y _DP _EO _PPV _bias + e_0+e_1 Adult Eq ERM 0.00 0.87 ± 0.00 0.92 ± 0.00 0.98 ± 0.00 0.13 ± 0.01 0.14 ± 0.01 0.14 ± 0.01 0.01 ± 0.00 0.42 ± 0.02 Eq M 0.00 0.72 ± 0.01 0.56 ± 0.00 0.88 ± 0.02 0.03 ± 0.00 0.22 ± 0.05 0.06 ± 0.03 0.01 ± 0.00 0.57 ± 0.06 Eq C 0.00 0.84 ± 0.01 0.55 ± 0.00 0.61 ± 0.02 0.05 ± 0.01 0.08 ± 0.00 0.01 ± 0.00 0.01 ± 0.00 0.54 ± 0.05 Nat ERM 0.19 0.88 ± 0.00 0.93 ± 0.00 0.99 ± 0.00 0.32 ± 0.01 0.22 ± 0.01 0.06 ± 0.01 0.00 ± 0.00 0.38 ± 0.01 Nat M 0.19 0.77 ± 0.00 0.57 ± 0.01 0.85 ± 0.01 0.04 ± 0.00 0.31 ± 0.02 0.42 ± 0.03 0.17 ± 0.00 0.65 ± 0.07 Nat C 0.19 0.85 ± 0.00 0.60 ± 0.00 0.59 ± 0.01 0.15 ± 0.01 0.09 ± 0.02 0.28 ± 0.00 0.10 ± 0.00 0.49 ± 0.04 S50 ERM 0.25 0.89 ± 0.00 0.93 ± 0.00 0.99 ± 0.00 0.40 ± 0.03 0.28 ± 0.02 0.18 ± 0.01 0.02 ± 0.01 0.37 ± 0.03 S50 M 0.25 0.79 ± 0.00 0.58 ± 0.01 0.85 ± 0.02 0.04 ± 0.00 0.40 ± 0.03 0.49 ± 0.02 0.23 ± 0.00 0.68 ± 0.04 S50 C 0.25 0.86 ± 0.00 0.62 ± 0.01 0.55 ± 0.00 0.17 ± 0.01 0.08 ± 0.04 0.40 ± 0.01 0.15 ± 0.00 0.51 ± 0.05 S75 ERM 0.27 0.89 ± 0.00 0.93 ± 0.00 0.98 ± 0.00 0.46 ± 0.02 0.37 ± 0.06 0.31 ± 0.01 0.03 ± 0.01 0.38 ± 0.04 S75 M 0.27 0.81 ± 0.00 0.58 ± 0.01 0.83 ± 0.01 0.05 ± 0.01 0.46 ± 0.04 0.55 ± 0.01 0.25 ± 0.00 0.63 ± 0.02 S75 C 0.27 0.86 ± 0.00 0.63 ± 0.01 0.55 ± 0.00 0.18 ± 0.01 0.20 ± 0.00 0.49 ± 0.01 0.17 ± 0.01 0.50 ± 0.02 Dutch Eq ERM 0.00 0.90 ± 0.00 0.74 ± 0.00 0.76 ± 0.00 0.03 ± 0.00 0.10 ± 0.01 0.03 ± 0.00 0.03 ± 0.00 0.30 ± 0.00 Eq M 0.00 0.71 ± 0.01 0.53 ± 0.00 0.81 ± 0.01 0.02 ± 0.01 0.15 ± 0.02 0.13 ± 0.01 0.01 ± 0.00 0.64 ± 0.03 Eq C 0.00 0.90 ± 0.00 0.55 ± 0.01 0.58 ± 0.01 0.01 ± 0.01 0.05 ± 0.01 0.04 ± 0.01 0.01 ± 0.01 0.36 ± 0.01 Nat ERM 0.31 0.89 ± 0.00 0.75 ± 0.00 0.77 ± 0.00 0.20 ± 0.01 0.05 ± 0.01 0.13 ± 0.00 0.16 ± 0.00 0.36 ± 0.00 Nat M 0.31 0.71 ± 0.01 0.52 ± 0.00 0.76 ± 0.01 0.01 ± 0.00 0.27 ± 0.02 0.46 ± 0.01 0.31 ± 0.00 0.70 ± 0.01 Nat C 0.31 0.89 ± 0.00 0.63 ± 0.00 0.57 ± 0.01 0.19 ± 0.01 0.03 ± 0.00 0.18 ± 0.03 0.17 ± 0.01 0.39 ± 0.02 S50 ERM 0.44 0.90 ± 0.00 0.77 ± 0.00 0.77 ± 0.00 0.30 ± 0.00 0.05 ± 0.00 0.27 ± 0.01 0.22 ± 0.00 0.35 ± 0.01 S50 M 0.44 0.71 ± 0.00 0.52 ± 0.00 0.79 ± 0.01 0.01 ± 0.01 0.31 ± 0.03 0.63 ± 0.00 0.44 ± 0.00 0.73 ± 0.01 S50 C 0.44 0.89 ± 0.00 0.68 ± 0.01 0.56 ± 0.01 0.27 ± 0.01 0.03 ± 0.01 0.34 ± 0.01 0.23 ± 0.01 0.38 ± 0.01 S75 ERM 0.53 0.90 ± 0.00 0.79 ± 0.00 0.76 ± 0.00 0.36 ± 0.01 0.06 ± 0.01 0.44 ± 0.02 0.25 ± 0.00 0.34 ± 0.01 S75 M 0.53 0.71 ± 0.00 0.52 ± 0.00 0.80 ± 0.01 0.02 ± 0.01 0.34 ± 0.02 0.75 ± 0.01 0.53 ± 0.00 0.75 ± 0.01 S75 C 0.53 0.89 ± 0.00 0.71 ± 0.00 0.56 ± 0.00 0.33 ± 0.01 0.03 ± 0.01 0.50 ± 0.02 0.28 ± 0.00 0.36 ± 0.02 Crime Eq ERM 0.00 0.78 ± 0.00 0.65 ± 0.00 0.71 ± 0.00 0.18 ± 0.01 0.22 ± 0.01 0.12 ± 0.00 0.08 ± 0.00 0.65 ± 0.01 Eq M 0.00 0.61 ± 0.03 0.60 ± 0.00 0.77 ± 0.03 0.04 ± 0.05 0.13 ± 0.05 0.06 ± 0.03 0.02 ± 0.02 0.94 ± 0.18 Eq C 0.00 0.81 ± 0.01 0.60 ± 0.02 0.62 ± 0.03 0.06 ± 0.03 0.12 ± 0.04 0.10 ± 0.04 0.03 ± 0.01 0.67 ± 0.08 Nat ERM 0.45 0.85 ± 0.00 0.77 ± 0.00 0.74 ± 0.00 0.35 ± 0.01 0.20 ± 0.02 0.35 ± 0.01 0.16 ± 0.00 0.47 ± 0.03 Nat M 0.45 0.65 ± 0.01 0.55 ± 0.02 0.77 ± 0.01 0.02 ± 0.03 0.33 ± 0.05 0.59 ± 0.03 0.45 ± 0.01 0.82 ± 0.01 Nat C 0.45 0.85 ± 0.02 0.70 ± 0.03 0.62 ± 0.01 0.31 ± 0.07 0.13 ± 0.10 0.28 ± 0.05 0.22 ± 0.03 0.47 ± 0.06 S50 ERM 0.57 0.85 ± 0.00 0.79 ± 0.00 0.66 ± 0.00 0.43 ± 0.02 0.20 ± 0.02 0.50 ± 0.01 0.23 ± 0.00 0.46 ± 0.01 S50 M 0.57 0.67 ± 0.02 0.55 ± 0.02 0.73 ± 0.03 0.05 ± 0.01 0.28 ± 0.06 0.71 ± 0.04 0.55 ± 0.01 0.80 ± 0.05 S50 C 0.57 0.85 ± 0.01 0.74 ± 0.02 0.57 ± 0.05 0.40 ± 0.07 0.28 ± 0.16 0.53 ± 0.09 0.29 ± 0.03 0.49 ± 0.06 S75 ERM 0.64 0.86 ± 0.00 0.80 ± 0.00 0.67 ± 0.02 0.43 ± 0.04 0.19 ± 0.03 0.64 ± 0.03 0.29 ± 0.00 0.45 ± 0.00 S75 M 0.64 0.63 ± 0.02 0.55 ± 0.04 0.77 ± 0.04 0.02 ± 0.01 0.36 ± 0.06 0.81 ± 0.01 0.64 ± 0.01 0.86 ± 0.03 S75 C 0.64 0.84 ± 0.02 0.77 ± 0.02 0.61 ± 0.01 0.41 ± 0.02 0.15 ± 0.02 0.60 ± 0.04 0.34 ± 0.02 0.45 ± 0.03 Credit Eq ERM 0.00 0.74 ± 0.00 0.62 ± 0.00 0.62 ± 0.00 0.02 ± 0.01 0.05 ± 0.01 0.01 ± 0.00 0.01 ± 0.00 0.57 ± 0.01 Eq M 0.00 0.67 ± 0.00 0.53 ± 0.00 0.68 ± 0.01 0.00 ± 0.00 0.10 ± 0.03 0.07 ± 0.04 0.00 ± 0.00 0.62 ± 0.07 Eq C 0.00 0.75 ± 0.01 0.52 ± 0.00 0.52 ± 0.01 0.01 ± 0.01 0.03 ± 0.02 0.01 ± 0.01 0.00 ± 0.00 0.52 ± 0.05 Nat ERM 0.02 0.75 ± 0.00 0.62 ± 0.00 0.62 ± 0.00 0.03 ± 0.00 0.06 ± 0.01 0.04 ± 0.00 0.00 ± 0.00 0.53 ± 0.02 Nat M 0.02 0.68 ± 0.00 0.53 ± 0.00 0.68 ± 0.01 0.01 ± 0.00 0.08 ± 0.00 0.10 ± 0.00 0.02 ± 0.00 0.61 ± 0.03 Nat C 0.02 0.76 ± 0.00 0.52 ± 0.00 0.52 ± 0.01 0.01 ± 0.01 0.02 ± 0.01 0.01 ± 0.01 0.01 ± 0.00 0.49 ± 0.03 S50 ERM 0.11 0.75 ± 0.00 0.63 ± 0.00 0.62 ± 0.00 0.07 ± 0.00 0.08 ± 0.00 0.19 ± 0.01 0.08 ± 0.00 0.49 ± 0.03 S50 M 0.11 0.68 ± 0.01 0.52 ± 0.01 0.69 ± 0.02 0.01 ± 0.00 0.14 ± 0.03 0.28 ± 0.04 0.11 ± 0.00 0.55 ± 0.06 S50 C 0.11 0.75 ± 0.00 0.54 ± 0.01 0.53 ± 0.00 0.04 ± 0.01 0.01 ± 0.01 0.18 ± 0.01 0.09 ± 0.00 0.54 ± 0.09 S75 ERM 0.17 0.76 ± 0.00 0.64 ± 0.00 0.62 ± 0.00 0.10 ± 0.01 0.06 ± 0.02 0.28 ± 0.01 0.13 ± 0.00 0.50 ± 0.09 S75 M 0.17 0.69 ± 0.00 0.53 ± 0.01 0.70 ± 0.02 0.01 ± 0.01 0.17 ± 0.01 0.38 ± 0.02 0.17 ± 0.00 0.53 ± 0.05 S75 C 0.17 0.75 ± 0.00 0.56 ± 0.00 0.53 ± 0.01 0.07 ± 0.00 0.04 ± 0.04 0.33 ± 0.05 0.14 ± 0.00 0.41 ± 0.10 COMPAS Eq ERM 0.00 0.68 ± 0.00 0.64 ± 0.00 0.65 ± 0.01 0.14 ± 0.01 0.14 ± 0.01 0.04 ± 0.01 0.06 ± 0.00 0.74 ± 0.04 Eq M 0.00 0.63 ± 0.02 0.56 ± 0.01 0.66 ± 0.00 0.00 ± 0.00 0.04 ± 0.02 0.03 ± 0.01 0.01 ± 0.01 0.79 ± 0.05 Eq C 0.00 0.71 ± 0.01 0.53 ± 0.01 0.56 ± 0.02 0.01 ± 0.01 0.03 ± 0.02 0.02 ± 0.02 0.00 ± 0.00 0.66 ± 0.02 Nat ERM 0.11 0.70 ± 0.00 0.66 ± 0.00 0.65 ± 0.00 0.16 ± 0.01 0.15 ± 0.02 0.06 ± 0.01 0.02 ± 0.00 0.68 ± 0.00 Nat M 0.11 0.63 ± 0.00 0.56 ± 0.01 0.65 ± 0.02 0.02 ± 0.01 0.09 ± 0.03 0.17 ± 0.03 0.10 ± 0.01 0.84 ± 0.03 Nat C 0.11 0.71 ± 0.01 0.53 ± 0.01 0.54 ± 0.01 0.05 ± 0.03 0.06 ± 0.03 0.06 ± 0.04 0.09 ± 0.01 0.66 ± 0.01 S50 ERM 0.25 0.71 ± 0.00 0.68 ± 0.01 0.65 ± 0.00 0.21 ± 0.01 0.15 ± 0.01 0.23 ± 0.01 0.13 ± 0.00 0.70 ± 0.03 S50 M 0.25 0.64 ± 0.00 0.56 ± 0.01 0.63 ± 0.02 0.02 ± 0.01 0.11 ± 0.02 0.33 ± 0.06 0.24 ± 0.00 0.78 ± 0.04 S50 C 0.25 0.69 ± 0.00 0.56 ± 0.01 0.54 ± 0.00 0.10 ± 0.02 0.04 ± 0.02 0.27 ± 0.01 0.20 ± 0.01 0.68 ± 0.03 S75 ERM 0.35 0.73 ± 0.00 0.70 ± 0.00 0.66 ± 0.00 0.25 ± 0.01 0.16 ± 0.01 0.35 ± 0.01 0.21 ± 0.00 0.67 ± 0.08 S75 M 0.35 0.64 ± 0.01 0.58 ± 0.02 0.65 ± 0.02 0.04 ± 0.00 0.20 ± 0.06 0.50 ± 0.01 0.33 ± 0.01 0.83 ± 0.03 S75 C 0.35 0.70 ± 0.01 0.58 ± 0.01 0.54 ± 0.00 0.11 ± 0.03 0.09 ± 0.04 0.41 ± 0.03 0.29 ± 0.01 0.68 ± 0.10 A.2 Ablations: Probe Ablation Refer to Fig. 8 and Table 3 for details on probe evaluation. For the marginal alignment methods, the empirical probe AUROCAUROC lies below the bound obtained from the exact per-group MADE densities, while the within-class probe stays high or rises above ERMERM. For the class-conditional alignment methods, the within-class probe stays within the class bound, and the marginal probe rises to the base-rate floor of P2P_2. Figure 8: Probe AUROCAUROC to decode for A from Z under six probe families (shift-50, reported mean ± std over all seeds). Dotted: (1+TV)/2(1+TV)/2 for the marginal flow. Table 3: Probe ablation at shift 50. Methods: ERM, Marginal (M: Z⟂AZ A), and Conditional (C: Z⟂A|YZ A Y). Columns 3–8 report AUC of probe families predicting A from Z: LR (logistic regression), GB (gradient-boosted trees), MLP-k (k hidden layers), and LR−A-A (logistic control baseline excluding A). Right columns report the original FNF adversary balanced accuracy (Adv BA), the certified balanced accuracy bound (1+TV)/2(1+TV)/2 (Bound), and the maximum within-class probe AUC (Max W). Reported as mean ± sd over three seeds. Probe AUC (A from Z) Dataset Method LR GB MLP-0 MLP-1 MLP-2 LR−A-A Adv BA Bound Max W Adult ERM 0.93 ± 0.00 0.93 ± 0.00 0.93 ± 0.00 0.93 ± 0.00 0.93 ± 0.00 0.51 ± 0.02 0.86 ± 0.00 – 0.99 ± 0.00 M 0.56 ± 0.00 0.57 ± 0.01 0.57 ± 0.01 0.57 ± 0.01 0.58 ± 0.01 0.51 ± 0.01 0.55 ± 0.01 0.55 ± 0.00 0.85 ± 0.02 C 0.61 ± 0.01 0.62 ± 0.00 0.62 ± 0.01 0.62 ± 0.00 0.62 ± 0.00 0.51 ± 0.01 0.58 ± 0.01 0.53 ± 0.00 0.56 ± 0.00 Dutch census ERM 0.75 ± 0.00 0.77 ± 0.00 0.77 ± 0.00 0.77 ± 0.00 0.77 ± 0.00 0.51 ± 0.04 0.68 ± 0.00 – 0.77 ± 0.00 M 0.51 ± 0.00 0.51 ± 0.00 0.51 ± 0.01 0.51 ± 0.00 0.51 ± 0.00 0.49 ± 0.00 0.50 ± 0.00 0.51 ± 0.00 0.79 ± 0.01 C 0.67 ± 0.01 0.67 ± 0.01 0.67 ± 0.01 0.68 ± 0.01 0.68 ± 0.01 0.51 ± 0.01 0.60 ± 0.00 0.56 ± 0.00 0.56 ± 0.01 Comm. & Crime ERM 0.79 ± 0.00 0.76 ± 0.00 0.78 ± 0.00 0.77 ± 0.00 0.78 ± 0.00 0.58 ± 0.08 0.69 ± 0.01 – 0.66 ± 0.00 M 0.54 ± 0.03 0.51 ± 0.02 0.51 ± 0.02 0.54 ± 0.02 0.52 ± 0.03 0.52 ± 0.07 0.51 ± 0.00 0.54 ± 0.01 0.73 ± 0.03 C 0.73 ± 0.03 0.72 ± 0.03 0.74 ± 0.02 0.73 ± 0.02 0.73 ± 0.02 0.54 ± 0.10 0.64 ± 0.01 0.55 ± 0.01 0.58 ± 0.04 Taiwan credit ERM 0.61 ± 0.00 0.63 ± 0.00 0.63 ± 0.00 0.63 ± 0.00 0.63 ± 0.00 0.48 ± 0.05 0.58 ± 0.00 – 0.62 ± 0.00 M 0.51 ± 0.01 0.52 ± 0.01 0.51 ± 0.00 0.51 ± 0.01 0.51 ± 0.01 0.49 ± 0.01 0.50 ± 0.00 0.52 ± 0.00 0.69 ± 0.02 C 0.53 ± 0.00 0.54 ± 0.01 0.54 ± 0.01 0.54 ± 0.00 0.54 ± 0.01 0.49 ± 0.02 0.50 ± 0.00 0.52 ± 0.00 0.53 ± 0.00 COMPAS ERM 0.68 ± 0.00 0.67 ± 0.00 0.68 ± 0.00 0.68 ± 0.00 0.68 ± 0.01 0.53 ± 0.06 0.60 ± 0.00 – 0.65 ± 0.00 M 0.52 ± 0.01 0.56 ± 0.01 0.55 ± 0.03 0.56 ± 0.01 0.56 ± 0.01 0.50 ± 0.01 0.52 ± 0.01 0.56 ± 0.00 0.63 ± 0.02 C 0.55 ± 0.01 0.56 ± 0.01 0.55 ± 0.01 0.55 ± 0.02 0.55 ± 0.02 0.52 ± 0.01 0.52 ± 0.01 0.60 ± 0.01 0.54 ± 0.01 A.3 Ablation: Strength of representation invariance, hyperparameters and thresholds Refer to Fig. 9 for sweeps γ for the marginal alignment method. DAD_A, ΔDP _DP and AUROC both decrease with γ; ΔEO _EO do not. On Dutch it’s par with ERMERM level for γ≤0.1γ≤ 0.1 and rises to 0.310.31 at γ=1γ=1 as DA|YD_A Y increases from 0.610.61 to 0.790.79; on Adult it is above ERMERM for all γ; on COMPAS it stays within seed std of ERMERM. Refer to Table 4, which demonstrates that changing the feature set does not change the conclusion: ΔDP≤0.10 _DP≤ 0.10 in every config while ΔEO _EO stays at par or above ERMERM. Table 5 compares the Youden threshold with a default 0.50.5: at default, the marginal alignment method predicts few positives under low prevalence on Adult, Credit and COMPAS, so both gaps shrink counterintuitively; on Dutch and Crime it holds for both thresholds. Figure 9: Marginal alignment method as a function of the invariance strength γ (shift-50, selected feature set, MADE width 50). Reported mean ± std over seeds. Table 4: Marginal alignment method at γ=1γ=1 for every feature set and MADE width in the sweep (shift 50, mean ± std over all seeds). Dataset Feature set MADE n fits AUROC DAD_A DA|YD_A Y ΔDP _DP ΔEO _EO exact TV Adult paper6 50 3 0.79 ± 0.00 0.58 ± 0.01 0.89 ± 0.01 0.08 ± 0.00 0.38 ± 0.03 0.103 ± 0.003 Adult paper6 100 3 0.79 ± 0.00 0.59 ± 0.01 0.87 ± 0.02 0.09 ± 0.01 0.36 ± 0.04 0.102 ± 0.005 Adult paper6_age (selected) 50 3 0.79 ± 0.00 0.58 ± 0.01 0.85 ± 0.02 0.04 ± 0.00 0.40 ± 0.03 0.096 ± 0.007 Adult paper6_age 100 3 0.80 ± 0.00 0.58 ± 0.00 0.84 ± 0.02 0.05 ± 0.01 0.38 ± 0.02 0.093 ± 0.024 Dutch census core6 50 3 0.72 ± 0.00 0.52 ± 0.01 0.81 ± 0.01 0.01 ± 0.01 0.27 ± 0.02 0.022 ± 0.003 Dutch census core6 100 3 0.72 ± 0.01 0.52 ± 0.00 0.81 ± 0.00 0.01 ± 0.01 0.30 ± 0.01 0.022 ± 0.003 Dutch census core6_h (selected) 50 3 0.71 ± 0.00 0.52 ± 0.00 0.79 ± 0.01 0.01 ± 0.01 0.31 ± 0.03 0.023 ± 0.002 Dutch census core6_h 100 3 0.71 ± 0.00 0.51 ± 0.01 0.78 ± 0.00 0.01 ± 0.01 0.32 ± 0.02 0.026 ± 0.004 Comm. & Crime socio4 (selected) 50 3 0.67 ± 0.02 0.55 ± 0.02 0.73 ± 0.03 0.05 ± 0.01 0.28 ± 0.06 0.070 ± 0.020 Comm. & Crime socio4 100 3 0.64 ± 0.03 0.56 ± 0.03 0.76 ± 0.02 0.04 ± 0.03 0.28 ± 0.05 0.069 ± 0.015 Comm. & Crime socio6 50 3 0.65 ± 0.02 0.56 ± 0.03 0.74 ± 0.01 0.09 ± 0.03 0.30 ± 0.09 0.085 ± 0.013 Comm. & Crime socio6 100 3 0.68 ± 0.02 0.58 ± 0.02 0.73 ± 0.04 0.10 ± 0.07 0.22 ± 0.02 0.063 ± 0.011 Taiwan credit core5 (selected) 50 3 0.68 ± 0.01 0.52 ± 0.01 0.69 ± 0.02 0.01 ± 0.00 0.14 ± 0.03 0.038 ± 0.005 Taiwan credit core5 100 3 0.69 ± 0.00 0.53 ± 0.01 0.69 ± 0.01 0.01 ± 0.01 0.12 ± 0.03 0.039 ± 0.005 Taiwan credit extended8 50 3 0.65 ± 0.01 0.53 ± 0.01 0.70 ± 0.01 0.01 ± 0.01 0.22 ± 0.04 0.022 ± 0.002 Taiwan credit extended8 100 3 0.65 ± 0.02 0.58 ± 0.01 0.71 ± 0.03 0.01 ± 0.01 0.25 ± 0.03 0.025 ± 0.005 COMPAS paper7 (selected) 50 3 0.64 ± 0.00 0.56 ± 0.01 0.63 ± 0.02 0.02 ± 0.01 0.11 ± 0.02 0.125 ± 0.007 COMPAS paper7 100 3 0.63 ± 0.02 0.57 ± 0.01 0.64 ± 0.04 0.02 ± 0.03 0.10 ± 0.03 0.116 ± 0.016 COMPAS paper7_nosex 50 3 0.63 ± 0.00 0.60 ± 0.01 0.65 ± 0.04 0.04 ± 0.02 0.10 ± 0.02 0.143 ± 0.018 COMPAS paper7_nosex 100 3 0.63 ± 0.02 0.60 ± 0.01 0.65 ± 0.01 0.04 ± 0.03 0.07 ± 0.02 0.131 ± 0.007 Table 5: Optimal Youden threshold τ (for validation) vs. a default 0.50.5 (shift 50, reported mean ± std over all seeds). Dataset Method Youden τ Acc (τ) ΔDP _DP (τ) ΔEO _EO (τ) Acc (0.5) ΔDP _DP (0.5) ΔEO _EO (0.5) Adult ERM 0.22 ± 0.02 0.78 ± 0.02 0.40 ± 0.03 0.28 ± 0.02 0.84 ± 0.00 0.24 ± 0.01 0.39 ± 0.03 Adult Z⟂AZ A 0.17 ± 0.02 0.69 ± 0.02 0.04 ± 0.00 0.40 ± 0.03 0.78 ± 0.00 0.01 ± 0.01 0.06 ± 0.03 Adult Z⟂A|YZ A Y 0.17 ± 0.00 0.75 ± 0.02 0.17 ± 0.01 0.08 ± 0.04 0.82 ± 0.00 0.11 ± 0.01 0.06 ± 0.01 Dutch census ERM 0.43 ± 0.02 0.82 ± 0.00 0.30 ± 0.00 0.05 ± 0.00 0.83 ± 0.00 0.29 ± 0.00 0.06 ± 0.00 Dutch census Z⟂AZ A 0.40 ± 0.04 0.64 ± 0.00 0.01 ± 0.01 0.31 ± 0.03 0.65 ± 0.01 0.01 ± 0.01 0.22 ± 0.03 Dutch census Z⟂A|YZ A Y 0.41 ± 0.04 0.81 ± 0.00 0.27 ± 0.01 0.03 ± 0.01 0.81 ± 0.00 0.28 ± 0.01 0.02 ± 0.01 Comm. & Crime ERM 0.45 ± 0.03 0.77 ± 0.01 0.43 ± 0.02 0.20 ± 0.02 0.75 ± 0.01 0.39 ± 0.01 0.27 ± 0.00 Comm. & Crime Z⟂AZ A 0.44 ± 0.03 0.62 ± 0.02 0.05 ± 0.01 0.28 ± 0.06 0.61 ± 0.02 0.04 ± 0.03 0.19 ± 0.04 Comm. & Crime Z⟂A|YZ A Y 0.50 ± 0.02 0.75 ± 0.03 0.40 ± 0.07 0.28 ± 0.16 0.76 ± 0.03 0.40 ± 0.07 0.25 ± 0.18 Taiwan credit ERM 0.18 ± 0.02 0.76 ± 0.01 0.07 ± 0.00 0.08 ± 0.00 0.85 ± 0.00 0.05 ± 0.00 0.03 ± 0.00 Taiwan credit Z⟂AZ A 0.20 ± 0.03 0.72 ± 0.03 0.01 ± 0.00 0.14 ± 0.03 0.82 ± 0.00 0.00 ± 0.00 0.02 ± 0.01 Taiwan credit Z⟂A|YZ A Y 0.17 ± 0.03 0.73 ± 0.05 0.04 ± 0.01 0.01 ± 0.01 0.85 ± 0.00 0.03 ± 0.01 0.02 ± 0.02 COMPAS ERM 0.39 ± 0.04 0.64 ± 0.01 0.21 ± 0.01 0.15 ± 0.01 0.68 ± 0.00 0.20 ± 0.01 0.14 ± 0.01 COMPAS Z⟂AZ A 0.41 ± 0.03 0.62 ± 0.01 0.02 ± 0.01 0.11 ± 0.02 0.62 ± 0.00 0.02 ± 0.02 0.05 ± 0.03 COMPAS Z⟂A|YZ A Y 0.39 ± 0.07 0.66 ± 0.01 0.10 ± 0.02 0.04 ± 0.02 0.65 ± 0.00 0.08 ± 0.00 0.05 ± 0.01 A.4 Additional Details: Datasets and per-seed results Dataset details, group definitions, and training-split prevalences per condition are reported in Table 6 for accountability. Table 6: Dataset statistics and prevalence shift configurations. Binary groups (A=0A=0 vs. A=1A=1) and prediction targets (Y=1Y=1) are: Adult: female vs. male, income >>50K; Dutch: female vs. male, high-level occupation; Crime: Black population share below vs. at or above median, violent crime rate above median; Taiwan credit: female vs. male, default next month; COMPAS: Caucasian vs. African-American, two-year recidivism. Rightmost columns report the training-split base rates (π0 _0 / π1 _1) across synthetic shift regimes. Selected feature sets: Adult paper6_age (workclass, education, marital status, occupation, relationship, race, binned age), Dutch core6_h (age, education, economic status, current activity, marital status, household position, household size), Crime socio4 (median income, poverty, unemployment, two-parent families; quantile-binned), Credit core5 (education, marriage, binned age, last repayment status, binned limit), COMPAS paper7 (sex, age category, charge degree, binned priors, three juvenile counts). Training Prevalences (π0 _0 / π1 _1) Dataset Samples Natural π0/π1 _0/ _1 Equalized Natural Shift-50 Shift-75 Adult income 48,842 0.11 / 0.30 0.11 / 0.11 0.11 / 0.31 0.06 / 0.31 0.03 / 0.31 (29,312 / 4,880 / 4,885 / 9,765) Dutch census 60,420 0.33 / 0.63 0.33 / 0.33 0.33 / 0.62 0.20 / 0.62 0.11 / 0.62 (36,252 / 6,042 / 6,042 / 12,084) Comm. & Crime 1,994 0.27 / 0.71 0.26 / 0.26 0.26 / 0.69 0.15 / 0.69 0.08 / 0.69 (1,196 / 199 / 199 / 400) Taiwan credit 30,000 0.21 / 0.24 0.21 / 0.21 0.21 / 0.25 0.12 / 0.25 0.06 / 0.25 (18,000 / 3,000 / 3,000 / 6,000) COMPAS 5,278 0.39 / 0.52 0.39 / 0.39 0.39 / 0.54 0.24 / 0.54 0.14 / 0.54 (3,166 / 528 / 528 / 1,056) Table 7: Detailed results for the shift-50 prevalence condition per selected seeds. Dataset Method Seed AUROC DAD_A DA|YD_A Y ΔDP _DP ΔEO _EO ΔPPV _PPV Δbias _bias Adult ERM 42 0.89 0.93 0.99 0.42 0.30 0.17 0.00 Adult ERM 123 0.89 0.93 0.99 0.36 0.27 0.19 0.02 Adult ERM 456 0.89 0.93 0.99 0.40 0.27 0.18 0.02 Adult M-FNF (Z⟂AZ A) 42 0.79 0.58 0.86 0.04 0.37 0.48 0.23 Adult M-FNF (Z⟂AZ A) 123 0.79 0.57 0.86 0.05 0.43 0.47 0.23 Adult M-FNF (Z⟂AZ A) 456 0.79 0.58 0.83 0.04 0.41 0.51 0.23 Adult C-FNF (Z⟂A|YZ A Y) 42 0.84 0.66 0.83 0.21 0.41 0.47 0.11 Adult C-FNF (Z⟂A|YZ A Y) 123 0.84 0.65 0.78 0.20 0.41 0.48 0.11 Adult C-FNF (Z⟂A|YZ A Y) 456 0.84 0.66 0.81 0.21 0.48 0.50 0.11 Dutch census ERM 42 0.90 0.77 0.76 0.30 0.05 0.26 0.22 Dutch census ERM 123 0.90 0.77 0.77 0.30 0.05 0.29 0.22 Dutch census ERM 456 0.90 0.77 0.77 0.30 0.05 0.27 0.21 Dutch census M-FNF (Z⟂AZ A) 42 0.71 0.51 0.79 0.01 0.28 0.63 0.44 Dutch census M-FNF (Z⟂AZ A) 123 0.70 0.51 0.80 0.00 0.32 0.63 0.44 Dutch census M-FNF (Z⟂AZ A) 456 0.71 0.52 0.79 0.01 0.33 0.63 0.45 Dutch census C-FNF (Z⟂A|YZ A Y) 42 0.87 0.64 0.61 0.26 0.07 0.29 0.27 Dutch census C-FNF (Z⟂A|YZ A Y) 123 0.86 0.64 0.61 0.26 0.06 0.29 0.27 Dutch census C-FNF (Z⟂A|YZ A Y) 456 0.86 0.64 0.62 0.26 0.06 0.29 0.27 Comm. & Crime ERM 42 0.85 0.79 0.66 0.43 0.18 0.50 0.23 Comm. & Crime ERM 123 0.85 0.79 0.66 0.41 0.23 0.52 0.23 Comm. & Crime ERM 456 0.85 0.79 0.66 0.44 0.19 0.50 0.23 Comm. & Crime M-FNF (Z⟂AZ A) 42 0.67 0.53 0.76 0.05 0.35 0.74 0.55 Comm. & Crime M-FNF (Z⟂AZ A) 123 0.65 0.57 0.72 0.06 0.27 0.73 0.56 Comm. & Crime M-FNF (Z⟂AZ A) 456 0.69 0.54 0.70 0.05 0.23 0.66 0.55 Comm. & Crime C-FNF (Z⟂A|YZ A Y) 42 0.81 0.71 0.74 0.36 0.18 0.54 0.32 Comm. & Crime C-FNF (Z⟂A|YZ A Y) 123 0.82 0.73 0.73 0.40 0.28 0.55 0.31 Comm. & Crime C-FNF (Z⟂A|YZ A Y) 456 0.82 0.74 0.56 0.39 0.31 0.56 0.30 Taiwan credit ERM 42 0.75 0.63 0.62 0.07 0.09 0.20 0.08 Taiwan credit ERM 123 0.75 0.63 0.62 0.08 0.08 0.19 0.08 Taiwan credit ERM 456 0.75 0.64 0.62 0.07 0.08 0.20 0.08 Taiwan credit M-FNF (Z⟂AZ A) 42 0.67 0.52 0.67 0.01 0.16 0.29 0.11 Taiwan credit M-FNF (Z⟂AZ A) 123 0.68 0.52 0.72 0.01 0.11 0.24 0.11 Taiwan credit M-FNF (Z⟂AZ A) 456 0.69 0.51 0.69 0.01 0.15 0.31 0.11 Taiwan credit C-FNF (Z⟂A|YZ A Y) 42 0.71 0.54 0.60 0.03 0.07 0.26 0.09 Taiwan credit C-FNF (Z⟂A|YZ A Y) 123 0.70 0.54 0.62 0.04 0.08 0.25 0.09 Taiwan credit C-FNF (Z⟂A|YZ A Y) 456 0.72 0.54 0.57 0.03 0.08 0.25 0.10 COMPAS ERM 42 0.71 0.68 0.65 0.20 0.13 0.23 0.13 COMPAS ERM 123 0.72 0.69 0.65 0.22 0.16 0.23 0.13 COMPAS ERM 456 0.71 0.68 0.65 0.22 0.15 0.24 0.14 COMPAS M-FNF (Z⟂AZ A) 42 0.63 0.56 0.65 0.02 0.13 0.38 0.24 COMPAS M-FNF (Z⟂AZ A) 123 0.64 0.58 0.63 0.01 0.10 0.26 0.25 COMPAS M-FNF (Z⟂AZ A) 456 0.64 0.56 0.61 0.02 0.09 0.35 0.24 COMPAS C-FNF (Z⟂A|YZ A Y) 42 0.66 0.61 0.62 0.17 0.12 0.20 0.18 COMPAS C-FNF (Z⟂A|YZ A Y) 123 0.69 0.63 0.62 0.18 0.12 0.21 0.17 COMPAS C-FNF (Z⟂A|YZ A Y) 456 0.68 0.63 0.62 0.18 0.12 0.21 0.17 Table 7 reports every seed for the four methods at the shift-50 prevalence condition in Table 1. The training seed variation is small relative to the between-method differences on all datasets except Communities & Crime (400 test rows, 38–151 positives in A=0A=0)/ It is the only dataset where the training seed variation of ΔEO _EO and ΔPPV _PPV is comparable to the method effect. A.5 Additional Details: Calibration by groups Table 8 reports the calibration by groups results for the selected models under shift-50 prevalence condition. Under marginal alignment: the low-prevalence group A=0A=0 becomes strongly ’over-confident’ (Adult b0b_0: 0.96→0.060.96→ 0.06; Dutch 0.98→0.510.98→ 0.51) and the high-prevalence group ’under-confident’ (b1=1.34b_1=1.34–1.841.84), the intercepts can be seen in opposite directions (a0<0<a1a_0<0<a_1), and Δcal _cal increases for all datasets, as anticipated by P2P_2. Similar dependency on training prevalence is also discussed by Kina and Petersen (2026) for calibration by groups: a model fitted to a P⁡(Y∣A)P(Y A) carries that prevalence, and the mismatch may appears as group-specific miscalibration under Q. . Class-conditional alignment brings back both slopes to what they were for ERMERM but leaves the intercept gap at par with or above ERMERM (P4P_4). Under prevalence shift at deployment (Fig. 4), Δbias _bias increases for every method, including at ERMERM, because a score calibrated to the training prevalence πa _a is off by logitπaQ−logitπalogit\,π^Q_a-logit\, _a in the large once the test prevalence is πaQπ^Q_a; when the shift is group-specific, this gap differs between groups. Per-run values for all 2,0402,040 evaluations are summarized in the table. Table 8: Calibration by groups at shift 50: 15-bin ECE, Cox recalibration slope (Cox, 1958) bgb_g and fixed-slope intercept aga_g (logits), signed mean bias y¯g−p¯g y_g- p_g, and the two between-group gaps Δbias _bias and Δcal _cal. Methods: ERM, M (marginal alignment, Z⟂AZ A), and C (class-conditional alignment, Z⟂A|YZ A Y). Datasets: Adult, Dutch census (Dutch), Communities & Crime (Crime), Taiwan credit (Credit), and COMPAS. Reported as mean ± std over all three seeds. Dataset Method 0ECE_0 1ECE_1 0 b_0 1 b_1 0 a_0 1 a_1 ¯−¯ y_0- p_0 ¯−¯ y_1- p_1 _bias _cal Adult ERM 0.02 ± 0.00 0.02 ± 0.00 0.96 ± 0.03 0.85 ± 0.01 −0.29-0.29 ± 0.08 0.03 ± 0.12 −0.013-0.013 ± 0.004 0.004 ± 0.016 0.02 ± 0.01 0.32 ± 0.06 M 0.13 ± 0.01 0.11 ± 0.01 0.06 ± 0.04 1.34 ± 0.05 −1.36-1.36 ± 0.06 0.75 ± 0.06 −0.116-0.116 ± 0.007 0.112 ± 0.009 0.23 ± 0.00 2.12 ± 0.01 C 0.08 ± 0.01 0.08 ± 0.01 0.79 ± 0.03 0.88 ± 0.02 −1.20-1.20 ± 0.14 0.60 ± 0.09 −0.077-0.077 ± 0.013 0.075 ± 0.010 0.15 ± 0.00 1.80 ± 0.07 Dutch ERM 0.12 ± 0.00 0.10 ± 0.00 1.01 ± 0.02 0.90 ± 0.01 −1.24-1.24 ± 0.05 0.75 ± 0.02 −0.120-0.120 ± 0.004 0.095 ± 0.003 0.22 ± 0.00 2.00 ± 0.03 M 0.23 ± 0.00 0.22 ± 0.01 0.51 ± 0.08 1.65 ± 0.06 −1.25-1.25 ± 0.01 1.07 ± 0.04 −0.224-0.224 ± 0.003 0.219 ± 0.007 0.44 ± 0.00 2.32 ± 0.03 C 0.12 ± 0.01 0.11 ± 0.00 0.98 ± 0.03 0.91 ± 0.02 −1.16-1.16 ± 0.05 0.89 ± 0.04 −0.121-0.121 ± 0.006 0.114 ± 0.005 0.23 ± 0.01 2.05 ± 0.07 Crime ERM 0.18 ± 0.02 0.12 ± 0.00 0.45 ± 0.01 0.74 ± 0.01 −1.28-1.28 ± 0.04 0.82 ± 0.01 −0.126-0.126 ± 0.006 0.104 ± 0.002 0.23 ± 0.00 2.10 ± 0.03 M 0.28 ± 0.02 0.29 ± 0.00 0.43 ± 0.53 1.48 ± 0.18 −1.46-1.46 ± 0.01 1.41 ± 0.02 −0.263-0.263 ± 0.003 0.290 ± 0.005 0.55 ± 0.01 2.87 ± 0.03 C 0.17 ± 0.04 0.16 ± 0.01 0.71 ± 0.26 0.74 ± 0.01 −1.21-1.21 ± 0.24 1.13 ± 0.01 −0.137-0.137 ± 0.026 0.153 ± 0.007 0.29 ± 0.03 2.34 ± 0.24 Credit ERM 0.03 ± 0.00 0.05 ± 0.00 0.96 ± 0.01 0.95 ± 0.00 −0.34-0.34 ± 0.02 0.35 ± 0.02 −0.034-0.034 ± 0.003 0.046 ± 0.002 0.08 ± 0.00 0.68 ± 0.01 M 0.06 ± 0.00 0.05 ± 0.00 0.50 ± 0.05 1.22 ± 0.01 −0.50-0.50 ± 0.02 0.36 ± 0.02 −0.058-0.058 ± 0.003 0.053 ± 0.003 0.11 ± 0.00 0.86 ± 0.02 C 0.05 ± 0.00 0.05 ± 0.00 0.94 ± 0.02 0.89 ± 0.01 −0.44-0.44 ± 0.02 0.35 ± 0.01 −0.045-0.045 ± 0.002 0.046 ± 0.002 0.09 ± 0.00 0.79 ± 0.02 COMPAS ERM 0.13 ± 0.00 0.06 ± 0.01 0.63 ± 0.01 0.75 ± 0.03 −0.69-0.69 ± 0.02 0.09 ± 0.02 −0.116-0.116 ± 0.003 0.018 ± 0.004 0.13 ± 0.00 0.78 ± 0.01 M 0.18 ± 0.01 0.10 ± 0.01 0.52 ± 0.14 0.86 ± 0.07 −0.81-0.81 ± 0.02 0.38 ± 0.01 −0.157-0.157 ± 0.004 0.086 ± 0.002 0.24 ± 0.00 1.19 ± 0.01 C 0.13 ± 0.01 0.11 ± 0.01 0.78 ± 0.01 0.62 ± 0.02 −0.72-0.72 ± 0.07 0.37 ± 0.04 −0.125-0.125 ± 0.013 0.073 ± 0.006 0.20 ± 0.01 1.09 ± 0.05 A.6 Additional Details: Score-level alignment We constrain the score distribution, whereas alignment methods constrain the representation. We look for (i) whether aligning the score R directly, rather than Z, changes the outcome, and (i) whether a weaker constraint that matches only the class-conditional mean of the score suffices for separation. We evaluate this on Dutch at shift-50 with the ERM model (Table 9). Score-level alignment. It reproduce the representation-level results: marginal alignment gives ΔDP=0.006 _DP=0.006 with ΔEO=0.26 _EO=0.26 and Δbias=0.44 _bias=0.44 (FNF: 0.0070.007, 0.310.31, 0.440.44), and within-class alignment gives ΔEO=0.024 _EO=0.024 (C-FNF: 0.0280.028) with ΔDP _DP and Δbias _bias at the ERM level. Aligning one scalar costs less utility than aligning Z (AUROC 0.810.81 versus 0.710.71 for the marginal target), but the fairness conclusions remain unchanged, as expected for statements about p⁡(r∣a)p(r a) and p⁡(r∣y,a)p(r y,a). Conditional mean matching. We retrain the ERM head with the penalty λ∑y([R∣Y=y,A=0]−[R∣Y=y,A=1])2λ _y (E[R Y=y,A=0]-E[R Y=y,A=1] )^2 estimated per mini-batch, λ∈1,10,100λ∈\1,10,100\. The class-conditional mean gaps fall from +0.041+0.041 (y=0y=0) and −0.028-0.028 (y=1y=1) under ERM to 0.0000.000 on train and test for λ≥100λ≥ 100 (also at λ=103,104λ=10^3,10^4). It does not deliver separation: ΔEO _EO stays at 0.060.06–0.080.08 (ERM 0.0550.055) and DA|YD_A Y does not fall, because the group-specific shape of p⁡(r∣y,a)p(r y,a) beyond its mean is untouched. It does move calibration: Δbias _bias rises to 0.410.41. Table 9: Score-level alignment and class-conditional mean matching on Dutch, shift-50, test split, mean over three seeds. Method AUROC D_A | D_A Y _DP _EO _PPV _NPV _bias _cal ERM 0.896 0.70 0.64 0.296 0.055 0.27 0.28 0.22 2.00 Score R⟂AR A 0.813 0.54 0.77 0.006 0.258 0.55 0.35 0.44 3.68 Score R⟂A|YR A Y (within-class map) 0.881 0.67 0.59 0.257 0.024 0.36 0.28 0.24 2.11 Mean matching, λ=1λ=1 0.896 0.70 0.63 0.291 0.051 0.29 0.28 0.22 1.95 Mean matching, λ=10λ=10 0.889 0.69 0.66 0.270 0.082 0.27 0.30 0.28 1.79 Mean matching, λ=100λ=100 0.883 0.69 0.66 0.266 0.081 0.30 0.30 0.41 1.87 FNF Z⟂AZ A (γ=1γ=1), repr. level 0.707 0.52∗ 0.79∗ 0.007 0.313 0.63 0.21 0.44 2.32 C-FNF Z⟂A|YZ A Y (γ=1γ=1), repr. level 0.888 0.68∗ 0.56∗ 0.275 0.028 0.34 0.27 0.23 2.05 Appendix B Appendix 2: CheXpert B.1 Additional Details: Dataset details and cohorts We perform a classification task on Edema and Cardiomegaly from frontal CheXpert radiographs using recorded sex (A=0A=0: female; A=1A=1: male). Images with unknown sex are excluded; Uncertain pathology labels are positive, and blank labels are negative. The shift50 intervention removes half of female-positive images separately within each partition. Both training and evaluation use the shifted distribution; this experiment does not test a new deployment shift. Table 10 reports the retained cohorts. Table 10: shift50 cohorts. Patient partitions are fixed across seeds; image-level deletion changes the retained patient counts. Task Partition Images Patients π0 _0 π1 _1 Edema Train 107,688 37,580 0.198 0.313 Checkpoint 9,050 3,134 0.195 0.325 Development 8,872 3,138 0.202 0.333 Probe val. 17,186 6,271 0.206 0.310 Test 35,078 12,551 0.205 0.309 Cardiomegaly Train 111,980 38,082 0.081 0.159 Checkpoint 9,363 3,175 0.093 0.161 Development 9,225 3,168 0.082 0.173 Probe val. 17,896 6,351 0.085 0.168 Test 36,525 12,693 0.085 0.160 Similar to the tabular setting, we use three training seeds (42, 123, 456). ImageNet-init DenseNet121 models with 256×256256× 256 RGB input and standard normalization. We use BCE, AdamW (lr 10−410^-4, decay of 10−510^-5), a batch size of 32, and at most 30 epochs with early stopping. It uses a patience of 7 epochs after the first 10 epochs and minimizes validation BCE. We adapt continuous Fair Normalizing Flows (Balunović et al., 2022). Marginal training uses the analogous marginal KL divided by d. We use γ∈.05,.50,.95γ∈\.05,.50,.95\, 60 epochs, batch size 256, AdamW (learning rate .001.001, weight decay .0001.0001), gradient clipping at five, and a ten-epoch ramp. Each KL direction uses 128 samples from its fitted prior. Checkpoint selection (passing criteria) minimizes validation MMD (worst-label conditional MMD for conditional FNF), subject to retaining at least 90% of ERM validation AUROC. All 36 FNF fits completed without fallback; all fitted GMMs converged. The existing MMD and adversarial-adapter experiments are ancillary comparisons; the marginal-versus-conditional analysis here holds the FNF architecture fixed. Results are detailed in Tables 11 and 12. Evaluation. The frozen representation Z and scalar score R are audited separately, both marginally and within each outcome class. Probes use up to 40,000 training examples; the RBF-SVM uses at most 2,000 per sex. Hyperparameters and MLP early stopping use only probe-validation balanced BCE. Decision metrics use a common, sex-independent threshold selected by validation Youden’s index. Calibration uses 15 equal-width ECE bins and logistic recalibration. Figure 10: Invariance strength changes the fairness trade-off. Lines and bands show means ± std. Conditional alignment lowers EOEO gaps at strong alignment, while marginal alignment produces smaller DPDP gaps. Table 11: Fairness and calibration for every invariance method strength (mean ± seed reported). Δbias _ bias is the absolute between-sex difference in mean residual Y−RY-R; Δcal _ cal is the absolute difference in fixed-slope calibration intercepts. Task Method ΔDP _ DP ΔEO _ EO Max ECE Δbias _ bias Δcal _ cal Edema ERM 0.147±0.0020.147± 0.002 0.102±0.0070.102± 0.007 0.020±0.0090.020± 0.009 0.013±0.0060.013± 0.006 0.092±0.0370.092± 0.037 M .05 0.132±0.0020.132± 0.002 0.088±0.0050.088± 0.005 0.037±0.0030.037± 0.003 0.018±0.0050.018± 0.005 0.160±0.0430.160± 0.043 M .50 0.070±0.0050.070± 0.005 0.019±0.0050.019± 0.005 0.037±0.0020.037± 0.002 0.062±0.0040.062± 0.004 0.504±0.0120.504± 0.012 M .95 0.009±0.0090.009± 0.009 0.048±0.0140.048± 0.014 0.069±0.0060.069± 0.006 0.101±0.0060.101± 0.006 0.797±0.0710.797± 0.071 C .05 0.134±0.0050.134± 0.005 0.094±0.0050.094± 0.005 0.034±0.0090.034± 0.009 0.020±0.0090.020± 0.009 0.165±0.0640.165± 0.064 C .50 0.092±0.0090.092± 0.009 0.042±0.0060.042± 0.006 0.039±0.0040.039± 0.004 0.046±0.0080.046± 0.008 0.387±0.0550.387± 0.055 C .95 0.054±0.0130.054± 0.013 0.016±0.0020.016± 0.002 0.051±0.0060.051± 0.006 0.072±0.0070.072± 0.007 0.582±0.0290.582± 0.029 Cardiomegaly ERM 0.130±0.0080.130± 0.008 0.092±0.0090.092± 0.009 0.022±0.0030.022± 0.003 0.013±0.0080.013± 0.008 0.188±0.1020.188± 0.102 M .05 0.104±0.0250.104± 0.025 0.070±0.0320.070± 0.032 0.030±0.0060.030± 0.006 0.019±0.0060.019± 0.006 0.266±0.0870.266± 0.087 M .50 0.049±0.0030.049± 0.003 0.010±0.0030.010± 0.003 0.031±0.0060.031± 0.006 0.044±0.0020.044± 0.002 0.635±0.0320.635± 0.032 M .95 0.005±0.0030.005± 0.003 0.051±0.0160.051± 0.016 0.050±0.0020.050± 0.002 0.070±0.0010.070± 0.001 0.962±0.0180.962± 0.018 C .05 0.119±0.0250.119± 0.025 0.091±0.0310.091± 0.031 0.028±0.0080.028± 0.008 0.016±0.0080.016± 0.008 0.246±0.1100.246± 0.110 C .50 0.061±0.0030.061± 0.003 0.022±0.0040.022± 0.004 0.027±0.0010.027± 0.001 0.041±0.0020.041± 0.002 0.592±0.0180.592± 0.018 C .95 0.036±0.0040.036± 0.004 0.014±0.0080.014± 0.008 0.035±0.0010.035± 0.001 0.052±0.0010.052± 0.001 0.737±0.0080.737± 0.008 Table 12: All settings: utility and strongest demographic probe AUROC (mean ± sample std over all training seeds). M/C denote marginal/conditional objectives, followed by γ. Task Method AUROC AUPRC DA(Z)D_A(Z) DA|Y(Z)D_A Y(Z) DA|Y(R)D_A Y(R) Edema ERM 0.839±0.0030.839± 0.003 0.646±0.0070.646± 0.007 0.978±0.0020.978± 0.002 0.978±0.0020.978± 0.002 0.613±0.0190.613± 0.019 M .05 0.835±0.0050.835± 0.005 0.641±0.0110.641± 0.011 0.693±0.0220.693± 0.022 0.678±0.0220.678± 0.022 0.583±0.0110.583± 0.011 M .50 0.835±0.0020.835± 0.002 0.640±0.0060.640± 0.006 0.631±0.0340.631± 0.034 0.624±0.0290.624± 0.029 0.519±0.0080.519± 0.008 M .95 0.830±0.0020.830± 0.002 0.630±0.0050.630± 0.005 0.623±0.0420.623± 0.042 0.629±0.0430.629± 0.043 0.555±0.0080.555± 0.008 C .05 0.829±0.0030.829± 0.003 0.632±0.0080.632± 0.008 0.675±0.0130.675± 0.013 0.664±0.0240.664± 0.024 0.582±0.0100.582± 0.010 C .50 0.837±0.0030.837± 0.003 0.642±0.0070.642± 0.007 0.613±0.0040.613± 0.004 0.599±0.0130.599± 0.013 0.537±0.0100.537± 0.010 C .95 0.834±0.0040.834± 0.004 0.636±0.0080.636± 0.008 0.596±0.0160.596± 0.016 0.590±0.0230.590± 0.023 0.513±0.0030.513± 0.003 Cardiomegaly ERM 0.848±0.0050.848± 0.005 0.531±0.0100.531± 0.010 0.978±0.0030.978± 0.003 0.978±0.0020.978± 0.002 0.602±0.0090.602± 0.009 M .05 0.841±0.0020.841± 0.002 0.514±0.0090.514± 0.009 0.703±0.0190.703± 0.019 0.696±0.0230.696± 0.023 0.607±0.0160.607± 0.016 M .50 0.845±0.0030.845± 0.003 0.524±0.0060.524± 0.006 0.650±0.0210.650± 0.021 0.636±0.0250.636± 0.025 0.551±0.0110.551± 0.011 M .95 0.840±0.0040.840± 0.004 0.507±0.0110.507± 0.011 0.644±0.0280.644± 0.028 0.646±0.0230.646± 0.023 0.533±0.0080.533± 0.008 C .05 0.839±0.0040.839± 0.004 0.507±0.0140.507± 0.014 0.673±0.0120.673± 0.012 0.663±0.0120.663± 0.012 0.592±0.0190.592± 0.019 C .50 0.846±0.0030.846± 0.003 0.526±0.0080.526± 0.008 0.630±0.0050.630± 0.005 0.623±0.0040.623± 0.004 0.537±0.0020.537± 0.002 C .95 0.844±0.0040.844± 0.004 0.521±0.0080.521± 0.008 0.617±0.0020.617± 0.002 0.616±0.0040.616± 0.004 0.515±0.0010.515± 0.001 B.2 Additional Details: Probe Results Strong conditional alignment did reduce conditional representation probe AUROC from ≈ .978.978 under ERMERM to .590.590 (Edema) and .616.616 (Cardiomegaly), but none of the 18 conditional representations pass the marginal or conditional near-chance AUROCAUROC criterion (which is, in fact is unrealistic for medical images. Figure 11 shows why a single probe family is insufficient. Score-level results are stronger: conditional FNF at γ=.95γ=.95 passes all three test criteria for Edema in 3/3 seeds. Neither task passes the conditional AUROCAUROC criterion (Table 13), but are sufficient to support our theoretical findings. Figure 11: Conditional probe-family audit at γ=.95γ=.95. Top: representations; bottom: scores. Table 13: Conditional score audit. Upper bounds use the within-scope Bonferroni correction; passing upper bound ≤0.55≤ 0.55. All three Edema test runs also pass the balanced-accuracy and log-loss interval checks. Task Seed Test DA|Y(R)D_A Y(R) Test upper Dev. upper Test pass Edema 42 0.515 0.540 0.591 Yes 123 0.510 0.541 0.563 Yes 456 0.514 0.545 0.572 Yes Cardiomegaly 42 0.514 0.557 0.637 No 123 0.516 0.559 0.618 No 456 0.515 0.558 0.606 No B.3 Additional Details: Calibration and operating-point analysis At γ=.95γ=.95, conditional FNF improves mean AUROCAUROC, EOEO, and worst-group ECEECE relative to marginal FNF on both tasks, but increases DPDP gaps. Detailed in Table 14; Fig. 13. The EOEO comparison depends on the operating point (Fig. 13). All results support our propositions, though extreme thresholds can shrink gaps by predicting the same label for everyone. Stronger alignment is therefore not preferable, and reduced gaps alone do not establish improved predictions. Figure 12: Calibration costs vs. improved EOEO. Means ± std across strengths. Figure 13: Fairness gaps dependency on threshold. ERMERM and FNF at γ=.95γ=.95, evaluated on the same test cohorts. Bands show training seed std; the vertical dotted line marks the default .5.5 threshold. Table 14: Sex-calibration at γ=.95γ=.95 (mean ± std reported). αa _a is fitted separately with slope fixed at one. Negative intercepts and negative mean residuals ⟹ overprediction. Ideal slope is 1; ideal intercept and mean residual are 0. Task Method A ECE Slope bab_a Intercept αa _a y¯a−r¯a y_a- r_a Edema ERM 0 0.010±0.0050.010± 0.005 0.978±0.0540.978± 0.054 0.001±0.0340.001± 0.034 −0.000±0.004-0.000± 0.004 1 0.020±0.0090.020± 0.009 0.926±0.0440.926± 0.044 0.093±0.0710.093± 0.071 0.013±0.0100.013± 0.010 M .95 0 0.069±0.0060.069± 0.006 0.875±0.0370.875± 0.037 −0.561±0.057-0.561± 0.057 −0.069±0.006-0.069± 0.006 1 0.036±0.0040.036± 0.004 0.858±0.0560.858± 0.056 0.236±0.0300.236± 0.030 0.032±0.0030.032± 0.003 C .95 0 0.051±0.0060.051± 0.006 0.867±0.0790.867± 0.079 −0.416±0.036-0.416± 0.036 −0.050±0.007-0.050± 0.007 1 0.029±0.0070.029± 0.007 0.861±0.0760.861± 0.076 0.166±0.0430.166± 0.043 0.023±0.0050.023± 0.005 Cardiomegaly ERM 0 0.015±0.0040.015± 0.004 0.923±0.0830.923± 0.083 −0.136±0.142-0.136± 0.142 −0.009±0.010-0.009± 0.010 1 0.018±0.0080.018± 0.008 0.870±0.1020.870± 0.102 0.052±0.1110.052± 0.111 0.005±0.0090.005± 0.009 M .95 0 0.050±0.0020.050± 0.002 0.851±0.0300.851± 0.030 −0.718±0.031-0.718± 0.031 −0.050±0.002-0.050± 0.002 1 0.023±0.0010.023± 0.001 0.864±0.0200.864± 0.020 0.244±0.0250.244± 0.025 0.021±0.0020.021± 0.002 C .95 0 0.035±0.0010.035± 0.001 0.857±0.0360.857± 0.036 −0.534±0.008-0.534± 0.008 −0.035±0.001-0.035± 0.001 1 0.024±0.0030.024± 0.003 0.818±0.0180.818± 0.018 0.204±0.0160.204± 0.016 0.017±0.0010.017± 0.001 Appendix C Appendix 3: Fairness Metrics Do Not Establish Invariance Failure to achieve demographic erasure using popular methods. We initially used more popular methods, specifically DANN (Ganin et al., 2016) and class-conditional DANN Long et al. (2018). Neither achieved its objective, i.e., to erase demographic information in our CheXpert experiments. Across eight adversary strengths and five seeds, probes were still able to recover demographic information from learned representations. Fairness gaps can sometimes decrease (and even be zero) because diagnostic performance also decreases as a function of demographic erasure. We leave it to future work to investigate why certain methods fail to erase demographic signals even with strong constraints while others succeed, and why it is easy on some datasets and not on others. This also shows that improved fairness metrics alone do not establish invariance. Our broader exploratory comparison revealed substantial variation across methods and hyperparameters: some reduced demographic decodability while retaining useful prediction, whereas others retained demographic information or substantially reduced predictive performance. Moreover, nonlinear probes recovered information that linear probes missed, highlighting the importance of evaluating erasure with multiple probe families (also discussed in Appendix 2). Beyond DANN and CDANN, we also explored MMD (Gretton et al., 2006) and HSIC (Pérez-Suay et al., 2017) penalties during training and their alternative variants, such as adversarial ensemble variational bottlenecks with MMD (Gretton et al., 2006), learning Fair Representations (LFR) (Zemel et al., 2013), Gaussian moment transport, LEACE (Belrose et al., 2023), LEOPARD (Ravfogel et al., 2022), Obliviate (Shakibania et al., 2026), random Fourier features (Rahimi and Recht, 2007) followed by LEACE. Across 51 configurations in the single-seed Edema study, LFR, FNF, Gaussian transport, LEACE, LEOPARD, and RFF–LEACE did not meet our expectation to reduce demographic decodability. As the probe-comparison plot illustrates in Fig 14, low linear decodability sometimes hides nonlinear leakage. Some methods like MMD and HSIC did achieve near-chance decodability but with reduced task utility (classification AUROC). Similarly, Obliviator also costs predictive performance; variational bottlenecks and gender-aware channels retained decodable information. An analysis is detailed in Table 16. Note that this analysis does not reflect a limitation or general failure of the methods. Table 15: Evaluation across pathologies and objectives. Task AUROC, marginal and class-conditional sex probe AUROCs, demographic parity (DPDP) gap, and equalized odds (EOEO) gap. Results are reported as mean ± std. Sex Probe AUROC ↓ Fairness Gaps ↓ Pathology Method Task AUROC ↑ Marginal = Y=0 = Y=1 DP Gap EO Gap Edema DANN 0.842 ± 0.003 0.950 ± 0.005 0.950 ± 0.006 0.938 ± 0.008 0.069 ± 0.023 0.033 ± 0.025 CDANN 0.845 ± 0.003 0.952 ± 0.006 0.955 ± 0.007 0.940 ± 0.012 0.111 ± 0.015 0.074 ± 0.025 Cardiomegaly DANN 0.842 ± 0.004 0.947 ± 0.006 0.951 ± 0.006 0.928 ± 0.018 0.054 ± 0.031 0.037 ± 0.015 CDANN 0.843 ± 0.006 0.944 ± 0.030 0.957 ± 0.008 0.924 ± 0.027 0.105 ± 0.028 0.078 ± 0.042 (a) Nonlinear probes reveal demographic information missed by linear probes. (b) Diagnostic AUROC and ΔEO _EO across adversarial strengths. (c) Atelectasis: small fairness gaps coexist with strong demographic decodability. Figure 14: Evaluating demographic erasure on CheXpert. (a) Comparison of linear and nonlinear probes in the single-seed exploratory Edema study. (b–c) Summary of DANN and CDANN experiments across adversarial strengths and seeds. Chance probe AUROC does not establish independence, and marginal probes alone do not assess conditional invariance. Method Configs. Task AUROC ↑ Probe AUROC ↓ Pass Main Finding / Limitation Baselines and Controls ERM 1 0.840 0.961 0/1 Strong demographic leakage. ERM adapters 3 0.833 0.554–0.803 0/3 Compression reduced leakage but did not meet the criterion. PCA + ERM 2 0.838 0.813–0.886 0/2 Substantial residual leakage. Constant output 1 0.500 0.502 1/1 Trivial removal through complete task collapse. Learned Representations MMD 5 0.802–0.819 0.514–0.650 1/5 The 2-D setting passed development and test auditing. HSIC 5 0.774–0.818 0.521–0.619 1/5 The 2-D setting passed, with reduced task utility. Adversarial ensemble 3 0.812–0.828 0.521–0.771 1/3 Strongest setting passed test but missed development criterion. Variational bottleneck + MMD 3 0.804–0.806 0.502–0.536 3/3 Single-sample passes; mean and repeated-release audits failed. LFR 3 0.834–0.836 0.610–0.758 0/3 Retained demographic information. NF 6 0.771–0.838 0.561–0.854 0/6 Reduced leakage but did not meet strict criterion. Post-Hoc Representation Transformations Gaussian moment transport 2 0.832–0.839 0.650–0.853 0/2 Moment matching left demographic information recoverable. LEACE 2 0.836–0.838 0.962–0.963 0/2 Strong residual leakage in the tested implementation. LEOPARD 3 0.829–0.834 0.834–0.876 0/3 Nonlinear probes recovered substantial information. RFF + LEACE 3 0.818–0.836 0.831–0.905 0/3 Near-chance linear probes concealed nonlinear leakage. Obliviator 3 0.749–0.759 0.500–0.514 3/3 Only one setting met AUROC-loss budget; AUPRC fell substantially. Score-Only Mechanisms Sex-blind score channel 3 0.825–0.829 0.510–0.512 3/3 Also passed repeated-release auditing; backbone unchanged. Sex-aware score channel 3 0.836–0.837 0.513–0.517 3/3 Repeated releases exposed demographic information. Table 16: Exploratory demographic-erasure experiments on CheXpert Edema at natural prevalence (one training seed; 51 configurations). Results do not reflect general limitations of the evaluated methods. C.1 Additional Details: Clarifications Regarding using sensitive attributes A. In Section. 2. Acquisition variables such as scanner, site, or protocol are nuisance factors that can affect Z, which is unwanted, and are handled by domain-invariant methods. We use A to denote demographic attributes in this work that apply to people; usage and interpretation may differ if we consider nuisance variables.

Instant research discovery