Paper deep dive
Counterfactual Contrastive Analysis
Yunlong He, Pietro Gori
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 8/20/2026, 5:32:07 AM
Summary
This paper proposes a classifier-free approach for Visual Counterfactual Explanations (VCEs) based on Contrastive Analysis (CA). The method disentangles generative factors into common and salient components using a StyleGAN2-based framework (CS-StyleGAN) and refines the output in the F-space to preserve details. Unlike existing classifier-dependent methods, this approach operates on data distributions, making it robust to classifier biases. The method is evaluated on three medical imaging datasets (BloodMNIST, OCTMNIST, BraTS2023), demonstrating superior generation quality and disentanglement compared to baselines.
Entities (12)
Relation Signals (9)
CS-StyleGAN → isevaluatedon → BraTS2023
confidence 99% · We evaluate our method on three medical imaging datasets: ... BraTS2023...
CS-StyleGAN → isevaluatedon → BloodMNIST
confidence 99% · We evaluate our method on three medical imaging datasets: BloodMNIST...
CS-StyleGAN → isevaluatedon → OCTMNIST
confidence 99% · We evaluate our method on three medical imaging datasets: ... OCTMNIST...
CS-StyleGAN → isbasedon → StyleGAN2
confidence 98% · Our approach leverages the high-quality synthesis and well-structured latent space of StyleGAN2.
CS-StyleGAN → uses → F-space
confidence 95% · We use the feature space F, instead than the usual W-space, to improve detail preservation.
CS-StyleGAN → outperforms → Double InfoGAN
confidence 92% · our pSp-based method... consistently outperforms all CA baselines on both reconstruction and swapping.
CS-StyleGAN → outperforms → MM-cVAE
confidence 92% · our pSp-based method... consistently outperforms all CA baselines on both reconstruction and swapping.
CS-StyleGAN → outperforms → SepVAE
confidence 92% · our pSp-based method... consistently outperforms all CA baselines on both reconstruction and swapping.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Visual Counterfactual Explanations (VCEs) aim to explain image classifiers by generating minimally edited and realistic versions of an input image that change the classifier's prediction. Existing VCE methods are inherently classifier-dependent and therefore susceptible to classifier biases and failure modes, such as sensitivity to shortcut features and calibration errors. In this paper, we propose a classifier-free approach for visual counterfactual generation based on Contrastive Analysis (CA). Given two datasets corresponding to different classes (e.g., healthy and patients), we disentangle the generative factors that are common across the two datasets from those that are salient to each dataset, and generate counterfactual images by swapping only the salient factors. By operating directly on data distributions rather than decision boundaries, our method provides model-agnostic VCEs that are less sensitive to classifier biases. Our approach leverages the high-quality synthesis and well-structured latent space of StyleGAN2. We use the feature space F, instead than the usual W-space, to improve detail preservation. Unlike conventional CA approaches, which typically assume salient factors in only one dataset, we introduce an adapted framework and loss functions for VCE that allow multiple salient factors in each dataset. We evaluate our method on three medical imaging datasets and demonstrate superior counterfactual generation quality compared to existing approaches.
Tags
Links
- Source: https://arxiv.org/abs/2608.19032v1
- Canonical: https://arxiv.org/abs/2608.19032v1
Trouble viewing inline? Open PDF directly →
Full Text
42,702 characters extracted from source content.
Expand or collapse full text
Counterfactual Contrastive Analysis Yunlong He Affiliation: LTCI, Télécom Paris, Institut Polytechnique de Paris, France Pietro Gori Thanks: Corresponding author. Affiliation: E-mail pietro.gori@telecom-paris.fr Abstract Visual Counterfactual Explanations (VCEs) aim to explain image classifiers by generating minimally edited and realistic versions of an input image that change the classifier’s prediction. Existing VCE methods are inherently classifier-dependent and therefore susceptible to classifier biases and failure modes, such as sensitivity to shortcut features and calibration errors. In this paper, we propose a classifier-free approach for visual counterfactual generation based on Contrastive Analysis (CA). Given two datasets corresponding to different classes (e.g., healthy and patients), we disentangle the generative factors that are common across the two datasets from those that are salient to each dataset, and generate counterfactual images by swapping only the salient factors. By operating directly on data distributions rather than decision boundaries, our method provides model-agnostic VCEs that are less sensitive to classifier biases. Our approach leverages the high-quality synthesis and well-structured latent space of StyleGAN2. We use the feature space F, instead than the usual W-space, to improve detail preservation. Unlike conventional CA approaches, which typically assume salient factors in only one dataset, we introduce an adapted framework and loss functions for VCE that allow multiple salient factors in each dataset. We evaluate our method on three medical imaging datasets and demonstrate superior counterfactual generation quality compared to existing approaches. Keywords: Visual Counterfactual Explanations Contrastive Analysis 1 Introduction With the rapid scaling of modern AI models, the ability to explain black-box decisions has become critical, particularly in medical imaging. Explainability often governs whether models can be trusted and accepted for real-world use. Visual counterfactual explanations (VCEs) [14, 2, 9, 15] are widely studied as a way to probe a classifier’s behavior, seeking the smallest realistic changes to an input image that would alter its prediction. VCEs for image classifiers are typically realized by generating a counterfactual (CF) image as an edited version of the original input. This task is often posed as a constrained generation problem with three objectives. First, the CF must be valid, meaning that it changes the classifier’s prediction to the desired outcome. Second, the edited image should remain realistic, staying on the image manifold and avoiding unnecessary changes to irrelevant content. Third, the induced changes should be interpretable and minimal in a semantic sense, so that the explanation clearly indicates the key evidence responsible for the classifier’s decision change. Most VCE methods instantiate these objectives within a generative framework, leveraging VAEs [27], GANs [20, 32], or diffusion models (DMs) [15, 14, 35, 16]. Among them, DMs currently achieve state-of-the-art performance for realistic CF image generation, typically guiding the generation process with classifier-driven losses to reach a label-flipping outcome, while penalizing deviations to preserve plausibility and similarity to the input. However, classifier guidance can induce changes aligned with the classifier biases and failure modes, such as sensitivity to shortcut features [35] and calibration errors, rather than realistic changes in the underlying data distribution, potentially reducing edit fidelity. In addition, these methods offer limited insight into which semantic factors drive the prediction flip, thereby limiting their interpretability. In this paper, we introduce a classifier-free approach for visual counterfactual generation grounded in the Contrastive Analysis (CA) framework [1, 34, 7, 22, 23, 10]. Instead of generating CFs based on a classifier’s decision boundary, we operate at the data-distribution level, as in CA. Given two class-specific datasets, the proposed method aims to uncover the common and salient (i.e., class-specific) generative factors. CFs can then be generated by swapping only the salient factors, while preserving the common ones. Most CA methods are based on deep generative models, such as VAEs [6, 34, 30, 38, 3, 19] and GAN[7, 8]. While reconstruction can regularize the salient space by suppressing common information, it is insufficient on its own and typically requires additional constraints for clean latent separation [34, 22, 23, 28]. A recent work [10] extended CA to high-quality image datasets using modern GANs and diffusion models and reported that StyleGAN2 [17] achieves diffusion-level quality while enabling substantially faster inference. In parallel, advances in StyleGAN editing have shown that translating edits from the W-space to the generator’s intermediate feature space (F-space) improves fine-grained details [5]. Inspired by these findings, we propose a StyleGAN-based CA framework that leverages latent disentanglement and F-space refinement for VCEs on medical imaging datasets. Our contributions are: 1. A classifier-free VCE generative framework grounded in CA that enables controllable, classifier-agnostic CF generation via disentangled latent representations, yielding high-fidelity results and high semantic interpretability. 2. A CA-based framework, supporting both background-target and multi-salient assumptions, based on StyleGAN2’s high-quality synthesis and well-structured F-space, instead of standard W-space, for better detail preservation. 3. Experiments on three medical imaging datasets demonstrate improved disentanglement performance over CA baselines and superior generation quality to prior state-of-the-art VCE approaches with faster image editing (0.25s/image). Figure 1: Overview of the proposed CA-based framework for CF generations. Left: we first disentangle each input into a common c and a salient factor s, enabling reconstruction and CF editing by swapping the salient factors between images from X and Y. Right: we then refine the synthesized image by adjusting F-space features according to the feature shift Δfy→x=f(cy+sy)−f(cy+sx) f_y→ x=f(c_y+s_y)-f(c_y+s_x) common-salient factor manipulation. 2 Method Problem Statement. In the CA setting, we consider two datasets X=xii=1NX=\x_i\_i=1^N and Y=yjj=1MY=\y_j\_j=1^M. Most prior work adopts a background-target (BT) assumption, where only one dataset (typically Y) contains a dataset-specific (or salient) pattern (e.g., a pathology) absent from the other, while both share common content (e.g., healthy anatomy). We generalize CA to a multiple-salient (MS) setting, where both X and Y contain their own salient patterns that are absent in the other. Our goal is to learn a common generative factor c∈ℝdcc ^d_c capturing information shared across X and Y, and salient generative factors sx∈ℝdsxs_x ^d_sx, sy∈ℝdsys_y ^d_sy that capture variations unique to X and Y, respectively. Common and Salient (CS)-StyleGAN. In the context of StyleGAN, given samples x∈Xx∈ X and y∈Yy∈ Y, an encoder E (i.e., pSp [26]) maps them to latent codes wx=E(x)w_x=E(x) and wy=E(y)w_y=E(y), where wx,wy∈ℝdww_x,w_y ^d_w. We introduce a CS separator Hcs,ϕH_cs,φ, which consists of one common branch ϕcC_ _c and two salient branches x,ϕsxS_x, _sx and y,ϕsyS_y, _sy. For brevity, we omit the parameters ϕφ in the following. We define the latent factors as: cx=(wx)c_x=C(w_x) and cy=(wy)c_y=C(w_y). In the MS setting, we set sx=x(wx)s_x=S_x(w_x) and sy=y(wy)s_y=S_y(w_y), while in the BT setting, we remove xS_x and use yS_y only (all xS_x-related terms are omitted). Latent Regularization. We first train the HcsH_cs separator with the following reconstruction loss in the StyleGAN2 latent space: ℒlat=‖(wx)+x(wx)−wx‖22+‖(wy)+y(wy)−wy‖22,L_lat=\|C(w_x)+S_x(w_x)-w_x\|_2^2+\|C(w_y)+S_y(w_y)-w_y\|_2^2, (1) where each term encourages the sum of the common output and the corresponding salient output to reconstruct the StyleGAN2 latent (wxw_x or wyw_y). To enforce that the salient factors capture only dataset-specific information, we additionally penalize cross-dataset activations, by minimizing ‖y(wx)‖22\|S_y(w_x)\|_2^2 and ‖x(wy)‖22\|S_x(w_y)\|_2^2 (we use only ‖y(wx)‖22\|S_y(w_x)\|_2^2 under the BT assumption, as explained in Fig. 1). The loss in Eq. (1) is necessary but insufficient. In particular, it does not enforce (i) common consistency, i.e., that the common factors capture the same information across X and Y, nor (i) common-salient independence, i.e., that common and salient factors do not share information. To promote common consistency, we impose distributional alignment of the common factors by introducing a discriminator D that predicts whether (w)C(w) originates from X or Y, while the separator HcsH_cs is trained to make this prediction impossible: ℒD=minHcsmax(−wy[log((wy))]−wx[log(1−((wx)))]).L_D= _H_cs _D (-E_w_y[ (C(w_y))]-E_w_x[ (1-D(C(w_x)))] ). (2) Here, D is optimized to discriminate X from Y using the common codes, whereas HcsH_cs is optimized adversarially so that (wx)C(w_x) and (wy)C(w_y) become indistinguishable. To encourage common–salient statistical independence, we introduce two regressors, ℛxR_x and ℛyR_y, that attempt to predict the salient outputs x(wx)S_x(w_x) and y(wy)S_y(w_y) from the corresponding common outputs (wx)C(w_x) and (wy)C(w_y), respectively. If the regressors can accurately predict salient output from the common one, this indicates that (w)C(w) still contains salient information and the separation is incomplete. We therefore train the separator adversarially against the regressors by minimizing the following objective: ℒR=minHcsmaxℛ(CLOSE _R= _H_cs _R ( −wx[∥ℛx((wx))−x(wx)∥22]−wy[∥ℛx((wy))] -E_w_x\! [\|R_x(C(w_x))\!-\!S_x(w_x)\|_2^2 ]\!-E_w_y\! [\|R_x(C(w_y)) ] (3) −wy[∥ℛy((wy))−y(wy)∥22]−wx[∥ℛy((wx))]). -E_w_y\! [\|R_y(C(w_y))\!-\!S_y(w_y)\|_2^2 ]\!-E_w_x\! [\|R_y(C(w_x)) ] ). This discourages (w)C(w) from encoding salient patterns. We regress s from c because c is typically bigger (more information) and thus tends to absorb s. Image-Space Losses. We adopt an image-space reconstruction loss combining pixel and perceptual terms. Given a real image i and its reconstruction i i, we use ℒrec(i,i^)=‖i−i^‖22+‖V(i)−V(i^)‖22L_rec(i, i)=\|i- i\|_2^2+\|V(i)-V( i)\|_2^2, where the first term is a pixel-wise ℓ2 _2 loss, and the second term is the LPIPS perceptual loss [37]. The function V(⋅)V(·) extracts perceptual features using a pretrained VGG16 network [31]. We use this Eq. to reconstruct both x and y, minimizing the image-space loss: ℒimg=ℒrec(x,G(cx+sx))+ℒrec(y,G(cy+sy)),L_img=L_rec(x,G(c_x+s_x))+L_rec(y,G(c_y+s_y)), (4) where G is the pretrained StyleGAN2 generator. The final loss to train HcsH_cs is: ℒHcs=λlatℒlat+λDℒD+λRℒR+λimgℒimgL_H_cs= _latL_lat+ _DL_D+ _RL_R+ _imgL_img (5) F-space Refinement. We first learn common and salient factors in the W space of StyleGAN2. Then, we add a refinement module that edits intermediate StyleGAN feature space (F space), as in [5]. Let f(⋅)f(·) denote an intermediate feature of the generator G, e.g., the output of layer l=9l=9. Given the learned latent factors, we define F-space feature shifts for the two swap directions as: Δfx→y=f(cx+sx)−f(cx+sy);Δfy→x=f(cy+sy)−f(cy+sx). f_x→ y=f(c_x+s_x)-f(c_x+s_y);\;\; f_y→ x=f(c_y+s_y)-f(c_y+s_x). (6) Intuitively, Δfx→y f_x→ y captures the change induced by replacing the salient factor of x with the one of y while keeping cxc_x fixed, and Δfy→x f_y→ x is defined analogously. To condition the shifts on the source image, we extract features from reconstructions using a pretrained F-space encoder EfE_f: fx=Ef(G(cx+sx))f_x\!=\!E_f(G(c_x\!+\!s_x)) and fy=Ef(G(cy+sy))f_y\!=\!E_f(G(c_y\!+\!s_y)). As in [5], we concatenate f and Δf f, and pass them to an adapter FAdptF_Adpt, yielding: f^x→y=FAdpt(fx⊕Δfx→y) f_x\!→\!y\!=\!F_Adpt(f_x\! \! f_x\!→\!y) and f^y→x=FAdpt(fy⊕Δfy→x) f_y\!→\!x\!=\!F_Adpt(f_y\! \! f_y\!→\!x), where ⊕ denotes concatenation. Decoding the adapted features with G yields the edited images. FAdptF_Adpt is trained with: ℒx→y=ℒrec(G(cx+sy),G(f^x→y))L_x→ y=L_rec(G(c_x+s_y),\;G( f_x→ y)) and ℒy→x=ℒrec(G(cy+sx),G(f^y→x))L_y→ x=L_rec(G(c_y+s_x),\;G( f_y→ x)). We also include a real-image reconstruction term to preserve performance on real inputs: ℒreal=ℒrec(x,x^)+ℒrec(y,y^)L_real=L_rec(x,\; x)+L_rec(y,\; y), where x^=G(FAdpt(Ef(x))) x=G(F_Adpt(E_f(x))) and y^=G(FAdpt(Ef(y))) y=G(F_Adpt(E_f(y))). Finally, we encourage realism with an adversarial loss ℒadv=ℒadv-x+ℒadv-yL_adv=L_adv-x+L_adv-y, where ℒadv-xL_adv-x is applied to G(f^y→x)G( f_y→ x) against x, and ℒadv-yL_adv-y to G(f^x→y)G( f_x→ y) against y. The overall refinement objective is: ℒrefine=ℒx→y+ℒy→x+ℒreal+λadvℒadvL_refine=L_x→ y+L_y→ x+L_real+ _advL_adv. 3 Experiments Datasets. We evaluate our method on three 2D medical imaging datasets: BloodMNIST[36] (Eosinophil Vs Neutrophil; Train=2500/Test=500), OCTMNIST[36] (Normal Vs Choroidal neovascularization; Train=17000/Test=3000 + Diabetic macular edema Vs Drusen; Train=7000/Test=2000), and BraTS2023 [24] (Healthy Vs Tumor; Train=8000/Test=2000). For each dataset, we define two groups, denoted X and Y, based on the provided labels. We construct balanced splits such that X and Y contain the same number of images in both training and test sets. All images are preprocessed to a resolution of 256×256256× 256. For BloodMNIST and OCTMNIST, we resize the 224×224224× 224 images from the MedMNIST+ release to 256×256256× 256. For MRI scans, we use zero-padding to obtain 256×256256× 256 images while preserving the original aspect ratio. Implementation Details. We train the proposed CS-StyleGAN in two stages. In Stage 1, we optimize the separator HcsH_cs to disentangle common and salient factors in the +W^+ space. We first warm up HcsH_cs for 2,000 steps using only ℒlatL_lat and ℒimgL_img to obtain informative latent factors. We then alternate between updating D and ℛR and updating HcsH_cs: with HcsH_cs fixed, D and ℛR are updated by maximizing the objectives in Eqs. (2)–(3) using separate Adam optimizers with a learning rate of 10−410^-4; with D and ℛR fixed, HcsH_cs is updated by minimizing Eq. (5) using an Adam optimizer with a learning rate of 10−310^-3. In Stage 2, using the common and salient factors learned in Stage 1 and their corresponding F-space feature shifts, we optimize the refinement module (FAdptF_Adpt) with ℒrefineL_refine. We use the Ranger optimizer employed in SFE [5], with our learning rate set to 10−410^-4. For each dataset, the pSp encoder E, StyleGAN2 generator G, and F-space encoder EfE_f are pretrained on the corresponding training set before the two-stage training procedure. Detailed network architectures are provided in the supplementary material. The implementation code is available at https://github.com/BioMedTP/CF_Contrastive_Analysis. Baselines and Evaluation Metrics. We compare our method with CA baselines M-cVAE [34], SepVAE [23], and Double InfoGAN [7], and with diffusion-based VCE methods: ACE [15], DiME [14], FastDiME [35], and TIME [16]. We also consider off-the-shelf T2I DMs as CF alternatives (not CF-specific methods), including FLUX [4] and SDXL [25] adapted with LoRA [12]. We evaluate the proposed method on CA and CF generation using widely adopted metrics to evaluate synthesis quality: L2, LPIPS [37], MS-SSIM [33], and FID [11]. Furthermore, we assess whether swapped outputs match the post-edit domain distribution using FID, reporting FIDX→Y_X\!→\!Y (FID between X images after swapping to Y and real Y) and FIDY→X_Y\!→\!X (swapped Y vs. real X). We evaluate editing success using the same U-Net-based classifier architecture as in [35], trained on real images from the corresponding training set. We report the classification accuracy with respect to the desired post-edit label. About latent separation quality, we train a classifier on the learned latent factors to predict the class (X or Y). Higher accuracy indicates the class is encoded in the corresponding (common or salient) space. Please note that in the BT assumption, the information about X is entirely encoded in c. Eventually, following VCE prior works [35, 15], we report FID, sFID, Flip Ratio (FR), Mean Absolute Difference (MAD) and Bidirectional KL Divergence (BKL) for measuring counterfactual quality. Figure 2: Comparisons with CA baselines on BraTS, BloodMNIST, and OCTMNIST (Normal vs. CNV) (top to bottom). Reconstruction: use the common and salient factors from original images. Salient Swap: swap salient factors between x and y while keeping common factors fixed during generation. Table 1: Quantitative comparison results on multiple datasets. Best results in bold, second-best underlined (within each dataset block). Ref: F-space refinement; BT/MS Ass.: background-target/multi-salient assumptions. Reconstruction (X) Reconstruction (Y) Swap Dataset (X vs. Y) Method PSNR↑ MSE↓ MS-SSIM↑ LPIPS↓ FID↓ PSNR↑ MSE↓ MS-SSIM↑ LPIPS↓ FID↓ FIDX→Y↓_X\!→\!Y FIDY→X↓_Y\!→\!X Acc (X+Y)↑ BraTS (Healthy vs. Tumored) MMc-VAE 20.32 0.008 0.852 0.163 89.86 21.06 0.007 0.951 0.157 86.36 98.63 92.77 0.753 SepVAE 23.24 0.006 0.933 0.156 115.2 23.75 0.006 0.930 0.152 109.4 113.7 111.9 0.821 Double InfoGAN 23.96 0.005 0.925 0.151 125.1 25.44 0.004 0.958 0.145 128.3 137.2 135.4 0.901 Ours (pSp-cs) 24.03 0.004 0.91 0.054 44.46 24.75 0.003 0.90 0.055 30.10 37.10 45.50 0.939 Ours (pSp-cs-Ref) 38.19 0.0002 0.99 0.004 2.64 38.20 0.0002 0.99 0.005 2.57 25.17 32.56 0.914 BloodMNIST (Eos. vs. Neutrophil) MMc-VAE 18.27 0.015 0.556 0.378 175.1 19.42 0.011 0.643 0.357 179.6 181.7 183.9 0.819 SepVAE 18.04 0.016 0.551 0.421 182.8 18.01 0.016 0.579 0.419 219.5 217.5 221.9 0.691 Double InfoGAN 19.45 0.015 0.567 0.472 95.97 19.70 0.014 0.606 0.491 104.7 149.50 135.7 0.646 Ours (pSp-cs) 22.35 0.006 0.725 0.139 73.46 22.43 0.006 0.760 0.139 70.42 84.05 88.61 0.979 Ours (pSp-cs-Ref) 41.99 0.0001 0.998 0.004 2.887 42.54 0.0001 0.998 0.004 2.590 43.37 45.90 0.981 OCTMNIST (DME vs. Drusen) MMc-VAE 23.44 0.005 0.805 0.404 128.05 26.19 0.003 0.867 0.390 117.71 126.49 128.17 0.665 SepVAE 23.89 0.005 0.822 0.400 107.84 26.26 0.003 0.872 0.395 105.92 131.22 143.67 0.597 Double InfoGAN 16.84 0.026 0.592 0.369 181.38 18.89 0.017 0.698 0.300 179.87 189.06 199.42 0.832 Ours (BT Ass.) 23.79 0.005 0.806 0.111 51.10 24.99 0.004 0.830 0.084 42.62 65.66 79.13 0.627 Ours (BT Ass. Ref) 38.24 0.0002 0.992 0.0143 5.832 39.02 0.0001 0.994 0.012 3.294 44.67 63.62 0.689 Ours (MS Ass.) 26.50 0.005 0.814 0.108 48.64 24.70 0.004 0.837 0.083 39.43 59.21 73.79 0.876 Ours (MS Ass. Ref) 38.09 0.0001 0.994 0.014 5.356 39.00 0.0001 0.995 0.012 3.911 39.24 55.92 0.863 Comparison with CA Baselines. Fig. 2 qualitatively compares our method with CA baselines for reconstruction and salient swapping. Our reconstructions are sharper and better preserve fine anatomical structures. For salient swapping, our approach transfers the salient pattern (tumors in BraTS, eosinophil–neutrophil staining differences in BloodMNIST, and CNV-related abnormalities in OCTMNIST) while preserving shared anatomy (healthy brain tissues, global cell layout/nuclear morphology, and normal retinal layer structure, respectively). Table 1 reports quantitative results, where our pSp-based method, with or without F-space refinement (-Ref), consistently outperforms all CA baselines on both reconstruction and swapping. Notably, with the MS setting, our method better handles the challenging DME vs. Drusen case, achieving the best reconstruction metrics and the highest swapping quality (FID and classification accuracy of 0.876). Finally, Table 3 evaluates latent separation between the learned common (C) and salient (S) factors. Salient patterns are mainly captured in S rather than in C, and our method attains the lowest separation gap Δ across all methods, indicating improved semantic disentanglement over prior approaches. In Table 3, we also report an ablation study. Table 2: Latent separation results. Mean ± std of 5-fold cross-validated classification accuracy to distinguish between the class of X or Y using logistic regression and using as features either C, SxS_x or SyS_y. Δ=|0.5−C|+|1.0−S1|(+|1.0−S2|CLOSE = 0.5-C + 1.0-S_1 (+ 1.0-S_2 ). BraTS (Healthy vs. Tumored) OCTMNIST (DME vs. Drusen) Model C SyS_y Δ C SxS_x SyS_y Δ M-cVAE 0.68 ± 0.02 0.73 ± 0.02 0.45 0.61 ± 0.01 0.69± 0.03 – 0.42 SepVAE 0.67 ± 0.02 0.92 ± 0.01 0.25 0.64 ± 0.02 0.94 ± 0.01 – 0.20 Double Info. 0.65 ± 0.01 0.86 ± 0.01 0.29 0.76 ± 0.02 0.61 ± 0.02 – 0.65 Ours (BT Ass.) 0.58 ± 0.08 0.95 ± 0.01 0.13 0.57 ± 0.08 0.96 ± 0.006 – 0.11 Ours (MS Ass.) – – – 0.54 ± 0.05 0.97 ± 0.008 0.98 ± 0.003 0.09 Expected 0.5 1.0 0 0.5 1.0 1.0 0 Table 3: Ablation study on the BraTS dataset: Healthy (X) vs. Tumored (Y). Latent Separation Image Edit Quality Model C S Δ FIDX→Y_X\!→\!Y FIDY→X_Y\!→\!X Acc. Base (ℒlat+ℒimgL_lat\!+\!L_img) 0.74 ± 0.010 0.73 ± 0.02 0.51 39.78 55.75 0.535 Base + ℒDL_D 0.64 ± 0.02 0.92 ± 0.01 0.22 37.03 53.21 0.638 Base + ℒD+ℒDiscMIL_D\!+\!L_DiscMI 0.68 ± 0.01 0.86 ± 0.01 0.32 38.78 55.04 0.593 Base + ℒD+ℒRL_D\!+\!L_R 0.58 ± 0.08 0.95 ± 0.01 0.13 37.10 45.50 0.939 Base + ℒD+ℒR+ℒrefineL_D\!+\!L_R\!+\!L_refine – – – 25.17 35.56 0.913 Table 4: CF explanation comparison on BraTS. Best in bold, second-best underlined. Metrics for CF edits (X→ : add tumors; Y→ : remove tumors) and runtime. CF: X→ (add tumors) CF: Y→ (remove tumors) Method FID sFID L1 FR MAD BKL FID sFID L1 FR MAD BKL Time (s/img) ACE 38.553 51.428 0.003 0.977 0.942 0.043 39.436 52.841 0.005 0.895 0.855 0.120 21.37 ± 0.34 DiME 38.378 50.421 0.026 0.947 0.934 0.054 48.752 60.722 0.025 0.741 0.723 0.228 294.12 ± 47.44 FastDiME 32.431 44.735 0.017 0.905 0.873 0.108 51.391 64.007 0.016 0.696 0.658 0.285 16.07 ± 0.64 FastDiME-2+ 32.865 45.283 0.017 0.912 0.876 0.106 52.191 64.753 0.016 0.707 0.668 0.277 30.69 ± 0.91 TIME 76.704 91.064 0.012 0.389 0.293 0.475 88.706 100.074 0.012 0.341 0.247 0.413 17.59 ± 0.40 SDXL-LoRA 51.258 60.179 0.027 0.731 0.727 0.230 59.006 69.185 0.028 0.824 0.817 0.164 4.95 ± 0.67 Ours 27.533 38.441 0.021 0.953 0.946 0.047 37.492 47.527 0.022 0.914 0.906 0.074 0.25 ± 0.005 Counterfactual Explanations Based on CA. Table 4 reports quantitative comparisons on BraTS Healthy (X) and Tumored (Y) for two counterfactual directions: adding tumors (X→YX\!→\!Y) and removing tumors (Y→XY\!→\!X). Overall, our CA-based swapping achieves the best image fidelity in both directions, as indicated by the lowest FID/sFID scores, and is better or on par with diffusion-based CF baselines (ACE, DiME, and FastDiME) in terms of counterfactual effectiveness (i.e., FR/MAD/BKL). In Fig. 3, we observe that diffusion-based methods struggle to completely remove large tumors from Y images, and the added tumors in X→YX\!→\!Y have often limited size and/or are misplaced. In addition, text-conditioned editing (e.g., FLUX) or learning per-image embeddings (e.g., TIME) can drift away from the real image manifold, while fine-tuning large diffusion models with LoRA (e.g., SDXL-LoRA) may alter common content beyond the desired tumor-related changes. By contrast, our approach explicitly swaps salient generative factors between X and Y, improving image fidelity and offering better controllability for tumor-specific counterfactual manipulation. Thanks to the well structured latent space of StyleGAN2, we can also interpolate between salient factors. Given a pair of samples (x,y)(x,y) from two domains/classes, we can generate a continuous counterfactual trajectory by scaling the salient change with a scalar α∈[0,1]α∈[0,1], e.g., G(αf^x→y)G(α f_x→ y). This is shown in Fig. 4, where the predicted class probability changes smoothly along the trajectory, suggesting that the generated counterfactuals form a coherent explanation path. Figure 3: Comparisons of CF generation. Cols. 2–5 show CF outputs from SOTA diffusion-based methods. Cols. 6–8 show CF outputs from T2I diffusion models. Figure 4: Interpolation along salient factors. Gradual interpolation from sxs_x to sys_y (and viceversa) using an interpolation weight α (→10\!→\!1) during generation. The classifier prediction changes consistently along the trajectory, illustrating gradual insertion/removal of class-specific evidence while preserving common content. 4 Conclusion We propose a classifier-free, CA-based framework for VCEs that yields strong common–salient disentanglement and high-quality CF generations. Interesting perspectives might be adapting the proposed method to recent VLM, as in [18], and studying its identifiability. Similarly to all recent VCE [14, 15, 35] and CA [7, 22] works, we note that also our method also, without strong assumptions, is not identifiable [13, 21] and does not enable causal discovery. This is generally impossible from purely observational single-modal data. An interesting direction, in line with causal representation learning theory [29], would be using additional prior or longitudinal data, to further constrain the learning process. References [1] A. Abid, M. J. Zhang, V. K. Bagaria, and J. Zou (2018) Exploring patterns enriched in a dataset with contrastive principal component analysis. Nature Communications. Cited by: §1. [2] M. Augustin, V. Boreiko, F. Croce, and M. Hein (2022) Diffusion visual counterfactual explanations. In NeurIPS, Cited by: §1. [3] S. Benaim, M. Khaitov, T. Galanti, and L. Wolf (2019) Domain intersection and domain difference. In ICCV, Cited by: §1. [4] Black Forest Labs et al. (2025) FLUX.1 Kontext: flow matching for in-context image generation and editing in latent space. Cited by: §3. [5] D. Bobkov, V. Titov, A. Alanov, and D. Vetrov (2024) The devil is in the details: StyleFeatureEditor for detail-rich StyleGAN inversion and high quality image editing. In CVPR, Cited by: §1, §2, §2, §3, §5.3. [6] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan (2016) Domain separation networks. In NeurIPS, Cited by: §1. [7] F. Carton, R. Louiset, and P. Gori (2024) Double InfoGAN for contrastive analysis. In AISTATS, Cited by: §1, §3, §4. [8] A. Gonzalez-Garcia, J. van de Weijer, and Y. Bengio (2018) Image-to-image translation for cross-domain disentanglement. In NeurIPS, Cited by: §1. [9] Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee (2019) Counterfactual Visual Explanations. In ICML, Cited by: §1. [10] Y. He, G. Lesné, Z. Liu, M. Soumm, and P. Gori (2025) Learning common and salient generative factors between two image datasets. Cited by: §1, §5.1. [11] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter (2017) GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, Cited by: §3. [12] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In ICLR, Cited by: §3. [13] A. Hyvärinen, H. Sasaki, and R. E. Turner (2019) Nonlinear ICA using auxiliary variables and generalized contrastive learning. In AISTATS, Cited by: §4. [14] G. Jeanneret, L. Simon, and F. Jurie (2022) Diffusion models for counterfactual explanations. In ACCV, Cited by: §1, §1, §3, §4. [15] G. Jeanneret, L. Simon, and F. Jurie (2023) Adversarial counterfactual visual explanations. In CVPR, Cited by: §1, §1, §3, §4. [16] G. Jeanneret, L. Simon, and F. Jurie (2024) Text-to-image models for counterfactual explanations: a black-box approach. In WACV, p. 4757–4767. Cited by: §1, §3. [17] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila (2020) Analyzing and improving the image quality of StyleGAN. In CVPR, Cited by: §1, §5.1, §5.3. [18] T. Kazimi, R. Allada, and P. Yanardag (2025) Explaining in Diffusion: Explaining a Classifier with Diffusion Semantics. In CVPR, Cited by: §4. [19] M. Kleinman, A. Achille, S. Soatto, and J. C. Kao (2023) Gács–körner common information variational autoencoder. In NeurIPS, Cited by: §1. [20] O. Lang, Y. Gandelsman, M. Yarom, Y. Wald, G. Elidan, A. Hassidim, W. T. Freeman, P. Isola, A. Globerson, M. Irani, and I. Mosseri (2021) Explaining in style: training a GAN to explain a classifier in stylespace. In ICCV, Cited by: §1. [21] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem (2019) Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML, Cited by: §4. [22] R. Louiset, E. Duchesnay, A. Grigis, B. Dufumier, and P. Gori (2024) SepVAE: a contrastive VAE to separate pathological patterns from healthy ones. In MIDL, Cited by: §1, §4. [23] R. Louiset, E. Duchesnay, A. Grigis, and P. Gori (2024) Separating common from salient patterns with contrastive representation learning. In ICLR, Cited by: §1, §3. [24] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al. (2015) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE TMI. Cited by: §3. [25] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach (2024) SDXL: improving latent diffusion models for high-resolution image synthesis. In ICLR, Cited by: §3. [26] E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or (2021) Encoding in style: a StyleGAN encoder for image-to-image translation. In CVPR, Cited by: §2, §5.3. [27] P. Rodríguez, M. Caccia, A. Lacoste, L. Zamparo, I. Laradji, L. Charlin, and D. Vazquez (2021) Beyond trivial counterfactual explanations with diverse valuable explanations. In ICCV, Cited by: §1. [28] E. H. Sánchez, M. Serrurier, and M. Ortner (2020) Learning disentangled representations via mutual information estimation. In ECCV, Cited by: §1. [29] B. Schölkopf and J. v. Kügelgen (2022) From Statistical to Causal Learning. In ICM, Cited by: §4. [30] K. A. Severson, S. Ghosh, and K. Ng (2019) Unsupervised learning with contrastive latent variable models. In AAAI, Cited by: §1. [31] K. Simonyan and A. Zisserman (2015) Very deep convolutional networks for large-scale image recognition. In ICLR, Cited by: §2. [32] S. Singla, B. Pollack, J. Chen, and K. Batmanghelich (2020) Explanation by progressive exaggeration. In ICLR, Cited by: §1. [33] Z. Wang, E. P. Simoncelli, and A. C. Bovik (2003) Multiscale structural similarity for image quality assessment. In Asilomar Conference on Signals, Systems and Computers, Cited by: §3. [34] E. Weinberger, N. Beebe-Wang, and S. Lee (2022) Moment matching deep contrastive latent variable models. In AISTATS, Cited by: §1, §3. [35] N. Weng, P. Pegios, E. Petersen, A. Feragen, and S. Bigdeli (2024) Fast diffusion-based counterfactuals for shortcut removal and generation. In ECCV, Cited by: §1, §3, §4, §5.3. [36] J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni (2023) MedMNIST v2: a large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data. Cited by: §3, §3. [37] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, Cited by: §2, §3. [38] K. Zou, S. Faisan, F. Heitz, and S. Valette (2022) Joint disentanglement of labels and their features with VAE. In ICIP, Cited by: §1. Supplementary Material 5 Architecture Details 5.1 CS Separator As described in Section 2, the CS separator Hcs,ϕH_cs,φ contains one common branch and one or two salient branches, depending on the contrastive analysis setting. In the background–target (BT) setting, it consists of the common branch CϕcC_ _c and the salient branch Sy,ϕsyS_y, _s_y. In the multiple-salient (MS) setting, an additional salient branch Sx,ϕsxS_x, _s_x is introduced to capture features specific to dataset X. All branches share the same architecture but have independent parameters. Following He et al. [10], we implement each branch as an MLP adapted from StyleGAN2’s Z-to-W mapping network [17]. Each branch comprises four linear layers with equalized learning-rate scaling, each followed by a LeakyReLU activation with a negative slope of 0.20.2. Unlike the original StyleGAN2 Z-to-W mapping network, in which each linear layer applies a single 512×512512× 512 weight matrix to a 512-dimensional latent vector, each linear layer in our separator is parameterized by a weight tensor (k)∈ℝL×512×512A^(k) ^L× 512× 512 to process the +W^+ code w=[w1,…,wL]⊤∈ℝL×512w=[w_1,…,w_L] ^L× 512. Here, k=1,…,4k=1,…,4 indexes the linear layer, and L denotes the number of style vectors in the +W^+ code. For a generator with a native output resolution of R×R× R, L is given by L=2log2(R)−2L=2 _2(R)-2. In our experiments, all StyleGAN2 generators are pretrained at a native resolution of 256×256256× 256, yielding L=14L=14. This design assigns a position-specific weight matrix to each style position. Since each layer preserves the dimensionality of the latent code, it maps from ℝB×L×512R^B× L× 512 to ℝB×L×512R^B× L× 512, where B denotes the batch size. 5.2 Regularization Networks D and R As described in Section 2, we employ a domain discriminator D and a dependency regressor R to regularize the common and salient representations. Given a common representation c∈ℝL×512c ^L× 512, the domain discriminator first flattens the complete representation into a single vector. Since our StyleGAN2 generator operates at a native resolution of 256×256256× 256, we have L=14L=14, resulting in a 7168-dimensional input vector. The discriminator is implemented as a binary linear classifier with a single fully connected layer mapping the 7168-dimensional input to one output logit. No hidden layer or output activation is used, and the network directly produces a single domain-classification logit. The dependency regressor R predicts the salient representation from the corresponding common representation. Given a batch of common representations c∈ℝB×L×512c ^B× L× 512, the batch and style dimensions are merged to form a matrix in ℝ(BL)×512R^(BL)× 512. The regressor consists of two fully connected layers that map this matrix from (BL)×512(BL)× 512 to (BL)×512(BL)× 512, with a ReLU activation between the two layers and no activation after the output layer. The output is then reshaped back to obtain the predicted salient factor s^∈ℝB×L×512 s ^B× L× 512. 5.3 Other Architectures and Implementations For the remaining network components, we use the pSp architecture [26] for E and the StyleGAN2 architecture [17] for G. The F-space refinement stage builds upon SFE [5], retaining the original architectures of its F-space encoder EfE_f and trainable adapter FAdptF_Adpt. For the original StyleGAN2 Z-to-W mapping network on which the separator architecture is based, we refer to the implementation provided at https://n.labml.ai/gan/stylegan/index.html. For the StyleGAN2 generator G, we use the PyTorch implementation by rosinality, available at https://github.com/rosinality/stylegan2-pytorch. The same pretrained G is used throughout both training stages and evaluation. To compute ℒadvL_adv, we use the discriminator pretrained jointly with G. For image-level evaluation, we adopt the U-Net-based classifier used in [35]. The classifier is trained on real images from the corresponding training set and subsequently applied to the edited images to assess whether they exhibit the intended target attributes. For latent-space evaluation, we quantify the separation between the learned common and salient factors by training a logistic regression classifier to predict the dataset label from each factor. The classifier is implemented in scikit-learn. Each factor is flattened and normalized before being provided to the classifier. 6 Additional Results Fig. 5 illustrates the effects of different training components on salient-factor swapping. With only the base reconstruction objectives (W/o Reg.), the swaps produce limited class-specific changes. Adding ℒDL_D improves tumor removal in the y→xy→ x direction but not tumor addition in the x→yx→ y direction. Adding ℒRL_R improves common–salient separation, enabling clearer bidirectional edits while better preserving the source anatomy. Finally, F-space refinement enhances local details and image fidelity without changing the semantic effect of the edit. These results are consistent with the quantitative ablation in Table 3. Fig. 6 compares the BT and MS assumptions on OCTMNIST. While both variants reconstruct the inputs well, the BT variant produces limited swapping effects because it cannot separately model the class-specific patterns of DME and drusen. The proposed MS variant, with two salient branches, enables more meaningful bidirectional swaps while preserving the common retinal structure. Figure 5: Qualitative ablation study of the training components on BraTS. The first column shows inputs x∈Xx∈ X and y∈Yy∈ Y from the healthy and tumor datasets, respectively. Columns 2–5 show the corresponding salient-factor swapping results under different training configurations. Figure 6: Comparison of the BT and MS assumptions on OCTMNIST. The input images x and y are sampled from the DME and drusen datasets, respectively. The remaining columns show the reconstruction and salient-factor swapping results under the BT and MS assumptions. Figs. 7–10 show additional reconstruction and salient-factor swapping results. Each pair of columns, separated by dashed lines, corresponds to an input pair (x,y)(x,y). From top to bottom, the rows show the real images, their reconstructions, and the swapping results. The swapped images are obtained by combining the common representation of each source image with the salient representation of the other image. Figs. 7–9 use the BT assumption, whereas Fig. 10 uses the MS assumption. Under the BT assumption, the salient factors of x is absent, whereas the MS assumption models separate salient factors for both x and y. Figure 11 provides additional interpolation examples on BraTS and OCTMNIST. Figure 7: Additional examples on BloodMNIST under the BT assumption. The input images x and y are sampled from the eosinophil and neutrophil datasets, respectively. Figure 8: Additional examples on BraTS under the BT assumption. The input images x and y are sampled from the healthy and tumor MRI scans, respectively. Figure 9: Additional examples on OCTMNIST under the BT assumption. The input images x and y are sampled from the normal and choroidal neovascularization (CNV) datasets, respectively. Figure 10: Additional examples on OCTMNIST under the MS assumption. The input images labeled x and y are sampled from the DME and drusen datasets, respectively. Figure 11: Additional interpolation results on BraTS and OCTMNIST. The upper two rows correspond to a healthy image x and a tumor image y from BraTS, and the lower two rows to a normal image x and a CNV image y from OCTMNIST. The first row of each pair interpolates from sxs_x to sys_y while preserving cxc_x, and the second interpolates from sys_y to sxs_x while preserving cyc_y.