Paper deep dive
On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures
Mohamed Amine Kina
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 3:59:07 AM
Summary
This paper investigates the failure of boundary-seeking knowledge distillation methods, specifically Contrastive Abductive Knowledge Extraction (CAKE), when applied to bottlenecked generative architectures like autoencoders. The authors demonstrate that CAKE, which works for classifiers by synthesizing samples near decision boundaries, fails for autoencoders because the shared low-dimensional latent bottleneck creates severe gradient conflicts when independently sampled contrastive targets are applied to coupled pixel-wise outputs. Experiments on MNIST show that CAKE variants produce poor reconstruction fidelity compared to a simple 'Single Noise Pass' baseline, which bypasses gradient conflicts by directly projecting noise through the teacher model.
Entities (8)
Relation Signals (6)
CaKE → designedfor → Classifier
confidence 97% · CAKE achieves this for classifiers by synthesizing samples near the teacher’s decision boundary.
CaKE → failsin → Autoencoder
confidence 95% · We show theoretically and empirically that CAKE fundamentally fails in autoencoders because the shared latent manifold strictly constrains optimization.
Autoencoder → hasconstraint → Latent Manifold
confidence 94% · The decoder acts as an array of tightly coupled, feature-level classifiers constrained by a shared low-dimensional bottleneck.
CaKE → causes → Gradient Conflict
confidence 92% · Independently sampling contrastive targets for these coupled outputs violates the geometry of the learned latent manifold and produces severe gradient conflicts
Single Noise Pass → outperforms → CaKE
confidence 90% · Direct noise propagation through the teacher decisively outperforms all other strategies with a Foreground mIoU of 0.1891.
Single Noise Pass → bypasses → Gradient Conflict
confidence 88% · Single Noise Pass avoids this entirely by using the teacher as a direct projector... without any gradient optimization
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Data-free knowledge distillation transfers the knowledge encoded in a teacher model to a student model without access to the original training data. Prior work such as Contrastive Abductive Knowledge Extraction (CAKE) achieves this for classifiers by synthesizing samples near the teacher's decision boundary. In this work, we investigate whether this boundary-seeking principle extends to autoencoder distillation through experiments on the MNIST dataset . To enable a direct comparison, we reformulate continuous reconstruction as a dense, per-feature classification task, allowing the decoder to output categorical logits. We show that boundary-seeking objectives are fundamentally ill-posed in bottlenecked generative architectures. CAKE operates on a single, instance-level objective, but a decoder acts as an array of tightly coupled, feature-level classifiers constrained by a shared low-dimensional bottleneck. Independently sampling contrastive targets for these coupled outputs violates the geometry of the learned latent manifold and produces severe gradient conflicts instead of informative boundary samples. Manifold-aware synthesis bypasses these conflicts entirely and establishes an effective baseline for data-free generative distillation.
Tags
Links
- Source: https://arxiv.org/abs/2607.15919v1
- Canonical: https://arxiv.org/abs/2607.15919v1
Trouble viewing inline? Open PDF directly →
Full Text
19,651 characters extracted from source content.
Expand or collapse full text
On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures Abstract Data-free knowledge distillation transfers the knowledge encoded in a teacher model to a student model without access to the original training data. Prior work such as Contrastive Abductive Knowledge Extraction (CAKE) achieves this for classifiers by synthesizing samples near the teacher’s decision boundary. In this work, we investigate whether this boundary-seeking principle extends to autoencoder distillation through experiments on the MNIST dataset . To enable a direct comparison, we reformulate continuous reconstruction as a dense, per-feature classification task, allowing the decoder to output categorical logits. We show that boundary-seeking objectives are fundamentally ill-posed in bottlenecked generative architectures. CAKE operates on a single, instance-level objective, but a decoder acts as an array of tightly coupled, feature-level classifiers constrained by a shared low-dimensional bottleneck. Independently sampling contrastive targets for these coupled outputs violates the geometry of the learned latent manifold and produces severe gradient conflicts instead of informative boundary samples. Manifold-aware synthesis bypasses these conflicts entirely and establishes an effective baseline for data-free generative distillation. 1 Introduction Knowledge distillation enables model compression, continual learning, and adaptation. However, traditional distillation assumes access to original training data, an assumption frequently violated due to privacy constraints or data unavailability (Liuetal2024). To address this, Contrastive Abductive Knowledge Extraction (CAKE) (Braun et al., 2024) enables data-free distillation by generating synthetic samples that probe a teacher’s decision boundary via contrastive diffusion. While data-free distillation is actively explored for generative models like masked and variational autoencoders (Bai et al., 2023; Ye and Bors, 2023), CAKE’s diffusion process fundamentally relies on a classification-based decision surface. This raises a natural question: can CAKE’s boundary-seeking principles extend to bottlenecked generative architectures? We investigate this by reformulating continuous reconstruction into a dense, per-feature classification task, directly applying CAKE to autoencoders. We make two contributions. First, we show theoretically and empirically that CAKE fundamentally fails in autoencoders because the shared latent manifold strictly constrains optimization. Second, we propose a single noise forward pass as an intuitive baseline for data-free generative distillation. 2 Background 2.1 Knowledge Distillation and CAKE Knowledge distillation (Hinton et al., 2015) transfers knowledge encoded in a teacher model fTf^T into a student model fSf^S. In supervised classification, the student minimises cross-entropy on ground-truth labels alongside a temperature-scaled matching loss on the teacher’s soft outputs. CAKE (Braun et al., 2024) achieves data-free extraction by targeting the teacher’s decision boundaries directly, termed abductive knowledge, rather than approximating the original distribution or relaxing class priors like other teacher-agnostic methods (Shin and Choi, 2024). CAKE generates contrastive sample pairs assigned to different classes, diffusing them toward opposing sides of the boundary. Stochastic noise then sweeps these samples along the surface, generating a diverse synthetic dataset capable of training a student classifier. 2.2 Autoencoders An autoencoder comprises an encoder qϕq_φ and a decoder pθp_θ. The encoder maps an input x to a low-dimensional latent representation =qϕ()z=q_φ(x), and the decoder reconstructs it as ^=pθ() x=p_θ(z). Training minimises reconstruction loss, typically mean squared error: ℒAE()=‖−pθ(qϕ())‖2L_AE(x)=\|x-p_θ(q_φ(x))\|^2 The bottleneck z forces the network to retain only essential structural information, effectively learning a manifold approximating the support of the training data distribution. 3 CAKE’s Inapplicability to Autoencoders 3.1 Adapting CAKE to the Autoencoder Setting Fig. 1: Qualitative comparison of student MNIST reconstructions across distillation strategies . While Standard and Bounded CAKE fail to recover recognizable digit geometry due to severe gradient conflicts, the Single Noise Pass strategy effectively preserves global structure and stroke boundaries. CAKE is naturally defined for classifiers, where the teacher maps an input to a single global class distribution. To test whether the same mechanism can be transferred to autoencoders, we reformulate the MNIST autoencoder as a pixel-wise classification model. Instead of reconstructing continuous grayscale values, each pixel is assigned to one of C intensity bins, corresponding to the background and foreground levels. The decoder therefore outputs: fθ()∈ℝN×Cf_θ(z) ^N× C where N is the total number of pixels, and each has an independent C-class logit vector. In our specific MNIST setup, N=784N=784 and C=4C=4. Given an input image x, the encoder produces a d-dimensional latent code (where d=12d=12 in our experiments): =gϕ()z=g_φ(x) This construction makes the autoencoder superficially compatible with CAKE: one can sample a target class map y and optimize a synthetic input x to probe the teacher’s decision boundaries. Following the standard CAKE formulation, the full extraction objective combines three components: a target classification loss, a contrastive loss to diffuse sample pairs toward the boundary, and a Total Variation (TV) prior: ℒ(,)=λclsℒcls(,)+λcontrℒcontr+λTVℒTV()L(x,y)= _clsL_cls(x,y)+ _contrL_contr+ _TVL_TV(x) To adapt this for the autoencoder, the classification component ℒclsL_cls must be evaluated densely across the output space: ℒcls(,)=∑j=1NCE(fθ,j(gϕ()),yj)L_cls(x,y)= _j=1^NCE (f_θ,j(g_φ(x)),y_j ) yj∈1,…,Cy_j∈\1,…,C\ is the independently sampled target class for pixel j. However, this objective reveals the central mismatch between CAKE and autoencoder distillation. In CAKE for standard classifiers, a synthetic sample is optimized toward one global class label. In the autoencoder case, a single d-dimensional bottleneck must satisfy N pixel-level labels simultaneously. This exposes the fundamental architectural flaw of applying boundary-seeking diffusion to dense generative models: independently sampling targets for highly coupled output features guarantees severe gradient conflicts across the shared latent manifold. 3.2 Latent Manifold Constraint The decoder does not represent all possible C-class images. Its outputs are constrained by the d-dimensional latent bottleneck learned from the training data. Thus, the set of possible reconstructions is a small structured subset of the full combinatorial output space: ℳθ=argmaxfθ():∈ℝd⊂1,…,CNM_θ= \ f_θ(z)\;:\;z ^d \⊂\1,…,C\^N When a contrastive distillation method like CAKE generates a target class map y by independently sampling pixel-wise classes, the resulting target is drawn from CNC^N possible combinations (e.g., 47844^784 combinations for our specific setup). Because these independent pixel targets are unconstrained by the underlying data distribution, they almost never correspond to a valid geometric structure. Consequently, they are overwhelmingly likely to fall outside the decoder’s learned manifold ℳθM_θ. 3.3 Gradient Conflict Under Random Pixel Targets The failure of CAKE in this setting follows directly from this coupling. When targets are sampled independently for all pixels, different pixels typically demand incompatible movements in the latent space. Applying the chain rule, the gradient of the dense classification objective with respect to the shared bottleneck is a sum of N terms: ∇ℒcls=∑j=1N∇CE(fθ,j(),yj) _zL_cls= _j=1^N _zCE (f_θ,j(z),y_j ) Each term attempts to move the same latent code toward a region where one specific pixel achieves its assigned target class. Since the targets are sampled independently, these directional vectors rarely agree. Satisfying the target for one pixel routinely moves the latent code away from the target region required by another pixel. Figure 2 visualizes this gradient conflict for two independent pixels (pixel550pixel_550 and pixel406pixel_406) using mathematically derived gradient vector fields. Because both pixels are conditioned on the same input, their shared latent embedding z (blue dot) occupies the exact same coordinate space. To satisfy the independently sampled target for pixel550pixel_550 (class 0), the gradient field demands a shift upward and to the left. Simultaneously, satisfying the target for pixel406pixel_406 (class 0) demands a contradictory shift to the right. Rather than discovering informative teacher boundaries, this severe gradient conflict tears the optimization in opposing directions, pushing the latent code toward unstable or off-manifold regions. This is not merely an artifact of the 2D PCA visualization. In the full 12-dimensional latent space, each pixel decision boundary is an 11-dimensional hypersurface. The same conflict remains: a single low-dimensional latent code is being asked to satisfy hundreds of independently sampled categorical constraints. The problem is therefore massively over-constrained. There are only 12 latent degrees of freedom, but 784 strictly enforced pixel-wise targets. 3.4 Why This Differs from Standard CAKE This structural mismatch explains why CAKE succeeds in settings such as the Two-Moons dataset or standard image classification, but does not transfer to autoencoders. In a standard classifier, the model maps an input to one unified, global label. CAKE only needs to move a synthetic sample toward a single decision boundary separating discrete global classes. Because the optimization is guided by a singular target, the high-dimensional input space provides more than enough degrees of freedom to satisfy it. The autoencoder’s decoder has the exact opposite structure. It maps a highly constrained, low-dimensional latent code to a high-dimensional, structured output. The target is not one class label, but an entire image-sized grid of independent pixel classes. Randomly assigning these pixel classes ignores the geometric correlations learned by the decoder: strokes are spatially coherent, neighboring pixels are highly dependent, and digit identity is represented globally. Consequently, almost all randomly generated target configurations represent off-manifold noise that cannot be produced by any latent code. Fig. 2: Gradient vector fields demonstrating latent coupling. The shared latent embedding z is pulled upward and to the left to satisfy the target for pixel550pixel_550 (left), but pulled to the right to satisfy pixel406pixel_406 (right). This opposing directional constraint illustrates why independently sampled targets fail in bottlenecked architectures. 4 Experimental Setup and Results To empirically validate the structural limitations of boundary-seeking optimization and evaluate alternative data-free extraction methods, we construct a controlled distillation pipeline. We first train a teacher autoencoder on the four-class pixel-wise MNIST task to full convergence. We then synthesize a transfer batch using various extraction strategies without accessing the original training data. Finally, a student autoencoder—architecturally identical to the teacher but with half the parameter count—is trained from scratch using the synthesized batch. Code to reproduce all experiments is available at (Github Repo) 4.1 Evaluation Metrics Because the autoencoder reconstruction is formulated as a dense classification task, standard image quality metrics (such as MSE or PSNR) are inapplicable. Instead, we evaluate the student’s fidelity to the teacher using strict categorical and spatial overlap metrics: • Overall Accuracy: The global proportion of pixels across the entire test set where the student’s predicted class perfectly matches the teacher’s prediction. • Foreground mIoU (F-mIoU): Our primary benchmark metric. It averages the Jaccard Index of the three foreground classes (c∈1,2,3c∈\1,2,3\), explicitly ignoring the massive class imbalance caused by the background: Foreground mIoU=13∑c=13|Pc∩Tc||Pc∪Tc|Foreground mIoU= 13 _c=1^3 |P_c∩ T_c||P_c∪ T_c| where PcP_c and TcT_c are the sets of predicted and target pixels for class c, respectively. While class-weighted accuracy and pixel-level F1 both handle class imbalance, neither captures exact geometric alignment as strictly as the Jaccard Index: accuracy under-penalizes structural hallucinations, and F1 is more forgiving of spatial misalignments. Since generative distillation demands precise reproduction of stroke boundaries and spatial coherence, Foreground mIoU is the most rigorous measure of structural fidelity. 4.2 Distillation Strategies We test five distinct strategies for synthesizing the transfer batch to systematically isolate the cause of optimization failure. 1. Standard CAKE: Following the exact implementation of the original CAKE paper, we initialize the synthetic batch X∼(0,I)X (0,I). The pixel-wise target map Y is sampled from a uniform distribution over the four classes. The input X is then optimized via gradient descent to match Y. 2. Bounded CAKE: A variant of the standard approach where the input initialization is restricted to a bounded Gaussian-like prior in the range [0,1][0,1]. This attempts to keep the initial noise closer to a standard image domain before optimization. 3. Shuffled Manifold Targets: To test whether the failure stems purely from unstructured targets, we initialize X∼(0,I)X (0,I) and perform a single forward pass through the teacher to generate a batch of valid, on-manifold output maps YbatchY_batch. We then shuffle these valid maps across the batch to use as optimization targets for X. 4. Shifted Manifold Targets: Similar to the shuffled approach, we generate valid output maps from the teacher. Instead of shuffling across the batch, we apply a spatial shift of two pixels to the output map. This preserves local stroke coherence while forcing the optimization to attempt a slight geometric translation. 5. Single Noise Pass: Diverging entirely from boundary-seeking optimization, this strategy performs zero gradient updates on the input space. We sample X∼(0,I)X (0,I) and simply pass it through the teacher autoencoder. The resulting output is directly used as the target for the student. 4.3 Results and Analysis Table 1 summarizes the performance of the student autoencoder across all five synthesis strategies. Table 1: Student Autoencoder Fidelity by Synthesis Strategy. Foreground mIoU is the primary benchmark for spatial feature learning. Synthesis Strategy Overall Acc. F-mIoU Standard CAKE 0.8149 0.0345 Bounded CAKE 0.7159 0.0789 Shuffled Targets 0.8178 0.0038 Shifted Targets 0.8174 0.0070 Single Noise Pass 0.8500 0.1891 Both CAKE variants fail to transfer geometric knowledge. Standard CAKE yields a near-zero Foreground mIoU of 0.0345, and bounding the initialization provides only marginal improvement (0.0789), leaving the fundamental gradient conflict unresolved. Counterintuitively, providing valid manifold targets (Experiments 3 and 4) produces the worst results overall. Though shuffled and shifted targets are geometrically coherent, they are arbitrarily paired with unrelated input noise, forcing the student to learn a fractured, discontinuous mapping. The high directional consistency of the resulting gradient signal (≈0.68≈ 0.68) actively drives the latent representation into a confident but geometrically mismatched state, causing catastrophic mode collapse. This shows that structured, mismatched targets are more destructive than pure noise. Direct noise propagation through the teacher decisively outperforms all other strategies with a Foreground mIoU of 0.1891. By passing input noise directly through the teacher without any gradient optimization, it sidesteps gradient conflicts entirely. Crucially, even off-distribution inputs produce outputs that preserve the teacher’s structural priors and learned Jacobian properties, giving the student smooth, functionally consistent training pairs rather than artificially decoupled ones. 5 Conclusion Boundary-seeking distillation methods like CAKE fail in bottlenecked generative architectures due to a fundamental structural incompatibility. A decoder’s outputs are tightly coupled through a shared low-dimensional bottleneck, so independently sampled contrastive targets produce severe gradient conflicts rather than informative boundary samples. This incompatibility is not merely empirical: it is a geometric inevitability arising from the mismatch between CAKE’s single-objective optimization and the massively overconstrained, feature-level coupling inherent to autoencoder decoders. Our experiments further reveal that geometrically valid but mismatched targets are worse than pure noise, actively driving the student into confident but structurally incorrect reconstructions through high-consistency, misdirected gradients. Single Noise Pass avoids this entirely by using the teacher as a direct projector: passing random noise through it yields on-manifold, functionally consistent training pairs without any gradient optimization, establishing a strong and surprisingly competitive baseline for data-free generative distillation. Promising directions include structured latent sampling strategies to improve transfer batch coverage, and extending the gradient conflict analysis to continuous VAE and diffusion-based architectures beyond the discrete MNIST setting explored here. References Y. Bai, Z. Wang, J. Xiao, C. Wei, H. Wang, A. Yuille, Y. Zhou, and C. Xie (2023) Masked autoencoders enable efficient knowledge distillers. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 24256–24265. External Links: Document Cited by: §1. S. Braun, M. Mundt, and K. Kersting (2024) Deep classifier mimicry without data access. Proceedings of Machine Learning Research 238, p. 4762–4770. External Links: Document Cited by: §1, §2.1. G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. arXiv. External Links: Document Cited by: §2.1. H. Shin and D. Choi (2024) Teacher as a lenient expert: teacher-agnostic data-free knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence 38, p. 14991–14999. External Links: Document Cited by: §2.1. F. Ye and A. G. Bors (2023) Continual variational autoencoder via continual generative knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence 37, p. 10918–10926. External Links: Document Cited by: §1.