Paper deep dive
DecoyFace: Beyond Obfuscation via Controllable and Imperceptible Identity Misdirection for Privacy-Preserving Face Recognition
Zhihan Ren, Lijun He, Xinyao Wang, Xinzhu Fu, Fan Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/21/2026, 5:07:06 AM
Summary
The paper introduces DecoyFace, a privacy-preserving face recognition framework that protects against feature inversion attacks and unauthorized reconstruction by honest-but-curious (HBC) servers. Unlike traditional methods that obfuscate features to produce distorted images, DecoyFace decomposes intermediate representations into a reconstruction-sensitive subspace and a complementary subspace. It injects decoy identity cues into the sensitive subspace while retaining true identity evidence in the complementary subspace. This steers unauthorized reconstructions toward a plausible but incorrect identity, reducing identity leakage while maintaining recognition accuracy for authorized users.
Entities (9)
Relation Signals (8)
Zhihan Ren → affiliatedwith → Xi'an Jiaotong University
confidence 95% · Zhihan Ren... are with... Xi’an Jiaotong University
DecoyFace → uses → Decoy-Guided Coherent Mixing
confidence 95% · we design a client-side Decoy-guided Coherent Mixing (DGCM) that injects decoy identity cues into the sensitive component
DecoyFace → uses → Authorized Canonicalization Module
confidence 95% · a server-side Authorized Canonicalization Module (ACM) that suppresses these cues for authorized verification
DecoyFace → uses → Reconstruction-Sensitive Subspace Decomposition
confidence 95% · we first decompose the transmitted feature into a reconstruction-sensitive subspace and its complementary subspace via Reconstruction-Sensitive Subspace Decomposition (RSSD)
DecoyFace → achievesfacevalidityof → 99.78%
confidence 90% · with over 99.78% face validity on LFW dataset
DecoyFace → mitigates → Honest-But-Curious Server
confidence 90% · This design addresses both attacker-side inversion from intercepted features and HBC server-side reconstruction from canonicalized representations
DecoyFace → reducesidentityleakageto → 2.93%
confidence 90% · substantially reducing identity leakage to 2.93% under U-Net attacks
DecoyFace → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Split face recognition reduces client-side computation but exposes intermediate features to feature inversion attacks and unauthorized analysis by honest-but-curious (HBC) servers. Existing privacy-preserving face recognition methods mainly aim to resist unauthorized reconstruction, typically producing features whose inversion yields visibly degraded results, which may reveal the existence of protection and motivate adaptive attacks. To address this issue, we propose DecoyFace, an imperceptible decoy-oriented framework that steers unauthorized reconstruction toward a plausible but incorrect identity while preserving recognition utility. The key idea is to decompose the intermediate representation into a reconstruction-sensitive subspace and its complementary subspace. The client injects decoy identity cues into the reconstruction-sensitive subspace, while limited recognition-relevant evidence from the true sample is retained in the complementary subspace. On the server side, an authorized canonicalization module suppresses decoy-dominant components and recovers a recognition-friendly representation. This design addresses both attacker-side inversion from intercepted features and HBC server-side reconstruction from canonicalized representations. Experiments show that DecoyFace preserves competitive recognition accuracy while substantially reducing identity leakage to 2.93% under U-Net attacks and 0.74% under Flow-Matching attacks while yielding visually plausible and imperceptible reconstructions, with over 99.78% face validity on LFW dataset.
Tags
Links
- Source: https://arxiv.org/abs/2607.17504v1
- Canonical: https://arxiv.org/abs/2607.17504v1
Trouble viewing inline? Open PDF directly →
Full Text
72,697 characters extracted from source content.
Expand or collapse full text
DecoyFace: Beyond Obfuscation via Controllable and Imperceptible Identity Misdirection for Privacy-Preserving Face Recognition Zhihan Ren, Lijun He, Xinyao Wang, Xinzhu Fu, Fan Li Zhihan Ren, Lijun He, Xinyao Wang, Xinzhu Fu, and Fan Li are with Shaanxi Key Laboratory of Deep Space Exploration Intelligent Information Technology, School of Information and Communications Engineering, Xi’an Jiaotong University, Xi’an 710049, China (e-mail: renzh@stu.xjtu.edu.cn, lijunhe@mail.xjtu.edu.cn, wangxy04@stu.xjtu.edu.cn, xinzhufu@stu.xjtu.edu.cn, lifan@mail.xjtu.edu.cn).This work is supported by the National Natural Science Foundation of China under Grant No. 62531012, the Sichuan Science and Technology Program under Grant 2026YFHZ0205, and the XJTU Research Fund for AI Science, No.2025YXYC004. (Corresponding author: Lijun He.) Abstract Split face recognition reduces client-side computation but exposes intermediate features to feature inversion attacks and unauthorized analysis by honest-but-curious (HBC) servers. Existing privacy-preserving face recognition methods mainly aim to resist unauthorized reconstruction, typically producing features whose inversion yields visibly degraded results, which may reveal the existence of protection and motivate adaptive attacks. To address this issue, we propose DecoyFace, an imperceptible decoy-oriented framework that steers unauthorized reconstruction toward a plausible but incorrect identity while preserving recognition utility. The key idea is to decompose the intermediate representation into a reconstruction-sensitive subspace and its complementary subspace. The client injects decoy identity cues into the reconstruction-sensitive subspace, while limited recognition-relevant evidence from the true sample is retained in the complementary subspace. On the server side, an authorized canonicalization module suppresses decoy-dominant components and recovers a recognition-friendly representation. This design addresses both attacker-side inversion from intercepted features and HBC server-side reconstruction from canonicalized representations. Experiments show that DecoyFace preserves competitive recognition accuracy while substantially reducing identity leakage to 2.93% under U-Net attacks and 0.74% under Flow-Matching attacks while yielding visually plausible and imperceptible reconstructions, with over 99.78% face validity on LFW dataset. I Introduction With the rapid development of deep learning (DL), face recognition (FR) systems have been widely deployed in applications such as mobile authentication, smart access control, and edge-assisted identity verification. In many real-world scenarios, edge devices have limited computational and memory resources, making it difficult to run a full FR model locally. A common solution is to adopt a client-server architecture that offloads compute-intensive stages to a cloud server Kang et al. (2017); Teerapittayanon et al. (2017). However, the transmission of intermediate representations in edge-cloud face recognition raises privacy concerns at both the communication and server computation stages. (1) On the communication side, a malicious attacker may eavesdrop on the intermediate feature transmitted from the edge device to the cloud server. The attacker then attempts feature inversion from the intercepted representation Singh et al. (2024); Li et al. (2023); Chen et al. (2024); Zhang et al. (2024); Lei et al. (2025); Ren et al. (2026). (2) On the cloud server side, the honest-but-curious (HBC) server follows the recognition protocol but may additionally inspect the received feature or recovered canonicalized representations for unauthorized reconstruction. Therefore, the desired defense should not only preserve recognition accuracy for authorized verification but also prevent both intercepted and server-observed representations from revealing the original identity. Figure 1: Comparison between existing PPFR methods (Top Row) and the proposed DecoyFace framework (Bottom Row). Existing PPFR methods primarily obfuscate transmitted features, producing distorted reconstructions that reveal the presence of protection. In contrast, DecoyFace steers unauthorized inversion toward a realistic but misleading decoy identity111The decoy face shown for illustration is cropped from this URL. . As summarized in Table I, prior studies can be categorized according to their operating domain and privacy objective. Image-domain approaches, such as deepfake defense methods Li et al. (2025); Wang et al. (2025b, c), perturb the input image to disrupt downstream misuse, while virtual-ID methods Yuan et al. (2022); Wang et al. (2024) directly synthesize a substitute identity. Nevertheless, our objective differs from virtual-ID generation methods. Rather than replacing the user identity with a new enrolled identity for downstream recognition, we preserve recognition for the true identity on the authorized path while misleading unauthorized inference on the leaked representation. In the frequency and feature domains, feature encryption Hsu et al. (2012); Ao and Boddeti (2025) seeks to make transmitted representations hard to exploit, and privacy-preserving face recognition (PPFR) methods Ji et al. (2022); Wang et al. (2023); Mi et al. (2022, 2024); Jin et al. (2024); Dai et al. (2025) mainly learn inversion-resistant features that preserve recognition utility while degrading visual recovery. However, many obfuscation-oriented PPFR methods treat failed or visibly distorted reconstructions as successful privacy outcomes. In summary, existing methods face two limitations: (1) Limited protection coverage: They mainly protect against feature leakage during wireless transmission, while overlooking unauthorized reconstruction performed by HBC servers from recovered representations; (2) Lacking Concealed Defense Capabilities: Visibly distorted reconstructions may reveal the presence of protection, thereby encouraging attackers to refine their inversion models or adopt stronger attacks. It is necessary to redirect unauthorized reconstruction toward a plausible but incorrect identity, making the privacy protection less perceptible and harder to adapt against. TABLE I: Comparison of different privacy protection directions for FR. A denotes the original identity, while B denotes the decoy identity. Category Domain Privacy objective Authorized utility Attacker outcome No protection None None Preserved Original ID A Deepfake defense Li et al. (2025); Wang et al. (2025b, c) Image Perturbed image N/A Not reliably A Virtual ID Yuan et al. (2022); Wang et al. (2024) Image Generate virtual ID Preserved for virtual ID Virtual ID Feature encryption Hsu et al. (2012); Ao and Boddeti (2025) Feature Encrypted features Preserved Hard to exploit PPFR Wang et al. (2023); Mi et al. (2022); Jin et al. (2024); Dai et al. (2025) Frequency / Feature Inversion-resistant features Preserved Distorted recovery Ours Feature Misdirection to decoy ID B Preserved Reconstructed as decoy ID B To address these limitations, we propose DecoyFace, an imperceptible decoy-oriented privacy-preserving framework for face recognition. Unlike existing obfuscation-oriented PPFR methods, DecoyFace preserves authorized recognition utility while misleading unauthorized reconstruction toward a plausible but incorrect decoy identity. This shift from reconstruction failure to identity misdirection distinguishes our method from existing privacy-preserving approaches and provides a new perspective for secure feature transmission in edge-cloud FR. More importantly, we consider not only attacker-side feature inversion from intercepted features, but also unauthorized reconstruction by an HBC server from the server-side canonicalization representation. This broader threat model better reflects the privacy risks of practical split face recognition systems. Fig. 1 illustrates the conceptual difference between existing obfuscation-oriented PPFR methods and the proposed decoy-based misdirection framework. The key idea is to exploit the asymmetry between unauthorized reconstruction and authorized recognition in the intermediate feature space. Specifically, we first decompose the transmitted feature into a reconstruction-sensitive subspace and its complementary subspace via Reconstruction-Sensitive Subspace Decomposition (RSSD). The former captures the feature directions that are most influential for attacker-side inversion, while the latter serves as a carrier of recognition-relevant evidence in a form less susceptible to reconstruction. Based on this decomposition, we design a client-side Decoy-guided Coherent Mixing (DGCM) that injects decoy identity cues into the sensitive component, and a server-side Authorized Canonicalization Module (ACM) that suppresses these cues for authorized verification. By treating leaked features as opportunities for controlled deception, DecoyFace induces attackers to reconstruct a realistic yet incorrect identity. The main contributions are summarized as follows: 1. Any Where: leakage protection across wireless transmission and HBC servers. Unlike existing privacy-preserving FR methods that mainly defend against feature leakage during wireless transmission, DecoyFace jointly protects both the communication side and the HBC server side, where unauthorized reconstruction may be performed from recovered representations. 2. Any Way: imperceptible resistance to both machine-vision and human inspection. We develop a reconstruction-sensitive privacy protection mechanism, which decomposes the feature into reconstruction-sensitive and insensitive subspaces, injects decoy cues into the sensitive component, and enables authorized recognition while avoiding perceptible distortions that may expose the presence of protection. 3. Any Who: identity-controllable misdirection. We introduce a decoy-oriented privacy objective that steers unauthorized reconstruction toward a plausible but incorrect identity, instead of degrading reconstruction quality. This design reduces true-identity leakage while preserving facial plausibility in unauthorized outputs, thus enabling stealthy identity misdirection. I Related Work I-A Face Reconstruction Attacks Face reconstruction attacks aim to recover private visual content from transmitted or stored representations. Recent studies have shown that intermediate features in split inference can leak rich visual and identity-related information, making them vulnerable to feature inversion. Singh et al. Singh et al. (2024) benchmarked split inference under different split points and adversarial knowledge settings, showing that leaked features can be reconstructed with nontrivial fidelity. Benefiting from the growth of generative models in image synthesis and reconstruction Song and Ermon (2019); Ren et al. (2023); He et al. (2024); Ren et al. (2024); Wang et al. (2025a); Bai et al. (2026), subsequent work has strengthened attackers by introducing stronger generative priors. GLASS Li et al. (2023) proposed GAN-based attacks that search in a pretrained latent space to improve the realism and stability of recovered images. More recent approaches further exploit diffusion Song and Ermon (2019); Song et al. (2020) or flow-based Lipman et al. (2022) priors. DRAG Lei et al. (2025) used guided diffusion to reconstruct higher-fidelity images from intermediate representations. FIA-Flow Ren et al. (2026) studied data-efficient black-box inversion and showed that semantically faithful recovery is possible even without access to the victim model’s internal parameters. These results indicate that intermediate features are sufficient to recover identifiable faces in split face recognition. I-B Privacy-Preserving Face Recognition Existing PPFR methods can be broadly grouped into three lines. The first line relies on cryptographic protection, such as homomorphic-encryption-based matching or end-to-end encrypted face recognition, which offers strong confidentiality guarantees but often introduces substantial computational overhead and deployment complexity Hsu et al. (2012); Ao and Boddeti (2025). The second line operates in the image domain. Deepfake defense Li et al. (2025); Wang et al. (2025b, c) and identity-cloaking methods perturb the original face image to prevent malicious reuse, while virtual-ID or cancelable-face methods generate substitute identities for downstream recognition. These methods are valuable for image sharing and media protection, but they do not directly address privacy leakage from intermediate features during split inference. Moreover, virtual-ID methods Yuan et al. (2022); Wang et al. (2024) usually preserve utility for the substitute identity rather than the true enrolled identity. The third and most relevant line is representation-domain PPFR methods, which attempt to preserve recognition utility while suppressing unauthorized reconstruction from transmitted features. AdvFace Wang et al. (2023) learns privacy-preserving adversarial facial features as a plug-in defense against unknown reconstruction models. MinusFace Mi et al. (2024) removes reconstruction-friendly cues via trainable feature subtraction and randomized transformation. FaceObfuscator Jin et al. (2024) further constructs gradient-descent-resistant obfuscated features, while FracFace Dai et al. (2025) disrupts spatial regularities in the frequency domain to weaken visual cues exploitable by generative reconstruction. However, most existing PPFR methods aim to make unauthorized reconstruction fail or appear visually distorted. By contrast, DecoyFace adopts a misdirection-oriented design that aims to preserve verification utility for the server while steering unauthorized reconstruction toward a plausible but incorrect identity. I Preliminary Figure 2: Overview of the proposed DecoyFace framework. The edge device decomposes the intermediate representation, injects decoy cues into the reconstruction-sensitive component, and constructs a protected complementary carrier via DGCM. The cloud server then suppresses the decoy-dominant component and canonicalizes the received feature for authorized face recognition. I-A Threat Model I-A1 Attacker’s Scenarios and Knowledge In this work, we consider a threat model involving both a malicious attacker and an honest-but-curious server. The malicious attacker can intercept transmitted features during edge-cloud communication. The client encoder FcF_c maps a face image x to an intermediate feature z=Fc(x)z=F_c(x) and transmits z to a cloud server FsF_s for identity verification. The attacker is assumed to have black-box query access to the client model. This assumption follows recent feature inversion studies and reflects practical queryable edge clients Ren et al. (2026); Zhang et al. (2024); Chen et al. (2024). It can submit inputs xix_i and obtain the corresponding output features ziz_i, but cannot access the model architecture or parameters. In addition, we consider an HBC server that follows the normal recognition protocol but may additionally analyze the received protected features and the server-side recovered representations for unauthorized privacy inference. This models a stronger and more realistic privacy risk in deployed split FR systems, where the server is trusted for utility but not necessarily for privacy. I-A2 Attacker’s Strategy Using queried image-feature pairs xi,zi\x_i,z_i\, the attacker trains a reconstruction model to reconstruct face images from intercepted features. In particular, the HBC server may apply the server-side projection and canonicalization used for authorized recognition and then attempt reconstruction from that de-randomized representation. I-B Defender Objective Existing PPFR methods mainly aim to transmit a protected feature z~ z that preserves recognition utility while reducing reconstruction fidelity, so that reconstructions from z~ z become visually distorted. In contrast, our goal is to reduce true-identity leakage while maintaining the facial plausibility of unauthorized outputs. This objective is more targeted than merely degrading reconstruction quality, because it steers unauthorized reconstructions toward a plausible yet incorrect identity rather than merely revealing that protection has been applied. Accordingly, the defender should satisfy the following three requirements simultaneously: 1. Authorized utility: The protected feature should remain discriminative for the true identity under the authorized recognition pipeline. 2. Attacker-side privacy: The protected feature should reduce similarity to the original image and suppress identity leakage while maintaining facial plausibility in unauthorized reconstruction. 3. Server-side privacy: Even after server-side authorized processing, the HBC server should not enable faithful reconstruction of the original face from the recovered feature. IV Method IV-A Overview of the Proposed Framework Given a face image xAx_A of identity A, the client encoder extracts an intermediate feature zA=Fc(xA)z_A=F_c(x_A). DecoyFace transforms zAz_A into a protected representation z~ z that preserves utility for authorized verification while biasing unauthorized reconstruction toward a decoy identity. As shown in Fig. 2, Reconstruction-Sensitive Subspace Decomposition (RSSD) separates the feature space into a reconstruction-sensitive subspace and a complementary subspace. Based on this decomposition, Decoy-Guided Coherent Mixing (DGCM) injects decoy identity cues into the sensitive subspace while preserving limited true-identity evidence in the complementary component. Finally, the Authorized Canonicalization Module (ACM) provides a constrained recovery path that supports authorized verification without fully restoring the original protected feature. Overall, DecoyFace redirects leakage by injecting decoy cues into reconstruction-sensitive directions while retaining limited recognition-relevant evidence in the complementary subspace. The resulting representation is easier for the authorized server to canonicalize than for an attacker to exploit directly. This asymmetry is the basis of the proposed client-server PPFR design. Figure 3: Construction and illustration of the reconstruction-sensitive subspace. Perturbations restricted to the RSS induce substantially larger reconstruction changes than perturbations of equal magnitude applied to the complementary subspace. IV-B Reconstruction-Sensitive Subspace Decomposition The transmitted feature serves two competing purposes: it must preserve identity information for recognition, yet it also exposes information that can be exploited for unauthorized reconstruction. Direct perturbation does not distinguish directions that primarily support recognition from those that primarily support reconstruction, and therefore makes it difficult to inject decoy cues in a controlled manner. To address this problem, we explicitly decompose the intermediate feature space into a reconstruction-sensitive subspace (RSS) and a complementary subspace, enabling the two roles to be manipulated separately. As shown in Fig. 3, we estimate the subspace using an offline proxy inversion network Φ(⋅) (·), whose architecture need not match the attacker used in the evaluation. For each input image xix_i, the client encoder produces the intermediate feature zi=Fc(xi)∈ℝC×H×Wz_i=F_c(x_i) ^C× H× W, and the proxy attacker reconstructs x^i=Φ(zi) x_i= (z_i). We define the reconstruction loss as ℒrec=ℓ1(x^i,xi).L_rec= _1( x_i,x_i). To measure reconstruction sensitivity with respect to the intermediate feature, we compute the gradient of the reconstruction loss gi=∇ziℒrecg_i= _z_iL_rec. Figure 4: Detailed design of Decoy-Guided Coherent Mixing (DGCM). DGCM first applies channel-shuffled grouping (CSG) to partition complementary features into groups, followed by Instance Norm (IN) to normalize group-wise statistics. A routing mask preserves a small subset of true groups, while the remaining groups are replaced by coherent or independently sampled decoy groups. Inv-CSG maps the mixed groups back to the original feature layout to form the protected complementary carrier. Intuitively, feature directions whose perturbations consistently induce large reconstruction changes across samples are more likely to encode recoverable facial structure. Therefore, we identify the RSS from the dominant eigensubspace of the gradient covariance. g¯i[c]=1HW∑h=1H∑w=1Wgi[c,h,w],μg=1N∑i=1Ng¯i, g_i[c]= 1HW _h=1^H _w=1^Wg_i[c,h,w], _g= 1N _i=1^N g_i, (1) Σ=1N−1∑i=1N(g¯i−μg)(g¯i−μg)⊤∈ℝC×C. = 1N-1 _i=1^N( g_i- _g)( g_i- _g) ^C× C. (2) The dominant eigenvectors correspond to feature perturbations that consistently produce large reconstruction variations across samples, and are treated as reconstruction-sensitive. We then perform eigendecomposition of Σ=UΛU⊤ =U U . Let UrecU_rec denote the top-r eigenvectors. These vectors define the dominant RSS, and the corresponding projection matrix is Prec=UrecUrec⊤∈ℝC×CP_rec=U_recU_rec ^C× C. Its complementary projection is defined as PN=I−PrecP_N=I-P_rec. Accordingly, any intermediate feature z can be decomposed as z=Precz+PNz,z=P_recz+P_Nz, (3) where PreczP_recz denotes the component that is most influential for reconstruction. As shown in Fig. 3, perturbations restricted to the RSS produce substantially larger reconstruction changes than perturbations of the same magnitude applied to the complementary subspace. By contrast, perturbations of comparable magnitude in the insensitive subspace, Φ(zi+PNϵ) (z_i+P_Nε), have a much weaker visual effect. This observation motivates using PrecP_rec as the primary subspace for decoy injection, and treating PNP_N as the complementary carrier for preserving recognition-relevant information on the authorized path. It also enables a matched recovery rule on the server side, where the decoy-dominant component can be suppressed through projection and subsequent canonicalization. This decomposition provides the structural basis for the subsequent asymmetric design: decoy cues are injected into the RSS, whereas the complementary subspace is used as a constrained carrier for recognition. IV-C Client-side Protected Feature Generation Given the RSSD, the next question is how to construct a protected feature that remains discriminative for the authorized server while redirecting unauthorized inversion toward a decoy identity. Replacing the sensitive component is insufficient because the complementary branch would remain unchanged. An unchanged complementary branch may still preserve excessive true-identity information and may also create statistical inconsistency between the two branches, thereby weakening controllability under both inversion and server-side recovery. Therefore, we design a structured client-side mixing process that retains only a limited amount of true-identity evidence while replacing the remaining complementary groups with auxiliary complementary components from decoy samples. To reduce fragmentation among independently replaced groups, we further design a coherent-subset group mixing policy, under which a subset of the replaced groups shares a small number of auxiliary sources. Let xAx_A be an input image from the true identity A, and let yBy_B be a decoy sample from another identity ℬB. Their encoder features are zA=Fc(xA),zB=Fc(yB).z_A=F_c(x_A), z_B=F_c(y_B). We decompose them into reconstruction-sensitive and insensitive components: rB=PreczB,nA=PNzA,r_B=P_recz_B, n_A=P_Nz_A, (4) where rBr_B denotes the reconstruction-sensitive component extracted from identity ℬB and nAn_A denotes the complementary component of the true sample A. Meanwhile, we maintain an auxiliary decoy sample set yaux=yjj=1My_aux=\y_j\_j=1^M, which is not tied to the current target decoy identity B. The corresponding auxiliary decoy feature bank N=njj=1MD_N=\n_j\_j=1^M is defined as nj=PNFc(yj),nj∈ℝC×H×W,n_j=P_NF_c(y_j), n_j ^C× H× W, (5) where each njn_j is extracted from an auxiliary sample yjy_j and projected onto the reconstruction-insensitive subspace. The auxiliary components serve both to fill the suppressed complementary groups and to reduce the risk that the server reconstructs a clean complementary branch associated with the true identity. The detailed design of DGCM is illustrated in Fig. 4. We operate on the complementary branch at the group level. The key idea is to retain only a limited subset of complementary groups from the true sample and to replace the remaining groups with auxiliary complementary groups drawn from ND_N. This design controls the amount of true-sample evidence exposed to the server while making the transmitted complementary branch less cleanly attributable to the true sample. Specifically, a channel-shuffled grouping operator (CSG) Γ(⋅) (·) partitions the feature into G groups: Γ:ℝ :R →C×H×W∏g=1Gℝcg×H×W,∑g=1Gcg=C, ^C× H× W→ _g=1^GR^c_g× H× W, _g=1^Gc_g=C, (6) uA u_A =Γ(nA)=(uA,1,…,uA,G), = (n_A)=(u_A,1,…,u_A,G), vj v_j =Γ(nj)=(vj,1,…,vj,G). = (n_j)=(v_j,1,…,v_j,G). To control how much true-sample evidence is retained, we randomly select K complementary groups from nAn_A, yielding a binary routing mask m∈0,1Gm∈\0,1\^G with ‖m‖0=K\|m\|_0=K. Groups with mg=1m_g=1 are preserved from nAn_A, while the remaining suppressed groups with mg=0m_g=0 are replaced by auxiliary groups from ND_N. A straightforward replacement strategy is to sample the source of each suppressed group independently. However, fully independent replacement tends to produce a fragmented complementary branch, in which replaced groups originate from unrelated decoy samples and thus exhibit inconsistent cross-group statistics, which may be exploited by the attacker. To alleviate this issue, we introduce a coherent-subset group mixing policy that divides the replaced groups into coherent and independently replaced subsets. Let 0=g∈1,…,G:mg=0G_0=\g∈\1,…,G\:m_g=0\ denote the set of suppressed groups. We uniformly sample a coherent subset ∼Unif(S⊆0:|S|=sc),S (\S _0:|S|=s_c\ ), (7) where sc≤|0|s_c≤|G_0|. The remaining groups 0∖G_0 are independently replaced. We further sample a small shared source index set ∼Unif(J⊆1,…,M:|J|=L),J (\J \1,…,M\:|J|=L\ ), (8) where L≤scL≤ s_c and L≪ML M. Here, S specifies which suppressed groups use coherent replacement, while J provides the shared pool of auxiliary source indices available to these groups. Conditioned on S and J, the source index for each suppressed group is sampled as ρg∼Unif(),g∈,Unif(1,…,M),g∈0∖. _g casesUnif(J),&g ,\\ Unif(\1,…,M\),&g _0 . cases (9) In this way, only a subset of the replaced groups exhibits controlled source sharing. This encourages higher cross-group source consistency in the replaced branch, while keeping the number of preserved true-sample groups unchanged. Before mixing, we normalize each group independently to reduce source-dependent magnitude mismatch: u^A,g=IN(uA,g),v^j,g=IN(vj,g), u_A,g=IN\! (u_A,g ), v_j,g=IN\! (v_j,g ), (10) where IN(⋅)IN(·) denotes the InstanceNorm Ulyanov et al. (2016) operator. The mixed complementary feature is constructed group-wise as u~g=u^A,g,mg=1,v^ρg,g,mg=0, u_g= cases u_A,g,&m_g=1,\\ v_ _g,g,&m_g=0, cases (11) Finally, we apply the inverse grouping operator to recover the mixed complementary component: n~=Γ−1(u~1,…,u~G). n= ^-1( u_1,…, u_G). (12) This design improves local consistency in the mixed complementary branch without increasing the amount of retained true-sample evidence. Specifically, the amount of retained true-sample information is still determined by the K routed true groups selected by m. The coherent-subset group mixing policy only reorganizes the replaced groups and never introduces additional information from nAn_A. The complementary branch is not a clean residual of the true sample, but a constrained carrier that preserves only limited true-identity evidence while exhibiting more stable cross-group statistics than fully independent replacement. The final protected feature is constructed as z~=n~+rB. z= n+r_B. (13) The reconstruction-sensitive branch is dominated by decoy identity cues, whereas the complementary branch contains only sparse routed evidence from the true sample, together with auxiliary complementary content from other samples. Importantly, the main decoy feature rBr_B is sampled once per protected view. In contrast, the complementary replacement groups are drawn from the auxiliary bank and are not tied to the identity of ℬB. Even if the server can suppress the decoy, the remaining complementary branch does not coincide with the original nA=PNzAn_A=P_Nz_A. Before transmission, we apply a sample-wise random global scaling transform to the protected feature z~ z. Specifically, we sample an integer exponent k∈−t,−t+1,…,t−1,tk∈\-t,-t+1,…,t-1,t\, set s=2ks=2^k, and transmit zt=sz~z_t=s z. The scaling factor is not revealed to the server and is instead handled implicitly by server-side canonicalization. The server-side ACM is designed to absorb this magnitude variation through matched canonicalization. Consequently, the transmitted feature is no longer a simple perturbation of the true sample, but a structured mixture whose leakage is biased toward the decoy identity and whose complementary branch is intentionally constructed to be only partially recoverable on the authorized server side. IV-D Server-side Recognition After receiving the protected feature ztz_t, the server performs a matched recovery procedure that suppresses the reconstruction-sensitive decoy component and restores a representation more suitable for downstream recognition. Since the decoy cue is injected primarily through the RSS, the server first suppresses this decoy-dominant component by projecting the received feature onto the reconstruction-insensitive subspace: zs z_s =PNzt=sPN(n~+rB) =P_Nz_t=sP_N( n+r_B) (14) =sPNn~+sPNPreczB=sPNn~, =sP_N n+sP_NP_recz_B=sP_N n, This cancellation holds because rB∈range(Prec)r_B (P_rec), and PNPrec=0P_NP_rec=0. This projection suppresses the RSS component that contains the explicitly injected decoy cues and retains only the projected complementary carrier, still scaled by the unknown factor s. As a result, zsz_s is expected to preserve the recognition-relevant evidence from the routed complementary groups, while discarding the decoy-dominant component injected into the RSS. To reduce the magnitude perturbation introduced during transmission, the server further performs group-wise canonicalization on zsz_s. We adopt quantile-based calibration instead of normalization because the former is less sensitive to outlier activations introduced by group replacement and feature mixing. We partition zsz_s into G groups and rescale each group independently according to its q-quantile magnitude: zcal(g)=τsQq(|zs(g)|)+ϵzs(g),g=1,…,G,z_cal^(g)= _sQ_q(|z_s^(g)|)+ε\,z_s^(g), g=1,…,G, (15) where zs(g)z_s^(g) denotes the g-th feature group, Qq(⋅)Q_q(·) denotes the q-quantile computed over the absolute values of all elements in the group, τs _s is the target server-side magnitude, and ξ is a small constant for numerical stability. After calibrating all groups, we reassemble them to obtain the calibrated feature ncal=Γ−1(zcal(1),…,zcal(G))n_cal= ^-1(z^(1)_cal,…,z^(G)_cal). The server then forms a canonicalized recognition feature by combining the calibrated complementary component with a fixed anchor in the RSS: zcan=μrec+ncal,z_can= _rec+n_cal, (16) where μrec _rec is defined as the mean reconstruction-sensitive component of the decoy bank μrec=1M∑j=1MPrecFc(yj), _rec= 1M _j=1^MP_recF_c(y_j), (17) and serves as a fixed reference in the PrecP_rec subspace, thereby reducing distribution shift after the decoy-dominant component has been removed. The canonicalized feature zcanz_can is finally fed into the server backbone for recognition: e=Fs(zcan),p=Harc(e),e=F_s(z_can), p=H_arc(e), (18) where HarcH_arc denotes the ArcFace classification head Deng et al. (2019). From the attacker’s perspective, the transmitted feature is dominated by reconstruction-sensitive decoy cues and thus tends to produce a plausible but incorrect face under inversion. In contrast, the authorized server applies the matched projection and scale calibration to recover a more stable recognition feature for downstream authentication. IV-E Training Objective The client-side protection process is stochastic because the routing mask, complementary replacement, and transport-time scaling vary across protected views. Identity supervision alone is insufficient because it may preserve class separability while still permitting large view-dependent variation across protected samples of the same input. The server backbone FsF_s and the ArcFace classification head HarcH_arc are trained jointly on the recovered features. During training, we generate V protected views for each sample using different client-side randomization, and recover their corresponding server-side features zcanvv=1V\z^v_can\_v=1^V. The overall training objective is ℒ=ℒid+λconsℒcons.L=L_id+ _consL_cons. (19) For identity supervision, we apply the ArcFace classification loss on the canonicalized feature: ℒid=1V∑v=1VCE(Harc(Fs(zcanv)),yi).L_id= 1V _v=1^VCE (H_arc(F_s(z^v_can)),\,y_i ). (20) where CE(⋅)CE(·) denotes the cross-entropy loss, and yiy_i is the ground-truth identity label of the training image xix_i. This term encourages the recovered representation to remain discriminative for the true identity after client-side protection and server-side recovery. To reduce variation across protected views, we further impose a multi-view embedding consistency loss. Let e^v=Fs(zcanv)‖Fs(zcanv)‖2 e^v= F_s(z^v_can)\|F_s(z^v_can)\|_2 (21) denote the normalized embedding of the v-th protected view. The consistency loss is defined as ℒcons=2V(V−1)∑1≤u<v≤V‖e^u−e^v‖22.L_cons= 2V(V-1) _1≤ u<v≤ V \| e^u- e^v \|_2^2. (22) This term stabilizes the server-side embedding against stochastic client-side protection, which is important because the same input may generate multiple protected views with different routed groups, auxiliary assignments, and transport-time scales. V Experiments V-A Datasets and Metrics We use MS1Mv2 Guo et al. (2016) as the training set and evaluate on LFW Huang et al. (2008), AgeDB Moschoglou et al. (2017), CFP-FP Sengupta et al. (2016), CALFW Zheng et al. (2017), CPLFW Zheng and Deng (2018), IJB-B Whitelam et al. (2017), and IJB-C Maze et al. (2018). We report TAR@FAR=10−410^-4 on IJB-B/IJB-C and verification accuracy (Acc) on the other datasets. We evaluate the method on authorized recognition and privacy protection against both malicious attackers and the HBC server. The experiments follow the edge-cloud split FR setting, where the client transmits intermediate features to the server for identity verification. Unlike conventional PPFR evaluations that primarily degrade reconstruction quality, our evaluation also examines whether the reconstructed results are redirected toward a decoy identity while preserving utility for the authorized server. Cosine similarity (COS) in the identity embedding space is used to assess identity consistency. In addition, to better reflect the decoy-oriented goal of our method, we use three identity-related metrics to evaluate reconstructed results. Let xix_i be the original image with identity label yiy_i, and let x^i x_i be the reconstructed image. We first apply a face detector d(⋅)d(·) to x^i x_i and use its maximum detection confidence as the face-validity score. Given a threshold θf _f, the valid reconstruction set and the face validity ratio (FVR) are defined as =i∣d(x^i)>θf,FVR=||N.V=\i d( x_i)> _f\, = |V|N. (23) For identity evaluation, we use an independent face recognition model ϕ(⋅)φ(·), with all embeddings normalized to unit length. Let T denote the closed-set identity gallery, and let ct_c be the identity template of class c∈c . For each valid reconstruction, the predicted identity is obtained by top-1 nearest-template matching: y^i=argmaxc∈ϕ(x^i)⊤c. y_i= _c φ( x_i) t_c. (24) Thus, a reconstruction is regarded as leaking the original identity if y^i=yi y_i=y_i, and as being redirected to another identity if y^i≠yi y_i≠ y_i. Let yidy_i^d denote the target decoy identity assigned to sample i. We define =i∈∣y^i=yi, =\i y_i=y_i\, (25) ℛ =i∈∣y^i≠yi, =\i y_i≠ y_i\, =i∈∣y^i=yid. =\i y_i=y_i^d\. Here, O denotes original-identity leakage, ℛR denotes redirection to any non-original identity, and ⊆ℛD denotes successful redirection to the target decoy identity. The Identity Leakage Ratio (ILR), Identity Redirection Ratio (IRR), and Decoy Hit Ratio (DHR) metrics are defined as ILR=||||,IRR=|ℛ|||,DHR=||||.ILR= |O||V|, = |R||V|, = |D||V|. (26) When ||=0|V|=0, the identity-related metrics are undefined and reported as N/A. Since existing baselines do not define a target decoy identity, DHR is only reported in the ablation studies. TABLE I: Comparison of authorized face verification accuracy on standard benchmarks and overall privacy outcome. Bold denotes the best result in each column. Method LFW AgeDB CFP-FP CALFW CPLFW IJB-B IJB-C Privacy Protected Imperceptible DCTDP 99.68 97.42 95.26 95.65 90.05 93.70 95.46 × × DuetFace 99.82 97.83 97.87 95.92 92.37 94.00 95.51 × × Cloak 99.52 95.80 96.54 95.12 90.80 79.95 81.66 × × AdvFace 99.35 94.85 93.89 94.77 89.68 75.22 82.37 × × MinusFace 99.67 96.13 95.03 95.38 90.18 93.37 94.70 ✓ × FaceObfuscator 99.08 95.72 94.66 94.73 90.23 89.19 91.46 ✓ × FracFace 99.60 96.08 94.26 95.42 88.18 85.57 87.34 ✓ × 99.77 97.20 95.54 95.63 91.15 91.10 93.35 ✓ ✓ Figure 5: Qualitative reconstruction results under U-Net and Flow-Matching attackers. DecoyFace produces plausible reconstructions that are visually closer to the decoy identity than to the original identity. V-B Implementation Details We adopt IR-50 He et al. (2016) as the backbone and split it into a client-side encoder FcF_c and a server-side recognizer FsF_s after the second residual stage, resulting in an intermediate feature of size 64×56×5664× 56× 56. For the RSSD, the complementary projection is constructed based on a reconstruction-sensitive subspace with rank r=32r=32. The complementary branch is partitioned group-wise with group size 4, yielding G=16G=16 groups in total, among which K=3K=3 groups preserve routed carrier information from the true sample while the remaining groups are replaced using auxiliary complementary decoy features from DND_N. The auxiliary complementary bank size is set to M=5000M=5000, which is sampled from CelebA Liu et al. (2015). For the coherent-subset policy, sc∼U[8,12]s_c U[8,12], suppressed groups are reassigned to share L∼U[2,4]L U[2,4] auxiliary sources, while the remaining suppressed groups keep their independent assignments. To obtain the RSS, we use a U-Net Ronneberger et al. (2015) proxy inversion model trained with 30,00030,000 iterations on CelebA. For transport-time perturbation, we adopt the scale-only variant of the global transformation, where the exponent k is sampled from [−3,3][-3,3]. For server-side canonicalization, we use the group-wise quantile normalizer, with q=0.95q=0.95 and τs=0.8 _s=0.8. For defender training, we use SGD with momentum 0.9, weight decay 5×10−45× 10^-4. The initial learning rate is 0.1, and it decays by a factor of 0.1 at epochs 4, 8, and 12. The batch size is 256, and training runs for 15 epochs. Following the multi-view formulation in Section IV, we generate two protected views for each training sample, and λcons _cons is 0.1. For the attacker, we use a U-Net-based network Ronneberger et al. (2015) and a Flow-Matching-based (FM-based) network Ren et al. (2026). Both attackers are trained on CelebA Liu et al. (2015) for 100,000 iterations. TABLE I: Privacy utility comparison results on the LFW. Method U-Net Attacker Flow-Matching Attacker COS ↓ ILR ↓ IRR ↑ FVR ↑ COS ↓ ILR ↓ IRR ↑ FVR ↑ DCTDP 0.9292 100.00 0.00 99.92 0.7187 99.99 0.01 99.91 DuetFace 0.9561 100.00 0.00 99.88 0.7581 100.00 0.00 99.93 MinusFace 0.3500 83.61 16.39 87.02 0.1960 38.28 61.72 78.03 FaceObfuscator -0.0051 N/A N/A 0.00 -0.0079 N/A N/A 0.00 FracFace 0.4424 95.96 4.04 60.16 0.2617 59.46 40.54 89.15 Cloak 0.9420 100.00 0.00 99.90 0.6997 100.00 0.00 99.83 AdvFace 0.9408 100.00 0.00 99.83 0.7162 100.00 0.00 99.91 0.0267 2.93 97.07 99.88 0.0137 0.74 99.26 99.78 Figure 6: Privacy–utility visualization on the LFW. The markers of DCTDP, DuetFace, AdvFace, and Cloak are jittered to avoid overlap. TABLE IV: Privacy utility comparison results on the AgeDB. Method U-Net Attacker Flow-Matching Attacker COS ↓ ILR ↓ IRR ↑ FVR ↑ COS ↓ ILR ↓ IRR ↑ FVR ↑ DCTDP 0.9431 99.95 0.05 98.91 0.7085 99.86 0.14 98.98 DuetFace 0.9641 100.00 0.00 97.55 0.7414 99.80 0.20 98.12 MinusFace 0.4551 89.55 10.45 89.75 0.2604 53.46 46.54 83.57 FaceObfuscator 0.0010 N/A N/A 0.00 -0.0218 N/A N/A 0.00 FracFace 0.5427 97.51 2.49 70.73 0.3253 68.33 31.67 91.92 Cloak 0.9307 100.00 0.00 98.22 0.6881 99.86 0.14 98.24 AdvFace 0.9414 100.00 0.00 97.79 0.7201 99.81 0.19 98.54 0.0240 3.38 96.62 99.78 0.0132 0.90 99.10 99.52 Figure 7: Privacy–utility visualization on the AgeDB. The markers of DCTDP, DuetFace, AdvFace, and Cloak are jittered to avoid overlap. TABLE V: Privacy utility comparison results on the CFP-FP. Method U-Net Attacker Flow-Matching Attacker COS ↓ ILR ↓ IRR ↑ FVR ↑ COS ↓ ILR ↓ IRR ↑ FVR ↑ DCTDP 0.9170 100.00 0.00 96.23 0.6936 100.00 0.00 96.16 DuetFace 0.9528 100.00 0.00 95.43 0.7391 100.00 0.00 95.89 MinusFace 0.3584 86.95 13.05 73.91 0.2063 48.72 51.28 56.09 FaceObfuscator -0.0182 N/A N/A 0.00 -0.0097 N/A N/A 0.00 FracFace 0.4662 97.93 2.07 56.34 0.2872 74.08 25.92 72.69 Cloak 0.9139 100.00 0.00 95.53 0.6895 99.98 0.02 99.98 AdvFace 0.8759 100.00 0.00 89.68 0.6510 99.76 0.24 89.82 0.0247 3.56 96.44 99.23 0.0177 1.79 98.21 98.72 Figure 8: Privacy–utility visualization on the CFP-FP. The markers of DCTDP, DuetFace, and Cloak are slightly jittered to avoid overlap. TABLE VI: Privacy utility comparison results on the CALFW. Method U-Net Attacker Flow-Matching Attacker COS ↓ ILR ↓ IRR ↑ FVR ↑ COS ↓ ILR ↓ IRR ↑ FVR ↑ DCTDP 0.9400 100.00 0.00 99.58 0.7112 99.97 0.03 99.67 DuetFace 0.9606 100.00 0.00 99.27 0.7405 99.97 0.03 99.44 MinusFace 0.4044 91.20 8.80 90.45 0.2273 47.48 52.52 80.88 FaceObfuscator -0.0036 N/A N/A 0.00 -0.0082 N/A N/A 0.00 FracFace 0.5106 98.10 1.90 67.97 0.2923 66.33 33.67 92.01 Cloak 0.9358 100.00 0.00 99.43 0.6801 99.97 0.03 99.97 AdvFace 0.9429 100.00 0.00 99.32 0.7476 99.49 0.51 99.49 0.0297 3.86 96.14 99.85 0.0157 1.15 98.85 99.64 Figure 9: Privacy–utility visualization on the CALFW. The markers of DCTDP, DuetFace, AdvFace, and Cloak are jittered to avoid overlap. TABLE VII: Privacy utility comparison results on the CPLFW. Method U-Net Attacker Flow-Matching Attacker COS ↓ ILR ↓ IRR ↑ FVR ↑ COS ↓ ILR ↓ IRR ↑ FVR ↑ DCTDP 0.9021 100.00 0.00 93.31 0.7221 99.98 0.02 93.24 DuetFace 0.9502 100.00 0.00 92.27 0.7858 100.00 0.00 92.24 MinusFace 0.3502 80.52 19.48 66.13 0.1985 40.54 59.46 51.08 FaceObfuscator -0.0118 N/A N/A 0.00 -0.0070 N/A N/A 0.00 FracFace 0.4072 95.40 4.60 42.73 0.2630 63.84 36.16 68.35 Cloak 0.9223 100.00 0.00 92.46 0.7146 100.00 0.00 90.81 AdvFace 0.9302 100.00 0.00 92.19 0.7476 99.99 0.01 92.57 0.0301 2.63 97.37 99.45 0.0216 1.59 98.41 98.98 Figure 10: Privacy–utility visualization on the CPLFW. The markers of DCTDP, DuetFace, AdvFace, and Cloak are jittered to avoid overlap. We compare the proposed method with representative PPFR methods222As official implementations of AdvFace, FaceObfuscator, and FracFace are unavailable, we reproduce them based on the details reported in their papers. For MinusFace, we report the IJB-B and IJB-C results from the original paper., including DCTDP Ji et al. (2022), DuetFace Mi et al. (2022), MinusFace Mi et al. (2024), FaceObfuscator Jin et al. (2024), FracFace Dai et al. (2025), Cloak Mireshghallah et al. (2021), and AdvFace Wang et al. (2023). These baselines cover several major design strategies, including frequency-domain collaboration, adversarial feature obfuscation, and inversion-resistant feature learning. All experiments were conducted on NVIDIA A100 GPUs. V-C Recognition Accuracy Table I reports the authorized recognition performance on standard face verification benchmarks. Overall, DecoyFace preserves competitive recognition performance across all datasets. Although DuetFace and MinusFace achieve higher Acc on some benchmarks, DecoyFace provides a more favorable privacy–utility trade-off by maintaining competitive authorized recognition performance while achieving identity misdirection rather than identity leakage or perceptible obfuscation. Specifically, DecoyFace achieves 99.77%99.77\% on LFW, 97.20%97.20\% on AgeDB, 95.54%95.54\% on CFP-FP, 95.63%95.63\% on CALFW, and 91.15%91.15\% on CPLFW. On the more challenging template-based benchmarks, it obtains 91.10%91.10\% TAR@FAR=10−410^-4 on IJB-B and 93.35%93.35\% on IJB-C. In particular, it outperforms FaceObfuscator, FracFace, Cloak, and AdvFace on most verification benchmarks. These results suggest that the proposed DecoyFace preserves most of the identity-discriminative information required for authorized verification, despite deliberately perturbing the transmitted representation to protect privacy. It is worth noting that DecoyFace does not pursue recognition accuracy alone, but a better privacy–utility trade-off, as shown in the following section. Some methods improve utility but still leak the original identity under inversion. By contrast, DecoyFace retains competitive verification utility while explicitly targeting identity redirection, which better aligns with the practical objective of PPFR. V-D Evaluation of Privacy Protection V-D1 Privacy Against Malicious Attackers - Any Way Tables I-VII and Figs. 5-10 report reconstruction privacy against malicious attackers under a U-Net-based attacker and a FM-based attacker. Fig. 5 shows qualitative reconstruction examples under both U-Net-based and FM-based attackers. Prior methods either preserve the original identity in visually plausible reconstructions or produce severely degraded outputs that expose the presence of protection. In contrast, DecoyFace maintains facial plausibility while visually redirecting the reconstruction toward the decoy identity, which is consistent with its low ILR and high FVR. It can be observed that prior methods mainly fall into two patterns, whereas DecoyFace exhibits a markedly different privacy profile. ① Privacy Leakage Defense: The first group, including DCTDP, DuetFace, Cloak, and AdvFace, preserves highly face-valid reconstructions, with FVR remaining close to 100%100\%. However, these methods still exhibit severe identity leakage: their ILR remains close to 100%100\%, whereas IRR is close to 0%0\%. This means that although the reconstructed faces remain visually plausible, they are still recognized as the original identity in almost all cases. Therefore, preserving visual realism while slightly degrading reconstruction fidelity is insufficient for preventing identity leakage. ② Effective-but-obvious Defenses: The second group corresponds to stronger suppression-oriented defenses. FaceObfuscator collapses the reconstruction, leading FVR to 0%, so identity metrics become undefined rather than indicating successful misdirection. FracFace and MinusFace partially reduce privacy leakage, but still fail to maintain high face validity and strong identity redirection simultaneously. These results suggest that prior methods primarily trade off between two unsatisfactory extremes: either the reconstructed result still reveals the original identity, or the reconstruction quality deteriorates to the point that the defense becomes obvious. In contrast, DecoyFace achieves a different privacy profile. Under the malicious-attacker setting, DecoyFace maintains consistently high face validity, with FVR ranging from 98.72%98.72\% to 99.88%99.88\%, indicating that the reconstructed outputs are still plausible faces. At the same time, the reconstructed results exhibit very weak consistency with the original identity, as reflected by the low COS across all benchmarks. More importantly, DecoyFace reduces ILR to 2.63%2.63\%–3.86%3.86\% under the U-Net attacker and 0.74%0.74\%–1.79%1.79\% under the FM attacker. This means that in most cases, the attacker still obtains a valid face, but it is recognized as a non-original identity rather than the true subject. This observation is consistent with the decoy-oriented objective of DecoyFace: the defense does not merely make reconstruction fail, but instead redirects privacy leakage toward a realistic yet incorrect identity. TABLE VIII: Privacy under HBC Server-Side Reconstruction. Dataset Method COS↓ ILR↓ IRR↑ FVR↑ LFW FaceObfuscator 0.4898 88.33 11.67 89.45 0.0059 0.04 99.96 99.98 AgeDB FaceObfuscator 0.5185 85.36 14.64 90.57 0.0007 0.08 99.92 99.99 CFP-FP FaceObfuscator 0.4228 85.36 14.64 78.89 0.0046 0.34 99.66 99.93 CALFW FaceObfuscator 0.5038 89.41 10.59 89.25 0.0067 0.07 99.93 100.00 CPLFW FaceObfuscator 0.4134 83.57 16.43 70.64 0.0094 0.08 99.92 99.94 V-D2 Privacy Against HBC Servers - Any Where Different from the malicious attacker that reconstructs from the transmitted protected feature ztz_t, the HBC server trains its reconstruction model directly on the server-side canonicalized feature zcanz_can. We assume that the HBC server knows the recovery pipeline, including the projection PNP_N, quantile calibration, and the RSS anchor μrec _rec. This setting is more challenging because zcanz_can is the representation used for authorized recognition and therefore inevitably retains certain identity-discriminative evidence. Since most existing baselines do not introduce an additional server-side canonicalization module comparable to zcanz_can, we include FaceObfuscator as a representative baseline for the HBC comparison, as shown in Table VIII. FaceObfuscator still suffers from substantial original-identity leakage, with ILR ranging from 83.57% to 89.41%, whereas DecoyFace reduces ILR to 0.04%–0.34% while keeping FVR between 99.93% and 100.00%. Existing PPFR methods mainly protect the transmitted representation, but do not explicitly prevent a server-side recognition feature from being exploited for inversion. In contrast, DecoyFace does not recover the original intermediate feature on the server side. The RSS component is suppressed by PNP_N, while the complementary carrier has already been sparsified and mixed with auxiliary identities through DGCM. Quantile calibration and RSS anchor μrec _rec restore a stable distribution for recognition, but do not restore a clean complementary branch of the true sample. Therefore, zcanz_can remains sufficiently discriminative for authorized verification, yet lacks the visual and identity-complete evidence required for faithful reconstruction by the HBC server. TABLE IX: Component ablations of DecoyFace. Setting Acc ↑ ILR ↓ DHR ↑ FVR ↑ w/o RSSD 61.37 94.32 0.40 92.47 w/o DGCM 91.97 100.00 0.00 92.83 w/o ACM 66.02 2.63 30.19 99.45 w/o CSG 90.12 53.24 7.12 95.57 w/o IN 89.20 53.29 6.40 95.45 w/o Coherent 90.11 86.69 0.07 94.72 w/o Scale 90.68 72.89 2.88 94.20 w/o Calib 81.38 2.63 30.19 99.45 w/o μrec _rec 83.68 2.63 30.19 99.45 w/o ℒconsL_cons 88.77 2.63 30.19 99.45 91.15 2.63 30.19 99.45 TABLE X: EFFECT OF GROUP NUMBER G AND PRESERVED TRUE-SAMPLE GROUPS K IN DGCM. G K Acc ↑ ILR ↓ DHR ↑ FVR ↑ 16 2 80.98 0.12 26.78 99.88 16 3 91.15 2.63 30.19 99.45 16 4 91.17 83.48 4.92 94.20 32 6 88.65 63.65 5.08 94.88 V-E Ablation Studies We conduct ablation studies on the CPLFW dataset to analyze the contribution of each component in DecoyFace with the U-Net attacker. V-E1 Effect of core components Table IX summarizes the effect of the major components in the proposed framework. Removing RSSD severely degrades both utility and privacy, reducing the Acc to 61.37% on CPLFW, while increasing ILR to 94.32% and reducing DHR to 0.40%. This indicates that RSSD is essential for separating reconstruction-dominant directions from the complementary carrier used for recognition. Removing DGCM leads to a different failure mode. Although the recognition Acc remains relatively high, the privacy protection completely fails, as ILR reaches 100.00% and DHR drops to 0.00%. This result confirms that explicitly constructing the decoy carrier is necessary for steering unauthorized reconstruction away from the original identity. Removing ACM mainly damages authorized utility, with the Acc dropping from 91.15% to 66.02%. This confirms that the matched server-side ACM is critical for suppressing the decoy-dominant component and restoring a recognition-friendly representation. V-E2 Effect of client-side DGCM We further analyze the detailed design choices in DGCM. Removing the CSG, IN, coherent-subset group mixing policy, and random scaling consistently weakens the identity redirection capability. Without CSG, ILR increases to 53.24%, and DHR decreases to 7.12%, suggesting that randomized grouping helps disrupt residual true-identity traces in the complementary branch. Without IN, ILR rises to 53.29%, indicating that statistical alignment is important for stabilizing the mixed representation. The coherent mixing strategy is particularly important. Removing it increases ILR to 86.69% and reduces DHR to 0.07%, which shows that independently replaced groups introduce fragmented complementary evidence and weaken decoy-consistent reconstruction. Removing the random scale also increases ILR to 72.89% and reduces DHR to 2.88%, verifying that sample-wise magnitude variation provides an additional barrier against feature inversion. Notably, all variants maintain high FVR, which indicates that the degradation is not caused by reconstruction collapse. Instead, these components mainly determine whether a valid reconstructed face is attributed to the original identity or redirected to the decoy identity. V-E3 Effect of server-side ACM design The server-side components mainly affect authorized recognition rather than attacker-side privacy. The attacker-side privacy metrics remain unchanged because the malicious attacker reconstructs from the transmitted protected feature before server-side processing. Removing quantile calibration reduces the Acc to 81.38%, while the privacy metrics remain unchanged. A similar trend is observed when removing the reconstruction-sensitive anchor μrec _rec, where the Acc drops to 83.68%. Removing the consistency loss decreases the Acc to 88.77%, indicating that multi-view consistency helps the server learn embeddings that are robust to stochastic client-side protection. Overall, these results show that quantile calibration, the reconstruction-sensitive anchor, and the consistency constraint mainly improve the stability of authorized recognition. V-E4 Effect of Group Number and Preserved Groups Table X studies how the group number G and the number of preserved groups K affect recognition and privacy. With G=16G=16 and K=2K=2, the model achieves a very low ILR of 0.12%, but the Acc drops to 80.98%, indicating that preserving too few true-sample groups can weaken authorized verification. Increasing K to 3 improves the Acc to 91.15%, while keeping ILR low at 2.63% and maintaining a high DHR of 30.19%. However, further increasing K to 4 causes ILR to rise sharply to 83.48%, showing that the attacker can exploit excessive true-identity evidence. When the group number is increased to G=32G=32 with K=6K=6, ILR remains high at 63.65%, and Acc drops to 88.65%, suggesting that overly fine-grained grouping may fragment the mixed representation and weaken coherent decoy guidance. Therefore, we adopt G=16G=16 and K=3K=3 as the default configuration. Figure 11: Qualitative results with different decoy identities. The original face is fixed, while each row uses a different target decoy. V-E5 Effect of Different Decoy - Any Who To further illustrate the controllability of decoy-oriented misdirection, Fig. 11 presents qualitative examples with different decoys. Given the same original face, changing the assigned decoy leads to different unauthorized reconstructions under both U-Net-based and FM-based attackers. The reconstructed faces remain visually plausible, but are shifted away from the original identity and toward the selected decoys. These results confirm that DecoyFace can achieve controllable identity misdirection. VI Conclusion We propose DecoyFace, a decoy-oriented privacy-preserving framework for split face recognition. Unlike existing PPFR methods that mainly suppress reconstruction quality, DecoyFace aims to misdirect unauthorized reconstruction toward a plausible but incorrect identity while preserving recognition utility for the authorized server. The framework is built upon RSSD, client-side DGCM, and server-side ACM. This design injects decoy information into reconstruction-sensitive directions, limits true-identity exposure in the complementary carrier, and enables the server to recover a canonicalized representation for verification. Experiments demonstrate that DecoyFace preserves competitive recognition accuracy while significantly reducing true-identity leakage. Under both malicious attacker-side inversion and HBC server-side reconstruction, DecoyFace maintains high face validity and redirects most unauthorized reconstructions away from the original identity. These results show that privacy protection in split face recognition need not rely solely on perceptible obfuscation or reconstruction collapse. Instead, decoy-based misdirection provides a stealthier and more controllable privacy-preserving direction. References [1] W. Ao and V. N. Boddeti (2025) CryptoFace: end-to-end encrypted face recognition. In IEEE Conf. Comput. Vis. Pattern Recog., p. 19197–19206. Cited by: TABLE I, §I, §I-B. [2] W. Bai, S. Chen, W. Chen, and H. Sun (2026) Blind inversion using latent diffusion priors. IEEE Trans. Image Process.. Cited by: §I-A. [3] D. Chen, S. Li, Y. Zhang, C. Li, S. Kundu, and P. A. Beerel (2024) DIA: diffusion based inverse network attack on collaborative inference. In IEEE Conf. Comput. Vis. Pattern Recog., p. 124–130. Cited by: §I, §I-A1. [4] W. Dai, B. Li, N. Dong, G. Bai, and J. S. Dong (2025) FracFace: breaking the visual clues—fractal-based privacy-preserving face recognition. In Adv. Neural Inform. Process. Syst., Cited by: TABLE I, §I, §I-B, §V-B. [5] J. Deng, J. Guo, N. Xue, and S. Zafeiriou (2019) Arcface: additive angular margin loss for deep face recognition. In IEEE Conf. Comput. Vis. Pattern Recog., p. 4690–4699. Cited by: §IV-D. [6] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao (2016) Ms-celeb-1m: a dataset and benchmark for large-scale face recognition. In Eur. Conf. Comput. Vis., p. 87–102. Cited by: §V-A. [7] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In IEEE Conf. Comput. Vis. Pattern Recog., p. 770–778. Cited by: §V-B. [8] L. He, Z. Ren, W. Zhang, F. Li, and S. Mei (2024) Unsupervised pansharpening based on double-cycle consistency. IEEE Transactions on Geoscience and Remote Sensing 62, p. 1–15. Cited by: §I-A. [9] C. Hsu, C. Lu, and S. Pei (2012) Image feature extraction in encrypted domain with privacy-preserving sift. IEEE Trans. Image Process. 21 (11), p. 4593–4607. Cited by: TABLE I, §I, §I-B. [10] G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller (2008) Labeled faces in the wild: a database forstudying face recognition in unconstrained environments. In Workshop on faces in Real-Life Images: detection, alignment, and recognition, Cited by: §V-A. [11] J. Ji, H. Wang, Y. Huang, J. Wu, X. Xu, S. Ding, S. Zhang, L. Cao, and R. Ji (2022) Privacy-preserving face recognition with learnable privacy budgets in frequency domain. In Eur. Conf. Comput. Vis., p. 475–491. Cited by: §I, §V-B. [12] S. Jin, H. Wang, Z. Wang, F. Xiao, J. Hu, Y. He, W. Zhang, Z. Ba, W. Fang, S. Yuan, et al. (2024) FaceObfuscator: defending deep learning-based privacy attacks with gradient descent-resistant features in face recognition. In 33rd USENIX Security Symposium (USENIX Security 24), p. 6849–6866. Cited by: TABLE I, §I, §I-B, §V-B. [13] Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang (2017) Neurosurgeon: collaborative intelligence between the cloud and mobile edge. ACM SIGARCH Computer Architecture News 45 (1), p. 615–629. Cited by: §I. [14] W. Lei, J. Chen, and S. Chen (2025) DRAG: data reconstruction attack using guided diffusion. In International Conference on Machine Learning, p. 33883–33901. Cited by: §I, §I-A. [15] J. Li, C. Luo, H. Zhang, Y. Cao, X. Liao, and X. Cao (2025) Anti-fake vaccine: safeguarding privacy against face swapping via visual-semantic dual degradation. Int. J. Comput. Vis. 133 (4), p. 2025–2043. Cited by: TABLE I, §I, §I-B. [16] Z. Li, M. Yang, Y. Liu, J. Wang, H. Hu, W. Yi, and X. Xu (2023) GAN you see me? enhanced data reconstruction attacks against split inference. Adv. Neural Inform. Process. Syst. 36, p. 54554–54566. Cited by: §I, §I-A. [17] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2022) Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Cited by: §I-A. [18] Z. Liu, P. Luo, X. Wang, and X. Tang (2015) Deep learning face attributes in the wild. In Int. Conf. Comput. Vis., p. 3730–3738. Cited by: §V-B, §V-B. [19] B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller, C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, J. Cheney, et al. (2018) Iarpa janus benchmark-c: face dataset and protocol. In 2018 international conference on biometrics (ICB), p. 158–165. Cited by: §V-A. [20] Y. Mi, Y. Huang, J. Ji, H. Liu, X. Xu, S. Ding, and S. Zhou (2022) Duetface: collaborative privacy-preserving face recognition via channel splitting in the frequency domain. In ACM Int. Conf. Multimedia, p. 6755–6764. Cited by: TABLE I, §I, §V-B. [21] Y. Mi, Z. Zhong, Y. Huang, J. Ji, J. Xu, J. Wang, S. Wang, S. Ding, and S. Zhou (2024) Privacy-preserving face recognition using trainable feature subtraction. In IEEE Conf. Comput. Vis. Pattern Recog., p. 297–307. Cited by: §I, §I-B, §V-B. [22] F. Mireshghallah, M. Taram, A. Jalali, A. T. T. Elthakeb, D. Tullsen, and H. Esmaeilzadeh (2021) Not all features are equal: discovering essential features for preserving prediction privacy. In Proceedings of the Web Conference, p. 669–680. Cited by: §V-B. [23] S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kotsia, and S. Zafeiriou (2017) Agedb: the first manually collected, in-the-wild age database. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., p. 51–59. Cited by: §V-A. [24] Z. Ren, L. He, J. Liang, X. Fu, H. Bi, and F. Li (2026) What your features reveal: data-efficient black-box feature inversion attack for split dnns. In IEEE Conf. Comput. Vis. Pattern Recog., p. 13366–13375. Cited by: §I, §I-A, §I-A1, §V-B. [25] Z. Ren, L. He, and J. Lu (2023) Context aware edge-enhanced gan for remote sensing image super-resolution. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 17, p. 1363–1376. Cited by: §I-A. [26] Z. Ren, L. He, and P. Zhu (2024) Super-resolution learning strategy based on expert knowledge supervision. Remote Sensing 16 (16), p. 2888. Cited by: §I-A. [27] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, p. 234–241. Cited by: §V-B, §V-B. [28] S. Sengupta, J. Chen, C. Castillo, V. M. Patel, R. Chellappa, and D. W. Jacobs (2016) Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision, p. 1–9. Cited by: §V-A. [29] A. Singh, V. Sharma, R. Sukumaran, J. Mose, J. Chiu, J. Yu, and R. Raskar (2024) Simba: split inference—mechanisms, benchmarks and attacks. In Eur. Conf. Comput. Vis., p. 214–232. Cited by: §I, §I-A. [30] Y. Song and S. Ermon (2019) Generative modeling by estimating gradients of the data distribution. Adv. Neural Inform. Process. Syst. 32. Cited by: §I-A. [31] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2020) Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. Cited by: §I-A. [32] S. Teerapittayanon, B. McDanel, and H. Kung (2017) Distributed deep neural networks over the cloud, the edge and end devices. In 2017 IEEE 37th international conference on distributed computing systems, p. 328–339. Cited by: §I. [33] D. Ulyanov, A. Vedaldi, and V. Lempitsky (2016) Instance normalization: the missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022. Cited by: §IV-C. [34] K. Wang, L. He, H. Fu, H. Bi, and F. Li (2025) Beyond hallucinations: a multimodal-guided task-aware generative image compression for ultra-low bitrate. arXiv preprint arXiv:2512.06344. Cited by: §I-A. [35] L. Wang, Z. Li, X. Zhang, S. Ji, and S. Guo (2025) Faceswapguard: safeguarding facial privacy from deepfake threats through identity obfuscation. arXiv preprint arXiv:2502.10801. Cited by: TABLE I, §I, §I-B. [36] T. Wang, Y. Zhang, X. Xiao, L. Yuan, Z. Xia, and J. Weng (2024) Make privacy renewable! generating privacy-preserving faces supporting cancelable biometric recognition. In ACM Int. Conf. Multimedia, p. 10268–10276. Cited by: TABLE I, §I, §I-B. [37] T. Wang, S. Niu, H. Cheng, X. Zhang, and Y. Wang (2025) Nullswap: proactive identity cloaking against deepfake face swapping. In Int. Conf. Comput. Vis., p. 9945–9954. Cited by: TABLE I, §I, §I-B. [38] Z. Wang, H. Wang, S. Jin, W. Zhang, J. Hu, Y. Wang, P. Sun, W. Yuan, K. Liu, and K. Ren (2023) Privacy-preserving adversarial facial features. In IEEE Conf. Comput. Vis. Pattern Recog., p. 8212–8221. Cited by: TABLE I, §I, §I-B, §V-B. [39] C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller, N. Kalka, A. K. Jain, J. A. Duncan, K. Allen, et al. (2017) Iarpa janus benchmark-b face dataset. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., p. 90–98. Cited by: §V-A. [40] Z. Yuan, Z. You, S. Li, Z. Qian, X. Zhang, and A. Kot (2022) On generating identifiable virtual faces. In ACM Int. Conf. Multimedia, p. 1465–1473. Cited by: TABLE I, §I, §I-B. [41] S. Q. Zhang, Z. Li, C. Guo, S. Mahloujifar, D. Dangwal, E. Suh, B. De Salvo, and C. Liu (2024) Unlocking visual secrets: inverting features with diffusion priors for image reconstruction. arXiv preprint arXiv:2412.10448. Cited by: §I, §I-A1. [42] T. Zheng, W. Deng, and J. Hu (2017) Cross-age LFW: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08197. Cited by: §V-A. [43] T. Zheng and W. Deng (2018) Cross-pose lfw: a database for studying cross-pose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep 5 (7), p. 5. Cited by: §V-A.