Paper deep dive
Calibrating One-Round Membership Inference with Neighbors
Francesco Rita, Jie Zhang, Florian TramĆØr
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The state-of-the-art Membership Inference (MI) methods calibrate their signal separately for each example using reference models, auxiliary models trained to exclude the target. This paradigm scales poorly to modern large models, however, whose training is too expensive to replicate. This has motivated one-round settings, where only a single trained model is available; but without reference models the per-example calibration that drives the strongest attacks can no longer be estimated, leaving the membership signal weak. We ask whether neighbors of the target point can recover this calibration without training any additional model. Our key observation is that reference models serve only to reveal how an example behaves under models not trained on it, and that querying the target model on nearby samples yields the same information. We propose two complementary ways to obtain such neighbors, and show that querying them against an early training checkpoint further sharpens the signal. We evaluate across three image classification datasets and three training setups, showing that neighbors yield strong membership signals and competitive attack performance at no additional training cost.
Tags
Links
- Source: https://arxiv.org/abs/2609.36331v1
- Canonical: https://arxiv.org/abs/2609.36331v1
Trouble viewing inline? Open PDF directly ā
Full Text
62,291 characters extracted from source content.
Expand or collapse full text
Calibrating One-Round Membership Inference with Neighbors DOI: 10.1145/3847352.3848120Conference: 19th Workshop on Artificial Intelligence and Security; November 15ā19, 2026; The Hague, Netherlands19th Workshop on Artificial Intelligence and Security (AISec ā26), November 15ā19, 2026, The Hague, NetherlandsISBN: 979-8-4007-3031-3/2026/11CCS: Computing methodologies Machine learningCCS: General and reference EvaluationCCS: Security and privacy Francesco Rita Affiliation: ETH Zurich, Zurich, Switzerland , Jie Zhang Affiliation: ETH Zurich, Zurich, Switzerland and Florian TramĆØr Affiliation: ETH Zurich, Zurich, Switzerland Ā© c Figure 1. Comparison between Steinke et al. (2023) and our best method. On the left we report the empirical ĻµĪµ in the DP-SGD setting across CIFAR-10, CIFAR-100 and CINIC-10. On the right we report the TPR at 0.1% FPR on CIFAR-10 across three privacy settings: undefended models, RelaxLoss and DP-SGD.Two bar plots illustrating the improvement in privacy leakage achieved by our best-performing method compared with steinke2023privacy. The left panel reports the empirical $Īµ$ in the DP-SGD setting across CIFAR-10, CIFAR-100, and CINIC-10. The right panel shows the true positive rate (TPR) at a 0.1\% false positive rate (FPR) on CIFAR-10 under three privacy settings: undefended models, RelaxLoss, and DP-SGD. Across all datasets and privacy settings, our method consistently achieves higher privacy leakage than the baseline. Abstract. The state-of-the-art Membership Inference (MI) methods calibrate their signal separately for each example using reference models, auxiliary models trained to exclude the target. This paradigm scales poorly to modern large models, however, whose training is too expensive to replicate. This has motivated one-round settings, where only a single trained model is available; but without reference models the per-example calibration that drives the strongest attacks can no longer be estimated, leaving the membership signal weak. We ask whether neighbors of the target point can recover this calibration without training any additional model. Our key observation is that reference models serve only to reveal how an example behaves under models not trained on it, and that querying the target model on nearby samples yields the same information. We propose two complementary ways to obtain such neighbors, and show that querying them against an early training checkpoint further sharpens the signal. We evaluate across three image classification datasets and three training setups, showing that neighbors yield strong membership signals and competitive attack performance at no additional training cost. Keywords: machine learning; privacy; audit; membership inference; DP-SGD ā c-license: by 1. Introduction Membership inference attacks (Shokri et al., 2017) have become a standard tool for auditing the privacy of machine learning models. Given a trained model and a target sample, an MIA decides whether that sample was part of the modelās training set. Because models tend to behave differently on data they have memorized than on unseen data, the success of such an attack offers an empirical, sample-level measure of how much a model leaks about its training data. The strongest MIAs achieve this by calibrating the targetās behavior on a per-example basis. Rather than asking whether the loss on a sample is low in absolute terms, they ask whether it is surprisingly low for that particular sample, accounting for the fact that some examples are intrinsically easier than others (Carlini et al., 2022). This calibration is precisely what separates state-of-the-art attacks from naive loss thresholding, and it is obtained through reference (or shadow) models: auxiliary models that reveal how the target model would score a sample when that sample is excluded from training (Shokri et al., 2017; Carlini et al., 2022; Du et al., 2025; Bai et al., 2025). This paradigm, however, has grown increasingly impractical with the advent of highly scalable architectures such as Transformers (Vaswani et al., 2023) and the large foundation models they enabled (Touvron et al., 2023; DeepSeek-AI et al., 2025). Training even a single such model can be prohibitively expensive; training the tens or hundreds of reference models that calibration requires is often out of reach. This has motivated growing interest in the one-round setting, where the auditor has access to a single trained model and cannot retrain new models. The one-round regime is especially central to the auditing of differentially private (DP) training, where one seeks an empirical lower bound on the privacy parameter ĻµĪµ from a single training run, precisely to avoid the cost of repeated retraining (Steinke et al., 2023; Mahloujifar et al., 2024; Xiang et al., 2025). Illustration of the key difference between the shadow-model-based approach and our method. The traditional approach fixes a target sample and evaluates it across multiple shadow models to construct a loss distribution. Our method instead fixes the target model and evaluates it on multiple neighboring samples of the target, yielding an approximation of the same loss distribution without the need to train or query multiple models. Figure 2. Comparison between shadow-model based methods and our method. Instead of characterizing the loss distribution by querying the canary against different shadow models, our method queries different samples neighboring the canary against the target model alone.Illustration of the key difference between the shadow-model-based approach and our method. The traditional approach fixes a target sample and evaluates it across multiple shadow models to construct a loss distribution. Our method instead fixes the target model and evaluates it on multiple neighboring samples of the target, yielding an approximation of the same loss distribution without the need to train or query multiple models. Removing reference models, however, removes the very ingredient that makes modern attacks strong. Without them, the per-example calibration can no longer be estimated, and existing one-round methods fall back on a global loss threshold or simple heuristics (Steinke et al., 2023). The resulting signal is weak, particularly at the low false-positive rates that matter most for membership inference (Carlini et al., 2022), which in turn loosens DP audits. Recovering per-example calibration without reference models is therefore the common bottleneck for both one-round attacks and one-run DP auditing. In this work, we show that this calibration can be recovered using nothing beyond the target model itself. Our starting point is the observation that reference models serve a single purpose: to reveal how the target sample would score under models that did not train on it. We obtain the same information differently, by querying the target model on samples that neighbor the target point. Intuitively, the modelās behavior on a non-member sample close to the target approximates the behavior a reference model would exhibit on the target itself. Figure 2 contrasts the two views: where reference-based attacks query the target sample against many models, we query many neighbors against a single model. Building on this idea, we characterize what makes a neighbor useful and propose two complementary ways to construct neighbors, by retrieving them from held-out data (Similarity Neighbors) and by synthesizing them with a diffusion model (Synthetic Neighbors); in the context of internal audits, we further show that querying neighbors against an early training checkpoint sharpens the signal further. Across three datasets and three privacy regimes, our attacks substantially outperform the one-round baseline of Steinke et al. (2023) and close much of the gap to reference-based attacks, without training a single additional model, as previewed in Figure 1. 2. Background and Preliminary 2.1. Membership Inference: Setup and Threat Model Let F denote a model trained on a dataset D drawn from a data distribution D. Membership inference (MI) asks, for a given example x, whether xāDxā D. Following the standard formulation (Shokri et al., 2017; Yeom et al., 2018), an MI adversary A outputs a score reflecting its belief that x was a training member; thresholding this score yields a membership decision, and the scoreās quality is measured by the resulting true-positive/false-positive trade-off. Within this framework, we refer to canaries as the subset of training examples that are most vulnerable to membership inference. Threat model. We study the most constrained and most realistic regime, which we refer to as one-round membership inference. The adversary is given a single target model FtF_t and an example x, and may query FtF_t to obtain a per-example signal such as the loss or confidence on x. Crucially, the adversary cannot train additional models. This rules out the reference-model machinery on which the strongest attacks rely, and it is the setting faced by an auditor or adversary confronting a single deployed model. 2.2. From Loss Thresholding to Reference-Based Attacks The simplest attack thresholds the per-example loss: if Lā”(Ft,x)L(F_t,x) is small, predict āmemberā (Yeom et al., 2018). This is weak because examples differ in intrinsic difficulty. Some examples have low loss whether or not they were trained on, while others have high loss regardless, so a single global threshold conflates easy non-members with true members. The membership signal comes not from the magnitude of the loss but from calibrating that loss to the specific example. Reference-model (or shadow-model) attacks supply this calibration (Shokri et al., 2017; Carlini et al., 2022). By training auxiliary models on data drawn from D, the adversary estimates how a model behaves on x when x is, or is not, in its training set, and calibrates the targetās behavior against this reference. The Likelihood Ratio Attack (LiRA) (Carlini et al., 2022) makes this precise: it casts membership as a per-example hypothesis test, comparing the targetās score on x against the distribution of scores that models not trained on x assign to x. 2.3. Offline LiRA, Formally Let sā”(F,x)s(F,x) be a membership score for example x under model F. A standard choice is the logit-scaled confidence (Carlini et al., 2022) (1) sā”(F,x)=logā”p1āp,p=Fā(x)y,s(F,x)\;=\; p1-p, p=F(x)_y, where Fā(x)yF(x)_y is the predicted probability on the true label y; larger s indicates stronger memorization of x by F. The logit scaling is used because s is approximately Gaussian, whereas the raw confidence is not. Offline LiRA assumes access only to out modelsāmodels not trained on x---and performs a one-sided test.11 1 This is the appropriate baseline for our setting: the online variant additionally trains in models that include x, which requires retraining with the target example and is therefore unavailable in the one-round regime. For brevity, we henceforth write LiRA to denote its offline variant unless stated otherwise. Concretely, it trains a set of out shadow models F1,ā¦,Fn\F_1,ā¦,F_n\, none of which is trained on x, and approximates the out score distribution as a Gaussian (2) 1=ā”(Ī¼out,Ļout2),D_1\;=\;N\! ( _out,\, _out^2 ), with (3) Ī¼out=1nāāi=1nsā”(Fi,x),Ļout2=1nāāi=1n(sā”(Fi,x)āĪ¼out)2. _out= 1n _i=1^ns(F_i,x), _out^2= 1n _i=1^n (s(F_i,x)- _out )^2. Membership is then assessed by how surprising the targetās score is under 1D_1: (4) Īā”(x)= 1āĪ¦ā”(sā”(Ft,x)āĪ¼outĻout), (x)\;=\;1- \! ( s(F_t,x)- _out _out ), where Ī¦ is the standard normal CDF. A small Īā”(x) (x)āthe targetās score lying in the upper tail of 1D_1āindicates that x is a member. The entire procedure reduces to one object: the per-example out distribution 1D_1, estimated from a set of out scores sā”(Fi,x)i=1n\s(F_i,x)\_i=1^n. Its strength comes entirely from this per-example calibration, and its cost comes entirely from obtaining the n out models needed to form it. 2.4. One-Round MI and One-Run DP Auditing The reference models underlying Eq. (3) are precisely what the one-round setting denies. Without the ability to train out models, 1D_1 cannot be estimated per example, and existing one-round methods fall back on a global loss threshold or simple per-sample heuristicsādiscarding the calibration that drives the signal. The result is a faint membership signal, especially in the low-false-positive-rate regime that matters most for MI (Carlini et al., 2022). The same constraint motivates the auditing of differentially private (DP) training (Dwork et al., 2006). DP auditing derives an empirical lower bound on the privacy parameter Īµ from the distinguishability of members and non-members, and has moved decisively toward the single-run regime (Steinke et al., 2023): rather than retraining a model many times, one inserts many canary examples into a single training run and infers Īµ from their membership scores, precisely to avoid the prohibitive cost of repeated training. The tightness of such an auditāhow close the empirical Īµ comes to the true privacy lossāis governed by the separation between member and non-member scores. When each canary is scored by its global loss, this separation is poor, and the audit is correspondingly loose, forcing either an underestimate of Īµ or an impractically large number of canaries. Recovering per-example calibration without out models is therefore the common bottleneck for both one-round attacks and one-run auditing. In Section 3, we address it directly: we estimate the out distribution 1D_1 using the target modelās scores on neighbors of x, requiring nothing beyond the target model itself. 3. Calibrating One-Round MI with Neighbors LiRA owes its strength to a single object: the per-example out distribution 1D_1 of Eq. (2), whose mean and variance are estimated from the scores that out shadow models assign to the example under testāthe canary, in the auditing setting. As noted in Section 2.3, the shadow models serve no purpose beyond producing this set of scores. We exploit this observation directly: rather than training shadow models, we form the same set of scores by querying the target model FtF_t on samples that neighbor the canary (see Figure 2). The intuition, which we make precise below, is that the targetās behavior on nearby non-member samples proxies the canaryās behavior across out models. Formally, our attack is a single substitution in the LiRA estimator. Given a set of n neighbors x~1,ā¦,x~n\ x_1,ā¦, x_n\ of the canary x, we replace the out-model scores sā”(Fi,x)\s(F_i,x)\ with the targetās scores on the neighbors and estimate (5) Ī¼^out=1nāāj=1nsā”(Ft,x~j),Ļ^out2=1nāāj=1n(sā”(Ft,x~j)āĪ¼^out)2, Ī¼_out= 1n _j=1^ns(F_t, x_j), Ļ^2_out= 1n _j=1^n (s(F_t, x_j)- Ī¼_out )^2, yielding ^1=ā”(Ī¼^out,Ļ^out2) D_1=N( Ī¼_out, Ļ^2_out). We then score the canary by the tail probability of sā”(Ft,x)s(F_t,x) under ^1 D_1, exactly as in LiRA (Eq. (4)). The full procedure is given in Algorithm 1; it requires no model training and queries only FtF_t, making it applicable in the one-round setting. Figure 3. Comparison of shadow model-based and neighbor-driven confidence distributions for a member canary. The in (blue) and out (red) distributions are estimated using shadow models trained with and without the canary, respectively. Neighbor distributions are obtained by querying Stable Diffusion img2img generations against the target model. Neighbors that are too similar (strength 0.45) resemble the member (in) distribution, whereas stronger perturbations (strength 0.75) better approximate the non-member (out) distribution.Comparison of confidence distributions obtained from shadow models and synthetic neighbors for a member sample. Neighbors that are too similar to the target sample produce confidence values that closely follow the shadow-model in distribution, whereas sufficiently perturbed neighbors better approximate the out distribution. 3.1. What Makes a āGoodā Neighbor The validity of the substitution in Eq. (5) rests on one assumption, which we state explicitly. Assumption 1 (Local smoothness). The target modelās membership score sā”(Ft,ā )s(F_t,Ā·) varies smoothly over the data manifold in a neighborhood of the canary x. Consequently, for a non-member x~ x sufficiently close to x, the score sā”(Ft,x~)s(F_t, x) approximates the score an out model assigns to x, i.e. a sample from 1D_1. Assumption 1 exposes a distance trade-off intrinsic to our estimator. To faithfully approximate 1D_1, each neighbor must be close enough to the canary to be representative of its behavior. Yet being too close is harmful: if the canary was seen during training, the target memorizes it, and a neighbor that is too similar has its score pulled toward the member regime, contaminating ^1 D_1 and shrinking the very membership signal we seek. For member canaries the trade-off depends further on intrinsic learning difficulty and the modelās memorization capacity: a hard-to-memorize sample exhibits similar scores regardless of membership, tolerating closer neighbors, whereas an easy-to-memorize sample has a wide membership-dependent score gap, requiring neighbors to lie further away. Because membership labels are unavailable at inference time, this trade-off must be respected whether or not the canary is a member. Even if one could condition on membership, the optimal distance would still depend on canary- and model-specific characteristics that vary considerably across samples and settings. These factors make it difficult to formulate a single distance threshold whose tuning reliably optimizes the trade-off across all canaries. We therefore do not attempt to pin down one optimal distance. Instead, we keep neighbors within a controlled similarity bandāclose to the canary, yet not too closeāand operationalize this band through two complementary construction mechanisms: the intrinsic trainātest gap when retrieving neighbors from the test set (Section 3.2), and the denoising strength when synthesizing them (Section 3.3). Both mechanisms use a notion of distance to locate candidates near the canary, but neither relies on a single tuned distance threshold as its decision criterion. Algorithm 1 Neighbor approximation of LiRA. In the Early Checkpoint variant, the neighbor scores in line 2 are computed against an early checkpoint of the target model. 0: target model FtF_t, canary x, number of neighbors n 1: x~1,ā¦,x~nāā”(x)\ x_1,ā¦, x_n\ā neighbors(x) search / generate neighbors 2: Soutāsā”(Ft,x~1),ā¦,sā”(Ft,x~n)S_outā\s(F_t, x_1),ā¦,s(F_t, x_n)\ 3: Ī¼^outāā”(Sout) Ī¼_outā mean(S_out) 4: Ļ^out2āā”(Sout) Ļ^2_outā var(S_out) 5: return 1āā”(sā”(Ft,x),Ī¼^out,Ļ^out2)1- GaussianTail (s(F_t,x);\, Ī¼_out, Ļ^2_out ) 3.2. Similarity Neighbors The test set is a natural source of neighbors: by construction it consists entirely of non-member data, shares the training distribution, and is held at the intrinsic trainātest distance from any training sampleāprecisely the upper edge of the band described in Section 3.1. Exploiting this, we select each canaryās neighbors by retrieving the most similar test samples. Concretely, given a canary we first discard all test samples not in its class. We then embed the canary and each remaining test sample with OpenAIās CLIP ViT-B/32 model (Radford et al., 2021) and compute their cosine similarity. To obtain a more robust score, we repeat this over 1818 augmentations of the input pair (horizontal shifts, vertical shifts, and vertical flips) and average the resulting scores. Simpler variants that omit the class filter or the augmentations are possible; we adopt this formulation as it yields the best empirical performance. As shown in Section 4.5, it already outperforms Steinke et al. (2023) in most scenarios. 3.3. Synthetic Neighbors Our second construction shifts from existing data to synthetic neighbors. This offers greater flexibility, as the generated samples can be controlled through the choice of generative model, its hyperparameters, and optional text prompting---and it applies even when the test set contains no suitable neighbors. We generate neighbors with the image-to-image pipeline of Stable Diffusion22 2 We use the stable-diffusion-v1-5 implementation, as the original RunwayML model is deprecated. (Rombach et al., 2022), which produces variations of an input image by partially noising and then denoising it. The pipeline is governed by three core hyperparameters: ā¢ Number of steps. The total number of denoising iterations. ā¢ Strength. A value in [0,1][0,1] controlling the diffusion step from which denoising begins, effectively determining how much the output may deviate from the input. ā¢ Guidance scale. The influence of the text prompt on generation. Strength is the mechanism by which we control the similarity band of Section 3.1: to enforce sufficient distance between canary and neighbors, we consider only medium-to-high strength values, granting higher variance from the original image. To assess the potential of this strategy, we report the highest score across number of steps ā30,35,40ā\30,35,40\, strength ā0.55,0.65,0.75,0.85ā\0.55,0.65,0.75,0.85\, and guidance scale ā1,3,5ā\1,3,5\. For guidance scale 11 we use an empty prompt; for larger values we use a short class-representative prompt (e.g., āA dogā, āAn airplaneā). As shown in Section 4.5, this approach consistently outperforms both the baseline Steinke et al. (2023) and our first proposal. Figure 4. TPR at 0.1% FPR of our attack when querying synthetic neighbors against early training checkpoints, from a real example.Attack performance measured across successive training checkpoints for a representative sample. Rather than increasing monotonically, the TPR at 0.1\% FPR peaks during the early stages of training and subsequently declines. 3.4. Early Checkpoints To ease the distance trade-off of Section 3.1, we propose evaluating the neighborsā scores against an early checkpoint of the target model rather than the fully trained one. We call this the Early Checkpoint (E.C.) variant. Unlike our base attacks, this variant relaxes the black-box assumption, as it requires access to intermediate training snapshots rather than the fully trained model alone; it is therefore best suited to internal auditing settings. The intuition is that before memorization sets in, the model has acquired generalization but has not yet overfit specific training examples. At this stage the score landscape around a member canary resembles that of non-member samples, so even closer neighbors serve as good non-member representatives. Figure 4 corroborates this intuition: the attack TPR peaks in the early epochs and progressively declines as training continues. Since we cannot know a priori when memorization begins, we periodically snapshot the model during training and select the checkpoint yielding the highest privacy leakage. Crucially, the early checkpoint is used only to compute the neighbor scores that characterize ^1 D_1; the canary score is still evaluated against the fully trained model. In Algorithm 1 this amounts to replacing FtF_t with an early checkpoint exclusively in line 2. We note that this variant relaxes the black-box assumption: it requires access to intermediate training states. This is essentially free in the one-run auditing setting, where the auditor controls training and can save checkpoints at no cost, but it is a stronger assumption for an external attacker, for whom it applies only when such checkpoints are available. In Section 4.5 we show that, where applicable, E.C. consistently improves performance across all our methods. 4. Experiments Following Aerni et al. (2024), we adopt their model architectures and training pipelines, in accordance with their recommendation to evaluate privacy defenses on high-utility models. We evaluate each attack across three datasets and three privacy settings, resulting in nine experimental scenarios. In every scenario, the objective is to infer the membership of 500 canaries across 64 independently trained models. Canaries are randomly selected from the training set and assigned independently to half of the models. Each model is trained on all non-canary samples together with its assigned canaries. This assignment ensures that every canary appears equally often as a member and a non-member sample across the ensemble, enabling a balanced evaluation of membership inference performance. To increase memorization and maximize privacy leakage, the labels of all canaries are flipped prior to training. Figure 5. Settings where our Similarity Neighbors approach significantly outperforms Steinke et al. (2023).Three grouped bar charts comparing privacy leakage estimates from steinke2023privacy versus our Similarity Neighbors method. Left: empirical epsilon under DP-SGD on CIFAR-10, CIFAR-100, and CINIC-10, with our method 1.15x, 1.13x, and 1.30x higher respectively. Middle: TPR at 0.1\% FPR for undefended models on CIFAR-10 and CINIC-10, with our method 3.37x and 2.85x higher. Right: TPR at 0.1\% FPR under RelaxLoss on CIFAR-10 and CINIC-10, with our method 19.16x and 3.12x higher. In all panels, our approach yields higher leakage estimates than the baseline. Figure 6. Settings where our Synthetic Neighbors approach significantly outperforms our Similarity Neighbors approach.Three grouped bar charts comparing privacy leakage estimates from our Similarity Neighbors method versus our Synthetic Neighbors method. Left: TPR at 0.1\% FPR for undefended models on CIFAR-100 and CINIC-10, with Synthetic Neighbors 1.28x and 1.09x higher. Middle: TPR at 0.1\% FPR under RelaxLoss on CIFAR-100 and CINIC-10, with Synthetic Neighbors 1.40x and 1.67x higher. Right: TPR at 0.1\% FPR under DP-SGD on CIFAR-100 and CINIC-10, with Synthetic Neighbors 2.64x and 6.03x higher. In all panels, Synthetic Neighbors yields higher leakage estimates than Similarity Neighbors. 4.1. Datasets, Defenses and Metrics We conduct experiments on CIFAR-10, CIFAR-100, and CINIC-10 (Darlow et al., 2018), three widely used image-classification benchmarks in the privacy literature. To increase the distributional difference between CIFAR-10 and CINIC-10, we remove all CIFAR-10 images from CINIC-10, as their presence would otherwise introduce overlap between the two datasets and reduce the diversity of our evaluation. We then subsample the remaining data to 50k training and 10k test examples, matching the scale of CIFAR-10 while preserving class balance. Prior auditing work primarily focuses on estimating the differential privacy parameter ĻµĪµ and therefore evaluates attacks exclusively against DP-SGD (Steinke et al., 2023). Our objective, however, is not to obtain the tightest privacy guarantees, but rather to assess the quality of membership scores that can be computed without auxiliary models. Under this perspective, privacy leakage serves as a proxy for score quality rather than an end goal in itself. Consequently, the evaluation need not to be restricted to ĻµĪµ estimation or to differentially private models. We therefore consider three privacy regimes: undefended training, RelaxLoss (Chen et al., 2022), and DP-SGD, representing no protection, empirical leakage mitigation, and formal privacy guarantees, respectively. To evaluate privacy leakage, we report TPR at 0.1% FPR in all settings and, for DP-SGD models, the empirical ĻµĪµ. Following Carlini et al. (2022), we adopt TPR at low FPR because low false-positive rates are critical in realistic membership inference scenarios and provide a direct measure of attack effectiveness. For DP-SGD, we additionally estimate ĻµĪµ using the procedure of Steinke et al. (2023), evaluating all pairs k+,kāā10,20,ā¦,250k_+,k_-ā10,20,ā¦,250, retaining the tightest estimate for each model, and reporting the average across the 64-model ensemble. Since TPR at low FPR considers only the highest-scored members, while ĻµĪµ estimation aggregates over both members and non-members, the two metrics can diverge. Intuitively, ĻµĪµ captures average-case leakage: it reflects how well the attack discriminates overall across the score distribution. TPR at low FPR, by contrast, is a worst-case metric: it measures whether the attack can confidently identify any members, regardless of its behavior on the rest. To see why these can disagree, consider two attacks with identical ĻµĪµ estimates: one concentrates its correct predictions at the top of the ranking (high TPR at low FPR), the other at the bottom (near-zero TPR at low FPR). This asymmetry motivates reporting both metrics jointly in DP settings. 4.2. Similarity Neighbors In Figure 5, we present representative settings where our Similarity Neighbors approach significantly outperforms Steinke et al. (2023). Looking at both metrics, the results follow a consistent trend. Under DP-SGD , our attack achieves modest but meaningful improvements over the baseline in terms of empirical ĻµĪµ across all three datasets, suggesting that our scores are consistently stronger in the traditional one-round DP auditing setting. The TPR results reveal a particularly pronounced effect. Against undefended models, our attack substantially outperforms the baseline across both CIFAR-10 and CINIC-10, indicating that our method is far more effective at identifying members when no empirical defense is active. The gap becomes even more dramatic under RelaxLoss, where our attack achieves almost 20Ć20Ć higher TPR on CIFAR-10, demonstrating that our approach is considerably more robust to this defense than the baseline. Overall, these results highlight the importance of per-sample calibrated scores. They further confirm that strong calibrated signals can be effectively achieved within a one-round setting through neighbor-based approximations relying solely on test dataāwithout the need for shadow model training or auxiliary data collection. 4.3. Synthetic Neighbors Figure 6 reports a selection of settings in which Synthetic Neighbors outperforms Similarity Neighbors, focusing on the cases where the performance gap is most pronounced. Against undefended models, Synthetic Neighbors already yields a substantial advantage on CIFAR-100, reaching nearly 96% TPR, while the gap narrows on CINIC-10. Under RelaxLoss, Similarity Neighbors suffer a marked degradation across both datasets, whereas Synthetic Neighbors maintain considerably higher TPR, indicating greater resilience to this form of defense. The contrast is most striking under DP-SGD: despite both methods being heavily suppressed by the strong privacy guarantees, Synthetic Neighbors achieve disproportionately higher TPR across both datasets, pointing to a qualitative difference in attack effectiveness. Figure 7. Settings where our Early Checkpoint (E.C.) variants significantly outperform the standard attacks.Two rows of three grouped bar charts comparing standard attacks against early-checkpoint (E.C.) variants, for CIFAR-10, CIFAR-100, and CINIC-10. Top row: Similarity Neighbors versus Similarity Neighbors E.C. Left panel shows empirical epsilon under DP-SGD, with E.C. 1.03x, 1.07x, and 1.02x higher across the three datasets. Middle panel shows TPR at 0.1\% FPR under DP-SGD, with E.C. 1.35x, 2.93x, and 1.57x higher. Right panel shows TPR at 0.1\% FPR for undefended models, with E.C. 1.03x, 1.32x, and 1.11x higher. Bottom row: Synthetic Neighbors versus Synthetic Neighbors E.C., same three panel types. Left panel shows empirical epsilon under DP-SGD, with E.C. 1.28x, 1.29x, and 1.23x higher. Middle panel shows TPR at 0.1\% FPR under DP-SGD, with E.C. 1.27x, 2.03x, and 1.59x higher. Right panel shows TPR at 0.1\% FPR for an undefended model, with E.C. 1.03x, 1.03x, and 1.12x higher. In all panels, the early-checkpoint variant yields higher leakage estimates than the standard attack. These results highlight a consistent and defense-agnostic advantage of Synthetic Neighbors, which we attribute to the flexibility of the generative approach. Unlike Similarity Neighbors, whose effectiveness is inherently limited by the availability of similar samples in the test set, Synthetic Neighbors can produce informative neighbors for virtually any canary, making the attack robust to cases where the test distribution fails to provide neighbors of adequate quality. 4.4. Early Checkpoints The results reported in Figure 7 consistently demonstrate that the Early Checkpoint (E.C.) variants outperform their standard counterparts across both DP-SGD and undefended settings, both metrics, and all three datasets. For Similarity Neighbors, empirical epsilon improvements under DP-SGD are modest, with the E.C. variant providing only marginal gains across all datasets. The picture changes more clearly when looking at TPR at 0.1% FPR, where the E.C. variant achieves notably stronger results, particularly on CIFAR-100. In the undefended setting, the improvement is most dramatic on CIFAR-100, where the standard variant reaches 74.9% while the E.C. variant nearly saturates at 99.0%, with more moderate but still consistent gains on CIFAR-10 and CINIC-10. For Synthetic Neighbors, the E.C. variant delivers more substantial empirical epsilon improvements than in the similarity-based case, with consistent gains across all three datasets. TPR improvements under DP-SGD are consistent across all datasets, with CIFAR-100 again showing the largest relative gain. In the undefended setting, the E.C. variant approaches saturation on CIFAR-100 and delivers meaningful gains on CINIC-10, while CIFAR-10 shows a smaller margin due to the standard variant already performing strongly there. Taken together, these results make a compelling case for the E.C. variants. The gains are not isolated to a specific attack family, dataset, or defense regime ā they are systematic. The effect is especially pronounced in the most challenging settings, precisely where standard attacks struggle most. This robustness across diverse experimental conditions strongly supports querying neighbors against early checkpoints as a broadly effective strategy, consistently yielding stronger membership inference signals regardless of the underlying neighbor generation method. 4.5. Main Results Considering the results reported in the Sections 4.2, 4.3 and 4.4, Synthetic Neighbors E.C. emerges as our strongest proposal. In Tables 1 and 2, we therefore expand the direct comparison with Steinke et al. (2023), already anticipated in Figure 1. We further report results for LiRA, as upper bound on our method. In Table 1, we measure the privacy leakage against DP-SGD in terms of ĻµĪµ, showing that our attack yields improvements over the baseline ranging from roughly 0.5 to 0.6 across the datasets, while consistently retaining more than 50% of the LiRA performance. In Table 2, we measure the TPR at 0.1% FPR across all nine settings. Against undefended models, we exhibit only marginal performance loss compared to LiRA, outperforming Steinke et al. (2023) by a factor of ā¼3.5Ć 3.5Ć on both CIFAR-10 and CINIC-10. On CIFAR-100, the higher task complexity limits the maximum achievable test accuracy, leading to a more pronounced train-test accuracy gap, which makes overfitting particularly pronounced and drives nearly all attacks to saturate near 100% TPR. Against DP-SGD, despite the restricted privacy leakage, our approach still outperforms the baseline by a factor of approximately 4Ć4Ć on CIFAR-10, 2Ć2Ć on CIFAR-100, and 5Ć5Ć on CINIC-10. Against RelaxLoss, our method again substantially outperforms the baseline across all datasets, by a factor of 20Ć on CIFAR-10 and 5Ć on CINIC-10, while achieving an intermediate TPR between it and LiRA on CIFAR-100. Overall, Synthetic Neighbors E.C. consistently and significantly closes the gap to the LiRA upper bound across all privacy settings and datasets, substantially outperforming Steinke et al. (2023) in every setting. These results highlight that more powerful and calibrated MI scores directly translate into tighter empirical lower bounds on ĻµĪµ in the context of one-round DP auditing, and, more broadly, into a more accurate estimation of the true privacy leakage of a model. Table 1. Comparison between Steinke et al. (2023) (baseline), our Synthetic Neighbors E.C. attack and LiRA (upper bound) against DP-SGD, in terms of empirical ĻµĪµ. Empirical Epsilon Steinke et al. (2023) Ours LiRA DP-SGD CIFAR-10 1.29 1.78 2.99 CIFAR-100 1.24 1.88 3.16 CINIC-10 0.79 1.38 2.37 Table 2. Comparison between Steinke et al. (2023) (baseline), our Synthetic Neighbors E.C. attack and LiRA (upper bound) across all privacy settings, in terms of TPR at 0.1% FPR. TPR@0.1%FPR (%) Steinke et al. (2023) Ours LiRA Undefended CIFAR-10 23.38 84.98 97.03 CIFAR-100 99.28 99.03 99.73 CINIC-10 22.88 79.10 91.29 DP-SGD CIFAR-10 0.65 2.51 12.21 CIFAR-100 2.19 4.43 13.89 CINIC-10 0.33 1.80 4.43 RelaxLoss CIFAR-10 1.06 21.45 69.24 CIFAR-100 72.68 84.64 98.31 CINIC-10 10.96 53.73 78.46 5. Related Work 5.1. Reference-Based Membership Inference Attacks As anticipated in Section 2.2, Membership inference attacks (MIAs) have traditionally relied on shadow modelsāāalso referred to as reference modelsāāas a mechanism for approximating the behavior of a target model. The shadow-model paradigm was introduced by Shokri et al. (2017), who proposed training multiple auxiliary models on datasets drawn from the same distribution as the target model and subsequently using their outputs to train an attack classifier that distinguishes members from non-members. This work established the canonical black-box MIA framework and demonstrated that membership information can be inferred from model predictions alone. Subsequent work sought to improve both the efficiency and fidelity of this paradigm. Salem et al. (2018) showed that effective membership inference can be achieved with significantly fewer shadow models and even without a dedicated attack classifier, suggesting that the core membership signal lies in the statistical differences between member and non-member predictions rather than in the complexity of the attack model itself. More recently, Li et al. (2022) proposed ā -Leaks, arguing that posterior probabilities discard valuable information and showing that shadow models trained to characterize the distribution of pre-softmax logits can substantially improve attack performance. These developments reflect a broader shift from heuristic attack-classifier pipelines toward statistically grounded methods that explicitly model the distributions induced by training membership. This evolution culminated in the Likelihood Ratio Attack (LiRA) (Carlini et al., 2022), which formulates membership inference as a hypothesis-testing problem. By explicitly modeling uncertainty and focusing on the low-false-positive-rate regime, LiRA consistently outperforms previous shadow-model-based attacks and is widely regarded as the state-of-the-art reference-model attack, making it the primary benchmark against which modern membership inference methods are evaluated. Despite providing stronger membership signals, Online LiRA comes at a higher computational cost than Offline LiRA, as it relies also on in modelsāmodels trained on the target sample. Building on the notation introduced in Section 2.3, alongside the set of out shadow models F1,ā¦,Fn\F_1,ā¦,F_n\, Online LiRA trains a set of in shadow models F1ā²,ā¦,Fnā²\F _1,ā¦,F _n\ to approximate the in score distribution as a Gaussian (6) 0=ā”(Ī¼in,Ļin2),D_0\;=\;N\! ( _in,\, _in^2 ), with (7) Ī¼in=1nāāi=1nsā”(Fiā²,x),Ļin2=1nāāi=1n(sā”(Fiā²,x)āĪ¼in)2. _in= 1n _i=1^ns(F _i,x), _in^2= 1n _i=1^n (s(F _i,x)- _in )^2. Membership is then assessed by the targetās score likelihood ratio between 0D_0 and 1D_1: (8) Īā”(x)=ā”(sā”(Ft,x),Ī¼in,Ļin2)ā”(sā”(Ft,x),Ī¼out,Ļout2) (x)\;=\; N\! (s(F_t,x); _in,\, _in^2 )N\! (s(F_t,x); _out,\, _out^2 ) 5.2. Reference-Free Membership Inference Attacks The high computational cost of shadow model training has motivated a parallel line of work seeking membership signals that require no reference models. The simplest such approaches are metric-based, exploiting statistics directly computable from the target model. Yeom et al. (2018) proposed thresholding on the modelās loss, exploiting the observation that members tend to incur lower loss than non-members. Bertran et al. (2023) proposed an alternative route via quantile regression, training a lightweight auxiliary model to directly estimate the target modelās score distribution without any shadow training, reporting competitive performance against LiRA at low false-positive rates. For language models specifically, Carlini et al. (2021) refined loss-based inference by normalizing perplexity against zlib entropy, partially correcting for the intrinsic complexity of each sample, while token-level methods such as Min-K% (Shi et al., 2024) and Min-K%++ (Zhang et al., 2025) aggregate statistics over the lowest-probability tokens, grounded in the intuition that non-members are more likely to contain outlier tokens. Beyond general-purpose attacks, shadow-free methods have also been proposed for specific settings, including recommender systems (Chi et al., 2024) and federated learning (Deng et al., 2026). 5.3. One-Round DP Audits Although reference-free attacks reduce the per-attack cost, reliably estimating the privacy leakageāsuch as TPR at low FPRāstill requires evaluating a large number of samples across multiple models. One-round auditing addresses this orthogonal bottleneck in DP settings by shifting the auditing target to a formal ĻµĪµ lower bound, constraining the access to a single trained target model. While averaging over multiple runs yields more representative estimates, a statistically valid ĻµĪµ bound can in principle be obtained from a single training run. Steinke et al. (2023) first introduced ORA by auditing randomly included canary points into a single training run, and converting membership guesses into an ĻµĪµ lower bound via a binomial tail bound under (Ļµ,Ī“)(Īµ,Ī“)-DP. Mahloujifar et al. (2024) tightened this by grounding the conversion in the full f-DP trade-off curve and Xiang et al. (2025) further derived the theoretically optimal membership decoder under f-DP via order statistics over per-canary privacy loss scores. Despite these advances largely contributing to refine the statistical procedure that converts membership guesses to privacy lower bounds, the membership signal itself has been largely taken for grantedātypically model loss or gradient projections. On the theoretical side, Keinan et al. (2026) show that interference between canary signals is the dominant barrier to tight auditing in DP-SGD, directly motivating the design of signals with higher per-canary distinguishability. Notably, Liu et al. (2022) substitute the loss-based signal with a quantile regression-based membership inference attack, building on previous work from Bertran et al. (2023). Rather than relying on a global loss threshold, their method trains an auxiliary model to predict sample-specific score quantiles, yielding per-example membership decisions that better account for heterogeneity across canaries and reporting substantial gains in black-box settings. However, this still requires training an auxiliary model, albeit one that is independent of the target and can be kept arbitrarily small. In contrast, our method requires no additional model training whatsoever, bridging the gap between the statistical power of reference-based attacks and the computational efficiency demanded by one-run auditing. 5.4. Neighbors and synthetic data in Membership Inference Attacks The use of neighboring or synthetic samples is not new in the MIA literature. Early works leveraged generative models to compensate for the lack of access to the target modelās training distribution. For instance, Zhang et al. (2022) enrich the available data with synthetic samples, enabling the training of shadow models even when only limited information about the original training set is available. Beyond data augmentation, synthetic samples can also be used to probe the local behavior of a model around a target point. Wen et al. (2023) argue that a target sample alone provides limited information for membership inference. They therefore employ adversarial tools to optimize the generation of diverse perturbations around a canary sample, allowing for a more informative exploration of its surrounding region and improving inference performance. This intuition has recently gained particular attention also in the context of MIAs against large language models (LLMs), where neighboring samples are used as reference points to characterize model behavior. Mattern et al. (2023) observe that reference-model-based attacks rely on the often unrealistic assumption of access to samples drawn from the training distribution. To overcome this limitation, they compare the modelās score on the target sample with scores obtained on synthetically generated neighboring texts, eliminating the need for reference data. Similarly, Galli et al. (2024) exploit neighboring samples by querying noisy variants of the target instance, obtained through stochastic perturbations in the embedding space at inference time. Finally, Mozaffari and Marathe (2024) extend this idea by generating semantically perturbed versions of the target text and training a neural classifier to distinguish members from non-members based on the modelās responses across these perturbations. 6. Limitations As discussed in Section 3, the core difficulty underlying our approach is that the optimal distance between a canary and its neighbors depends on canary- and model-specific factors that are not known a priori, and that cannot be resolved without membership labels. We are unable to formulate a metric that reliably quantifies neighbor quality ahead of time; instead, we rely on constructions (similarity-based retrieval, diffusion strength, early checkpoints) that implicitly control this trade-off. Developing a principled, canary-specific criterion for neighbor quality remains an open problem, and we see it as the most promising direction to further close the gap to reference-based attacks. Beyond this core limitation, each of our proposed constructions carries its own constraints. Since Similarity Neighbors are retrieved from the test set, the attackās effectiveness for a given canary is inherently limited by whether the test set contains samples that lie in an adequate similarity band around it; when no such neighbors exist, the estimated ^1 D_1 is a poor approximation and the attack underperforms. Synthetic Neighbors, in turn, require a generative model capable of producing suitable variations of the input, which may not be readily available or effective for all data modalities or tasks, and depend on hyperparameters whose optimal range must be explored per setting; this exploration can be more or less costly depending on how thoroughly it is performed. As noted in Section 3.4, the Early Checkpoint variant further relaxes the black-box assumption, as it requires access to intermediate training checkpoints that are not generally available to an external attacker; it is therefore best suited to internal auditing settings, where the auditor controls training and checkpoints come at no additional cost, rather than to external attacks. Finally, the scale of our models and experiments is limited, for two reasons. First, prior work on one-round DP auditing, including the baseline (Steinke et al., 2023), evaluates on datasets and models of similar scale, so our choice keeps our results directly comparable to existing literature. Second, our evaluation protocol requires training 64 independent models across nine experimental scenarios to obtain reliable estimates of attack performance; replicating this protocol at the scale of large foundation models would demand computational resources far beyond what is feasible in this work. We leave the evaluation of our method on larger-scale models, including LLMs to future work. 7. Conclusion In this work, we introduce two novel classes of black-box membership inference attacks that require no additional model training, making them suitable for one-round audits. While prior work on one-round auditing focuses on refining the procedure that estimates the privacy leakage from membership scores, we address a complementary and largely unexplored direction: the design of powerful membership scores that require no auxiliary models. We propose to approximate the out score distribution of Offline LiRA by querying neighbors of the canary on the target model alone, and instantiate this idea in two ways: the Similarity Neighbors attack, which retrieves neighbors from the test set using a CLIP-based similarity metric, and the Synthetic Neighbors attack, which generates neighbors via the image-to-image pipeline of Stable Diffusion. We further propose an Early Checkpoint variant of both attacks, which evaluates neighbors on intermediate snapshots of the target model to mitigate the effect of memorization on the in canaries. Our experimental evaluation across three datasets and three privacy defenses demonstrates that both proposals consistently outperform Steinke et al. (2023). The Synthetic Neighbors attack and its Early Checkpoint variant emerge as the strongest attacks in virtually every setting, achieving particularly large TPR gains in the DP-SGD setting and maintaining strong performance under Undefended and RelaxLoss scenarios. The Similarity Neighbors attack, on the other hand, provides a simpler yet powerful alternative that requires no additional data generation, making it an attractive option when synthetic generation is not feasible. More broadly, our results suggest that meaningful approximations of state-of-the-art MIAs are achievable in one-round settings. We hope this work encourages further investigation into model-free membership inference attacks, and contributes to a more comprehensive understanding of one-round audits beyond the DP-SGD setting. References Aerni et al. (2024) M. Aerni, J. Zhang, and F. TramĆØr Evaluations of machine learning privacy defenses are misleading. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ā24, p. 1271ā1284. External Links: Link, Document Cited by: Ā§4. Bai et al. (2025) L. Bai, Q. Ye, X. Zhang, S. Zhang, Z. Liang, J. Xu, and H. Hu Toward efficient inference attacks: shadow model sharing via mixture-of-experts. External Links: 2510.13451, Link Cited by: Ā§1. Bertran et al. (2023) M. Bertran, S. Tang, M. Kearns, J. Morgenstern, A. Roth, and Z. S. Wu Scalable membership inference attacks via quantile regression. External Links: 2307.03694, Link Cited by: Ā§5.2, Ā§5.3. Carlini et al. (2022) N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer Membership inference attacks from first principles. External Links: 2112.03570, Link Cited by: Ā§1, Ā§1, Ā§2.2, Ā§2.3, Ā§2.4, Ā§4.1, Ā§5.1. Carlini et al. (2021) N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel Extracting training data from large language models. External Links: 2012.07805, Link Cited by: Ā§5.2. Chen et al. (2022) D. Chen, N. Yu, and M. Fritz RelaxLoss: defending membership inference attacks without losing utility. External Links: 2207.05801, Link Cited by: Ā§4.1. Chi et al. (2024) X. Chi, X. Zhang, Y. Wang, L. Qi, A. Beheshti, X. Xu, K. R. Choo, S. Wang, and H. Hu Shadow-free membership inference attacks: recommender systems are more vulnerable than you thought. External Links: 2405.07018, Link Cited by: Ā§5.2. Darlow et al. (2018) L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey CINIC-10 is not imagenet or cifar-10. External Links: 1810.03505, Link Cited by: Ā§4.1. DeepSeek-AI et al. (2025) DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai, J. Liang, J. Guo, J. Ni, J. Li, J. Wang, J. Chen, J. Chen, J. Yuan, J. Qiu, J. Li, J. Song, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Xu, L. Xia, L. Zhao, L. Wang, L. Zhang, M. Li, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, N. Tian, P. Huang, P. Wang, P. Zhang, Q. Wang, Q. Zhu, Q. Chen, Q. Du, R. J. Chen, R. L. Jin, R. Ge, R. Zhang, R. Pan, R. Wang, R. Xu, R. Zhang, R. Chen, S. S. Li, S. Lu, S. Zhou, S. Chen, S. Wu, S. Ye, S. Ye, S. Ma, S. Wang, S. Zhou, S. Yu, S. Zhou, S. Pan, T. Wang, T. Yun, T. Pei, T. Sun, W. L. Xiao, W. Zeng, W. Zhao, W. An, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, X. Q. Li, X. Jin, X. Wang, X. Bi, X. Liu, X. Wang, X. Shen, X. Chen, X. Zhang, X. Chen, X. Nie, X. Sun, X. Wang, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yu, X. Song, X. Shan, X. Zhou, X. Yang, X. Li, X. Su, X. Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Y. Zhang, Y. Xu, Y. Xu, Y. Huang, Y. Li, Y. Zhao, Y. Sun, Y. Li, Y. Wang, Y. Yu, Y. Zheng, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Tang, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Wu, Y. Ou, Y. Zhu, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Zha, Y. Xiong, Y. Ma, Y. Yan, Y. Luo, Y. You, Y. Liu, Y. Zhou, Z. F. Wu, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Huang, Z. Zhang, Z. Xie, Z. Zhang, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Xu, Z. Wu, Z. Zhang, Z. Li, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Gao, and Z. Pan DeepSeek-v3 technical report. External Links: 2412.19437, Link Cited by: Ā§1. Deng et al. (2026) G. Deng, S. Chen, Y. Luo, Y. Liu, S. Wang, Z. Cai, L. Liu, X. Jia, and S. Fu Toward efficient membership inference attacks against federated large language models: a projection residual approach. External Links: 2604.21197, Link Cited by: Ā§5.2. Du et al. (2025) Y. Du, J. Li, Y. Chen, K. Zhang, Z. Yuan, H. Xiao, B. Ribeiro, and N. Li Cascading and proxy membership inference attacks. External Links: 2507.21412, Link Cited by: Ā§1. Dwork et al. (2006) C. Dwork, F. McSherry, K. Nissim, and A. Smith Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference, p. 265ā284. Cited by: Ā§2.4. Galli et al. (2024) F. Galli, L. Melis, and T. Cucinotta Noisy neighbors: efficient membership inference attacks against llms. External Links: 2406.16565, Link Cited by: Ā§5.4. Keinan et al. (2026) A. Keinan, M. Shenfeld, and K. Ligett How well can differential privacy be audited in one run?. External Links: 2503.07199, Link Cited by: Ā§5.3. Li et al. (2022) S. Li, Y. Wang, Y. Li, and Y. Tan L-leaks: membership inference attacks with logits. External Links: 2205.06469, Link Cited by: Ā§5.1. Liu et al. (2022) Y. Liu, Z. Zhao, M. Backes, and Y. Zhang Membership inference attacks by exploiting loss trajectory. External Links: 2208.14933, Link Cited by: Ā§5.3. Mahloujifar et al. (2024) S. Mahloujifar, L. Melis, and K. Chaudhuri Auditing f-differential privacy in one run. External Links: 2410.22235, Link Cited by: Ā§1, Ā§5.3. Mattern et al. (2023) J. Mattern, F. Mireshghallah, Z. Jin, B. Schƶlkopf, M. Sachan, and T. Berg-Kirkpatrick Membership inference attacks against language models via neighbourhood comparison. External Links: 2305.18462, Link Cited by: Ā§5.4. Mozaffari and Marathe (2024) H. Mozaffari and V. J. Marathe Semantic membership inference attack against large language models. External Links: 2406.10218, Link Cited by: Ā§5.4. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever Learning transferable visual models from natural language supervision. External Links: 2103.00020, Link Cited by: Ā§3.2. Rombach et al. (2022) R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer High-resolution image synthesis with latent diffusion models. External Links: 2112.10752, Link Cited by: Ā§3.3. Salem et al. (2018) A. Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes ML-leaks: model and data independent membership inference attacks and defenses on machine learning models. External Links: 1806.01246, Link Cited by: Ā§5.1. Shi et al. (2024) W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer Detecting pretraining data from large language models. External Links: 2310.16789, Link Cited by: Ā§5.2. Shokri et al. (2017) R. Shokri, M. Stronati, C. Song, and V. Shmatikov Membership inference attacks against machine learning models. External Links: 1610.05820, Link Cited by: Ā§1, Ā§1, Ā§2.1, Ā§2.2, Ā§5.1. Steinke et al. (2023) T. Steinke, M. Nasr, and M. Jagielski Privacy auditing with one (1) training run. External Links: 2305.08846, Link Cited by: Table 3, Table 3, Table 3, Table 4, Table 4, Table 4, Figure 1, Figure 1, Ā§1, Ā§1, Ā§1, Ā§2.4, Ā§3.2, Ā§3.3, Figure 5, Figure 5, Ā§4.1, Ā§4.1, Ā§4.2, Ā§4.5, Ā§4.5, Ā§4.5, Table 1, Table 1, Table 1, Table 2, Table 2, Table 2, Ā§5.3, Ā§6, Ā§7. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. RoziĆØre, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample LLaMA: open and efficient foundation language models. External Links: 2302.13971, Link Cited by: Ā§1. Vaswani et al. (2023) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin Attention is all you need. External Links: 1706.03762, Link Cited by: Ā§1. Wen et al. (2023) Y. Wen, A. Bansal, H. Kazemi, E. Borgnia, M. Goldblum, J. Geiping, and T. Goldstein Canary in a coalmine: better membership inference with ensembled adversarial queries. External Links: 2210.10750, Link Cited by: Ā§5.4. Xiang et al. (2025) Z. Xiang, T. Wang, H. Xiao, Y. Tian, and D. Wang Tight privacy audit in one run. External Links: 2509.08704, Link Cited by: Ā§1, Ā§5.3. Yeom et al. (2018) S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha Privacy risk in machine learning: analyzing the connection to overfitting. External Links: 1709.01604, Link Cited by: Ā§2.1, Ā§2.2, Ā§5.2. Zhang et al. (2025) J. Zhang, J. Sun, E. Yeats, Y. Ouyang, M. Kuo, J. Zhang, H. F. Yang, and H. Li Min-k%++: improved baseline for detecting pre-training data from large language models. External Links: 2404.02936, Link Cited by: Ā§5.2. Zhang et al. (2022) Y. Zhang, H. Zhou, P. Wang, and G. Yang Black-box based limited query membership inference attack. IEEE Access 10, p. 1ā1. External Links: Document Cited by: Ā§5.4. Empirical Epsilon TPR@0.1%FPR (%) DP-SGD CIFAR-10 CIFAR-100 CINIC-10 CIFAR-10 CIFAR-100 CINIC-10 Steinke et al. (2023) 1.2867 1.2427 0.7917 0.6500 2.1875 0.3312 Similarity Neighbors 1.4851 1.4062 1.0305 0.5250 0.8250 0.1875 Sim. Neighbors E.C. 1.5294 1.4981 1.0504 0.7063 2.4188 0.2938 Synthetic Neighbors 1.3926 1.4549 1.1216 1.9688 2.1812 1.1312 Synt. Neighbors E.C. 1.7832 1.8837 1.3849 2.5063 4.4250 1.8000 LiRA 2.9934 3.1560 2.3675 12.2125 13.8875 4.4250 Table 3. Experimental results against DP-SGD, across CIFAR-10, CIFAR-100 and CINIC-10. We measure both ĻµĪµ and TPR at 0.1% FPR. Comparison between Steinke et al. (2023) (baseline), our Similarity and Synthetic Neighbors attacks with their Early Checkpoint (E.C.) variants and LiRA (upper bound). Undefended RelaxLoss TPR@0.1%FPR (%) CIFAR-10 CIFAR-100 CINIC-10 CIFAR-10 CIFAR-100 CINIC-10 Steinke et al. (2023) 23.3813 99.2812 22.8813 1.0563 72.6750 10.9625 Similarity Neighbors 78.9062 74.8875 65.2125 20.2437 61.9687 34.1500 Sim. Neighbors E.C. 81.1063 99.0062 72.4125 20.9663 82.0187 34.4750 Synthetic Neighbors 82.6375 95.8625 70.7562 19.9875 86.6188 57.0500 Synt. Neighbors E.C. 84.9813 99.0250 79.1000 21.4500 84.6361 53.7312 LiRA 97.0250 99.7313 91.2937 69.2375 98.3062 78.4563 Table 4. Experimental results against undefended models and RelaxLoss, across CIFAR-10, CIFAR-100 and CINIC-10. We evaluate the TPR at 0.1% FPR. Comparison between Steinke et al. (2023) (baseline), our Similarity and Synthetic Neighbors attacks with their Early Checkpoint (E.C.) variants and LiRA (upper bound). Appendix A Open Science The source code and scripts required to reproduce our experiments are available on GitHub at ethz-spylab/one_round_MI. Instructions about how to get started and the structure of the repository are contained in the README.md file. In Tables 3 and 4 we report all our experimental results. Appendix B Use of Generative AI As stated in Section 3.3, we used the stable-diffusion-v1-5 generative model to perform our attack relying on synthetic neighbors. Moreover, the writing of this paper has been assisted by LLMs.