Paper deep dive

Subgroup Membership Inference Audits of Differentially Private Synthetic Text

Yidan Sun, Viktor Schlegel, Srinivasan Nandakumar, Siew Kei Lam, Anil Anthony Bharath

Year: 2026Venue: arXiv preprintArea: cs.CRType: PreprintEmbeddings: 131

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 9/10/2026, 3:25:11 AM

Summary

This paper introduces a subgroup-targeted membership inference audit framework for differentially private (DP) synthetic text. It demonstrates that while DP reduces average privacy leakage, it fails to protect high-risk subgroups uniformly. Leakage is concentrated in a small fraction of records (top decile), and which records leak depends on the specific release mechanism rather than intrinsic record attributes, challenging the ability to triage risk based on data alone.

Entities (7)

Differential Privacy · privacy-mechanism · 98%Membership Inference Attack · attack-method · 97%Synthetic Text · data-type · 95%Subgroup-Targeted Membership Inference Game · methodology · 93%DP-SGD · generator · 92%Canary · attack-method · 90%RMIA · attack-method · 90%

Relation Signals (6)

Differential Privacy → reduces → average privacy leakage

confidence 95% · DP is effective at the aggregate level: it substantially reduces average leakage at every budget we test.

Privacy Leakage → concentratesin → top decile of records

confidence 94% · under DP, a tenth of the records carries roughly 40% of it.

Record Leakage Risk → dependson → release mechanism

confidence 93% · which records leak proves to be a property of the release mechanism rather than of the record alone

Differential Privacy → failstoprotectuniformly → high-risk subgroups

confidence 92% · the protection DP delivers in practice is uneven... the noise removes more of the measured leakage from random records than from high-risk ones

RMIA → assumes → model access

confidence 88% · These attacks assume stronger adversary knowledge: a release-based adversary lacks access to the generator

Canary → evaluates → artificial sequences

confidence 85% · Canary-based attacks... insert crafted worst-case sequences... evaluate the privacy leakage risk of synthetic examples instead of naturally occurring ones

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Synthetic data releases are increasingly proposed in the literature as a means of sharing realistic data replicas in lieu of sensitive private datasets. Even when the worst-case privacy leakage of such releases is bounded by means of differential privacy (DP), in practice a residual risk remains. Membership inference attack (MIA) audits are conducted to empirically quantify this risk. However, existing methods only measure average-case risk for randomly drawn records, which might conceal the risk to vulnerable subgroups. To highlight this issue, we define a subgroup-targeted membership inference game in which the target pool is an explicit parameter, and instantiate it with an audit of 32 proxies under three scenarios with different levels of attacker knowledge, across four datasets, three generators (DP-SGD fine-tuning, API-based prompting, and activation steering), and five privacy budgets. The audit shows that synthetic releases leak subgroup membership and that prior attacks systematically underestimate this leakage. DP is effective at the aggregate level: it substantially reduces average leakage at every budget we test. Three observations temper this picture. First, the remaining leakage is concentrated rather than spread out: under DP, a tenth of the records carries roughly 40% of it. Second, the protection DP delivers in practice is uneven: within its worst-case guarantee, the noise removes more of the measured leakage from random records than from high-risk ones---and a merged-pool audit that scores both record types against shared negatives confirms this at the record level. Third, \emph{which} records leak proves to be a property of the release mechanism rather than of the record alone, so record-level risk cannot be assessed independently of the release.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

127,679 characters extracted from source content.

Expand or collapse full text

Subgroup Membership Inference Audits of Differentially Private Synthetic Text Yidan Sun1 Affiliation: * Viktor Schlegel1 Srinivasan Nandakumar1 Siew Kei Lam2 Anil Anthony Bharath1 Affiliation: 3 Abstract Synthetic data releases are increasingly proposed in the literature as a means of sharing realistic data replicas in lieu of sensitive private datasets. Even when the worst-case privacy leakage of such releases is bounded by means of differential privacy (DP), in practice a residual risk remains. Membership inference attack (MIA) audits are conducted to empirically quantify this risk. However, existing methods only measure average-case risk for randomly drawn records, which might conceal the risk to vulnerable subgroups. To highlight this issue, we define a subgroup-targeted membership inference game in which the target pool is an explicit parameter, and instantiate it with an audit of 32 proxies under three scenarios with different levels of attacker knowledge, across four datasets, three generators (DP-SGD fine-tuning, API-based prompting, and activation steering), and five privacy budgets. The audit shows that synthetic releases leak subgroup membership and that prior attacks systematically underestimate this leakage. DP is effective at the aggregate level: it substantially reduces average leakage at every budget we test. Three observations temper this picture. First, the remaining leakage is concentrated rather than spread out: under DP, a tenth of the records carries roughly 40% of it. Second, the protection DP delivers in practice is uneven: within its worst-case guarantee, the noise removes more of the measured leakage from random records than from high-risk ones—and a merged-pool audit that scores both record types against shared negatives confirms this at the record level. Third, which records leak proves to be a property of the release mechanism rather than of the record alone, so record-level risk cannot be assessed independently of the release. September 9, 2026 Keywords Privacy Auditing ⋅· Membership Inference ⋅· Differential Privacy ⋅· Synthetic Text 1 Introduction Figure 1: Overview: a pool-parameterized membership game (left), the audit instantiating it (middle), and the central finding (right) — DP removes less leakage from atypical records than from random ones, and the residual concentrates on a top decile that no record attribute predicts. Synthetic data bear the potential to be used as a privacy-preserving substitute for direct data sharing in high-stakes domains such as healthcare [1], finance [2, 3], and law [4, 5] By releasing generated records instead of the original dataset, practitioners hope to retain utility while reducing disclosure risk [6]. However, synthetic data are not inherently safe: when the generation pipeline overfits or memorizes training examples, the released synthetic dataset can still reveal whether particular records were used in training [7, 8]. This risk is most fundamentally quantified empirically via Membership Inference Attacks (MIA) [9]: an adversary is tasked to determine, given a target record and the output of a computation (i.e., synthetic dataset) whether the target was used in the computation (i.e. membership). Repeating this over an evaluation set of target records indicates the average ability of an adversary to distinguish members from non-members [10, 11]. Reducing (or bounding) the adversary’s MIA capability gives a notion of plausible deniability: if it is impossible to tell whether an individual’s data was used in the computation, all subsequent inferences regarding that data (e.g., attribute inference) might not hold. Conducting MIA audits on synthetic texts is hard: an attacker needs to extract information from a whole dataset (i.e. hundred of individual records) and for unstructured data like text, no obvious aggregate statistics, as is the case for e.g., tabular data [6], exist. Therefore existing privacy studies for synthetic data tend to omit membership inference audits [1, 12]. A tempting alternative is to evaluate membership inference of random private records on the model that was trained to generate synthetic data, such as the state-of-the-art RMIA [11] attack. These attacks assume stronger adversary knowledge: a release-based adversary lacks access to the generator, and cannot be applied to training-free synthesis methods [13, 14]. Canary-based attacks [7], meanwhile, insert crafted worst-case sequences (potentially multiple times), which, while yielding upper bounds on memorization, evaluate the privacy leakage risk of synthetic examples instead of naturally occurring ones. However, sampling random (or artificial) target records reports a generic average-case risk and can miss an important detail: privacy leakage may be concentrated on a small, high-risk subset of records: samples that are unusually distinctive relative to the background distribution [15, 16, 17], rather than distributed uniformly across the population. Low empirical MIA scores in average-case scenarios may therefore underestimate privacy harms for such outlier samples. To investigate this issue, we study subgroup-targeted membership inference for synthetic data releases. We formalize a membership inference game parameterized by the target pool: target records are drawn from a designated pool, chosen based on a desired characteristic. The rest of the private data to synthesise from are drawn from its complement. Because the pool can be arbitrarily defined, the same formulation can be used to conduct the high-risk subgroups audits, random control audits, and—by taking unions of pools—a design in which different kinds of records are scored against a common reference distribution. This formulation isolates whether pool membership creates disproportionate inference risk, and is particularly relevant for privacy auditing, where concentrated-risk failures matter more than average-case results. To conduct a rigorous membership inference audit within this framework, we adapt normalized-evidence membership inference attacks [7, 11] to synthetic data releases: The exact likelihood-ratio test—how much more probable the observed release dataset is, when the target is a member —is intractable to calculate for a whole dataset (rather than model weights): therefore we replace it with easily measurable proxy scores, such as n-gram overlap and embedding distance, all normalized by scenario-dependent calibration. This allows us to cleanly separate the evidence of proxy, which is purely extracted from releases (lexical or embedding-based), for a target record, from scenario-specific calibration, which normalizes that evidence according to the attacker’s knowledge, and the same proxy family can be calibrated under progressively stronger attacker capabilities. This allows us to model attacker capability as an explicit axis of the audit: We study three release-based attacker scenarios—a release-only attacker, a raw-reference attacker, and a generator-assisted release attacker—to measure how much membership leakage is inferable from the release alone and what auxiliary calibration and generator access can add. Finally, because harm may be concentrated within a pool, we complement aggregate metrics with per-record vulnerability scores obtained from repeated attack episodes, and ask whether a data holder could predict, from intrinsic record attributes, for which records membership is more likely to be inferred. Our experiments across four datasets, three DP text generators, and five privacy budgets show that this risk is real and structured: it concentrates on few records, differs across pools, and depends on the mechanism. On high-risk records, the best attacker reaches AUC 0.790.79–0.830.83, identifying 34–61% of true members at a false-positive rate below 5% when the generator is non-private, 39–52% of them exposed to an attacker that sees nothing but the release, and leakage persists above chance at every DP budget (AUC 0.550.55–0.650.65, TPR up to ≈ 18% against a 5% chance rate). DP is effective in aggregate—average AUC falls substantially at every budget. But its protection is uneven in two ways. Across pools, DP removes less leakage from high-risk records than from random ones: without DP, targeting a high-risk pool buys the attacker little or nothing, yet under every DP budget we test, the high-risk pools remain more distinguishable than random controls. And within a pool, the residual leakage is concentrated: a tenth of the records carries roughly 40% of the per-record vulnerability—up to 72%—at every ε , though which records occupy that top decile changes from release to release. Contributions. • A pool-parameterized membership game whose target pool is an explicit parameter, instantiating high-risk audits, random controls, and union pools within one formulation (§3.2). • A comprehensive release-based audit: 32 proxies, three attacker scenarios, four datasets, three generators, five budgets. Leakage persists above chance at every budget, added attacker knowledge reshuffles which proxy wins rather than how much leaks, and no prior fixed attack matches the audit (best AUC in 69/78 cells) (§5.1–5.3). • DP’s protection is uneven: it removes less leakage from high-risk pools than from random controls, and a tenth of the records carries ∼40% 40\% of what remains at every ε (§5.4, §5.6). • Which records leak is a property of the mechanism, not the record: vulnerability does not transfer across generators and no intrinsic attribute predicts it, so risk cannot be triaged from the data alone and must be audited per release (§5.5). • A two-stage audit protocol. Within-pool scores are only meaningful relative to their own pool, so we screen with full within-pool games and adjudicate flagged cells with a merged-pool audit that scores all record types against shared negatives (§5.6). 2 Related Work Membership inference against models. Membership inference attacks (MIA) ask whether a specific record was part of a model’s training set, and have become a standard empirical measure of privacy leakage since the shadow-model formulation of Shokri et al. [9]. Carlini et al. [10] recast MIA from first principles, showing that per-example calibration and evaluation at low false-positive rates (rather than average accuracy) are essential to expose high-confidence leakage; LiRA and its successors operationalize this through likelihood-ratio tests against shadow models. RMIA [11] reduces the cost of this approach by normalizing a target’s membership evidence against a model-averaged baseline that approximates the marginal probability of the record, yielding strong attacks with far fewer reference models. Our work inherits RMIA’s normalized-evidence view and Carlini et al.’s [10] emphasis on low-FPR evaluation, but departs from the model-access assumption common to this line: our adversary never queries a model or observes its likelihoods, and instead reasons only from a released synthetic dataset. Privacy of synthetic data. A parallel line studies whether releasing synthetic records, rather than a trained model, protects the underlying data. Stadler et al. [6] showed that synthetic tabular data does not provide privacy for free and can remain vulnerable to inference attacks, and subsequent frameworks such as TAPAS [18] and density-based attacks like DOMIAS [19] formalized membership inference directly against synthetic releases, typically by comparing a target’s local density under the synthetic distribution to that under a reference distribution. These works focus on tabular data and average-case risk. We adopt the same release-based threat model—the adversary sees only synthetic outputs and auxiliary reference data—but target text generation, high-risk subgroups, and the low-FPR regime rather than aggregate tabular attackability. Auditing synthetic text and canaries. For text, memorization is commonly audited by inserting crafted canaries and measuring their exposure [16]. Canary [7] adapts normalized-evidence MIA to LLM-generated synthetic text: because release likelihoods are unavailable, it replaces them with tractable proxy signals computed from synthetic outputs, and amplifies detectability by repeating each canary nrep>1n_rep>1 times. This yields a valuable upper bound on worst-case memorization, but measures the exposure of artificial sequences designed to be memorized rather than of real records. Our audit differs in target and intent: we insert naturally occurring subgroup records once, so the reported leakage reflects the exposure of genuine members of a high-risk population, and we study a family of heterogeneous proxies together with scenario-dependent calibration rather than a fixed signal. Closest to our setting, SYNBENCH [8] benchmarks DP text generators with a release-based membership attack on natural records, sharing our IN/OUT reference construction, but audits average-case risk with a single dataset-level n-gram signal and aggregate AUC. We instead make concentrated risk the object of the audit: a subgroup-targeted lone-record game with matched random controls, heterogeneous proxies under explicit attacker-capability scenarios, and evaluation at low-FPR operating points and the level of individual records. Differentially private text generation. Differential privacy [20] is the standard defense for private data synthesis. Two design points are representative of current DP text generators. DP-SGD–based methods [21] fine-tune language models with clipped, noised gradients and then sample synthetic text, enforcing privacy during training. Inference-based methods such as Aug-PE [13] instead privatize corpus statistics and prompt a foundation model to match them, enforcing privacy through the released statistics rather than model training. We audit both, across a range of budgets, and connect our game to DP through its add-one-record adjacency relation, which upper-bounds any adversary’s achievable ROC at a given budget. Subgroup, outlier, and disparate memorization. A growing body of work shows that memorization and privacy risk are not uniform across a dataset. Outliers and atypical examples are memorized more readily [15, 16], and privacy harms and DP’s utility costs can fall unevenly across subgroups [17]. Most closely related, prior MIA analyses have observed that average-case metrics can understate the risk to the most vulnerable records [10, 22]. We build on this observation but make the subgroup the explicit object of the audit: we define a high-risk subgroup, formalize a subgroup-targeted release-based game with a matched random-target control, and characterize both whether the subgroup is disproportionately exposed and how that exposure is distributed within the subgroup. To our knowledge, this concentrated-risk, release-based formulation for synthetic text has not been studied under an explicit range of attacker capabilities. 3 Method Overview. We formalize subgroup-targeted membership inference for synthetic data release as a release-based hypothesis test parameterized by the target pool: a selection rule is used to decide which records the adversary must infer membership for, and running the same game with different selection rules —high-outlierness, rare-label, uniformly random, and composites of two—separates the effect of targeting from the leakage a release exhibits regardless of targeting (Sections 3.2–3.3). The game asks the adversary to distinguish two worlds: under H1H_1 the training set contains the target (one record drawn from the pool) while under H0H_0 (the null world) it contains no record of the pool at all. Because extracting likelihoods from synthetic releases is intractable, the adversary scores membership through a family of lexical and embedding-space proxies (Section 3.4), calibrated under three progressively stronger attacker scenarios (Sections 3.5). Repeated episodes yield a per-record vulnerability score V⁡(x)V(x) for the concentration analysis of Section 5.4. 3.1 Problem Setting Let π be a distribution over records x∈x , and C=(x1,…,xM)C=(x_1,…,x_M) a finite candidate corpus drawn i.i.d. from π. Let further D^←Gε(D) D← G_ (D) a mechanism to generate synthetic data, applied to a private training set D⊆CD C of size N. The adversary observes a target x, the release D D, and scenario-dependent side information auxaux (Section 3.5) and has to predict whether x∈Dx∈ D. Target pools. We treat the targeted records as a parameter of the audit. A selection rule r maps the corpus to a pool r=σr(C)⊆CG_r= _r(C) C with |r|=n|G_r|=n; the target is then drawn uniformly from the pool, πr∼Uniform⁡(r) _r (G_r), and the private training records uniformly from its complement, π¯r∼Uniform⁡(C∖r) π_r (C _r). Pools are defined relative to the population (as estimated by the corpus): whether a record is an Outlier (a ranking over C), Random (a uniform draw from C) or features Rare labels, depends on the population. 3.2 Pool-Parameterized Membership Inference Game Definition 1 (Pool-Parameterized MI on Synthetic Release). Fix a selection rule r, r=σr(C)G_r= _r(C). 1. Sample a target x∼πrx _r and a fair bit b←$0,1b \$←\0,1\. 2. If b=1b=1, set D:=D0∪xD:=D_0∪\x\ with D0∼(π¯r)N−1D_0 ( π_r)^N-1; if b=0b=0, draw D∼(π¯r)ND ( π_r)^N. 3. Release D^←Gε(D) D← G_ (D) and give (x,D^,aux)(x, D,aux) to the adversary. 4. The adversary predicts b^=Score(x;D^,aux)≥β b=1\Score(x; D,aux)≥β\. We write AdvrAdv_r for the adversary’s advantage in the game under rule r, reported as ROC AUC and, where relevant, TPR at low FPR. Equivalently the attack distinguishes H1:D=D0∪xH_1:D=D_0∪\x\ from H0:D∼(π¯r)NH_0:D ( π_r)^N (null world), with x∼πrx _r in both worlds: only membership status differs, so the game isolates the evidence a release carries about pool records. Taking r=Randomr= Random gives πr=π¯r=Unif⁡(C) _r= π_r=Unif(C), recovering the average-case game, so Advr−AdvRandomAdv_r-Adv_ Random measures what targeting buys the adversary. Unlike RMIA and Canary, which fix the target set implicitly, the pool is an explicit parameter, which is what makes the selected-vs-control comparison possible. Remark 1 (Composite pools). For r=r1⊔r2r=r_1 r_2, step 1 also records the constituent the target came from and π¯r π_r is uniform over C∖(r1∪r2)C (G_r_1 _r_2). Both constituents are then scored against one reference distribution, so their advantages are directly comparable and per-record risk can be regressed on attributes over the union of the two ranges—neither of which holds when Advr1Adv_r_1 and Advr2Adv_r_2 come from separate within-pool games (§5.6). 3.3 Target-Pool Selection Rules We study four rules: • Outlier: the n records of highest outlierness according to a specified measure, representing atypical records. • Rare: records with a low-frequency label. • Random: n records drawn uniformly from C: a control that removes selection while preserving everything else. • r1⊔r2r_1 r_2: a composite pool joining the two constituent pools (Remark 1). Under every rule the remaining records are partitioned into pools of private candidates auxiliary reference data by the same procedure (§4.2). The generation pipeline, |C||C|, N, pool construction, proxy family, and how negatives are sampled is fixed. Thus, instantiations differ only in r, so any observable difference is attributable to the choice of the pool. A target pool generated by one single rule supports a within-pool game whose scores are calibrated against that pool’s own complement—cheap (one release each) but pool-relative and not comparable across rules (§5.6). Composite pools remove this at the cost of one extra release per pair. We therefore screen with single rule pools and use composite pools for the cross-rule and per-record comparisons that screening flags. 3.4 Release-Based Proxy Auditing An ideal attack would directly compare release likelihoods, Λr(x,D^)=P⁡(D^∣H1)/P⁡(D^∣H0) _r(x; D)=P( D H_1)/P( D H_0) with P⁡(D^∣H1)=D0∼(π¯r)N−1[P⁡(D^∣D0∪x)]P( D H_1)=E_D_0 ( π_r)^N-1[P( D D_0∪\x\)] and P⁡(D^∣H0)=D∼(π¯r)N[P⁡(D^∣D)]P( D H_0)=E_D ( π_r)^N[P( D D)]. As these are intractable to compute, the adversary instead measures a set of proxy metrics ϕ⁡(x,R)φ(x;R), where the observation object R is the target release D D, a raw reference set Dref(j)D_ref^(j), or a synthetic reference release D^ref(j) D_ref^(j), depending on the scenario. The same set of proxies is used for all setups (§3.6); only the reference objects available for calibration change, and two game instantiations differ only through π¯r π_r. Some proxies are based on probability (n-gram log-probability, Gaussian log-likelihood); others (similarity, overlap, distance) are treated as empirical membership-evidence scores compared against a scenario-dependent baseline. 3.5 Attacker Scenarios and Reference-Based Calibration For each selection rule, we study three attackers, all with access to D D and x, but with differing access to auxiliary reference information. Scenario 1 (release-only) has no reference data and no generator access, so it uses unnormalized evidence Scoreϕ(1)=ϕ⁡(x,D^)Score^(1)_φ=φ(x; D)—the weakest form of attacker with the lowest assumptions. Scenario 2 (raw-reference) cannot query the generator but has access to auxiliary data from π¯r π_r. Scenario 3 (generator-assisted) additionally has release-level query access to the same pipeline. Raw evidence ϕ⁡(x,D^)φ(x; D) conflates membership signal with the target’s intrinsic difficulty. Following RMIA [11] and Canary [7], we calibrate against the support x receives when its membership is marginalized, μϕ(x)=12(⁡[ϕ⁡(x,D^)∣H1]+⁡[ϕ⁡(x,D^)∣H0])μ^φ(x)= 12 (E[φ(x; D) H_1]+E[φ(x; D) H_0] ). the release-based analogue of RMIA’s Pr⁡(x) (x), estimated without bias by averaging over references that contain x in exactly half of the cases (RMIA Eq. 9; Canary Eq. 3). We estimate μϕ(x)μ^φ(x) with a common IN/OUT construction shared by the two reference-based scenarios: build MrefM_ref reference sets from an auxiliary pool drawn from π¯r π_r, insert the target into exactly Mref/2M_ref/2 of them (IN) and leave the rest (OUT), and average proxy evidence, μ^ϕ(x)=1Mref∑j=1Mrefϕ⁡(x,R(j))=12(μ^INϕ(x)+μ^OUTϕ(x)), μ^φ(x)= 1M_ref _j=1^M_refφ (x;R^(j) )= 12 ( μ^φ_IN(x)+ μ^φ_OUT(x) ), where R(j)R^(j) is the raw set Dref(j)D_ref^(j) in Scenario 2 and the generated release D^ref(j)←Gε(Dref(j)) D_ref^(j)← G_ (D_ref^(j)) in Scenario 3. Only Scenario 3 estimates μϕ(x)μ^φ(x) in the release space; Scenario 2 applies the same construction to raw data and thus gives a generator-free calibration of target difficulty. Writing μ^(s)ϕ(x) μ^φ_(s)(x) for the scenario-s baseline (rawraw for s=2s=2, synsyn for s=3s=3), the score is Scoreϕ(s)(x,D^)=ϕ⁡(x,D^),s=1,ϕ⁡(x,D^)−μ^(s)ϕ(x),s∈2,3,ϕ in log-space,ϕ⁡(x,D^)μ^(s)ϕ(x)+δ,s∈2,3 otherwise.Score^(s)_φ(x; D)= casesφ(x; D),&s=1,\\[6.0pt] φ(x; D)- μ^φ_(s)(x),& array[t]@l@s∈\2,3\,\\ φ in log-space, array\\[10.0pt] φ(x; D) μ^φ_(s)(x)+δ,&s∈\2,3\ otherwise. cases i.e. difference normalization for log-space proxies and ratio normalization for non-logarithmic (nonnegative) ones, matching each proxy’s scale. Comparing the three scenarios characterizes how targeted risk scales with available auxiliary information, and specifically whether release-level generator access adds power beyond reference calibration alone. 3.6 Proxy-Based Scoring We use m=32m=32 proxies Φ=ϕ1,…,ϕm =\ _1,…, _m\, each a distinct form of release support or memorization (Appendix 7.2). We report attacks per individual proxy, as a preliminary study found that a learned meta-classifier does not outperform individual proxies (see Appendix 7.6). 3.7 Attack Episodes and Sample-Level Vulnerability Each attack is run over T episodes—repeated plays of Definition 1, each drawing fresh membership bits, assembling D, generating a release, and scoring every target. Let st(x)s_t(x) be the score of x in episode t, and T1(x),T0(x)T_1(x),T_0(x) the member and non-member episodes for x (we keep targets with at least one of each and require a minimum count per condition). The sample-level vulnerability is V⁡(x)=1|T1(x)|∑t∈T1(x)st(x)−1|T0(x)|∑t∈T0(x)st(x).V(x)= 1|T_1(x)| _t∈ T_1(x)s_t(x)- 1|T_0(x)| _t∈ T_0(x)s_t(x). (1) Since raw scores are not comparable across proxies, V⁡(x)V(x) is computed within a fixed proxy and scenario, and cross-proxy/cross-scenario analyses use rank statistics. For the same reason V⁡(x)V(x) is not comparable across rules run as separate within-pool games—they normalize against different complements—which is why composite pools (Remark 1) are needed to make that comparison well posed. We then study the distribution of V⁡(x)V(x), rank-based concentration, cross-scenario consistency, and its correlation with attributes such as outlierness, lexical rarity, duplication, and length, to determine whether leakage is broad or concentrated on a few records. 4 Experiment Setup Our experiments broadly aim to investigate the following five research questions: RQ1. Can synthetic text releases leak membership of high-risk records under DP? RQ2. How does this risk scale with the attacker’s knowledge? RQ3. Do existing release-based attacks suffice for subgroup auditing? RQ4. Does DP protect the records within a target pool uniformly? RQ5. Can a data holder predict which records are at risk, and is the high-risk pool more exposed than ordinary records? This section describes the setup of the experiments needed to pursue these questions. 4.1 Datasets and Domains We evaluate on four datasets across three domains, spanning two orders of magnitude in length and label density (Table 1), to test whether subgroup-targeted leakage persists across data sources with different linguistic and label structure. In healthcare we use N2C2’08 [23], clinical discharge summaries annotated for obesity and co-morbidities, and PsyTAR [24], social-media posts labeled for adverse drug effects—formal clinical documentation versus informal user-generated discourse. In finance we use DMSAFN [25], financial-news text. In legal we use EurLex [26], EU legislative documents annotated with EuroVoc concepts, reduced to the eight most frequent. N2C2 PsyTAR DMSAFN EurLex Domain health health fin. legal N 604 5102 3876 4942 |||Y| 16 7 3 8 Comb. 429 38 3 95 Card. (mn/mx) 4.8/11 1.3/5 1.0/1 1.6/6 Words (mn/md) 1124/1070 14/12 23/21 191/119 Tok. md/p95/mx 1.8k/3.2k/4.0k 15/40/194 26/57/134 166/852/2.0k Table 1: Corpus-level statistics. Comb.: distinct label combinations; Card.: positive labels/record; mn/md/mx = mean/median/max. Tokens use the Llama-3.2-1B tokenizer. DMSAFN is single-label, the rest multi-label. 4.2 Target-Pool Construction The game of Section 3.2 is parameterized by a target pool P⊂fullP _full. We construct three pools—two notions of “high risk” and a control—and their compositions. Outlier pool (geometric). We encode all records with a per-dataset sentence encoder (see 7.1) and take records flagged by a high-percentile distance-to-mean threshold combined with an optional Local Outlier Factor criterion for locally isolated records. Rare-label pool (semantic). As an embedding-free notion of high risk, we select records whose label combinations fall below a frequency threshold (see 7.1). Notably, the rare-label pool shares zero records with the outlier pool O where applicable, even though by design, they are not mutually exclusive. Random pools (control). For each dataset we draw three independent uniform samples (rand0/1/2), size-matched to the outlier pool, that indicate how much outlier subgroup MIA signal stand out against a random-population control for the dataset. The rare-label pools are compared against the same controls. Induced partition. Each pool P induces its own partition full=P∪C∪RD_full=P∪ C∪ R (disjoint), where C builds the private training sets and R supplies attacker-side iid auxiliary data. The partition is shared across the three attacker scenarios, giving a consistent comparison of attack strength across both scenarios and pools. 4.3 Synthetic Text Generators We use three baselines at distinct design points for DP text synthesis. Aug-PE [13] is API-based: it estimates privatized feature statistics from the corpus and iteratively queries an LLM to generate and evolve matching text, enforcing privacy through the statistics rather than through DP training. DP-Transformers (DP-Gen) [21] is end-to-end neural: it fine-tunes with DP-SGD (per-example clipping, additive noise) and samples by standard decoding, enforcing privacy during training. EPSVec [14] steers a frozen LLM with a privatized dataset vector—per-layer hidden-state differences between sensitive examples and model-generated references, clipped and Gaussian-noised, then added at every decoding step—enforcing privacy once at vector release and decoupling the privacy cost from the number of samples. Backbones. All three share the same backbone family and scale (Llama-3.2-1B), isolating the privacy mechanism from generator capacity; DP-Gen and EPSVec use the base checkpoint, while Aug-PE’s prompt-driven generation requires the instruction-tuned one. 4.4 Baselines We compare against the two release-based MIA families applicable to our threat model, re-instantiated in our game under the identical protocol (same pools, releases, trial plan). Canary’s Echo [7] scores membership from the release through two signal families—n-gram support under a model fit on the synthetic text, and embedding similarity (SIMembSIM_emb, k=25k=25)—optionally calibrated against reference models via RMIA [11]; we port all four published arms unchanged, fitting references on raw text in Scenario 2 and shadow-generator releases in Scenario 3. Their targets are canaries injected 12–16 times while ours are real records appearing once, so their published AUCs are not head-to-head comparable. DOMIAS [19] scores log⁡pG(x)−log⁡pR(x) p_G(x)- p_R(x); its published settings have n/d≥78n/d≥ 78 whereas ours has n/d≈0.26n/d≈ 0.26–11, where a full-covariance Gaussian is rank-deficient, so we sweep four estimators (Gaussian/KDE × raw/PCA, fit only on attacker-accessible data) and report the best. For fairness, each family is collapsed to its best arm per (dataset, generator, ε ) cell, mirroring our own worst-case selection. 4.5 Proxy Families Proxies are scores ϕ⁡(x,R)φ(x;R) as in Section 3.5, with the observation object R set by the scenario; calibration is per feature, and Table 8 lists all proxies and hyperparameters. Lexical. A retrieve-then-compare design: x is a BM25 query retrieving its top-k candidates from R (k=50k=50), over which we compute surface-overlap statistics ranging from character n-gram containment (n=5n=5)11 1 Unlike prior audits that fit the n-gram model on the whole release [7, 8], we fit it only on the top-k retrieved candidates, localizing evidence to where single-record memorization surfaces. to an entity/number F1, each aggregated by max and mean (Table 8); an add-1 bigram model fit on the candidates scores verbatim reuse via log⁡p⁡(x) p(x). Embedding-space. Records are embedded by the per-dataset scoring encoder; relative to R we collect neighborhood similarity (cosine, dot-product, CSLS), distance and density statistics (Euclidean, Mahalanobis), and a Gaussian log-likelihood ℓR(x) _R(x) (Table 8). 4.6 Evaluation Protocol Fix a target pool P (Section 4.2). We run the game of Definition 1 over 100 game instances, each producing a dedicated release D^i←Gε(Di) D_i← G_ (D_i) per (dataset, generator, ε ) setting; the private set DiD_i is built from the candidate pool C induced by P. For bi=1b_i=1, the positive target xi+∈Px_i^+∈ P is fixed across rounds and scored once. For bi=0b_i=0, DiD_i contains no record of P, so any x∈Px∈ P is a valid non-member; we pre-select a fixed pool i⊂PP_i⊂ P of K=20K=20 candidates drawn without replacement, distinct across negative instances. Positives and negatives are thus drawn from the same pool P, so the resulting ROC measures membership signal within P rather than any difference between pools. Across T=50T=50 rounds, one candidate is drawn uniformly from iP_i per negative instance; the pools i\P_i\ and per-round assignments k⁡(t)\k(t)\ are fixed across all generators and ε , so cross-method differences reflect the generator’s effect rather than negative-sampling variance. The round-t evaluation set is ℰ(t)=(xi+,D^i):bi=1∪(xi,k⁡(t)−,D^i):bi=0.E^(t)= \(x_i^+, D_i):b_i=1 \∪ \(x_i,k(t)^-, D_i):b_i=0 \. (2) The protocol is identical for every pool instantiation; only P (and hence the induced C and R) changes. Setup and intervals. Scenarios follow §3.5 with Mref=4M_ref=4 (two IN, two OUT) per target. All T=50T=50 rounds share the same 100 releases and differ only in sampled negatives, so the reported CIs capture negative-sampling variability alone: they are narrow by construction and do not reflect uncertainty over independent pipeline re-runs. 4.7 Evaluation Metrics AUC-ROC. Our primary metric is AUC-ROC, the probability that a random member scores higher than a random non-member. We fit one ROC per round over ℰ(t)E^(t) and report the mean of the resulting T=50T=50 values with a 95% CI, μ^±1.96σ^/T μ± 1.96\, σ/ T. Unless noted, each setting reports the proxy with the highest mean AUC. Since that proxy is chosen on the same episodes it is scored on, these numbers are an empirical worst-case audit: what the best proxy would have revealed in hindsight, not what a deployable attack achieves. The meta-classifier (Appendix 7.6) selects on training episodes only and is the deployable counterpart. TPR at low FPR. A useful membership attack must be confident, not just better than chance, so we also report TPR at FPR ≤0.05≤ 0.05: the fraction of members recovered while falsely flagging at most 5% of non-members. By default this uses the AUC-selected proxy. 5 Experimental Results Table 2: Subgroup-targeted MI across privacy budgets ε (columns) and attacker scenarios S1–S3 (rows), for the three generators. Each cell reports the best-proxy AUC with TPR%@FPR≤ 5% as a subscript (worst-case audit, §4.7); Shaded cells mark the best scenario per dataset and ε , AUC ties broken by TPR; the superscript indicates the winning proxy family, Lexical or Embedding, respectively. Subcaptions indicate the generator and distribution of best proxies per best scenario. (a) DP-Gen (11 L vs. 9 E). ∞ ∞ 4 2 1 .5 .5 PsyTAR S1 .7946 .5912 .5913 .60E [rgb]0.6602,0.2617,0.1211E_11 .614 S2 .80L [rgb]0.1211,0.3711,0.6641L_53 .59L [rgb]0.1211,0.3711,0.6641L_10 .576 .579 .64L [rgb]0.1211,0.3711,0.6641L_6 S3 .7645 .5715 .59L [rgb]0.1211,0.3711,0.6641L_11 .578 .637 DMSAFN S1 .7939 .60L [rgb]0.1211,0.3711,0.6641L_4 .61L [rgb]0.1211,0.3711,0.6641L_5 .644 .646 S2 .79E [rgb]0.6602,0.2617,0.1211E_34 .555 .597 .65L [rgb]0.1211,0.3711,0.6641L_5 .647 S3 .7342 .522 .556 .616 .65L [rgb]0.1211,0.3711,0.6641L_16 N2C2’08 S1 .554 .5816 .578 .5713 .569 S2 .6011 .5918 .63E [rgb]0.6602,0.2617,0.1211E_13 .6112 .63E [rgb]0.6602,0.2617,0.1211E_9 S3 .64E [rgb]0.6602,0.2617,0.1211E_14 .59L [rgb]0.1211,0.3711,0.6641L_4 .5610 .61E [rgb]0.6602,0.2617,0.1211E_9 .578 EurLex S1 .8052 .5911 .566 .58L [rgb]0.1211,0.3711,0.6641L_9 .57L [rgb]0.1211,0.3711,0.6641L_6 S2 .8360 .5710 .565 .589 .5610 S3 .83E [rgb]0.6602,0.2617,0.1211E_61 .59E [rgb]0.6602,0.2617,0.1211E_12 .58E [rgb]0.6602,0.2617,0.1211E_4 .559 .548 (b) Aug-PE (14 L vs. 6 E). ∞ ∞ 4 2 1 .5 .5 PsyTAR S1 .5911 .5914 .5912 .5913 .6112 S2 .64L [rgb]0.1211,0.3711,0.6641L_13 .62L [rgb]0.1211,0.3711,0.6641L_12 .65L [rgb]0.1211,0.3711,0.6641L_14 .63E [rgb]0.6602,0.2617,0.1211E_15 .63L [rgb]0.1211,0.3711,0.6641L_12 S3 .561 .568 .614 .585 .597 DMSAFN S1 .61L [rgb]0.1211,0.3711,0.6641L_4 .61L [rgb]0.1211,0.3711,0.6641L_5 .616 .61L [rgb]0.1211,0.3711,0.6641L_5 .61L [rgb]0.1211,0.3711,0.6641L_4 S2 .605 .5911 .604 .604 .594 S3 .5811 .598 .62E [rgb]0.6602,0.2617,0.1211E_9 .588 .554 N2C2’08 S1 .606 .5913 .5616 .5810 .5715 S2 .64E [rgb]0.6602,0.2617,0.1211E_10 .59L [rgb]0.1211,0.3711,0.6641L_15 .62E [rgb]0.6602,0.2617,0.1211E_13 .6012 .62E [rgb]0.6602,0.2617,0.1211E_10 S3 .637 .598 .607 .62E [rgb]0.6602,0.2617,0.1211E_9 .6012 EurLex S1 .598 .57L [rgb]0.1211,0.3711,0.6641L_5 .59L [rgb]0.1211,0.3711,0.6641L_7 .58L [rgb]0.1211,0.3711,0.6641L_8 .5710 S2 .596 .558 .5910 .565 .586 S3 .59L [rgb]0.1211,0.3711,0.6641L_9 .553 .569 .587 .61L [rgb]0.1211,0.3711,0.6641L_11 (c) EPSVec (16 L vs. 4 E). ∞ ∞ 4 2 1 .5 .5 PsyTAR S1 .5912 .5813 .59L [rgb]0.1211,0.3711,0.6641L_14 .5910 .59L [rgb]0.1211,0.3711,0.6641L_11 S2 .589 .5814 .5910 .598 .576 S3 .60L [rgb]0.1211,0.3711,0.6641L_19 .64L [rgb]0.1211,0.3711,0.6641L_15 .599 .60L [rgb]0.1211,0.3711,0.6641L_6 .579 DMSAFN S1 .615 .614 .61L [rgb]0.1211,0.3711,0.6641L_5 .61L [rgb]0.1211,0.3711,0.6641L_3 .613 S2 .63L [rgb]0.1211,0.3711,0.6641L_16 .617 .603 .6115 .62L [rgb]0.1211,0.3711,0.6641L_15 S3 .569 .63L [rgb]0.1211,0.3711,0.6641L_10 .5611 .6010 .569 N2C2’08 S1 .5817 .5513 .565 .5813 .586 S2 .60E [rgb]0.6602,0.2617,0.1211E_12 .60E [rgb]0.6602,0.2617,0.1211E_6 .62E [rgb]0.6602,0.2617,0.1211E_10 .618 .6012 S3 .5711 .559 .587 .63L [rgb]0.1211,0.3711,0.6641L_6 .63L [rgb]0.1211,0.3711,0.6641L_9 EurLex S1 .576 .5711 .586 .6312 .59L [rgb]0.1211,0.3711,0.6641L_7 S2 .58L [rgb]0.1211,0.3711,0.6641L_6 .589 .59L [rgb]0.1211,0.3711,0.6641L_6 .64E [rgb]0.6602,0.2617,0.1211E_13 .566 S3 .566 .59L [rgb]0.1211,0.3711,0.6641L_8 .587 .618 .586 5.1 Subgroup Membership Leakage Across Generators and Budgets Answering RQ1, Tables 2(a), 2(b), and 2(c) report best-feature AUC and TPR at FPR ≤ 5% for the outlier pools under each generator, scenario and ε combination. Table 3 repeats the audit on the rare-label pools. Per-combination confidence intervals are given in Tables 9–10 and 12 (Appendix 7.5). Non-privately trained generators expose high-risk records at practically exploitable rates. When the generator is fine-tuned on the private data without DP, releasing synthetic text is close to releasing a membership oracle for the high-risk pool. Under DP-SGD fine-tuning at ε=∞ =∞, the best proxy reaches AUC 0.79–0.83 on PsyTAR, DMSAFN, and EurLex, identifying 34–61% of true members at a false-positive rate below 5% (Table 2(a)). A release-only attacker (S1) is already within 0.010.01–0.040.04 AUC of the best calibrated attacker for these combinations: with strong memorization, no auxiliary data is needed. This cannot be attributed to a serendipitous choice of proxy: even the deployable fusion attack (Appendix 7.6, Tables 13 and 16), which selects and weights proxies by nested cross-validation without seeing the evaluation data, still reaches AUC 0.760.76–0.790.79. The spike is specific to fine-tuning: MIA scores on data generated by training-free generators is far below the reported DPGen numbers, but it also doesn’t drop to chance levels, with scores largely unaffected by the application of DP (Tables 2(b) and 2(c)). DP suppresses most of this leakage, but not all of it. At every finite budget we test, subgroup membership remains inferable above chance. Across all datasets and scenarios under DP-SGD fine-tuning, the AUC for the best-performing proxy is no more than 0.520.52–0.650.65, and TPR@FPR ≤ 5% reaches 18%—3.6× the chance rate—at ε=4 =4 on N2C2’08 and 16% at ε=0.5 =0.5 on DMSAFN. The drop from the non-private ceiling is large (from 0.800.80 to 0.590.59 on average), confirming that DP-SGD effectively removes a large proportion of the membership signal. Table 3: Rare-label pool results for DP-Gen and EPSVec, formatted in the same way as Table 2. DP-Gen EPSVec ∞ ∞ 4 1 ∞ ∞ 4 1 PsyTAR S1 .82/49 .59E/13 .57/9 .57/8 .60E/16 .58E/9 S2 .84/46 .53/11 .55/4 .54/6 .56/7 .57/8 S3 .86L/59 .56/9 .58L/11 .59E/7 .58/13 .57/8 N2C2’08 S1 .62/14 .53/5 .53/7 .52/3 .56/16 .56E/6 S2 .69/15 .55/2 .56/3 .55/6 .56/4 .55/12 S3 .77L/28 .56L/8 .58E/3 .58E/11 .66L/10 .55/3 EurLex S1 .63/13 .57E/9 .56/6 .62E/9 .62E/10 .62E/9 S2 .61/11 .56/8 .57L/6 .58/8 .62/11 .58/6 S3 .64L/23 .55/7 .55/5 .59/12 .56/3 .57/6 Rare-label pools confirm the pattern and expose leakage the outlier audit misses. When the pool is selected by label statistics alone, similar observations apply. Under non-private fine-tuning, the rare pools leak at least as strongly as the outlier pools on the same corpora—best AUC 0.86 on PsyTAR (TPR 59%) versus 0.80 for the outlier pool—and under DP they sit on the same floor (0.53–0.66 across generators and budgets; Table 3). Notably, for N2C2’08 the non-private audit stays relatively weak (0.640.64 AUC), while its rare-label pool—records with at least eight co-morbidities—reaches 0.770.77 with TPR 28% (Table. 3), suggesting that high-risk subgroups that have a disproportionately high membership signal differ across datasets. 5.2 Scaling with Attacker Side Information Figure 2: Effect of attacker side information over all 60 (generator, dataset, ε ) settings. (a) Change in best-feature AUC relative to the release-only attacker S1 (grey: individual settings; black: mean with 95 % CI). (b) Paired change of the median-performing and the best-performing proxy of each view at each step of the scenario ladder. Answering RQ2: our three scenarios add capability monotonically (§3.5), and a natural expectation is that more capability yields a stronger attack. Our results indicate that this is not the case across all 60 combination of generator, dataset and ε , raw-reference calibration (S2) shifts the best-feature AUC by +0.009+0.009 on average (95%95\,\% CI ±0.007± 0.007) relative to S1, and generator access (S3) by −0.004-0.004 (±0.009± 0.009). S2 yields a statistically detectable but practically negligible gain (+0.009+0.009, 95%95\% CI ±0.007± 0.007); S3 is indistinguishable from zero (−0.004±0.009-0.004± 0.009). (Figure 2a). Side information changes which proxy wins, not how much leaks: under S2 the median embedding proxy loses 0.0130.013 AUC while the best gains 0.020.02 (Figure 2b), so the attack improves only if the attacker re-selects its proxy afterward—and the winning scenario and channel (L vs. E) indeed change from cell to cell across Tables 2(a)–2(c), with no majority. These findings underline the importance of considering (and optimizing for) the best proxy for each setting (i.e, dataset, background knowledge and DP protection) separately, to not under-report MIA scores due to sub-par proxy choice. 5.3 Comparison with Prior Release-Based Attacks Table 4: Prior release-based attacks (Top: Outlier pool, bottom: rare label pool) re-instantiated in the subgroup game, across generators. Cells are AUC / TPR@FPR≤ 0.05. Per ε block the best AUC of the three methods is in bold and the runner-up is underlined. Δ is ours minus the stronger baseline. Superscripts name the winning variant and are coloured by the channel it uses, on the same scale as the L/E tags on ours. canary arms: n n-gram, nR n-gram++RMIA, s SIMembSIM_emb, sR SIMembSIM_emb ratio. domias estimators: g Gaussian, k KDE, pg PCA32++Gaussian, pk PCA32++KDE. The superscript on ours marks whether the winning proxy is Lexical or Embedding. ε=∞ =∞ ε=1 =1 Dataset Generator canary domias ours Δ canary domias ours Δ PsyTAR DP-Gen 0.6300.630nR / 21% 0.739¯ 0.739g / 54% 0.8040.804L / 53% +0.065+0.065 0.5920.592n / 12% 0.597¯ 0.597k / 9% 0.6010.601E / 11% +0.004+0.004 PsyTAR Aug-PE 0.587¯ 0.587n / 14% 0.5360.536k / 10% 0.6370.637L / 13% +0.050+0.050 0.588¯ 0.588n / 11% 0.5630.563k / 7% 0.6310.631E / 15% +0.043+0.043 PsyTAR EPSVec 0.5950.595n / 13% 0.5520.552k / 7% 0.595¯ 0.595L / 19% +0.000+0.000 0.594¯ 0.594n / 11% 0.5210.521k / 9% 0.5970.597L / 6% +0.004+0.004 DMSAFN DP-Gen 0.6510.651nR / 22% 0.717¯ 0.717g / 42% 0.7890.789E / 34% +0.072+0.072 0.604¯ 0.604n / 3% 0.5750.575k / 10% 0.6470.647L / 5% +0.043+0.043 DMSAFN Aug-PE 0.603¯ 0.603n / 2% 0.5150.515k / 5% 0.6120.612L / 4% +0.010+0.010 0.602¯ 0.602n / 3% 0.4940.494k / 11% 0.6130.613L / 5% +0.012+0.012 DMSAFN EPSVec 0.602¯ 0.602n / 3% 0.4980.498pk / 2% 0.6300.630L / 16% +0.028+0.028 0.601¯ 0.601n / 3% 0.5130.513k / 7% 0.6110.611L / 3% +0.010+0.010 N2C2’08 DP-Gen 0.607¯ 0.607sR / 5% 0.5580.558pk / 9% 0.6360.636E / 14% +0.029+0.029 0.5040.504s / 8% 0.552¯ 0.552pk / 14% 0.6150.615E / 9% +0.063+0.063 N2C2’08 Aug-PE 0.570¯ 0.570nR / 19% 0.5550.555pk / 7% 0.6400.640E / 10% +0.070+0.070 0.5190.519s / 2% 0.545¯ 0.545k / 11% 0.6230.623E / 9% +0.077+0.077 N2C2’08 EPSVec 0.588¯ 0.588nR / 9% 0.5840.584pk / 10% 0.6050.605E / 12% +0.017+0.017 0.585¯ 0.585sR / 15% 0.5700.570pk / 9% 0.6260.626L / 6% +0.041+0.041 EUR-Lex DP-Gen 0.7090.709nR / 34% 0.809¯ 0.809g / 61% 0.8340.834E / 61% +0.025+0.025 0.5280.528sR / 7% 0.540¯ 0.540k / 7% 0.5790.579L / 9% +0.040+0.040 EUR-Lex Aug-PE 0.5170.517n / 5% 0.536¯ 0.536k / 10% 0.5940.594L / 9% +0.058+0.058 0.5160.516n / 7% 0.531¯ 0.531pk / 4% 0.5840.584L / 8% +0.054+0.054 EUR-Lex EPSVec 0.5160.516n / 6% 0.578¯ 0.578k / 13% 0.5820.582L / 6% +0.004+0.004 0.5350.535sR / 8% 0.587¯ 0.587g / 9% 0.6440.644E / 13% +0.056+0.056 ε=∞ =∞ ε=1 =1 Dataset Generator canary domias ours Δ canary domias ours Δ PsyTAR DP-Gen 0.7990.799nR / 43% 0.801¯ 0.801g / 56% 0.8570.857L / 59% +0.055+0.055 0.564¯ 0.564s / 6% 0.5460.546g / 4% 0.5840.584L / 11% +0.019+0.019 PsyTAR EPSVec 0.592¯ 0.592sR / 7% 0.5750.575pg / 7% 0.5950.595E / 7% +0.003+0.003 0.5640.564s / 8% 0.5860.586pg / 6% 0.575¯ 0.575E / 9% −0.011-0.011 N2C2’08 DP-Gen 0.679¯ 0.679nR / 33% 0.6020.602g / 15% 0.7710.771L / 28% +0.093+0.093 0.5910.591nR / 9% 0.5500.550pk / 4% 0.583¯ 0.583E / 3% −0.008-0.008 N2C2’08 EPSVec 0.5990.599nR / 6% 0.5280.528g / 4% 0.584¯ 0.584E / 11% −0.015-0.015 0.5290.529sR / 2% 0.532¯ 0.532pg / 2% 0.5570.557E / 6% +0.025+0.025 EUR-Lex DP-Gen 0.5600.560s / 9% 0.597¯ 0.597pk / 10% 0.6380.638L / 23% +0.041+0.041 0.570¯ 0.570s / 8% 0.5390.539g / 12% 0.5710.571L / 6% +0.000+0.000 EUR-Lex EPSVec 0.6210.621s / 11% 0.5850.585g / 9% 0.621¯ 0.621E / 9% −0.001-0.001 0.611¯ 0.611s / 9% 0.5630.563pk / 7% 0.6160.616E / 9% +0.005+0.005 Answering RQ3: Section 5.2 suggests that the most informative proxy moves from setting to setting, so any attack that commits to one signal would under-measure leakage. We test this directly by re-instantiating the two release-based MIA families applicable to our threat model under the same protocol: Canary [7] and DOMIAS [19] (see details in 4.4). The two baselines fall short for different reasons. DOMIAS is structurally confined to one kind of evidence: all four of its estimators are density models over embeddings, so leakage that surfaces as verbatim wording cannot be captured more explicitly than via embedding similarity. Canary can be instantiated with both n-gram and embedding similarity proxies but only one fixed signal of each kind, whereas the informative signal varies more finely: within each family (L or. E), the best proxy is different based on the specific setup (indicated by superscripts of the Ours columns in Table 4). This variance holds for both baselines as well (Table 4, superscripts in Canary and DOMIAS columns): For Canary, the best proxy alternates between the choices of n-gram, RMIA-combined, and similarity forms, and for DOMIAS’s between its Gaussian, KDE, and PCA estimators—so even within a single family there is no configuration that is guaranteed to perform best across different settings. Table 4 also shows that our MIA setup significantly outperforms the baselines, on 30 of 36 settings (across both outlier and rare label pools). Notably, our methods recovers AUC up to 0.83 on real records that were inserted only once, which is above the 0.62–0.77 reported for Canary, when canaries were injected 12–16 times. 5.4 Aggregate vs. Individual Protection: Leakage Concentration under DP Figure 3: Sorted per-record vulnerability V⁡(x)V(x) for four representative cells (note the differing y-scales). The dark slice is the positive mass held by the top decile of records (dashed line: decile boundary); the printed percentage is the top-decile leakage share S10S_10 (Eq. 3; ≈10%≈10\% under even leakage). Left: non-private fine-tuning leaks broadly; right: under suppressed memorization the residual is top-heavy. Answering RQ4, this section asks how the leakage of §5.1 is distributed across records, measured by the per-record vulnerability V⁡(x)V(x) (Eq. 1), computed as follows. For each combination of dataset, generator and ε , the scores of its best proxy—the same worst-case selection as the main tables—are sign-aligned and z-normalized, so magnitudes are comparable directly. V⁡(x)V(x) is then the record’s mean score over the combinations’s episodes in which it was the inserted member, minus its mean over those in which it served as a non-member candidate. A record receives a score only if it was both member and non-member at least once. Of the roughly 60 records in a pool, 19–36 qualify. Top-decile leakage share. Order a combination’s n qualifying records by decreasing V, write V(i)+=max⁡V(i),0V^+_(i)= \V_(i),0\, and let k=max⁡1,⌊0.1n+0.5⌋k= \1, 0.1\,n+0.5 \. The top-decile leakage share is S10=∑i=1kV(i)+∑i=1nV(i)+,S_10\;=\; _i=1^kV^+_(i) _i=1^nV^+_(i), (3) the share of the combination’s total positive vulnerability that falls on its top tenth of records; records with V≤0V≤ 0 contribute to neither sum. Even leakage gives S10≈10%S_10≈ 10\%; values approaching 100% mean the release’s residual risk is disproportionately concentrated in a small subset records. S10S_10 is the percentage printed in each panel of Figure 3. It is reported in full together with the share of records scoring V>0V>0 for all 55 configurations in Appendix 7.8 (Table 19), and the medians and ranges below are computed based on that table. Without DP, fine-tuned releases leak broadly, not selectively. Under non-private DP-SGD fine-tuning, most subgroup records (not a vulnerable few) carry positive membership signal. In the four ε=∞ =∞ fine-tuning combinations, 77–94% of qualifying records (median 82%) score V>0V>0, per-record magnitudes reach 5050–100100 times those under any DP budget, and concentration is modest: S10S_10 is 20–37% (median 30%), the lowest of any regime we audit. Under fine-tuning, DP thins the leakage and shifts it toward the top decile. In addition to changing overall membership inference risk, DP changes how this risk is distributed: it is concentrated in a small subset of records. Across DP-Gen’s 16 DP combinations, the share of records scoring positive falls from a median of 82% to 61%, while S10S_10 rises from a median of 30% to 40% (range 28–69%) and exceeds the even-spread value in 15 of 16 combinations threefold, against 2 of 4 without DP (Appendix 7.8). While the mean observations suggest a trend, individual configurations deviate in both directions. For example, DMSAFN at ε=4 =4 is strongly concentrated (29% positive, S10=69%S_10=69\%) while the same dataset at ε=0.5 =0.5 remains broad (71%, S10=28%S_10=28\%). Therefore, we report this as suggestive rather than conclusive evidence for membership signal concentration under DP. Concentration is the default wherever bulk memorization is absent—with or without DP. For the two mechanisms that don’t exhibit the no-DP leakage spike, the shape of the vulnerability distribution is already top-heavy at zero noise and barely moves with the budget. EPSVec’s ε=∞ =∞ combinations have S10S_10 of 36–82% (median 47%; PsyTAR reaches 82% with no noise at all), statistically indistinguishable from its DP combinations (median 41%); Aug-PE moves from a median of 31% at its zero-noise setting to 40% under noise. Pooling all 60 combinations, S10S_10 exceeds three times the even-spread value in 50 (median 39%, range 20–82%). Concentration therefore tracks the absence of bulk memorization—whether achieved by DP noise or by mechanism design—rather than DP itself, the record-level counterpart of the mechanism-relative picture of §5.1. Reporting only average AUC scores misses the concentration of membership inference risk. Releases with the same AUC can have very different top-decile shapes, and the concentration of risk in the top decile is a per-release property and not tied to a specific set of records. For DP, a median of 40% (up to max 72%) of the residual positive vulnerability is concentrated in the top decile and, yet an aggregate audit that certifies a release by mean AUC alone would report only the near-floor we reported in Section 5.1. Whether a record inhabits the top decile changes across different configurations, in line with prior findings [27]. A high S10S_10 suggests to audit each release rather than maintaining a list of high-risk records; the next section shows that the top decile’s membership is in fact hard to predict from record attributes alone. 5.5 Intrinsic Attributes Do Not Predict Which Records Leak If a data holder could identify at-risk records from the private data alone, triage would be possible without a costly audit. We test five release-independent attributes—text length, embedding outlierness (the selection statistic of §4.2), near-duplication, rare-n-gram density, and entity/numeral density—against the within-pool rank of V⁡(x)V(x) in all 42 pools the audit covers. Scoring V⁡(x)V(x) requires choosing one scenario, proxy pair per pool; we report a conservative rule (chosen on an 80% training partition) as primary and the main tables’ worst-case rule as a sensitivity analysis. Computation details, two selection-free rules under which every conclusion is unchanged, and the full per-pool matrix are in Appendices 7.9 and 7.10. Under the conservative rule, no attribute predicts vulnerability Figure 4: Spearman ρ between five release-independent record attributes and per-record vulnerability rank in the random pools, per generator. Per-dataset columns are Fisher-z combinations of that dataset’s three disjoint draws and carry no test; all pools the twelve draws (337 records) with Holm correction over the five attributes within a generator. Bold: nominal p<0.05p<0.05; boxed: Holm-significant. The outlier and rare-label pools are in Figure 5, Appendix 7.9. Table 5: Outlier vs. Rare-label subgroups vs. the same random-group control for DP-Gen (top) and EPSVec (bottom), evaluated under the identical protocol. Each cell is the best AUC over the candidate proxy set, selected independently per cell so neither group is handicapped by the other’s winner; ± is a 95% normal interval over the evaluation trials (S 4.6), in units of 10−310^-3 (i.e. ± 9 means ± 0.009). The control pools 3 independent draws of the random pool (rand0/1/2), so its interval 1.96sehr2+sectl21.96 se_hr^2+se_ctl^2 absorbs both the negative-candidate and the pool draw. Gap is defined as subgroup −- control. Bold gaps exclude zero. ε=∞ =∞ ε=4 =4 Dataset Cal. Control Outlier Gap Rare-label Gap Control Outlier Gap Rare-label Gap PsyTAR S1 0.8040.804L ± 9 0.7910.791L ± 5 −0.012-0.012 ± 10 0.8200.820L ± 5 +0.017+0.017 ± 11 0.5320.532L ± 8 0.5860.586E ± 12 +0.054+0.054 ± 14 0.5880.588E ± 9 +0.056+0.056 ± 11 S2 0.8100.810L ± 9 0.8040.804L ± 4 −0.006-0.006 ± 10ns 0.8350.835L ± 6 +0.025+0.025 ± 11 0.5250.525E ± 7 0.5870.587L ± 10 +0.062+0.062 ± 12 0.5310.531E ± 10 +0.006+0.006 ± 12ns N2C2’08 S1 0.5940.594L ± 7 0.5510.551L ± 13 −0.043-0.043 ± 15 0.6220.622L ± 10 +0.028+0.028 ± 12 0.5540.554E ± 9 0.5760.576L ± 10 +0.022+0.022 ± 13 0.5250.525L ± 11 −0.029-0.029 ± 15 S2 0.6560.656L ± 6 0.6000.600L ± 9 −0.056-0.056 ± 11 0.6880.688L ± 10 +0.032+0.032 ± 12 0.5500.550E ± 9 0.5900.590L ± 8 +0.040+0.040 ± 12 0.5460.546E ± 10 −0.004-0.004 ± 14ns EUR-Lex S1 0.6650.665E ± 5 0.8040.804E ± 6 +0.139+0.139 ± 8 0.6250.625E ± 10 −0.040-0.040 ± 11 0.5510.551L ± 7 0.5870.587E ± 9 +0.036+0.036 ± 11 0.5750.575E ± 12 +0.023+0.023 ± 14 S2 0.6550.655E ± 7 0.8310.831E ± 4 +0.176+0.176 ± 8 0.6060.606E ± 9 −0.049-0.049 ± 11 0.5520.552L ± 12 0.5700.570E ± 9 +0.018+0.018 ± 15 0.5650.565E ± 12 +0.013+0.013 ± 16ns ε=∞ =∞ ε=4 =4 Dataset Cal. Control Outlier Gap Rare-label Gap Control Outlier Gap Rare-label Gap PsyTAR S1 0.5510.551L ± 7 0.5940.594L ± 9 +0.044+0.044 ± 11 0.5700.570E ± 11 +0.019+0.019 ± 13 0.5390.539L ± 6 0.5850.585L ± 9 +0.046+0.046 ± 11 0.5970.597E ± 11 +0.058+0.058 ± 12 S2 0.5500.550L ± 8 0.5820.582L ± 10 +0.032+0.032 ± 13 0.5430.543E ± 12 −0.008-0.008 ± 14ns 0.5260.526L ± 5 0.5780.578L ± 10 +0.052+0.052 ± 11 0.5600.560E ± 10 +0.033+0.033 ± 12 N2C2’08 S1 0.5640.564E ± 10 0.5760.576L ± 9 +0.012+0.012 ± 13ns 0.5200.520E ± 9 −0.044-0.044 ± 13 0.5600.560E ± 8 0.5460.546L ± 10 −0.014-0.014 ± 13 0.5620.562E ± 10 +0.002+0.002 ± 13ns S2 0.5550.555E ± 9 0.6050.605E ± 11 +0.049+0.049 ± 14 0.5490.549E ± 11 −0.007-0.007 ± 14ns 0.5400.540E ± 10 0.5950.595E ± 10 +0.055+0.055 ± 14 0.5630.563L ± 11 +0.024+0.024 ± 15 EUR-Lex S1 0.5340.534L ± 9 0.5750.575L ± 10 +0.041+0.041 ± 13 0.6210.621E ± 12 +0.087+0.087 ± 15 0.5400.540L ± 8 0.5730.573L ± 8 +0.033+0.033 ± 11 0.6180.618E ± 12 +0.078+0.078 ± 14 S2 0.5350.535E ± 6 0.5820.582L ± 11 +0.048+0.048 ± 12 0.5830.583E ± 12 +0.048+0.048 ± 14 0.5360.536E ± 6 0.5810.581L ± 10 +0.045+0.045 ± 12 0.6170.617E ± 11 +0.081+0.081 ± 13 Of the 210 attribute–pool tests, none survives Holm correction. Pooling the twelve random pools by Fisher’s z (Figure 4) surfaces only weak tendencies, all under DP-SGD fine-tuning: near-duplication −0.17-0.17 (Holm p=0.01p=0.01), entity density −0.15-0.15 (p=0.03p=0.03), outlierness +0.12+0.12 (p=0.09p=0.09); for EPSVec every combined ρ lies within ±0.12± 0.12. Correlations of this size cannot be meaningfully used to predict membership inference risk from records. Worst-case selection sharpens the tendencies but cannot change their meaning Re-selecting the proxy on all episodes strengthens the tendencies described above: outlierness +0.20+0.20, near-duplication −0.20-0.20, entity density −0.15-0.15, all clearing correction. Correlations for EPSVec remain essentially zero (Table 21). Our findings suggest that which records a release exposes is dictated by the choice of release mechanism, DP budget, and other audit configurations and is not easily predictable from the records. Making triaging a priori infeasible. 5.6 High-Risk vs. Random Pools In this section we answer the second part of RQ5, i.e., whether high-risk records are easier to infer membership comparing with ordinary records. We proceed in two steps. First (§5.6), we run the within-pool game on each outlier and rare-label pool and compare its separability against a random pool under an identical protocol. These gaps measure how distinguishable a pool’s records are as members versus as non-members, but the non-members are still selected within the pool, it cannot say whether an individual high-risk record is more exposed (Remark 1). Second (§5.6), we answer the record-level question with an experiment with composite pools: we merge an outlier pool with a random pool, which allows us to score both record types against the same negatives, making the comparison well posed—run on two configurations (EurLex × DP-Gen × ϵ=∞ε=∞; PsyTAR × DP-Gen × ϵ=4ε=4), which in the first step are observed having statistically significant difference in AUC between outlier and control pools. Within-pool distinguishability: outlier and rare-label pools vs. random pools Table 5 reports the gap in within-pool AUC between high-risk pools with random controls, for DP-Gen and EPSVec, respectively. The two generators behave differently without DP and exhibit similar behavior once DP is applied. First, under non-private fine-tuning (DP-Gen, ε=∞ =∞), outlier pools are not easier to attack: on two of three datasets they are significantly less distinguishable than the controls (gaps −0.01-0.01 to −0.06-0.06; EUR-Lex is the exception, +0.14+0.14/+0.18+0.18), while the rare-label gaps are small and mixed in sign (+0.02+0.02 to +0.03+0.03 on PsyTAR and N2C2’08, −0.04/0.05-0.04/0.05 on EUR-Lex). Second, adding DP to fine-tuning inverts this trend: at ε=4 =4 all six of DP-Gen’s outlier gaps are positive (+0.02+0.02 to +0.06+0.06, all significantly), and its rare-label gaps turn mixed (−0.03-0.03 to +0.06+0.06). Third, the training-free EPSVec shows inverted trends regardless of noise: 11 of 12 outlier gaps are positive (+0.01+0.01 to +0.06+0.06), its rare-label gaps peak on EUR-Lex (+0.05+0.05 to +0.09+0.09), and its control AUC never exceeds 0.560.56. Thus, when the generator memorizes broadly, as is the case for non-private DP-Gen, ordinary records are as easy do infer membership from as much as atypical ones. However, without broad memorisation, e.g., because of the use of DP or training-free methods, membership signal from random pools are closer to chance than while target from high-risk pools are left as the more distinguishable targets. Membership inference game with unified pools Table 6: Merged-pool audits of the two flagged cells. Both record types sit in one pool and face the same negative candidates, so Δ (outlier-origin −- random-origin members) directly measures which records are easier to attack; absolute values are against the merged pool’s negatives and are not comparable to the within-pool tables. Brackets: 95% bootstrap over member records; bold: interval excludes zero. Cell Outl. Rand. Δ EUR-Lex, ε=∞ =∞ AUC 0.73 0.67 +0.066+0.066 [-0.03,+0.16] TPR 0.50 0.12 +0.378+0.378 [+0.25,+0.51] PsyTAR, ε=4 =4 AUC 0.58 0.46 +0.127+0.127 [+0.02,+0.24] TPR 0.07 0.03 +0.040+0.040 [-0.02,+0.11] TPR at FPR≤ 5%. So far, the statements regarding membership signal differences between random and outlier targets were made at population level. This leaves open one final question: is an outlier record easier to attack than a randomly drawn one? To answer it directly, we merge the outlier pool with a random pool (the pools share no record), rerun the membership game on that merged pool, schedule every record as the inserted member equally often, and draw negative candidates from the whole merged pool. With this setup, both outliers and random examples are attacked under identical conditions and their scores can be compared directly. Table 6 shows the results: without DP the outlier records’ extra exposure shows up at the low-FPR end of the ROC; under DP it survives only in the ranking, i.e., outlier records still rank systematically higher than ordinary records in the attack’s score ordering. Without DP (EUR-Lex, ε=∞ =∞), outlier records are far more vulnerable: at 5% false positives, 50% of outlier members are identified against 12% of random members (Δ=+0.38 =+0.38), while the overall AUC difference is modest and not significant (+0.07+0.07). Under DP (PsyTAR, ε=4 =4), the advantage changes character: at 5% false positives neither type can be identified with confidence (TPR 0.070.07 vs. 0.030.03; the +0.04+0.04 difference has an interval that spans zero), yet outlier members still rank above random members across the full score range, with an AUC difference of +0.127+0.127—the one gap in the table whose interval excludes zero. 6 Conclusion and Limitations Auditing synthetic text releases with a game where the membership target pool is an explicit parameter changes what a membership inference attack experiment can report. Beyond confirming that subgroup membership leaks and that prior attacks under-report it, our auditing framework produced findings we believe are useful to the community: DP protects membership inference signal from high-risk groups less than from random controls. How much residual membership signal a release exposes is a property of the release mechanism, not of the private records, so no release-independent triage of at-risk records is feasible according to our analyses. This implies that average-case MIA performance reports and attempts at attribute-based screening of vulnerable records may both miss the risk to records. Our findings suggest that per-release MIA auditing is the preferred way to identify at-risk records. Our conclusions have the following limitations: First, the corpora are small (604–5,102 records) and the pools smaller (16–71), with these pool sizes, no correlation below |ρ|≈0.33|ρ|≈ 0.33–0.500.50 can reach significance. Second, our reported uncertainty estimates (as reported by confidence intervals) capture negative-sampling variance but not the variance of drawing the pool itself: re-estimating the latter would mean repeating the full pipeline over fresh pools, and a single configuration of pool, generator and ε takes about 27-30h to run on a L40S GPU for its 100 releases (EurLex, DP-Gen, one outlier pool)—replicating every combination over multiple pool draws is beyond our budget. Third, our adversary is strictly release-based and our generators span three mechanism families; attacks with model access, larger corpora, and further mechanisms may further refine our conclusions. Our membership inference game and the whole pipeline we release are designed to make such extensions cheap to run. Acknowledgments This research is part of the IN-CYPHER programme and is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. References [1] Y. Wu, V. Schlegel, W. Del-Pinto, S. Nandakumar, I. Zahid, Y. Sun, U. F. Omar, A. Jasmine, A. Kaliya-Perumal, C. S. Tham, G. Connors, A. A. Bharath, and G. Nenadic (2025) Term2Note: synthesising differentially private clinical notes from medical terms. arXiv preprint arXiv:2509.10882. External Links: Link Cited by: §1, §1. [2] S. A. Assefa, D. Dervovic, M. Mahfouz, R. E. Tillman, P. Reddy, and M. Veloso (2020) Generating synthetic data in finance: opportunities, challenges and pitfalls. In Proceedings of the First ACM International Conference on AI in Finance (ICAIF ’20), p. 44:1–44:8. External Links: Document Cited by: §1. [3] Synthetic Data Expert Group (2024) Using synthetic data in financial services. Technical report Financial Conduct Authority, London, UK. External Links: Link Cited by: §1. [4] Z. Zhao and A. Field (2025) Controlled generation for private synthetic text. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, p. 32720–32735. External Links: Link Cited by: §1. [5] S. M. Bellovin, P. K. Dutta, and N. Reitinger (2019) Privacy and synthetic datasets. Stanford Technology Law Review 22 (1), p. 1–52. External Links: Link Cited by: §1. [6] T. Stadler, B. Oprisanu, and C. Troncoso (2022) Synthetic data–anonymisation groundhog day. In 31st USENIX Security Symposium (USENIX Security 22), p. 1451–1468. Cited by: §1, §1, §2. [7] M. Meeus, L. Wutschitz, S. Zanella-Béguelin, S. Tople, and R. Shokri (2025) The canary’s echo: auditing privacy risks of llm-generated synthetic text. arXiv preprint arXiv:2502.14921. Cited by: §1, §1, §1, §2, §3.5, §4.4, §5.3, §7.3, footnote 1. [8] Y. Sun, V. Schlegel, S. Nandakumar, I. Zahid, Y. Wu, Y. Wu, H. Li, J. Zhang, W. Del-Pinto, G. Nenadic, et al. (2025) SynBench: a benchmark for differentially private text generation. arXiv preprint arXiv:2509.14594. Cited by: §1, §2, footnote 1. [9] R. Shokri, M. Stronati, C. Song, and V. Shmatikov (2017) Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), p. 3–18. Cited by: §1, §2. [10] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer (2022) Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), p. 1897–1914. Cited by: §1, §2, §2, §2, §7.3. [11] S. Zarifzadeh, P. Liu, and R. Shokri (2023) Low-cost high-power membership inference attacks. arXiv preprint arXiv:2312.03262. Cited by: §1, §1, §1, §2, §3.5, §4.4, §7.3. [12] X. Yue, H. Inan, X. Li, G. Kumar, J. McAnallen, H. Shajari, H. Sun, D. Levitan, and R. Sim (2023) Synthetic text generation with differential privacy: a simple and practical recipe. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1321–1342. Cited by: §1. [13] C. Xie, Z. Lin, A. Backurs, S. Gopi, D. Yu, H. A. Inan, H. Nori, H. Jiang, H. Zhang, Y. T. Lee, B. Li, and S. Yekhanin (2024) Differentially Private Synthetic Data via Foundation Model APIs 2: Text. In Proceedings of the 41st International Conference on Machine Learning, p. 54531–54560. External Links: Link Cited by: §1, §2, §4.3, §7.4. [14] A. Banayeeanzade, Q. Yang, D. Fu, S. Hong, E. Babinsky, A. Samuel, A. Kumar, R. Jia, and S. P. Karimireddy (2026) EPSVec: efficient and private synthetic data generation via dataset vectors. arXiv preprint arXiv:2602.21218. Cited by: §1, §4.3, §7.4. [15] V. Feldman (2020) Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd annual ACM SIGACT symposium on theory of computing, p. 954–959. Cited by: §1, §2. [16] N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song (2019) The secret sharer: evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), p. 267–284. Cited by: §1, §2, §2. [17] E. Bagdasaryan, O. Poursaeed, and V. Shmatikov (2019) Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems 32. Cited by: §1, §2. [18] F. Houssiau, J. Jordon, S. N. Cohen, O. Daniel, A. Elliott, J. Geddes, C. Mole, C. Rangel-Smith, and L. Szpruch (2022) Tapas: a toolbox for adversarial privacy auditing of synthetic data. arXiv preprint arXiv:2211.06550. Cited by: §2. [19] B. Van Breugel, H. Sun, Z. Qian, and M. van der Schaar (2023) Membership inference attacks against synthetic data through overfitting detection. arXiv preprint arXiv:2302.12580. Cited by: §2, §4.4, §5.3. [20] C. Dwork, F. McSherry, K. Nissim, and A. Smith (2006) Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, p. 265–284. Cited by: §2. [21] L. Wutschitz, H. A. Inan, and A. Manoel (2022) Dp-transformers: training transformer models with differential privacy. Note: https://w.microsoft.com/en-us/research/project/dp-transformers Cited by: §2, §4.3. [22] Y. Long, V. Bindschaedler, L. Wang, D. Bu, X. Wang, H. Tang, C. A. Gunter, and K. Chen (2018) Understanding membership inferences on well-generalized learning models. arXiv preprint arXiv:1802.04889. Cited by: §2. [23] O. Uzuner (2009) Recognizing obesity and comorbidities in sparse data. Journal of the American Medical Informatics Association 16 (4), p. 561–570. External Links: Link, Document, https://academic.oup.com/jamia/article-pdf/16/4/561/2302602/16-4-561.pdf Cited by: §4.1. [24] M. Zolnoori, K. W. Fung, T. B. Patrick, P. Fontelo, H. Kharrazi, A. Faiola, Y. S. S. Wu, C. E. Eldredge, J. Luo, M. Conway, et al. (2019) A systematic approach for developing a corpus of patient reported adverse drug events: a case study for ssri and snri medications. Journal of biomedical informatics 90, p. 103091. Cited by: §4.1. [25] Daniel-ML (2024) Sentiment analysis for financial news v2. Note: https://huggingface.co/datasets/Daniel-ML/sentiment-analysis-for-financial-news-v2 Cited by: §4.1. [26] I. Chalkidis, E. Fergadiotis, P. Malakasiotis, and I. Androutsopoulos (2019) Large-scale multi-label text classification on EU legislation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 6314–6322. External Links: Document, Link Cited by: §4.1. [27] N. Carlini, M. Jagielski, C. Zhang, N. Papernot, A. Terzis, and F. Tramer (2022) The privacy onion effect: memorization is relative. Advances in Neural Information Processing Systems 35, p. 13263–13276. Cited by: §5.4. 7 Appendix 7.1 The details of definition of target pool Dataset Embedder Outlier Rare Control rule |||T| rule |||T| |||T| N2C2’08 BioSimCSE-BioLinkBERT d97d_97 + LOFk=3 19 card. ≥8≥ 8 63 19 PsyTAR sentence-t5-base d99d_99 + LOFk=10 60 freq. ≤10≤ 10 54 60 DMSAFN sentence-t5-base d99d_99 + LOFk=3 71 — — 71 EurLex bge-m3 d99d_99 50 freq. ≤3≤ 3 55 50 Table 7: Subgroup construction. dpd_p selects records beyond the p-th percentile of distance to the embedding centroid; LOFk is Local Outlier Factor with k neighbours; outlier pools are the union of the two criteria. Rare pools use label statistics: card. is label cardinality, freq. is the corpus frequency of a record’s label combination. Control pools match the outlier pool size and are drawn uniformly at random, with three independent seeds per dataset. For each dataset we construct three target pools (as shown in Table. 7). The outlier pool captures geometric atypicality in a domain-appropriate embedding space; the rare pool captures statistical atypicality in label space; the control pool is a uniform random sample, replicated over three seeds, and serves as the null condition against which subgroup effects are measured. The rarity axis is chosen per dataset because the corpora admit different notions of rarity. In N2C2, 354 of 429 label combinations occur exactly once, so combination frequency is uninformative and we instead define rarity by co-morbidity count. DMSAFN is single-label with three classes and therefore admits no rare-combination subgroup at all; it contributes outlier and control conditions only. 7.2 Details of Proxies Table 8 groups the m=32m=32 proxies of §3.6 into 13 families; all families run unchanged in all three scenarios, with only the observation object R changing (§3.5). The count expands from 13 to 32 because most families contribute several concrete features: retrieval-based overlap statistics are aggregated by both max and mean over the top-k candidates, and the neighborhood-based embedding families are instantiated at several neighborhood definitions (nearest neighbor, top-k average, radius count). The families are chosen to cover distinct signatures of memorization rather than to be independent: verbatim reuse (character n-gram containment, longest common substring), near-verbatim paraphrase (ROUGE-L, embedding similarity), distributional support (bigram log-probability, Gaussian log-likelihood), and rarity-localized copying (rare-n-gram and entity/number overlap), the last targeting the identifier-dense records typical of clinical and legal text. Within each view the proxies are highly rank-correlated by construction—they measure the same copying signal at different granularities—which is why the audit reports the per-proxy envelope rather than a learned combination, and why the fusion of Appendix 7.6 finds little complementary signal to harvest. Proxy family View What it measures Probability-like Used in scenarios n-gram log-probability Lexical Likelihood of the target text under an n-gram language model fitted on texts from the observation object R Yes S1 / S2 / S3 Gaussian log-likelihood Embedding Log-density of the target embedding under a Gaussian fitted on embeddings from R Yes S1 / S2 / S3 BM25 relevance Lexical Retrieval relevance between the target and texts in R No S1 / S2 / S3 Character n-gram overlap Lexical Local lexical copying measured by containment or Jaccard overlap over character n-grams No S1 / S2 / S3 Sequence overlap Lexical Long-span copying or paraphrase-style overlap measured by ROUGE-L, token LCS, or longest common substring No S1 / S2 / S3 Rare phrase overlap Lexical Overlap on rare or informative n-grams, optionally weighted by IDF No S1 / S2 / S3 Entity / number overlap Lexical Exact overlap on numbers, identifiers, and entity-like strings No S1 / S2 / S3 Cosine similarity Embedding Angular similarity to nearest neighbors in the embedding space of R No S1 / S2 / S3 Dot-product similarity Embedding Raw embedding affinity to nearest neighbors in R No S1 / S2 / S3 Euclidean proximity Embedding Distance-based closeness and local density in the embedding space of R No S1 / S2 / S3 Mahalanobis proximity Embedding Covariance-aware closeness to the embedding manifold induced by R No S1 / S2 / S3 CSLS similarity Embedding Hubness-corrected nearest-neighbor similarity in the embedding space of R No S1 / S2 / S3 Anomaly score Embedding Normality or atypicality under an embedding-space detector fitted on R No S1 / S2 / S3 Table 8: Proxy families used in our attacks. Each proxy is defined as a raw score ϕ⁡(x,R)φ(x;R) computed for target record x with respect to an observation object R. Depending on the attacker capability scenario, R is the target synthetic release (S1), a raw reference dataset for calibration (S2), or a synthetic reference release for calibration (S3). Probability-like proxies arise from explicitly fitted probabilistic models; the remaining proxies are empirical similarity, overlap, density, or anomaly signals. Scenario-specific calibration is handled separately and is not part of the proxy definition itself. 7.3 Reference Construction: Sharing and Its Caveats As in shadow-model practice [10, 11, 7], reference sets are shared across targets rather than built per target: several targets are inserted into each set, so each target is IN for half of the MrefM_ref references (Mref=4M_ref=4; Scenario 3 likewise reuses the generated reference releases across targets). This introduces two approximations. First, references share one auxiliary pool rather than being resampled i.i.d., so the baseline conditions on that pool and misses dataset-level sampling variance (Canary does the same, as worst-case auditing). Second, OUT references may contain other records of rG_r while the null world contains none, which can inflate the OUT baseline and make the audit conservative; the effect is largest for rules selecting mutually similar records and vanishes for Random. 7.4 Privacy accounting. We evaluate budgets ε∈∞,4,2,1,0.5 ∈\∞,4,2,1,0.5\, with ∞ the non-private setting. DP-Gen: accounted at δDP=1/N2 _DP=1/N^2 via the PRV accountant, with ℓ2 _2 clip norm 1.01.0, noise multiplier calibrated to the target (ε,δDP)( , _DP) over 10 epochs at effective batch size 16 (PsyTAR, DMSAFN), 8 (EurLex), 4 (N2C2’08). Aug-PE: 10 refinement rounds, each a Gaussian-noised nearest-neighbor histogram query on the private data; following [13] we compose across rounds at δDP=1/(Nlog⁡N) _DP=1/(N N) and calibrate the per-round noise multiplier. EPSVec: the cost is incurred once at vector release, and is split across two private steps. Stage 1 selects k=2k=2 few-shot exemplars per class by a DP histogram over a candidate pool, taking one sixth of the budget (εhist=ε/6 _hist= /6, δhist=10−6 _hist=10^-6); the remaining 5ε/65 /6 is spent on the per-layer difference vectors. At generation the released vectors are injected at layers 8–11 with scaling factor α=1.4α=1.4, the value reported by Banayeeanzade et al. [14]; at ε=∞ =∞ we use α=0.2α=0.2, since without privacy noise we find that α=1.4α=1.4 yields low-quality texts. 7.5 More Results of Our MIA Auditing Tables 9–12 repeat the main tables with per-cell 95% intervals (negative-sampling variability only, §4.6). Dataset Scenario =∞ =∞ 4 2 1 0.5 0.5 PsyTAR S1 0.79 (0.79–0.80) 45.7% 0.59 (0.57–0.60) 12.0% 0.59 (0.58–0.59) 12.5% 0.60E (0.59–0.61) 11.2% 0.61 (0.60–0.63) 4.2% S2 0.80L (0.80–0.81) 53.2% 0.59L (0.58–0.60) 9.8% 0.57 (0.56–0.59) 5.9% 0.57 (0.56–0.58) 9.2% 0.64L (0.63–0.65) 6.3% S3 0.76 (0.75–0.76) 45.3% 0.57 (0.56–0.58) 14.7% 0.59L (0.58–0.60) 11.4% 0.57 (0.56–0.59) 7.8% 0.63 (0.62–0.64) 6.5% DMSAFN S1 0.79 (0.78–0.80) 38.5% 0.60L (0.59–0.62) 4.0% 0.61L (0.60–0.62) 5.2% 0.64 (0.63–0.65) 3.5% 0.64 (0.63–0.65) 6.3% S2 0.79E (0.78–0.80) 33.5% 0.55 (0.54–0.56) 4.6% 0.59 (0.58–0.60) 7.4% 0.65L (0.64–0.66) 5.2% 0.64 (0.63–0.65) 7.1% S3 0.73 (0.73–0.74) 41.8% 0.52 (0.51–0.53) 2.2% 0.55 (0.54–0.55) 6.1% 0.61 (0.60–0.62) 6.4% 0.65L (0.64–0.66) 16.0% N2C2’08 S1 0.55 (0.54–0.56) 3.9% 0.58 (0.57–0.59) 15.6% 0.57 (0.56–0.58) 7.9% 0.57 (0.56–0.58) 12.8% 0.56 (0.55–0.57) 8.7% S2 0.60 (0.59–0.61) 11.1% 0.59 (0.58–0.60) 18.4% 0.63E (0.61–0.64) 12.6% 0.61 (0.59–0.62) 12.0% 0.63E (0.62–0.64) 9.2% S3 0.64E (0.63–0.65) 13.9% 0.59L (0.58–0.60) 4.0% 0.56 (0.55–0.57) 9.8% 0.61E (0.60–0.63) 8.6% 0.57 (0.56–0.58) 7.9% EurLex S1 0.80 (0.80–0.81) 52.0% 0.59 (0.58–0.60) 10.7% 0.56 (0.55–0.57) 5.7% 0.58L (0.57–0.59) 8.7% 0.57L (0.56–0.58) 6.4% S2 0.83 (0.83–0.84) 60.2% 0.57 (0.56–0.58) 10.0% 0.56 (0.55–0.57) 4.7% 0.58 (0.56–0.59) 9.0% 0.56 (0.55–0.57) 10.0% S3 0.83E (0.83–0.84) 61.4% 0.59E (0.58–0.60) 11.9% 0.58E (0.57–0.59) 3.7% 0.55 (0.53–0.56) 9.0% 0.54 (0.53–0.55) 7.5% • Each cell reports the best-feature AUC-ROC with 95 % CI via normal approximation over 50 evaluation rounds sharing a fixed set of 100 releases (CI = mean ± 1.96 σ/50 50; the interval reflects negative-sampling variability only, see Section 4.6) / maxd⁡TPR⁡(d) _d\,TPR(d) s.t. FPR⁡(d)≤0.05FPR(d)≤ 0.05 for that same feature. • Shaded cell = best scenario for that (dataset, ε ), ranked by AUC and tie-broken by TPR. The superscript marks whether the winning proxy is Lexical or Embedding: 11 vs. 9 of 20 highlighted cells. Individual feature names are given in the companion CSV rather than here: several proxies are rank-equivalent (e.g. Gaussian log-likelihood and the Mahalanobis family), so the view is identified but the particular feature within a tied group is not. Table 9: Overall subgroup-targeted membership inference performance across privacy budgets and attacker capability scenarios for DP-Gen. The best-performing feature is selected on the evaluation episodes (empirical worst-case audit; see Section 4.7). Dataset Scenario =∞ =∞ 4 2 1 0.5 0.5 PsyTAR S1 0.59 (0.59–0.60) 11.8% 0.58 (0.58–0.59) 12.7% 0.59L (0.58–0.60) 13.8% 0.59 (0.58–0.60) 10.0% 0.59L (0.58–0.60) 10.5% S2 0.58 (0.57–0.59) 8.7% 0.58 (0.57–0.59) 13.9% 0.59 (0.58–0.60) 10.3% 0.59 (0.58–0.60) 7.6% 0.57 (0.57–0.58) 5.8% S3 0.60L (0.58–0.61) 19.0% 0.64L (0.63–0.66) 14.8% 0.59 (0.58–0.60) 9.1% 0.60L (0.59–0.61) 6.3% 0.57 (0.56–0.58) 8.5% DMSAFN S1 0.61 (0.60–0.63) 5.3% 0.61 (0.60–0.62) 3.8% 0.61L (0.60–0.62) 4.8% 0.61L (0.60–0.62) 3.1% 0.61 (0.60–0.62) 3.3% S2 0.63L (0.62–0.64) 16.1% 0.61 (0.60–0.62) 7.4% 0.60 (0.58–0.61) 3.0% 0.61 (0.60–0.61) 14.5% 0.62L (0.61–0.63) 15.3% S3 0.56 (0.55–0.57) 8.5% 0.63L (0.62–0.64) 9.9% 0.56 (0.55–0.56) 11.3% 0.60 (0.59–0.61) 10.4% 0.56 (0.55–0.57) 8.8% N2C2’08 S1 0.58 (0.57–0.59) 16.5% 0.55 (0.54–0.56) 12.8% 0.56 (0.55–0.57) 5.3% 0.58 (0.57–0.59) 13.2% 0.58 (0.57–0.59) 5.8% S2 0.60E (0.59–0.62) 11.7% 0.60E (0.59–0.60) 6.2% 0.62E (0.61–0.63) 9.9% 0.61 (0.60–0.62) 8.3% 0.60 (0.59–0.61) 12.4% S3 0.57 (0.56–0.58) 11.3% 0.55 (0.54–0.56) 8.8% 0.58 (0.57–0.59) 7.3% 0.63L (0.62–0.63) 6.2% 0.63L (0.62–0.64) 8.6% EurLex S1 0.57 (0.57–0.58) 5.6% 0.57 (0.56–0.58) 10.5% 0.58 (0.57–0.59) 6.1% 0.63 (0.63–0.64) 11.8% 0.59L (0.58–0.60) 7.2% S2 0.58L (0.57–0.59) 5.5% 0.58 (0.57–0.59) 9.4% 0.59L (0.58–0.60) 6.1% 0.64E (0.63–0.65) 12.6% 0.56 (0.55–0.57) 6.2% S3 0.56 (0.55–0.57) 6.3% 0.59L (0.58–0.60) 7.8% 0.58 (0.57–0.59) 7.2% 0.61 (0.60–0.62) 8.4% 0.58 (0.57–0.59) 6.0% Table 10: Overall subgroup-targeted membership inference performance across privacy budgets and attacker capability scenarios for EPSVec. Winning proxy is Lexical or Embedding: 16 vs. 4 of 20 highlighted cells; cell format, shading and notes as in Table 9. Dataset Scenario =∞ =∞ 4 2 1 0.5 0.5 PsyTAR S1 0.59 (0.58–0.60) 10.9% 0.59 (0.58–0.60) 13.7% 0.59 (0.58–0.60) 12.4% 0.59 (0.59–0.60) 13.3% 0.61 (0.60–0.62) 11.7% S2 0.64L (0.63–0.65) 13.3% 0.62L (0.61–0.64) 12.4% 0.65L (0.64–0.66) 13.5% 0.63E (0.62–0.64) 15.2% 0.63L (0.62–0.64) 12.2% S3 0.56 (0.54–0.57) 1.1% 0.56 (0.55–0.57) 8.0% 0.61 (0.60–0.63) 4.4% 0.58 (0.57–0.60) 4.5% 0.59 (0.58–0.60) 7.4% DMSAFN S1 0.61L (0.60–0.62) 4.2% 0.61L (0.60–0.62) 5.2% 0.61 (0.60–0.62) 5.5% 0.61L (0.60–0.62) 5.2% 0.61L (0.60–0.62) 4.3% S2 0.60 (0.59–0.61) 4.7% 0.59 (0.58–0.59) 10.5% 0.60 (0.58–0.61) 3.6% 0.60 (0.59–0.62) 4.3% 0.59 (0.58–0.60) 4.1% S3 0.58 (0.57–0.60) 11.3% 0.59 (0.58–0.61) 7.6% 0.62E (0.61–0.63) 8.9% 0.58 (0.57–0.59) 8.4% 0.55 (0.54–0.55) 3.5% N2C2’08 S1 0.60 (0.59–0.61) 5.5% 0.59 (0.58–0.60) 13.2% 0.56 (0.55–0.56) 15.7% 0.58 (0.57–0.59) 10.1% 0.57 (0.56–0.58) 14.7% S2 0.64E (0.63–0.65) 9.9% 0.59L (0.59–0.60) 14.9% 0.62E (0.61–0.63) 12.5% 0.60 (0.59–0.61) 12.2% 0.62E (0.61–0.63) 9.9% S3 0.63 (0.62–0.64) 6.6% 0.59 (0.58–0.59) 8.4% 0.60 (0.59–0.60) 6.6% 0.62E (0.61–0.63) 8.7% 0.60 (0.60–0.61) 12.3% EurLex S1 0.59 (0.58–0.60) 8.1% 0.57L (0.56–0.58) 5.4% 0.59L (0.58–0.60) 7.4% 0.58L (0.58–0.59) 7.8% 0.57 (0.56–0.58) 9.7% S2 0.59 (0.58–0.60) 6.3% 0.55 (0.54–0.56) 8.0% 0.59 (0.58–0.60) 10.0% 0.56 (0.55–0.57) 4.9% 0.58 (0.57–0.59) 5.7% S3 0.59L (0.59–0.60) 8.8% 0.55 (0.54–0.56) 2.8% 0.56 (0.55–0.57) 9.0% 0.58 (0.57–0.59) 6.5% 0.61L (0.60–0.62) 10.8% Table 11: Overall subgroup-targeted membership inference performance across privacy budgets and attacker capability scenarios for Aug-PE. Winning proxy is Lexical or Embedding: 14 vs. 6 of 20 highlighted cells; cell format, shading and notes as in Table 9. Generator Dataset Scenario =∞ =∞ 4 1 DP-Gen PsyTAR S1 0.82 (0.82–0.83) 49.4% 0.59E (0.58–0.60) 13.1% 0.57 (0.56–0.58) 9.3% S2 0.84 (0.83–0.84) 46.4% 0.53 (0.52–0.54) 11.2% 0.55 (0.54–0.56) 4.0% S3 0.86L (0.85–0.86) 58.8% 0.56 (0.55–0.58) 9.3% 0.58L (0.57–0.59) 10.8% N2C2’08 S1 0.62 (0.61–0.63) 13.7% 0.53 (0.51–0.54) 5.4% 0.53 (0.52–0.54) 6.9% S2 0.69 (0.68–0.70) 15.3% 0.55 (0.54–0.56) 2.2% 0.56 (0.56–0.57) 3.2% S3 0.77L (0.76–0.78) 28.0% 0.56L (0.56–0.57) 8.2% 0.58E (0.58–0.59) 2.8% EurLex S1 0.63 (0.62–0.64) 13.2% 0.57E (0.56–0.59) 8.7% 0.56 (0.55–0.57) 6.1% S2 0.61 (0.60–0.61) 10.8% 0.56 (0.55–0.58) 8.2% 0.57L (0.56–0.58) 5.7% S3 0.64L (0.63–0.65) 22.9% 0.55 (0.54–0.56) 7.4% 0.55 (0.54–0.57) 4.5% EPSVec PsyTAR S1 0.57 (0.56–0.58) 7.8% 0.60E (0.59–0.61) 15.8% 0.58E (0.56–0.59) 8.7% S2 0.54 (0.53–0.55) 5.5% 0.56 (0.55–0.57) 6.5% 0.57 (0.56–0.58) 7.6% S3 0.59E (0.58–0.61) 7.4% 0.58 (0.57–0.59) 12.5% 0.57 (0.56–0.59) 7.7% N2C2’08 S1 0.52 (0.51–0.53) 2.9% 0.56 (0.55–0.57) 15.5% 0.56E (0.55–0.56) 5.7% S2 0.55 (0.54–0.56) 5.8% 0.56 (0.55–0.57) 3.5% 0.55 (0.54–0.56) 11.5% S3 0.58E (0.58–0.59) 11.4% 0.66L (0.65–0.67) 9.5% 0.55 (0.54–0.56) 3.3% EurLex S1 0.62E (0.61–0.63) 9.2% 0.62E (0.61–0.63) 9.9% 0.62E (0.60–0.63) 9.4% S2 0.58 (0.57–0.59) 8.0% 0.62 (0.61–0.63) 10.8% 0.58 (0.57–0.59) 6.4% S3 0.59 (0.58–0.60) 11.9% 0.56 (0.55–0.57) 3.1% 0.57 (0.57–0.58) 6.3% • Each cell reports the best-feature AUC-ROC with 95 % CI via normal approximation over 50 evaluation rounds sharing a fixed set of 100 releases (CI = mean ± 1.96 σ/50 50; the interval reflects negative-sampling variability only, see Section 4.6) / maxd⁡TPR⁡(d) _d\,TPR(d) s.t. FPR⁡(d)≤0.05FPR(d)≤ 0.05 for that same feature. This sweep covers ε∈∞,4,1 ∈\∞,4,1\ only; the remaining budgets were not run for this group. • Shaded cell = best scenario for that (dataset, ε ), ranked by AUC and tie-broken by TPR. The superscript marks whether the winning proxy is Lexical or Embedding: 7 vs. 11 of 18 highlighted cells. Individual feature names are given in the companion CSV rather than here: several proxies are rank-equivalent (e.g. Gaussian log-likelihood and the Mahalanobis family), so the view is identified but the particular feature within a tied group is not. Table 12: Overall rare-label subgroup-targeted membership inference performance across generators, privacy budgets and attacker capability scenarios. The best-performing feature is selected on the evaluation episodes (empirical worst-case audit; see Section 4.7). 7.6 Meta-Classifier: A Negative Result Table 13: Meta-classifier (fusion of all 32 proxies) on the outlier group, DP-Gen releases: AUC, TPR and the gain over the best single proxy. Generator Dataset Scenario =∞ =∞ 4 2 1 0.5 0.5 DP-Gen PsyTAR S1 0.76 (0.76–0.77) 51.5% −-0.027 L 0.52 (0.51–0.54) 7.2% −-0.062 E 0.56 (0.54–0.58) 9.7% −-0.025 L 0.54 (0.52–0.55) 8.0% −-0.066 E 0.55 (0.53–0.56) 6.8% −-0.068 L S2 0.76 (0.75–0.77) 46.9% −-0.041 L 0.53 (0.51–0.55) 8.3% −-0.060 L 0.56 (0.55–0.58) 10.4% −-0.013 L 0.51 (0.50–0.53) 8.9% −-0.060 L 0.54 (0.53–0.56) 7.9% −-0.093 L S3 0.71 (0.70–0.72) 44.1% −-0.049 L 0.45 (0.43–0.46) 4.8% −-0.117 E 0.56 (0.55–0.58) 10.3% −-0.024 L 0.51 (0.50–0.53) 7.9% −-0.059 L 0.53 (0.51–0.55) 5.5% −-0.098 L DMSAFN S1 0.70 (0.68–0.71) 28.3% −-0.093 E 0.64 (0.62–0.65) 9.3% ++0.034 L 0.55 (0.54–0.57) 7.1% −-0.057 L 0.55 (0.54–0.57) 5.9% −-0.084 L 0.56 (0.54–0.57) 6.3% −-0.081 L S2 0.69 (0.67–0.70) 29.8% −-0.102 E 0.52 (0.50–0.53) 6.8% −-0.036 L 0.50 (0.48–0.51) 5.0% −-0.094 L 0.52 (0.50–0.54) 7.5% −-0.123 L 0.54 (0.53–0.56) 7.3% −-0.100 L S3 0.69 (0.68–0.71) 29.4% −-0.040 E 0.63 (0.61–0.64) 10.8% ++0.109 E 0.45 (0.43–0.46) 4.8% −-0.096 E 0.54 (0.52–0.56) 6.2% −-0.075 E 0.58 (0.56–0.60) 7.2% −-0.062 L N2C2’08 S1 0.49 (0.47–0.51) 7.5% −-0.062 L 0.53 (0.51–0.55) 5.5% −-0.043 L 0.48 (0.46–0.50) 3.8% −-0.093 E 0.53 (0.52–0.55) 7.4% −-0.038 L 0.46 (0.44–0.48) 3.2% −-0.101 E S2 0.54 (0.52–0.55) 9.2% −-0.065 L 0.57 (0.55–0.59) 8.2% −-0.019 L 0.53 (0.51–0.55) 6.2% −-0.092 E 0.56 (0.54–0.57) 8.2% −-0.048 E 0.53 (0.51–0.55) 6.3% −-0.097 E S3 0.55 (0.53–0.57) 5.9% −-0.089 E 0.49 (0.47–0.51) 4.2% −-0.102 L 0.48 (0.46–0.51) 3.8% −-0.079 E 0.52 (0.50–0.54) 6.9% −-0.098 E 0.44 (0.42–0.46) 4.2% −-0.131 E EurLex S1 0.79 (0.78–0.80) 50.0% −-0.013 E 0.48 (0.46–0.50) 4.2% −-0.105 E 0.47 (0.45–0.49) 5.8% −-0.092 L 0.44 (0.43–0.46) 4.2% −-0.136 L 0.48 (0.46–0.50) 4.7% −-0.098 L S2 0.79 (0.78–0.80) 49.9% −-0.037 E 0.49 (0.47–0.51) 4.0% −-0.078 E 0.45 (0.43–0.47) 4.4% −-0.108 L 0.43 (0.41–0.44) 3.9% −-0.150 L 0.44 (0.42–0.46) 5.0% −-0.125 L S3 0.79 (0.79–0.80) 48.1% −-0.041 E 0.49 (0.47–0.50) 4.2% −-0.100 E 0.49 (0.47–0.50) 5.9% −-0.093 E 0.47 (0.45–0.49) 3.1% −-0.078 L 0.45 (0.43–0.47) 4.7% −-0.089 L • Top line: AUC-ROC of the meta-classifier (mean ± 95 % CI, normal approximation over 50 evaluation rounds; CI == mean ± 1.96 σ/50σ/ 50). Every round draws the negative candidates from the same evaluation trial plan the main tables use, so a cell here and the corresponding main-table cell see identical data. Within a round, membership probabilities come from nested cross-validation (outer 5-fold out-of-fold prediction; inner 5-fold grid search over the number of selected features k∈5,8,12,allk∈\5,8,12,all\ and the ℓ2 _2 penalty C∈0.01,0.1,1,10C∈\0.01,0.1,1,10\ of a class-balanced logistic regression on standardised, mean-imputed features), so no record contributes to the model that scores it. • Bottom line: TPR at FPR ≤0.05≤ 0.05, then Δ = meta-classifier AUC −- AUC of the best of the 32 individual proxies, recomputed on the same rounds. That reference is an oracle — the proxy is chosen after seeing the evaluation data, which is what the main table reports — so Δ>0 >0 (shaded) means the fusion beats a single proxy no adversary could have picked in advance. The superscript gives that proxy’s view (Lexical / Embedding); its name, the modal (k,C)(k,C) and the number of scored records per round are in the companion CSV. • Across the 60 populated cells the fusion wins in 2 and loses in 58, mean Δ=−0.071 =-0.071. Table 14: Meta-classifier (fusion of all 32 proxies) on the outlier group, EPSVec releases: AUC, TPR and the gain over the best single proxy. Companion to Table 13. Generator Dataset Scenario =∞ =∞ 4 2 1 0.5 0.5 EPSVec PsyTAR S1 0.51 (0.50–0.53) 6.0% −-0.081 L 0.55 (0.54–0.57) 8.0% −-0.033 L 0.54 (0.52–0.56) 5.4% −-0.054 L 0.55 (0.54–0.57) 9.0% −-0.038 L 0.54 (0.52–0.55) 6.9% −-0.055 L S2 0.53 (0.51–0.54) 7.3% −-0.055 L 0.54 (0.52–0.55) 7.7% −-0.042 L 0.54 (0.53–0.56) 7.2% −-0.042 L 0.55 (0.53–0.57) 8.1% −-0.040 L 0.54 (0.53–0.56) 7.7% −-0.030 L S3 0.48 (0.46–0.49) 5.9% −-0.119 L 0.55 (0.54–0.57) 8.1% −-0.091 L 0.52 (0.51–0.54) 6.8% −-0.069 L 0.52 (0.50–0.54) 6.6% −-0.076 L 0.52 (0.51–0.54) 6.1% −-0.050 L DMSAFN S1 0.56 (0.54–0.58) 6.8% −-0.053 L 0.57 (0.56–0.59) 5.3% −-0.041 L 0.56 (0.54–0.57) 6.8% −-0.049 L 0.60 (0.59–0.62) 8.7% −-0.010 L 0.55 (0.53–0.56) 5.3% −-0.064 L S2 0.53 (0.51–0.55) 5.6% −-0.100 L 0.57 (0.55–0.58) 7.5% −-0.044 L 0.54 (0.53–0.56) 7.5% −-0.052 L 0.56 (0.54–0.57) 8.6% −-0.048 L 0.55 (0.53–0.56) 7.0% −-0.075 L S3 0.49 (0.48–0.51) 4.3% −-0.068 L 0.55 (0.53–0.57) 7.4% −-0.083 L 0.56 (0.54–0.58) 7.7% −-0.002 E 0.51 (0.50–0.53) 5.0% −-0.083 E 0.51 (0.49–0.53) 5.1% −-0.044 L N2C2’08 S1 0.42 (0.40–0.44) 3.5% −-0.155 L 0.43 (0.41–0.45) 3.4% −-0.116 L 0.44 (0.42–0.45) 3.0% −-0.126 E 0.56 (0.54–0.57) 7.5% −-0.025 L 0.46 (0.44–0.48) 3.8% −-0.118 E S2 0.50 (0.48–0.52) 6.2% −-0.102 E 0.48 (0.46–0.49) 4.7% −-0.118 E 0.49 (0.47–0.51) 3.5% −-0.125 E 0.51 (0.49–0.53) 4.9% −-0.101 E 0.51 (0.48–0.53) 7.3% −-0.091 E S3 0.43 (0.41–0.44) 3.4% −-0.145 L 0.47 (0.45–0.49) 4.2% −-0.078 E 0.44 (0.43–0.46) 3.5% −-0.137 E 0.51 (0.49–0.53) 7.2% −-0.115 L 0.49 (0.48–0.50) 4.2% −-0.138 L EurLex S1 0.42 (0.40–0.44) 2.9% −-0.155 L 0.46 (0.44–0.48) 4.8% −-0.113 L 0.48 (0.46–0.50) 7.5% −-0.105 L 0.54 (0.52–0.56) 7.3% −-0.093 E 0.48 (0.46–0.49) 5.6% −-0.109 L S2 0.41 (0.40–0.43) 4.0% −-0.168 L 0.45 (0.43–0.47) 5.0% −-0.131 L 0.45 (0.43–0.47) 5.5% −-0.134 L 0.55 (0.54–0.57) 9.6% −-0.089 E 0.44 (0.43–0.45) 3.8% −-0.119 L S3 0.52 (0.51–0.54) 4.4% −-0.038 E 0.49 (0.47–0.50) 5.3% −-0.103 L 0.46 (0.44–0.48) 4.0% −-0.120 L 0.50 (0.49–0.52) 5.4% −-0.107 L 0.48 (0.47–0.50) 5.0% −-0.100 E • Cell format, protocol and Δ as in Table 13. • Across the 60 populated cells the fusion wins in 0 and loses in 60, mean Δ=−0.084 =-0.084. Table 15: Meta-classifier (fusion of all 32 proxies) on the outlier group, Aug-PE releases: AUC, TPR and the gain over the best single proxy. Companion to Table 13. Generator Dataset Scenario =∞ =∞ 4 2 1 0.5 0.5 Aug-PE PsyTAR S1 0.54 (0.52–0.55) 8.3% −-0.054 L 0.53 (0.52–0.55) 7.5% −-0.056 L 0.55 (0.53–0.56) 7.4% −-0.046 L 0.56 (0.54–0.57) 9.3% −-0.038 L 0.53 (0.51–0.55) 6.6% −-0.076 L S2 0.56 (0.54–0.57) 7.7% −-0.079 L 0.55 (0.53–0.57) 8.3% −-0.076 L 0.57 (0.55–0.59) 9.7% −-0.073 L 0.55 (0.53–0.57) 8.6% −-0.082 E 0.56 (0.54–0.58) 7.9% −-0.074 L S3 0.52 (0.51–0.54) 7.9% −-0.035 L 0.53 (0.52–0.54) 6.3% −-0.028 E 0.56 (0.54–0.57) 9.2% −-0.056 L 0.52 (0.50–0.53) 6.4% −-0.069 L 0.53 (0.51–0.55) 6.9% −-0.055 L DMSAFN S1 0.56 (0.54–0.57) 6.4% −-0.055 L 0.61 (0.59–0.62) 8.1% −-0.002 L 0.56 (0.54–0.58) 6.7% −-0.050 L 0.56 (0.54–0.57) 6.2% −-0.057 L 0.58 (0.57–0.60) 6.2% −-0.026 L S2 0.51 (0.49–0.53) 4.8% −-0.089 L 0.58 (0.57–0.60) 8.7% −-0.002 L 0.53 (0.52–0.55) 6.0% −-0.060 L 0.52 (0.50–0.53) 5.9% −-0.083 L 0.53 (0.52–0.55) 5.5% −-0.059 L S3 0.49 (0.47–0.51) 6.5% −-0.093 E 0.64 (0.63–0.66) 14.5% ++0.048 L 0.48 (0.46–0.50) 5.2% −-0.141 E 0.46 (0.45–0.48) 5.0% −-0.113 L 0.46 (0.45–0.48) 3.6% −-0.083 E N2C2’08 S1 0.46 (0.44–0.49) 3.3% −-0.134 E 0.46 (0.44–0.48) 2.6% −-0.130 L 0.45 (0.44–0.47) 4.5% −-0.106 L 0.45 (0.43–0.47) 3.7% −-0.126 L 0.43 (0.42–0.45) 4.6% −-0.138 L S2 0.53 (0.52–0.55) 4.5% −-0.106 E 0.50 (0.48–0.52) 4.2% −-0.089 L 0.51 (0.49–0.52) 7.0% −-0.109 E 0.50 (0.48–0.52) 5.9% −-0.098 E 0.50 (0.48–0.52) 6.2% −-0.125 E S3 0.51 (0.49–0.53) 5.0% −-0.122 L 0.46 (0.44–0.47) 2.9% −-0.129 L 0.51 (0.50–0.53) 6.0% −-0.085 L 0.51 (0.50–0.53) 3.8% −-0.109 E 0.50 (0.48–0.51) 7.3% −-0.105 L EurLex S1 0.45 (0.43–0.46) 3.8% −-0.141 L 0.43 (0.41–0.45) 3.7% −-0.136 L 0.44 (0.42–0.46) 3.1% −-0.147 L 0.46 (0.44–0.48) 4.6% −-0.122 L 0.58 (0.55–0.60) 8.7% ++0.006 L S2 0.47 (0.45–0.49) 4.8% −-0.123 L 0.44 (0.42–0.46) 4.5% −-0.115 E 0.47 (0.45–0.49) 4.5% −-0.118 E 0.43 (0.41–0.45) 4.2% −-0.129 L 0.47 (0.45–0.49) 4.2% −-0.111 E S3 0.49 (0.47–0.51) 4.7% −-0.101 L 0.45 (0.43–0.47) 4.2% −-0.107 E 0.50 (0.48–0.52) 6.1% −-0.060 E 0.42 (0.41–0.44) 3.2% −-0.159 L 0.55 (0.53–0.56) 6.2% −-0.059 L • Cell format, protocol and Δ as in Table 13. • Across the 60 populated cells the fusion wins in 2 and loses in 58, mean Δ=−0.085 =-0.085. Table 16: Meta-classifier (fusion of all 32 proxies) on the rare-label group: AUC, TPR and the gain over the best single proxy. Generator Dataset Scenario =∞ =∞ 4 1 DP-Gen PsyTAR S1 0.78 (0.77–0.79) 45.7% −-0.036 L 0.48 (0.46–0.50) 5.6% −-0.112 E 0.48 (0.47–0.50) 5.1% −-0.089 E S2 0.79 (0.78–0.80) 47.4% −-0.050 L 0.42 (0.40–0.44) 5.1% −-0.110 E 0.46 (0.44–0.48) 4.5% −-0.094 E S3 0.83 (0.82–0.84) 52.6% −-0.026 L 0.45 (0.43–0.47) 4.4% −-0.115 L 0.49 (0.47–0.51) 5.7% −-0.097 L N2C2’08 S1 0.56 (0.55–0.58) 7.1% −-0.059 L 0.46 (0.44–0.48) 2.9% −-0.068 L 0.43 (0.41–0.45) 3.5% −-0.099 E S2 0.69 (0.68–0.71) 11.4% ++0.005 L 0.47 (0.46–0.49) 3.0% −-0.072 E 0.50 (0.49–0.52) 3.6% −-0.059 E S3 0.72 (0.71–0.73) 18.5% −-0.051 L 0.44 (0.42–0.46) 2.7% −-0.125 L 0.50 (0.49–0.51) 6.1% −-0.085 E EurLex S1 0.55 (0.53–0.56) 8.8% −-0.078 E 0.48 (0.46–0.51) 4.3% −-0.090 E 0.47 (0.45–0.49) 4.0% −-0.096 E S2 0.53 (0.51–0.54) 6.5% −-0.080 E 0.47 (0.46–0.49) 4.8% −-0.091 E 0.45 (0.43–0.47) 4.8% −-0.118 L S3 0.54 (0.53–0.56) 7.1% −-0.094 L 0.43 (0.42–0.45) 4.0% −-0.113 E 0.42 (0.41–0.44) 3.6% −-0.130 E EPSVec PsyTAR S1 0.46 (0.44–0.48) 3.9% −-0.113 E 0.50 (0.48–0.52) 4.6% −-0.096 E 0.48 (0.46–0.50) 4.7% −-0.097 E S2 0.44 (0.42–0.46) 3.6% −-0.102 E 0.48 (0.45–0.50) 5.8% −-0.084 E 0.52 (0.49–0.54) 6.4% −-0.054 E S3 0.54 (0.52–0.56) 5.2% −-0.056 E 0.50 (0.49–0.52) 7.4% −-0.079 E 0.48 (0.46–0.50) 5.5% −-0.092 E N2C2’08 S1 0.45 (0.44–0.47) 3.4% −-0.065 E 0.54 (0.52–0.55) 4.6% −-0.027 E 0.47 (0.45–0.49) 3.4% −-0.088 E S2 0.49 (0.47–0.50) 4.9% −-0.061 E 0.54 (0.52–0.56) 5.4% −-0.024 L 0.51 (0.50–0.53) 3.7% −-0.036 E S3 0.45 (0.43–0.47) 3.7% −-0.133 E 0.57 (0.55–0.58) 6.0% −-0.096 L 0.51 (0.50–0.52) 5.1% −-0.038 E EurLex S1 0.51 (0.49–0.54) 6.3% −-0.106 E 0.55 (0.53–0.57) 6.3% −-0.071 E 0.52 (0.49–0.54) 4.4% −-0.099 E S2 0.51 (0.48–0.53) 5.1% −-0.077 E 0.53 (0.51–0.55) 5.2% −-0.088 E 0.45 (0.43–0.47) 4.0% −-0.131 E S3 0.49 (0.47–0.51) 4.5% −-0.101 L 0.46 (0.44–0.48) 3.5% −-0.103 L 0.51 (0.49–0.54) 6.4% −-0.060 L • Top line: AUC-ROC of the meta-classifier (mean ± 95 % CI, normal approximation over 50 evaluation rounds; CI == mean ± 1.96 σ/50σ/ 50). Every round draws the negative candidates from the same evaluation trial plan the main tables use, so a cell here and the corresponding main-table cell see identical data. Within a round, membership probabilities come from nested cross-validation (outer 5-fold out-of-fold prediction; inner 5-fold grid search over the number of selected features k∈5,8,12,allk∈\5,8,12,all\ and the ℓ2 _2 penalty C∈0.01,0.1,1,10C∈\0.01,0.1,1,10\ of a class-balanced logistic regression on standardised, mean-imputed features), so no record contributes to the model that scores it. • Bottom line: TPR at FPR ≤0.05≤ 0.05, then Δ = meta-classifier AUC −- AUC of the best of the 32 individual proxies, recomputed on the same rounds. That reference is an oracle — the proxy is chosen after seeing the evaluation data, which is what the main table reports — so Δ>0 >0 (shaded) means the fusion beats a single proxy no adversary could have picked in advance. The superscript gives that proxy’s view (Lexical / Embedding); its name, the modal (k,C)(k,C) and the number of scored records per round are in the companion CSV. • Across the 54 populated cells the fusion wins in 1 and loses in 53, mean Δ=−0.082 =-0.082. Tables 13–16 evaluate a fusion attack that combines all 32 proxies through a class-balanced logistic regression with nested cross-validation (outer 5-fold out-of-fold prediction; inner 5-fold selection of the feature count and ℓ2 _2 penalty), so that no record contributes to the model that scores it. The comparison is deliberately unfavorable to the fusion: its reference is the oracle best single proxy, selected per cell after seeing the evaluation data—the same worst-case selection the main tables report. The fusion does not close the gap to the oracle. Across the 234 populated (generator, dataset, scenario, ε ) cells it wins in 4 and loses in 200, with a mean deficit of Δ=−0.077 =-0.077 AUC (outlier pools: 2/602/60 under DP-Gen, 0/600/60 under EPSVec, 1/451/45 under Aug-PE; rare pools: 1/541/54). Its absolute performance is nonetheless informative: on the strong non-private fine-tuning cells the deployable fusion still reaches AUC 0.76–0.79 on the outlier pools and up to 0.83 on the rare pools, within 0.03–0.05 of the per-cell oracle, so the headline leakage of §5.1 does not depend on oracle proxy selection. Under DP, where the best single proxy sits at 0.55–0.65, the fusion typically falls to 0.44–0.57—often below the oracle by more than the oracle exceeds chance. We attribute the failure to the geometry of the proxy family rather than to the classifier: the proxies within each view are highly rank-correlated, so the fusion has few effectively independent signals to combine, while the per-round training sets (tens of positive records) are small relative to even the reduced feature dimension; in this regime the cross-validated model’s variance costs more than the (near-zero) complementary signal it could harvest. Two practical conclusions follow. First, the oracle best-proxy numbers in the main tables should be read as what they are—an upper envelope over the family—but the envelope is not an artifact of selection: a deployable attacker gets within a few points of it wherever leakage is strong. Second, there is no evidence that lexical and embedding evidence are complementary at the record level; the two views agree on which records look like members, which is consistent with the mechanism-bound picture of §5.5. 7.7 Extended Baseline Comparisons Tables 17 and 18 give the full per-cell results for both baseline families under every privacy budget and variant we swept (§4.4), complementing the ε∈∞,1 ∈\∞,1\ cells shown in §5.3. Each method is maximized over its own variants independently in each cell, so all three columns receive the same worst-case treatment. Across all 78 populated (dataset, generator, ε , pool) cells—the main-text and appendix budgets combined—our audit attains the highest AUC in 69, ties in 2, and is beaten in 7, every loss smaller than 0.020.02 AUC and confined to low-memorization configurations (EPSVec at any budget, or DP-SGD fine-tuning at ε≤4 ≤ 4). Table 17: Subgroup game: remaining privacy budgets; format, variant codes, shading and notes as in Table 4. ε=4 =4 ε=2 =2 ε=0.5 =0.5 Dataset Generator canary domias ours Δ canary domias ours Δ canary domias ours Δ PsyTAR DP-Gen 0.588¯ 0.588n / 14% 0.6040.604g / 14% 0.5870.587L / 10% −0.017-0.017 0.5890.589n / 13% 0.5710.571k / 7% 0.586¯ 0.586L / 11% −0.003-0.003 0.614¯ 0.614nR / 6% 0.5700.570k / 4% 0.6370.637L / 6% +0.023+0.023 PsyTAR Aug-PE 0.588¯ 0.588n / 14% 0.5230.523k / 8% 0.6240.624L / 12% +0.036+0.036 0.589¯ 0.589n / 10% 0.5180.518k / 6% 0.6460.646L / 14% +0.057+0.057 0.593¯ 0.593n / 15% 0.5580.558k / 8% 0.6310.631L / 12% +0.037+0.037 PsyTAR EPSVec 0.589¯ 0.589n / 10% 0.5270.527k / 5% 0.6450.645L / 15% +0.056+0.056 0.5970.597n / 15% 0.5530.553k / 7% 0.593¯ 0.593L / 14% −0.004-0.004 0.590¯ 0.590n / 11% 0.5010.501k / 9% 0.5910.591L / 10% +0.001+0.001 DMSAFN DP-Gen 0.594¯ 0.594n / 2% 0.5490.549pg / 12% 0.6040.604L / 4% +0.010+0.010 0.599¯ 0.599n / 3% 0.5470.547pg / 5% 0.6100.610L / 5% +0.011+0.011 0.632¯ 0.632nR / 11% 0.4870.487pk / 6% 0.6470.647L / 16% +0.014+0.014 DMSAFN Aug-PE 0.604¯ 0.604n / 4% 0.5370.537k / 4% 0.6080.608L / 5% +0.004+0.004 0.599¯ 0.599n / 3% 0.4540.454g / 4% 0.6210.621E / 9% +0.022+0.022 0.600¯ 0.600n / 3% 0.5370.537pk / 5% 0.6090.609L / 4% +0.009+0.009 DMSAFN EPSVec 0.599¯ 0.599n / 3% 0.5350.535k / 4% 0.6340.634L / 10% +0.035+0.035 0.600¯ 0.600n / 3% 0.4990.499k / 6% 0.6080.608L / 5% +0.007+0.007 0.600¯ 0.600n / 3% 0.4910.491pk / 7% 0.6210.621L / 15% +0.021+0.021 N2C2’08 DP-Gen 0.5230.523s / 6% 0.580¯ 0.580pk / 8% 0.5900.590L / 4% +0.010+0.010 0.5180.518s / 9% 0.568¯ 0.568pk / 7% 0.6250.625E / 13% +0.057+0.057 0.5660.566sR / 11% 0.615¯ 0.615pk / 7% 0.6290.629E / 9% +0.014+0.014 N2C2’08 Aug-PE 0.4960.496nR / 8% 0.555¯ 0.555pk / 9% 0.5930.593L / 15% +0.038+0.038 0.5250.525nR / 6% 0.610¯ 0.610pk / 9% 0.6170.617E / 13% +0.007+0.007 0.5300.530s / 2% 0.555¯ 0.555k / 8% 0.6220.622E / 10% +0.067+0.067 N2C2’08 EPSVec 0.4770.477s / 5% 0.559¯ 0.559pk / 6% 0.5950.595E / 6% +0.036+0.036 0.5620.562sR / 12% 0.595¯ 0.595k / 12% 0.6170.617E / 10% +0.022+0.022 0.4730.473sR / 5% 0.525¯ 0.525pk / 3% 0.6290.629L / 9% +0.104+0.104 EUR-Lex DP-Gen 0.5170.517n / 7% 0.565¯ 0.565g / 9% 0.5890.589E / 12% +0.024+0.024 0.5200.520n / 6% 0.545¯ 0.545g / 6% 0.5800.580E / 4% +0.035+0.035 0.5190.519n / 7% 0.537¯ 0.537g / 5% 0.5750.575L / 6% +0.038+0.038 EUR-Lex Aug-PE 0.543¯ 0.543nR / 4% 0.5420.542g / 8% 0.5680.568L / 5% +0.025+0.025 0.5130.513n / 6% 0.542¯ 0.542k / 7% 0.5890.589L / 7% +0.048+0.048 0.5530.553nR / 3% 0.604¯ 0.604k / 13% 0.6050.605L / 11% +0.001+0.001 EUR-Lex EPSVec 0.5170.517n / 7% 0.559¯ 0.559g / 8% 0.5890.589L / 8% +0.030+0.030 0.5270.527nR / 7% 0.553¯ 0.553k / 13% 0.5860.586L / 6% +0.032+0.032 0.5170.517sR / 3% 0.583¯ 0.583k / 9% 0.5860.586L / 7% +0.002+0.002 Table 18: Rare-label pools; format, variant codes, shading and notes as in Table 4. ε=4 =4 Dataset Generator canary domias ours Δ PsyTAR DP-Gen 0.571¯ 0.571s / 11% 0.5310.531pg / 7% 0.5880.588E / 13% +0.018+0.018 PsyTAR EPSVec 0.572¯ 0.572s / 8% 0.5700.570pg / 6% 0.5970.597E / 16% +0.025+0.025 N2C2’08 DP-Gen 0.4950.495n / 2% 0.552¯ 0.552pk / 4% 0.5650.565L / 8% +0.013+0.013 N2C2’08 EPSVec 0.5460.546nR / 11% 0.573¯ 0.573pg / 3% 0.6610.661L / 10% +0.088+0.088 EUR-Lex DP-Gen 0.568¯ 0.568s / 5% 0.5580.558pk / 14% 0.5750.575E / 9% +0.007+0.007 EUR-Lex EPSVec 0.615¯ 0.615s / 11% 0.5690.569g / 8% 0.6180.618E / 10% +0.002+0.002 7.8 Per-Cell Vulnerability Distribution Table 19 reports, for every audited (generator, dataset, ε ) cell, the two statistics behind §5.4: the share of qualifying records with V⁡(x)>0V(x)>0 and the top-decile leakage share S10S_10 (Eq. 3), computed with the cell’s best proxy under the same worst-case selection as the main tables. The medians, ranges, and cell counts quoted in §5.4 are computed over this table. Reading across regimes, the two statistics move in opposite directions: the four non-private fine-tuning cells are the only ones that combine a large positive share with a small S10S_10, while every low-memorization cell pairs a thinner positive share with a heavier top decile—the per-cell form of the pattern §5.4 reports in aggregate. Table 19: Shape of the per-record vulnerability distribution in every audited cell. Each entry is the share of qualifying records with V⁡(x)>0V(x)>0 / the top-decile leakage share S10S_10 (Eq. 3); n is the number of qualifying records; even leakage would give S10≈10S_10≈ 10. V is computed per cell under that cell’s best proxy (the main table’s worst-case selection). For Aug-PE the columns are its noise levels mapped to the matching budgets, as in Table 11. Shaded: S10S_10 above the same row’s ε=∞ =∞ value, i.e. leakage that DP noise made more concentrated; deeper shading marks a larger excess (up to +40+40 points). Generator Dataset n ε=∞ =∞ 44 22 11 0.50.5 DP-Gen PsyTAR 35 77 / 31 46 / 58 54 / 57 63 / 34 63 / 38 DMSAFN 31 84 / 29 29 / 69 52 / 49 39 / 59 71 / 28 N2C2’08 19 79 / 37 79 / 38 68 / 39 68 / 58 68 / 38 EUR-Lex 32 94 / 20 59 / 34 69 / 31 56 / 42 59 / 42 EPSVec PsyTAR 35 63 / 82 49 / 69 54 / 47 69 / 41 51 / 51 DMSAFN 31 55 / 36 52 / 41 39 / 54 52 / 38 55 / 33 N2C2’08 19 42 / 55 47 / 59 53 / 32 74 / 32 74 / 41 EUR-Lex 32 44 / 39 56 / 35 53 / 36 81 / 49 56 / 37 Aug-PE PsyTAR 35 60 / 34 54 / 42 63 / 37 66 / 35 51 / 36 DMSAFN 31 58 / 49 45 / 57 52 / 72 68 / 33 68 / 37 N2C2’08 19 74 / 25 58 / 42 58 / 46 79 / 24 42 / 54 EUR-Lex 32 66 / 28 50 / 59 53 / 44 62 / 40 62 / 41 7.9 Attribute Analysis: Computation and Robustness Figure 5: Full attribute–vulnerability matrix: the outlier and rare-label pools alongside the random pools of Figure 4. Cell format and significance marks as there; within each block, all is the Fisher-z pool with Holm correction over the block’s five attributes. None of the 70 per-pool tests in the selected pools reaches nominal significance. Outlierness is range-restricted in the outlier pools, which are selected on that statistic. Figure 5 extends Figure 4 to the outlier and rare-label pools: no attribute reaches nominal significance in any of them, so the null result of the random pools is not an artifact of how records were selected. Computing the correlations. Every entry of Figure 4 and Table 20 is a Spearman rank correlation computed inside one pool. For each pool we take its qualifying records—those scored at least once as the inserted member and at least once as a non-member candidate within the 100-instance universe (16–36 per pool)—and their vulnerability V⁡(x)V(x) (Eq. 1), pooled over the pool’s privacy budgets: scores are the selected proxy, sign-aligned, z-normalized within each release set, and averaged over all member and all non-member episodes before the difference is taken. The five attributes are computed from the record and the fixed pools only, never from a release or an attack proxy: token length (spaCy tokens); outlierness, the Euclidean distance from the record’s embedding to the corpus centroid under the dataset’s encoder, i.e. the selection statistic of §4.2; near-duplication, the maximum cosine similarity to any other record of that pool’s private candidate pool C; rare-n-gram density, the fraction of the record’s token bigrams whose document frequency over the background B=C∪RB=C∪ R falls at or below the occurrence-weighted 10th percentile; and entity/numeral density, the number of numeral, identifier, all-caps, e-mail, and URL spans (overlaps merged) per token. The background is resolved per pool—each pool induces its own C and R (§4.2)—with the record itself excluded. Per-pool entries report ρ between the attribute and the within-pool rank of V, with Holm correction over all attribute–pool tests; the combined entry pools the twelve random pools of a generator (three independent draws per dataset, pairwise disjoint in records) by Fisher’s z, weighting each pool by n−3n-3, with Holm correction over the five attributes. Outlierness is marked † in the outlier columns because those pools were selected on it (§5.5). The per-pool detection floor is the significance boundary implied by the pool size: for a two-sided test at α=0.05α=0.05 with n records, |ρ|min=t0.975,n−2/t0.975,n−22+n−2|ρ|_ =t_0.975,n-2/ t_0.975,n-2^2+n-2, i.e. 0.330.33 at n=36n=36 and 0.460.46 at n=19n=19. A null in a single pool therefore means “no correlation as large as ∼ 0.4,” not “no correlation”; detecting effects of the size the random-pool combination suggests (|ρ|≈0.15|ρ|≈ 0.15) requires pooling, which is why inference is performed on the blocks’ combined columns. Table 20: Spearman correlation between five release-independent record attributes and per-record vulnerability rank, per generator, pooled over budgets. Left: the outlier pools; middle: the rare-label pools; right: Fisher-z combination over the 12 disjoint random pools (337 records). No cell is significant after Holm correction; per-pool detection floors are |ρ|≥0.33|ρ|≥ 0.33–0.460.46 (raw). † : range-restricted in the outlier pools. ∗ : the only directionally consistent effect (9/12 pools positive, Holm p=0.09p=0.09). Outlier pools Rare-label pools Random pools Attribute PsyTAR DMSAFN N2C2’08 EUR-Lex PsyTAR N2C2’08 EUR-Lex combined ρ DP-Gen length +.29+.29 +.00+.00 +.08+.08 −.24-.24 −.16-.16 +.17+.17 +.25+.25 +.05+.05 outlierness† −.18-.18 +.04+.04 +.00+.00 +.23+.23 +.16+.16 −.00-.00 +.25+.25 +.12+.12∗ near-duplication +.11+.11 −.06-.06 +.06+.06 −.10-.10 −.26-.26 +.11+.11 −.30-.30 −.17-.17 rare n-gram density −.04-.04 +.01+.01 +.10+.10 +.06+.06 +.17+.17 +.17+.17 +.03+.03 +.01+.01 entity/numeral density +.18+.18 −.22-.22 +.05+.05 −.10-.10 −.12-.12 +.20+.20 −.04-.04 −.15-.15 EPSVec length −.02-.02 +.18+.18 +.06+.06 +.09+.09 +.13+.13 +.16+.16 −.02-.02 +.04+.04 outlierness† −.16-.16 −.20-.20 +.10+.10 +.18+.18 +.00+.00 −.02-.02 −.15-.15 +.01+.01 near-duplication +.05+.05 +.29+.29 +.05+.05 +.13+.13 +.09+.09 +.04+.04 +.01+.01 +.12+.12 rare n-gram density −.06-.06 −.12-.12 +.17+.17 −.13-.13 −.29-.29 +.06+.06 −.20-.20 −.01-.01 entity/numeral density −.23-.23 +.17+.17 +.03+.03 −.00-.00 +.19+.19 −.02-.02 −.30-.30 −.05-.05 Aug-PE length −.30-.30 −.17-.17 +.25+.25 +.15+.15 – – – – outlierness† −.03-.03 −.24-.24 +.32+.32 +.05+.05 – – – – near-duplication +.06+.06 +.03+.03 +.39+.39 +.00+.00 – – – – rare n-gram density +.26+.26 −.14-.14 −.08-.08 +.11+.11 – – – – entity/numeral density −.08-.08 +.09+.09 +.01+.01 −.12-.12 – – – – Selection rules. Both rules search the same space—every proxy of §4.5 crossed with every scenario and budget available to the pool—and return one (scenario, proxy) pair applied to all of the pool’s budgets. The conservative rule takes the pair with the best AUC on an 80% training partition of instances. This is a pre-commitment, not independence: V⁡(x)V(x) still spans all instances, so selection has seen 80% of the episodes behind the tested correlations, but the proxy is fixed by a deterministic criterion that never sees the remaining fifth and is never revised afterwards. The worst-case rule is the main tables’: argmax AUC over all episodes. It is the audit’s natural instrument and the more sensitive of the two, but its argmax ranges over the very episodes being tested, so its p-values are optimistic—which is why the conservative rule is primary and Table 21 treats the worst-case rule as a sensitivity analysis. 7.10 Robustness to the proxy-selection rule Because the selected proxy determines V⁡(x)V(x), we recompute the full attribute–pool matrix for every generator and every pool under three alternative rules: the main table’s worst-case rule (argmax AUC over all episodes), and two selection-free rules that fix the same proxy in every pool—the lexical containment_max (Scenario 2) and the embedding cos_max (Scenario 1). Tables 21 and 22 summarize the outcome. Two conclusions are invariant. First, no attribute–pool test is significant after Holm correction under any rule (00 of 9595, 9595, and 2020 tests for the three generators), so the cell-level null of §5.5 does not depend on how the proxy was chosen. Second, the direction of every DP-SGD random-pool tendency is stable: outlierness is positive, and near-duplication and entity density negative, under all four rules; under activation steering every combined ρ stays within ±0.12± 0.12 under every rule. What depends on the rule is significance. The paper’s rule is the most conservative: outlierness is marginal under it (Holm p=0.09p=0.09) but clears correction under all-episode selection (+0.20+0.20, p=0.002p=0.002) and under the fixed embedding proxy (+0.32+0.32)—the latter being the natural consequence of selecting and scoring in the same embedding geometry, not evidence of a stronger effect. Individual pools are noisy: relative to the paper’s rule the median change in a per-pool ρ is at most 0.120.12 but the maximum reaches 0.960.96 in the smallest pools (n=16n=16–1919), which is why we interpret pooled directions only. Even under the most favourable rule the strongest attribute reaches |ρ|=0.32|ρ|=0.32, far below what record-level triage would require. Table 21: Table 20 under four proxy-selection rules, per generator, over every pool the table covers (high-risk, rare-label, and random). Columns: number of attribute–pool tests significant after Holm correction (of N); the Fisher-z combined ρ over that generator’s random pools per attribute; and the largest change in any per-pool ρ relative to the paper’s rule. Selection rule sig./N/N length outlierness near-dup. rare n-gram entity/num. max |Δρ|| ρ| DP-Gen 80% split (paper) 0/90 +0.05+0.05 +0.12+0.12 −0.17-0.17 +0.01+0.01 −0.15-0.15 – all episodes 0/90 +0.01+0.01 +0.20+0.20 −0.20-0.20 +0.04+0.04 −0.15-0.15 0.40 fixed containment_max 0/90 −0.05-0.05 +0.18+0.18 −0.19-0.19 +0.02+0.02 −0.13-0.13 0.70 fixed cos_max 0/90 −0.17-0.17 +0.32+0.32 −0.24-0.24 +0.13+0.13 −0.11-0.11 0.96 EPSVec 80% split (paper) 0/95 +0.04+0.04 +0.01+0.01 +0.12+0.12 −0.01-0.01 −0.05-0.05 – all episodes 0/95 +0.10+0.10 +0.02+0.02 −0.03-0.03 +0.01+0.01 +0.01+0.01 0.64 fixed containment_max 0/95 +0.08+0.08 −0.08-0.08 +0.06+0.06 +0.03+0.03 −0.04-0.04 0.64 fixed cos_max 0/95 +0.04+0.04 +0.01+0.01 +0.00+0.00 +0.08+0.08 −0.04-0.04 0.65 Table 22: Random-pool combined correlation between each attribute and per-record vulnerability under four proxy-selection rules (Fisher-z over 12 pools for DP-Gen and 12 pools for EPSVec). Each entry is ρ with the Holm-corrected p over the five attributes in parentheses; bold marks p<0.05p<0.05. The paper’s rule is the most conservative: the same tendencies that are marginal under it reach significance under all-episode (worst-case) selection and under fixed proxies. Attribute 80% split (paper) all episodes fixed containment_max fixed cos_max DP-Gen length +0.05+0.05 (0.725) +0.01+0.01 (0.895) −0.05-0.05 (0.707) −0.17-0.17 (0.009) outlierness +0.12+0.12 (0.088) +0.20+0.20 (0.002) +0.18+0.18 (0.006) +0.32+0.32 (<<0.001) near-duplication −0.17-0.17 (0.013) −0.20-0.20 (0.002) −0.19-0.19 (0.004) −0.24-0.24 (<<0.001) rare n-gram density +0.01+0.01 (0.826) +0.04+0.04 (0.895) +0.02+0.02 (0.761) +0.13+0.13 (0.048) entity/numeral density −0.15-0.15 (0.033) −0.15-0.15 (0.023) −0.13-0.13 (0.067) −0.11-0.11 (0.048) EPSVec length +0.04+0.04 (1.000) +0.10+0.10 (0.372) +0.08+0.08 (0.742) +0.04+0.04 (1.000) outlierness +0.01+0.01 (1.000) +0.02+0.02 (1.000) −0.08-0.08 (0.725) +0.01+0.01 (1.000) near-duplication +0.12+0.12 (0.172) −0.03-0.03 (1.000) +0.06+0.06 (0.824) +0.00+0.00 (1.000) rare n-gram density −0.01-0.01 (1.000) +0.01+0.01 (1.000) +0.03+0.03 (0.954) +0.08+0.08 (0.782) entity/numeral density −0.05-0.05 (1.000) +0.01+0.01 (1.000) −0.04-0.04 (0.954) −0.04-0.04 (1.000) 7.11 Cross-mechanism consistency of per-record vulnerability On identical records scored with the identical proxy (the DP-Gen pool’s selected proxy), the rank correlation of V⁡(x)V(x) between DP-SGD fine-tuning and each other mechanism is near zero (Table 23): mean −0.09-0.09 against activation steering over 16 pools (−0.12-0.12 over the random pools alone) and +0.07+0.07 against API-based prompting over the three outlier pools that have both. The per-record channel also differs in magnitude: without DP, fine-tuning yields mean |V|=1.99|V|=1.99 on PsyTAR, whereas prompting yields 0.040.04–0.530.53 at every noise level and never exhibits the no-DP spike. Both correlations are attenuated by the low reliability of V under the weaker mechanisms (within a mechanism, V at different budgets correlates at |ρ|≤0.35|ρ|≤ 0.35) and are therefore upper bounds on the transferable component rather than point estimates. Table 23: Cross-mechanism consistency of per-record vulnerability. Each entry is the Spearman correlation between V⁡(x)V(x) under DP-Gen and V⁡(x)V(x) under the named generator, on the same records scored with the same proxy (the DP-Gen pool’s selected proxy); parentheses give the number of records qualifying under both. Aug-PE has no random-pool releases. Pool n vs. EPSVec vs. Aug-PE PsyTAR outlier 35 +0.01+0.01 (35) +0.34+0.34 (35) DMSAFN outlier 31 −0.05-0.05 (31) −0.05-0.05 (31) N2C2’08 outlier 19 +0.28+0.28 (19) −0.08-0.08 (19) EUR-Lex outlier 32 −0.13-0.13 (32) −0.14-0.14 (32) PsyTAR rand0 32 +0.08+0.08 (32) – PsyTAR rand1 30 +0.07+0.07 (30) – PsyTAR rand2 36 −0.06-0.06 (36) – DMSAFN rand0 34 −0.34-0.34 (34) – DMSAFN rand1 31 −0.16-0.16 (31) – DMSAFN rand2 34 +0.04+0.04 (34) – N2C2’08 rand0 18 −0.03-0.03 (18) – N2C2’08 rand1 17 −0.41-0.41 (17) – N2C2’08 rand2 16 −0.55-0.55 (16) – EUR-Lex rand0 32 +0.05+0.05 (32) – EUR-Lex rand1 26 −0.13-0.13 (26) – EUR-Lex rand2 31 −0.02-0.02 (31) – mean −0.09-0.09 +0.02+0.02

Instant research discovery