Paper deep dive
Stealing Part of a Production Language Model
Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, Florian Tramèr
Models: GPT-3.5-turbo, GPT-3.5-turbo-instruct, OpenAI Ada, OpenAI Babbage, Pythia
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/12/2026, 8:05:21 PM
Summary
The paper introduces a novel model-stealing attack that extracts the embedding projection layer of black-box transformer language models. By exploiting the low-rank nature of the final projection layer, the authors demonstrate that they can recover the hidden dimension size and the full projection matrix of production models like OpenAI's Ada, Babbage, and GPT-3.5-turbo, as well as Google's PaLM-2, using API queries.
Entities (6)
Relation Signals (3)
Model Stealing → recovers → Embedding Projection Layer
confidence 95% · Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model
Model Stealing → targets → Black-box Language Models
confidence 95% · We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models
OpenAI → developed → GPT-3.5-turbo
confidence 90% · We also recover the exact hidden dimension size of the gpt-3.5-turbo model
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \$20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under $2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
96,639 characters extracted from source content.
Expand or collapse full text
Stealing Part of a Production Language Model Nicholas Carlini 1 Daniel Paleka 2 Krishnamurthy (Dj) Dvijotham 1 Thomas Steinke 1 Jonathan Hayase 3 A. Feder Cooper 1 Katherine Lee 1 Matthew Jagielski 1 Milad Nasr 1 Arthur Conmy 1 Itay Yona 1 Eric Wallace 4 David Rolnick 5 Florian Tramèr 2 Abstract We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like Open- AI’s ChatGPT or Google’s PaLM-2. Specifically, our attack recovers theembedding projection layer(up to symmetries) of a transformer model, given typical API access. For under $20 USD, our attack extracts the entire projection matrix of OpenAI’sadaandbabbagelanguage mod- els. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of thegpt-3.5- turbomodel, and estimate it would cost under $2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possi- ble future work that could extend our attack. 1. Introduction Little is publicly known about the inner workings of today’s most popular large language models, such as GPT-4, Claude 2, or Gemini. The GPT-4 technical report states it “contains no [...] details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar” (OpenAI et al., 2023). Similarly, the PaLM-2 paper states that “details of [the] model size and architecture are withheld from external publication” (Anil et al., 2023). This secrecy is often ascribed to “the competi- tive landscape” (because these models are expensive to train) and the “safety implications of large-scale models” (OpenAI et al., 2023) (because it is easier to attack models when more information is available). Nevertheless, while these models’ weights and internal details are not publicly accessible, the models themselves are exposed via APIs. 1 Google DeepMind 2 ETH Zurich 3 University of Washington 4 OpenAI 5 McGill University. Proceedings of the41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024. Copyright 2024 by the author(s). In this paper we ask:how much information can an adver- sary learn about a production language model by making queries to its API?This is the question studied by the field ofmodel stealing(Tramèr et al., 2016): the ability of an ad- versary to extract model weights by making queries its API. Contributions.We introduce an attack that can be applied to black-box language models, and allows us to recover the completeembedding projection layerof a transformer language model. Our attack departs from prior approaches that reconstruct a model in abottom-upfashion, starting from the input layer. Instead, our attack operatestop-down and directly extracts the model’s last layer. Specifically, we exploit the fact that the final layer of a language model projects from the hidden dimension to a (higher dimen- sional) logit vector. This final layer is thus low-rank, and by making targeted queries to a model’s API, we can extract its embedding dimension or its final weight matrix. Stealing this layer is useful for several reasons. First, it reveals thewidthof the transformer model, which is often correlated with its total parameter count. Second, it slightly reduces the degree to which the model is a complete “black- box”, which so might be useful for future attacks. Third, while our attack recovers only a (relatively small) part of the entire model, the fact that it is at all possible to stealany parameters of a production model is surprising, and raises concerns that extensions of this attack might be able to recover more information. Finally, recovering the model’s last layer (and thus hidden dimension) may reveal more global information about the model, such as relative size differences between different models. Our attack is effective and efficient, and is applicable to production models whose APIs expose full logprobs, or a “logit bias”. This included Google’s PaLM-2 and OpenAI’s GPT-4 (Anil et al., 2023; OpenAI et al., 2023); after respon- sible disclosure, both APIs have implemented defenses to prevent our attack or make it more expensive. We extract the embedding layer of several OpenAI models with a mean squared error of10 −4 (up to unavoidable symmetries). We apply a limited form of our attack togpt-3.5at a cost of un- der $200 USD and, instead of recovering the full embedding layer, recover just the size of the embedding dimension. 1 arXiv:2403.06634v2 [cs.CR] 9 Jul 2024 Stealing Part of a Production Language Model Responsible disclosure.We shared our attack with all ser- vices we are aware of that are vulnerable to this attack. We also shared our attack with several other popular services, even if they were not vulnerable to our specific attack, be- cause variants of our attack may be possible in other settings. We received approval from OpenAI prior to extracting the parameters of the last layers of their models, worked with OpenAI to confirm our approach’s efficacy, and then deleted all data associated with the attack. In response to our attack, OpenAI and Google have both modified their APIs to intro- duce mitigiations and defenses (like those that we suggest in Section 8) to make it more difficult for adversaries to perform this attack. 2. Related Work Model stealing attacks (Tramèr et al., 2016) aim to recover the functionality of a black-box model, and optimize for one of two objectives (Jagielski et al., 2020): 1.Accuracy: the stolen model ˆ f should match the perfor- mance of the target modelfon some particular data domain. For example, if the target is an image clas- sifier, we might want the stolen model to match the target’s overall accuracy on ImageNet. 2.Fidelity: the stolen model ˆ fshould be functionally equivalent to the target modelfon all inputs. That is, for any valid inputp, we want ˆ f(p)≈f(p). In this paper, we focus on high-fidelity attacks. Most prior high-fidelity attacks exploit specific properties of deep neu- ral networks with ReLU activations. Milli et al. (2019) first showed that if an attacker can computegradientsof a target two-layer ReLU model, then they can steal a nearly bit- for-bit equivalent model. Jagielski et al. (2020) observed that if the attacker only has query access to model outputs, they can approximate gradients with finite differences. Sub- sequent work extended these attacks to efficiently extract deeper ReLU models (Carlini et al., 2020; Rolnick & Kord- ing, 2020; Shamir et al., 2023). Unfortunately, none of these approaches scale to production language models, because they (1) accept tokens as inputs (and so performing finite differences is intractable); (2) use activations other than Re- LUs; (3) contain architectural components such as attention, layer normalization, residual connections, etc. that current attacks cannot handle; (4) are orders-of-magnitude larger than prior extracted models; and (5) expose only limited- precision outputs. Other attacks aim to recover more limited information, or assume a stronger adversary. Wei et al. (2020) show that an adversary co-located on the same server as the LLM can recover the sizes of all hidden layers. Zanella-Beguelin et al. (2021) assume a model with a public pretrained encoder and a private final layer, and extract the final layer; our Section 4.2 is quite similar to their method. Others have attempted to recover model sizes by correlating performance on published benchmarks with model sizes in academic papers (Gao, 2021). 3. Problem Formulation We study models that take a sequence of tokens drawn from a vocabularyXas input. LetP ( X ) denote the space of probability distributions overX. We study parameterized modelsf θ :X N →P ( X ) that produce a probability distri- bution over the next output token, given an input sequence ofNtokens. The model has the following structure: f θ (p) =softmax(W·g θ (p)),(1) whereg θ :X N →R h is another parameterized model that computes hidden states,Wis anl×hdimensional matrix (theembedding projection matrix), andsoftmax:R l → [0, 1] l is the softmax function applied to the resultinglogits: softmax(z) = " e z 1 P l i=1 e z i ,..., e z l P l i=1 e z i # . Note that the hidden dimension size is much smaller than the size of the token dictionary, i.e.,h≪l. For example, LLaMA (Touvron et al., 2023) choosesh∈ 4096, 5120, 6656, 8192andl=32,000, and there is a recent trend towards increasingly large token sizes; GPT-4, for example, has a≈100,000 token vocabulary. Threat model.Throughout the paper, we assume that the adversary does not have any additional knowledge about the model parameters. We assume access to a modelf θ , hosted by a service provider and made available to users through a query interface (API)O. We assume thatOis a perfect oracle: given an input sequencep, it producesy=O ( p ) without leaking any other information aboutf θ than what can be inferred from(p,y). For example, the adversary cannot infer anything aboutf θ via timing side-channels or other details of the implementation of the query interface. Different open-source and proprietary LLMs offer APIs with varying capabilities, which impact the ability to per- form model extraction attacks and the choice of attack algo- rithm. A summary of the different APIs we study, and our motivation for doing so, is presented in Table 1. The logits API is a strawman threat model where the API provides log- its for all tokens in the response to a given prompt. We begin with this toy setting, as the attack techniques we develop here can be reused in subsequent sections, where we will first reconstruct the logits from more limited information (e.g., log-probabilities for only the top few tokens) and then run the attack. 2 Stealing Part of a Production Language Model Table 1.Summary of APIs APIMotivation All Logits §4Pedagogy & basis for next attacks Top Logprobs, Logit-bias §5 Current LLM APIs (e.g., OpenAI) No logprobs, Logit-bias §FPotential future constrained APIs 4. Extraction Attack for Logit-Vector APIs In this section, we assume the adversary can directly view the logits that feed into the softmax function for every token in the vocabulary (we will later relax this assumption), i.e., O ( p ) ←W·g θ ( p ) . We develop new attack techniques that allow us to perform high-fidelity extraction of (a small part of) a transformer. Section 4.1 demonstrates how we can identify the hidden dimensionhusing the logits API and Section 4.2 presents an algorithm that can recover the matrixW. 4.1. Warm-up: Recovering Hidden Dimensionality We begin with a simple attack that allows an adversary to re- cover the size of the hidden dimension of a language model by making queries to the oracleO(Algorithm 1). The tech- niques we use to perform this attack will be the foundation for attacks that we further develop to perform complete extraction of the final embedding projection matrix. Algorithm 1Hidden-Dimension Extraction Attack Require:Oracle LLMOreturninglogits 1:Initializento an appropriate value greater thanh 2:Initialize an empty matrixQ=0 n×l 3:fori=1tondo 4:p i ←RandPrefix()▷Choose a random prompt 5:Q i ←O(p i ) 6:end for 7:λ 1 ≥λ 2 ≥·≥λ n ←SingularValues(Q) 8: count←arg max i log∥λ i ∥−log∥λ i+1 ∥ 9:returncount Intuition.Suppose we query a language model on a large number of different random prefixes. Even though each output logit vector is anl-dimensional vector, they all actu- ally lie in ah-dimensional subspace because the embedding projection layer up-projects fromh-dimensions. Therefore, by querying the model “enough” (more thanhtimes) we will eventually observe new queries are linearly dependent of past queries. We can then compute the dimensionality of this subspace (e.g., with SVD) and report this as the hidden dimensionality of the model. Formalization.The attack is based on the following straightforward mathematical result: Lemma 4.1.LetQ ( p 1 ,...p n ) ∈R l×n denote the matrix with columnsO ( p 1 ) ,...,O ( p n ) of query responses from the logit-vector API. Then h≥rank ( Q ( p 1 ,...p n )) . Further, if the matrix with columnsg θ ( p i ) (i=1, ...,n) has rankhandWhas rankh, then h=rank ( Q ( p 1 ,...p n )) . Proof. We haveQ=W·H, whereHis ah×nma- trix whose columns areg θ (p i )(i=1,...,n). Thus, h≥rank ( Q ) . Further, ifHhas rankh(with the second assumption), thenh=rank ( Q ) . Assumptions.In Lemma 4.1, we assume that both the matrix with columnsg θ ( p i ) and the matrixWhave rank h. These matrices have eitherhrows orhcolumns, so both have rank at mosth. Moreover, it is very unlikely that they have rank< h: this would require the distribution ofg θ ( p ) to be fully supported on a subspace of dimension< hacross allp i we query, or allh≪lcolumns ofWto lie in the same(h−1)dimensional subspace ofR l (the output space of logits). In practice we find this assumption holds for all larger models (Table 2) and when different normalization layers are used (Appendix B.1). Practical considerations.Since the matrixQis not com- puted over the reals, but over floating-point numbers (possi- bly with precision as low as 16-bits or 8-bits for production neural networks), we cannot naively take the rank to be the number of linearly independent rows. Instead, we use a practicalnumerical rankofQ, where we order the singular valuesλ 1 ≥λ 2 ≥·≥λ n , and identify the largestmulti- plicativegap λ i λ i+1 between consecutive singular values. A large multiplicative gap arises when we switch from large “actual” singular values to small singular values that arise from numerical imprecision. Figure 2 shows these gaps. Algorithm 1 describes this attack. Experiments.In order to visualize the intuition behind this attack, Figure 1 illustrates an attack against the Pythia- 1.4b LLM. Here, we plot the magnitude of the singular values ofQas we send an increasing numbernof queries to the model. When we send fewer than 2048 queries it is impossible to identify the dimensionality of the hidden space. This is becausen < h, and so then×ldimensional matrixQhas full rank andnnontrivial singular values. But once we make more than2048queries to the model, and thusn > h, the number of numerically significant singular values does not increase further; it is capped at exactly2048. In Figure 2 we plot the difference (in log-space) between subsequent singular values. As we can see, the largest 3 Stealing Part of a Production Language Model 050010001500200025003000 Sorted Singular Values 10 4 10 3 10 2 10 1 10 0 10 1 10 2 10 3 Magnitude 1024 queries 1536 queries 2048 queries 2560 queries 3072 queries Figure 1.SVD can recover the hidden dimensionality of a model when the final output layer dimension is greater than the hidden dimension. Here we extract the hidden dimension (2048) of the Pythia 1.4B model. We can precisely identify the size by obtaining slightly over 2048 full logit vectors. 20002020204020602080 Sorted Singular Values 10 4 10 3 10 2 10 1 10 0 10 1 Difference between consecuitive singular values Figure 2.Our extraction attack recovers the hidden dimension by identifying a sharp drop in singular values, visualized as a spike in the difference between consecutive singular values. On Pythia- 1.4B, a 2048 dimensional model, the spike occurs at 2047 values. difference occurs at (almost exactly) the 2048th singular value—the true hidden dimensionality of this model. We now analyze the efficacy of this attack across a wider range of models: GPT-2 (Radford et al., 2019) Small and XL, Pythia (Biderman et al., 2023) 1.4B and 6.9B, and LLaMA (Touvron et al., 2023) 7B and 65B. The results are in Table 2: our attack recovers the embedding size nearly perfectly, with an error of 0 or 1 in five out of six cases. Our near perfect extraction has one exception: GPT-2 Small. On this 768 dimensional model, our attack reports a hidden dimension of 757. In Appendix A we show that this “failure” is caused by GPT-2 actually having an effective hidden dimensionality of757despite having768dimensions. Cheaper Dimension ExtractionNote thatlbeing exactly equal to the vocabulary size is not crucial. Formally, taking onlyl ′ < lrows ofQdoes not change the number of nonzero singular values, except in the unlikely case that the resulting submatrix is of smaller rank. Hence, we can choose a subset ofl ′ tokens and extract the dimension from logits on these tokens alone, as long asl ′ > h. 4.2. Full Layer Extraction (Up to Symmetries) We extend the attack from the prior section to recover the final output projection matrixWthat maps from the final hidden layer to the output logits. Method:LetQbe as defined in Algorithm 1. Now rewrite Q=U·Σ·V ⊤ with SVD. Previously we saw that the number of large enough singular values corresponded to the dimension of the model. But it turns out that the matrixU actually directly represents (a linear transformation of) the final layer! Specifically, we can show thatU·Σ=W·G for someh×hmatrixGin the following lemma. Lemma 4.2.In the logit-API threat model, under the as- sumptions of Lemma 4.1:(i)The method above recovers ̃ W=W·Gfor someG∈R h×h ;(i)With the addi- tional assumption thatg θ (p)is a transformer with residual connections, it is impossible to extractWexactly. Proof.See Appendix C.□ Note that we could also useQ=W·Gforn=l. The SVD construction above gains numerical precision ifn > l. Experiments.For the six models considered previously, we evaluate the attack success rate by comparing the root mean square (RMS) between our extracted matrix ̃ W= U·Σand the actual weight matrix, after allowing for a h×haffine transformation. Concretely, we solve the least squares system ̃ W·G≈WforG, which reduces toh linear least squares problems, each withlequations andh unknowns. Then, we report the RMS ofWand ̃ W·G. The results are in Table 2. As a point of reference, the RMS between a randomly initialized model and the actual weights is2·10 −2 , over100–500×higher than the error of our reconstruction. In Appendices C and H, we show that reconstruction is pos- sible up anorthogonaltransformation (approximatelyh 2 /2 missing parameters, as opposed toh 2 for reconstruction up to an affine transformation), and that this is tight under some formal assumptions. However, we only have an efficient algorithm for reconstruction up to affine transformations. 4 Stealing Part of a Production Language Model Table 2.Our attack succeeds across a range of open-source models, at both stealing the model size, and also at reconstructing the output projection matrix (up to invariances; we show the root MSE). ModelHidden DimStolen SizeWRMS GPT-2 Small (fp32)768757±14·10 −4 GPT-2 XL (fp32)16001599±16·10 −4 Pythia-1.4 (fp16)20482047±13·10 −5 Pythia-6.9 (fp16)40964096±14·10 −5 LLaMA 7B (fp16)40964096±28·10 −5 LLaMA 65B (fp16)81928192±25·10 −5 5. Extraction Attack for Logit-Bias APIs The above attack makes a significant assumption: that the adversary can directly observe the complete logit vector for each input. In practice, this is not true: no production model we are aware of provides such an API. Instead, for example, they provide a way for users to get the top-K(by logit) token log probabilities. In this section we address this challenge. 5.1. Description of the API In this section we develop attacks for APIs that return log probabilities for the topKtokens (sorted by logits), and where the user can specify a real-valued biasb∈R |X| (the “logit bias”) to be added to the logits for specified tokens before the softmax, i.e., O(p,b)←TopK ( logsoftmax ( Wg θ (p) +b )) =TopK Wg θ (p)+b−log X i exp ( Wg θ (p)+b ) i ! ·1 ! . whereTopK ( z ) returns theKhighest entries ofz∈R l and their indices. Many APIs (prior to this paper) provided such an option for their state-of-the-art models (OpenAI, 2024; Google, 2024). In particular, the OpenAI API supports mod- ifying logits for at most300tokens, and the logit bias for each token is restricted to the range[−100, 100](OpenAI, 2023). All that remains is to show that we can uncover the full logit vector for distinct prompt queries through this API. In this section, we develop techniques for this purpose. Once we have recovered multiple complete logit vector, we can run the attack from Section 4.2 without modification. 5.2. Evaluation Methodology Practical attacks must beefficient, both to keep the cost of extraction manageable and to bypass any rate limiters or other filters in the APIs. We thus begin with two cost definitions that we use to measure the efficacy of our attack. Token cost:the number of tokens the adversary sends to (or receives from) the model during the attack. Most APIs charge users per-token, so this metric represents the mone- tary cost of an attack (after scaling by the token cost). Query cost:the total duration of the attack. Most APIs place a limit on the number of queries an adversary can make in any given interval, and so some attacks may be faster but cost more (by sending more tokens per query). In the remainder of this section we develop several attacks under varying attack assumptions and optimizing for either token cost,query cost, or both. 5.3. Extraction Attack for Top-5 Logit Bias APIs We develop a technique to compute the logit vector for any prefixpvia a sequence of queries with varying logit biases. To begin,suppose that the API returned the topKlogits. Then we could recover the complete logit vector for an arbitrary promptpby cycling through different choices for the logit bias and measuring the top-klogits each time. In particular, for an API with top-5 logits we can send a sequence of queries O(p,b k =b k+1 =...=b k+4 =B), fork∈0,5,10,...,|X| with a large enoughB. Each query thus promotes five different tokensk,k+1,...,k+4into the top-5, which allows us to observe their logits. By subtracting the biasB and merging answers from all of these queries, we recover the entire logit vector. Unfortunately, we cannot use this attack directly because all production APIs we are aware of returnlogprobs(the log of the softmax output of the model) instead of the logitsz i . The problem now is that when we apply a logit biasBto thei-th token and observe that token’s logprob, we get the value y B i =z i +B−log X j̸=i exp(z j ) +exp(z i +B) wherez i are the original logits. We thus get an additional bias-dependent term which we need to deal with. We pro- pose two approaches. Our first approach relies on a common “reference” token that lets us learn the relative difference between all logits (this is the best we can hope for, since the softmax is invariant under additive shifts to the logits). Suppose the top token for a prompt isR, and we want to learn the relative difference between the logits of tokensiandR. We add a large bias Bto tokenito push it to the top-5, and then observe the logprobs of both tokeniandR. We have: y B R −y B i −B=z R −z i . Since we can observe 5 logprobs, we can compare the ref- erence tokenRto four tokens per query, by adding a large 5 Stealing Part of a Production Language Model bias that pushes all four tokens into the top 5 (along with the reference token). We thus issue a sequence of queries O(p,b i =b i+1 =b i+2 =b i+3 =B) fori∈0, 4, 8,·,|X|. This recovers the logits up to the free parameterz R that we set to0. Query cost.This attack reveals the value of K-1 logits with each query to the model (theK-th being used as a reference point), for a cost of1/(K−1)queries per logit. In Appendix E we present a second, more sophisticated method that allows us to recoverKlogits per query, i.e., a cost of1/K, by viewing each logprob we receive as a linear constraint on the original logits. Token cost.Recall that our attack requires that we learn the logits for several distinct prompts; and so each prompt must be at least one token long. Therefore, this attack costs at least two tokens per query (one input and one output), or a cost of1/2for each token of output. But, in practice, many models (likegpt-3.5-turbo) include a few tokens of overhead along with every single query. This increases the token cost per logit to 2+∆ 4 where∆is the number of overhead tokens; forgpt-3.5-turbowe report∆=7. An improved cost-optimal attack.It is possible to gen- eralize the above attack to improveboththe query cost and token cost. Instead of issuing queries to the model that reveal 4 or 5 logit values for a single generated token, we might instead hope to be able to send a multi-token query [p 0 p 1 p 2 ...p n ]and then ask for the logprob vector for each prefix of the prompt[p 0 ],[p 0 p 1 ],[p 0 p 1 p 2 ] etc. OpenAI’s API did allow for queries of this form in the past, by providing logprobs forprompttokens as well as generated tokens by combining thelogprobandecho parameters; this option has since been removed. Now, it is only possible to view logprobs ofgenerated tokens. And since only the very last token is generated, we can only view four logprobs for this single longer query. This, however, presents a potential approach to re- duce the query and token cost: if there were some way to cause the model to emit a specific sequence of tokens [p n+1 p n+2 ... p n+m ], then we could inspect the log- prob vector of each generated token. We achieve this as follows: we fix a tokenxand four other tokens, and force the model to emit[x x ... x]. In- stead of supplying a logit bias ofBfor each of the five tokens, we supply a logit bias ofBfor tokenx, andB ′ < B for the other four tokens. IfB ′ is large enough so that the other tokens will be brought into the top-5 outputs, we will still be able to learn the logits for those tokens. As long as B ′ is small enough so that the model will always complete the initial promptp 0 with tokenx(and not any other), then we will be able to collect the logits on several prompts of the form[p 0 x x ... x]. Analysis.It is easy to see that the query cost of this attack is 1 4m , wheremis the expansion factor. Further, since each query requires1+mtokens, the token cost is 1+m 4m . (Or, 1+m+∆if the API has an overhead of∆tokens.) Note that ifm=1, i.e., there is no expansion, this attack reduces to our first attack and the analysis similarly gives a query cost of 1 /4and a token cost of 1 /2. 5.4. Extraction Attack for top-1 Binary Logit Bias APIs In light of our attacks, it is conceivable that model providers introduce restrictions on the above API. We now demon- strate that an attack is possible even if the API only returns the top logprob (K=1in the API from Section 5.1), and the logit bias is constrained to only take one of two values. API.We place two following further restrictions on the logit bias API (Section 5.1): first, we setK=1, and only see the most likely token’s logprob; and second, each logit bias entrybis constrained to be in−1, 0. These constraints would completely prevent the attacks from the prior section. We believe this constraint is significantly tighter than any practical implementation would define. Method.At first it may seem impossible to be able to learn any information about a tokentif it is not already the most likely token. However, note that if we query the model twice, once without any logit bias, and once with a logit bias of−1 for tokent, then the top token will beslightlymore likely with a bias of−1, with exactly how slight depending on the valueof tokent’s logprob. Specifically, in Appendix D we show the logprob equals( 1 /e−1) −1 (exp(y top −y ′ top )−1) wherey top andy ′ top are the logprobs of the most likely token when querying with logit bias of0and−1. Analysis.This attack requires1query and token per logprob extracted. However, as we will show in the evaluation, this attack is much less numerically stable than the previously- discussed attacks, and so may require more queries to reach the same level of accuracy. 6. Logprob-free attacks Due to space constraints, in Appendix F, we show we can still extract logitswithout logprob access, although with a higher cost. Intuitively, even without logprobs (as long as we still have logit bias) it is possible to perform binary search to increase and decrease the logits for every token until increasing any token by epsilon will make it the most likely. At this point, the logit bias vector corresponds directly to the (relative) logits of each token relative to every other. 6 Stealing Part of a Production Language Model Table 3.Attack success rate on five different black-box models Dimension ExtractionWeight Matrix Extraction ModelSize# QueriesCost (USD)RMS# QueriesCost (USD) OpenAIada1024✓<2·10 6 $15·10 −4 <2·10 7 $4 OpenAIbabbage2048✓<4·10 6 $27·10 −4 <4·10 7 $12 OpenAIbabbage-0021536✓<4·10 6 $2 † <4·10 6†+ $12 OpenAIgpt-3.5-turbo-instruct ∗ ✓<4·10 7 $200 † <4·10 8†+ $2,000 †+ OpenAIgpt-3.5-turbo-1106 ∗ ✓<4·10 7 $800 † <4·10 8†+ $8,000 †+ ✓ Extracted attack size was exactly correct; confirmed in discussion with OpenAI. ∗ As part of our responsible disclosure, OpenAI has asked that we do not publish this number. † Attack not implemented to preserve security of the weights. + Estimated cost of attack given the size of the model and estimated scaling ratio. Table 4.Average error at recovering the logit vector for each of the logit-estimation attacks we develop. Our highest precision, and most efficient attack, recovers logits nearly perfectly; other attacks approach this level of precision but at a higher query cost. AttackLogprobs Bits of precision Queries per logit logprob-4 (§5.3)top-523.00.25 logprob-5 (§E)top-511.50.64 logprob-1 (§5.4)top-16.11.0 binary search (§F.1)✗7.210.0 hyperrectangle (§F.2)✗15.75.4 one-of-n (§F.3)✗18.03.7 By performing the binary search one token at a time, we can develop an effective (but inefficient) attack that requires Nlog( B ε )whereNis the number of logits,Bis an upper bound on the gap between any two logits andεis the desired tolerance. An improved attack is possible by noticing that it is possible to perform binary search on multiple tokens in parallel. Because the adversary gets to view the arg-max sampled token, by modifying multiple tokens at the same time we can learn information faster and therefore improve attack efficiency. 7. Evaluation We now study the efficacy of our practical stealing attack. 7.1. Logit Validation We begin by validating that the attacks developed in the prior sections can effectively recover the full logit vector given a limited query interface. In Table 4 we report the average number of bits of agreement between the true logit vector and the recovered logit vector, as well as the (amortized) number of queries required to recover one full logit vector. Generally, attacks that operate under stronger threat mod- els have higher precision. But theoretical improvements are not always practical: the theoretically stronger attack from §E that learns 5 logprobs per query in practice re- quires more queries and recovers logits with lower fidelity. This is because this attack is numerically unstable: it re- quires a potentially ill-conditioned matrix, and therefore can require re-querying the API after adjusting the logit bias. Our strongest logprob-free attack is highly efficient, and recovers18bits of precision at just3.7queries per logit. In Appendix G we theoretically analyze how far this is from optimal, and find it is within a factor of two. 7.2. Stealing Parts of Production Models We now investigate our ability to steal production language models, focusing on five of OpenAI’s models available on 1 January 2024:ada,babbage,babbage-002,gpt-3.5- turbo-instruct , andgpt-3.5-turbo-1106. We selected these models because these were the only production models which were able to receive advance permission to attempt an extraction attack; we are exceptionally grateful to OpenAI for allowing us to perform this research using their models. Given the results from the prior section, we chose to imple- ment the improved 4-logprob attack (Section 5.3) because it is both the most query efficient attack and also the most precise attack. Switching to a different attack algorithm would increase our total experiment cost significantly, and so we do not perform these ablation studies. Both our hidden-dimension-stealing and entire-layer- stealing attack worked for all five of these models. The size we recover from the model perfectly matches the ac- tual size of the original model, as confirmed by OpenAI. For the first three models, we report in Table 3 the size we recover because (1) the sizes of these models was never pre- viously confirmed, but (2) they have now been deprecated and so disclosing the size is not harmful. In discussions with OpenAI, we decided to withhold disclosure of the size of gpt-3.5-turbomodels, but we confirmed with them that the number our attack reported was accurate. 7 Stealing Part of a Production Language Model When running the full layer-stealing attack, we confirmed that our extracted weights are nearly identical to the ac- tual weights, with error<7·10 −4 , up to anh×hmatrix product as discussed previously. Table 3 reports the RMS between our extracted weight matrix and the actual model weights, after “aligning” the two by anh×htransform. 8. Defenses It would be possible to prevent or mitigate this attack in a number of different ways, albeit with loss of functionality. 8.1. Prevention Remove logit bias.Perhaps the simplest defense would be to outright remove the logit bias parameter from the API. Unfortunately, there are several legitimate use cases of this parameter. For example, several works use logit bias in order to perform controlled or constrained generation (Jiang et al., 2023; Yang & Klein, 2021), to shift generation and mimic fine-tuning the model (Liu et al., 2024; Mitchell et al., 2024), or other reasons (Ren et al., 2023; Lee et al., 2022). Replace logit bias with a block-list.Instead of offering a logit bias, model developers could replace it with a block- list of tokens the model is prohibited from emitting. This would support (some) of the functionality discussed in the prior section, but would still prevent our attack. Architectural changes.Instead of modifying the API, we could instead make changes to the model. Our attack only works because the hidden dimensionhis less than the output dimensionl. This suggests a natural architectural defense: split the final layer into two layers, one that goes fromh→tand thent→lwheret > land a nonlinearity was placed in between. This is not very efficient though, as the last linear layer is large (quadratic in the vocabulary size). Post-hoc altering the architecture.We can also modify the hidden dimensionhfor the final layer after the model is trained. In particular, we can expand the dimensionality ofWby concatenating extra weight vectors that are orthog- onal to the original matrix. We set the singular values for these weights to be small enough to not materially affect the model’s predictions, while also being large enough to look realistic. Then, during the model’s forward pass, we concatenate a vector of random Gaussian noise to the final hidden vectorg θ (p)before multiplying byW. Figure 7 shows an example of this, where we expand GPT-2 small to appear as if it was 1024 dimensional instead of 768 di- mensions. This misleads the adversary into thinking that the model is wider than it actually is. 8.2. Mitigations Logit bias XOR logprobs.Our attack is10×cheaper when an adversary can supply both a logit bias and also view output logprobs. This suggests a natural mitigation: prohibit queries to the API that make use ofbothlogit bias and logprobs at the same time. This type of defense is com- mon in both the security and machine learning community: for example, in 2023 OpenAI removed the ability to com- bine bothechoandlogprobs, but with either alone being allowed; this defense would behave similarly. Noise addition.By adding a sufficient amount of noise to the output logits of any given query, it would be possible to prevent our attack. However, logit-noise has the potential to make models less useful. We perform some preliminary experiments on this direction in Appendix I. Rate limits on logit bias.Our attack requires that we are able to learn at leasthlogit values for each promptp. One defense would be to allow logit-bias queries to the model, but only allowT= ̃ h/5logit bias queries for any given promptpto prevent an adversary from learning if a model has hidden dimension ̃ hor smaller. Unfortunately this has several significant drawbacks: the threshold has to be independent ofh(or learning the thresh- old would revealh); the system would need to maintain state of all user queries to the API; and preventing Sybil attacks requires a global pool of user queries, which can present significant privacy risks (Debenedetti et al., 2023). Detect malicious queries.Instead of preventing any queries that might leak model weights, an alternate strategy could be to implement standard anti-abuse tools todetect any patterns of malicious behavior. Several proposals of this form exist for prior machine learning attacks, including model stealing (Juuti et al., 2019; Pal et al., 2021) and ad- versarial examples (Chen et al., 2020). Unfortunately, these defenses are often vulnerable to attack (Feng et al., 2023), and so any mitigation here would need to improve on the state-of-the-art to be truly robust. 9. Future Work We are motivated to study this problem not because we ex- pect to be able to steal an entire production transformer model bit-for-bit, but because we hope to conclusively demonstrate that model stealing attacks are not just of aca- demic concern but can be practically applied to the largest production models deployed today. We see a number of potential directions for improving on this attack. Breaking symmetry with quantized weights.Large pro- duction models are typically stored “quantized”, where each weight is represented in just 4 or 8 bits. In principle, this quantization could allow an adversary to recover a nearly 8 Stealing Part of a Production Language Model bit-for-bit copy of the matrixW: while there exist an infi- nite number of matricesW·G, only one will be discretized properly. Unfortunately, this integer-constrained problem is NP-hard in general (similar problems are the foundation for an entire class of public key cryptosystems). But this need not imply that the problem is hard on all instances. Extending this attack beyond a single layer.Our attack recovers a single layer of a transformer. We see no obvious methodology to extend it beyond just a single layer, due to the non-linearity of the models. But we invite further research in this area. Removing the logit bias assumption.All our attacks re- quire the ability to pass a logit bias. Model providers includ- ing Google and OpenAI provided this capability when we began the writing of this paper, but this could change. (In- deed, it already has, as model providers begin implementing defenses to prevent this attack.) Other API parameters could give alternative avenues for learning logit information. For example, unconstrainedtemperatureandtop-kparame- ters could also leak logit values through a series of queries. In the long run, completely hiding the logit information might be challenging due both to public demand for the feature, and ability of adversaries to infer this information through other means. Exploiting the stolen weights.Recovering a model’s em- bedding projection layer might improve other attacks against that model. Alternatively, an attacker could infer details about a provider’sfinetuningAPI by observing changes (or the absence thereof) in the last layer. In this paper, we fo- cus primarily on the model extraction problem and leave exploring downstream attacks to future work. Practical stealing of other model information.Existing high-fidelity model stealing attacks are “all-or-nothing” at- tacks that recover entire models, but only apply to small ReLU networks. We show that stealing partial information can be much more practical, even for state-of-the-art models. Future work may find that practical attacks can steal many more bits of information about current proprietary models. 10. Conclusion As the field of machine learning matures, and models tran- sition from research artifacts to production tools used by millions, the field of adversarial machine learning must also adapt. While it is certainly useful to understand the potential applicability of model stealing to three-layer 100-neuron ReLU-only fully-connected networks, at some point it be- comes important to understand to what extent attacks can be actually applied to the largest production models. This paper takes one step in that direction. We give an existence proof that it is possible to steal one layer of a production language model. While there appear to be no immediate practical consequences of learning this layer, it represents the first time thatanyprecise information about a deployed transformer model has been stolen. Two imme- diate open questions are (1) how hazardous these practical stealing attacks are and (2) whether they pose a greater threat to developers and the security of their models than black- box access already does via distillation or other approximate stealing attacks. Our attack also highlights how small design decisions in- fluence the overall security of a system. Our attack works because of the seemingly-innocuouslogit-biasandlogprobs parameters made available by the largest machine learning service providers, including OpenAI and Google—although both have now implemented mitigations to prevent this at- tack or make it more expensive. Practitioners should strive to understand how system-level design decisions impact the safety and security of the full product. Overall, we hope our paper serves to further motivate the study of practical attacks on machine learning models, in order to ultimately develop safer and more reliable systems. Impact Statement This paper is the most recent in a line of work that demon- strates successful attacks on production models. As such, we take several steps to mitigate the near-term potential harms of this research. As discussed throughout the paper, we have worked closely with all affected products to ensure that mitigations are in place before disclosing this work. We have additionally sent advance copies of this paper to allpotentiallyaffected parties, even if we were unable to precisely verify our attack. Long-term, we believe that openly discussing vulnerabilities that have practical impact is an important strategy for ensur- ing safe machine learning. This vulnerability exists whether or not we report on it. Especially for attacks that are simple to identify (as evidenced by the concurrent work of Fin- layson et al. (2024) that discovered this same vulnerability), malicious actors are also likely to discover the same vulner- ability whether or not we report on it. By documenting it early, we can ensure future systems remain secure. Acknowledgements We are grateful to Andreas Terzis and the anonymous re- viewers for comments on early drafts of this paper. We are grateful to Joshua Achiam for helping to write the code for post-hoc modifying the model architecture. We are also grateful to OpenAI for allowing us to attempt our extraction attack on their production models. 9 Stealing Part of a Production Language Model References Anil, R. et al. PaLM 2 Technical Report, 2023. Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. Biderman, S. Common LLM settings, 2024. URLhttps: //rb.gy/2afqlw. Accessed February 1, 2024. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. InInternational Conference on Machine Learning, 2023. Cancedda, N. Spectral filters, dark signals, and attention sinks, 2024. Carlini, N., Jagielski, M., and Mironov, I. Cryptanalytic extraction of neural network models. InAnnual Interna- tional Cryptology Conference, 2020. Chen, S., Carlini, N., and Wagner, D. Stateful detection of black-box adversarial attacks. InProceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence, 2020. Chiu, J.openlogprobs, 2024.URLhttps:// github.com/justinchiu/openlogprobs. Ac- cessed February 1, 2024. Debenedetti, E., Severi, G., Carlini, N., Choquette-Choo, C. A., Jagielski, M., Nasr, M., Wallace, E., and Tramèr, F. Privacy side channels in machine learning systems.arXiv preprint arXiv:2309.05610, 2023. Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit Optimizers via Block-wise Quantization.ICLR, 2022. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield- Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits. 2021. URLhttps://transformer-circuits. pub/2021/framework/index.html. Feng, R., Hooda, A., Mangaokar, N., Fawaz, K., Jha, S., and Prakash, A. Stateful defenses for machine learning models are not yet secure against black-box attacks. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, p. 786–800, 2023. Finlayson, M., Swayamdipta, S., and Ren, X. Logits of api-protected llms leak proprietary information.arXiv preprint arXiv:2403.09539, 2024. Gao, L. On the sizes of OpenAI API models.https: //blog.eleuther.ai/gpt3-model-sizes/, 2021. Google.Changelog1.38.0,2024.URL https://cloud.google.com/python/docs/ reference/aiplatform/1.38.0/changelog. Accessed January 30, 2024. Gurnee, W., Horsley, T., Guo, Z. C., Kheirkhah, T. R., Sun, Q., Hathaway, W., Nanda, N., and Bertsimas, D. Univer- sal neurons in gpt2 language models, 2024. Hayase, J., Borevkovic, E., Carlini, N., Tramèr, F., and Nasr, M. Query-based adversarial prompt generation.arXiv preprint arXiv:2402.12329, 2024. Jagielski, M., Carlini, N., Berthelot, D., Kurakin, A., and Papernot, N. High accuracy and high fidelity extraction of neural networks. InUSENIX Security Symposium, 2020. Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. InEMNLP, 2023. Juuti, M., Szyller, S., Marchal, S., and Asokan, N. PRADA: protecting against DNN model stealing attacks. InEu- roS&P, 2019. Lee, K.-H., Nachum, O., Yang, M. S., Lee, L., Free- man, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., and Mordatch, I. Multi-game decision transformers. InAdvances in Neural Information Pro- cessing Systems, 2022. Liu, A., Han, X., Wang, Y., Tsvetkov, Y., Choi, Y., and Smith, N. A. Tuning language models by proxy.arXiv preprint arXiv:2401.08565, 2024. Milli, S., Schmidt, L., Dragan, A. D., and Hardt, M. Model reconstruction from model explanations. InProceedings of the Conference on Fairness, Accountability, and Trans- parency, 2019. Mitchell, E., Rafailov, R., Sharma, A., Finn, C., and Man- ning, C. D. An emulator for fine-tuning large language models using small language models. InICLR, 2024. Morris, J. X., Zhao, W., Chiu, J. T., Shmatikov, V., and Rush, A. M. Language model inversion.arXiv preprint arXiv:2311.13647, 2023. OpenAI.Using logit bias to define token probability, 2023.URLhttps://help.openai.com/en/ articles/5247780-using-logit-bias-to- 10 Stealing Part of a Production Language Model define-token-probability. Accessed Febraury 1, 2024. OpenAI.Create chat completion, 2024.URL https://platform.openai.com/docs/api- reference/chat/create . Accessed January 30, 2024. OpenAI et al. GPT-4 Technical Report, 2023. Pal, S., Gupta, Y., Kanade, A., and Shevade, S. Stateful detection of model extraction attacks.arXiv preprint arXiv:2107.05166, 2021. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language Models are Unsupervised Mul- titask Learners. Technical report, OpenAI, 2019. URL https://rb.gy/tm8qh. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kun- coro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J.-B., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., Weidinger, L., Gabriel, I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., and Irving, G. Scaling language models: Methods, analysis and insights from training gopher, 2022. Ren, J., Zhao, Y., Vu, T., Liu, P. J., and Lakshminarayanan, B. Self-evaluation improves selective generation in large language models.arXiv preprint arXiv:2312.09300, 2023. Rolnick, D. and Kording, K. Reverse-engineering deep relu networks. InInternational Conference on Machine Learning, 2020. Shamir, A., Canales-Martinez, I., Hambitzer, A., Chavez- Saab, J., Rodrigez-Henriquez, F., and Satpute, N. Poly- nomial time cryptanalytic extraction of neural network models.arXiv preprint arXiv:2310.08708, 2023. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and efficient founda- tion language models.arXiv preprint arXiv:2302.13971, 2023. Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., and Risten- part, T. Stealing machine learning models via prediction APIs. InUSENIX Security Symposium, 2016. Veit, A., Wilber, M. J., and Belongie, S. J. Residual net- works behave like ensembles of relatively shallow net- works. InAdvances in Neural Information Processing Systems, p. 550–558, 2016. Wei, J., Zhang, Y., Zhou, Z., Li, Z., and Al Faruque, M. A. Leaky DNN: Stealing deep-learning model secret with GPU context-switching side-channel. InIEEE/IFIP In- ternational Conference on Dependable Systems and Net- works (DSN), 2020. Yang, K. and Klein, D. FUDGE: Controlled text generation with future discriminators. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), ACL, 2021. Zanella-Beguelin, S., Tople, S., Paverd, A., and Köpf, B. Grey-box extraction of natural language models. InIn- ternational Conference on Machine Learning, p. 12278– 12286. PMLR, 2021. Zhang, B. and Sennrich, R. Root mean square layer normal- ization.NeurIPS, 2019. 11 Stealing Part of a Production Language Model A. What’s Going On With GPT-2 Small? Our attack nearly perfectly extracts the model size of all models—except for GPT-2 Small where our extracted size of 757 is off by 11 from the correct 768. Why is this? In Figure 3 we directly inspect this model’s final hidden activation vector across10, 000different model queries and perform SVD of the resulting activation matrix. We see that despite GPT-2 actually having 768 potential hidden neurons, there are only757different activation directions. Thus, while this model istechnicallya 768 dimensional model, in practice it behaves as if it was a 757 (i.e, the rank of the embedding matrix is 757) dimensional model, and our attack has recovered this effective size. However, when running the model in higher float64 precision, we find that indeed all dimensions are used, but that the smallest dozen or so singular values are much smaller than the other singular values, an observation made by concurrent work (Cancedda, 2024). 600650700750800850900 Sorted Singular Values 10 0 10 1 10 2 10 3 Magnitude (a).Singular values of GPT-2 Small (default bfloat16 precision) 600650700750800850900 Sorted Singular Values (computed in float64) 10 10 10 8 10 6 10 4 10 2 10 0 10 2 Magnitude (b).Singular values of GPT-2 Small (higher float64 precision) Figure 3.Singular values of final hidden activations of GPT-2 Small. B. Accounting for Normalization Layers B.1. LayerNorm Does Not Affect Our RankhAssumption Almost all LLMs that have publicly available architecture details use LayerNorm (Ba et al., 2016) or RMSNorm (Zhang & Sennrich, 2019) just before applying the output projectionW(Biderman, 2024). LayerNorm begins with a centering step, which projects its input onto a(h−1)-dimensional subspace (and RMSNorm does not). In theory, this could break our assumption that the rank of the matrix with columnsg θ ( p i ) (i=1, ...,n) has rankh(Lemma 4.1). In practice, all LLMs we surveyed (Biderman, 2024) enabled the LayerNorm bias, which means the matrices had full rankh(besides GPT-2 Small: see Appendix A). B.2. Stealing Architectural Details About Normalization Layers B.2.1. THEORY The difference between LayerNorm and RMSNorm (Appendix B.1) could enable attackers to deduce whether models used LayerNorm or RMSNorm. If an attacker recovered an initial logit-vector API query responseO ( p 0 ) , then they could apply Lemma 4.1 toO ( p 1 ) −O ( p 0 ) ,...,O ( p n ) −O ( p 0 ) . 1 From the description of the API at the top of Section 4.1, it follows thatO ( p i ) −O ( p 0 ) =W(g θ ( p i ) −g θ ( p 0 ) ). This subtraction ofgterms occurs immediately after LayerNorm, 1 Throughout this appendix section, we assume the sum of logit outputs is always 0. We can calculate centered logits from logprobs by subtracting the mean logits across the vocab dimension. 12 Stealing Part of a Production Language Model so cancels the LayerNorm bias term. Hence, if we apply the Lemma 4.1 attack with this subtraction modification to a model using LayerNorm, then the resultant ‘h’ output will be smaller by 1 (due to Appendix B.1). This would imply the model used LayerNorm rather than RMSNorm, because RMSNorm does not project onto a smaller subspace and so would not have a decrease in ‘h’ value if we were to use this subtraction trick. B.2.2. RESULTS To confirm that the method from Appendix B.2.1 works, we test whether we can detect whether the GPT-2, Pythia and LLAMA architectures use LayerNorm or RMSNorm from their logit outputs alone. We found that the technique required two adjustments before it worked on models with lower than 32-bit precision (it always worked with 32-bit precision). i) We do not subtractO ( p 0 ) from logits queries, but instead subtract the mean logits over all queries, i.e. 1 n P n i=1 O ( p i ) . Since the average of several points in a common affine subspace still lie on that affine subspace, this doesn’t change the conclusions from Appendix B.2.1. i) We additionally found it helped to calculate this mean in lower precision, before casting to 64-bit precision to calculate the compact SVD. The results are in Figure 4. We plot the singular value magnitudes (as in Figure 1) and show thatthere is a drop in thehth singular value for the architectures using LayerNorm, but not for architecture using RMSNorm: 40804085409040954100410541104115 i 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 log 10 || i || LLAMA-7B (bfloat16) Normal attack Subtracted bias h (a) LLAMA-7B (RMSNorm). 158015851590159516001605161016151620 i 2 1 0 1 2 log 10 || i || GPT-2 XL (float32) Normal attack Subtracted bias h (b) GPT-2 XL (LayerNorm). 510051055110511551205125513051355140 i 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 log 10 || i || Pythia-12B (float16) Normal attack Subtracted bias h (c) Pythia-12B (LayerNorm). Figure 4.Detecting whether models useLayerNormorRMSNormby singular value magnitudes. Is this attack practical for real models? We perform the same attack on the logprobs we obtained foradaandbabbage. 2 We see in Figure 5a-b that indeed the drop in thehth singular values occurs for these two models that use LayerNorm (GPT-3’s architecture was almost entirely inherited from GPT-2): 10051010101510201025103010351040 0.2 0.0 0.2 0.4 0.6 0.8 ada Normal attack Subtracted bias h (a)adausesLayerNorm. 20302035204020452050205520602065 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 babbage Normal attack Subtracted bias h (b)babbageusesLayerNorm. 40804085409040954100410541104115 0.55 0.60 0.65 0.70 0.75 gopher-7b (bfloat16) Normal attack Subtracted bias h (c) Gopher-7B usesRMSNorm. Figure 5. Stress-testing the LayerNorm extraction attack on models behind an API (a-b), and models using both RMSNorm and biases (c). As a final stress test, we found that all open language models that use RMSNorm do not use any bias terms (Biderman, 2024). Therefore, we checked that our attack would not give a false positive when applied to a model with RMSNorm but with biases. We chose Gopher-7B (Rae et al., 2022), a model with public architectural details but no public weight access, 2 Unfortunately, we deleted the logprobs for GPT-3.5 models before we created this attack due to security constraints. 13 Stealing Part of a Production Language Model that uses RMSNorm but also biases (e.g. on the output logits). In Figure 5c we show that indeed thehth singular value does not decrease for this model that uses RMSNorm. C. Proof of Lemma 4.2 Restating the lemma from Section 4.2: Lemma 4.2In the logit-API threat model, under the assumptions of Lemma 4.1:(i)The method from Section 4.2 recovers ̃ W=W·Gfor someG∈R h×h ;(i)With the additional assumption thatg θ (p)is a transformer with residual connections, it is impossible to extractWexactly. We first give a short proof of (i): Proof.(i) To show we can recover ̃ W=W·G, recall Lemma 4.1: we have access toQ ⊤ =W·Hfor someH∈R h×n . Using the compact SVD ofQfrom the method in Section 4.2,W·H·V=U·Σ. We knowG:=H·V∈R h×h , hence if we take ̃ W=U·Σ, we have ̃ W=W·G. Proving Lemma 4.2(i) requires several steps due to the complexity of the transformer architecture: we progressively strengthen the proof to apply to models with no residual connections (C.1), models with residual connections (C.2), models with RMSNorm (C.4), LayerNorm (C.5) and normalization with anεterm (C.6). C.1. Proof of Lemma 4.2(i) in Models With Fully-connected Layers Proof of Lemma 4.2(i).As a gentle warmup, we prove (i) under the additional assumption that the model does not use normalization layers (LayerNorm or RMSNorm) in its architecture. To prove (i) we show it is possible to find a two distinct sets of model parametersθ,θ ′ with different embedding projection matrices that result in identical API outputs. We begin with a simpler case whereg θ does not have residual connections but a fully connected (FC) final layer. In this case, for any invertibleh×hmatrixS, we have thatg θ ( p ) =Sg θ ′ ( p ) whereθ ′ is the same asθexcept that the weights of the final FC layer are pre-multiplied byS −1 . Hence, ifg θ has a final FC layer, it is impossible to distinguish between the embedding projection layerWacting ong θ and the embedding projection layerW·Sacting ong θ ′ , given access to the output of the APIOonly. C.2. Proof of Lemma 4.2(i) With Residual Layers More generally, ifg θ is composed of residual layers but no normalization layers theng θ (p) = P i L i (p), whereL i (p)is the output of theith residual layer in the model, ignoring the skip connection (Elhage et al., 2021; Veit et al., 2016). Assume also that eachL i has a final layer that is a fully connected linear layer and a linear input layer (this assumption is true for both attention and MLP modules in transformers without normalization layers). Constructingθ ′ such that eachL i has input weights pre-multiplied byS −1 and output FC weights multiplied byS, we haveg θ ′ (p) = P i SL i (p) =S·g θ (p)by linearity. Finally, by using a new embedding projection matrix(S −1 ) ⊤ ·W ⊤ and calculating ((S −1 ) ⊤ ·W ⊤ ) ⊤ ·g θ ′ (p) =W·g θ (p),(2) we have shown that logit outputs are identical and so again we cannot distinguish these transformers by queryingOandO ′ alone. C.3. Normalization Layers and Orthogonal Matrices In Sections C.3-C.6 we can no longer use general invertible matricesSin our arguments, and must instead use orthogonal matrices, matricesUsuch thatU ⊤ U=I. In models with LayerNorm, we specialise further, too (Appendix C.5). Lemma C.1.The RMSNorm operation is equal tox7→Wn(x) +bwhereWis a diagonal matrix. Proof.RMSNorm is conventionally written as x7→ w·x q 1 h P i x 2 i +b(3) 14 Stealing Part of a Production Language Model wherewis multiplied elementwise by normalizedx. Clearly this can be written as a diagonal matrix. Further, we can multiply this diagonal matrix by √ hto cancel that factor in the denominator of Equation (3). Sincen(x) =x/||x||=x P i q x 2 i we get the result. Intuitively, the proof in Appendix C.2 relied on pre-multiplying the input projection weight of layers by a matrixS −1 , so that this cancelled the rotationSapplied to the model’s hidden state (called the ‘residual stream’ in mechanistic interpretability literature (Elhage et al., 2021)). Formally, if we let the input projection layer beM, we were using the fact that MS −1 ( Sx ) =Mx . However, since models with normalization layers use these before the linear input projection, the result of applyingSto the hidden state, if we apply the same procedure, produces the activation (MS −1 )(Wn(Sx) +b)(4) but since in generalnandSdo not commute, we cannot conclude that theStransformations preserve the transformer’s outputs. We will show that if we takeS=Uan orthogonal matrix, then we still get a general impossibility result. To do this, we will need a simple result from linear algebra: Lemma C.2.Letx∈R h . Then the normalization mapn(x):= x ||x|| commutes with orthogonal matricesU. Proof of Lemma C.2.We need to show that Ux ||x|| = Ux ||Ux|| . This is true sincex ⊤ U ⊤ Ux=x T x, so||Ux||=||x||. C.4. Proof of Lemma 4.2(i) in Models With RMSNorm In Lemma C.2, we showed that orthogonal matricesUcommute with normalization. Hence if we multiply all layer output weights byU, but pre-multiply all layer input projection weights byWU ⊤ W −1 , then the effect of the linear projection layer is (MWU ⊤ W −1 )(Wn(Ux) +b) = (MWU ⊤ W −1 )(WUn(x) +b) =M(Wn(x) +b)(5) which is identical to the original model. Applying this procedure to all layers added to the hidden state (using the different Wdiagonal matrices each time) gives us a modelg θ ′ (p)such thatg θ ′ (p) =Ug θ ′ (p)so a different embedding projection matrixWU ⊤ will give identical outputs to the original modelg θ (p)(with embedding projectionW). Note that we ignore what happens tobin the above arguments, since any sequence of affine maps applied to a constant b∈R h yields a constantb ′ ∈R h , and we can just useb ′ instead ofbing θ ′ . C.5. Proof of Lemma 4.2(i) in Models With LayerNorm The LayerNorm operation is the composition of a centering operationx7→x− ̄xwith RMSNorm (i.e. first centering is applied, then RMSNorm). Therefore the identical argument to Appendix C.4 goes through, besides the fact that we needU to also commute with the centering operation. Since the centering operation fixes a(h−1)dimensional subspace defined by1 T x=0where1∈R h is the vector of ones, it is enough to impose an additional condition thatU 1∈−1,1. C.6. Proof of Lemma 4.2(i) in Models With Normalizationε̸=0 We now extend to realistic models where theεin the denominator of LayerNorm is not 0. We can do this because the only fact we used aboutx7→n(x)was thatx7→n(Ux)was identical tox7→Un(x). In turn Lemma C.2 relied on ||Ux||=||x||due to orthogonality. But adjustingn(x)ton ′ (x):=x q 1 h ||x|| 2 +ε(i.e. normalization with an epsilon), since||x||=||Ux||,n ′ commutes withU, and so the proofs in Appendix C.4 and Appendix C.5 still work when usingn ′ instead ofn. Therefore finally, we have proven the impossibility result Lemma 4.2(i) in all common model architectures (all non-residual networks that end with dense layers, and all transformers from Biderman (2024)). 15 Stealing Part of a Production Language Model D. Derivation of Binarized Logprob Extraction (Section 5.4) To begin, observe that we can write y top =logit top −log X i exp(logit i ) y ′ top =logit top −log exp(logit t −1) + X i̸=t exp(logit i ) LetN= P i exp ( logit i ) andp=exp ( logit t ) /N. Then, we can rewrite y top =logit top −logN y top =logit top −log(N+ (1/e−1)pN) Subtracting the two, we get y top −y ′ top =log ( 1+ (1/e−1)p ) =⇒p= exp(y top −y ′ top )−1 1/e−1 . Related work.Concurrent work (Morris et al., 2023) discusses a similar but weaker two-query logprob extraction. Their attack requires a logit bias larger thanlogit top −logit i and top-2 logprob access; our attack works as soon the logit bias is allowed to be nonzero, and with top-1 logprob access. E. Efficient Recovery of Logits From TopkLogprobs APIs In Section 5.3 of the main body, we presented a simple and practical method for extracting the entire logits vector via multiple queries to an API that only provides the top few logprobs and accepts a logit bias with each query. In this section we present more efficient methods. The method we presented earlier uses a reference token. We set this to some arbitrary value (e.g.,0) and then compare the logits for all other tokens to this one. This approach is numerically stable, but is slightly wasteful: of the topKlogprobs returned by the API, one is always the reference token. Hence, we only recoverK−1logits per query with this method. In this appendix, we present linear algebraic methods that are able to recoverKlogits per query to the top-Klogprobs API. Setting:Recall that there is an unknown vectorz=W·g θ (p)∈R ℓ (i.e., the logits for a given promptp) that we want to recover. We can make multiple queries to the API with the same promptO(p,b). Each query is specified by a vector b∈R ℓ (a.k.a. the logit bias). We receive answers of the form(i,a i (z,b))∈N×R, whereiis a token index anda i (z,b) is a logprob: a i (z,b) =log exp(z i +b i ) P ℓ j exp(z j +b j ) ! =z i +b i −log ℓ X j exp(z j +b j ) .(6) Each query may receive multiple answers (namely, theKlargesta i (z,b)values). For notational simplicity, we denote multiple answers to one query the same way as multiple queries each returning one answer. Suppose queriesb 1 ,·,b m were asked and we receivedmanswers(i 1 ,a i 1 (z,b 1 ))←O(p,b 1 ),·,(i m ,a i m (z,b m ))←O(p,b m ). Our goal is to computezfrom the answersa i (z,b). E.1. Warmup: Single Logprob API (K=1) As a starting point, suppose the API only returns the single largest logprob (i.e.,K=1). The approach from Section 5.3 cannot work in this setting because we cannot obtain the logprob of both the reference token and another token at the same time, meaning we can recover less than1logit per query. The high-level idea to overcome this problem is that, instead of normalizing logits relative to a reference token, we shall normalize the logits to be logprobs. That is, we recover the logits with the normalization P j exp(z j ) =1. With this normalization it is no longer necessary to include a reference token in every query. 16 Stealing Part of a Production Language Model Fix a token indexiand letb i =Bandb j =0for allj̸=i. We query the API with this logit bias and assume thatBis large enough that tokeniis returned: (i,a i (z,b))←O(p,b). From Equation 6, a i (z,b) =z i +b i −log ℓ X j exp(z j +b j ) =z i +B−log exp(z i +B) + X j̸=i exp(z j ) =z i +B−log exp(z i +B)−exp(z i ) + ℓ X j exp(z j ) , =⇒z i +B−a i (z,b) =log exp(z i +B)−exp(z i ) + ℓ X j exp(z j ) , =⇒exp(z i +B−a i (z,b)) =exp(z i +B)−exp(z i ) + ℓ X j exp(z j ), =⇒exp(z i +B−a i (z,b))−exp(z i +B) +exp(z i ) = ℓ X j exp(z j ), =⇒exp(z i )· ( exp(B−a i (z,b))−exp(B) +1 ) = ℓ X j exp(z j ), =⇒exp(z i ) = P ℓ j exp(z j ) exp(B−a i (z,b))−exp(B) +1 , =⇒z i =log ℓ X j exp(z j ) −log ( exp(B−a i (z,b))−exp(B) +1 ) . Thus if we normalize P ℓ j exp(z j ) =1, we have z i =−log ( exp(B−a i (z,b))−exp(B) +1 ) .(7) E.2. RecoveringKLogits FromKLogprobs The approach from the previous subsection extends to the setting where each API query returns the topKlogprobs. In practice we work withK=5. We are able to recoverKlogits. Again, instead of using a reference token to normalize the logits, we will normalize P j exp(z j ) =1. However, in this setting we will need to solve aK-by-Ksystem of linear equations. FixKtoken indicesi 1 ,·,i K and letb i k =B fork∈ 1,·,Kandb j =0for allj/∈ i 1 ,·,i K . We query the API with this logit bias and assume thatBis large enough that the logprobs fori 1 ,·,i K are returned as the topK logprobs: (i 1 ,a i 1 (z,b)),(i 2 ,a i 2 (z,b)),·,(i K ,a i K (z,b))←O(p,b). Letz∈R ℓ be the (unknown) logits and letN= P i exp(z i )be the normalizing constant. For eachk∈1,·,K, we 17 Stealing Part of a Production Language Model have a i k (z,b) =z i k +B−log X i∈i 1 ,·,i K exp(z i +B) + X i/∈i 1 ,·,i K exp(z i ) =z i k +B−log (e B −1) X i∈i 1 ,·,i K exp(z i ) + ℓ X i exp(z i ) =z i k +B−log (e B −1) X i∈i 1 ,·,i K exp(z i ) +N , =⇒z i k +B−a i k (z,b) =log (e B −1) X i∈i 1 ,·,i K exp(z i ) +N , =⇒exp(z i k +B−a i k (z,b)) = (e B −1) X i∈i 1 ,·,i K exp(z i ) +N, And therefore we can conclude exp(B−a i k (z,b))·exp(z k )−(e B −1) X i∈i 1 ,·,i K exp(z i ) =N. This linear system of equations can be expressed in matrix form: A· exp(z i 1 ) exp(z i 2 ) . . . exp(z i K ) = N N . . . N , whereAis aK×Kmatrix with entries A k,j = ( exp(B−a i k (z,b))−(e B −1)ifj=k −(e B −1)ifj̸=k. Note thatAis a rank-one perturbation of a diagonal matrix, that is, if1is the all-ones vector, then A=diag 1≤k≤K (exp(B−a i k (z,b)))−(e B −1)11 T , wherediag 1≤k≤K (exp(B−a i k (z,b))) denotes a diagonal matrix with thek-th diagonal entry beingexp(B−a i k (z,b)) . Inverting a diagonal matrix is easy and thus we can use the Sherman-Morrison formula to compute the inverse ofA: A −1 =diag 1≤k≤K (exp(a i k (z,b)−B))) + (e B −1) diag 1≤k≤K (exp(a i k (z,b)−B)))11 T diag 1≤k≤5 (exp(a i k (b)−B))) 1−(e B −1)1 T diag 1≤k≤5 (exp(a i k (b)−B)))1 =diag(v) + (e B −1) v T 1−(e B −1)1 T v , 18 Stealing Part of a Production Language Model wherev∈R K is the vector with entriesv k =exp(a i k (z,b)−B). Hence exp(z i 1 ) exp(z i 2 ) . . . exp(z i K ) =A −1 · N N . . . N = diag(v) + (e B −1) v T 1−(e B −1)1 T v ·1·N = v+ (e B −1)v T 1 1−(e B −1)1 T v ·N = 1+ (e B −1)1 T v 1−(e B −1)1 T v ·N·v = N 1−(e B −1) P j v j ·v, =⇒z i k =log A −1 1N k =log Nv k 1−(e B −1) P K j v j ! =log Nexp(a i k (z,b)−B) 1−(e B −1) P K j exp(a i j (z,b)−B) ! =logN+a i k (z,b)−B−log 1−(e B −1) K X j exp(a i j (z,b)−B) =logN+a i k (z,b)−B−log 1−(1−e −B ) K X j exp(a i j (z,b)) . If we normalizeN=1, this gives us a formula for computing the logits: z i k =a i k (z,b)−B−log 1−(1−e −B ) K X j exp(a i j (z,b)) .(8) Note that settingK=1yields the same result as in Equation 7. Recovery using Equation 8 is more efficient than the method in Section 5.3, as we recoverKlogitsz i 1 ,z i 2 ,·,z i K rather than justK−1logits. However, ifBis large, numerical stability may be an issue. (And, ifBis small, the logit bias may be insufficient to force the API to output the desired tokens by placing them in the topK.) Specifically, asB→ ∞, we have(1−e −B ) P K j exp(a i j (z,b))→1and so the logarithm in Equation 8 tends tolog(1−1) =−∞; this means we may have catastrophic cancellation. Related work.Two works published during the responsible disclosure period use a similar procedure, and deal with numerical issues in different ways. (Chiu, 2024) start with a lowBfor the whole vocabulary, then increaseBand ask for all tokens that haven’t appeared before, and repeat until all tokens are covered. (Hayase et al., 2024) use the method in Appendix E.1, and setB=−ˆz i , whereˆz i is an estimate ofz i inherent to their application. It is possible variants of this method have been discussed before our or these works, but we are not aware of further references. E.3. General Method In general, we may not have have full control over which logprobs the API returns or which logit bias is provided to the API. Thus we generalize the linear algebraic approach above to reconstruct the logits from arbitrary logit biases and tokens. 19 Stealing Part of a Production Language Model Suppose queriesb 1 ,·,b m were asked and we receivedmanswers(i 1 ,a i 1 (z,b 1 ))←O(p,b 1 ),...,(i m ,a i m (z,b m ))← O(p,b m ). (If a query returns multiple answers, we can treat this the same as multiple queries each returning one answer.) As before, rearranging Equation 6 gives the following equations. ∀k∈[m]exp(a i k (z,b k i k )) = exp(z i k +b k i k ) P ℓ j exp(z j +b k j ) . ∀k∈[m] ℓ X j exp(z j +b k j ) =exp(z i k +b k i k −a i k (z,b k )). ∀k∈[m] ℓ X j exp(z j )·exp(b k j ) =exp(z i k )·exp(b k i k −a i k (z,b k )). ∀k∈[m] ℓ X j exp(b k j )−I[j=i k ]·exp(b k i k −a i k (z,b k )) ·exp(z j ) =0. A· exp(z 1 ) exp(z 2 ) . . . exp(z ℓ ) = 0 0 . . . 0 , where∀k∈[m]∀j∈[ℓ]A k,j =exp(b k j )· 1−I[j=i k ]·exp(−a i k (z,b k )) . HereI[j=i k ]is1ifj=i k and0otherwise. IfAis invertible, then this linear system can be solved to recover the logitsz. Unfortunately,Ais not invertible: Indeed, we know that the solution cannot be unique because shifting all the logits by the same amount yields the exact same answersa i (z,b) =a i (z+1,b). That is, we expect a one-dimensional space of valid solutions toA·exp(z) =0. To deal with this we simply add the constraint thatz 1 =0or, equivalently,exp(z 1 ) =1. This corresponds to the system b A·exp(z) = 1 0·0 A · exp(z 1 ) exp(z 2 ) . . . exp(z ℓ ) = 1 0 . . . 0 . (We could also normalize P ℓ i exp(z i ) =1 . This corresponds to the first row of b Abeing all1s instead of one1.) This is solvable as long as the augmented matrix has a nonzero determinant det b A =det 1 0·0 A =det(A 1:m,2:ℓ ).(9) HereA 1:m,2:d denotesAwith the first column removed. Note that we are settingm=ℓ−1. This is the minimum number of query-answer pairs that we need. If we have more (i.e.,m≥ℓ), then the system is overdetermined. Having the system be overdetermined is a good thing; the extra answers can help us recover the logprobs with more precision. The least squares solution to the overdetermined system is given by b A T b A· exp(z 1 ) exp(z 2 ) . . . exp(z ℓ ) = b A T 1 0 . . . 0 .(10) This provides a general method for recovering the (normalized) logits from the logprobs API. Related work.(Zanella-Beguelin et al., 2021) have an almost identical method, although they operate in the setting of a publicly known encoder and reconstructing the last layer. 20 Stealing Part of a Production Language Model F. Extraction From Logprob-free APIs A more conservative API provider may remove access to the combination of logit bias and logprobs entirely. Indeed, after disclosing our attack to OpenAI, they removed the ability for logit bias to impact the top logprobs—thus preventing the attacks from the prior sections. To exploit situations such at this, we further develop several logprob-free attacks that recover the complete logit vector by performing binary search on the logit bias vector, albeit at increased cost. 3 API:Some APIs provide access to a logit bias term, but do not provide any information about the logprobs. Thus, we have, O(p,b) =ArgMax ( logsoftmax ( W·g θ (p) +b )) . whereArgMax ( z ) returns the index of the highest coordinate in the vectorz∈R l . In this section, we will use the notation b=i:zto denote that the bias is set tozfor tokeniand0for every other token. We also useb=to denote that no logit bias is used. Finally, we assume that the bias is restricted to fall within the range[−B,B]. What can be extracted?The attacks developed in this Section reconstruct the logit vector up to an additive (∞-norm) error ofε. F.1. Warm-up: Basic Logprob-free Attack Method.We make one simple insight for our logprob-free attacks: sampling with temperature 0 produces the token with the largest logit value. By adjusting the logit bias for each token accordingly, we can therefore recover every token’s logit value through binary search. Formally, letpbe the prompt, and relabel tokens so that the token with index0is the most likely token in the response top, given byO(p,b=). For each tokeni̸=0, we run a binary search over the logit bias term to find the minimal valuex i ≥0such that the model emits tokeniwith probability 1. This recovers all logits (like all prior attacks, we lose one free variable due to the softmax). Algorithm 2Learning logit differences α i ←−B,β i ←0 whileβ i −α i > εdo ifO p,b=i:− α i +β i 2 =0then β i ← α i +β i 2 else α i ← α i +β i 2 end if Return α i +β i 2 end while Analysis.This attack, while inefficient, correctly extracts the logit vector. Lemma F.1.For every tokenisuch thatlogit i −logit 0 ≥−B, Algorithm 2 outputs a value that is at mostεaway from the logit i −logit 0 in at mostlog B ε API queries. Proof.The API returns the (re-ordered) token0as long as the logit bias added is smaller thanlogit i −logit 0 . By the assumption, we know thatlogit i −logit 0 ∈[−B, 0]. The algorithm ensures thatβ i ≥logit i −logit 0 ≥α i at each iteration, as can be seen easily by an inductive argument. Further,β i −α i decreases by a factor of2in each iteration, and hence at termination, we can see that the true value oflogit i −logit 0 is sandwiched in an interval of lengthε. Furthermore, it is clear that the number of iterations is at mostlog 2 B ε and hence so is the query cost of this algorithm. Limitations of the approach.Iflogit i −logit 0 <−2Bit is easy to see there is no efficient way to sample the tokeni, hence no way to find information aboutlogit i without logprob access. There is a way to slightly increase the range for 3 We release supplementary code that deals with testing these attacks without direct API queries athttps://github.com/ dpaleka/stealing-part-lm-supplementary. 21 Stealing Part of a Production Language Model −2B≤logit i −logit 0 ≤ −Bby adding negative logit biases to the tokens with the largest logit values, but we skip the details since for most models, for the prompts we use, the every token satisfieslogit i −logit 0 >−B. Related work.Concurrent work (Morris et al., 2023) has discussed this method of extracting logits. F.2. Improved Logprob-free Attack: Hyperrectangle Relaxation Center We can improve the previous attack by modifying the logit bias of multiple tokens at once. API:We use the same API as in the previous section, with the additional constraint that theOaccepts at mostN+1 tokens in the logit bias dictionary. We again first run a queryO(p,b=)to identify the most likely token and set its index to0. Our goal is to approximatelogit i −logit 0 forNdifferent tokens. IfN < l−1, we simply repeat the same algorithm for different batches ofNtokens l−1 N times. Algorithm 3Learning logit differences with multi-token calls α i ←−B,β i ←0∀i=1,...,N C=logit:logit i −logit 0 ≤B∀i=1,...,N forTroundsdo b i ←− α i +β i 2 fori=0,...,N k←O(p,b=0:b 0 , 1:b 1 ,...,N:b N ) forj̸=kdo C ←C∩logit:logit k +b k ≥logit j +b j end for fori=0,...,Ndo α i ←min logit∈C logit i −logit 0 β i ←max logit∈C logit i −logit 0 end for end for Return[α i ,β i ]∀i∈0,...,N Method.Our approach queries the API with the logit bias set for several tokens in parallel. The algorithm proceeds in rounds, where each round involves querying the API with the logit bias set for several tokens. Suppose that the query returns tokenkas output when the logit bias was set toi:b i fori=1,...,land the prompt isp. Then, we know thatlogit k +b k ≥logit j +b j for allj̸=kby the definition of the API. This imposes a system of linear constraints on the logits. By querying the model many times, and accumulating many such systems of equations, we can recover the logit values more efficiently. To do this, we accumulate all such linear constraints in the setC, and at the end of each round, compute the smallest and largest possible values forlogit i −logit 0 by solving a linear program that maximizes/minimizes this value over the constraint setC. Thus, at each round, we can maintain an interval that encloseslogit i −logit 0 , and refine the interval at each round given additional information from that round’s query. AfterTrounds (whereTis chosen based on the total query budget for the attack), we return the tightest known bounds on each logit. Lemma F.2.Suppose thatlogit i −logit 0 ∈[−B, 0]for alli=1,...,l. Then, Algorithm 3 returns an interval[α i ,β i ]such thatlogit i −logit 0 ∈[α i ,β i ]for eachisuch thatlogit i −logit 0 ∈[−B, 0]. Furthermore, each round in the algorithm can be implemented in computation timeO(N 3 )(excluding the computation required for the API call). Proof. Algorithm 3 maintains the invariant thatlogit i −logit 0 ∈[α i ,β i ]in each round. We will prove by induction that this is true and that the true vector of logits always lies inC. Note that by the assumption stated in the Lemma, this is clearly true at the beginning of the first round. Suppose that this is true afterK < Trounds. Then, in theK+1-th round, the constraints added are all valid constraints for the true logit vector, since the API returning tokenkguarantees that logit k +b k ≥logit j +b j for allj̸=k. Hence, by induction, the algorithm always ensures thatlogit i −logit 0 ∈[α i ,β i ]. 22 Stealing Part of a Production Language Model In Appendix F.2.1, we show the LP to computeα i ,β i for allican be seen as an all-pairs shortest paths problem on graph with edge weightsc jk =min rounds b j −b k where the minimum is taken over all rounds where the token returned wask. This ensures the computation complexity of maintaining the logit difference intervals isO(N 3 ). F.2.1. SHORTEST-PATHFORMULATION OF THELOGPROB-FREEATTACKLP It is actually possible to improve the computational efficiency of the hyperrectangle relaxation of the polytopeC. Here we show how to formulate this problem as a shortest path problem on a weighted graph. This enables us to quickly compute the exact[α i ,β i ]for alli∈1,...,Nafter each query. Lemma F.3.LetG= (0, 1,...,N,E)be a weighted directed graph without negative cycles. LetP ⊂R n+1 be the solution set of a system of linear inequalities: logit i −logit j ≤c ji ∀j c ji −→i∈E Then iflogit 0 =0, we have max x∈C logit i =distance inGfrom0toi. Proof.Lete 0j 1 ,e j 1 j 2 ,...,e j m−1 i be the edges of the minimum distance path from0toiinG. We have logit i ≤logit j m−1 +c j m−1 i ≤... ≤logit 0 + m−1 X t=1 c j t+1 j t = m−1 X t=1 c j t+1 j t , hence the shortest path is an upper bound onlogit i . To prove feasibility, we claim that settinglogit i to be the distance from0 toisatisfies all the inequalities. Assume some inequalitylogit i −logit j ≤c ji is violated. Then we can go from0→j→i inGwith a total weight oflogit j +c ji <logit i , which contradicts the assumption thatlogit i is the distance from0toi. To apply this to our setting, note that (1) all constraints, even the initialα i ≤logit i ≤β i , are of the required form; (2) the graph has no negative cycles because the true logits give a feasible solution. (3) we can get the lower bounds by applying the same procedure to the graph induced by inequalities on−logit i . We can find the distances from0to all other vertices using the Bellman-Ford algorithm inO(N 3 )time. IfN=300, this is at most comparable to the latency ofO. Since onlyNedges of the graph update at each step, we note that the heuristic of just updating and doing a few incremental iterations of Bellman-Ford gets[α i ,β i ]to high precision in practice. The number of API queries and the token cost, of course, remains the same. F.3. Improved Logprob-free Attack: Better Queries on Hyperrectangles The main issue of the previous approach is that some tokens are sampled more often than others, even in the case our prior for thelogitvector is uniform over[−B, 0]. This is because the "centering of the hyperrectangle" logit bias does not partition the hyperrectangle into equally-sized parts labeled by the argmax coordinate. For example, ifβ i −α i ≪β j −α j , under an uniform prior over[α i ,β i ]×[α j ,β j ],jwill be much more likely to be the output token thani. Hence, in Algorithm 3 we rarely get constraints lower-boundinglogit i in terms of other logits, which makes for weaker relaxations ofC. Our solution is to bias tokens so that the output token distribution is closer to uniform; in particular, biasing the token with the smallestβ t −α t (the0token) to have probability exactly1/(N+1)given an uniform prior over the hyperrectangle. One logit bias that satisfies this is: b i =−(1−c)α i −cβ i ∀i=0,...,N wherec=exp(−log(N+1)/N).(11) We now run Algorithm 3, with one simple modification: we replaceb i =− α+β 2 withb=−(1−c)α−cβ. As can be seen in Table 4, the modified algorithm outperforms the method in F.2 significantly. 23 Stealing Part of a Production Language Model The goal of balanced sampling of all output tokens can be approached in many ways. For example, we could tunecin the above expression; bias tokens whichOhasn’t returned previously to be more likely; or solve for the exact logit bias that separatesC(or some relaxation) into equal parts. However, we show in Appendix G that, under some simplifying assumptions, the queries/logit metric of this method in Table 4 is surprisingly close to optimal. G. How Far Are Our Logprob-Free Attacks From Optimal? In the logprob-free API, we have produced attacks capable of recovering logits and ultimately the embedding hidden dimension and embedding matrix up to a similarity transform. We now provide lower bounds on the minimum number of queries required byanyattacker attempting model stealing under the logprob-free API threat model. Lemma G.1.Assume the entries oflogit∈R l are i.i.d. uniform over[−B, 0]. To recover the vectorlogitup to∞-norm errorε, the number of queries toO(p,·)we need is at least: llog 2 (B/ε) log 2 (l) . Proof. The information content of a single logit value in[−B, 0]up to∞-norm errorεislog 2 (B/ε), assuming a uniform prior overε-spaced points in the interval. Since the logits are independent, the information encoded inllogit values up to ∞-norm errorεisllog 2 (100/ε). Any single query toO, no matter how well-crafted, yields at mostlog 2 (l)bits, because the output is one ofldistinct values. The minimum number of queries required is at least the total information content divided by the information per query, yielding the lower boundllog 2 (B/ε)/log 2 (l). The restriction of biasing at mostNtokens at a time gives us a lower bound of llog 2 (B/ε) log 2 (N) queries, which is a factor oflog 2 (l)/log 2 (N)worse. ForN=300andl≈100,000, this is only a factor of2. ForB=100andN=300, we thus need at least log 2 (B/ε) log 2 (N) ≈0.81+0.12 log 2 (1/ε) queries per logit. If we want between 6 and 23 bits of precision, the lower bound corresponds to 1.53 to 3.57 queries per logit. We see that the best logprob-free attack in Table 4 is only about 1 query per logit worse than the lower bound. The main unrealistic assumption in Lemma G.1 is that the prior over the logit values is i.i.d. uniform over an interval. A better assumption might be that most of the logit values come from a light-tailed unimodal distribution. We leave more realistic lower bounds and attacks that make use of this better prior to future work. H. RecoveringWup to an orthogonal matrix In this section, we present an algorithm for extractingWup to an orthogonalh×hmatrix, instead of a non-singularh×h matrix as in Appendix C. This algorithm requires solving a system ofO(h 2 )linear equations and is hence prohibitive for large models in production, some of which haveh >1000. However, we present proof that this technique works in practice in a notebook 4 on Pythia-14M, despite the assumptions bellow (Appendix H.1). H.1. Assumptions We make a few simplifying assumptions: 4 See here: RecoveringWup to an orthogonal matrix (Colab notebook) 24 Stealing Part of a Production Language Model 1. We merge the final normalization layer weightsγintoWby linearity. 5 2.We assume that the output matrix always outputs 0-centered logits (as in Appendix B.2.1). This isn’t a restrictive assumption since model logprobs are invariant to a bias added to the logit outputs. 3. We assume the numerical precision is high enough that during the final normalization layer, the hidden states are on a sphere. 4. There is no degenerate lower-dimensional subspace containing allg θ (p)for all our queriesp. 5. We assume theεin RMSNorm/LayerNorm is negligible. H.2. Methodology Intuition.The main idea is to exploit the structure imposed by the last two layers of the model: LayerNorm/RMSNorm, which projects model internal activations to a sphere, and the unembedding layer. The model’s logits lie on an ellipsoid of rankhdue to Lemma H.1: Lemma H.1.The image of an ellipsoid under a linear transformation (such as the unembedding layer) is itself an ellipsoid. Thus, we can project the model’s logits outputs to that subspace usingUthe truncated SVD matrix;X=U ⊤ Q. An ellipsoid is defined with the quadratic form as in Lemma H.2: Lemma H.2.An ellipsoid is defined by a single semipositive-definite symmetric matrixAand a center by the following equation:(x−c) ⊤ A(x−c) =1. By expanding this formulax ⊤ Ax−2Acx+c ⊤ Ac=1, substitutingd=Acwe end up with a system that all (projected) model outputs should satisfy. This system is linear in ( h+1 2 ) entries ofA(asAis symmetric), and theh+1entries ofd. The system doesn’t constrain the free termc ⊤ Ac, and we can get rid of it by subtractingx 0 from other outputs of the model, effectively forcing the ellipsoid to pass through the origin, and thus satisfyc ⊤ Ac=1. Using Cholesky decomposition on the PSD symmetric matrixA=M ⊤ , we would end up withMa linear mapping from the unit-sphere to the (origin-passing) ellipsoid, which is equivalent toWup to an orthogonal projection (see Lemma H.3). This should be centered atWb=c−x 0 . Procedure.The procedure we took in order to recoverWup to an orthogonal matrix as implemented in the notebook?? is described as follows at a high level (we expand in detail on several of the steps below): 1. Collect ( h+1 2 ) + (h+1)logits and store them in a matrixQ. 2. Shift to origin:Q←Q−Q[0]. 3. Perform SVD on Q:UΣV ⊤ =Q. 4. Find the model’s hidden dimensionh(e.g. using Section 4.2) and truncateU 5. Solvex ⊤ Ax−2dx=0using SVD/QR to find the nullspace, such that •d=Ac •c ⊤ Ac=1 •Ais symmetric. This process has time complexity:O(h 6 )(in many cases much faster in practice). 6. Findc=A −1 d, and scalecandAto satisfyc ⊤ Ac=1. 7. Use Cholesky decomposition to findMs.t.M ⊤ M=A. 5 For a full explanation of this method of rewriting the unembedding matrix, see Appendix A.1, ‘Folding LayerNorm’ in Gurnee et al. (2024). The intution is thatγis another linear transformation applied just beforeW(a diagonal matrix). Therefore together they are together one matrix multiplication. 25 Stealing Part of a Production Language Model 8. ObtainW=U·M −1 ·Ofor some orthogonal matrixO. In steps 3-4, we use the compact SVD on the query output matrixQ=U·Σ·V ⊤ . HereQ∈R l×n ,U∈R l×h ,Σ∈R h×h , andV ⊤ ∈R h×n . Note that the pointsg θ (p)lie on a sphere inR h , andU ⊤ ·W∈R h×h , henceU ⊤ ·W·g θ (p)lie on an ellipsoid inR h . From now on, it is convenient to work with the pointsX=U ⊤ ·Q; As centered-ellipsoids are equivalently defined byx ⊤ Ax=1for some positive semidefinite (symmetric) matrixA∈R h×h , this implies that we can writeA=M ⊤ ·Mfor someM(step 7 of the procedure) which will work sinceAis positive semidefinite and symmetric since the system is non-degenerate because the ellipse is rank. Importantly, to motivate step 8 of the procedure, we use Lemma H.3. Lemma H.3.W=U·M −1 ·Ofor some orthogonal matrixO. Proof.We know thatg θ (p i )lie on a sphere. The equation (x i −c) ⊤ A(x i −c) =1is equivalent to (x i −c) ⊤ M ⊤ M(x i − c) =1, which is equivalent to∥M(x i −c)∥=1. This means thatM(x i −c)lie on a sphere. BecauseM(x i −c) = M·U ⊤ ·W·g θ (p i ) , we have thatM·U ⊤ ·Wis a norm-preserving transformation on the pointsg θ (p i ). By the assumption thatg θ (p i )are not in a degenerate lower-dimensional subspace, we have thatM·U ⊤ ·W=:Ois a norm-preserving endomorphism ofR h , hence an orthogonal matrix. This directly impliesW=U·M −1 ·Oas claimed. I. Quantization and Noise I.1. Quantization Quantization is a popular strategy for decreasing a model’s memory footprint and speeding up inference. In addition to these benefits, using lower-precision number representations also effectively adds noise. As noted in Section 8.2, adding noise to the output logits could prevent our attack. A natural question that follows is, does quantization add sufficient noise to make our attack ineffective or more difficult to carry out? For a simple test, we quantize Llama-7B at both 8-bits and 4-bits, and compare our baseline attack (Section 4.1) to the default 16-bit implementation. We quantize usingbitsandbytes(Dettmers et al., 2022), which HuggingFace supports for out-of-the-box quantization of model weights and lower-precision inference (Figure 6). We observe no meaningful differences at different levels of quantization; querying each model results in recovering the same same embedding matrix dimensionhin the same number of queries. Given that 8-bit and 4-bit quantization are generally observed to not have a large impact on performance, this is perhaps an unsurprising result; any noise from quanitization does not seem to have a meaningful impact on the logits (in the context of our attack). I.2. Noise One natural defense to our attacks is to obfuscate the logits by adding noise. This will naturally induce a tradeoff between utility and vulnerability—more noise will result in less useful outputs, but increase extraction difficulty. We empirically measure this tradeoff in Figure 5(c). We consider noise added directly to the logits, that is consistent between different queries of the same prompt. To simulate this, we directly add noise to our recovered logits, and recompute the extracted embedding matrix. For GPT-2, we measure the RMSE between the true embedding matrix and the embedding matrix extracted with a specific noise level; foradaandbabbage, we measure the RMSE between the noisy extracted weights and the weights we extracted in the absence of noise. We normalize all embedding matrices (to haveℓ 2 norm 1) before measuring RMSE. 26 Stealing Part of a Production Language Model 0100020003000400050006000 Sorted Singular Values 10 4 10 2 10 0 10 2 10 4 Magnitude llama-7B llama-7B-8bit llama-7B-4bit (a).Sortedsingularvaluesfor 1024, 2048, 4096, 8192queries. 40704080409041004110 Sorted Singular Values 0 2 4 6 8 10 Difference between consecutive singular values llama-7B llama-7B-8bit llama-7B-4bit (b).Differences between consecutive sorted singular values. 10 4 10 3 10 2 10 1 10 0 Noise Scale 10 7 10 6 10 5 10 4 Embedding Matrix MSE ada babbage GPT-2 (c).RMSE of extracted embeddings at various noise variances. Figure 6. In (a, b), recovering the embedding matrix dimensionhfor Llama-7B at different levels of precision: 16-bit (default), 8-bit, and 4-bit. We observe no meaningful differences, with respect to our attack, at different levels of quantization. In (c), the RMSE between extracted embeddings as a function of the standard deviation of Gaussian noise added to the logits. 02004006008001000 Sorted Singular Values 0 200 400 600 800 1000 Singular Value Original 02004006008001000 Sorted Singular Values 0 200 400 600 800 1000 Singular Value With Spoofed Dimension Figure 7.On the left, we plot the singular values that are extracted using our attack on GPT-2 small—the estimated hidden dimension is near 768. On the right, we post-hoc extend the dimensionality of the weight matrix to 1024, as described in Section 8. This misleads the adversary into thinking the model is wider than it actually is. 27