Paper deep dive

Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

Róbert Csordás, Christopher Potts, Christopher D. Manning, Atticus Geiger

Year: 2024Venue: arXiv preprintArea: Representation AnalysisType: EmpiricalEmbeddings: 91

Models: GRU (sizes 48-1024)

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 7:57:44 PM

Summary

The paper challenges the 'Strong Linear Representation Hypothesis' (LRH) by demonstrating that small gated recurrent neural networks (RNNs) solve sequence repetition tasks using non-linear 'onion representations'—where information is encoded via magnitudes rather than linear directions. While larger RNNs eventually develop linear subspaces consistent with the LRH, the authors provide evidence that interpretability research must account for non-linear mechanisms.

Entities (4)

Gated Recurrent Units · model-architecture · 99%Linear Representation Hypothesis · theory · 98%Distributed Alignment Search · methodology · 97%Onion Representations · representation-mechanism · 95%

Relation Signals (3)

Distributed Alignment Search → evaluates → Linear Representation Hypothesis

confidence 95% · To test this hypothesis, we employed a variant of distributed alignment search (DAS)... This revealed that the larger GRUs do in fact have linear subspaces

Onion Representations → contradicts → Linear Representation Hypothesis

confidence 94% · In this paper, we present a counterexample to this strong LRH: when trained to repeat an input token sequence, gated recurrent neural networks (RNNs) learn to represent the token at each position with a particular order of magnitude

Gated Recurrent Units → utilizes → Onion Representations

confidence 92% · GRUs solve a repeat task by learning a scaling factor corresponding to each sequence position, leading to layered onion-like representations.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The Linear Representation Hypothesis (LRH) states that neural networks learn to encode concepts as directions in activation space, and a strong version of the LRH states that models learn only such encodings. In this paper, we present a counterexample to this strong LRH: when trained to repeat an input token sequence, gated recurrent neural networks (RNNs) learn to represent the token at each position with a particular order of magnitude, rather than a direction. These representations have layered features that are impossible to locate in distinct linear subspaces. To show this, we train interventions to predict and manipulate tokens by learning the scaling factor corresponding to each sequence position. These interventions indicate that the smallest RNNs find only this magnitude-based solution, while larger RNNs have linear representations. These findings strongly indicate that interpretability research should not be confined by the LRH.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

90,561 characters extracted from source content.

Expand or collapse full text

Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations Róbert Csordás Stanford University rcsordas@stanford.edu &Christopher Potts Stanford University cgpotts@stanford.edu D. Manning Stanford University manning@stanford.edu &Atticus Geiger Pr(Ai)2R Group atticusg@gmail.com Abstract The Linear Representation Hypothesis (LRH) states that neural networks learn to encode concepts as directions in activation space, and a strong version of the LRH states that models learn only such encodings. In this paper, we present a counterexample to this strong LRH: when trained to repeat an input token sequence, gated recurrent neural networks (RNNs) learn to represent the token at each position with a particular order of magnitude, rather than a direction. These representations have layered features that are impossible to locate in distinct linear subspaces. To show this, we train interventions to predict and manipulate tokens by learning the scaling factor corresponding to each sequence position. These interventions indicate that the smallest RNNs find only this magnitude-based solution, while larger RNNs have linear representations. These findings strongly indicate that interpretability research should not be confined by the LRH. 111Our code is public: https://github.com/robertcsordas/onion_representations Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations Róbert Csordás Stanford University rcsordas@stanford.edu Christopher Potts Stanford University cgpotts@stanford.edu Christopher D. Manning Stanford University manning@stanford.edu Atticus Geiger Pr(Ai)2R Group atticusg@gmail.com 1 Introduction It has long been observed that neural networks encode concepts as linear directions in their representations Smolensky (1986), and much recent work has articulated and explored this insight as the Linear Representation Hypothesis (LRH; Elhage et al. 2022; Park et al. 2023; Guerner et al. 2023; Nanda et al. 2023; Olah 2024). A strong interpretation of the LRH says that such linear encodings are entirely sufficient for a mechanistic analysis of a deep learning model Smith (2024). In this paper, we present a counterexample to the Strong LRH by showing that recurrent neural networks with Gated Recurrent Units (GRUs; Cho et al. 2014) learn to represent the token at each position using magnitude rather than direction when solving a simple repeat task (memorizing and generating a sequence of tokens). This leads to a set of layered features that are impossible to locate in distinct linear subspaces. We refer to the resulting hidden states as ‘onion representations’ to evoke how sequence position can be identified by iteratively peeling off these magnitude changes from the positions before it (Figure 1). In our experiments, this is the only solution found by the smallest networks (hidden size 48, 64); the larger networks (128, 512, 1024) learn to store input tokens in position-specific linear subspaces, consistent with the LRH, though we find these linear representations are compatible with onion-based mechanisms as well. (a) Token Embeddings (b) Onion for (a,b,c,d)(a,b,c,d)( a , b , c , d ). (c) Onion for (a,⋅,c,d)⋅(a,·,c,d)( a , ⋅ , c , d ). (d) Onion for (a,c,c,d)(a,c,c,d)( a , c , c , d ). Figure 1: We find that GRUs solve a repeat task by learning a scaling factor corresponding to each sequence position, leading to layered onion-like representations. In this simplified illustration, the learned token embeddings (a) are rescaled to have magnitudes proportional to their sequence positions (b). To change an element of the sequence, remove (c) and replace (d) the token embedding at the given positional magnitude. The layered nature of the representations makes them non-linear; any direction will cross-cut multiple layers of the onion. We made this surprising finding in a hypothesis-driven fashion. Our Hypothesis 1 was that GRUs would store each token in a linear subspace. To test this hypothesis, we employed a variant of distributed alignment search (DAS; Geiger et al. 2024b; Wu et al. 2023) that uses a Gumbel softmax to select dimensions for intervention. This revealed that the larger GRUs do in fact have linear subspaces for each position, but we found no evidence of this for the smaller ones (section 5). This led to Hypothesis 2: GRUs learn to represent input bigrams in linear subspaces. A DAS-based analysis supports this for the medium-sized models but not for the smallest ones (section 6). This left the task success of the smallest models to be explained. For the smallest models, we observed that the update gates of the GRUs got gradually lower as the sequence progressed. This led to Hypothesis 3: onion representations. To evaluate this hypothesis, we learned interventions on the hidden vector encoding a sequence of tokens that replaces token A with token B at position j. The intervention adds the scaled difference of learned embeddings for A and B, where the scaling factor is determined by the position j with learned linear and exponential terms. Across positions, this intervention works with ≈90% accuracy, demonstrating the existence of layered features stored at different scales. The existence of non-linear representations is a well-formed theoretical possibility. For example, under the framework of Geiger et al. (2024a) and Huang et al. (2024), any bijective function can be used to featurize a hidden vector, and interventions can be performed on these potentially non-linear features. However, the typical causal analysis of a neural networks involves only interventions on linear representations (see Section 2 for a brief review of such methods). We hope that our counterexample to the strong version of the LRH spurs researchers to consider methods that fall outside of this class, so that we do not overlook concepts and mechanisms that our models have learned. 2 Related Work The Linear Representation Hypothesis Much early work on ‘word vectors’ was guided by the idea that linear operations on vectors could identify meaningful structure (Mikolov et al., 2013; Arora et al., 2016; Levy and Goldberg, 2014). More recently, Elhage et al. (2022) articulated the Linear Representation Hypothesis (LRH), which says that (1) features are represented as directions in vector space and (2) features are one-dimensional (see also Elhage et al. 2022; Park et al. 2023; Guerner et al. 2023; Nanda et al. 2023). Engels et al. 2024 challenged (2) by showing some features are irreducibly multi-dimensional. Olah (2024) subsequently argued that (1) is the more significant aspect of the hypothesis, and it is the one that we focus on here. Smith (2024) adds important nuance to the LRH by distinguishing a weak version (some concepts are linearly encoded) from a strong one (all concepts are linearly encoded). Our concern is with the strong form; there is ample evidence that linear encoding is possible, but our example shows that other encodings are possible. In onion representations, multiple concepts can be represented in a linear subspace by storing each concept at a different order of magnitude, i.e., a ‘layer’ of the onion, and any direction will cross-cut multiple layers of the onion. Intervention-based Methods Recent years have seen an outpouring of new methods in which interventions are performed on linear representations, e.g., entire vectors Vig et al. (2020); Geiger et al. (2020); Finlayson et al. (2021); Wang et al. (2023), individual dimensions of weights Csordás et al. (2021) and hidden vectors Giulianelli et al. (2018); De Cao et al. (2020); Davies et al. (2023), linear subspaces Ravfogel et al. (2020); Geiger et al. (2024b); Belrose et al. (2023), or linear features from a sparse dictionary Marks et al. (2024); Makelov et al. (2024). These methods have provided deep insights into how neural networks operate. However, the vast and varied space of non-linear representations is woefully underexplored in a causal setting. RNNs Recurrent Neural Networks (RNNs) were among the first neural architectures used to process sequential data (Elman, 1990, 1991). Many variants arose to help networks successfully store and manage information across long sequences, including LSTMs (Hochreiter and Schmidhuber, 1997) and GRUs (Cho et al., 2014). Bidirectional LSTMs provided the basis for one of the first large-scale pretraining efforts (ELMo; Peters et al. 2018). With the rise of Transformer-based models (Vaswani et al., 2017), RNNs fell out of favor somewhat, but the arrival of structured state-space models Gu et al. (2021b, a); Gu and Dao (2023); Dao and Gu (2024) has brought RNNs back into the spotlight, since such models seek to replace the Transformer’s potentially costly attention mechanisms with recurrent connections. We chose GRUs for our studies, with an eye towards better understanding structured state space models as well. N=4848N=48N = 48 N=6464N=64N = 64 N=128128N=128N = 128 N=256256N=256N = 256 N=512512N=512N = 512 N=10241024N=1024N = 1024 Exact-Match Accuracy 0.95 ±plus-or-minus± 0.01 0.97 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 Table 1: Exact-match accuracy (mean of 5 runs; ±plus-or-minus± 1 s.d.) for GRUs of different sizes trained on the repeat task. 3 Models In this paper, we focus on how RNNs solve the repeat task. As noted in section 2, this question has taken on renewed importance with the development of structured state-space models that depend on recurrent computations and are meant to provide efficient alternatives to transformers. Define an RNN as t=f⁢(t−1,t)subscriptsubscript1subscript h_t=f( h_t-1, x_t)italic_hitalic_t = f ( italic_hitalic_t - 1 , italic_xitalic_t ), 0=0subscript00 h_0=0italic_h0 = 0, where f⁢(⋅,⋅)⋅f(·,·)f ( ⋅ , ⋅ ) is the state update function, t∈1,…,T1…t∈\1,…,T\t ∈ 1 , … , T is the current timestep, t∈ℝNsubscriptsuperscriptℝ x_t ^Nitalic_xitalic_t ∈ blackboard_RN is the current input, and t∈ℝNsubscriptsuperscriptℝ h_t ^Nitalic_hitalic_t ∈ blackboard_RN is the state after receiving the input tsubscript x_titalic_xitalic_t. The output of the model is t=g⁢(t)subscriptsubscript y_t=g( h_t)italic_yitalic_t = g ( italic_hitalic_t ). Vectorized inputs tsubscript x_titalic_xitalic_t are obtained with a learned embedding ∈ℝNS×Nsuperscriptℝsubscript E ^N_S× Nitalic_E ∈ blackboard_RNitalic_S × N, using the indexing operator t=⁢[it]subscriptdelimited-[]subscript x_t= E[i_t]italic_xitalic_t = italic_E [ iitalic_t ], where it∈1,…,NSsubscript1…subscripti_t∈\1,…,N_S\iitalic_t ∈ 1 , … , Nitalic_S is the index of the token at timestep t. In our experiments, we use GRU cells over the more widely-used LSTM cells because they have a single state to intervene on, as opposed to the two states of the LSTM. GRU-based RNNs defined as: tsubscript z_titalic_zitalic_t =σ⁢(z⁢t+z⁢t+z)absentsubscriptsubscriptsubscriptsubscriptsubscript =σ ( W_z x_t+ U_z h_t+% b_z )= σ ( italic_Witalic_z italic_xitalic_t + italic_Uitalic_z italic_hitalic_t + italic_bitalic_z ) (1) tsubscript r_titalic_ritalic_t =σ⁢(r⁢t+r⁢t+r)absentsubscriptsubscriptsubscriptsubscriptsubscript =σ ( W_r x_t+ U_r h_t+% b_r )= σ ( italic_Witalic_r italic_xitalic_t + italic_Uitalic_r italic_hitalic_t + italic_bitalic_r ) (2) tsubscript u_titalic_uitalic_t =tanh⁡(h⁢t+h⁢(t⊙t)+h)absentsubscriptℎsubscriptsubscriptℎdirect-productsubscriptsubscriptsubscriptℎ = ( W_h x_t+ U_h( r_t% h_t)+ b_h )= tanh ( italic_Witalic_h italic_xitalic_t + italic_Uitalic_h ( italic_ritalic_t ⊙ italic_hitalic_t ) + italic_bitalic_h ) (3) tsubscript h_titalic_hitalic_t =(1−t)⊙t−1+t⊙tabsentdirect-product1subscriptsubscript1direct-productsubscriptsubscript =(1- z_t) h_t-1+ z_t u_t= ( 1 - italic_zitalic_t ) ⊙ italic_hitalic_t - 1 + italic_zitalic_t ⊙ italic_uitalic_t (4) For output generation, we use g⁢(t)=softmax⁢(t⁢o+o)subscriptsoftmaxsubscriptsubscriptsubscriptg( h_t)=softmax( h_t W_o+ b_o)g ( italic_hitalic_t ) = softmax ( italic_hitalic_t italic_Witalic_o + italic_bitalic_o ). The learned parameters are weights ∗,∗∈ℝN×Nsubscriptsubscriptsuperscriptℝ W_*, U_* ^N× Nitalic_W∗ , italic_U∗ ∈ blackboard_RN × N, and biases ∗∈ℝNsubscriptsuperscriptℝ b_* ^Nitalic_b∗ ∈ blackboard_RN. We will investigate how the final hidden state Lsubscript h_Litalic_hitalic_L of a GRU represents an input token sequence =i1,i2,…⁢iLsubscript1subscript2…subscripti=i_1,i_2,… i_Li = i1 , i2 , … iitalic_L. The final state is a bottle-neck between the input token sequence and the output. 4 Repeat Task Experiments Our over-arching research question is how different models learn to represent abstract concepts. The repeat task is an appealingly simple setting in which to explore this question. In this task, the network is presented with a sequence of random tokens =i1,i2,…,iLsubscript1subscript2…subscripti=i_1,i_2,…,i_Li = i1 , i2 , … , iitalic_L, where each ijsubscripti_jiitalic_j is chosen with replacement from a set of symbols NSsubscriptN_SNitalic_S and the length L is chosen at random from 1⁢…⁢Lmax1…subscriptmax\1… L_max\ 1 … Lmax . This is followed by a special token, iL+1=‘S’subscript1‘S’i_L+1=`S'iitalic_L + 1 = ‘S’, that indicates the start of the repeat phase. The task is to repeat the input sequence: yL+1+j=ijsubscript1subscripty_L+1+j=i_jyitalic_L + 1 + j = iitalic_j. The variables in this task will represent positions in the sequence and take on token values. As a preliminary step, we evaluate RNN models on the repeat task. The core finding is that all of the models solve the task. This sets us up to explore our core interpretability hypotheses in sections 5–7. 4.1 Setup For our experiments, we generate 1M random sequences of the repeat task. The maximum sequence length is Lmax=9subscriptmax9L_max=9Lmax = 9, and the number of possible symbols is NS=30subscript30N_S=30Nitalic_S = 30. For testing, we generate an additional 5K examples using the same procedure, ensuring that they are disjoint at the sequence level from those included in the train set. We use the same model weights during both the input and decoding phases. During the input phase, we ignore the model’s outputs. No loss is applied to these positions. We use an autoregressive decoding phase: the model receives its previous output as input in the next step. We investigate multiple hidden state sizes, from N=4848N=48N = 48 to N=10241024N=1024N = 1024. We train using a batch size of 256, up to 40K iterations, which is sufficient for each model variants to converge. We use an AdamW optimizer with a learning rate of 10−3superscript10310^-310- 3 and a weight decay of 0.10.10.10.1. 4.2 Results Table 1 reports on model performance at solving the repeat task. It seems fair to say that all the models solve the task; only the smallest model comes in shy of a perfect score, but it is at 95%. Overall, these results provide a solid basis for asking how the models manage to do this. This is the question we take up for the remainder of the paper. Intervention N=4848N=48N = 48 N=6464N=64N = 64 N=128128N=128N = 128 N=256256N=256N = 256 N=512512N=512N = 512 N=10241024N=1024N = 1024 Linear Unigram 0.00 ±plus-or-minus± 0.00 0.00 ±plus-or-minus± 0.00 0.01 ±plus-or-minus± 0.00 0.18 ±plus-or-minus± 0.03 0.91 ±plus-or-minus± 0.08 1.00 ±plus-or-minus± 0.00 Linear Bigram 0.01 ±plus-or-minus± 0.00 0.01 ±plus-or-minus± 0.00 0.54 ±plus-or-minus± 0.05 0.97 ±plus-or-minus± 0.05 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 Onion Unigram 0.83 ±plus-or-minus± 0.03 0.87 ±plus-or-minus± 0.03 0.89 ±plus-or-minus± 0.04 0.91 ±plus-or-minus± 0.08 0.95 ±plus-or-minus± 0.01 0.94 ±plus-or-minus± 0.04 Table 2: Intervention accuracy (mean of 5 runs; ±plus-or-minus± 1 s.d.) for GRUs of different sizes trained on the repeat task. 5 Hypothesis 1: Unigram Variables Intuitively, to solve the repeat task, the token at each position will have a different feature in the state vector Lsubscript h_Litalic_hitalic_L (the boundary between the input and output phrases). In line with the LRH, we hypothesize these features will be linear subspaces. 5.1 Interchange Intervention Data In causal abstraction analysis Geiger et al. (2021), interchange interventions are used to determine the content of a representation by fixing it to the counterfactual value it would have taken on if a different input were provided. These operations require datasets of counterfactuals. To create such examples, we begin with a random sequence y of length L consisting of elements of our vocabulary. We then sample a set of positions I⊆1,…,L1…I \1,…,L\I ⊆ 1 , … , L , where each position k has a 50% chance of being selected. To create the base b, we copy y and then replace each bksubscriptb_kbitalic_k with a random token, for k∈Ik∈ Ik ∈ I. To create the source s, we copy y and then replace each sjsubscripts_jsitalic_j with a random token, for j∉Ij∉ Ij ∉ I. Here is a simple example with I=1,313I=\1,3\I = 1 , 3 : y =b d a cabsentb d a c = b d a c= b d a c b =X d Y cabsentX d Y c = X d Y c= X d Y c s =b 4 a 1absentb 4 a 1 = b 4 a 1= b 4 a 1 Our core question is whether we can replace representations obtained from processing b with those obtained from processing s in a way that leads the model to predict y in the decoding phase. 5.2 Method: Interchange Interventions on Unigram Subspaces Our goal is to localize each position k in the input token sequence to a separate linear subspaces SksubscriptS_kSitalic_k of Lsubscript h_Litalic_hitalic_L. We will evaluate our success using interchange interventions. For each position in k∈Ik∈ Ik ∈ I, we replace the subspace SksubscriptS_kSitalic_k in the hidden representation Lbsubscriptsuperscriptb h^b_Litalic_hbitalic_L for base input sequence b with the value it takes in Lssubscriptsuperscripts h^s_Litalic_hsitalic_L for source input sequence s. The resulting output sequence should exactly match y. If we succeed, we have shown that the network has linear representations for each position in a sequence. There is no reason to assume that the subspaces will be axis-aligned. Thus, we use Distributed Alignment Search (DAS) and train a rotation matrix ∈ℝN×Nsuperscriptℝ R ^N× Nitalic_R ∈ blackboard_RN × N to map hitalic_h into a new rotated space ¯ hover¯ start_ARG italic_h end_ARG. However, a remaining difficulty is to determine which dimensions in the rotated space belong to which position. The size of individual subspaces may differ: for example, the first input of a repeated sequence, b1subscript1b_1b1, is always present, and the probability of successive inputs decreases due to the random length of the input sequences. Thus, the network might decide to allocate a larger subspace to the more important variables that are always present, maximizing the probability of correct decoding for popular sequence elements. To solve this problem, we learn an assignment matrix ∈0,1N×(L+1)superscript011 A∈\0,1\^N×(L+1)italic_A ∈ 0 , 1 N × ( L + 1 ) that assigns dimensions of the axis-aligned representation ¯ hover¯ start_ARG italic_h end_ARG with at most one sequence position. Allowing some dimensions to be unassigned provides the possibility for the network to store other information that is outside of these positions, such as the input length. We can learn this assignment matrix by defining a soft version of it ^∈ℝN×(L+1)^superscriptℝ1 A ^N×(L+1)over start_ARG italic_A end_ARG ∈ blackboard_RN × ( L + 1 ), and taking the hard gumbel-softmax (Jang et al., 2017; Maddison et al., 2017) with straight-through estimator (Hinton, 2012; Bengio et al., 2013) over its columns for each row (r∈1⁢…⁢N1…r∈\1… N\r ∈ 1 … N ) independently: ⁢[r]delimited-[] A[r]italic_A [ r ] =gumbel_softmax⁢(^⁢[r])absentgumbel_softmax^delimited-[] =gumbel\_softmax( A[r])= gumbel_softmax ( over start_ARG italic_A end_ARG [ r ] ) (5) For intervening on the position k∈ℕk ∈ blackboard_N, we replace dimensions of the rotated state ¯ hover¯ start_ARG italic_h end_ARG, that are 1111 in ⁢[⋅,v]⋅ A[·,v]italic_A [ ⋅ , v ]. Specifically, intervention ^superscript h^bover start_ARG italic_h end_ARGb is defined: ¯superscript¯ h^bover¯ start_ARG italic_h end_ARGb =⁢absentsuperscript = R h^b= italic_R italic_hbold_b (6) ¯superscript¯ h^sover¯ start_ARG italic_h end_ARGs =⁢absentsuperscript = R h^s= italic_R italic_hbold_s (7) ¯^superscript^¯ h^bover start_ARG over¯ start_ARG italic_h end_ARG end_ARGb =⁢[⋅,v]⊙¯+(1−⁢[⋅,v])⊙¯absentdirect-product⋅superscript¯direct-product1⋅superscript¯ = A[·,v] h^s+(1- A[% ·,v]) h^b= italic_A [ ⋅ , v ] ⊙ over¯ start_ARG italic_h end_ARGs + ( 1 - italic_A [ ⋅ , v ] ) ⊙ over¯ start_ARG italic_h end_ARGb (8) ^superscript h^bover start_ARG italic_h end_ARGb =⊺⁢¯^absentsuperscript⊺superscript^¯ = R h^b= italic_R⊺ over start_ARG over¯ start_ARG italic_h end_ARG end_ARGb (9) When learning the rotation matrix Ritalic_R and assignment matrix Aitalic_A, we freeze the parameters of the already trained GRU network. We perform the intervention on the final state of the GRU, after encoding the input sequences, and use the original GRU to decode the output sequence y^^y yover start_ARG y end_ARG from the intervened state ^Lsubscriptsuperscript h^b_Lover start_ARG italic_h end_ARGbitalic_L. We update Ritalic_R and Aitalic_A by back-propogating with respect to the cross entropy loss between the output sequence y^^y yover start_ARG y end_ARG and the expected output sequence after intervention y. 5.3 Results We use the same training set as the base model to train the intervention model, and we use the same validation set to evaluate it. The first row of Table 2 shows the accuracy of the unigram intervention. It works well for “big” models, with N≥512512N≥ 512N ≥ 512. In these cases, we can confidentially conclude that the model has a separate linear subspace for each position in the sequence. 5.4 Discussion The above results suggest that the model prefers to store each input element in a different subspace if there is “enough space” in its representations relative to the task. However, Hypothesis 1 seems to be incorrect for autoregressive decoders where N<512512N<512N < 512. Since these models do solve our task, we need to find an alternative explanation for how they succeed. This leads us to Hypothesis 2. 6 Hypothesis 2: Bigram Variables Our second hypothesis is a minor variant of Hypothesis 1. Here, we posit that, instead of representing variables for unigrams, the model instead stores tuples of inputs (it,it+1)subscriptsubscript1(i_t,i_t+1)( iitalic_t , iitalic_t + 1 ) we call bigram variables. 6.1 Intervention Data We create counterfactual pairs using the same method as we used for Hypothesis 1 (section 5.1). In this case, each token itsubscripti_tiitalic_t affects two bigram variables (if present). Thus, the subspace replacement intervention must be performed on both of these variables. This also means that, for each k∈Ik∈ Ik ∈ I, the tokens sk−1subscript1s_k-1sitalic_k - 1 and sk+1subscript1s_k+1sitalic_k + 1 in the source sequence input must match bt−1subscript1b_t-1bitalic_t - 1 and bt+1subscript1b_t+1bitalic_t + 1 in the base sequence, because the bigram at position t−11t-1t - 1 depends on (it−1,it)subscript1subscript(i_t-1,i_t)( iitalic_t - 1 , iitalic_t ) and the bigram at t depends on (it,it+1)subscriptsubscript1(i_t,i_t+1)( iitalic_t , iitalic_t + 1 ). 6.2 Method: Interchange Interventions on Bigram Subspaces For a sequence of length L, there are L−11L-1L - 1 bigram variables. To try to identify these, we use the same interchange intervention method described in section 5.2. Because targeting a single position in the base input sequence requires replacing two bigram variables, we intervene on only a single token at a time. Otherwise, the randomized sequence could be too close to the original, and most of the subspaces would be replaced at once, thereby artificially simplifying the task. 6.3 Results We show the effectiveness of bigram interventions in the middle row of Table 2. The intervention is successful on most sizes, but fails for the smallest models (N≤6464N≤ 64N ≤ 64). 6.4 Discussion We hypothesize that the models prefer to learn bigram representations because of their benefits for autoregressive input: the current input can be compared to each of the stored tuples, and the output can be generated from the second element of the tuple. This alone would be enough to repeat all sequences which have no repeated tokens. Because our models solve the task with repeat tokens, an additional mechanism must be involved. Regardless, bigrams could provide a powerful representation that is advantageous for the model. Two additional remarks are in order. First, successful unigram interventions entail successful bigram interventions; a full argument is given in Appendix E.1. Second, one might worry that our negative results for smaller models trace to limitations of DAS on the small models. Appendix E.2 addresses this by showing DAS succeeding on a non-autoregressive control model (N≤6464N≤ 64N ≤ 64) that solves the copy task. This alleviates the concern, suggesting that the small autoregressive model does not implement the bigram solution and highlighting the role of autoregression in the bigram solution. However, we still do not have an explanation for how the smallest models (N≤6464N≤ 64N ≤ 64) manages to solve the repeat task; Hypotheses 1 and 2 are unsupported as explanations for this model. This in turn leads us to Hypothesis 3. (a) The first 64 channels of GRU with N=10241024N=1024N = 1024. The model learns to store variables in different, axis-aligned subspaces. Gates close sharply, freezing individual subspaces at different times. For all channels, please refer to Figure 5 in the Appendix. (b) GRU with N=6464N=64N = 64 learns a “onion representation”, using different scales of the same numbers to represent the variables. The gates close gradually and synchronously in the input phase, providing the exponentially decaying scaling needed to represent different positions in the sequence. Figure 2: The input gate tsubscript z_titalic_zitalic_t in GRUs learning different representations Yellow is open; dark blue is closed; y-axis is the channel; x axis is the position. Both models use input gates to let in different proportions of each dimension across the sequence in order to store the positions of the input tokens. The large model (left) sharply turns off individual channels to mark position; in contrast, the small model (right) gradually turns off all channels. 7 Hypothesis 3: Onion Representations In an effort to better understand how the smallest GRUs solve the repeat task, we inspected the gate values tsubscript z_titalic_zitalic_t as defined in equation 1 from the GRU definition (section 3). Figure 2a visualizes the first 64 input gates for the N=10241024N=1024N = 1024 model (Appendix figure 5 is a larger diagram with all the gates). The x-axis is the sequence (temporal dimension) and the y-axis depicts the gate for each dimension. One can see that this model uses gates to store inputs by closing position-dependent channels sharply, creating a position-dependent subspace for each input. (This gating pattern is consistent across all inputs.) Figure 2b shows all the gates for the N=6464N=64N = 64 model. Here, the picture looks substantially different. This model gradually closes its gates simultaneously, suggesting that the network might be using this gate to encode token positions. This led us to Hypothesis 3: RNNs learn to encode each position in a sequence as a magnitude. This hypothesis relies heavily on the autoregressive nature of the GRU, the discriminative capacity of the output classifier g⁢(t)subscriptg( h_t)g ( italic_hitalic_t ), and the sequential nature of the problem. Multiple features can be stored in the same subspace, at different scales. When the GRU begins to generate tokens at timestep t=L+22t=L+2t = L + 2, if the scales st′subscriptsuperscript′s_t sitalic_t′ associated with position t′>tsuperscript′t >t′ > t are sufficiently small (st′≪stmuch-less-thansubscriptsuperscript′subscripts_t s_tsitalic_t′ ≪ sitalic_t), the output classifier yt=g⁢(t)subscriptsubscripty_t=g( h_t)yitalic_t = g ( italic_hitalic_t ) will be able to correctly decode the first input token i1subscript1i_1i1. In the following step, i1subscript1i_1i1 is fed back to the model as an input, and the model is able to remove the scaled representation corresponding to i1subscript1i_1i1 from tsubscript h_titalic_hitalic_t, obtaining t+1subscript1 h_t+1italic_hitalic_t + 1. In this new representation, the input with the next largest scale, i2subscript2i_2i2, will be dominant and will be decoded in the next step. This can be repeated to store a potentially long sequence in the same subspace, limited by the numerical precision. We call these ‘onion representations’ to invoke peeling back layers corresponding to sequence positions. Hypothesis 3 falls outside of the LRH. In linear representations, tokens are directions and each position has its own subspace. All positions are independently accessible; tokens can be read-out and manipulated given the right target subspace. Onion representations have very different characteristics. First, tokens have the same direction regardless of which position they are stored in; the magnitude of the token embedding determines the position rather than its direction. As a result, if multiple positions contain the same token, the same direction will be added twice with different scaling factors (see figure 1d where the token c occurs in positions 2 and 3). Second, because the memory is the sum of the scaled token embeddings, it is impossible to isolate the position associated with a given scale. Only the token with the most dominant scale can be extracted at a given time, by matching it to a dictionary of possible token directions. This is done by the final classifier for our GRUs. The autoregressive feedback for GRUs in effect peels off each layer, clearing access to the next variable. Appendix F provides a toy implementation of the onion solution to elucidate the underlying concepts. 7.1 Intervention Data For the causal analysis of onion representations, we do not use interchange interventions. Instead, we learn an embedding matrix for each token that encodes how the model represents that token in its hidden state vector. To replace a token in a sequence i1⁢…⁢iLsubscript1…subscripti_1… i_Li1 … iitalic_L, we add the difference of the embeddings for a new i^jsubscript i_jover start_ARG i end_ARGj and old ijsubscripti_jiitalic_j token scaled according to the target position j. Our goal is to intervene upon the hidden representation ^Lsubscript h_Lover start_ARG italic_h end_ARGL so that the sequence decoded is i1⁢…⁢i^j⁢…⁢iLsubscript1…subscript^…subscripti_1… i_j… i_Li1 … over start_ARG i end_ARGj … iitalic_L. We randomly sample i^jsubscript i_jover start_ARG i end_ARGj and use inputs from the GRU training data. Figure 3: The intervention described by Equations 10–13 where the input sequence is (a,b,c,d)(a,b,c,d)( a , b , c , d ) and the intervention is to fix the second position to be the token c. 7.2 Method: Onion Interventions To replace token ijsubscripti_jiitalic_j with token i^jsubscript i_jover start_ARG i end_ARGj, we add the difference of the corresponding token embeddings scaled by a factor determined by the position j. We parameterize this as: xitalic_x =⁢[ij]absentdelimited-[]subscript = E[i_j]= italic_E [ iitalic_j ] (10) ^ xover start_ARG italic_x end_ARG =⁢[i^j]absentdelimited-[]subscript = E[ i_j]= italic_E [ over start_ARG i end_ARGj ] (11) sitalic_s =⁢j+⁢j+absentsuperscript = g γ^j+ βj+ b= italic_g italic_γitalic_j + italic_β j + italic_b (12) ′superscript′ h italic_h′ =^+⊙(^−)absent^direct-product = h+ s ( x- x )= over start_ARG italic_h end_ARG + italic_s ⊙ ( over start_ARG italic_x end_ARG - italic_x ) (13) where ∈ℝNS×Nsuperscriptℝsubscript E ^N_S× Nitalic_E ∈ blackboard_RNitalic_S × N is the embedding for the tokens (distinct from the the GRU input embedding, learned from scratch for the intervention), and ,,,∈ℝNsuperscriptℝ g, γ, β, b ^Nitalic_g , italic_γ , italic_β , italic_b ∈ blackboard_RN are learned scaling parameters. Intuitively, sitalic_s is the scale used for the token in position j. Its main component is the exponential term γitalic_γ. In order to replace the token in the sequence, compute the difference of their embeddings, and scale them to the scale corresponding to the given position. Different channels in the state ∈ℝNsuperscriptℝ h ^Nitalic_h ∈ blackboard_RN might have different scales. Figure 3 depicts an example intervention, extending figure 1. 7.3 Results The last row of Table 2 shows that our onion intervention achieves significantly better accuracy on the small models compared to the alternative unigram and bigram interventions. For example, for N=6464N=64N = 64, the onion intervention achieves 87%percent8787\%87 % accuracy compared to the 1%percent11\%1 % of the bigram intervention. As a control, if we fix =11 γ=1italic_γ = 1 and =11 β=1italic_β = 1, we only reach 21%percent2121\%21 % accuracy. Figure 4: Accuracy of different probes on the final representation Lsubscript h_Litalic_hitalic_L of GRUs with N=6464N=64N = 64 and autoregressive input (mean of 5 runs; ±plus-or-minus± 1 s.d.). Only the probes that use autoregressive denoising can successfully decode the sequence. 7.4 Discussion Why do GRUs learn onion representations? In order to distinguish NSsubscriptN_SNitalic_S tokens stored in LmaxsubscriptmaxL_maxLmax possible positions, the model needs to be able to distinguish between NS×LmaxsubscriptsubscriptmaxN_S× L_maxNitalic_S × Lmax different directions in the feature space. In our experiments this is 300 possible directions, stored in a 64-dimensional vector space. In contrast, for onion representations, they only have to distinguish between NS=30subscript30N_S=30Nitalic_S = 30 directions at different orders of magnitude. Onion representations require unpeeling via autoregression. We train a variety of probes to decode the final representation Lsubscript h_Litalic_hitalic_L after encoding the input sequence of GRUs with N=6464N=64N = 64, which learn onion representation. We show our results in figure 4. The linear and MLP probes predict the entire sequence at once by mapping the hidden vector L∈ℝNsubscriptsuperscriptℝ h_L ^Nitalic_hitalic_L ∈ blackboard_RN to the logits for each timestep all∈ℝNS×Lmaxsubscriptallsuperscriptℝsubscriptsubscriptmax y_all ^N_S× L_maxitalic_yall ∈ blackboard_RNitalic_S × Lmax. The GRU Autoregressive (GRU – AR) probe is equivalent to the original model, and we use it as a check to verify that the decoding is easy to learn. The GRU – No input probe is similar, but unlike the original decoder of the model, it does not receive an autoregressive input. The probe results confirm that it’s not merely a free choice whether the decoder uses an autoregressive input or not: if an onion representation is learned during the training phase, it is impossible to decode it with a non-autoregressive decoder, contrary to the same-size models that are trained without an autoregressive input, shown in Table 4 in Appendix E.3. We also show the special probe we designed for onion representations in a similar spirit to the intervention described in section 7.2, which performs almost perfectly. More details can be found in Appendix E.3. What is the feature space of an onion representation? Together, the embeddings Eitalic_E learned for each token and the probe PP that predicts the token sequence form an encoder ℱFF that projects the hidden vector Lsubscript h_Litalic_hitalic_L into a new feature space: ℱ⁢(L)=ℱsubscriptabsent ( h_L)=F ( italic_hitalic_L ) = ⟨[(L)1],…, E[P( h_L)_1],…,⟨ italic_E [ P ( italic_hitalic_L )1 ] , … , [(L)L],L−∑j=2L[(L)j]⋅j⟩ E[P( h_L)_L], h_L- _j=2^% L E[P( h_L)_j]· s_j _E [ P ( italic_hitalic_L )L ] , italic_hitalic_L - ∑j = 2L italic_E [ P ( italic_hitalic_L )j ] ⋅ italic_sitalic_j ⟩ where the first L features are the token embeddings corresponding to the token sequence predicted by the probe and the final feature is what remains of the hidden state after those embeddings are removed. The inverse is a simple weighted sum: ℱ−1⁢()=L+1+∑j=1Lj⋅jsuperscriptℱ1subscript1superscriptsubscript1⋅subscriptsubscriptF^-1(f)=f_L+1+ _j=1^Lf_j% · s_jF- 1 ( f ) = fitalic_L + 1 + ∑j = 1L fitalic_j ⋅ italic_sitalic_j If the probe had perfect accuracy, this inverse would be perfect. Since our probe has 98% accuracy, there is a reconstruction loss when applying the featurizer and inverse featurizer (similar to sparse autoencoders, e.g., Bricken et al. 2023; Huben et al. 2024). This onion feature space is parameterized by an embedding for each token, a dynamic scaling factor, and a probe. In contrast, a single linear feature is just a vector. However, because ℱFF is (approximately) bijective, we know that ℱFF (approximately) induces an intervention algebra Geiger et al. (2024a) where each feature is modular and can be intervened upon separately from other features. Our embedding-based interventions are equivalent to onion interchange interventions. We evaluated the linear representations of large networks with interchange interventions that fixed a linear subspace to the value it would have taken on if a different token sequence were input to the model. There is a corresponding interchange intervention for onion representations. However, it turns out that these onion interchange interventions are equivalent to the scaled difference of embeddings used in our experiments (see Appendix B). Why do Onion interventions also work on large models? Surprisingly, the onion intervention works well on the big models that have linear representations of position (N≥256256N≥ 256N ≥ 256). We hypothesize that this is possible because all of the models start with gates open before closing them in a monotonic, sequential manner as the input sequence is processed. This enables the scaling-based onion intervention to simulate the actual gating pattern sufficiently closely to be able to perform the intervention well enough. The intervention cannot express arbitrarily sharp gate transitions but can compensate for them by creating an ensemble with different decay factors for the different channels. From Table 5 in the Appendix, it can be seen that the onion intervention achieves significantly worse performance on the small non-autoregressive models that use linear representations compared to the autoregressive ones. This is expected, as the onion intervention cannot express an arbitrary gating pattern that might be learned by these models. 8 Discussion and Conclusion The preceding experiments show that GRUs learn highly structured and systematic solutions to the repeat task. It should not be overlooked that two of these solutions (those based in unigram and bigram subspaces) are consistent with the general guiding intuitions behind the LRH and so help to illustrate the value of testing hypotheses in that space. However, our primary goal is to highlight the onion solution, as it falls outside the LRH. Our hope is that this spurs researchers working on mechanistic interpretability to consider a wider range of techniques. The field is rapidly converging around methods that can only find solutions consistent with the LRH, as we briefly reviewed in section 2. In this context, counterexamples to the LRH have significant empirical and theoretical value, as Olah (2024) makes clear: But if representations are not mathematically linear in the sense described above [in a definition of the LRH], it’s back to the drawing board – a huge number of questions like “how should we think about weights?” are reopened. Our counterexample is on a small network, but our task is also very simple. Very large networks solving very complex tasks may also find solutions that fall outside of the LRH. There is also a methodological lesson behind our counterexample to the LRH. Much interpretability work is guided by concerns related to AI safety. The reasoning here is that we need to deeply understand models if we are going to be able to certify them as safe and robust, and detect unsafe mechanisms and behaviors before they cause real harm. Given such goals, it is essential that we analyze these models in an unbiased and open-minded way. 9 Limitations The generality of onion representations. Onion representations are well fit for memorizing a sequence in order or in reverse order, but they cannot provide a general storage mechanism with arbitrary access patterns. It is unclear if such representations are useful in models trained on more complex real-world tasks. Using GRU models. Our exploration is limited to GRU models, which themselves might have less interest in the current Transformer-dominated state of the field. However, we suspect that the same representations are beneficial for other gated RNNs as well, such as LSTMs. Although we have a reason to believe that such representations can emerge in Transformers and state space models as well, we do not verify this hypothesis empirically. Onion representations only emerge in small models. This might indicate that onion representations are not a problem for bigger models used in practice. However, this might not be the case: LLMs, which are much bigger, operate on an enormous feature space using a relatively small residual stream. Thus, the pressure to compress representations and the potential for similar representations to emerge could be well motivated there as well. Numerical precision. The number of elements that can be stored in onion representations depends on the numerical precision of the data type used for the activations. We found that the network finds it easy to use these representations even with 16-bit floating point precision (bf16), potentially because multiple redundant channels of the state can be used as an ensemble. It remains unclear what the capacity of such representations is. 10 Acknowledgements Christopher D. Manning is a CIFAR Fellow. This research is in part supported by a grant from Open Philanthropy. References Arora et al. (2016) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to PMI-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385–399. Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. Preprint arXiv:1607.06450. Belrose et al. (2023) Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. LEACE: perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Bengio et al. (2013) Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. Preprint arXiv:1308.3432. Bricken et al. (2023) Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread. Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar. Association for Computational Linguistics. Csordás et al. (2021) Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. 2021. Are neural nets modular? Inspecting functional modularity through differentiable weight masks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Dao and Gu (2024) Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML). Davies et al. (2023) Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. 2023. Discovering variable binding circuitry with desiderata. CoRR, abs/2307.03637. De Cao et al. (2020) Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. 2020. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3243–3255, Online. Association for Computational Linguistics. Elhage et al. (2022) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Transformer Circuits Thread. Elman (1990) Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. Elman (1991) Jeffrey L. Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7(2):195–225. Engels et al. (2024) Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. 2024. Not all language model features are linear. CoRR, abs/2405.14860. Finlayson et al. (2021) Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1828–1843, Online. Association for Computational Linguistics. Geiger et al. (2024a) Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. 2024a. Causal abstraction: A theoretical foundation for mechanistic interpretability. Preprint, arXiv:2301.04709. Geiger et al. (2021) Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 9574–9586. Geiger et al. (2020) Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163–173, Online. Association for Computational Linguistics. Geiger et al. (2024b) Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. 2024b. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, 1-3 April 2024, Los Angeles, California, USA, volume 236 of Proceedings of Machine Learning Research, pages 160–187. PMLR. Giulianelli et al. (2018) Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem H. Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels, Belgium, November 1, 2018, pages 240–248. Association for Computational Linguistics. Gu and Dao (2023) Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Gu et al. (2021a) Albert Gu, Karan Goel, and Christopher Ré. 2021a. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396. Gu et al. (2021b) Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. 2021b. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Advances in Neural Information Processing Systems, volume 34, pages 572–585. Curran Associates, Inc. Guerner et al. (2023) Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, and Ryan Cotterell. 2023. A geometric notion of causal probing. CoRR, abs/2307.15054. Hinton (2012) Geoffrey Hinton. 2012. Neural networks for machine learning. Coursera, video lectures. Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. Huang et al. (2024) Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. 2024. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8669–8687, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. Huben et al. (2024) Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations. Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparametrization with gumbel-softmax. In Int. Conf. on Learning Representations (ICLR), Toulon, France. Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014. Linguistic regularities in sparse and explicit word representations. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 171–180, Ann Arbor, Michigan. Association for Computational Linguistics. Maddison et al. (2017) Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. In Int. Conf. on Learning Representations (ICLR), Toulon, France. Makelov et al. (2024) Aleksandar Makelov, George Lange, and Neel Nanda. 2024. Towards principled evaluations of sparse autoencoders for interpretability and control. Preprint, arXiv:2405.08366. Marks et al. (2024) Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. CoRR, abs/2403.19647. Mikolov et al. (2013) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia. Association for Computational Linguistics. Nanda et al. (2023) Neel Nanda, Andrew Lee, and Martin Wattenberg. 2023. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2023, Singapore, December 7, 2023, pages 16–30. Association for Computational Linguistics. Olah (2024) Christopher Olah. 2024. What is a linear representation? what is a multidimensional feature? Transformer Circuits Thread. Park et al. (2023) Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. CoRR, abs/2311.03658. Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7237–7256. Association for Computational Linguistics. Smith (2024) Lewis Smith. 2024. The ‘strong’ feature hypothesis could be wrong. LessWrong. Smolensky (1986) Paul Smolensky. 1986. Neural and conceptual interpretation of PDP models. In James L. McClelland, David E. Rumelhart, and the PDP Research Group, editors, Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Psychological and Biological Models, volume 2, pages 390–431. MIT Press. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Wang et al. (2023) Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations. Wu et al. (2023) Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah D. Goodman. 2023. Interpretability at scale: Identifying causal mechanisms in alpaca. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Appendix Appendix A Performance of the Non-Autoregressive GRUs We show the performance of all our models in Table 3, both autoregressive and those that do not receive autoregressive feedback during the decoding phase. All models solve the task well, except the smallest N=4848N=48N = 48 model without autoregressive decoding. The model finds it hard to distinguish between NS×Lmax=300subscriptsubscriptmax300N_S× L_max=300Nitalic_S × Lmax = 300 different directions in the 48-dimensional space. On the other hand, onion representations learned with autoregressive decoding work well even in these small models. Variant N=4848N=48N = 48 N=6464N=64N = 64 N=128128N=128N = 128 N=256256N=256N = 256 N=512512N=512N = 512 N=10241024N=1024N = 1024 Autoregressive 0.95 ±plus-or-minus± 0.01 0.97 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 No input 0.88 ±plus-or-minus± 0.11 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 Table 3: Exact-match accuracy (mean of 5 runs; ±plus-or-minus± 1 s.d.) for GRUs of different sizes trained on the repeat task results, with and without autoregressive input during the decoding. Appendix B Onion Interchange Interventions For position j and input token sequences a1,…,aLsubscript1…subscripta_1,…,a_La1 , … , aitalic_L and b1,…,bMsubscript1…subscriptb_1,…,b_Mb1 , … , bitalic_M, define the onion interchange intervention to be asuperscript ^afitalic_a =ℱ⁢(a)absentℱsuperscript =F( h^a)= F ( italic_hitalic_a ) bsuperscript ^bfitalic_b =ℱ⁢(b)absentℱsuperscript =F( h^b)= F ( italic_hitalic_b ) ^asuperscript h^aover start_ARG italic_h end_ARGa =ℱ−1⁢(1a,…,jb,…⁢La,L+1a)absentsuperscriptℱ1subscriptsuperscript1…subscriptsuperscript…subscriptsuperscriptsubscriptsuperscript1 =F^-1(f^a_1,…,f^b_j,% …f^a_L,f^a_L+1)= F- 1 ( fitalic_a1 , … , fitalic_bitalic_j , … fitalic_aitalic_L , fitalic_aitalic_L + 1 ) However, observe that that is simply the intervention of adding in the difference of the embeddings bjsubscriptb_jbitalic_j and ajsubscripta_jaitalic_j scaled according to the position j from Equations 10–13: ^asuperscript h^aover start_ARG italic_h end_ARGa =ℱ−1⁢(1a,…,jb,…⁢La,L+1a)absentsuperscriptℱ1subscriptsuperscript1…subscriptsuperscript…subscriptsuperscriptsubscriptsuperscript1 =F^-1(f^a_1,…,f^b_j,% …f^a_L,f^a_L+1)= F- 1 ( fitalic_a1 , … , fitalic_bitalic_j , … fitalic_aitalic_L , fitalic_aitalic_L + 1 ) =ℱ−1⁢(⁢[a1],…,⁢[bj],…⁢[aL],L+1a)absentsuperscriptℱ1delimited-[]subscript1…delimited-[]subscript…delimited-[]subscriptsubscriptsuperscript1 =F^-1( E[a_1],…, E[b_j],… % E[a_L],f^a_L+1)= F- 1 ( italic_E [ a1 ] , … , italic_E [ bitalic_j ] , … italic_E [ aitalic_L ] , fitalic_aitalic_L + 1 ) =L+1a+∑k=1Lk⋅⁢[ak]+(⁢[bj]−⁢[aj])⋅jabsentsubscriptsuperscript1subscriptsuperscript1⋅subscriptdelimited-[]subscript⋅delimited-[]subscriptdelimited-[]subscriptsubscript =f^a_L+1+Σ^L_k=1 s_k· E[a_% k]+( E[b_j]- E[a_j])· s_j= fitalic_aitalic_L + 1 + ∑Litalic_k = 1 italic_sitalic_k ⋅ italic_E [ aitalic_k ] + ( italic_E [ bitalic_j ] - italic_E [ aitalic_j ] ) ⋅ italic_sitalic_j =a+(⁢[bj]−⁢[aj])⋅jabsentsuperscript⋅delimited-[]subscriptdelimited-[]subscriptsubscript = h^a+( E[b_j]- E[a_j])· s_j= italic_hitalic_a + ( italic_E [ bitalic_j ] - italic_E [ aitalic_j ] ) ⋅ italic_sitalic_j This means the success of our intervention ^ hover start_ARG italic_h end_ARG to replace the token in a1,…,aLsubscript1…subscripta_1,…,a_La1 , … , aitalic_L at position j with a new token t entails the success of any onion interchange interventions where we patch from an input sequence b1,…,bMsubscript1…subscriptb_1,…,b_Mb1 , … , bitalic_M with bj=tsubscriptb_j=tbitalic_j = t. The learned token embeddings for onion representations creates a semantics for tokens that is externtal to the underlying model, so interchange interventions on the feature space have to do with the token embeddings rather than the representations actually created on the given source input. This is not the case for linear interchange interventions, where the value of the subspace intervention that must be performed is computed directly from the hidden representation created for the second input token sequence. Appendix C Probe Accuracy For All Models We show the accuracy of all of our probes in all models that we trained in Table 4. Linear and MLP probes work well when the learned solution respects LRH. Onion probes work well even for our smallest autoregressive models. We can see that autoregressive GRU can successfully decode all sequences, as expected, proving that relearning the decoding phase is a relatively easy learning problem. However, non-autoregressive GRUs are unable to decode sequences from onion representations. For more details, refer to sections 5–7. Decoder Variant N=4848N=48N = 48 N=6464N=64N = 64 N=128128N=128N = 128 N=256256N=256N = 256 N=512512N=512N = 512 N=10241024N=1024N = 1024 Linear Autoregressive 0.01 ±plus-or-minus± 0.00 0.01 ±plus-or-minus± 0.00 0.31 ±plus-or-minus± 0.03 0.89 ±plus-or-minus± 0.03 0.97 ±plus-or-minus± 0.00 0.99 ±plus-or-minus± 0.01 No input 0.31 ±plus-or-minus± 0.10 0.89 ±plus-or-minus± 0.05 0.98 ±plus-or-minus± 0.02 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 MLP Autoregressive 0.02 ±plus-or-minus± 0.00 0.04 ±plus-or-minus± 0.00 0.55 ±plus-or-minus± 0.04 0.98 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 No input 0.53 ±plus-or-minus± 0.25 0.95 ±plus-or-minus± 0.04 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 Onion Autoregressive 0.92 ±plus-or-minus± 0.02 0.97 ±plus-or-minus± 0.01 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 No input 0.76 ±plus-or-minus± 0.08 0.96 ±plus-or-minus± 0.01 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 GRU - autoregressive Autoregressive 0.97 ±plus-or-minus± 0.01 0.98 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 No input 0.92 ±plus-or-minus± 0.02 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 GRU - no input Autoregressive 0.10 ±plus-or-minus± 0.02 0.25 ±plus-or-minus± 0.08 0.86 ±plus-or-minus± 0.01 0.99 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 No input 0.77 ±plus-or-minus± 0.07 0.98 ±plus-or-minus± 0.01 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 Table 4: Probe accuracy (mean of 5 runs; ±plus-or-minus± 1 s.d.). Appendix D GRU Models Without Autoregressive Decoding In principle, RNN models do not need an autoregressive feedback loop during the decoding phase to be able to produce a consistent output. Given that we found that the network often relies on storing bigrams (section 6) or on onion representations (section 7), both of which benefit from autoregressive feedback, we asked what representation the models learn without such a mechanism. Thus, we changed our GRU model to receive only special PAD tokens during the decoding phase. We show the intervention accuracies in Table 5. We can see that the model is heavily based on storing unigrams, and the intervention now works down to N=10241024N=1024N = 1024. For the N=6464N=64N = 64 case, the models store bigrams. No intervention works well for the N=4848N=48N = 48 non-autoregressive model, but that model also does not perform well on the validation set (see Table 3). The model is unable to to learn onion representation at any scale, since the autoregressive input is required for that, as shown in figure 4. This experiment also confirms that our subspace intervention method introduced in section 5.2 works well even for models with N=6464N=64N = 64. Intervention N=4848N=48N = 48 N=6464N=64N = 64 N=128128N=128N = 128 N=256256N=256N = 256 N=512512N=512N = 512 N=10241024N=1024N = 1024 Linear Unigram 0.06 ±plus-or-minus± 0.07 0.37 ±plus-or-minus± 0.17 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.01 Linear Bigram 0.18 ±plus-or-minus± 0.04 0.95 ±plus-or-minus± 0.06 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 1.00 ±plus-or-minus± 0.00 Onion Unigram 0.24 ±plus-or-minus± 0.02 0.41 ±plus-or-minus± 0.04 0.76 ±plus-or-minus± 0.01 0.92 ±plus-or-minus± 0.01 0.96 ±plus-or-minus± 0.01 0.98 ±plus-or-minus± 0.00 Table 5: Intervention accuracy for GRUs without an autoregressive input in the decoding phase, with different sizes, trained on the repeat task (mean of 5 runs; ±plus-or-minus± 1 s.d.). Figure 5: All 1024 channels of the GRU gate tsubscript z_titalic_zitalic_t shown in Figure 2a. All channels follow similar patterns. Appendix E Additional Discussion of the Bigram Interventions E.1 Successful Unigram Interventions Entail Successful Bigram Interventions With bigram interventions, in addition to copying a token to the randomized sequence, we also copy its neighborhood and replace two variables. In contrast, unigram interventions only move the corrupted token and replace its corresponding variable. Thus, the unigram intervention performs a subset of movements performed by the bigram. This means that if the unigram intervention is successful, it is guaranteed that the bigram intervention will be successful as well. E.2 Verifying the Expressivity of the Subspace Intervention Obtaining negative results for the unigram intervention on smaller models (N<512512N<512N < 512) might raise the question of whether our intervention is expressive enough to capture the relatively small subspaces of these models. In order to verify this, we trained a GRU model without autoregressive input (Appendix D) during the decoding phase. By doing this, we eliminate some of the advantages provided by bigram representations. Since GRUs are RNNs, they can learn a decoding state machine without relying on seeing the output generated so far. We confirm this in Table 3. In these modified networks, unigram interventions are successful down to N=128128N=128N = 128, and the bigram intervention is successful on all scales. We show the detailed results in Table 5. E.3 The Onion-probe We designed a probe for onion representations similarly to the intervention described in section 7.2. We take the final representation after encoding the sequence, Lsubscript h_Litalic_hitalic_L, and decode yL+1=i1⁢…⁢y2⁢L=iLsubscript1subscript1…subscript2subscripty_L+1=i_1… y_2L=i_Lyitalic_L + 1 = i1 … y2 L = iitalic_L from it as follows: tsubscript s_titalic_sitalic_t =⁢t−L+⁢(t−L)+absentsuperscript = g γ^t-L+ β(t-L)+ b= italic_g italic_γitalic_t - L + italic_β ( t - L ) + italic_b (14) ytsubscript y_tyitalic_t =argmax⁢g⁢(t−1)absentargmaxsubscript1 =argmaxg( h_t-1)= argmax g ( italic_hitalic_t - 1 ) (15) tsubscript h_titalic_hitalic_t =t−1−st⁢[yt]absentsubscript1subscriptdelimited-[]subscript = h_t-1-s_t E[y_t]= italic_hitalic_t - 1 - sitalic_t italic_E [ yitalic_t ] (16) As a denoising classifier g⁢()g( h)g ( italic_h ) we use a 2 layer MLP with a layernorm (Ba et al., 2016) on its inputs g⁢()=softmax⁢(o2⁢max⁡(0,LN⁢(⁢o1+o1))+o2)softmaxsubscriptsubscript20LNsubscriptsubscript1subscriptsubscript1subscriptsubscript2g( h)=softmax ( W_o_2 (0,LN( h% W_o_1+ b_o_1))+ b_o_2 )g ( italic_h ) = softmax ( italic_Witalic_o start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT max ( 0 , LN ( italic_h italic_Witalic_o start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + italic_bitalic_o start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) ) + italic_bitalic_o start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ), where LN⁢(⋅)LN⋅LN(·)LN ( ⋅ ) is the layernorm. Layernorm is not strictly necessary, but it greatly accelerates the learning of the probe, so we decided to keep it. Appendix F Toy Model Implementing Onion Representations To show more clearly how a model can learn to represent sequence elements in different scales, we constructed a toy model that uses prototypical onion representations: stsubscript s_tsitalic_t =1,if ⁢t=1−1,if ⁢t=L+1γ⁢st−1otherwiseabsentcases1if 11if 1subscript1otherwise = cases1,&if t=1\\ -1,&if t=L+1\\ γ s_t-1&otherwise cases= start_ROW start_CELL 1 , end_CELL start_CELL if t = 1 end_CELL end_ROW start_ROW start_CELL - 1 , end_CELL start_CELL if t = L + 1 end_CELL end_ROW start_ROW start_CELL γ sitalic_t - 1 end_CELL start_CELL otherwise end_CELL end_ROW (17) 1subscript1 h_1italic_h1 =0absent0 =0= 0 (18) t+1subscript1 h_t+1italic_hitalic_t + 1 =t+st⁢tabsentsubscriptsubscriptsubscript = h_t+s_t x_t= italic_hitalic_t + sitalic_t italic_xitalic_t (19) tsubscript y_titalic_yitalic_t =softmax⁢(t⁢o+o)absentsoftmaxsubscriptsubscriptsubscript =softmax ( h_t W_o+ b_o )= softmax ( italic_hitalic_t italic_Witalic_o + italic_bitalic_o ) (20) where st∈ℝsubscriptℝs_t _t ∈ blackboard_R is a scalar state representing the current scale, γ∈ℝγ γ ∈ blackboard_R represents the difference in the scales used for different variables, and t∈ℝNsubscriptsuperscriptℝ h_t ^Nitalic_hitalic_t ∈ blackboard_RN is the vector memory. In a real RNN, both the vector memory and the current scale are part of a single state vector. In our experiments, we use a fixed γ=0.40.4γ=0.4γ = 0.4. The inputs are embedded in the same way as for our GRU model: t=⁢[it]subscriptdelimited-[]subscript x_t= E[i_t]italic_xitalic_t = italic_E [ iitalic_t ], where it∈ℕsubscriptℕi_t _t ∈ blackboard_N is the input token and ∈ℝNS×Nsuperscriptℝsubscript E ^N_S× Nitalic_E ∈ blackboard_RNitalic_S × N is the embedding matrix. The only learnable parameters of this model are the embedding matrix, Eitalic_E and the parameters of the output projection, o∈ℝN×Nsubscriptsuperscriptℝ W_o ^N× Nitalic_Witalic_o ∈ blackboard_RN × N and o∈ℝNsubscriptsuperscriptℝ b_o ^Nitalic_bitalic_o ∈ blackboard_RN. The idea behind this model is based on the fact that a linear layer followed by a softmax operation is able to ‘denoise’ the representation tsubscript h_titalic_hitalic_t. γ is chosen as <0.5absent0.5<0.5< 0.5, because in that case the contribution to the hidden state tsubscript h_titalic_hitalic_t of all future t′>tsuperscript′t >t′ > t positions will be lower than the contribution of input tsubscript x_titalic_xitalic_t. Thus, tsubscript x_titalic_xitalic_t will dominate all t′subscriptsuperscript′ h_t italic_hitalic_t′ for all t′>tsuperscript′t >t′ > t. Thus, when decoding from t′subscriptsuperscript′ h_t italic_hitalic_t′, Eq. 20, followed by the argmax used in greedy decoding, the model will always recover the first, most dominant itsubscripti_tiitalic_t that is not yet decoded from the model. Then, this token is autoregressively fed back to the next step, where it is subtracted from t′subscriptsuperscript′ h_t italic_hitalic_t′, letting the next token dominate the representation t′+1subscriptsuperscript′1 h_t +1italic_hitalic_t′ + 1. This allows storing an arbitrary sequence at different scales of the representation tsubscript h_titalic_hitalic_t. All 5 seeds of this model that we trained achieve perfect validation accuracy.

Instant research discovery