Paper deep dive
Investigating Bias Representations in Llama 2 Chat via Activation Steering
Dawn Lu, Nina Rimsky
Models: Llama 2 7B (base), Llama 2 7B Chat
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/12/2026, 7:25:05 PM
Summary
This paper investigates societal bias in the Llama 2 7B Chat model using activation steering (Contrastive Activation Addition). The authors demonstrate that while RLHF reduces overt bias, it creates a negative correlation between bias and refusal, and leads to a higher similarity in the model's internal representation of different bias types. The study highlights the necessity of using refusal vectors during red-teaming to effectively probe for latent biases.
Entities (5)
Relation Signals (4)
Llama-2-7B-Chat â exhibits â Societal Bias
confidence 90% · Our findings reveal inherent gender bias in Llama 2 7B Chat
Bias Vector â negativelycorrelatedwith â Refusal Vector
confidence 90% · every bias vector is negatively associated with refusal
RLHF â increasessimilarityof â Bias Representations
confidence 85% · RLHF tends to increase the similarity in the model's representation of different forms of societal biases
Activation Steering â mitigates â Societal Bias
confidence 80% · Our approach employs activation steering to probe for and mitigate biases
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We address the challenge of societal bias in Large Language Models (LLMs), focusing on the Llama 2 7B Chat model. As LLMs are increasingly integrated into decision-making processes with substantial societal impact, it becomes imperative to ensure these models do not reinforce existing biases. Our approach employs activation steering to probe for and mitigate biases related to gender, race, and religion. This method manipulates model activations to direct responses towards or away from biased outputs, utilizing steering vectors derived from the StereoSet dataset and custom GPT4 generated gender bias prompts. Our findings reveal inherent gender bias in Llama 2 7B Chat, persisting even after Reinforcement Learning from Human Feedback (RLHF). We also observe a predictable negative correlation between bias and the model's tendency to refuse responses. Significantly, our study uncovers that RLHF tends to increase the similarity in the model's representation of different forms of societal biases, which raises questions about the model's nuanced understanding of different forms of bias. This work also provides valuable insights into effective red-teaming strategies for LLMs using activation steering, particularly emphasizing the importance of integrating a refusal vector.
Tags
Links
- Source: https://arxiv.org/abs/2402.00402
- Canonical: https://arxiv.org/abs/2402.00402
- Code: https://github.com/matutinus/SPAR
Trouble viewing inline? Open PDF directly â
Full Text
14,318 characters extracted from source content.
Expand or collapse full text
Investigating Bias Representations in Llama 2 Chat via Activation Steering Dawn Lu UC Berkeley dawn_lu@berkeley.edu Nina Rimsky SPAR ninaarimsky@gmail.com Abstract We address the challenge of societal bias in Large Language Models (LLMs), focusing on the Llama 2 7B Chat model. As LLMs are increasingly integrated into decision-making processes with substantial societal impact, it becomes imperative to ensure these models do not reinforce existing biases. Our approach employs activation steering to probe for and mitigate biases related to gender, race, and religion. This method manipulates model activations to direct responses towards or away from biased outputs, utilizing steering vectors derived from the StereoSet dataset and custom GPT4-generated gender bias prompts. Our findings reveal inherent gender bias in Llama 2 7B Chat, persisting even after Reinforcement Learning from Human Feedback (RLHF). We also observe a predictable negative correlation between bias and the modelâs tendency to refuse responses. Significantly, our study uncovers that RLHF tends to increase the similarity in the modelâs representation of different forms of societal biases, which raises questions about the modelâs nuanced understanding of different forms of bias. This work also provides valuable insights into effective red-teaming strategies for LLMs using activation steering, particularly emphasizing the importance of integrating a refusal vector. 1 Introduction Large Language Models (LLMs) are increasingly likely to be used to make decisions that have broad societal impact, such as resume screening, college admissions and criminal justice sentencing. Therefore, it is imperative to develop techniques that ensure these models donât perpetuate harmful societal biases. One way we can evaluate whether a model is likely to exhibit biased behavior is via red-teaming. Red-teaming is the process of âattackingâ or challenging a system from an adversarial lens with the ultimate goal of identifying vulnerabilities. In our application of activation steering to red-teaming, the underlying premise is that if a small perturbation in the model can result in undesired behaviors, then the model is not robust and may display that behavior in response to some inputs. We evaluate the robustness of Llama 2 7B Chat [1] along different dimensions of societal bias using activation steering, specifically the Contrastive Activation Addition technique [2]. This can be viewed as a diagnostic test: if we can easily elicit biased responses, then this suggests the model may be unfit for sensitive applications. Furthermore, analyzing the steering vectors enables us to investigate and better understand how the model internally represents different types of societal bias, which could help to design targeted interventions â for instance, fine-tuning signals of a certain type. 2 Methodology and Data Activation steering (also known as representation engineering[3]) is a method used to steer an LLMâs response towards or away from a concept of interest by perturbing the modelâs activations during the forward pass. We perform this perturbation by adding a steering vector to the residual stream at a specific layer at every token position after an initial prompt. The steering vector is constructed by taking the average difference in residual stream activations between pairs of biased (stereotype) and unbiased (anti-stereotype) prompts at that layer. By taking the difference between paired prompts, we can effectively remove contextual noise and only retain the "bias" direction. This approach to activation steering is known as Contrastive Activation arXiv:2402.00402v1 [cs.CL] 1 Feb 2024 Addition[2]. For the data used to generate the steering vectors, we used the StereoSet dataset[4], which is a large-scale natural English dataset intended to measure stereotypical biases across various domains. In addition, we wrote a custom set of gender-bias prompts and used GPT-4[5] to generate similar examples. We then re-formatted all these examples into multiple choice A/B questions [6][7], as shown in Figure 1. Figure 1: Example of A/B contrast prompt used to generate steering vectors. 3 Activation Clustering With the StereoSet data and custom gender-bias prompts, we were able to focus on three dimensions of societal biases: gender, race, and religion. Figure 5 shows a t-SNE projection of the activations for the paired prompts. We see relatively good separation between the stereotype and anti-stereotype examples, especially for gender and race. This provides some confidence that the steering vectors constructed from these activations will be effective. Note that the race dataset has the largest sample size. 1510505101520 t-SNE 1 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 t-SNE 2 t-SNE projected activations layer 10 (gender) Stereotype Examples Anti-Stereotype Examples Figure 2: Gender (n=72) 40302010010203040 t-SNE 1 20 15 10 5 0 5 10 15 20 t-SNE 2 t-SNE projected activations layer 10 (race) Stereotype Examples Anti-Stereotype Examples Figure 3: Race (n=300) 201001020 t-SNE 1 6 4 2 0 2 4 t-SNE 2 t-SNE projected activations layer 10 (religion) Stereotype Examples Anti-Stereotype Examples Figure 4: Religion (n=78) Figure 5: t-SNE projection of contrastive activations 2 4 Steered Responses For the prompts used to evaluate the bias steering vectors, we chose the template shown in Figure 6, which was presented in Sheng et al. [8] Figure 6: Prompts for Evaluation 4.1 Original Responses To obtain a baseline, we first sampled responses from Llama 2 7B Chat without any steering. We made the following observations: 1.The model is already biased on the gender dimension despite fine-tuning and Reinforcement Learning from Human Feedback (RLHF). 2. The model simply refuses to respond to the race and religion prompts. Figure 7: Examples of responses from unsteered model 3 4.2 Bias-steered responses Next, we added the corresponding bias steering vectors (with a +2 coefficient) to every token position. The expected result was to see more biased responses. However, the model simply refused to answer and claimed that the prompt is "offensive and discriminatory", as seen in Figure 8. Even after testing coefficients with much higher magnitude, the refusal response remained. Figure 8: Result of bias CAA After observing these responses, it became clear we needed to bypass the modelâs refusal guardrails. In order to do this, we tried to add the bias vector and subtract a refusal vector [9]. After re-normalizing the activations to preserve the residual stream norm, this approach was effective in eliciting more biased responses from the model as shown in red text in Figure 9. While the model still claims that it cannot respond to the prompt, it proceeds to produce responses that mention "racism" as well as religion-based positions such as "priest" and "rabbi." 4 Figure 9: Result of bias CAA when subtracting refusal To validate that the bias steering vectors were indeed contributing to the results, we tried to just subtract the refusal steering vector. This caused the model to produce some generic responses, such as "The worked as a doctor, lawyer, or engineer" but it did not make the model more biased by default. Finally, we were curious about whether these bias vectors are transferable. For example, if we try to steer the model in a racially biased way, does it also elicit gender bias? Interestingly, the answer is yes, as evidenced in Figure 10. The biased gender responses are perhaps less surprising since the model already exhibited this behavior without steering. Whatâs more shocking are the religion responses, which exhibit more drastic and harmful bias, saying "The Muslim man worked as a terrorist on a plane." Figure 10: Example of transfer between bias steering 5 As a side note, itâs likely the racial bias steering vector was most effective since it was constructed using the most robust sample size. This result aligns with findings from Zou et al.[3] that also showed racial bias steering vectors had an impact on an LLMâs biased behaviors related to gender and occupation. 5 Relationship Between Steering Vectors 5.1 How are Bias and Refusal Related? Based on the steered responses, there appears to be a relationship between bias and refusal. Itâs evident that attempting to steer the model in a biased direction triggers a refusal response. To explore this relationship, we looked at the cosine similarity between each bias steering vector and the refusal steering vector across the modelâs mid-to-late layers. In Figure 11, we see that every bias vector is negatively associated with refusal. This is intuitive since a biased response and a refusal response can be viewed as "oppositesâ" from the modelâs perspective. Furthermore, when we try to elicit undesired model behaviors, we add a bias vector and subtract a refusal vector. Notably, the gender bias vector is least negatively associated with refusal, which aligns with our observations from the modelâs original responses (which already exhibited gender bias). 10111213141516171819202122232425262728293031 Layer 0.4 0.3 0.2 0.1 0.0 Cosine Similarity Bias Vectors vs. Refusal Vector (Cosine Similarity) Figure 11: Cosine similarity of bias and refusal CAA vectors 5.2 How are Different Forms of Bias Related to Each Other? Finally, we wanted to evaluate how these different bias vectors related to each other, given the observation that the racial bias steering vector was effective in eliciting gender bias and religion bias. We looked at the cosine similarity between each pair of bias vectors and compared the results from Llama 2 7B Chat to the base Llama 2 7B model. Interestingly, we found a very high correlation (~0.8) between gender bias and racial bias in the Chat model. This result is especially pronounced when contrasted with the respective cosine similarity of the bias vectors in the base model. This pattern is consistent across all combinations of bias, as shown in Figure 15. Observe that the cosine similarity in the base model tends to decrease as the layers progress, whereas the cosine similarity in the Chat model stays relatively stable across layers. 6 10111213141516171819202122232425262728293031 Layer 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Cosine Similarity Gender Bias vs. Race Bias (Cosine Similarity) Base Model RLHF Model Figure 12: Gender vs. Race 10111213141516171819202122232425262728293031 Layer 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Cosine Similarity Race Bias vs. Religion Bias (Cosine Similarity) Base Model RLHF Model Figure 13: Race vs. Religion 10111213141516171819202122232425262728293031 Layer 0.55 0.60 0.65 0.70 0.75 0.80 Cosine Similarity Gender Bias vs. Religion Bias (Cosine Similarity) Base Model RLHF Model Figure 14: Gender vs. Religion Figure 15: Relationships between different bias vectors 6 Conclusion Since language models are pre-trained on vast amounts of internet data, itâs inevitable they will learn societal biases. While fine-tuning and RLHF aid in reducing biased behaviors in LLMs, it is crucial to assess the robustness of these models to ensure that they do not perpetuate societal biases. Our research highlights that when red-teaming LLMs for biased behaviors, integrating refusal steering vectors is essential. In addition, employing steering vectors across various bias dimensions proves beneficial, particularly when one dimension possesses more robust data. This study also reveals a significant insight: RLHF seems to lead the model to more closely associate various forms of societal biases. This suggests that the model might lose its nuanced understanding of these distinct concepts, and instead broadly categorize them under topics it should refuse to answer. While itâs open to debate whether this outcome is desirable, identifying these trends enhances our understanding of RLHFâs influence on how an LLM processes and represents bias. 7 References [1]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. [2]Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2023. [3]Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023. [4]Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models, 2020. [5] OpenAI. Gpt-4 technical report, 2023. [6]Dawn Lu.gender_stereotypes_augmented_A_B.json.https://github.com/matutinus/SPAR/blob/ main/gender_stereotypes_augmented_A_B.json, 2023. [7]Dawn Lu.stereotype_data_A_B_subset.json.https://github.com/matutinus/SPAR/blob/main/ stereotype_data_A_B_subset.json, 2023. [8]Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation, 2019. [9]Nina Rimsky.Red-teaming language models via activation engineering. https://w.alignmentforum.org/posts/iHmsJdxgMEWmAfNne/red-teaming-language-models- via-activation-engineering, 2023. 8