Instant research discovery
Search and browse arXiv CS/AI/ML papers, enriched with AI-generated insights.
Generate novel research ideas grounded in real arXiv papers with Brainstorm.
PreviousNext
| Paper | Published | Area | Tags | Intel | Citations |
|---|---|---|---|---|---|
Search and browse arXiv CS/AI/ML papers, enriched with AI-generated insights.
Generate novel research ideas grounded in real arXiv papers with Brainstorm.
| Paper | Published | Area | Tags | Intel | Citations |
|---|---|---|---|---|---|
Search and browse arXiv CS/AI/ML papers, enriched with AI-generated insights.
Generate novel research ideas grounded in real arXiv papers with Brainstorm.
| Paper | Published | Area | Tags | Intel | Citations |
|---|---|---|---|---|---|
| Paper | Published | Area | Tags | Intel | Citations |
|---|---|---|---|---|---|
| Reward Model Ensembles Help Mitigate Overoptimization David Krueger, Robert Kirk, Thomas Coste, Usman Anwar Published: 2023-10-04Area: Alignment TrainingCitations: 190 Tags: ai-safety, alignment-training, empirical | 2023-10-04 | Alignment Training | ai-safety, alignment-training, empirical | E7 / R3 (96%) | 190 |
| Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models Dahua Lin, Linda Petzold, Qi Zhang, William Yang Wang Published: 2023-10-04Area: Adversarial RobustnessCitations: 261 Tags: adversarial-robustness, ai-safety, alignment-training, empirical | 2023-10-04 | Adversarial Robustness | adversarial-robustness, ai-safety, alignment-training, empirical | E7 / R3 (98%) | 261 |
| A Long Way to Go: Investigating Length Correlations in RLHF Greg Durrett, Jiacheng Xu, Prasann Singhal, Tanya Goyal Published: 2023-10-05Area: Alignment TrainingCitations: 223 Tags: ai-safety, alignment-training, empirical | 2023-10-05 | Alignment Training | ai-safety, alignment-training, empirical | E6 / R3 (95%) | 223 |
| SALMON: Self-Alignment with Principle-Following Reward Models Chuang Gan, David Cox, Hongxin Zhang, Qinhong Zhou Published: 2023-10-09Area: Alignment TrainingCitations: 55 Tags: ai-safety, alignment-training, empirical | 2023-10-09 | Alignment Training | ai-safety, alignment-training, empirical | E6 / R3 (97%) | 55 |
| Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation Danqi Chen, Kai Li, Mengzhou Xia, Samyak Gupta Published: 2023-10-10Area: Adversarial RobustnessCitations: 437 Tags: adversarial-robustness, ai-safety, alignment-training, empirical | 2023-10-10 | Adversarial Robustness | adversarial-robustness, ai-safety, alignment-training, empirical | E8 / R4 (96%) | 437 |
| Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations Yifei Wang, Yisen Wang, Zeming Wei Published: 2023-10-10Area: Adversarial RobustnessCitations: 423 Tags: adversarial-robustness, ai-safety, alignment-training, empirical | 2023-10-10 | Adversarial Robustness | adversarial-robustness, ai-safety, alignment-training, empirical | E6 / R4 (95%) | 423 |
| Understanding the Effects of RLHF on LLM Generalisation and Diversity Christoforos Nalmpantis, Edward Grefenstette, Eric Hambro, Ishita Mediratta Published: 2023-10-10Area: Alignment TrainingCitations: 295 Tags: ai-safety, alignment-training, empirical | 2023-10-10 | Alignment Training | ai-safety, alignment-training, empirical | E6 / R3 (94%) | 295 |
| Compositional Preference Models for Aligning LMs Dongyoung Go, Germ谩n Kruszewski, Jos Rozen, Marc Dymetman Published: 2023-10-17Area: Alignment TrainingCitations: 26 Tags: ai-safety, alignment-training, empirical | 2023-10-17 | Alignment Training | ai-safety, alignment-training, empirical | E6 / R4 (94%) | 26 |
| A General Theoretical Paradigm to Understand Learning from Human Feedback Bilal Piot, Daniele Calandriello, Daniel Guo, Mark Rowland Published: 2023-10-18Area: Alignment TrainingCitations: 894 Tags: ai-safety, alignment-training, theoretical | 2023-10-18 | Alignment Training | ai-safety, alignment-training, theoretical | E5 / R3 (97%) | 894 |
| Improving Generalization of Alignment with Human Preferences through Group Invariant Learning Haoran Huang, Qi Zhang, Rui Zheng, Shihan Dou Published: 2023-10-18Area: Alignment TrainingCitations: 24 Tags: ai-safety, alignment-training, empirical | 2023-10-18 | Alignment Training | ai-safety, alignment-training, empirical | E4 / R3 (95%) | 24 |
| Safe RLHF: Safe Reinforcement Learning from Human Feedback Jiaming Ji, Josef Dai, Mickel Liu, Ruiyang Sun Published: 2023-10-19Area: Alignment TrainingCitations: 574 Tags: ai-safety, alignment-training, empirical | 2023-10-19 | Alignment Training | ai-safety, alignment-training, empirical | E5 / R4 (94%) | 574 |
| Specific versus General Principles for Constitutional AI Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie Published: 2023-10-20Area: Alignment TrainingCitations: 45 Tags: ai-safety, alignment-training, empirical | 2023-10-20 | Alignment Training | ai-safety, alignment-training, empirical | E5 / R3 (95%) | 45 |
| The History and Risks of Reinforcement Learning and Human Feedback Nathan Lambert, Thomas Krendl Gilbert, Tom Zick Published: 2023-10-20Area: Alignment TrainingCitations: 50 Tags: ai-safety, alignment-training, safety-evaluation, survey | 2023-10-20 | Alignment Training | ai-safety, alignment-training, safety-evaluation, survey | E5 / R3 (96%) | 50 |
| Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases Rishabh Bhardwaj, Soujanya Poria Published: 2023-10-22Area: Safety EvaluationCitations: 23 Tags: ai-safety, alignment-training, empirical, safety-evaluation | 2023-10-22 | Safety Evaluation | ai-safety, alignment-training, empirical, safety-evaluation | E5 / R3 (95%) | 23 |
| SuperHF: Supervised Iterative Learning from Human Feedback Gabriel Mukobi, Gitta Kutyniok, Kush Bhatia, Oliver Fong Published: 2023-10-25Area: Alignment TrainingCitations: 13 Tags: ai-safety, alignment-training, empirical | 2023-10-25 | Alignment Training | ai-safety, alignment-training, empirical | E5 / R4 (96%) | 13 |
| Zephyr: Direct Distillation of LM Alignment Alexander M. Rush, Cl茅mentine Fourrier, Edward Beeching, Kashif Rasul Published: 2023-10-25Area: Alignment TrainingCitations: 542 Tags: ai-safety, alignment-training, empirical | 2023-10-25 | Alignment Training | ai-safety, alignment-training, empirical | E6 / R3 (97%) | 542 |
| AI Alignment: A Comprehensive Survey Aidan O'Gara, Borong Zhang, Boyuan Chen, Brian Tse Published: 2023-10-30Area: Surveys & ReviewsCitations: 320 Tags: ai-safety, alignment-training, interpretability, survey, surveys-reviews | 2023-10-30 | Surveys & Reviews | ai-safety, alignment-training, interpretability, survey, surveys-reviews | E7 / R4 (97%) | 320 |
| The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback Nathan Lambert, Roberto Calandra Published: 2023-10-31Area: Alignment TrainingCitations: 42 Tags: ai-safety, alignment-training, safety-evaluation, theoretical | 2023-10-31 | Alignment Training | ai-safety, alignment-training, safety-evaluation, theoretical | E5 / R3 (95%) | 42 |
| FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts Anyu Wang, Conglei Wang, Delong Ran, Jinyuan Liu Published: 2023-11-09Area: Multimodal SafetyCitations: 306 Tags: adversarial-robustness, ai-safety, alignment-training, empirical, multimodal-safety | 2023-11-09 | Multimodal Safety | adversarial-robustness, ai-safety, alignment-training, empirical, multimodal-safety | E5 / R3 (95%) | 306 |
| Alignment is not sufficient to prevent large language models from generating harmful information: A psychoanalytic perspective Jia Liu, Wei Ding, Zi Yin Published: 2023-11-14Area: Adversarial RobustnessCitations: 3 Tags: adversarial-robustness, ai-safety, alignment-training, empirical | 2023-11-14 | Adversarial Robustness | adversarial-robustness, ai-safety, alignment-training, empirical | E5 / R3 (94%) | 3 |
| Scheming AIs: Will AIs Fake Alignment During Training in Order to Get Power? Joe Carlsmith Published: 2023-11-14Area: Deception & FailureCitations: 59 Tags: ai-safety, alignment-training, deception-failure, theoretical | 2023-11-14 | Deception & Failure | ai-safety, alignment-training, deception-failure, theoretical | E6 / R3 (94%) | 59 |
| Backdoor Activation Attack: Attack Large Language Models using Activation Steering for Safety-Alignment Haoran Wang, Kai Shu Published: 2023-11-15Area: Adversarial RobustnessCitations: 24 Tags: adversarial-robustness, ai-safety, alignment-training, empirical | 2023-11-15 | Adversarial Robustness | adversarial-robustness, ai-safety, alignment-training, empirical | E5 / R3 (96%) | 24 |
| Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Framework Abraham Sanders, Bingsheng Yao, Dakuo Wang, Matthew Pisano Published: 2023-11-16Area: Adversarial RobustnessCitations: 33 Tags: adversarial-robustness, ai-safety, alignment-training, empirical | 2023-11-16 | Adversarial Robustness | adversarial-robustness, ai-safety, alignment-training, empirical | E6 / R3 (94%) | 33 |
| On the Exploitability of Reinforcement Learning with Human Feedback for Large Language Models Chaowei Xiao, Jiongxiao Wang, Junlin Wu, Muhao Chen Published: 2023-11-16Area: Alignment TrainingCitations: 32 Tags: ai-safety, alignment-training, empirical | 2023-11-16 | Alignment Training | ai-safety, alignment-training, empirical | E5 / R3 (95%) | 32 |
| Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks David Krueger, Edward Grefenstette, Ekdeep Singh Lubana, Hidenori Tanaka Published: 2023-11-21Area: Mechanistic Interp.Citations: 99 Tags: ai-safety, alignment-training, empirical, mechanistic-interp | 2023-11-21 | Mechanistic Interp. | ai-safety, alignment-training, empirical, mechanistic-interp | E5 / R3 (92%) | 99 |
| How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs Bingchen Zhao, Chenhang Cui, Cihang Xie, Haoqin Tu Published: 2023-11-27Area: Multimodal SafetyCitations: 108 Tags: ai-safety, alignment-training, benchmark, multimodal-safety, safety-evaluation | 2023-11-27 | Multimodal Safety | ai-safety, alignment-training, benchmark, multimodal-safety, safety-evaluation | E5 / R3 (95%) | 108 |
| The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning Abhilasha Ravichander, Bill Yuchen Lin, Chandra Bhagavatula, Khyathi Chandu Published: 2023-12-04Area: Alignment TrainingCitations: 276 Tags: ai-safety, alignment-training, empirical | 2023-12-04 | Alignment Training | ai-safety, alignment-training, empirical | E6 / R3 (95%) | 276 |
| Alignment for Honesty Ethan Chern, Graham Neubig, Pengfei Liu, Xipeng Qiu Published: 2023-12-12Area: Alignment TrainingCitations: 63 Tags: ai-safety, alignment-training, empirical | 2023-12-12 | Alignment Training | ai-safety, alignment-training, empirical | E5 / R3 (94%) | 63 |
| Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack Cong Liu, Wen Xiao, Yue Dong, Yufei Li Published: 2023-12-12Area: Adversarial RobustnessCitations: 10 Tags: adversarial-robustness, ai-safety, alignment-training, empirical | 2023-12-12 | Adversarial Robustness | adversarial-robustness, ai-safety, alignment-training, empirical | E6 / R3 (94%) | 10 |
| Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations Damai Dai, Deli Chen, Lei Li, Peiyi Wang Published: 2023-12-14Area: Alignment TrainingCitations: 725 Tags: ai-safety, alignment-training, empirical | 2023-12-14 | Alignment Training | ai-safety, alignment-training, empirical | E6 / R4 (96%) | 725 |