Paper deep dive

Agentic Large Language Models

Max van Duijn, Michiel van der Meer, Aske Plaat, Niki van Stein

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 1091

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 10/12/2026, 7:21:30 AM

Summary

This book, 'Agentic Large Language Models' by Max van Duijn, Michiel van der Meer, Aske Plaat, and Niki van Stein, provides a comprehensive overview of how Large Language Models (LLMs) function as agents. It covers foundational language modeling, training pipelines, scaling laws, and advanced reasoning techniques like Chain of Thought and self-reflection. The text explores multi-modal models, vision-action models, and metalearning. A significant portion is dedicated to agentic architectures, including tool use, retrieval-augmented generation (RAG), and multi-agent systems involving role-based interaction and emergent behavior. The book also addresses ethical considerations, safety, and synthetic data generation, supported by case studies on building LLMs, reasoning agents, multi-agent diagnosis, and societal simulations.

Entities (18)

Aske Plaat · person · 99%Max van Duijn · person · 99%Michiel van der Meer · person · 99%Niki van Stein · person · 99%Large Language Models · technology · 98%Agentic LLMs · concept · 97%Transformer · architecture · 96%Chain-of-Thought Prompting · method · 95%Retrieval-Augmented Generation · method · 95%BERT · model · 94%GPT · model · 94%Chinchilla Scaling Laws · concept · 93%Kaplan Scaling Laws · concept · 93%BALROG · system · 92%Theory of Mind · concept · 92%LangChain · software · 91%Self-RAG · method · 90%Model Collapse · risk · 89%

Relation Signals (14)

Niki van Stein → authorof → Agentic Large Language Models

confidence 99% · Agentic Large Language Models Max van Duijn Michiel van der Meer Aske Plaat Niki van Stein

Michiel van der Meer → authorof → Agentic Large Language Models

confidence 99% · Agentic Large Language Models Max van Duijn Michiel van der Meer Aske Plaat Niki van Stein

Aske Plaat → authorof → Agentic Large Language Models

confidence 99% · Agentic Large Language Models Max van Duijn Michiel van der Meer Aske Plaat Niki van Stein

Max van Duijn → authorof → Agentic Large Language Models

confidence 99% · Agentic Large Language Models Max van Duijn Michiel van der Meer Aske Plaat Niki van Stein

Agentic LLMs → combines → Large Language Models

confidence 97% · Agentic LLMs combine the power of large language models with the ability to act in the real world.

Retrieval-Augmented Generation → enhances → Agentic LLMs

confidence 95% · 12.6 Agentic RAG: Beyond Simple Retrieval . . . . . . . . . . . . . . . 254

Chain-of-Thought Prompting → usedfor → Multi-step Reasoning

confidence 95% · 6.1.3Chain of Thought Prompting . . . . . . . . . . . . . . . . . . . . . 128

BERT → typeof → Encoder-only Models

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models (LLMs) have taken the world by storm. Their ability to communicate with us in our own language, and to reason about our world, has made them a favorite tool for writing, answering questions, and much more. Their answers already influence our decisions and our actions. What would happen if LLMs themselves could act in the world directly? What would happen if LLMs could interact with each other? Agentic LLMs combine the power of large language models with the ability to act in the real world. In this book we will discuss how agentic LLMs work, how they act, how they interact, and what this means for us, as users.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

1,088,500 characters extracted from source content.

Expand or collapse full text

Agentic Large Language Models Max van Duijn Michiel van der Meer Aske Plaat Niki van Stein arXiv:2610.13455v1 [cs.AI] 9 Oct 2026 2 Contents 1 Introduction15 1.1Why this book? . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 1.2Concepts and Definitions . . . . . . . . . . . . . . . . . . . . . . . .15 1.2.1Agents that Assist, Agents that Interact . . . . . . . . . . . .16 1.2.2Agentic Harness . . . . . . . . . . . . . . . . . . . . . . . .16 1.2.3Multi-Modal Models . . . . . . . . . . . . . . . . . . . . . .16 1.2.4Training Data . . . . . . . . . . . . . . . . . . . . . . . . . .17 1.3Audience and Aim . . . . . . . . . . . . . . . . . . . . . . . . . . .18 1.4Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 1.5Course Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 1.6Approach, Scope and Overview . . . . . . . . . . . . . . . . . . . .20 1.7Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . .22 I Language Modeling25 2 Foundations of Language Modeling29 2.1Processing Natural Language . . . . . . . . . . . . . . . . . . . . . .29 2.1.1Language Understanding and Language Generation . . . . . .31 2.1.2Complexity of Natural Language . . . . . . . . . . . . . . . .32 2.1.3Distributional Representations . . . . . . . . . . . . . . . . .34 2.1.4Tokenization . . . . . . . . . . . . . . . . . . . . . . . . . .36 2.1.5Early Methods for Language Representation . . . . . . . . .37 2.2Machine Learning Methods For Natural Language Processing . . . .39 2.2.1Supervised Learning . . . . . . . . . . . . . . . . . . . . . .39 2.2.2Autoregressive Learning . . . . . . . . . . . . . . . . . . . .40 2.2.3Reinforcement Learning . . . . . . . . . . . . . . . . . . . .41 2.3Modeling Language . . . . . . . . . . . . . . . . . . . . . . . . . . .42 2.3.1Embeddings . . . . . . . . . . . . . . . . . . . . . . . . . .43 2.3.2Shifting to Contextualized Embeddings . . . . . . . . . . . .44 2.3.3Recurrent Neural Networks . . . . . . . . . . . . . . . . . .45 2.3.4Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . .48 2.3.5Transformers . . . . . . . . . . . . . . . . . . . . . . . . . .50 2.3.6Encoder-only (BERT) . . . . . . . . . . . . . . . . . . . . .52 3 4CONTENTS 2.3.7Decoder-only (GPT) . . . . . . . . . . . . . . . . . . . . . .53 2.4Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .54 2.4.1Open challenges . . . . . . . . . . . . . . . . . . . . . . . .54 2.4.2Looking Ahead . . . . . . . . . . . . . . . . . . . . . . . . .56 3 LLM Training Pipeline63 3.1Training Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63 3.1.1Encoder versus Decoder Models . . . . . . . . . . . . . . . .64 3.1.2Pretraining/Finetuning Paradigm . . . . . . . . . . . . . . . .64 3.1.3Open and Closed Models . . . . . . . . . . . . . . . . . . . .64 3.2Training Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . .65 3.2.1Acquire Data . . . . . . . . . . . . . . . . . . . . . . . . . .65 3.2.2Pretrain . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66 3.2.3Finetune . . . . . . . . . . . . . . . . . . . . . . . . . . . .69 3.2.4Preference Tuning . . . . . . . . . . . . . . . . . . . . . . .70 3.2.5Infer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71 3.2.6Concluding Remarks . . . . . . . . . . . . . . . . . . . . . .72 3.3Benchmarks, and Performance . . . . . . . . . . . . . . . . . . . . .72 3.3.1Common tasks . . . . . . . . . . . . . . . . . . . . . . . . .73 3.3.2Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73 3.3.3Common Benchmarks . . . . . . . . . . . . . . . . . . . . .74 3.4Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . .74 4 Scaling Laws and Efficiency83 4.1Scaling Laws: Compute, Data, Parameters . . . . . . . . . . . . . . .83 4.1.1Kaplan Scaling Laws . . . . . . . . . . . . . . . . . . . . . .84 4.1.2Chinchilla Scaling Laws . . . . . . . . . . . . . . . . . . . .86 4.1.3Compute-Optimal Curve . . . . . . . . . . . . . . . . . . . .87 4.1.4Emergent Abilities . . . . . . . . . . . . . . . . . . . . . . .89 4.1.5Inverse Scaling . . . . . . . . . . . . . . . . . . . . . . . . .90 4.1.6Grokking . . . . . . . . . . . . . . . . . . . . . . . . . . . .91 4.2Small Language Models . . . . . . . . . . . . . . . . . . . . . . . .92 4.2.1Model-centric Optimization . . . . . . . . . . . . . . . . . .92 4.2.1.1Reducing Pretraining Cost . . . . . . . . . . . . . .92 4.2.1.2Reducing Inference Cost . . . . . . . . . . . . . .95 4.2.2Data-centric Optimization . . . . . . . . . . . . . . . . . . .96 4.2.2.1Data Selection and Curation . . . . . . . . . . . . .96 4.2.2.2Synthetic Data Gathering . . . . . . . . . . . . . .96 4.2.2.3Sample-Efficient Learning . . . . . . . . . . . . . .97 4.2.3Popular models . . . . . . . . . . . . . . . . . . . . . . . . .97 4.3Inference-Time Compute . . . . . . . . . . . . . . . . . . . . . . . .99 4.4Sustainable Language Models . . . . . . . . . . . . . . . . . . . . .99 4.5Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 100 CONTENTS5 5 Case Study 1: Building LLMs from Scratch111 5.1Building a Chatbot . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 5.2Background: Karpathy’s nanochat . . . . . . . . . . . . . . . . . . . 112 5.3Experiment Preparation . . . . . . . . . . . . . . . . . . . . . . . . . 113 5.4Task 0: Model Hub warm-up. . . . . . . . . . . . . . . . . . . . . . . 114 5.5Task 1: Tokenization . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.5.1What to Do . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.5.2Deliverables . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.6Task 2: Pretraining . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.6.1What to Do . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.6.2Deliverables . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.7Task 3: Mid-Training and Supervised Finetuning . . . . . . . . . . . 116 5.7.1What to Do . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.7.2Deliverables . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.8Task 4: Inference, Deployment, and Critical Reflection . . . . . . . . 117 5.8.1Part A: Inference and Deployment . . . . . . . . . . . . . . . 117 5.8.2Part B: Critical Reflection . . . . . . . . . . . . . . . . . . . 118 5.9Recommended Reading . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.10 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 119 I Reasoning121 6 Multi-step Reasoning in LLMs125 6.1In-Context Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.1.1In-Context Learning . . . . . . . . . . . . . . . . . . . . . . 126 6.1.2Multi-step Reasoning Benchmarks . . . . . . . . . . . . . . . 127 6.1.3Chain of Thought Prompting . . . . . . . . . . . . . . . . . . 128 6.1.3.1Self Evaluation . . . . . . . . . . . . . . . . . . . 130 6.1.3.2Code Generation for Validation . . . . . . . . . . . 132 6.1.3.3Evolving code generation prompts . . . . . . . . . 132 6.2Self-Reflection and State . . . . . . . . . . . . . . . . . . . . . . . . 133 6.2.1LLMs are Memory-less Functions . . . . . . . . . . . . . . . 133 6.2.2Retrieval Augmentation and Long Context Windows . . . . . 134 6.2.3LLMs and Stateful Problems . . . . . . . . . . . . . . . . . . 134 6.2.4Agentic Harness for Prompt Improvement . . . . . . . . . . . 135 6.2.4.1Search-based Prompt Optimization . . . . . . . . . 136 6.2.4.2Self-Reflection . . . . . . . . . . . . . . . . . . . . 138 6.2.4.3Reinforcement Learning with Verifiable Rewards . 141 6.3Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 142 6.3.1Brittleness . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 6.3.2Thinking, Fast and Slow . . . . . . . . . . . . . . . . . . . . 143 6.3.3Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 6CONTENTS 7 Vision, Action and World Models151 7.1Multi-modal Vision Language Models . . . . . . . . . . . . . . . . . 151 7.1.1Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .152 7.1.2Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 7.1.3Learning Algorithm . . . . . . . . . . . . . . . . . . . . . . 154 7.1.4Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . .156 7.2Decisions & Vision Language Action Models . . . . . . . . . . . . . 158 7.2.1Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .158 7.2.2Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 7.2.3Learning Algorithm . . . . . . . . . . . . . . . . . . . . . . 163 7.2.4Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . .163 7.3World Models, Physics, and Planning . . . . . . . . . . . . . . . . . 164 7.3.1Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .165 7.3.2Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 7.3.3Learning Algorithm . . . . . . . . . . . . . . . . . . . . . . 169 7.3.4Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . .170 7.3.5*Model-Based Reinforcement Learning . . . . . . . . . . . . 171 7.3.5.1Model-free Reinforcement Learning . . . . . . . . 171 7.3.5.2Model-based Reinforcement Learning . . . . . . . 172 7.3.5.3Dimension Reduction . . . . . . . . . . . . . . . . 172 7.3.5.4Joint Embedding Predictive Architecture . . . . . . 173 7.3.5.5Agentic World Models . . . . . . . . . . . . . . . . 173 7.4Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 173 8 Metalearning181 8.1Why Metalearning? . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 8.1.1Deep Learning is Slow . . . . . . . . . . . . . . . . . . . . . 182 8.1.2Achieving Few-shot Learning . . . . . . . . . . . . . . . . . 182 8.2Inner/Outer Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 8.3LLM and Metalearning . . . . . . . . . . . . . . . . . . . . . . . . . 184 8.3.1Static LLM Metalearning . . . . . . . . . . . . . . . . . . . . 184 8.3.1.1Pretraining Pipeline . . . . . . . . . . . . . . . . . 184 8.3.1.2In-Context Learning . . . . . . . . . . . . . . . . . 184 8.3.1.3Chain of Thought . . . . . . . . . . . . . . . . . . 185 8.3.1.4Static Metalearning is Stable . . . . . . . . . . . . 185 8.3.2Dynamic LLM Metalearning . . . . . . . . . . . . . . . . . . 185 8.3.2.1Self-Reflection . . . . . . . . . . . . . . . . . . . . 185 8.3.2.2RLVR . . . . . . . . . . . . . . . . . . . . . . . . 186 8.4Stabilizing Dynamic Metalearning . . . . . . . . . . . . . . . . . . . 186 8.5Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 186 I Agents189 9 Interaction Styles for Agentic LLMs193 9.1Agents in AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 CONTENTS7 9.1.1Agents Act and Adapt in their Environment . . . . . . . . . . 193 9.1.1.1Robotic Assistants . . . . . . . . . . . . . . . . . . 194 9.1.1.2Agents with Beliefs, Desires, and Intentions . . . . 194 9.1.1.3Swarm Agents . . . . . . . . . . . . . . . . . . . . 194 9.1.2Agentic LLMs as Reinforcement Learning Agents . . . . . . 195 9.2Learning Mechanisms in Agentic LLMs . . . . . . . . . . . . . . . . 197 9.2.1Agentic Interaction Styles . . . . . . . . . . . . . . . . . . . 197 9.2.1.1A. Query Prompt . . . . . . . . . . . . . . . . . . 198 9.2.1.2B. In-Context Learning . . . . . . . . . . . . . . . 198 9.2.1.3C. Chain of Thought . . . . . . . . . . . . . . . . . 199 9.2.1.4D. Self-Reflection Harness . . . . . . . . . . . . . 199 9.2.1.5E. Reinforcement Finetuning Harness . . . . . . . . 201 9.3Memory Architectures for Agentic LLMs . . . . . . . . . . . . . . . 202 9.3.1The Need for Memory . . . . . . . . . . . . . . . . . . . . . 202 9.3.1.1Coherent Conversation . . . . . . . . . . . . . . . 202 9.3.1.2Adaptation . . . . . . . . . . . . . . . . . . . . . . 202 9.3.1.3Identity . . . . . . . . . . . . . . . . . . . . . . . . 203 9.3.1.4Continual Learning . . . . . . . . . . . . . . . . . 203 9.3.2Memory Architectures . . . . . . . . . . . . . . . . . . . . . 203 9.3.2.1Finetuning . . . . . . . . . . . . . . . . . . . . . . 203 9.3.2.2Long Context Windows and Compression . . . . . 204 9.3.2.3Explicit Memory Management . . . . . . . . . . . 204 9.4Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 205 10 Case Study 2: Reasoning and Reflection211 10.1 BALROG Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 10.1.1 Puzzles and Games . . . . . . . . . . . . . . . . . . . . . . . 212 10.1.2 BALROG Architecture . . . . . . . . . . . . . . . . . . . . . 213 10.1.3 Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 10.1.3.1 Prompt Builder . . . . . . . . . . . . . . . . . . . 214 10.1.3.2 Agents . . . . . . . . . . . . . . . . . . . . . . . . 214 10.2 Implementation Steps . . . . . . . . . . . . . . . . . . . . . . . . . . 215 10.2.1 Reproduce Paper . . . . . . . . . . . . . . . . . . . . . . . . 215 10.2.2 Different LLMs . . . . . . . . . . . . . . . . . . . . . . . . . 215 10.2.3 Different Puzzles . . . . . . . . . . . . . . . . . . . . . . . . 215 10.2.4 Computational Cost . . . . . . . . . . . . . . . . . . . . . . 215 10.2.5 Implement Own Reasoning Agent . . . . . . . . . . . . . . . 216 10.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 10.4 Self-Reflection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 10.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 220 IV Tools and Assistants223 11 How Agents Use Tools to Act in the World227 11.1 Why Tools Matter in Agentic LLMs . . . . . . . . . . . . . . . . . . 227 8CONTENTS 11.2 Tool Use Architectures . . . . . . . . . . . . . . . . . . . . . . . . . 228 11.3 APIs and Toolchains . . . . . . . . . . . . . . . . . . . . . . . . . . 228 11.3.1 LangChain Example . . . . . . . . . . . . . . . . . . . . . . 229 11.3.2 Computer and Browser Tools . . . . . . . . . . . . . . . . . 229 11.4 Model Context Protocol and Context Management . . . . . . . . . . 232 11.5 Robustness, Error Handling, and Recovery . . . . . . . . . . . . . . . 233 11.6 Safety and Security . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 11.7 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 237 12 Search and Retrieval in Agentic LLMs241 12.1 Retrieval in General . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 12.1.1 Bag of Words . . . . . . . . . . . . . . . . . . . . . . . . . . 242 12.1.2 Vector Space Models . . . . . . . . . . . . . . . . . . . . . . 242 12.1.3 Latent and Sparse Representations . . . . . . . . . . . . . . . 243 12.2 Why Retrieval Matters for Agentic LLMs . . . . . . . . . . . . . . . 244 12.3 Retrieval-Augmented Generation Pipeline . . . . . . . . . . . . . . . 244 12.4 Adaptive Retrieval Methods . . . . . . . . . . . . . . . . . . . . . . 246 12.4.1 Adaptive Retrieval via Self-RAG and Self-Ask . . . . . . . . 247 12.4.2 A Simple Dense Retrieval Example in Python . . . . . . . . . 248 12.4.3 Query Rewriting and Expansion . . . . . . . . . . . . . . . . 248 12.5 Graphs, Code, Tabular Data . . . . . . . . . . . . . . . . . . . . . . . 249 12.5.1 Graph-based retrieval example . . . . . . . . . . . . . . . . . 250 12.5.2 Code retrieval example using ASTs and embeddings . . . . . 251 12.5.3 Multi-modal and heterogeneous sources . . . . . . . . . . . . 251 12.5.4 Multi-modal retrieval example . . . . . . . . . . . . . . . . . 253 12.6 Agentic RAG: Beyond Simple Retrieval . . . . . . . . . . . . . . . . 254 12.6.1 Key Components of Agentic RAG . . . . . . . . . . . . . . . 254 12.6.2 Real-Time and Streaming RAG . . . . . . . . . . . . . . . . 255 12.7 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 12.8 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 257 13 From Tools to Assistants263 13.1 Agentic LLMs as Assistants . . . . . . . . . . . . . . . . . . . . . . 263 13.2 Conversational Assistants . . . . . . . . . . . . . . . . . . . . . . . . 264 13.3 Shopping Assistant . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 13.4 Flight Operations Assistant . . . . . . . . . . . . . . . . . . . . . . . 266 13.5 Medical Assistant . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 13.6 Trading Assistant . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 13.7 Automated Scientific Discovery . . . . . . . . . . . . . . . . . . . . 270 13.8 Coding Assistants and General Advances . . . . . . . . . . . . . . . 271 13.9 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 272 CONTENTS9 14 Case Study 3: Multi-Agent Diagnosis277 14.1 Scenario Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 14.2 Environment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 14.3 Implementation Steps . . . . . . . . . . . . . . . . . . . . . . . . . . 278 14.3.1 Step 0: Training Diagnostic Models . . . . . . . . . . . . . . 278 14.3.2 Step 1: Understanding and Running the Baseline . . . . . . . 279 14.3.3 Step 2: Replacing Rule-Based Planners . . . . . . . . . . . . 279 14.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 14.5 Reflection Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 280 14.6 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 282 V Behavior and Cognition283 15 Studying Behavior of Agentic LLMs287 15.1 What is LLM Behavior? . . . . . . . . . . . . . . . . . . . . . . . . 287 15.2 Probing LLM Behavior Experimentally . . . . . . . . . . . . . . . . 288 15.3 The Basic Social Toolkit . . . . . . . . . . . . . . . . . . . . . . . . 289 15.3.1 Functional and Pragmatic Linguistic Competence . . . . . . . 289 15.3.2 Politeness and Social Etiquette . . . . . . . . . . . . . . . . . 290 15.3.3 Sensitivity to Emotional and Affective Cues . . . . . . . . . . 291 15.3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 15.4 Strategic Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 15.4.1 Cooperation and Coordination in Social Dilemmas . . . . . . 294 15.4.2 Benchmarks Across Games and Models . . . . . . . . . . . . 296 15.4.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 15.5 Theory of Mind . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 15.5.1 Computational Theory of Mind . . . . . . . . . . . . . . . . 298 15.5.2 Boosting ToM: Enhancement Strategies . . . . . . . . . . . . 301 15.5.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 15.6 Moral Judgment and Behavior . . . . . . . . . . . . . . . . . . . . . 302 15.6.1 Background: Moral Psychology and Philosophy . . . . . . . 302 15.6.2 Probing Moral Judgment in LLMs . . . . . . . . . . . . . . . 303 15.6.3 From Judgment to Moral Behavior . . . . . . . . . . . . . . . 304 15.6.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 15.7 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 305 16 Role-Based Interaction in Multi-LLM Systems317 16.1 From Single-agent to Multi-agent . . . . . . . . . . . . . . . . . . . 317 16.1.1 Mulit-Agent Simulations: History and Present . . . . . . . . 318 16.1.2 Role-Based versus Emergent Interaction . . . . . . . . . . . . 318 16.1.3 Five Dimensions of Multi-Agent Collaboration . . . . . . . . 318 16.1.4 A Methodological Note . . . . . . . . . . . . . . . . . . . . . 319 16.2 Strategic Interaction Among LLM Agents . . . . . . . . . . . . . . . 319 16.2.1 Benchmarking Strategic Ability Across Games . . . . . . . . 320 16.2.2 Strategic Interaction in Single-Scenario Platforms . . . . . . . 322 10CONTENTS 16.2.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 16.3 Role-Based Task Solving and Teamwork . . . . . . . . . . . . . . . . 323 16.3.1 Complementary Pairs . . . . . . . . . . . . . . . . . . . . . . 324 16.3.2 Debate and Discussion . . . . . . . . . . . . . . . . . . . . . 325 16.3.3 Role-Based Software Teams . . . . . . . . . . . . . . . . . . 326 16.3.4 Social Role-Play . . . . . . . . . . . . . . . . . . . . . . . . 326 16.3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 16.4 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 329 17 Emergent Behavior in Multi-LLM Systems335 17.1 What Is Emergent Multi-Agent Behavior? . . . . . . . . . . . . . . . 335 17.1.1 From Agent-Based Modeling to Generative Agent Societies . 337 17.1.2 A Methodological Note . . . . . . . . . . . . . . . . . . . . . 337 17.2 Simulation Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . 338 17.2.1 Small Worlds . . . . . . . . . . . . . . . . . . . . . . . . . . 338 17.2.2 Societies of Thousands to a Million Agents . . . . . . . . . . 340 17.2.3 Simulation as a Scientific Instrument . . . . . . . . . . . . . 341 17.3 Emergent Conventions and Norms . . . . . . . . . . . . . . . . . . . 342 17.4 Evaluating Agents in Open World Environments . . . . . . . . . . . . 345 17.5 Risks of Emergent Behavior . . . . . . . . . . . . . . . . . . . . . . 346 17.6 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 346 18 Case Study 4: Simulating Societies353 18.1 Social Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 18.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 18.2.1 Generative Agents . . . . . . . . . . . . . . . . . . . . . . . 354 18.2.2 Bad is Stronger Than Good . . . . . . . . . . . . . . . . . . . 354 18.2.3 Why This Matters for Agent Design . . . . . . . . . . . . . . 355 18.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355 18.3.1 The Simulation Environment . . . . . . . . . . . . . . . . . . 355 18.3.2 Agent Architecture . . . . . . . . . . . . . . . . . . . . . . . 356 18.3.3 The Cognitive Loop . . . . . . . . . . . . . . . . . . . . . . 356 18.3.4 Two Architecture Variants . . . . . . . . . . . . . . . . . . . 356 18.3.5 Setup and Installation . . . . . . . . . . . . . . . . . . . . . . 357 18.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 18.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 358 VI Advanced Topics359 19 *Synthetic Data Generation363 19.1 The Importance of Data . . . . . . . . . . . . . . . . . . . . . . . . . 363 19.2 Distillation Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 19.2.1 Signal: The Distillation Objective . . . . . . . . . . . . . . . 366 19.2.2 Data Source: Teacher-Student Paradigm . . . . . . . . . . . . 366 19.2.3 Timing: Off-policy versus On-policy Distillation . . . . . . . 366 CONTENTS11 19.2.4 Teacher Identity: Self-Distillation and Self-Improvement . . . 367 19.3 Simulating Human-generated Content . . . . . . . . . . . . . . . . . 368 19.3.1 Generating Training Data . . . . . . . . . . . . . . . . . . . . 369 19.3.2 Generating Evaluation Data . . . . . . . . . . . . . . . . . . 371 19.3.3 Generating Agentic Data . . . . . . . . . . . . . . . . . . . . 371 19.4 Data Curation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 19.4.1 Extrinsic evaluation . . . . . . . . . . . . . . . . . . . . . . . 373 19.4.2 Intrinsic evaluation . . . . . . . . . . . . . . . . . . . . . . . 373 19.5 Risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 19.5.1 Model Collapse . . . . . . . . . . . . . . . . . . . . . . . . . 375 19.5.2 Bias Amplification . . . . . . . . . . . . . . . . . . . . . . . 375 19.5.3 Security Risks . . . . . . . . . . . . . . . . . . . . . . . . . 375 19.5.4 Privacy Preservation . . . . . . . . . . . . . . . . . . . . . . 376 19.5.5 Copyright and Terms of Use . . . . . . . . . . . . . . . . . . 376 19.6 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 376 20 *Advanced Agentic Architectures391 20.1 Transformer Architectures . . . . . . . . . . . . . . . . . . . . . . . 391 20.1.1 Transformers are Turing-complete . . . . . . . . . . . . . . . 392 20.1.2 Pretraining Backtracking on Decision Traces . . . . . . . . . 392 20.1.3 LoopedLLM and Recursive Multi-Agent Systems . . . . . . . 394 20.2 Agentic Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . 394 20.2.1 In-Context Reinforcement Learning . . . . . . . . . . . . . . 394 20.2.2 Self-Modifying Hyperagents . . . . . . . . . . . . . . . . . . 396 20.2.3 Meta Agents . . . . . . . . . . . . . . . . . . . . . . . . . . 397 20.3 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 399 21 *Mechanistic Interpretability for LLM Agents403 21.1 Opening the LLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 21.2 Hallucination and Faithfulness in Model Reasoning . . . . . . . . . . 405 21.3 The Residual Stream and Iterative Refinement . . . . . . . . . . . . . 406 21.4 Polysemanticity and Superposition . . . . . . . . . . . . . . . . . . . 406 21.5 The Logit Lens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408 21.5.1 Formal Mechanism . . . . . . . . . . . . . . . . . . . . . . . 408 21.5.2 From the Logit Lens to the Tuned Lens . . . . . . . . . . . . 409 21.5.3 Applications and Interpretations . . . . . . . . . . . . . . . . 410 21.6 Circuit Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 21.6.1 The Prototypical Circuit: Induction Heads . . . . . . . . . . . 411 21.6.2 Methodologies for Circuit Discovery . . . . . . . . . . . . . 411 21.7 Activation Patching and Causal Interventions . . . . . . . . . . . . . 412 21.8 Sparse Autoencoders: Dictionary Learning for Feature Discovery . . . 414 21.8.1 Monosemanticity and Shadow Models . . . . . . . . . . . . . 415 21.9 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 416 21.9.1 Outlook and Emerging Directions . . . . . . . . . . . . . . . 416 21.9.2 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . 416 12CONTENTS 22 *Cognitive Perspectives on Agentic LLMs423 22.1 From Behavior to Cognition . . . . . . . . . . . . . . . . . . . . . . 423 22.1.1 Three Levels of Evidence . . . . . . . . . . . . . . . . . . . 424 22.1.2 Competence and Performance . . . . . . . . . . . . . . . . . 427 22.1.3 The Unit of Analysis: Model, System, Environment? . . . . . 427 22.2 Understanding Linguistic Content . . . . . . . . . . . . . . . . . . . 428 22.2.1 The Skeptical Case . . . . . . . . . . . . . . . . . . . . . . . 428 22.2.2 Replies to the Skeptical Case . . . . . . . . . . . . . . . . . . 429 22.2.3 Evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 22.2.4 The Agentic Perspective . . . . . . . . . . . . . . . . . . . . 430 22.2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 430 22.3 World Models and Spatial Cognition . . . . . . . . . . . . . . . . . . 431 22.3.1 Evidence For . . . . . . . . . . . . . . . . . . . . . . . . . . 431 22.3.2 Evidence Against . . . . . . . . . . . . . . . . . . . . . . . . 432 22.3.3 The Agentic Perspective . . . . . . . . . . . . . . . . . . . . 433 22.3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 22.4 Social Cognition and Theory of Mind . . . . . . . . . . . . . . . . . 433 22.4.1 Beliefs at the Representational and Mechanistic Levels . . . . 434 22.4.2 Human and Machine Routes to Similar Behavior . . . . . . . 434 22.4.3 The Literal–Functional Gap Revisited . . . . . . . . . . . . 435 22.4.4 Mindreading LLMs . . . . . . . . . . . . . . . . . . . . . . . 436 22.4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 436 22.5 Metacognition, Introspection, and Self-Knowledge . . . . . . . . . . 436 22.5.1 Monitoring: Knowing What You Know . . . . . . . . . . . . 437 22.5.2 Introspection . . . . . . . . . . . . . . . . . . . . . . . . . . 438 22.5.3 The Agentic Perspective . . . . . . . . . . . . . . . . . . . . 438 22.5.4 Conslusion . . . . . . . . . . . . . . . . . . . . . . . . . . .439 22.6 Consciousness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439 22.6.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . 439 22.6.2 Functionalism and Theory-Driven Assessment . . . . . . . . 440 22.6.3 Rejections of Functionalism . . . . . . . . . . . . . . . . . . 442 22.6.4 Self-Reports and Inner Life . . . . . . . . . . . . . . . . . . 443 22.6.5 The Agentic Perspective . . . . . . . . . . . . . . . . . . . . 443 22.6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 444 22.7 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 444 VII Conclusion457 23 Future Directions for Agentic LLMs461 23.1 The State of Agentic LLMs . . . . . . . . . . . . . . . . . . . . . . . 461 23.1.1 Technology . . . . . . . . . . . . . . . . . . . . . . . . . . .461 23.1.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . 462 23.1.3 Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462 23.2 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . 462 23.2.1 Memory and Self-reflection . . . . . . . . . . . . . . . . . . 463 CONTENTS13 23.2.2 Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 23.2.3 Hallucination and Explainability . . . . . . . . . . . . . . . . 464 23.2.4 Emotional Behavior and Consciousness . . . . . . . . . . . . 464 23.2.5 Social Behavior . . . . . . . . . . . . . . . . . . . . . . . . . 465 23.2.6 Legal and Ethical Aspects . . . . . . . . . . . . . . . . . . . 465 23.2.7 Crime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 23.2.8 Limitations of Applications . . . . . . . . . . . . . . . . . . 466 23.2.8.1 AI Science . . . . . . . . . . . . . . . . . . . . . . 466 23.2.8.2 E-commerce . . . . . . . . . . . . . . . . . . . . . 467 23.2.8.3 Trading . . . . . . . . . . . . . . . . . . . . . . . . 467 23.2.8.4 Medicine . . . . . . . . . . . . . . . . . . . . . . . 468 23.3 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 468 14CONTENTS Chapter 1 Introduction Large language models (LLMs) have taken the world by storm. Their ability to com- municate with us in our own language, and to reason about our world, has made them a favorite tool for writing, answering questions, and much more. Their answers already influence our decisions and our actions. What would happen if LLMs themselves could act in the world directly? What would happen if LLMs could interact with each other? Agentic LLMs combine the power of large language models with the ability to act in the real world. In this book we will discuss how agentic LLMs work, how they act, how they interact, and what this means for us, as users. 1.1 Why this book? Artificial intelligence (AI) is changing the world. Large language models that act change the world in a new way. In order to understand the extent of these changes it is important for students and researchers of AI to understand both the technology itself and the applications that are now emerging. This understanding can help us influ- ence the direction of these developments, perhaps build in safeguards, and, specifically, it will help us to develop the field further. This book has been written for the graduate course on agentic LLMs that we teach at our university. In agentic LLMs two fields of AI research come together: the study of agents, and the study of large language models. Agents are changing the field of LLMs, and, conversely, LLMs are changing the field of agents. Although there are excellent books about LLMs and how they work (such as Jurafsky and Martin [2025]), the ability to act puts agentic LLMs in a class of their own. We think that there is a need for a course and for a book that explain how agentic LLMs work, and how they enable things that were not possible before. 1.2 Concepts and Definitions The strength of the language abilities of LLMs is impressive, they talk just like we do. Now that LLMs have become agentic, their horizon extends far beyond talking: LLMs 15 16CHAPTER 1. INTRODUCTION have capabilities for reasoning, for using tools such as writing and running code, for browsing the web to retrieve information, and for interacting with other agents or with the outside world. This brings us to two definitions that we use in this book. Agents perform actions, observe the effect of these actions and use the feedback they receive from the world to adapt their action policy. Agentic LLMs are agents that integrate the power of LLMs, such as to communicate using natural language, write code, or process information. 1.2.1 Agents that Assist, Agents that Interact The ability to act has increased the relevance and applicability of LLMs in many do- mains of society. How are agentic LLMs used? In this book, we distinguish two major classes of use cases for Agentic LLMs: (1) Agentic LLMs are used to assist their human users in application domains such as medicine, logistics, and finance (Part IV). (2) Agentic LLMs are used to interact with each other to perform tasks or function in simulations of human behavior (Part V). Many research papers have been published in recent years in both domains. We will provide overviews of key studies and trends throughout the book. 1.2.2 Agentic Harness From an engineering perspective, how do we construct agentic LLMs? Agentic LLMs consist of two elements: (1) the LLM, and (2) a piece of code that is called the harness. Agentic LLM = LLM + Harness The harness is a piece of computer code that wraps around the model and controls how it is called (a horse’s harness helps us guide the power of the noble animal in a productive direction). The harness performs tasks such as prompt construction, answer processing, calling external tools, state management, system prompt management, and skill management. The harness loops through the reasoning, acting, and observing that are necessary for task completion, without the need for human intervention to guide the process manually. The agentic harness can thus be seen as a versatile automatic prompt engineer (Part I, Chapter 6). Importantly, the harness may adapt the behavior policy of the agent based on feedback on its actions, using reinforcement learning (Part I, Chapter 9). 1.2.3 Multi-Modal Models Understanding natural language is crucial for agents that interact with users and the world around them. However, for many tasks agents must also process other modal- ities, such as visual information, or other text formats, such as spreadsheets or com- puter code. As such, the domain of agentic LLMs extends beyond natural language, 1.2. CONCEPTS AND DEFINITIONS17 Figure 1.1: Recursive self-improvement cycle connecting three categories of agentic LLM: reasoning, acting, and interacting (in red, corresponding to Parts I, I/IV and V). Concepts that influence a category are in green. and includes multi-modal, code, and world models. Therefore, please note that in this book we use the term LLM pragmatically—depending on the context it can refer to language-only systems, or it can have a broader scope: large transformer-based model that processes natural language, computer code, images, tool-commands, or other rel- evant modalities (Part I, Chapter 7). In cases where the distinction is important, we flag which variant of the definition applies. 1.2.4 Training Data Agentic LLMs mark another paradigm shift in LLM engineering, that is related to machine learning. For a long time, the learning-emphasis in the field of LLMs has been on learning during model training, but now there is an increasing role for learning at inference time, when the model is queried. This has two advantages. First, it creates flexibility and space for dynamic agentic behavior, as we will see in Chapters 8 and 9. Second, in an era where almost the entire internet has been used as data for pretraining models [Sutskever, 2024], it has become useful to generate additional, relevant, and high-quality data. By interacting with the world, agents can generate such new data themselves. Figure 1.1 illustrates this process. This interaction-data can be used for additional training. An example of how LLMs are trained on their own agentic actions are vision- language-action models that learn from robotic action sequences (Chapter 7). Another example can be found in Chapter 17, where societies of agents allow models to become more intelligent through interaction, just like humans learn new behaviors and ideas when they interact with one another. Thus, in addition to enabling useful applications, another driver of agentic LLMs is the opportunity to generate more training data, creating a recursive self-improvement loop. In Figure 1.1 this virtuous cycle of interactive data generation is shown. Shown in red are the main categories: reasoning (Part I), acting (Parts I/IV), and interacting (Part V). In green are concepts that influence a category, either directly, or in a feedback loop. 18CHAPTER 1. INTRODUCTION 1.3 Audience and Aim This book is primarily written for graduate students and researchers in artificial intel- ligence and computer science. It may also be of interest to students and scholars from other fields, such as the cognitive and social sciences or humanities, as long as they have an intrinsic drive to engage with machine learning and technological challenges. The book’s aim is to introduce and review the advancements of techniques and ap- plications in the field of agentic LLMs. We discuss the major concepts of agentic LLM engineering in some depth. We provide references to the scientific literature as well as links to source code and datasets so that you can experiment with building your own agents. Note that the field is new, and still very much in motion. A thorough under- standing of the basics is important in order to be able to follow the latest developments. Our aim is to provide you with both. We use this book for teaching the course Agentic LLMs that is part of the Mas- ter programs Computer Science (AI-track) and Creative Intelligence & Technology at Leiden University. A website with support material is available (lecture slides, assign- ments, news). 1 Dear reader, we hope that you will enjoy reading this book as much as we did writing it. Agentic LLM research is a new field, and this book is just as fresh. New developments occur every day. We very much welcome your feedback and suggestions for following editions of this book, via the website! 1.4 Prerequisites This book assumes that you have an undergraduate-level understanding of computer programming, artificial intelligence, machine learning, and natural language process- ing. We assume that you know how to use Python and PyTorch. We also assume that you understand principles of machine learning, such as the difference between clas- sification and regression, the difference between a training set and a test set, and the causes of overfitting and underfitting. You should know classical machine learning approaches such as decision trees, ensembles, and random forests. You should also be familiar with concepts of deep learning such as the architectures of artificial neu- ral networks (fully connected multilayer perceptrons, convolutional neural networks, and recurrent neural networks), learning by backpropagation, with loss functions (such as cross-entropy) and activation functions (such as ReLU). We also expect that you are familiar with reinforcement learning: you should know what a sequential decision problem is, understand the difference between agent and environment, and know how to use some policy learning algorithms (such as Q-learning, DQN, PPO, and SAC). Finally, you should have an understanding of natural language processing: what is tokenization, what are word embeddings, what is attention, and what is Huggingface. However, we also believe that if you do not have in-depth knowledge about some or all of these topics and concepts, you may still find value in reading this book. We provide brief overviews of essential topics, you will learn about agentic LLMs at a 1 https://agenticllms.liacs.nl 1.5. COURSE STRUCTURES19 worthwhile level, and find entry points for further study. For some of the topics, you may find it useful to have a suitable textbook at hand, 2 such as: • for classical machine learning: Alpaydin [2021], Flach [2012], • for deep supervised machine learning: Bishop and Bishop [2023], Goodfellow et al. [2016], Chollet and Chollet [2021], Stevens et al. [2020], • for reinforcement learning: Sutton and Barto [2018], Graesser and Keng [2019], Plaat [2022], Franc ̧ois-Lavet et al. [2018], • for natural language processing and transformers: Jurafsky and Martin [2025]. 1.5 Course Structures We use this book for the graduate course Agentic LLMs at our university. The book has been written to be both concise and comprehensive: we cover the full breadth of the field at a level suitable for a graduate course. To allow you to go deeper, we provide many pointers to the scientific literature, and some advanced topics are covered in the chapters and sections that are marked with a *. Different university programs may cover different material (see Table 1.1). The course that we teach is comprehensive, and consists of 14 lectures: an introductory lecture, then 3 lectures on part I (language modeling), 3 lectures on parts I (reasoning) and I (agents), 3 lectures on part IV (tools and assistants), and 3 lectures on part V (social simulation). The course ends with a final evaluation lecture. Our course includes as assignments the four case studies, which we grade in an in-person meeting with the student. We do not have a separate theory exam at the end, the course grade is the average of the assignment grades. Although you are of course free to use the book in any way that makes sense in your situation, we cover two main use cases for agentic LLMs: (1) assistants and (2) social simulation. When your program includes a separate in-depth course on LLMs, then a core program to focus on (1) agentic assistants can include just parts I, I and IV. For the assignments, we suggest case study 2 and 3. Alternatively, a core program to focus on (2) social simulation should cover parts I and I, and part V, using case study 2 and 4 for the assignments. Both the assistants and the social simulation approach can be augmented with part I (language modeling), as necessary, depending on how the education program at your University is structured. Part VI contains advanced topics. Chapter 19, on synthetic data generation, is suit- able for the assistants use case, Part IV. Chapter 21, on mechanistic interpretability, is suitable to go more in depth for Parts I and I, the language and reasoning mod- els. Chapter 20, on advanced architectures, is also suitable for Part I. Chapter 22, on cognitive architectures, is suitable for the social simulation use case, Part V. Table 1.1 summarizes the different options for use of the material in this book for three different course goals. 2 See Gunasekar et al. [2023] 20CHAPTER 1. INTRODUCTION Course Goal PartsCase StudyChaptersOptional ComprehensiveI, I, I, IV, V1, 2, 3, 41-1819-22 Core Assistants I, I, IV2, 36-142-5, 19 Core Social SimulationII, I, V2, 46-10, 11, 15-182-5, 22 Table 1.1: Mapping different Course Goals to Parts and Chapters in this Book 1.6 Approach, Scope and Overview Our approach is hands-on. For all new concepts we provide the theoretical background and we give examples of how to use them in practice. We discuss datasets, training algorithms, benchmarks, training environments, we provide pseudo-code of important agent algorithms, and we provide practical case studies to help you learn. This book is organized in chapters that each cover a comprehensive topic. The chapters are organized in seven parts. Part I provides a basic introduction to large language models. Chapter 2 discusses important natural language processing concepts such as tokenization, embeddings, re- current neural networks, the attention mechanism, the transformer architecture, and autoregressive training (the machine learning method that is central to how LLMs are trained). We also discuss further machine learning methods: supervised learning and reinforcement learning. Next, in Chapter 3 we discuss in some detail the training pipeline that is used to train LLMs. We discuss dataset acquisition, pretraining, finetun- ing, instruction tuning, value alignment, and inference. We also discuss benchmarks that are commonly used to test the performance of LLMs. In Chapter 4 we discuss scaling laws and small language models, and we discuss energy consumption. To be able to use agentic LLMs, you must understand the basics of pretraining and finetuning a transformer model. In the final chapter of Part I, we help you acquire this experience with a practical case study where we will guide you to train a transformer model from scratch. Part I provides an introduction into the concepts of reasoning, reflection, and world models, three concepts that are essential elements of (advanced) agentic LLMs. In Chapter 6 we start with in-context learning, and discuss how the chain of thought method has enabled LLMs to perform multi-step reasoning in simple math word prob- lems. We will discuss how LLMs are used to self-reflect on their own in-context learn- ing, to enhance their reasoning power, using a harness. So far, we have focused on models that use language. To act in the real world, our agents will also have to un- derstand image data, and other modalities. In Chapter 7 we go beyond language and discuss models that understand vision and language, and also models that learn from decision sequences and know how to perform robotic actions. To act in the world with- out walking into walls, agents need some understanding of physics. In Chapter 7 we discuss world models, including their relation to model-based reinforcement learning. At that point, we will have discussed a multitude of different ways in which the agents learn and interact with their environment. In Chapter 8 we discuss in more depth how they learn, and how this learning is a form of metalearning. Next, in Part I, Chapter 9 provides a historic review of agents in the perspective 1.6. APPROACH, SCOPE AND OVERVIEW21 of artificial intelligence, and we discuss different agentic interaction styles of agentic LLMs. Furthermore, in order to act in the world, agents need a form of memory. We discuss different agentic memory architectures in this chapter. We conclude Part I with a practical case study, where you can build a self-reflecting agent for challenging puzzles and games from the BALROG benchmarking suite. In Part IV we switch gears, from theoretical concepts to practical implementations. We provide a discussion of the frameworks that have been developed in the commu- nity to implement agentic LLMs, and for them to use tools to act in the world. Some are proprietary, but many are open, and can be downloaded from Github. In this part, we discuss agents that act and perform practical tasks for their users. In Chapter 11 we discuss toolchains, frameworks and APIs that have been developed to implement agentic LLMs. In Chapter 12 we discuss retrieval augmentation methods and other methods to search for up to date information that was not available when the LLM was trained. Chapter 13 provides an overview of practical applications of agentic LLMs. We discuss applications of LLMs such as shopping assistants, medical assistants, trad- ing assistants, and assistants that help perform scientific research. In the final chapter of Part IV we conclude with a practical case study of how to build a medical diagno- sis agent (also to help you understand its shortcomings, so that you will never blindly follow an agent’s medical advice). Part V takes a different point of view: we look at agentic LLMs through a behav- ioral and cognitive lens, rather than a technical one. After defining and introducing machine-behavioral research in general, Chapter 15 discusses single LLMs and agen- tic LLMs in controlled, experimental settings to test their social behavior. Chapter 16 shifts to multi-agent systems, first discussing role-based interactions, where agentic LLMs are built or prompted to perform a certain role and interact in goal-directed ways. Next, in Chapter 17 we move to systems comprising multiple agents that in- teract in open-ended ways and study emergent behaviors, such as cooperation and norm-enforcement. The case study allows you to experience social agentic behavior in-depth. The next part, Part VI, contains more advanced material. Chapter 19 discusses synthetic data generation. Like all forms of deep learning, LLMs remain to a large extent a black box, and we discuss techniques that can lift a corner of the veil. Chap- ter 20 discusses advanced architectures. We discuss transformers that are trained on decision traces, and looped and recursive LLMs. We also discuss advanced agentic ar- chitectures: in-context reinforcement learning, self-modifying agents and meta agents. Chapter 21 focuses on tools from mechanistic interpretability to map patterns of neu- ronal activity in transformer architectures, allowing to better understand the reasoning processes of LLMs. In Chapter 22 we review recent literature on behavior and cog- nition in Agentic LLMs, including advanced questions on Theory of Mind and con- sciousness. Finally, in Part VII we draw together the elements of agentic LLMs that are pro- vided in this book. In Chapter 23 we discuss what agentic LLMs currently can do, what they still can not do, and we conclude with a vision on what the future of agentic LLMs might bring. Note that some of the Chapters and Sections are marked with a *. These contain more advanced material, that might be better suited for a thesis than for a course. 22BIBLIOGRAPHY 1.7 Acknowledgments This book grew out of our fascination with agentic LLMs, a fascination that caused many discussions with students and colleagues, as well as exciting research. At some point we felt that we had to write this book and to develop a graduate course. We believe that agentic LLM is a topic of great importance, that many students in AI will want to learn about. Some of the material in this book is based on and inspired by earlier work (see, for example, van Duijn et al. [2023], Van Der Meer et al. [2024], Huisman et al. [2021], Moerland et al. [2023], Plaat [2022], Plaat et al. [2025b,a], van Stein and B ̈ ack [2024], Chen et al. [2026], van Wetten et al. [2025], Xing et al. [2025]). We thank all co-authors of these works—this book could not have been written without you! The case study on a society of generative agents in Chapter 18 was written and de- veloped by Dorota Morav ˇ cikov ́ a and Friso van der Veen (based on [Park et al., 2023]), who adapted their existing work on this topic to fit this book. We are also grateful for their generous efforts and sharp comments on earlier chapters. We thank our students and colleagues at our research institute. In particular, we thank Thomas B ̈ ack, Joost Batenburg, Joost Broekens, Xi Chen, Lennard Froma, Bas Haring, Anna Kononova, Tom van Kouwenhoven, Thomas Moerland, Rob van Nieuw- poort, Mike Preuss, Peter van der Putten, Zhaochun Ren, Derya Soydaner, Suzan Ver- berne, Fien van Wetten, Annie Wong, and Tarik Za ˇ ciragi ́ c for many fruitful discussions. Bibliography Ethem Alpaydin. Machine learning. MIT press, 2021. 19 C.M. Bishop and H. Bishop. Deep Learning: Foundations and Concepts. Computer Science. Springer International Publishing, 2023. ISBN 9783031454684. URL https://books.google.nl/books?id=0uTgEAAAQBAJ. 19 Xi Chen, Aske Plaat, and Niki van Stein. How does chain of thought think? mech- anistic interpretability of chain-of-thought reasoning with sparse autoencoding. In AAAI2026, arXiv preprint arXiv:2507.22928, 2026. 22 Francois Chollet and Franc ̧ois Chollet. Deep learning with Python. simon and schuster, 2021. 19 Peter Flach. Machine learning: the art and science of algorithms that make sense of data. Cambridge university press, 2012. 19 Vincent Franc ̧ois-Lavet, Peter Henderson, Riashat Islam, Marc G Bellemare, and Joelle Pineau. An introduction to deep reinforcement learning. arXiv preprint arXiv:1811.12560, 2018. 19 Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016. 19 BIBLIOGRAPHY23 Laura Graesser and Wah Loon Keng. Foundations of deep reinforcement learning: theory and practice in Python. Addison-Wesley Professional, 2019. 19 Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C ́ esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, S ́ ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https://arxiv.org/abs/2306.11644. 19 Mike Huisman, Jan N Van Rijn, and Aske Plaat. A survey of deep meta-learning. Artificial Intelligence Review, 54(6):4483–4541, 2021. 22 Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Intro- duction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models. Stanford, 3rd edition, 2025. URL https: //web.stanford.edu/ ̃ jurafsky/slp3/. Online manuscript released Au- gust 24, 2025. 15, 19 Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model- based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1):1–118, 2023. 22 Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behav- ior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. 22 Aske Plaat. Deep reinforcement learning. Springer, 2022. 19, 22 Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg. Agentic large language models, a survey. Journal of Artificial Intelligence Research, arXiv preprint arXiv:2503.23037, 84:29:1–29:74, 2025a. 22 Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas B ̈ ack. Multi-step reasoning with large language models, a survey. ACM Computing Surveys, arXiv preprint arXiv:2407.11511, 58(6):160:1–160:35, 2025b. 22 Eli Stevens, Luca Antiga, and Thomas Viehmann. Deep Learning with PyTorch: Build, train, and tune neural networks using Python tools. Manning, 2020. 19 Ilya Sutskever. Acceptance speech for test of time award. In Advances in Neural Information Processing Systems, Vancouver, British Columbia, Canada, 2024. 17 Richard S Sutton and Andrew G Barto. Reinforcement Learning. Adaptive Compu- tation and Machine Learning series. Bradford Books, Cambridge, MA, 2 edition, November 2018. 19 Michiel Van Der Meer, Enrico Liscio, Catholijn Jonker, Aske Plaat, Piek Vossen, and Pradeep Murukannaiah. A hybrid intelligence method for argument mining. Journal of Artificial Intelligence Research, 80:1187–1222, 2024. 22 24BIBLIOGRAPHY Max J van Duijn, Bram van Dijk, Tom Kouwenhoven, Werner de Valk, Marco R Spruit, and Peter van der Putten. Theory of mind in large language models: Examining performance of 11 state-of-the-art models vs. children aged 7-10 on advanced tests. arXiv preprint arXiv:2310.20320, 2023. 22 Niki van Stein and Thomas B ̈ ack. Llamea: A large language model evolutionary algo- rithm for automatically generating metaheuristics. arXiv preprint arXiv:2405.20132, 2024. 22 Fien van Wetten, Aske Plaat, and Max van Duijn. Baba is llm: Reasoning in a game with dynamic rules. arXiv preprint arXiv:2506.19095, 2025. 22 Peiwen Xing, Aske Plaat, and Niki van Stein. Cocomposer: Llm multi-agent collabo- rative music composition. arXiv preprint arXiv:2509.00132, 2025. 22 Part I Language Modeling 25 27 In this first part we discuss the basics of language modeling and machine learning. We start with basic principles of how language can be modeled, using tokens and em- beddings. Next, the attention concept is discussed, a concept that is at the foundation of the transformer architecture, and the success of large language models. We will also review two important machine learning techniques: supervised learning and reinforce- ment learning. In the next chapter, we become more concrete, and discuss the steps of a typical training pipeline, from pretraining and finetuning to inference. We also discuss common benchmarks in that chapter. The final content chapter of this first part is about the large part of large language models: we discuss how performance of LLMs scales with parameter count, training data and compute, and we delve into efficient models that benefit from scale while minimizing cost. At the end of the part, we turn to a hands-on case study, where you will learn how everything comes together when you create your own tiny large language model from scratch. 28 Chapter 2 Foundations of Language Modeling with LLMs What You Will Learn In this chapter you will learn how natural language can be processed in machine learning models. Specifically, you will be able to: • Describe the core concepts of language modeling, including tokenization and early word-representation methods, and explain what problem each was de- signed to solve. • Distinguish the learning regimes used to build a language model—supervised, autoregressive, and reinforcement—and state what each contributes to the final model. • Explain how embeddings represent word meaning as vectors, and predict how semantically related words are positioned relative to one another. • Explain how the attention mechanism forms flexible, context-dependent connec- tions between tokens, and trace how a query, key, and value interact for a given input. • Describe how the transformer architecture is built from attention, and identify its main components. • Contrast encoder (classification) and decoder (generation) models, and select the appropriate one for a given NLP task. 2.1 Processing Natural Language Language is the primary medium through which humans communicate. Language comes in forms spanning from printed books, digital media to spoken discourse. Lan- guage research has long been a focal point of multidisciplinary research. Within the context of artificial intelligence and computer science, language is of particular signifi- cance: we use it to articulate complex ideas, we formulate algorithms in programming 29 30CHAPTER 2. FOUNDATIONS OF LANGUAGE MODELING Figure 2.1: A typical conversation with ELIZA languages, and we use it to facilitate interaction between software and its users. For decades, a key aim of AI has been to unlock the ability for machines to under- stand human-produced language, as is argued by Jurafsky and Martin [2025] in their major textbook of the field. Research in this area is combined in the field of Natu- ral Language Processing (NLP) and its counterpart, Computational Linguistics. 1 In this chapter, we provide an overview of the complexities of capturing natural (human) language by computational means, and outline the evolution of various fundamental techniques over the years for achieving such understanding. One of the early goals of the fields has been to achieve the dream of practical ma- chine translation. Since its inception in the 1950’s [Hutchins, 1999], human language processing technology has evolved greatly. Extensive research progress has allowed it to have a profound impact across industries. This widespread industrial adoption has, in turn, emerged as a vital driving force for application-oriented development. In NLP, fundamental theoretical research and practical, real-world utility, go hand in hand. As a consequence, the application areas have expanded beyond machine translation. Recently, the impact of language technology on society has truly exploded. We interact on a daily basis with chatbots such as ChatGPT and Claude. We can do so in multiple languages, and enhanced search engines have been incorporated in many popular products. However, these developments stand on the shoulders of many earlier results. A famous early example is ELIZA, the psychotherapist chatbot developed in 1966 by Joseph Weizenbaum [Weizenbaum, 1966]. ELIZA was a rudimentary chatbot based on simple rule-based word substitutions (see Figure 2.1). ELIZA was already ascribed human-like qualities. Since these early days of rule-based systems for text processing, where we identify words based on specific delimiters (such as white space), we have come a long way. 1 Originally further apart, the two fields have been growing closer in recent years, one starting from the linguistic side, the other from the computational side. 2.1. PROCESSING NATURAL LANGUAGE31 Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland USA, June 23-24, 2014. c 2014 Association for Computational Linguistics The Stanford CoreNLP Natural Language Processing Toolkit Christopher D. Manning Linguistics & Computer Science Stanford University manning@stanford.edu Mihai Surdeanu SISTA University of Arizona msurdeanu@email.arizona.edu John Bauer Dept of Computer Science Stanford University horatio@stanford.edu Jenny Finkel Prismatic Inc. jrfinkel@gmail.com Steven J. Bethard Computer and Information Sciences U. of Alabama at Birmingham bethard@cis.uab.edu David McClosky IBM Research dmcclosky@us.ibm.com Abstract We describe the design and use of the Stanford CoreNLP toolkit, an extensible pipeline that provides core natural lan- guage analysis. This toolkit is quite widely used, both in the research NLP community and also among commercial and govern- ment users of open source NLP technol- ogy. We suggest that this follows from a simple, approachable design, straight- forward interfaces, the inclusion of ro- bust and good quality analysis compo- nents, and not requiring use of a large amount of associated baggage. 1 Introduction This paper describe the design and development of Stanford CoreNLP, a Java (or at least JVM-based) annotation pipeline framework, which provides most of the common core natural language pro- cessing (NLP) steps, from tokenization through to coreference resolution. We describe the original design of the system and its strengths (section 2), simple usage patterns (section 3), the set of pro- vided annotators and how properties control them (section 4), and how to add additional annotators (section 5), before concluding with some higher- level remarks and additional appendices. While there are several good natural language analysis toolkits, Stanford CoreNLP is one of the most used, and a central theme is trying to identify the attributes that contributed to its success. 2 Original Design and Development Our pipeline system was initially designed for in- ternal use. Previously, when combining multiple natural language analysis components, each with their own ad hoc APIs, we had tied them together with custom glue code. The initial version of the Tokeniza)on* Sentence*Spli0ng* Part4of4speech*Tagging* Morphological*Analysis* Named*En)ty*Recogni)on* Syntac)c*Parsing* Other*Annotators* Coreference*Resolu)on** Raw* text* Execu)on*Flow* Annota)on* Object* Annotated* text* (tokenize)* (ssplit)* (pos)* (lemma)* (ner)* (parse)* (dcoref)* (gender, sentiment) Figure 1: Overall system architecture: Raw text is put into anAnnotationobject and then a se- quence of Annotators add information in an analy- sis pipeline. The resultingAnnotation, containing all the analysis information added by the Annota- tors, can be output in XML or plain text forms. annotation pipeline was developed in 2006 in or- der to replace this jumble with something better. A uniform interface was provided for anAnnota- torthat adds some kind of analysis information to some text. AnAnnotatordoes this by taking in an Annotationobject to which it can add extra infor- mation. AnAnnotationis stored as a typesafe het- erogeneous map, following the ideas for this data type presented by Bloch (2008). This basic archi- tecture has proven quite successful, and is still the basis of the system described here. It is illustrated in figure 1. The motivations were: •To be able to quickly and painlessly get linguis- tic annotations for a text. •To hide variations across components behind a common API. •To have a minimal conceptual footprint, so the system is easy to learn. •To provide a lightweight framework, using plain Java objects (rather than something of heav- ier weight, such as XML or UIMA’s Common Analysis System (CAS) objects). 55 Figure 2.2: The standard NLP pipeline, as visualized by Manning et al. [2014] 2.1.1 Language Understanding and Language Generation Before considering how computers process language, we will consider how humans understand language. A key observation is that communication is a sequential process, and that languages are shaped accordingly. Words are uttered one at a time, and are also consumed one at a time. This parallel holds both for human and mechanical processing of language: language is a sequence of characters (when reading), syllables (when speaking), or entire words (for early computational models of language). We will now look deeper into the technology developed for working with natural language sequences. This technology is categorized as either natural language under- standing or natural language generation. We will briefly describe both categories. Natural Language Understanding Natural language understanding (NLU) focuses on developing methods for capturing meaning and intent. Approaches and techniques for NLU map unstructured human language elements onto a structured format such that a computer can interpret and act upon them. Such techniques can involve several layers of processing, inspired by linguistic analysis (as visualized in Figure 2.2), as follows: • Tokenization: determine boundaries between meaningful linguistic elements (such as words) to be used for further processing. • Syntactic Parsing: determine the grammatical structure (such as subjects, verbs, and objects). • Named Entity Recognition (NER): identify and categorize key entities such as names, dates, and locations. • Sentiment Analysis: detect the emotional tone of a text. • Co-reference Resolution: determine what person or items are referred to in ref- erential expressions (such as: the computer talked to me). 32CHAPTER 2. FOUNDATIONS OF LANGUAGE MODELING • Discourse Analysis: understand how the meaning of a sentence is influenced by preceding sentences. One of the primary challenges in NLU is dealing with ambiguity [Roth, 1998, Liu et al., 2023]. Human language is rife with ambiguity. Examples are polysemy (words with multiple meanings, such as bank) and context-dependent nuances that require world knowledge to decode correctly. Many more challenges exist; naturally occurring human language is as rich and complex as humans are. Natural Language Generation Natural language generation (NLG) focuses on de- veloping methods for producing readable, natural-sounding human language. Tasks range from free-form language generation, to the transformation of structured data or model representations into realistic utterances. An important NLG task is language modeling (LM). In language modeling, the task is to predict the next element in a sequence, given the previous context. Mathe- matically, this involves assigning a probability distribution over sequences of words. Traditionally, finding this distribution was achieved using simple count-based models, often relying on further simplifying assumptions to predict a word based on a fixed window of preceding words. Modern approaches use elaborate neural methods that can generate fluent sentences based on a long context, that often appear meaningful to a human observer. 2 Note that effective language generation requires more than just grammatical correctness; it re- quires coherence (logical flow) and cohesion (linguistic links). Current NLG systems use sampling methods, ensembles, and many other mechanisms to balance the creativ- ity and predictability of the text that they produce, to ensure that the output feels natural rather than robotic [Li and Jurafsky, 2017, Dong et al., 2022, Gatt and Krahmer, 2018]. Techniques for NLG are commonly categorized into two approaches: • Extractive Generation: Selecting and rearranging existing segments of text (often used in early summarization tools). • Abstractive Generation: Generating entirely new sentences from scratch (as mod- ern models such as ChatGPT do). Most LLMs use the abstractive approach. We will now discuss early, fundamental, approaches to language modeling. We will start by discussing the complexity of language, and then discuss the principle of distributional representation, tokenization methods, and early language modeling representations. 2.1.2 Complexity of Natural Language There are many reasons why natural language can be difficult for a machine to interpret. Language exhibits subtle nuances that carry meaning or distort it. Capturing the full nuance of language requires more than understanding its structure. Furthermore, the 2 Which is by itself not always difficult to achieve, as we saw in the ELIZA example. 2.1. PROCESSING NATURAL LANGUAGE33 context in which something is said plays a role in communication. Understanding goes beyond the words themselves to grasp the intentions, implications, and subtleties embedded in communication [Wittgenstein, 1953, Grice, 1975, Krause and Vossen, 2024]. Below, we introduce topics that are studied in the field of linguistics, each of which brings about a different view on what language is capable of expressing. Syntax Syntax rules govern the structure of sentences. In the study of linguistics, syntax refers to the set of rules, principles, and processes that regulate the structure of sentences in a given language. Linguistic syntax examines the underlying logic that allows a speaker to combine words into well-formed and meaningful sequences. In English, for example, the most common order is subject-verb-object: in cat bites dog the verb bites follows the subject cat, indicating that the meaning of the sentence is that it was the cat that was doing the biting (and not the dog). Morphology Morphology is the study of how words change shape depending on their grammatical function. How words change shape is related to syntax, and the morphological rules (and their exceptions) depend on the language (family). In the preceding example sentence, the shape of the verb to bite changed to bites, where the -s indicates simple present tense (it is happening now, by a singular actor, not by a posse of felines in ancient times). Semantics In linguistics, lexical semantics is the study of the meaning of words. In natural languages words derive their meaning from their relationship to other words. For instance, words with nearly identical meanings form synonyms, such as car and automobile or vanish and disappear; words with opposite meanings form antonyms, such as hot and cold or buy and sell. Many more relationships exist. In NLP, computational semantics is the study of how we can represent that mean- ing efficiently in a computer. A basic approach for storing a sentence in a computer memory would be to simply use a sequence of indices into a vector of the words of a language. However, this encoding only records the occurrence of the words, but not the relationships between the words; the semantics of the words are no longer present in this naive representation. Words derive their meaning from their relationship to other words, and we need a way to efficiently encode these relationships. As we will see later in this chapter, different solutions have been developed over the years, including the transformer ar- chitecture and the attention mechanism. Pragmatics In addition to the literal meaning of words, humans care about the intent of the speaker and the context of the utterance. The way language is employed in spe- cific social contexts determines its effect (meaning is use [Baker and Hacker, 2008]). The study of the implicit, everyday rules humans naturally follow to ensure that our communication is effective is called pragmatics. A set of foundational principles are the Gricean Maxims [Grice, 1975]: the unspo- ken cooperative principles that humans use to infer meaning. For instance, the maxim of quantity states that humans provide exactly as much information as is required for 34CHAPTER 2. FOUNDATIONS OF LANGUAGE MODELING the conversation. These maxims shape our communication, and help make our lan- guage understandable and cooperative. For a machine, interpreting sarcasm, subtext, or placeholders (e.g., “thingy”) requires knowledge of these rules, beyond adherance to a static dictionary. Discourse Language rarely occurs in isolated sentences; language is structured into larger units such as paragraphs, conversations, or full documents. The study of dis- course focuses on how sentences connect to form a cohesive and coherent whole. For a machine to truly interpret a text, it must be able to track entities across multiple sentences, a task known as co-reference resolution (e.g., determining whether the pro- noun it refers to the cat, the dog, or the bite from a previous sentence). Understanding discourse is vital for complex NLP tasks such as document summarization, machine translation, and sustaining long-form dialogue. In addition to these topics, other fields of study exist, such as phonetics (the sounds of a language that carry additional meaning in human communication), sociolinguistics (how language varies according to social factors like class or region), lexicography (the science of compiling dictionaries), and many more. As this overview may have highlighted, the generation and understanding of natural languages are large and challenging fields. It is important to realize that language is not static. Language varies over time, across geographical locations, between different cultures, and depending on the medium through which it is communicated :-). This variation makes the creation of language technology a difficult and never-ending quest. Before we introduce the recent technologies that provide at least a partial answer to these challenges, we discuss the fundamental assumptions that inspired the modern turn in NLP. Representing natural language in a computer is difficult because natural language is rich and complex. In the remainder of this section, we discuss the principle of dis- tributional representation, which states that the semantic meaning of a word is driven by the context that surrounds that word. Next, we will look at tokenization methods: methods for splitting sentences into words. Finally we will discuss how we can work with vectors of tokens to arrive at a representation that encodes meaning. 2.1.3 Distributional Representations How can we teach a machine to understand complex semantic relationships and gram- matical structures without manually encoding them all? The answer lies in the distribu- tional hypothesis, a foundational concept in modern NLP. The core idea was articulated in the late 1950s: You shall know a word by the company it keeps. (Firth, 1957) This suggests that the meaning of a word can be inferred from the words around it (con- texts) in which it typically appears. Words that appear in similar contexts are likely to have similar meanings. For example, if the words coffee and tea both frequently appear near verbs like drink, sip, or brew, then a machine may infer that there is a semantic relationship between them, without explicit definitions or other human-provided infor- mation. 2.1. PROCESSING NATURAL LANGUAGE35 Figure 2.3: Illustration of vector arithmetic with word2vec embeddings [Singerep, 2022]. To operationalize the distributional hypothesis for computational purposes, we will rely on linear algebra and turn to a framework known as vector semantics. We represent words as points in a high-dimensional continuous space. Words with similar meanings are mapped to points close to each other in this space, and we can measure semantic similarity using geometric distance measures such as cosine similarity (see Figure 2.3 and Section 2.3.1). Furthermore, we can make modifications to word meaning by doing vector arithmetic. Early implementations relied on sparse co-occurrence matrices to construct semantic vectors, where individual words were counted inside their context, much like other methods described in Section 2.1.5. Modern approaches favor dense, low-dimensional vectors, often referred to as word embeddings. These representations are typically learned by training a neural model to predict a word from its neighbors or vice versa, a breakthrough popularized by the word2vec architecture [Mikolov et al., 2013]. While learning vector-based representations for words is an effective way to capture meaning, maintaining a distinct vector for every unique word in a language results in an impractically large vocabulary. Furthermore, strict word-level approaches struggle to process rare or entirely un- seen words, as the model has no pre-existing vector for them. Hence, before a model can assign vectors to linguistic units, we must first determine what those fundamental units actually are. To address these inefficiencies and handle out-of-vocabulary inputs, modern methods use a systematic approach to break down raw text into more manage- able pieces. This process is called tokenization, which we will now discuss. 36CHAPTER 2. FOUNDATIONS OF LANGUAGE MODELING 2.1.4 Tokenization Before computers can ascribe meaning to language, they must first have a way to digest it. Assuming that we already have raw text at our disposal, whether scraped from the web or transcribed from speech, the immediate challenge is determining how to split these continuous character strings into meaningful units, or how to split sentences into words. This process is called tokenization. However, a robust tokenization strategy must do much more than simply divide sentences by spaces to obtain words; it needs to be able to handle the nuances of different languages and dialects, parse complex punctuation, and even account for modern additions like emojis, and fit these into a vocabulary of fixed size. This process of segmentation is a key step to many NLP pipelines. While dividing text might appear straightforward, ambiguities in punctuation complicate the process, such as in abbreviations (N.L.P. or NLP?). Simple rule-based segmenters frequently fail against the structural irregularities of natural language. Consequently, there are different approaches for constructing these units, as shown in Example 1. Example 1 (Tokenization Strategies). Consider the following sentence: “Leiden is beautiful in summer.” Word Tokenization This approach typically splits text by spaces and punctuation. ["Leiden", "is", "beautiful", "in", "summer", "."] Character Tokenization This approach breaks the text down into individual characters. ["L", "e", "i", "d", "e", "n", " ", "i", "s", " ", "b", "e", "a", "u", "t", "i", "f", "u", "l", " ", "i", "n", " ", "s", "u", "m", "m", "e", "r", "."] Subword Tokenization (e.g., WordPiece) This approach preserves common words as whole tokens but breaks down com- plex or rare words into foundational subwords. ["Lei", "##den", "is", "beautiful", "in", "summer", "."] Word-Level Tokenization This approach treats each space-separated string as a to- ken that is unique. While it preserves syntactic structure, it results in impractically large vocabularies and suffers from the out-of-vocabulary (OOV) problem, where models fail on words that were not seen during training. Large vocabularies are problematic due to the mechanics of one-hot encoding. If every word 푤 in a vocabulary푉 is represented as a vector x∈ 0, 1 |푉| , where only the index corresponding to 푤 is 1, the dimensionality|푉| scales linearly with the size of the 2.1. PROCESSING NATURAL LANGUAGE37 vocabulary. As these one-hot vectors will later form the input to our neural networks, for texts that do not contain all words in a certain vocabulary, we end up creating sparse, computationally inefficient representations that fail to capture semantic relationships between words. Character-Level Tokenization Pure character-level modeling seems like a clear so- lution to the problem of increasing vocabularies, and has been explored over the years [Mielke et al., 2021]. Such a method divides text into individual characters, which restricts the vocabulary to the size of the alphabet, virtually eliminating OOV issues. However, it significantly increases sequence length and the resulting computational burden is substantial. Furthermore, because isolated characters lack inherent semantic meaning, the model must learn word morphology before it can grasp broader linguistic patterns, increasing training time. Subword Tokenization Hence, the current standard for LLMs balances the previous two methods. It preserves common words as whole tokens while decomposing rare words into smaller, meaningful components. Notable implementations include: • Byte Pair Encoding (BPE): An iterative algorithm that begins with individual characters and merges the most frequent adjacent pairs into single tokens until a target vocabulary size is reached [Sennrich et al., 2016] (e.g., highest → [high, est]). BPE was originally proposed as a data compression algorithm [Gage, 1994]. • WordPiece: An iterative algorithm, similar to BPE, that merges pairs based on a likelihood criterion rather than on raw frequency, aiming to maximize the prob- ability of the training data [Wu et al., 2016]. 2.1.5 Early Methods for Language Representation After tokenization, the next question is how to combine our linguistic units (tokens) into coherent sentences, that carry meaning? We start with the observation that the ordering of these units in human language is dictated by strict syntactic and semantic rules. Now, if our computational approaches would be able to accurately produce sensi- ble text sequences, then we might say that they possess the same syntactic and semantic understanding (or at least that they are able to reproduce such understanding). How can we build mathematical representations that achieve this goal? A core task in this endeavor is language modeling. Mathematically speaking, a language model is a function that computes the joint probability of a sequence of words occurring together in a specific context. By calculating this probability, models can distinguish between a logical, fluent sentence and a random sequence of tokens. For example, in English, the sequence the cat sat is significantly more probable than sat cat the because it adheres to established syntactic rules. Because written language is sequential, estimating the joint probability of a sen- tence is inherently a sequence estimation problem. The overarching goal of language 38CHAPTER 2. FOUNDATIONS OF LANGUAGE MODELING modeling is to quantify the statistical likelihood that a given sentence, 푆, occurs within a specific language, based on its sequence of constituent words 푤 1 ,...,푤 푘 : 푃(푆) = 푃(푤 1 ,...,푤 푘 )(2.1) For sentences of arbitrary length, calculating this exact joint probability directly is computationally intractable, due to the exponentially large number of word combi- nations in any given vocabulary. To resolve this complexity, we often decompose the joint probability, using the chain rule of probability. By making specific mathematical assumptions regarding conditional independence, this decomposition allows the com- putation to remain tractable for real-world applications [Jurafsky and Martin, 2025]: 푃(푤 1 ,...,푤 푘 ) = 푃(푤 1 ) 푘 Ö 푖=2 푃(푤 푖 | 푤 1 ,...,푤 푖−1 )(2.2) By maximizing this joint probability, probabilistic models learn to favor grammat- ically and logically sound sequences. The learned probabilistic distributions effec- tively reverse engineer the complex, unwritten rules of human language [Manning and Schutze, 1999]. To estimate these probabilities from a segmented text corpus, early methods relied on maximum likelihood estimation (MLE). The simplest approach com- putes the marginal probability of a word based on its relative frequency in the corpus: 푃(푤) = count(푤 in corpus) total number of words (2.3) To make the computation of these conditional probabilities manageable, researchers rely on 푛-gram models. An 푛-gram is a contiguous sequence of 푛 items from a given sample of text. Instead of calculating the probability of a word given its entire preced- ing history, which quickly becomes computationally intractable, 푛-gram models apply the Markov assumption. This assumption states that the probability of the next word depends only on the preceding 푛 − 1 words, effectively truncating the context. For example, a bi-gram model (푛 = 2) approximates the history by looking solely at the immediately preceding word. The conditional probability is then calculated by taking the frequency of the two words occurring together and dividing it by the frequency of the first word alone, 푃(푤 푖 | 푤 1 ,...,푤 푖−1 ) ≈ 푃(푤 푖 | 푤 푖−1 ) = count(푤 푖−1 ,푤 푖 ) count(푤 푖−1 ) (2.4) Likewise, a tri-gram model (푛 = 3) conditions on the previous two words. This simplification drastically reduces the computational burden and makes it feasible to estimate probabilities directly from a corpus. Selecting the appropriate 푛 involves a critical trade-off. A larger 푛 captures richer contextual information and deeper syntactic dependencies, resulting in a model that generates a more natural language. Yet, increasing 푛 exponentially exacerbates the sparsity problem: highly specific word sequences appear far less frequently in the train- ing data, and the model will increasingly encounter zero probabilities. Conversely, a smaller 푛 yields robust, frequent counts but severely limits the model’s understanding of broader semantic meaning and long-distance dependencies within the sentence. 2.2. MACHINE LEARNING METHODS FOR NATURAL LANGUAGE PROCESSING39 In a standard language model computation, observing an unseen 푛-gram results in a probability of zero, as it nullifies the joint probability of the entire sentence via mul- tiplication. To mitigate this counterintuitive behavior, total counts are often smoothed. A common technique is Laplace (or add-one) smoothing, which adds a fictitious count of 1 to every possible vocabulary item. To maintain a valid probability distribution, the denominator is adjusted by adding the vocabulary size,|푉|: 푃 Laplace (푤) = count(푤 in corpus)+ 1 total number of words+|푉| (2.5) These probabilistic and count-based methods ultimately yield a numerical repre- sentation of language. A strong representation encodes the properties we intuitively grasp about a word: its meaning, its grammatical function, and its relationship with other words. In early NLP, these relative frequencies and co-occurrence counts were structured into vectors, representing words based on their distribution in the text. While count-based representations are tractable, they result in vectors that are high- dimensional and sparse, leading to computational inefficiencies and poor generalization on unseen data. Consequently, modern NLP has transitioned toward learning dense, low-dimensional vector representations, known as embeddings, using neural networks. Instead of relying on explicit counting, these models learn continuous representations as a byproduct of predicting a word from its surrounding context, capturing deeper semantic relationships in a more efficient format. These modern approaches will be discussed in Section 2.3. First we will have a brief intermezzo where we review three important machine learning approaches. 2.2 Machine Learning Methods For Natural Language Processing Having established a primer on language representation, we will now turn to the learn- ing algorithms that allow us to create the large models powering modern AI systems. LLMs are trained using methods from the broader field of machine learning. While machine learning is an extensive field of research, understanding LLMs requires fa- miliarity with three core paradigms, and how they are used in language modeling: supervised learning, autoregressive learning, and reinforcement learning. We briefly touch upon these essential methods and refer to standard works for deeper exploration. 2.2.1 Supervised Learning Supervised learning is the foundation of machine learning. It is named so because it relies on a set of labels, that must be given beforehand, that supervise the training process. This approach utilizes a dataset consisting of paired examples and labels, 퐷 = (푥 푖 , 푦 푖 ), to train the parameters 휃 of a predictive model 푓 휃 . The model can be used to compute an estimate ˆ푦 for each input 푥, denoted mathematically as 푓 휃 (푥) → ˆ푦. Using the model to compute this estimate is also known as inference, or the forward pass. 40CHAPTER 2. ML METHODS FOR NLP After the forward pass, we can use a loss function L to compute some kind of difference between the estimate ˆ푦 and the true label 푦. Common examples of loss functions are the sum of squared-difference loss from statistical regressionL = Í 푖 ( ˆ푦 푖 − 푦 푖 ) 2 , or the cross-entropy loss for classificationL =− Í 푐 푦 푐 log( ˆ푦 푐 ). The goal of a supervised training algorithm is to minimize the sum of the losses over the dataset of example pairs by adjusting the parameters. This is done in the backward pass. Here, the loss function is used to compute an error signal to adjust the parameters 휃 of the model 푓 so that the next forward pass achieves a lower loss (backpropagation of the error). Minimizing the loss function on the examples from a data set is a process called training. A single training run of forward and backward passes over all of the training data is called an epoch (for efficiency reasons, epochs can be split into training batches). After training, to see if our model 푓 휃 works well—if it generalizes to unseen examples—it is tested on new examples, that were not part of the training set. If the accuracy on the training set is high, we say that the model trains well (training converges). If the accuracy on the test set is also high, then we say that the model generalizes well and we have achieved our training goal. Informally speaking, if the accuracy of the model on the training set is high, but on the test set it is low, we say the model overfits on the training set. This usually means that the model has trained for too long or has too many parameters, so that it models not only the signal in the training set but also the noise. If the model has low accuracy on both training and test set, we say that it underfits. This is most likely due to too short training, or a model that is too simple (too few trainable parameters) to model the complexities of the data. Generally speaking, problems with few variables (low-dimensional problems, such as simple regression) require smaller models (fewer parameters) and less training data to perform well. Problems with many variables (high-dimensional problems, such as image classification) require larger models with many millions of parameters (often deep neural networks) and large training datasets to perform well. Indeed, break- through performance in image classification [Krizhevsky et al., 2012] was only achieved after the construction of a large dataset with millions of hand-labeled images, Imagenet [Deng et al., 2009]. Supervised learning is the basis of machine learning. Many good text books have been written about the topic [Bishop and Bishop, 2023, Goodfellow et al., 2016]. It is one of the core techniques in LLM training, for example in supervised finetuning, as we will see in the next chapter. 2.2.2 Autoregressive Learning Large language models are trained autoregressively. In this training approach, there is no dataset of pairs of labeled examples, but there is a lot of (textual) data available. Autoregressive learning is a form of self-supervised learning, without explicit labels, using the inherent sequential nature of the data, to predict the next token. In autoregressive pretraining of language models, our algorithm performs predic- tions over the dataset where each token 푦 = 푥 푡 is taken as a label for preceding example tokens x = 푥 푡−1 . Autoregressive learning is a core task of modern NLP, and is used in contemporary model training pipelines [Radford et al., 2018] and included in textbooks 41 Environment: 푇(푠,푎) → 푠 ′ , 푅(푠 ′ ) → 푟 Agent: 푟 improves 휋 휃 (푠) → 푎 (푠 ′ ,푟) 푎 Figure 2.4: Reinforcement Learning, after Sutton and Barto [2018] on NLP [Jurafsky and Martin, 2025]. We will provide more details of the methods in Section 2.3 and Chapter 3. 2.2.3 Reinforcement Learning Both supervised and autogregressive learning need a dataset of training data to train on—they train offline on pre-existing data. However, sometimes no dataset is available, but we may have access to a function that can tell us when we are in a good state or in a bad state. In that case, an online learning method called reinforcement learning (RL) can be used, where an agent performs an action 푎 to bring an environment from state 푠 to the next state 푠 ′ , after which the environment reports back the reward 푟(푠 ′ ) of that new state. (The method is called online because it generates its training data on the fly, while training.) The agent can then use this reward information to adjust a 휃-parameterized behavior policy function 휋 휃 (푠) → 푎 to know which action to perform in which state. When the problem at hand is as a sequential decision problem, actions 푎 take the system from state 푠 to another state 푠 ′ , and rewards depend on actions taken in the past (the credit assignment problem: to which actions should rewards be assigned?). Repeating the state/action/reward process can cause the agent to explore the entire state space, and learn an optimal policy 휋 ∗ (푠) of which action to take, for all states. Note that reinforcement learning, being online, is inherently circular. At the start, the policy 휋 has seen few rewards, and has useful information on which action to take for just a few states. If it would greedily use this information on which good action to take, it would traverse only a limited part of the environment, and the optimal policy that it would find would be only locally optimum. To find better optima, it should explore, and try random actions, ideally to cover the entire state space. Reinforcement is a form of active agentic learning from feedback from an environ- ment. A key observation is that it is typically easier to see if an output is good (reward function), than to create a full new one (labeled dataset). Many reinforcement learning algorithms have been developed that focus on good exploration methods. Reinforce- ment learning problems are formally modeled as a Markov decision process (MDP), consisting of a state representation 푆, an action function 퐴(푠), a transition function 푇(푠,푎) → 푠 ′ , and a reward function 푅(푠). Rewards are numeric indications of the quality of a state, provided by the environment. They can be positive or negative. The goal of an agent is to find the policy 휋(푠) of optimal actions 푎∗ in each state 푠. Fig- ure 2.4 illustrates how a reinforcement learning agent learns a policy with an action for each state, online, through interaction with the environment, which can compute 42CHAPTER 2. ML METHODS FOR NLP functions 푇 and 푅 (the environment will answer the agent’s question as to what the next state is that resulted from its action, and how good that state is). When the state space is small enough to fit in a computer memory, we can use ex- act tabular reinforcement learning methods, such as Q-learning [Watkins and Dayan, 1992]. For larger problems, function approximation methods are used (deep neural networks) such as deep Q-learning (DQN) [Mnih et al., 2015] or proximal policy op- timization (PPO) [Schulman et al., 2017]. In deep reinforcement learning, the policy is a parameterized neural network that is trained by backpropagation, as in supervised deep learning. Reinforcement learning is the basis of modern reasoning language models. Many text books have been written about reinforcement learning, such as the seminal work by Sutton and Barto [2018], as well as more educational texts [Graesser and Keng, 2019, Plaat, 2020, 2022]. In LLM training, reinforcement learning is used in reinforcement finetuning, and in multi-step reasoning, as we will see in Chapter 6. Furthermore, reinforcement learning formulates machine learning problems in terms of an agent that tries to learn how to interact in an environment, which fits the agentic LLM setting nicely. In agentic LLMs, it is used in in-context reinforcement learning for prompt optimization, in world models, and to model other agent behavior in relation to the environment, as we will see later in this book. Reinforcement learning is a popular machine learning approach since we can learn new environments where we do not have data available. The method finds its own data. When we consider using reinforcement learning, two questions should be an- swered: (1) is the problem at hand a sequential decision problem, and, if so, (2) what is the size of the state space, can we use exact tabular methods, or must we resort to approximation? Now that we have discussed the necessary language and machine learning princi- ples, we turn back to language, to the next step: modeling language. 2.3 Modeling Language After the basics of language representation and machine learning, we turn to the overar- ching paradigms used to build computational language modeling capabilities. The de- velopment of natural language processing is broadly characterized by three paradigms. Each paradigm represents a fundamental shift in the type of model constructed to cap- ture language and how the inherent complexity of human language is managed. We will see that as we move through the paradigms, we incorporate more techniques from machine learning, leveraging increases in compute power to process more data to train larger and larger neural networks for language processing. The paradigms are the sym- bolic, statistical, and neural approaches [Kautz, 2020, Deng and Liu, 2018]. Symbolic Methods Often referred to as Classical AI or rule-based systems, these early ELIZA-like symbolic approaches rely on explicit, human-defined instructions. By treating language as a formal system, symbolic models attempt to parse meaning deductively. While these systems are highly transparent and interpretable by design, 2.3. MODELING LANGUAGE43 they are brittle, as they struggle using rigid rules (if-then-else) to handle the natural vari- ation, idiomatic expressions, and ambiguities of real-world language. Scaling symbolic approaches to larger language tasks requires a labor-intensive, manual enumeration of many edge cases, which turned out to be unfeasible for all but the smallest problem domains. Statistical Methods To overcome the limitations of rigid rule sets, between 1980- 1990 the field experienced an empirical shift toward probabilistic modeling with the statistical methods. Instead of relying on explicit syntactic rules, statistical methods such as hidden Markov models and 푛-gram language models leverage large text cor- pora to compute the likelihood of word co-occurrences. This data-driven approach scales efficiently and handles linguistic ambiguity and noise far better than its sym- bolic predecessors. However, because these models typically rely on the Markov as- sumption, their representation of meaning is inherently limited to local word patterns, making them ill-equipped to capture long-distance dependencies and broader context. Neural Methods After approximately 2010, the contemporary state-of-the-art shifted to neural networks and deep learning. Rather than using logic or discrete counts, neu- ral methods represent linguistic units as dense, low-dimensional vectors in a continu- ous semantic space. Modern architectures, particularly transformers, employ attention mechanisms to model global context simultaneously across an entire sequence. This approach yields impressive generalization, high fluency in text generation, and cross- lingual capabilities. Nevertheless, these advantages come with significant trade-offs: neural models operate largely as black boxes, sacrificing the interpretability seen in symbolic methods, and they require large data sets and large computational resources. The contemporary pipelines for NLP training draw inspiration from all of these three paradigms. Before describing an example pipeline in the next chapter, we require a deeper understanding of the core components of these parts, as they are the building blocks for the systems on which agentic LLMs are based. In the next sections we will discuss many topics: word embeddings, contextualized embeddings, sequence modeling, attention, transformers, encoders (such as BERT), and decoders (such as GPT). 2.3.1 Embeddings Early models of neural embeddings operationalize semantic understanding of lexical items by defining a context window, a span of 푛 words to the left and right of a target word, and predicting the missing focus word. For the sentence There are two cookies left on the table, if we set a window of 푛 = 2, the context words for cookies are are, two, left, and on. By systematically sliding this window over a large corpus of text, we can count how many times each word appears in the context of every other word. These counts form the basis of a co-occurrence matrix, where each row represents a target word and each column represents a context word. The value in each cell is the number of times the column word appeared in the context of the row word. For example, the vector for cherry might have a high value in the column for pie (e.g., 442) and a low 44CHAPTER 2. ML METHODS FOR NLP value for computer (e.g., 2). The vector for strawberry would show a similar pattern, and hence reflect the semantical similarity between the two words. By representing words as vectors, we place them in a high-dimensional vector space where each dimension corresponds to a context word. In this space, words with similar meanings, which share similar contexts, will point in similar directions. This geometric relationship allows us to measure semantic similarity numerically. The most common method for this is cosine similarity, which measures the cosine of the angle 휃 between two vectors a and b, as in Equation 2.6. similarity = cos(휃) = a· b ∥a∥b∥ = Í 푛 푖=1 푎 푖 푏 푖 √︃ Í 푛 푖=1 푎 2 푖 √︃ Í 푛 푖=1 푏 2 푖 (2.6) Earlier, we argued that count-based vectors result in high-dimensional spaces that scale linearly with vocabulary size. Since the number of dimensions dictates the com- putational complexity of downstream models, we wish to keep the dimensionality in check and instead create dense embeddings. For this, we use the classical word2vec algorithm [Mikolov et al., 2013]. In this approach, a random embedding matrix 푊 is initialized and iteratively updated to match the intuition that similar words occur in similar contexts. This general framework uses two distinct techniques to achieve this: skip-gram and continuous bag-of-words, as shown in Figure 2.5. The first, skip-gram, predicts which context words are likely to surround a given target word. It updates the matrix to ensure that the cosine similarity between posi- tive samples, which are actual target and context pairs, is high, while driving down the similarity for negative samples, which are random word pairings. Because skip- gram treats every target-context pairing as a distinct training observation, it heavily penalizes incorrect predictions even for infrequent terms. Hence, skip-gram excels at capturing fine-grained semantic details and produces accurate representations even for rare words. The second, continuous bag-of-words (CBOW), performs the reverse operation. It predicts a single target word based on a combined set of surrounding context words. By averaging the context vectors before predicting the target, CBOW inherently smooths over the specific details of individual context words. This distributional smoothing is the reason for its distinct performance profile: CBOW trains significantly faster than skip-gram and is highly effective at capturing general syntactic relationships among frequent words, though it sacrifices some sensitivity to a rare vocabulary. 2.3.2 Shifting to Contextualized Embeddings Word2vec represented a major leap forward in the creation of high-quality word em- beddings. However, a major downside is that word2vec results in static embeddings. This means that a word is mapped to a single vector regardless of its usage (its con- text). This creates a fundamental problem with polysemy. Consider the word bank in two different sentences: 1. I need to go to the bank to withdraw some money. 2. The swimmer sat on the river bank. 2.3. MODELING LANGUAGE45 Figure 2.5: Visualizations of the skipgram (left) and continuous bag-of-words (right) approaches to constructing word embeddings [Zhang et al., 2023]. In a static model, bank has one fixed representation that must average or pick between these disparate meanings, resulting in a vector that is neither purely financial nor purely geographical or a weird blend of the two. Modern architectures (such as BERT and GPT, which are discussed later in this chapter) solve this by producing contextualized embeddings. Instead of a static look-up table, the model processes the entire sequence. The resulting vector for bank is calculated dynamically based on the surrounding to- kens (money, withdraw vs. river, swimmer), allowing the machine to represent the specific nuance intended by the speaker. 2.3.3 Recurrent Neural Networks While embeddings are capable of expressing the complex semantics of individual words, we still need an efficient way to ingest language as a sequence. Training models for classification tasks, for instance, requires independence assumptions to simplify com- putational dependencies. We could take a sliding window approach to concatenate all relevant word embeddings into a single vector to represent a sentence. However, this rapidly increases the size of the input to any classifier, and does not scale to long sequences of text. In humans, the expectation of what we observe next is conditioned on the things that we have seen (or heard) in the past. Previously, models approximated this intuition through the independence assumption, as seen in Equation 2.2. Formally, we introduce additional notation for our input 푥 occurring in sequence 푥 1 ,푥 2 ,...,푥 푡 from timestep 1 through 푡. In this formulation, predicting the next word can be modeled as 푃(푥 푡+1 | 푥 1 ,...,푥 푡 ). Alternatively, we call 푥 푡+1 the prediction 푦 푡 for timestep 푡. To effectively observe input in sequences and produce corresponding predictions, we must incorporate historical context into our models. This means that, in addition to the current input, we include the previous hidden state from our model. Recurrent neural networks (RNNs) are networks with such a cycle, where current values depend on the input as well as the network’s state at a previous timestep. Instead of only producing a prediction (e.g., a word or token) 푦 푡 , we now also keep track of a hidden state ℎ 푡 . Formally, this is expressed in Equations 2.7 and 2.8. Figure 2.6 illustrate the architecure of an RNN, and how we can picture its unrolling in time steps. 46CHAPTER 2. ML METHODS FOR NLP Figure 2.6: Unrolled RNN (illustration from Chris Olah’s blog [Olah et al., 2015]) Figure 2.7: Repeating module of an LSTM (illustration from Chris Olah’s blog [Olah et al., 2015]) ℎ 푡 = 푓(Uℎ 푡−1 + W푥 푡 + b h )(2.7) 푦 푡 = 푔(Vℎ 푡 + b y )(2.8) This introduces new trainable parameter matrices U, V, W (and subsequent bias terms b) to make sure our architecture can shape the hidden state and output logits accord- ingly. RNNs can theoretically process infinite-length sequences, and we no longer need the Markov assumption to account for a sparsity problem. Training RNNs is done using backpropagation through time (BPTT), where we un- roll the temporal dependencies to backpropagate the gradient from time step 푡 to earlier ones: 푡 −1 ,푡 −2 ,... . However, because the gradients for updating the model at a timestep depend recursively on the previous timesteps, this step is sensitive. Because of the re- peated matrix multiplications required by the chain rule, this leads to the problem of vanishing or exploding gradients [Bengio et al., 1994, Pascanu et al., 2013]. Control- ling the flow of information in RNNs is difficult. Information from early timesteps has relatively little impact on our gradients, compared to more recent information. In practice, learning salient signals with RNNs is often difficult. 2.3. MODELING LANGUAGE47 LSTMs The long short-term memory (LSTM) architecture was introduced to address the gradient problem [Hochreiter, 1997]. An LSTM works by controlling the flow of information. In an LSTM network, additional gates are added to construct a context vector, which the network will learn to add and update information to (see Figure 2.7). Below is the mathematical definition for a single time step 푡 for the LSTM architecture. The LSTM maintains two states: a context state (푐 푡 ) and a hidden state (ℎ 푡 ), and uses gates to control the flow of information: 푓 푡 = 휎(푈 푓 ℎ 푡−1 +푊 푓 푥 푡 ) 푘 푡 = 푐 푡−1 ⊙ 푓 푡 푔 푡 = tanh(푈 푔 ℎ 푡−1 +푊 푔 푥 푡 ) 푖 푡 = 휎(푈 푖 ℎ 푡−1 +푊 푖 푥 푡 ) 푗 푡 = 푔 푡 ⊙ 푖 푡 푐 푡 = 푗 푡 + 푘 푡 표 푡 = 휎(푈 표 ℎ 푡−1 +푊 표 푥 푡 ) ℎ 푡 = 표 푡 ⊙ tanh(푐 푡 ) In the notation above, ⊙ denotes the element-wise multiplication of two vectors, also called the Hadamard product. Besides LSTMs, there are subsequent variants such as the gated recurrent unit (GRU) that balance the computational complexity and the number of trainable parameters [Cho et al., 2014]. Recurrent neural networks played an important role in language modeling. For further explanations of how RNNs, LSTMs, and GRUs, work, see for example Olah et al. [2015], Bishop and Bishop [2023], Goodfellow et al. [2014]. Encoder-Decoder So far, we have formulated our task as the language modeling task: how to predict the next word, given a context of preceding words. However, there is an alternative task formulation that is useful to introduce. Consider the problem of machine translation. While this looks like the previous tasks of predicting token 푥 푡+1 based on inputs 푥 1 ,...,푥 푡 , we are now dealing with a sequence-to-sequence (seq2seq) problem: we need to process an input sentence 푥 1 ,...,푥 푛 in one language, and output a translated sentence 푦 1 ,..., 푦 푚 in another. Why is translation a seq2seq problem? Note that 푛 and 푚 can be different from each other and that translations may not preserve order, as in the English to Dutch translation example below: We will celebrate the relief of Leiden (7 words) Wij gaan Leids Ontzet vieren (5 words) The encoder-decoder architecture is a powerful framework designed to handle sequence- to-sequence tasks where the input and output lengths may differ. The encoder and decoder are two distinct RNNs that work together to map a variable-length input to a variable-length output. The job of the encoder is to process the input sequence one step at a time and compress the information into a fixed-length vector, often called the con- 48CHAPTER 2. ML METHODS FOR NLP Figure 2.8: Illustration of the Encoder-Decoder architecture text vector. 3 This forces the information into a bottleneck: compressing information of an input sequence 푥 1 ,...,푥 푡 into a single hidden vector. However, this process is difficult to model and there is a loss of information. See Figure 2.8 for an illustration of the architecture. The context vector acts as the bridge between the encoder and the decoder. It is the final hidden state of the encoder that is passed to the decoder as its initial hidden state. This vector encodes the semantic meaning of the entire source sentence, so the decoder can begin generating the target sequence accurately. The decoder is a second RNN that produces output tokens, conditioned on the context vector. Its goal is to generate the output sequence one token at a time. This continues until a special end-of-sequence token is generated. This approach works well, in principle. However, forcing a long input sentence into a single, fixed-length context vector causes information loss (the bottleneck problem). Moreover, RNNs struggle to remember the beginning of long sequences by the time they reach the end. These two problems combined prompted the development of a more dynamic approach for passing information between the encoder and decoder, which we will discuss next. 2.3.4 Attention In search for a solution to the bottleneck problem, researchers looked for ways to ef- ficiently pass information from across the entire sequence into the decoder. During the generation process, the decoder could then decide which of the parts in the input sequence were relevant for decoding the next word. This search has lead to advanced mechanisms for attention [Bahdanau et al., 2015, Kim et al., 2017]. The effect of the attention mechanism is illustrated in Figure 2.9, where translation performance (in terms of BLEU score [Papineni et al., 2002], higher is better) is plotted. The architecture using the attention mechanism (called RNNsearch, see Figure 2.9) shows a consistent ability to achieve high translation quality especially for long sentences. The standard RNN encoder-decoder model (RNNenc) suffers a sharp drop in performance as sentence length increases. The intuition behind attention is to include a linear combination of all encoder hidden states in the context vector, instead of relying on a single final hidden state. This allows the model to look back at specific parts of the input sequence that are 3 Not to be confused with the LSTM’s context vector. NLP research has repurposed terminology in relatively little time. Language in research papers is also a moving target. 2.3. MODELING LANGUAGE49 Figure 2.9: The bilingual evaluation understudy (BLEU) scores of generated transla- tions with respect to the lengths of the sentences [Bahdanau et al., 2015] relevant to the current word being generated. We can control how much focus each input token 푥 푖 receives by adjusting its weight 푎 푖 in the linear combination, as shown in Equation 2.9. Here, c 푗 is the context vector for the 푗 -th output, and ℎ 푖 represents the hidden state of the 푖-th input token. c 푗 = 푎 1 h 1 ,...,푎 푛 h 푛 (2.9) The weights 푎 (often called attention weights) are not fixed; they are computed dynamically. A key observation is that we can make use of the hidden states ℎ 푖 , which are a representation of the semantics of a token or word. We can find out how relevant one hidden state is to another, by comparing them, for example by computing their cosine similarity. The softmax-normalized dot product is a common approach: 1. Compare the current decoder state with each encoder hidden state by applying a similarity function (such as a dot product) to see how well they match. 2. Pass these scores through a softmax function so they sum to 1, effectively creat- ing a probability distribution over the input tokens. While attention improved accuracy, the challenge of efficient computation of RNNs still remained to be solved. Because RNNs process tokens one by one (sequentially), they cannot take full advantage of modern parallel computing hardware in GPUs. To be able to scale to large datasets the recurrence should be removed entirely. This can be achieved by going from a sequential structure to a parallel one with a fixed context width that processes all tokens in parallel. This is achieved through self- attention. Mathematically, this turns the computation of the contextualized embeddings into Equation 2.10. a 3 = 5 ∑︁ 푗=1 훼 3푗 x 푗 = 5 ∑︁ 푗=1 softmax( x 3 |z query · x 푗 |z key ) x 푗 |z value (2.10) 50CHAPTER 2. ML METHODS FOR NLP Attention(푄,퐾,푉) = softmax 푄퐾 푇 √ 푑 푘 푉(2.11) Where푄 represents the matrix representing the current focus, 퐾 represents the keys providing the information in the sequence to compare against, 푉 represents the matrix containing the actual content to be extracted, and 푑 푘 represents the dimension of the keys (used for scaling to prevent gradient vanishing). As the dimension 푑 푘 increases, the dot products 푄퐾 푇 can grow large, pushing the softmax function into regions where it has small gradients. Dividing by √ 푑 푘 counteracts this effect, ensuring more stable training. Together, the set of attention matrices 푄,퐾,푉 are called an attention layer or head. In practice, one attention layer captures a specific type of interaction between tokens. However, having just one type of interaction means that the model can only focus on one relationship at a time (an attention head). Since a word can have many different types of linguistic relationships, such as syntactical (which verb does this noun belong to?), semantic (is this word part of a metaphor?), or referential (who does he refer to?). By using multiple heads in tandem, we give the model the ability to represent multiple types of relationships. One head might learn to look for grammatical structure, while another learns to look for pronouns, and another for entities. The heads all work in parallel, and their results are concatenated into a single, rich representation. Together, these heads form a layer. This is shown in Equation 2.12. MultiHead(푄,퐾,푉) = Concat(head 1 ,..., head ℎ )푊 푂 (2.12) where head 푖 = Attention(푄푊 푄 푖 ,퐾푊 퐾 푖 ,푉푊 푉 푖 ) 2.3.5 Transformers How can we use the attention mechanism to build an architecture for language mod- eling? We will now discuss one such architecture, the transformer architecture, intro- duced in the seminal work by Vaswani et al. [2017]. The attention mechanism is the core of the transformer architecture. The original transformer architecture, shown in Figure 2.10, consists of both an encoder and a de- coder. The left side of the Figure shows the encoder stack, the right side shows the decoder stack. Most LLMs are decoder-only transformers. The Figure shows transformer blocks that each consist of a self-attention module, a feed-forward network, a normalization layer, and residual connections from input to output. In most LLMs many transformer blocks are stacked on each other, in layers. For example, GPT-2 consists of 12 layers or more, depending on the version of the model. Let us now have a closer look at the transformer architecture (please refer to the Figure). The full architecture consists of three main types of elements: embedding, transformer blocks, and output probabilities: 1. Embedding: First, the input text is tokenized, and projected into low dimensional embedding vectors. Positional encoding is added to the embedding to preserve 2.3. MODELING LANGUAGE51 Figure 2.10: Visualization of the Transformer architecture [Godoy, 2020, Vaswani et al., 2017] word order. 2. Transformer Blocks: The transformer blocks are the main feature of the archi- tecture. Most models consist of multiple layers of transformer blocks. Within each layer, each block consists of three elements: • Self-attention Mechanism: The attention mechanism allows long range connections between tokens, capturing contextual information and relation- ships between tokens. Multiple attention heads allow the model to consider these relationships from different perspectives, for example, for short-range syntactic links, and for broader semantic context. For an encoder stack, transformers use self-attention. For a decoder stack, they use masked self-attention, to prevent looking at the future. • Multi-layer Perceptron: A fully-connected feed-forward network operates on each token independently. The attention layer makes connections be- tween tokens, whereas the MLP can refine each token’s representation. • Normalization: In most transformer blocks typical deep learning methods such as normalization, regularization (dropouts), and residual connections are added, to prevent vanishing or exploding gradients, and to improve sta- ble training. 52CHAPTER 2. ML METHODS FOR NLP 3. Output Probabilities: linear and softmax layers transform the processed embed- dings into probabilities, to make the next-token predictions. The transformer architecture has two main advantages over earlier methods, such as RNN and LSTM networks. Attention-based transformers allow (1) large scale parallel processing, and (2) fine-grained long-range semantic connections—speed and semantic accuracy. Transformers have dominated language modeling since their introduction. Af- ter large scale high quality training corpuses were created, transformer-based LLMs achieved impressive performance in question-answering, data-to-text generation, and application-driven tasks. Researchers have been pursuing increasingly larger models with promises of even higher capabilities [Brown et al., 2020]. In Chapter 4 we will discuss how to scale performance further. The transformer architecture is a complex architecture. Excellent interactive tuto- rials exist online, with more details, 4 see also the case study in Chapter 5. 2.3.6 Encoder-only (BERT) While the original transformer model consists of a combined encoder-decoder struc- ture, modern models are usually either an encoder or a decoder, as they are found to have specific empirical strengths. Let us have a closer look at encoders. Encoders are trained to compress input data into meaningful representations (embeddings) using techniques such as masked language modeling (MLM). They are good at NLU tasks such as text classification and sentiment analysis, while decoders are good at NLG tasks such as text generation, paraphrasing and summarization. Encoder-only models are designed to produce high-quality vector representations (embeddings) of input text. Unlike decoders, which are causal (they can only look at past words), encoders are inherently bidirectional. In an encoder, every token in a sequence can attend to every other token simultaneously. This global context allows the model to build a complex semantic representation of the input. To train these models, we use masked language modeling. In this process, roughly 15% of the input tokens are replaced with a special [MASK] token. It is up to the model to then learn which of the tokens from its vocabulary should go in its place. Often the regular cross-entropy loss is used to update the model. The goal of MLM encoder training is for the model to use the surrounding context (both left and right) to predict what the masked word is. Eventually, this forces the model to learn grammar, semantics, and even some world logic. To improve learning, in addition to the [MASK] token, encoder training typically includes several special tokens: • [CLS] (Classification): Inserted at the very start. After passing through all layers, the final state of this token is used as the aggregate representation for the whole sequence. 4 See, for example, https://poloclub.github.io/transformer-explainer/ 2.3. MODELING LANGUAGE53 • [SEP] (Separator): Used to tell the model where one sentence ends and another begins, which is vital for tasks such as question answering. • [PAD] (Padding): Ensures all sequences in a training batch are the same length. The best-known encoder-only model may well be the bidirectional encoder represen- tations from transformers, better known as BERT [Devlin et al., 2018]. Encoder- only models such as BERT, and its successors RoBERTa [Liu et al., 2019] and AL- BERT [Lan et al., 2019], have become popular. Encoder-only models are quite suitable for transfer learning, where they embed inputs and are subsequently finetuned on downstream tasks, such as classification for certain application domains. The pipeline for constructing encoder-only models in- volves two phases: 1. Pretraining: First train the model on a large corpus of billions of words. It may learn, for example, that King is related to Man and Queen is related to Woman. The model does not yet know how to do a specific job yet, it just learns English. Pretraining is an expensive step, and may take a long time (weeks or months). 2. Finetuning: Second, take the pretrained model and add a small additional classi- fication layer on top (often called a classification head). Then, the whole model is trained again on task-specific data (such as spam detection, or sentiment anal- ysis). Because the model already understands language, it typically only needs a few thousand examples to become an expert at the new task. Finetuning is a cheap step, and may take just hours or minutes, whereas starting from scratch on the specific task would require a large dataset and much training time. After we have discussed encoder-only models, we will now switch to decoder-only models. 2.3.7 Decoder-only (GPT) In language generation tasks, decoder-only models are used. Decoders are trained to reconstruct data or generate sequences using autoregressive training—they often generate output one token at a time. This autoregressive dependency, where the next tokens depend on what has been generated so far, is a form of causal attention. We cannot use future tokens to derive the best prediction. In an encoder, a word can look at the words to its right. In a decoder, this is strictly forbidden. We use a look-ahead mask (often a triangular matrix) that sets the attention scores of future tokens to−∞. If the model is predicting the word apple in the phrase The boy ate the..., it should not be allowed to observe the word apple in the training data. The decoder must rely solely on the preceding context. Subsequently, while encoders use masked language modeling, decoders use (regular) causal language modeling, as shown in Equation 3.1 in the next chapter. As a consequence, when moving from encoders to decoders, the relationship be- tween pretraining and finetuning changes slightly: decoders learn to predict the next 54CHAPTER 2. ML METHODS FOR NLP word in pretaining, and are then fine-tuned with the same loss function, but with an- other type of data. This allows us to move to in-context learning (ICL) or prompt- learning: because decoders are trained to continue a sequence, you can simply give them a prompt (the context), and they will generate the answer without needing any weight updates. The learn, as it were, from the examples in the prompt [Brown et al., 2020]. The ease of use with which large decoders can be used through in-context learning, without having to do expensive task-specific training or finetuning, has cre- ated much interest in ICL. Decoder-only models, such as the generalized pretrained transformer (GPT) fam- ily [Radford et al., 2018, 2019, Brown et al., 2020, Ouyang et al., 2022, Achiam et al., 2023], have become quite successful and are now widely used in our society. Many other models have been developed, both open and closed. Together, the training mech- anisms that we discussed form the basis of the modern LLM pipeline, which we will explain in more detail in Chapter 3. 2.4 Concluding Remarks In this chapter, we have traced the remarkable evolution of natural language processing from its foundational linguistic roots to the neural architectures that power today’s large language models. We have argued that human language is inherently complex, defined by intricate rules of syntax, morphology, and deep semantic and pragmatic nuances. The breakthrough in modern NLP progressed through a series of paradigm shifts— symbolic, statistical, neural—delivering building blocks for contemporary language models that are built on dense embeddings, the attention mechanism, and increases in scale, for data, model size, and compute power. Encoder models are suitable for NLU, decoder models are suitable for NLG. Both are trained by pretraining, finetuning, and decoders allow prompt-based in-context learning. 2.4.1 Open challenges Although the performance of modern LLMs is impressive, and many useful use cases exists, important challenges remain. Understanding these challenges is critical for iden- tifying research gaps and for engineering more robust systems, that are also relevant for agentic LLMs. Performance Scaling Current language modeling assumes scaling laws: better gen- eralization is achieved through the triumvirate of more parameters, more data, and more compute. However, further scaling creates significant challenges. Achieving state-of-the-art performance may require models with billions of parameters, whose training costs millions of dollars in compute, bringing about a high environmental im- pact. How to achieve better generalization (the ability to perform tasks that the model was not trained on) using efficient architectures is an open question. Current research is exploring model pruning and quantization (both discussed in Chapter 4). Further- more, synthetic data generation (see Chapter 19) aims to achieve higher generalization 2.4. CONCLUDING REMARKS55 without more training data, but the sweet spot between model size and generalization capability remains elusive. Data Scarcity The majority of NLP training datasets are built for the English lan- guage, and so are major benchmarks, such as GLUE (general language understanding [Wang et al., 2018]) or SQuAD (question answering [Rajpurkar et al., 2016]). The data is often accumulated from large web-scrapes. Other high-resource languages, such as Chinese or Hindi, can also benefit from this approach, while low-resource languages (such as Dutch or Flemish) lack the digital footprint and the cultural presence that is necessary to train modern transformers from scratch. Because models inherit the cul- tural nuances of their training data, non-English speakers often experience degraded performance or algorithmic bias that favors Western perspectives and linguistic norms. Multilinguality Because of the differences in available resources, NLP rarely pro- vides a generic solution for each language. For instance, translating a model’s success from English to other languages introduces several linguistic complexities. Other lan- guages like Turkish, Finnish, or Arabic may be morphologically richer, as they convey significant information through prefixes and suffixes to verbs or nouns. A single word in Turkish might correspond to an entire sentence in English. Models that are not trained on such languages fail to reflect such linguistic richness. Furthermore, different languages may employ different character sets and scripts. This leads to problems in handling non-Latin scripts (for example, logographic scripts for Chinese, or Abjads for Hebrew and Arabic). These languages require new tokeniza- tion and embedding strategies that lead to sensible subwords. Language Evolution Languages are living entities that evolve faster than models can be retrained. Amongst others, this leads to the out-of-vocabulary (OOV) problem. Even with sub-word tokenization (such as byte-pair encoding (BPE) [Gage, 1994] or Word- Piece [Song et al., 2021]), models struggle with neologisms, slang, technical jargon, or names that did not exist at the time of the training cutoff. Furthermore, a model trained in 2022 may not understand the context of global events or cultural shifts in 2026 (what is the Gen Z stare?). Creating continual learning systems that update knowledge with- out forgetting previous information (catastrophic forgetting) is an open challenge [Wu et al., 2024]. Tokenization Beyond Text The boundary between text and other modalities is blur- ring as a consequence of the effort to make sense of information outside its traditional context. A challenge in providing this context is multi-modal representation: repre- senting images, audio, and video in the same latent space as text. While text is discrete (words/characters), audio and video are continuous signals that must be sampled and quantized. Mapping a frame of video or a segment of a waveform to a token that a language model can process alongside text requires complex architectures like Vi- sion Transformers (ViT) [Han et al., 2022] or Contrastive Language-Image Pre-training (CLIP) [Radford et al., 2021]. Chapter 7 will discuss multi-modal models. 56BIBLIOGRAPHY Interpretability Finally, an important open question is how these models work in- ternally. While we can measure that a model works, we often struggle to explain why it provided a specific answer. Furthermore, models often fail, or hallucinate, giving wrong answers in a convincing way. For high-stakes applications such as military, le- gal or medical NLP, the lack of transparency should be a significant barrier to adoption of language models. Techniques for interpretable language models, such as mechanis- tic interpretability, are discussed in Chapter 21. Beyond these challenges, language modeling faces many other socio-technical chal- lenges, including ethical alignment, long-form logical reasoning, and many others. As NLP continues to evolve, these open problems represent a dynamic frontier where solv- ing one challenges often reveals a host of deeper questions for the next generation of student, researchers and engineers. 2.4.2 Looking Ahead In this chapter we have discussed the basics of language modeling, from the early beginning, to modern decoder models. We are now ready to go deeper into the matter of agentic LLMs. In the next chapter, we will discuss in more detail how a training pipeline of a large language model works. Bibliography Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 54 Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:// arxiv.org/abs/1409.0473. 48, 49 Gordon P Baker and Peter Michael Stephan Hacker. Wittgenstein: Understanding and meaning: Volume 1 of an analytical commentary on the philosophical investigations, part I: Essays, volume 1. John Wiley & Sons, 2008. 33 Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157– 166, 1994. 46 C.M. Bishop and H. Bishop. Deep Learning: Foundations and Concepts. Computer Science. Springer International Publishing, 2023. ISBN 9783031454684. URL https://books.google.nl/books?id=0uTgEAAAQBAJ. 40, 47 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. BIBLIOGRAPHY57 Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 52, 54 KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. CoRR, abs/1409.1259, 2014. URL http://arxiv.org/abs/1409.1259. 47 Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 40 Li Deng and Yang Liu. Deep learning in natural language processing. Springer, 2018. 42 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018. 53 Chenhe Dong, Yinghui Li, Haifan Gong, Miaoxin Chen, Junxin Li, Ying Shen, and Min Yang. A survey of natural language generation. ACM Comput. Surv., 55(8), December 2022. ISSN 0360-0300. doi: 10.1145/3554727. URL https://doi. org/10.1145/3554727. 32 Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2): 23–38, 1994. 37, 55 Albert Gatt and Emiel Krahmer. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61:65–170, 2018. 32 Daniel Voigt Godoy.dl-visuals. https://github.com/dvgodoy/ dl-visuals/?tab=readme-ov-file, 2020. URL https://github. com/dvgodoy/dl-visuals/?tab=readme-ov-file. 51 Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672–2680, 2014. 47 Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016. 40 Laura Graesser and Wah Loon Keng. Foundations of deep reinforcement learning: theory and practice in Python. Addison-Wesley Professional, 2019. 42 H. P. Grice. Logic and Conversation, page 41–58. BRILL, December 1975. ISBN 9789004368576. doi: 10.1163/9789004368811 003. URL http://dx.doi. org/10.1163/9789004368811_003. 33 58BIBLIOGRAPHY Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision trans- former. IEEE transactions on pattern analysis and machine intelligence, 45(1):87– 110, 2022. 55 S Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997. 47 John Hutchins. Retrospect and prospect in computer-based translation. In Proceedings of Machine Translation Summit VII, pages 30–36, Singapore, Singapore, September 13-17 1999. URL https://aclanthology.org/1999.mtsummit-1.5/. 30 Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Intro- duction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models. Stanford, 3rd edition, 2025. URL https: //web.stanford.edu/ ̃ jurafsky/slp3/. Online manuscript released Au- gust 24, 2025. 30, 38, 41 Henry Kautz. The third ai summer, henry kautz,. AAAI 2020 Robert S Engelmore Memorial Award Lecture, 02 2020. 42 Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. Structured attention networks. arXiv preprint arXiv:1702.00887, 2017. 48 Lea Krause and Piek T.J.M. Vossen. The Gricean maxims in NLP—a survey. In Saad Mahamood, Nguyen Le Minh, and Daphne Ippolito, editors, Proceedings of the 17th International Natural Language Generation Conference, pages 470–485, Tokyo, Japan, September 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.inlg-main.39. URL https://aclanthology.org/2024. inlg-main.39/. 33 Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012. 40 Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909.11942, 2019. URL http://arxiv.org/abs/ 1909.11942. 53 Jiwei Li and Dan Jurafsky. Neural net models of open-domain discourse coherence. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 198–209, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1019. URL https://aclanthology. org/D17-1019/. 32 Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, and Yejin Choi. We’re afraid language mod- els aren’t modeling ambiguity. In Houda Bouamor, Juan Pino, and Kalika Bali, BIBLIOGRAPHY59 editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 790–807, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.51. URL https: //aclanthology.org/2023.emnlp-main.51/. 32 Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http: //arxiv.org/abs/1907.11692. 53 C. Manning and H. Schutze.Foundations of Statistical Natural Language Pro- cessing.Foundations of Statistical Natural Language Processing. MIT Press, 1999. ISBN 9780262133609. URL https://books.google.nl/books? id=ZL34DwAAQBAJ. 38 Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Kalina Bontcheva and Jingbo Zhu, editors, Proceedings of 52nd Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland, June 2014. Association for Computational Lin- guistics. doi: 10.3115/v1/P14-5010. URL https://aclanthology.org/ P14-5010/. 31 Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gall ́ e, Arun Raja, Chenglei Si, Wilson Y. Lee, Beno ˆ ıt Sagot, and Sam- son Tan. Between words and characters: A brief history of open-vocabulary mod- eling and tokenization in nlp, 2021. URL https://arxiv.org/abs/2112. 10508. 37 Tom ́ as Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Yoshua Bengio and Yann LeCun, editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL http: //arxiv.org/abs/1301.3781. 35, 44 Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. 42 Christopher Olah et al. Understanding lstm networks, 2015. URL http://colah. github.io/posts/2015-08-Understanding-LSTMs/. 46, 47 Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models to follow instructions with human feedback. Advances in Neu- ral Information Processing Systems, 35:27730–27744, 2022. 54 60BIBLIOGRAPHY Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. 48 Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Sanjoy Dasgupta and David McAllester, editors, Pro- ceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1310–1318, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR. URL https://proceedings.mlr.press/ v28/pascanu13.html. 46 Aske Plaat. Learning to play: reinforcement learning and games. Springer, 2020. 42 Aske Plaat. Deep reinforcement learning. Springer, 2022. 42 Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. OpenAI Blog, 2018. 40, 54 Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 54 Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 55 Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. 55 Dan Roth. Learning to resolve natural language ambiguities: A unified approach. In Jack Mostow and Chuck Rich, editors, Proceedings of the Fifteenth National Conference on Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference, AAAI 98, IAAI 98, July 26-30, 1998, Madison, Wisconsin, USA, pages 806–813. AAAI Press / The MIT Press, 1998. URL http://w. aaai.org/Library/AAAI/1998/aaai98-114.php. 32 John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 42 Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith, editors, Pro- ceedings of the 54th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL https://aclanthology.org/P16-1162/. 37 BIBLIOGRAPHY61 Singerep.File:word vector illustration.jpg.Wikimedia Commons, 2022. URL https://commons.wikimedia.org/wiki/File:Word_vector_ illustration.jpg. Accessed: 2026-01-08. Licensed under C BY-SA 4.0. 35 Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. Fast word- piece tokenization. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 2089–2103, 2021. 55 Richard S Sutton and Andrew G Barto. Reinforcement Learning. Adaptive Compu- tation and Machine Learning series. Bradford Books, Cambridge, MA, 2 edition, November 2018. 41, 42 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 50, 51 Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Ana- lyzing and interpreting neural networks for NLP, pages 353–355, 2018. 55 Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279– 292, 1992. 42 Joseph Weizenbaum. Eliza—a computer program for the study of natural language communication between man and machine. Commun. ACM, 9(1):36–45, January 1966. ISSN 0001-0782. doi: 10.1145/365153.365168. URL https://doi. org/10.1145/365153.365168. 30 Ludwig Wittgenstein. Philosophical Investigations. Basil Blackwell, Oxford, 1953. 33 Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholam- reza Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01364, 2024. 55 Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolf- gang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nis- hant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google’s neural ma- chine translation system: Bridging the gap between human and machine translation, 2016. URL https://arxiv.org/abs/1609.08144. 37 Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola. File:Continuous Bag of Words model (CBOW).svg.Wikimedia Commons, 2023.URL https://en.wikipedia.org/wiki/File:Continuous_Bag_of_ Words_model_(CBOW).svg.Accessed:2026-01-09. Original source: d2l-ai/d2l-en. Licensed under C BY-SA 4.0. 45 62BIBLIOGRAPHY Chapter 3 The Large Language Model Training Pipeline What You Will Learn In this chapter you will learn how language models are trained. Specifically, you will be able to: • Describe the major stages of a basic training pipeline and explain the purpose of the individual stage: data acquisition, pretraining, finetuning, preference tuning, and inference. • Distinguish pretraining from finetuning, and explain why preference tuning is applied after, and not before, them. • Explain how the encoder/decoder distinction shapes which training objective a model uses. • Identify common benchmarks used to evaluate language models, and state what capability each is meant to measure. 3.1 Training Models The training pipeline of a large language model follows a somewhat standard methodol- ogy: starting from a model architecture where the parameters are initialized randomly, we train language capabilities through a process that is called pretraining. During pretraining, we feed large text corpora to our models, which they are trained on autore- gressively. The result of this stage is a base model, which is generally not yet suited for specific, practical applications. Hence, we need to adapt it to a downstream task or pur- pose. This stage is called posttraining, where we show specific types of additional data to the model using which we elicit task-specific behavior, such as instruction-following or classification settings. 63 64CHAPTER 3. LLM TRAINING PIPELINE 3.1.1 Encoder versus Decoder Models As we saw in the previous chapter, there are two dominant styles of models. On the one hand, encoder models are able to create rich contextualized representations based on input data. However, most of the recent successes are based on the decoder models. While encoders are better at understanding tasks (classification, sentiment), decoders excel at generative tasks. Decoder models have grown in size, both in terms of number of trainable parameters, and the available training data. This leads to models that are surprisingly capable of generalization to unseen contexts. Let us have a look in more detail at the relevant steps in the decoder-only training pipeline. 3.1.2 Pretraining/Finetuning Paradigm A central feature of the success of current training pipelines is the use of transfer learn- ing. One of the standard applications of modern transformer models is to take a model that has been trained on a large, generic dataset, and then retrain it on a new task. The model is pretrained on the large, generic, corpus, and then finetuned on the new, spe- cific, task, for which a small dataset is present. Finetuning for the new task is quick, since the dataset is small, but also since the pretrained weights have been transferred. The finetuning-training does not have to start from scratch. This pretraining/finetuning distinction of the transfer learning paradigm is at the heart of large language model training pipelines. Rather than starting the training from scratch, the model can reuse already-learned linguistic knowledge in new scenarios, thus speeding up the training process. Pretraining of large, commercial models may take months. Usually, the new task is specific (such as question answering in a certain domain, or sentiment analysis). For the new task, a dedicated dataset will be used to finetune the model. During finetuning, a subset of the model’s weights are updated carefully to retain the general linguistic capabilities of the pretrained model, while gaining an understanding of the new task. Finetuning a model to a new task takes typically on the order of minutes to hours. 3.1.3 Open and Closed Models Popular LLMs are OpenAI’s ChatGPT series [Achiam et al., 2023, Ouyang et al., 2022], Meta’s LLaMa family [Touvron et al., 2023], Google’s Gemma [Chowdhery et al., 2023] and Gemini [Anil et al., 2023], Anthropic’s Claude family, Moonshot AI’s Kimi, DeepSeek [Guo et al., 2025], and Qwen [Bai et al., 2025], among others. There are different degrees of openness between these models. For example, a model’s training data can be made public, the network architecture of a model can be disclosed, and the trained weights can be released. However, the companies behind commercial models seldom publish all elements of their model and training procedure, as they seek to retain some competitive edge. In response, the scientific community has developed fully open-source models, such as BLOOM [Le Scao et al., 2023], Pythia [Biderman et al., 2023], and OLMo [Groeneveld et al., 2024], to contribute to the collective understanding of how large 3.2. TRAINING PIPELINE65 Acquire Large, Unlabeled Text Corpus Pretrain Trans- former (Self- supervised) Generalist Base Model (Supervised) Finetuning (SFT) Finetuned Model Preference Alignment Final Aligned LLM Inference (Prompt- ing) Labeled Dataset (prompt- answer pairs) In-context Learn- ing (at infer- ence time) Figure 3.1: Overview of the LLM training pipeline [Radford et al., 2019, Minaee et al., 2024]. language models work. Of these models all elements of the training pipeline are pub- lic, and they are the focus of this chapter. We also highlight certain aspects of some commercial models. Although the models are all based on the transformer architecture, there are small differences between individual components and their overall layout, for example to capitalize on hardware differences or specific optimizations developed by their respec- tive creators. For instance, Google optimized Gemini for their TPU-based compute infrastructure [Jouppi et al., 2023]. We discuss further improvements to the architec- ture extensively in Chapter 4. 3.2 Training Pipeline We will now describe in more detail the steps of a standard training pipeline. LLMs are constructed using an elaborate pipeline with different training phases [Radford et al., 2019, Minaee et al., 2024]. We will describe the phases, as shown in Figure 3.1. The combinations of individual phases are usually referred to as training recipes, as they denote the order in which the models were trained. At each phase, a new dataset may be used, together with a new loss function. Each phase is designed to imbue the model with a specific capability. This will become clear as we discuss the individual phases. 3.2.1 Acquire Data The first step in the pipeline is to acquire data. Typically, massive, general-purpose text corpora are used [Brown et al., 2020]. The corpora consist of raw text without the need for manual labeling. This data is typically aggregated from both curated and uncurated online sources, then refined through cleaning heuristics to mitigate the effect of harmful, private, or copyrighted content. The corpora that are used for pretraining are large. Modern LLMs are trained on a web-scale magnitude of data, often reaching the scale of trillions of tokens. Some examples of datasets are: • Common Crawl: a massive, non-profit snapshot of the web containing billions of pages; alternative sources include https://archive.org. While the size is unparalleled, the quality of the data is under discussion [Baack, 2024]. Many derivatives, such as C4 (Colossal Clean Crawled Corpus) present a filtered subset of English web text [Raffel et al., 2020] to enhance the quality of the data. 66CHAPTER 3. LLM TRAINING PIPELINE • Dolma 3: a data mixture that is designed to maximize performance across a diverse suite of tasks. It is used in the OLMo open source/open data project [Ettinger et al., 2025, Soldaini et al., 2024]. • SmolLM3: utilizes a large three-stage training strategy with 11.2 trillion tokens that dynamically mixes web, math, and code data to optimize reasoning and knowledge acquisition [Bakouch et al., 2025]. Recent research emphasizes that data diversity is as critical as size for improving cross- domain knowledge and downstream generalization [Gao et al., 2020]. The corpus called The Pile has been specifically created as a heterogeneous corpus. For exam- ple, models trained on The Pile show significant gains in coding efficiency (Bits Per Byte, BPB) 1 across disparate domains, such as GitHub repositories, medical papers, and philosophy texts (see Figure 3.2). It is therefore often used as a benchmark, where successful performance on Pile BPB serves as an indicator of a model’s world knowl- edge and complex reasoning abilities. Curation of data for pretraining requires defining complex data cleaning pipelines that transform raw data from multiple sources into a single collection of cleaned, plain text documents. Often, this process focuses on quality assurance and safety. The cu- ration process begins with source prioritization, where the mixture is weighted toward high-quality sources such as Wikipedia, books, and other known curated sources. Fur- ther, to ensure the integrity of the corpus, researchers apply noise reduction heuristics to strip away boilerplate text, navigation menus, and possibly inappropriate content. Furthermore, de-duplication is performed at both the URL and document levels; this is essential to prevent the model from memorizing redundant information and overfitting on repetitive data. Beyond accuracy, the pipeline must select data that matches safety and ethical con- straints. This too is commonly implemented through heuristic-based filtering. Fur- thermore, toxicity-detection systems are implemented to remove hate speech, although these methods often yield mixed results. A significant challenge in this phase is bias mitigation, as automated filters can inadvertently flag and suppress legitimate data writ- ten in specific dialects, such as African American English, leading to the accidental exclusion of diverse voices from the training set. 3.2.2 Pretrain The second step in the training pipeline is pretraining the model on the curated cor- pus, a process that yields a generalist natural language model. This stage uses a self- supervised transformer approach [Vaswani et al., 2017] on the unlabeled text. In order to acquire internal representations of language, conceptual semantic rela- tionships, and world knowledge, the model is trained using masked-token cloze-like tasks. In such tasks the objective is to predict masked tokens conditioned on a preced- ing context of large corpora of sequences (i.e., representations of text, image patches). This process is self-supervised (no human-annotated data is required) and is known to 1 Bits-Per-Byte is a tokenizer-agnostic metric from information theory that measures the average bits needed per symbol, reflecting coding efficiency, see also Section 3.3.2. 3.2. TRAINING PIPELINE67 Figure 3.2: Overview of the data sources inside The Pile dataset [Gao et al., 2020]. yield rich representations of language, relations between concepts, and world knowl- edge, in various forms. Formally, the language modeling objective maximizes the probability of the tokens in a sentenceU =푢 0 ,...,푢 푛 , based on a window 푘 of existing tokens (Equation 3.1). 퐿(U) is the language modeling loss. 퐿(U) = ∑︁ 푖 log푃(푢 푖 |푢 푖−푘 ,...,푢 푖−1 ;Θ)(3.1) Language models estimate these probabilities through trainable parameters Θ, op- timized via gradient updates. Given that model sizes range from millions to over a tril- lion parameters, large datasets are required to ensure convergence. During this training phase, teacher forcing is employed: the model makes a prediction, but the ground-truth token—rather than the model’s own output—is provided as the input for the next step. In contrast, during autoregressive generation at inference time, the model must rely on its own predictions as the input for subsequent tokens. Tokens are created by running the appropriate tokenizer, as explained in Section 2.1.4. In pretraining, a key challenge is to ensure that the model learns efficiently and re- mains stable throughout the trillions of gradient updates. Over time, different strategies have evolved that aim to keep the training process under control. One such strategy is curriculum learning. In human learning, we start by learning easy concepts (such as addition) before we tackle more difficult concepts (multiplication, logarithms). In language modeling, this idea is used in data mixtures, or data curricula. Rather than training on all data types simultaneously, researchers select a specific mixture of data sources to train on first, see the example in Figure 3.3. Creation of mixtures involves strategically ordering the data according to difficulty. In a training curriculum we start with easy-to-learn general web crawls, and then shift toward harder high-quality rea- 68CHAPTER 3. LLM TRAINING PIPELINE Figure 3.3: Example data mixtures for training SmolLM3 [Bakouch et al., 2025]. Figure 3.4: Training loss for the BERT model using the original architecture (a) and a rearranged architecture (b) [Shoeybi et al., 2020] soning data (math and code) in later stages. The underlying idea is that this helps the model converge quicker on more complex logic [Bengio et al., 2009]. Other mechanisms for stabilizing training are employed, as training at this scale is notoriously unstable and prone to loss spikes [Takase et al., 2025]. Stability is main- tained through standard machine learning techniques such as weight decay, layer nor- malization, and sophisticated learning rate schedulers. The use of specialized optimiz- ers and the monitoring of gradient norms helps to prevent parameters from diverging during the intensive pretraining process. An example of how switching embedding layers can lead to more stable training is shown in Figure 3.4. The left figure shows the training loss for the 336M and 752M BERT model. While the original architecture performs well on the 336M model, the modifications in (b) enable stable training with lower training loss [Shoeybi et al., 2020]. 3.2. TRAINING PIPELINE69 3.2.3 Finetune After pretraining comes posttraining, of which the first important training step is fine- tuning. LLMs are rarely used in an autocomplete generic predict-next-word setting, people rather use them to answer questions, give advice and recall information. Fine- tuning is the process of taking a pretrained model that has already learned general language patterns, and then train it further on a smaller, more specific dataset. In fine- tuning, we apply a general model to a specific (narrow) task on a labeled dataset, using a supervised learning approach. A popular efficient method is low-rank optimization, which finetunes a small subset of the parameters [Hu et al., 2021, Wei et al., 2022, Minaee et al., 2024]. Supervised Finetuning (SFT) Finetuning can be applied to many specialized tasks. The most general models use supervised finetuning to be able to follow instructions and perform reasoning steps [Ettinger et al., 2025]. The supervised finetuning step is especially important for smaller models, offering an efficient mechanism for significant improvements in general chat, reasoning, and function-calling capabilities. Supervised finetuning is also used for more specialized tasks, such as toxicity re- duction. WildChat [Zhao et al., 2024] provides over a million real-world user-ChatGPT interactions, capturing how humans prompt the LLM with varying complexity and tox- icity. For learning reasoning capabilities, OpenThoughts [Guha et al., 2025] contains step-by-step reasoning chains, distilled from advanced models to enhance problem- solving capabilities in math, science, and coding. Human-obtained data is supple- mented by synthetic data, which involves generating instructions and responses using larger teacher models (such as GPT-5) to fill specific data gaps and scale up training volume [Zhang et al., 2026]. Finetuning does not only pertain to general instruction-following or reasoning. Other tasks can be, for example, translation from one language to another, or question answering in health-care or customer service. Especially for industrial applications, where tasks are narrow and models are smaller, finetuning is a fruitful undertaking. In Chapter 11 we will discuss the use of tools by LLMs, a central aspect of agentic LLM. The training to use tools also takes place in the supervised finetuning step in the training pipeline [Wei et al., 2022, Minaee et al., 2024]. Furthermore, finetuning is instrumental in establishing a model’s persona and deep- ening its expertise. For instance, organizations can train models to adopt a specific tone of voice, for example so that a customer service bot maintains a consistent brand iden- tity, be that empathetic and formal or witty and casual. Finally, the injection of deep domain knowledge enables the model to master the intricate terminology and logic of niche fields, such as industry-specific patent information or debugging proprietary legacy code. Ultimately, finetuning transforms a generalist predictor into a specialized expert, often allowing smaller, more efficient models to outperform larger foundational models on specific tasks. 70CHAPTER 3. LLM TRAINING PIPELINE Figure 3.5: High-level overview of RLHF: training a reward model (left), and optimiz- ing a language model (the policy) with respect to that reward model Dong et al. [2024] 3.2.4 Preference Tuning A problem that surfaced in early models is that training tends to amplify extreme socially unacceptable texts in the training data and run wild, crowding out moderate points of view. Since then, an essential step for LLMs to become socially acceptable, is for pipelines to include a phase in which the model is tuned to behave in socially acceptable ways. This phase is called preference tuning. In this phase we align the finetuned model with user expectations (preference align- ment). The goal of this step is to improve the model to give socially acceptable answers, prevent hate speech, reduce hallucinations, and improve instruction adherence. Standard methods for preference tuning include reinforcement learning with hu- man feedback (RLHF) [Ouyang et al., 2022], which trains a reward model to approxi- mate human values, and direct preference optimization (DPO) [Rafailov et al., 2024], which optimizes the policy directly from preference data. A high-level overview of RLHF is shown in Figure 3.5. To address the bottleneck of human annotation, modern pipelines increasingly utilize reinforcement learning from AI feedback (RLAIF) [Bai et al., 2022], where a stronger model generates the preference signals. Achieving alignment with human values is not an easy task, and we usually dis- tinguish between safety alignment, value alignment and preference alignment. In the latter case, a complication is that preferences are subjective preferences (such as writ- ing style) that change per person. Value alignment On a higher level, value alignment focuses on ensuring that the model’s underlying principles and worldviews resonate with broad human ethics. Un- like individual preferences, this involves anchoring the model to high-level concepts such as fairness, helpfulness, and honesty. In practice, this means the model is gov- erned by a set of written principles (a constitution) to ensure that it adheres to the policies set out by the developer. However, this value alignment usually leads to a trade-off, where optimizing a model for safety and helpfulness also degrades its performance on other capabilities, such as factual recall or creativity [Lin et al., 2024]. This trade-off is otherwise known as the alignment tax. 3.2. TRAINING PIPELINE71 Figure 3.6: Comparison of RLHF (specifically, PPO) to RLVR (specifically, GRPO) method [Shao et al., 2024] Safety alignment Safety alignment is the process of setting hard boundaries on model behavior to prevent catastrophic or harmful outputs. While preference alignment might focus on tone, and value alignment on viewpoint, safety alignment focuses on adver- sarial robustness and the refusal of malicious prompts. Example methodologies for ensuring safety includes red teaming, where models are subjected to rigorous jailbreak attempts to identify vulnerabilities, and guard railing, where specific refusal boundaries for high-risk responses are filtered. Objective alignment Finally, for domains where objective truths exist, such as math- ematics or coding, methods known as reinforcement learning with verifiable rewards (RLVR) [Lambert et al., 2024] are employed. Unlike standard RLHF, RLVR opti- mizes for objective correctness by leveraging ground-truth signals to robustly improve reasoning capabilities. Examples include code that compiles, or correct mathematical solutions. In the latter case, the Group Relative Policy Optimization (GRPO) algorithm is often used [Guo et al., 2025], a variant of Proximal Policy Optimization (PPO) that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO. It does so by sampling multiple answers from the policy model and using their relative quality (as assessed by a verifier) to compute the rewards. Instead of relying on human labels for learning what counts as a good answer, the model gets direct rewards (+1, or -1) from a deterministic verifier [Guo et al., 2025, Shao et al., 2024], see Figure 3.6. 3.2.5 Infer The final step in the training pipeline is the inference step. This is the step in which the trained model is finally used. Here the user finally gets to interact with the model. Finally, we can use the fruits of our training efforts. At inference time, we use natural language prompts (instructions and questions) to obtain responses from the model. Prompting is the preferred way of using LLMs to have the model answer our questions. Inference results can be sensitive to the way that a prompt is formulated. The use of certain words, or re-ordering words in the prompt, 72CHAPTER 3. LLM TRAINING PIPELINE can influence the answer of the model. This sensitivity has led to prompt engineering and context engineering, skills that have become important for users of LLMs [Sahoo et al., 2024]. Interestingly, as we discussed in the previous chapter, in decoder models whose size is beyond hundreds of billions parameters a new learning method emerges: in- context learning (ICL) [Brown et al., 2020, Wei et al., 2022]. In-context learning is a method where the prompt contains a small number of examples, together with an instruction. These examples point the model in a certain direction and it uses them to learn the answer. ICL is a form of within-prompt few-shot learning. ICL is also a type of learning where no parameters of the model are changed—it is example learning without parameter training [Dong et al., 2022]. In the remainder of this book we will see many examples of the way in which in-context learning is used, such as in agentic harnesses. Inference-time in-context learning is a major driver of agentic LLM. 3.2.6 Concluding Remarks We have now described the phases of a typical LLM training pipeline. Note that the pipeline that we described is an example of a typical approach, and different models are trained in different ways. Current modern training pipelines are elaborate, and training is costly. Innovations to training pipelines are the topic of current research, see, for example, Guo et al. [2025], Du et al. [2025]. Being able to obtain predictions quickly is key, and multiple techniques have been developed to speed up the process. For instance, is is important to strike a balance between the reasoning effort and response quality. Research into test time scaling studies this trade-off. A higher inference-time effort costs more compute time, but may yield a better answer. We discuss these and other optimization techniques in Chapter 4. 3.3 Benchmarks, and Performance After the pipeline has been used to train the model, we wish to find out how well the model performs. LLMs are created using a variety of modeling techniques, for different tasks, on different datasets. As a result, they have often obtained remarkable capabilities. For example, LLMs perform well on natural language understanding tasks (GLUE, SQUAD, Xsum) [Wang et al., 2018, 2019, Rajpurkar et al., 2016, Narayan et al., 2018], translation [Kocmi et al., 2022, Papineni et al., 2002, Sennrich et al., 2015], question answering tasks [Roberts et al., 2020], and poetry composition [Zhang and Eger, 2024], to name just a few domains. The broad applicability of LLMs shows their versatility, although the rapid development of the field, as well as ongoing research efforts, raise many open questions related to the capabilities of LLMs in many domains and contexts. We will now discuss benchmarks that are commonly used to test the performance of our models. 3.3. BENCHMARKS, AND PERFORMANCE73 3.3.1 Common tasks Let us have a closer look at some common LLM tasks. The tasks can largely be grouped under the two language processing categories that we introduced previously: natural language understanding (NLU) and natural language generation (NLG). Depending on the category, the exact task formulation, and the underlying data, the performance metric will differ. We will discuss each separately. Note that performance is an ambiguous term. In NLP, it is often used in the con- text of task performance (accuracy), instead of computational performance (speed). While exact evaluation criteria differ depending on the task, modern evaluation prac- tices primarily focus on task performance. However, how many of the calls to the LLM we make, and how quickly we can get responses from the model, is also important to achieve high scores on contemporary benchmarks, as we will see in later chapters. 3.3.2 Metrics In order to monitor or report the (task) performance of LLMs, different metrics are used, depending on the stage of the training that we are in. Pretraining Metrics In pretraining, we are mostly interested in knowing whether the model is able to accurately predict the next word. The standard metric is perplexity, shown in Equation 3.2. Perplexity measures the model’s uncertainty when predicting the next token, and is directly related to the cross-entropy loss function. Low perplex- ity indicates that the model assigns high probability to the correct next tokens. For perplexity, lower is better. PPL(푋) = exp − 1 푁 푁 ∑︁ 푖=1 log푃(푥 푖 | 푥 <푖 ) ! (3.2) BPB =− 1 퐵 푁 ∑︁ 푖=1 log 2 푃(푥 푖 | 푥 <푖 )(3.3) Perplexity is computed over tokens; adjusting perplexity to stick to word boundaries needs careful normalization [Pimentel and Meister, 2024]. Moreover, even when ad- justed, it is still tokenizer-specific, as other tokenizers can lead to other subwords and hence other probability distributions. As an alternative, bits-per-byte (BPB), shown in Equation 3.3, normalizes the loss to the byte level, which enables fair comparisons between models that use different tokenization schemes [Borgeaud et al., 2022]. It measures the average number of bits required to encode a single byte of data. Posttraining Metrics In posttraining and inference, when the model is used for prob- lem solving, generation, or reasoning, we look at the deterministic success rates, based on ground truth data. Often, tasks are set up in multiple-choice question/answering formats, where there is a single correct answer. For these types of tasks, we commonly 74CHAPTER 3. LLM TRAINING PIPELINE report accuracy, the percentage of questions for which a correct response is given. For accuracy, higher is better. In the regular setup, the model needs to provide an answer in one unique attempt. However, since sampling from a language model is a stochastic process, we may re- quest additional attempts, and see if the answer is among them. This is formalized in the Pass@K metric: the percentage of problems solved correctly on the model’s 퐾 th attempt; higher is better. Common values for 퐾 include 1, 5, 10, 100. 3.3.3 Common Benchmarks While measuring the performance of LLMs remains an active area of research, many datasets have become commonplace for reporting their general capabilities. These datasets are designed to only be used for evaluating the quality of a model, and are not intended as additional training data. As such, they serve as semi-standardized benchmarks. Ongoing efforts for moving to field standards, such as MLCommons [Thiyagalingam et al., 2022], show an increasing need for objective and sound evalua- tion setups. Data Contamination Modern benchmarks run a risk of data contamination [Bal- loccu et al., 2024]. As pretraining data increases in size (often a considerable ratio of the total size of the internet), the line between in- and out-of-domain data becomes increasingly blurred, especially given that commercial vendors often do not specify the sources of their training data (either out of commercial interest or for other reasons). Hence, when we wish to measure true generalization to unseen data, we need to ensure that the model has not seen the data before (we wish to prevent data contamination). It is common for benchmarks to host a public, freely accessible dataset for quick experimentation by developers, but to maintain a private dataset that commercial ven- dors have restricted access to, to prevent data contamination. Furthermore, in order to maintain the benchmark’s usefulness into the future, developers often actively update the samples in the benchmark. In Table 3.1, we provide a non-exhaustive list of general-purpose benchmarks that are popular at the time of writing. 3.4 Concluding Remarks In the field of LLM development, it has become common for commercial LLM devel- opers (such as Anthropic, OpenAI, Google, DeepSeek, Mistral) to publish a tech report that informs the expert public on how the model behaves across popular benchmarks. Often, such as tech report is published as a preprint, without going through academic peer-review, and many details that would be necessary for full scientific reproducibility (such as data recipe, model architecture) are left out, for commercial reasons. Nonethe- less, these reports provide useful information to derive insights into what LLMs can and cannot do. First, we find that modern LLMs demonstrate a distinct performance hierarchy across benchmarks, as they have effectively solved broad knowledge tasks, but still 3.4. CONCLUDING REMARKS75 Benchmark Name Capability TestedTask FormatMetric Humanity’s LastExam (HLE) GeneralIntelligence, Multimodal Reasoning Short-Answer, Multiple-Choice Pass@1 Abstract Reasoning Challenge (ARC) General IntelligenceVisualGridPattern Completion Pass@1(Pri- vate Set) GPQADia- mond Expert-Level STEM Rea- soning Multiple-ChoicePass@1 AIME 2025Advanced Mathematical Reasoning Integer Answer (0-999)Pass@1 SWE-BenchAgentic Coding, Real- WorldSoftwareEngi- neering Code Patch GenerationPass@1 FortressAI Safety, National Secu- rity Risk Assessment Varies (Proprietary)Avg Risk Score (ARS) & Over- Refusal Score (ORS) MMLUBroad General Knowl- edge Multiple-ChoicePass@1 GSM8KGrade-School Mathemat- ical Reasoning Multi-Step ArithmeticPass@1 TruthfulQAFactual Correctness, Hal- lucination Detection Question AnsweringPass@1 Table 3.1: Overview of common LLM benchmarks with the metrics reported on them struggle to master complex reasoning and autonomous agentic tasks. On static knowl- edge and professional exam benchmarks (such as MMLU and TruthfulQA), frontier models (such as GPT-5.2, Gemini 3 Pro, and Claude 4.5 Opus) have reached, at the time of writing, near-saturation, consistently scoring above 85–90% and surpassing human expert baselines. The newest generation of reasoning models (such as OpenAI’s o1, DeepSeek-R1, and Qwen-3.5) have recently achieved great progress, pushing math and science pass rates from low percentages to expert levels. Despite these advances, the software en- gineering SWE-bench remains the ultimate stress test for reliability; while scores have risen from negligible (¡2%) to respectable levels (40-50% on verified subsets) for top coding agents, the gap between answering a coding question and autonomously fixing a real-world repository issue remains a significant hurdle for modern AI. Open Questions Open questions about the origin of LLMs capabilities still remain. As the pipelines have grown complex, both in terms of training (What data matters? 76BIBLIOGRAPHY What training steps are most important?) and inference (How to extract high quality reasoning traces?), drawing conclusions is difficult. For example, it is an open question whether pretraining or posttraining is more important, and whether knowledge comes mostly from pretraining, and posttraining only serves for finding a good task interface [Zhou et al., 2023]. The remaining parts of this book will discuss posttraining and inference-time rea- soning methods and agent training methods. But first, we will look at scaling laws, in the next chapter. Bibliography Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 64 Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 64 Stefan Baack. A critical analysis of the largest source for generative ai training data: Common crawl. In Proceedings of the 2024 ACM Conference on Fairness, Account- ability, and Transparency, pages 2199–2208, 2024. 65 Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 64 Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKin- non, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. 70 Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Pati ̃ no, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallou ́ edec, Kashif Rasul, Nathan Habib, Cl ́ ementine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivas- tav, Joshua Lochner, Xuan-Son Nguyen, Colin Raffel, Leandro von Werra, and Thomas Wolf. SmolLM3: smol, multilingual, long-context reasoner. https: //huggingface.co/blog/smollm3, 2025. 66, 68 Simone Balloccu, Patr ́ ıcia Schmidtov ́ a, Mateusz Lango, and Ondrej Dusek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 67–93, St. Julian’s, Malta, March 2024. As- sociation for Computational Linguistics. doi: 10.18653/v1/2024.eacl-long.5. URL https://aclanthology.org/2024.eacl-long.5/. 74 BIBLIOGRAPHY77 Yoshua Bengio, J ́ er ˆ ome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009. 68 Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023. 64 Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206–2240. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/borgeaud22a. html. 73 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 65, 72 Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023. 64 Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF. Transactions on Machine Learning Research, 2024.ISSN 2835-8856.URL https://openreview.net/forum?id= a13aYUU9eU. 70 Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022. 72 Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. 72 78BIBLIOGRAPHY Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morri- son, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, et al. Olmo 3, 2025. URL https://arxiv. org/abs/2512.13961. 66, 69 Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027. 66, 67 Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Co- han, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Acceler- ating the science of language models, 2024. URL https://arxiv.org/abs/ 2402.00838. 64 Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hri- tik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint arXiv:2506.04178, 2025. 69 Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qi- hao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 64, 71, 72 Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 69 Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nis- hant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings of the 50th annual international symposium on computer architecture, pages 1–14, 2023. 65 Tom Kocmi, Rachel Bawden, Ond ˇ rej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Had- dow, et al. Findings of the 2022 conference on machine translation (wmt22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1– 45, 2022. 72 BIBLIOGRAPHY79 Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. 71 Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ́ c, Daniel Hess- low, Roman Castagn ́ e, Alexandra Sasha Luccioni, Franc ̧ois Yvon, Matthias Gall ́ e, et al. Bloom: A 176b-parameter open-access multilingual language model, 2023. 64 Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, et al. Mitigating the alignment tax of rlhf. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 580–606, 2024. 70 Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024. 65, 69 Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018. 72 Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models to follow instructions with human feedback. Advances in Neu- ral Information Processing Systems, 35:27730–27744, 2022. 64, 70 Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. 72 Tiago Pimentel and Clara Meister. How to compute the probability of a word. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358–18375, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics.doi: 10.18653/v1/2024.emnlp-main.1020.URL https: //aclanthology.org/2024.emnlp-main.1020/. 73 Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 65 Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Er- mon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 70 80BIBLIOGRAPHY Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of trans- fer learning with a unified text-to-text transformer. Journal of Machine Learn- ing Research, 21(140):1–67, 2020. URL http://jmlr.org/papers/v21/ 20-074.html. 65 Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. 72 Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020. 72 Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024. 72 Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine trans- lation models with monolingual data. arXiv preprint arXiv:1511.06709, 2015. 72 Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 71 Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.URL https://arxiv.org/abs/ 1909.08053. 68 Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkin- son, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lam- bert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crys- tal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettle- moyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for lan- guage model pretraining research.In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15725–15788, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.840. URL https://aclanthology.org/2024. acl-long.840/. 66 Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. Spike no more: Stabi- lizing the pre-training of large language models, 2025. URL https://arxiv. org/abs/2312.16903. 68 BIBLIOGRAPHY81 Jeyan Thiyagalingam, Gregor von Laszewski, Junqi Yin, Murali Emani, Juri Papay, Gregg Barrett, Piotr Luszczek, Aristeidis Tsaris, Christine Kirkpatrick, Feiyi Wang, et al. Ai benchmarking for science: Efforts from the mlcommons science working group. In International Conference on High Performance Computing, pages 47–64. Springer, 2022. 74 Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 64 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 66 Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Ana- lyzing and interpreting neural networks for NLP, pages 353–355, 2018. 72 Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019. 72 Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022. 69, 72 Ran Zhang and Steffen Eger.Llm-based multi-agent poetry generation in non- cooperative environments. arXiv preprint arXiv:2409.03659, 2024. 72 Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Guoyin Wang, and Fei Wu. Instruction tuning for large language models: A survey. ACM Comput. Surv., 58(7), January 2026. ISSN 0360-0300. doi: 10.1145/3777411. URL https://doi.org/10.1145/ 3777411. 69 Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=Bl8u7ZRlbM. 69 Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021, 2023. 76 82BIBLIOGRAPHY Chapter 4 Scaling Laws and Efficient Large Language Models What You Will Learn In this chapter you will learn how performance of language models scales with data and training effort. Specifically, you will be able to: • Explain how model performance relates to compute, data, and model size through scaling laws, and predict the effect of increasing one factor while hold- ing the others fixed. • Describe what techniques can be used to develop smaller language models, and analyze the tradeoff of using smaller models with respect to compute require- ments and performance. • Recognize the difference between inference-time compute and training-time compute. • Reflect on sustainable language models and the environmental and privacy im- plications of large-scale deployment 4.1 Scaling Laws: Compute, Data, Parameters There are three primary factors involved in the training of LLMs: (1) the amount of compute, (2) the size of the dataset, and (3) the size of the model. Studies have shown that these three factors relate to the final performance of a model, measured as its loss on a held-out test set (퐿). This relation has been found to be both smooth and pre- dictable. The relationship takes the form of a power law: as any of the three resources is increased, the loss decreases proportionally to a negative power of that resource. For instance, doubling the model size might reduce the loss by a factor of 2 −훼 for some positive exponent 훼. Because the behavior is predictable, we can estimate how much a model will improve (for instance, from doubling the training dataset size) before committing to an expensive training run. 83 84CHAPTER 4. SCALING LAWS AND EFFICIENCY Figure 4.1: Loss on language modeling objective is a function of training compute, dataset size, and model size [Kaplan et al., 2020]. We define these three quantities (퐶, 퐷, 푁 ) as follows: • Compute (퐶): the total amount of arithmetic performed over the entire training run, measured in floating point operations (FLOPs). Note that this is a count of operations in total, not a rate per second. A widely used approximation re- lates it to the other two quantities as 퐶 ≈ 6푁퐷, since processing a single token touches each of the 푁 parameters roughly six times across the forward (twice) and backward pass (four times). • Dataset size (퐷): the total number of tokens the model is trained on. Note that a token is a subword unit defined by the tokenizer, so the same corpus corresponds to different values of 퐷 under different tokenizers. The count of 퐷 is the number of tokens as they are used during the training run. When data is repeated over several epochs, the repeated tokens still add to 퐷 but may yield diminishing returns compared to fresh data. • Model size (푁 ): the number of trainable parameters (weights) in the model ar- chitecture. This count usually refers only to the parameters that increase when scaling model depth, and therefore excludes the embedding matrix. Training large models comes at considerable cost, usually in the realm of tens of mil- lions of dollars, and up to months of compute time. Scaling laws allow us to predict whether a model will train well. 4.1.1 Kaplan Scaling Laws There are different empirical results for scaling laws that are found by fitting a function to empirical test data. For instance, in Figure 4.1, every point is a trained model on a fixed dataset with a similar experimental setup, while only changing the three primary variables 퐶, 퐷, 푁 . In early 2020, Kaplan et al. [2020] found how model performance is related to 퐶, 퐷, 푁 . They estimated, following empirical observations, that the loss follows independent power laws rekated to the three main variables: 퐿(푁) ∝ 푁 −훼 푁 and 퐿(퐷) ∝ 퐷 −훼 퐷 , with 훼 푁 ≈ 0.076 and 훼 퐷 ≈ 0.095. The individual power laws can be combined into a single optimizable equation, shown in Equation 4.1. This relationship was assumed to be generic: it would apply 4.1. SCALING LAWS: COMPUTE, DATA, PARAMETERS85 Figure 4.2: Larger models reach a given loss using fewer training tokens, illustrating greater sample efficiency [Kaplan et al., 2020] well as models of loss behavior for “other generative modeling tasks” [Kaplan et al., 2020, p. 5]. The authors further showed that, under no other constraints than compute budget, their derivations would point to optimal model size 푁 , an ideal dataset size 퐷, defined by optimal step size (푆) and batch size (퐵). 퐿(푁, 퐷) = h 푁 푐 푁 훼 푁 훼 퐷 + 퐷 푐 퐷 i 훼 퐷 (4.1) Specifically, they concluded that as the compute budget 퐶 increases, model size 푁 should be increased faster than data size 퐷 (푁 ∝ 퐶 0.73 and 퐷 ∝ 퐶 0.271 ), leading to the development of large models such as GPT-3. The negative exponents reflect diminishing returns: each doubling of model size or dataset size lowers the loss by only a few percent. At first glance, this recommendation seems to contradict the exponents themselves: since 훼 퐷 > 훼 푁 , data scaling reduces the loss faster than parameter scaling for the same relative increase in compute. The reason is that compute, not the resource itself, is the constraint: because 퐶 ≈ 6푁퐷, increasing 퐷 consumes compute just as increasing 푁 does. This means that 푁 is with becoming a bottleneck with increasing likelihood. Hence, once the compute cost for 퐷 is accounted for, allocating additional compute to 푁 yields a better return than allocating it to 퐷. This means that the optimal allocation favors larger models. Sample efficiency A separate observation of Kaplan et al. [2020] is that larger mod- els are more sample-efficient. For a fixed number of training tokens, a larger model reaches a lower loss, and to reach a given target loss it requires fewer tokens and fewer optimization steps than a smaller one. This effect is visualized in Figure 4.2. 1 퐵∝ 퐶 0.24 , 푆 ∝ 퐶 0.03 , 퐷 = 퐵· 푆 ∝ 퐶 0.27 86CHAPTER 4. SCALING LAWS AND EFFICIENCY 4.1.2 Chinchilla Scaling Laws In 2022, Hoffmann et al. [2022] reevaluated these findings with a model called Chin- chilla, and found, surprisingly, that most large models at the time were severely un- dertrained. This discrepancy can be traced back to the methodology of Kaplan et al. [2020]: their optimal allocation was extrapolated from the early, non-converged train- ing dynamics of many models rather than from runs trained to convergence, which systematically biased their conclusions toward larger models trained for fewer steps. The key insight from the Chinchilla study was that for every doubling of the com- pute budget, both the model size 푁 and the number of training tokens 퐷 should be increased in roughly equal proportions (푁 ∝ 퐶 0.5 and 퐷 ∝ 퐶 0.5 ). Their rule of thumb suggests training on approximately 20 tokens for every trainable parameter in the model to achieve compute optimality. This result has since been reproduced [Be- siroglu et al., 2024] and has been shown to be robust to sizable perturbations [Schaeffer et al., 2025]. In their scaling laws, the loss is modeled as a function of the parameter count 푁 and the number of tokens 퐷 that have been trained on, as follows: 퐿(푁, 퐷) = 퐸+ 퐴 푁 훼 + 퐵 퐷 훽 (4.2) The first term 퐸 captures the irreducible loss, corresponding to the entropy of natu- ral text. The second term 퐴 푁 훼 captures the sub-optimality of a model with finite param- eters 푁 , and the final term 퐵 퐷 훽 captures the sub-optimality due to training on a finite number of tokens 퐷. Although 훼 and 훽 here play the same role as Kaplan’s 훼 푁 and 훼 퐷 (the exponents governing how the loss falls off with model size and data) they are a different parameterization: they are fit jointly within this two-term form and therefore take different values. Fitting the Chinchilla scaling function on more than 400 models found the following parameter estimates: 퐸 ≈ 1.69, 퐴≈ 406.4, 퐵≈ 410.7, 훼≈ 0.34, 훽≈ 0.28(4.3) By optimizing this loss function under the constraint of a fixed compute budget 퐶 (recall 퐶 ≈ 6푁퐷), we can derive the compute-optimal model size 푁 표푝푡 and data size 퐷 표푝푡 for any budget 퐶: 푁 표푝푡 (퐶) = 퐺 퐶 6 푎 , 퐷 표푝푡 (퐶) = 1 퐺 퐶 6 푏 (4.4) where 퐺 = 훼퐴 훽퐵 1 훼+훽 is a constant, 푎 = 훽 훼+훽 ≈ 0.45 and 푏 = 훼 훼+훽 ≈ 0.55. Note that 푎 and 푏 are not independent: by construction 푎 + 푏 = 1. This follows directly from 퐶 ≈ 6푁퐷, since taking logarithms gives log퐶 = log 6+ log 푁 + log 퐷. Writing 푁 표푝푡 ∝ 퐶 푎 and 퐷 표푝푡 ∝ 퐶 푏 , this becomes log퐶 = const+ 푎 log퐶+ 푏 log퐶, which can only hold for all 퐶 if 푎+ 푏 = 1. The same constraint applies to Kaplan’s exponents (0.73+ 0.27 = 1). The difference for Chinchilla was how the budget is split between 푁 and 퐷. This rule is approximately linear because 푁 표푝푡 and 퐷 표푝푡 grow with the compute budget 퐶 at almost the same rate: their exponents 푎 ≈ 0.45 and 푏 ≈ 0.55 are close, 4.1. SCALING LAWS: COMPUTE, DATA, PARAMETERS87 Model푁 퐷 퐷/푁Regime Gopher [Rae et al., 2021]280B300B ≈ 1undertrained GPT-3 [Brown et al., 2020]175B300B ≈ 1.7undertrained Chinchilla [Hoffmann et al., 2022]70B1.4T20compute-optimal LLaMA-1 7B [Touvron et al., 2023a]7B1.0T ≈ 143overtrained LLaMA-2 7B [Touvron et al., 2023b]7B2.0T ≈ 286overtrained LLaMA-3 8B [Grattafiori et al., 2024]8B15T ≈ 1875overtrained Table 4.1: Token-to-parameter ratios of several well-known models relative to the Chinchilla rule of thumb 퐷 ≈ 20푁 . so the ratio 퐷 표푝푡 /푁 표푝푡 stays near a constant value of roughly 20 across many orders of magnitude of compute. The value is best understood as an empirical calibration an- chored on real compute-optimal runs rather than a number that results cleanly from the fitted coefficients. The Chinchilla model itself (70B parameters trained on 1.4T tokens) sits exactly at 퐷/푁 = 20. Table 4.1 shows several well-known models relative to this rule. Models below the ratio are undertrained (too large for their token budget), while models far above it are deliberately overtrained to obtain a small model that is cheap to run at inference time. Earlier flagship models such as GPT-3 and Gopher fall well below the line (퐷/푁 ≈ 1–2): they were made very large but trained on comparatively too few tokens. It is worth stressing that 퐷 ≈ 20푁 is optimal only for minimizing the training loss at a fixed training budget. When a model will later serve many inference requests, it is often deliberately over-trained on far more than 20 tokens per parameter, trading additional training cost for a smaller model that is cheaper to run at inference time. LLaMA-3 8B, for instance, is trained on roughly 1900 tokens per parameter. This deliberate extreme over-training is chosen so that a small model still reaches a low loss, and is cheap to run during inference. The picture so far is fairly clean: test loss falls smoothly, monotonically, and pre- dictably with compute, data, and parameters. However, in practice, various factors influence our results. We discuss four phenomena that each probe a different edge of that picture: (1) compute-optimal curves, (2) emergent abilities, (3) inverse scaling, and (4) grokking. 4.1.3 Compute-Optimal Curve Computer hardware and energy are finite resources. A practical question is how to split a fixed compute budget 퐶 between model size 푁 and dataset size 퐷 to obtain the best possible model. Scaling laws answer this through the compute-optimal frontier: the allocation of 퐶 that minimizes the loss. This frontier is located empirically with the IsoFLOP method [Hoffmann et al., 2022], shown in Figure 4.3. For various model sizes, we choose the number of training tokens such that the final FLOPs is a constant. The learning rate is adjusted using a cosine annealing schedule, with the cosine cycle length is set to match the target FLOP count. We find a clear valley in loss, meaning that for a given FLOP budget there 88CHAPTER 4. SCALING LAWS AND EFFICIENCY Figure 4.3: IsoFLOP curves [Hoffmann et al., 2022] is an optimal model size to train (left). Using the location of these valleys, we project optimal model size and number of tokens for larger models (center and right). In green, we show the estimated number of parameters and tokens for an optimal model trained with the compute budget of Gopher. For a fixed FLOP budget we train several models of different sizes: a model that is too small underfits, while a model that is too large is forced to train on too few tokens, so the resulting curve of loss against model size is U-shaped. Its minimum identifies the compute-optimal model size 푁 표푝푡 , from which the matching token count 퐷 표푝푡 = 퐶/6푁 표푝푡 follows via 퐶 ≈ 6푁퐷. Fitting a parabola to each such curve locates this minimum, and repeating the procedure across budgets finds how 푁 표푝푡 and 퐷 표푝푡 grow with compute. This parabola fitting must be done with care, and the simplification introduces sys- tematic biases in the estimated optimum [Czech et al., 2026]. These biases persist even on noise-free synthetic data and stem from three sources: (1) spacing the sampled model sizes too coarsely, (2) sampling that fails to locate the true minimum symmetri- cally, and (3) the genuine asymmetry of the loss surface, which a symmetric parabola cannot represent. At the scale of Llama-3, they estimate that such biases waste on the order of $1.4M of compute. As a remedy, they recommend fitting the parametric loss model 퐿(푁, 퐷) = 퐸+ 퐴/푁 훼 +퐵/퐷 훽 directly—Chinchilla Approach 3, the same form we used above to derive the 퐷 ≈ 20푁 rule—with a numerically robust optimizer, rather than reading the optimum off fitted parabolas. Complementary views on scaling [Paquette et al., 2024] study analytically solvable models (power-law random features trained with one-pass stochastic gradient descent) instead of fitting curves to trained models. They find that the compute-optimal expo- nent is not a single universal constant, but that the data-complexity/target-complexity plane instead splits into four primary phases (plus three sub-phases), each with its own exponent governing how 푁 opt grows with compute. The answer to the question which allocation rule to apply, and whether a fixed 20 tokens per parameter is even the right shape, depends on the regime of the problem. Scaling for finetuning The preceding works studied how pretraining scales. Do these scaling laws also govern finetuning? Zhang et al. [2024] study this question for both full-model finetuning, which updates all of the model’s weights, and parameter- efficient finetuning (PEFT), which freezes the pretrained weights and trains only a 4.1. SCALING LAWS: COMPUTE, DATA, PARAMETERS89 Figure 4.4: Eight examples of emergence in the few-shot prompting setting, each point is a separate model [Wei et al., 2022]. small set of added parameters (as in LoRA) [Hu et al., 2021]. The lesson most relevant to compute budgeting is that finetuning performance improves more from scaling the size of the base model than from collecting additional finetuning data, while adding more trainable PEFT parameters helps little. In practice, a task that underperforms is usually better served by starting from a larger pretrained model than by enlarging the adapter or gathering more finetuning examples. 4.1.4 Emergent Abilities Some of the most interesting phenomena in LLM scaling are emergent abilities: capa- bilities that are not present in smaller models but that appear suddenly as model size or compute cross a certain threshold [Wei et al., 2022]. Examples include multi-step rea- soning, few-shot translation, and specialized coding tasks, as shown in Figure 4.4. The ability to perform a task via few-shot prompting is emergent when a language model achieves random performance until a certain scale, after which performance signifi- cantly increases to well-above random. However, the suddenness of the appearance has been contested. Schaeffer et al. [2023] argue that emergent abilities might be a mirage caused by the choice of evalua- tion metrics. For instance, in Figure 4.5, we see that while non-smooth discrete metrics (such as accuracy or exact match) show sharp jumps—they are non-smooth, after all— underlying metrics (such as the cross-entropy loss or token-level probability) follow a smooth, predictable power law across scales. 90CHAPTER 4. SCALING LAWS AND EFFICIENCY Figure 4.5: Claimed emergent abilities evaporate upon changing the metric from a nonlinear (top) to a smooth (bottom) metric [Schaeffer et al., 2023] 4.1.5 Inverse Scaling While most tasks improve with scale, some exhibit inverse scaling, where larger mod- els perform worse or develop stronger biases. This may happen in tasks where a dis- tractor answer is more probable according to the language modeling objective than the correct answer, leading larger (and more confident) models to consistently choose the wrong option [McKenzie et al., 2023, Michaelov and Bergen, 2023]. An example of inverse scaling is hindsight neglect, a task that tests whether a bet was worth taking based on its expected value. In case the expected value is positive, the model should respond with yes, and in case of a negative expected value it should respond with no. The situation is as follows. The model is only shown few-shot exam- ples where the actual outcome always matches the expected value (that is, the bettor won money when the expected value was positive and lost money when the expected value was negative). The model is then asked a question about whether it was correct to take a bet where the expected value and the actual outcome do not match. Hind- sight neglect tests whether models can correctly generalize from instructions when the few-shot examples are correctly-labeled but misleading. Repetitive algebra tests how models respond to simple algebra questions when given a specific pattern of correctly- labeled few-shot examples. Figure 4.6 shows that models generally start around ran- dom performance, but fall below random performance at around 10 22 training FLOPs. 4.1. SCALING LAWS: COMPUTE, DATA, PARAMETERS91 Figure 4.6: Scaling behavior for the Hindsight Neglect (left) and Repetitive Algebra (right) tasks [McKenzie et al., 2023] Figure 4.7: Early memorization of the training set (blue) versus late generalization (red: grokking) on a modular addition task [Nanda et al., 2023] 4.1.6 Grokking Grokking is a relatively recent paradox in statistical learning theory. This theory holds that, conventionally, when a large model is presented with a small dataset, it will overfit, due to memorization of the examples. Grokking, then, occurs when we continue to train this too-large model, and, after much more training, the model suddenly finds a generalizing solution [Power et al., 2022]. While originally observed on small algorithmic tasks, such as modular addition, the grokking phenomenon suggests that scaling the number of training steps (over- training) can lead to qualitative shifts from memorization to understanding, even with- out increasing the dataset size 퐷, see Figure 4.7. The figure shows a data wall that initially limits scaling as the data is running out [Villalobos et al., 2024], but later, after much more training, the model groks a generalizing solution. Grokking is relevant to language modeling since we are rapidly running out of high- quality human text on the internet to satisfy the requirements on data 퐷 of Chinchilla 92CHAPTER 4. SCALING LAWS AND EFFICIENCY scaling for next-generation models. Because of these inherent limits, researchers are looking beyond simple scaling of 퐷. As an alternative, we can look at a variety of these new frontiers: (1) better architectural efficiency (such as Mixture of Experts), (2) inference-time scaling (such as giving models more compute during inference to think, as seen in OpenAI’s o-series), or (3) synthetic data generation (Chapter 19). 4.2 Small Language Models As the requirements for compute and data continue to grow, there is an increasing in- terest in small language models (SLMs): models that are small enough to run on con- sumer hardware or even mobile or edge devices while maintaining high performance. These models are often developed using techniques that optimize for efficiency without sacrificing too much performance. To understand the deployment landscape for these models, it is helpful to categorize common types of compute infrastructure: • Datacenters: High-performance GPUs (such as NVIDIA H100s) and TPUs, pri- marily used for pretraining large models and hosting high-throughput inference APIs. • Consumer Hardware: Standard desktop GPUs and high-end CPUs, where SLMs can be affordably finetuned or run locally by individual users. • Edge Devices: Mobile phones, Internet-of-Things devices, and laptops, which are increasingly equipped with Neural Processing Units (NPUs) specifically de- signed for low-power, efficient on-device inference. To fit models into the latter constrained environments, optimization techniques fo- cus on speeding up throughput, reducing latency, reducing memory usage, and increas- ing sample efficiency. The techniques for achieving these goals can be grouped into two categories: (1) model-centric optimization, which targets the architecture of the models and their optimization approaches, and (2) data-centric optimization, which targets the data used to train the models. We will discuss both categories. 4.2.1 Model-centric Optimization First we discuss the model-centric optimization techniques. We introduce the following techniques that allow for efficient training of smaller models, while still maintaining high task performance. Model-centric techniques can focus on pretraining/finetuning or inference. 4.2.1.1 Reducing Pretraining Cost Parameter-Efficient Fine-Tuning (PEFT) Methods such as Low-Rank Adaptation (LoRA) [Hu et al., 2021] allow for adapting a large model to a new task by training only a tiny fraction (often < 1%) of its parameters [Lialin et al., 2023]. The key observation is that the weight update learned during finetuning has a low intrinsic rank: although 4.2. SMALL LANGUAGE MODELS93 Figure 4.8: Low-Rank Adaptation reparametrization [Hu et al., 2021]. During finetun- ing, only 퐴 and 퐵 are trained a pretrained weight matrix 푊 0 ∈R 푑×푘 is full-rank, the change Δ푊 needed to adapt it to a new task can be approximated well by a low-rank matrix. Instead of updating 푊 0 directly, LoRA freezes it and represents the update as a product of two small matrices, 푊 = 푊 0 + Δ푊 = 푊 0 + 퐵퐴, 퐵 ∈R 푑×푟 , 퐴 ∈R 푟×푘 (4.5) where the rank 푟 ≪ min(푑, 푘). For an input 푥, the modified forward pass becomes ℎ = 푊 0 푥+ 훼 푟 퐵퐴푥,(4.6) where 훼 is a constant scaling factor that controls the magnitude of the update, roughly similar to a learning rate in standard training. This is visualized in Figure 4.8. We compute weight updates only for 퐴 and 퐵, reducing the number of trainable parameters from 푑× 푘 to 푟(푑+ 푘). At initialization 퐴 is drawn from a random Gaussian and 퐵 is set to zero, so Δ푊 = 0 and training starts from the pretrained model. Because Δ푊 can directly be merged back into 푊 0 after training, LoRA adds no inference latency, and a single base model can be served with many small, swappable task-specific adapters. Architectural Innovations A second family of techniques changes the model archi- tecture itself so that it is cheaper to run, rather than compressing a fixed architecture after the fact. Two influential examples are Grouped-Query Attention and Mixture of Experts. Grouped-Query Attention (GQA) [Ainslie et al., 2023] targets the memory band- width bottleneck of autoregressive decoding. In standard multi-head attention (MHA), every query head has its own key and value matrix, which we all need to use for com- puting attention scores. At the other extreme, multi-query attention (MQA) lets all query heads share a single key/value head, which shrinks the computation but can hurt expressive quality and training stability. GQA interpolates between these two: the query heads are divided into 푔 groups, and all heads within a group share one key/- value head. With 푔 = 1 it is equivalent to MQA, and with 푔 equal to the number of 94CHAPTER 4. SCALING LAWS AND EFFICIENCY Figure 4.9: Illustration of the sparsely gated Mixture of Experts (MoE) architecture on top of a decoder-only transformer [Zhang et al., 2025] heads it is equivalent to MHA, so 푔 trades quality against memory usage. Often, GQA is combined with the KV-caching optimizations discussed next in Section 4.2, and it is now the default attention variant in many open-weight models. Mixture of Experts (MoE) [Shazeer et al., 2017] decouples the total number of parameters from the compute spent per token. Instead of a single dense feed-forward block in each layer, a MoE layer contains many parallel “expert” sub-networks together with a lightweight gating network (router) that activates only a small number of experts (typically the top-1 or 2) for each token. Because only the chosen experts perform computation, the per-token cost stays close to that of a single dense block even as the total parameter count (and hence the model’s overall capacity) grows by an order of magnitude or more [Fedus et al., 2022]. This sparse activation is why models such as Mixtral and many recent frontier sys- tems can have hundreds of billions of total parameters while activating only a fraction of them per token. The trade-offs are (1) that all experts must still be held in mem- ory, (2) that the load must be balanced across experts (often via an auxiliary loss) so that no expert is over- or under-used, and (3) that routing introduces additional engi- neering complexity. The MoE architecture is visualized in Figure 4.9, which shows a top-푘 routing mechanism, configured with 푘 = 2, where the gating function selects the two highest-scoring FFN experts for each token based on the router. The selected ex- perts are evaluated in parallel, with their outputs aggregated using weighted summation [Zhang et al., 2025]. Knowledge Distillation A large teacher model is used to train a smaller student model by encouraging the student to match the teacher’s output distribution [Gu et al., 2023, Xu et al., 2024]. Crucially, the student learns from the teacher’s full soft prob- abilities (typically softened with a temperature) rather than only the single correct an- swer. These soft targets carry richer information: for instance, which wrong answers the teacher considers plausible. This lets the student approach the teacher’s quality at a fraction of its size. The same idea underpins the synthetic-data generation discussed 4.2. SMALL LANGUAGE MODELS95 below, where a frontier model effectively acts as the teacher, generating high-quality training data for a smaller student model. 4.2.1.2 Reducing Inference Cost Once a model is trained, it can be further optimized for deployment at inference time. Whereas training-time techniques focus on reducing the cost of producing the model, inference-time techniques target the cost of using it: the throughput (tokens generated per second), the latency (time to the first and each subsequent token), and the memory footprint required to serve a request. We discuss several techniques that are commonly used to reduce inference cost, which can be applied to both large and small models. Key–Value (KV) Caching During autoregressive generation, the model produces one token at a time, and each new token attends to all previous tokens. Recomputing the attention keys and values for the entire context at every step is quadratically expensive. However, when we predict a new token, the context remains the same. Instead of recomputing the attention values, we can store the key and value vectors of past tokens in a KV cache and reuse them. This turns per-step attention into a linear-cost lookup. This is the basic optimization that makes decoding practical, but it comes at a cost: the cache grows linearly with both sequence length and batch size and quickly dominates memory usage for long contexts. This problem has motivated both the architectural choices above (GQA and MQA shrink the cache by sharing key/value heads across query heads) and dedicated memory-management schemes such as PagedAttention, which stores the cache in non-contiguous blocks to reduce fragmentation and enable higher serving throughput [Kwon et al., 2023]. Pruning A more extreme method for reducing model size is to consider removing re- dundant or less important connections in the neural network [Ma et al., 2023]. Pruning can be unstructured (zeroing out individual weights) or structured (removing entire neurons, attention heads, or layers); the latter yields concrete speedups on standard hardware, whereas the former requires sparsity-aware kernels to realize its benefits. Quantization Another method of saving memory during inference is to reduce the size of the matrix that needs to be loaded into memory. Rather than making the ma- trix dimensions smaller, we can also lower the precision of weights entries (e.g., from 16-bit to 8-bit or 4-bit numbers). This significantly reduces memory usage and speeds up inference. However, in principle, lower precision means lower performance. The same idea can also be applied to the KV cache itself to keep long contexts in mem- ory. Quantization is a frequently used approach [Dettmers et al., 2022, Dettmers and Zettlemoyer, 2023, Xiao et al., 2023]. Alternatively, rather than casting all weights to the same precision, mixed precision modes have been proposed, with different levels of numerical precision for different parts of the computation, to balance speed and stability [Micikevicius et al., 2017]. This approach may suffer from non-deterministic results due to numerical instabilities [Yuan et al., 2025]. 96CHAPTER 4. SCALING LAWS AND EFFICIENCY Speculative Decoding A key technique for optimizing inference efficiency is spec- ulative decoding [Leviathan et al., 2023]. This approach uses a small draft model to generate multiple tokens quickly, which are then verified in parallel by a larger target model. This allows for faster inference (up to 2-3 fold speedup) without sacrificing the quality of the larger model, effectively leveraging the different points on the scaling curve. 4.2.2 Data-centric Optimization In addition to model-centric approaches, the literature emphasizes that data quality and curation are often more important than the size of the dataset when training efficient models [Luo et al., 2025]. Small models can perform well if they are trained on highly curated, high-quality data. Hence, it is important to create a good data recipe, in ad- dition to an efficient model architecture. Data-centric techniques broadly fall into two camps: selecting the most useful parts from a broader pool of data, and generating new synthetic data where good data is scarce. 4.2.2.1 Data Selection and Curation Rather than training on all available text, a growing body of work shows that care- fully choosing which data to train on, and in what proportions, yields better models for the same compute budget. For pretraining, NVIDIA’s CLIMB framework [Diao et al., 2026] embeds a large corpus into a semantic space, clusters it, and then itera- tively searches for the data mixture that minimises proxy-model loss. The resulting ClimbMix dataset (400B tokens) lets a sub-billion-parameter model outperform com- parable models trained on far larger but unfiltered corpora. The same principle applies at the finetuning stage: LIMA [Zhou et al., 2023] shows that supervised finetuning on only 1,000 carefully curated prompt–response pairs is enough to produce a strong instruction-following model, supporting the view that most knowledge is acquired dur- ing pretraining and that alignment mainly teaches the model a desired response style. 4.2.2.2 Synthetic Data Gathering When suitable data does not exist, it can be generated. The Textbooks Are All You Need approach, used for models such as Microsoft’s Phi series [Gunasekar et al., 2023], demonstrates that training on clean, synthetic textbook-quality data can allow a rela- tively small model of 1.3B parameters to outperform models ten times its size. Such pipelines typically rely on frontier models to generate the data, combined with LLM-as-a-judge scoring to filter for the most accurate and educational examples, and rigorous decontamination to ensure the synthetic data does not leak benchmark test sets and inflate reported performance. This approach is particularly valuable for reasoning data: by training on step-by-step reasoning trajectories produced by larger models, complex problem-solving skills can be distilled into a smaller architecture (see also Chapter 6) [Muennighoff et al., 2025, Arora and Zanette, 2026]. 4.2. SMALL LANGUAGE MODELS97 4.2.2.3 Sample-Efficient Learning Beyond synthetic data, there is an increased research focus on how models can learn more efficiently from limited, naturalistic data. Initiatives such as the BabyLM Chal- lenge [Warstadt et al., 2023] focus on training models under strict data constraints (e.g., 10M to 100M tokens). This mirrors the volume of language input a human child re- ceives during early development, driving innovation in sample-efficient architectures while providing insights into cognitively plausible language acquisition. 4.2.3 Popular models Having surveyed improvement techniques, it is useful to look at how popular large and small models apply them in practice. Models are commonly categorized by their open- ness: closed-weight (proprietary) models expose only an API, open-weight models re- lease their trained parameters, and open-source models additionally share the training data and code. In practice openness is a matter of degree rather than a strict category, and we may subsequently use an openness index to rate the transparency [Artificial Analysis, 2026]. Most open models can be found on repositories such as Hugging Face. Below, we briefly describe some of the most popular open-weight models that are designed to be efficient enough to run on consumer hardware, and the techniques they use to achieve this. This is not an exhaustive list, but rather a selection of influential models that exemplify the trends in efficiency techniques and their scales, at the time of writing. Phi series [Abdin et al., 2024]. The Phi-series are capable SLMs that are efficient enough to run locally on mobile devices. The core innovation behind the Phi models is their data-centric training methodology. Microsoft curates high-quality, heavily filtered web data combined with synthetic data generated by larger mod- els. This allows small models (such as the 3.8 billion parameter Phi-3-mini) to perform well, rivaling the reasoning capabilities of much larger models such as GPT-3.5 and Mixtral. Gemma series [Mesnard et al., 2024] Google emphasizes that Gemma models inherit Gemini’s strong generalist capabilities, particularly in text, reasoning, and cod- ing, while advancing the state-of-the-art for their specific size classes, but little is known about their training method or data. OLMo family [Ettinger et al., 2025] The OLMo (Open Language Model) family is developed by the Allen Institute for AI to be a completely open-source model. The OLMo project open-sourced the full training pipeline, including code and data: the pre-training dataset (Dolma), the training and evaluation code, train- ing logs, and intermediate checkpoints. This transparency allows researchers to study training dynamics, data influences, and algorithmic biases. Qwen series [Bai et al., 2023] The Qwen family represents a high-performing suite of foundational and instruction-tuned models, ranging from sub-billion dense mod- els to large Mixture-of-Experts (MoE) architectures. They are notable for their 98CHAPTER 4. SCALING LAWS AND EFFICIENCY multilingual capabilities and performance on mathematics, coding, and logical reasoning benchmarks. OpenELM [Mehta et al., 2024] Apple’s Efficient Language Model family is designed to run efficiently on edge devices. The layer-wise scaling strategy uses smaller latent dimensions in layers closer to the input and widens them closer to the output. This results in high accuracy for low parameter count. gpt-oss [Agarwal et al., 2025] OpenAI released two open-weight reasoning models that use an efficient mixture-of-expert architecture, optimized for agentic capa- bilities. Mixtral series [Jiang et al., 2024] These models popularized Sparse Mixture-of-Experts (SMoE) architectures in the open-weights ecosystem (e.g., Mixtral 8x7B), al- lowing for large parameter counts with few active parameters during inference, to maximize efficiency. Llama series [Grattafiori et al., 2024] Meta’s Llama models have become frequently used. Trained on large datasets (up to 15 trillion tokens for Llama 3) and rang- ing from small edge-friendly sizes to large 405B parameter frontier models, they established the standard for scaling laws in open models and catalyzed an ecosys- tem of finetuned models and derivatives. DeepSeek [Liu et al., 2025] DeepSeek-V3 is a large Mixture-of-Experts model (671B total parameters with 37B activated per token) notable for the stability of its training. It combines Multi-head Latent Attention (MLA) for a compressed key- value cache with an auxiliary-loss-free strategy for expert load balancing, and was trained in FP8 mixed precision, achieving frontier-level performance at a small compute cost. The line also includes the DeepSeek-R1 reasoning models. GLM [Zeng et al., 2026] Developed by Zhipu AI, the GLM (General Language Model) family is built on strong reasoning capabilities that are developed early in train- ing, and retained throughout the training process. Recent iterations are opti- mized for agentic workflows and software engineering, targeting tool-use and code-generation performance rather than text completion alone. Pythia [Biderman et al., 2023] EleutherAI’s Pythia is a suite of models (ranging from 70M to 12B parameters) designed to support research into training dynamics and scaling. Every model is trained on the same data in the same order, with intermediate checkpoints released per model, enabling controlled studies of how capabilities, memorization, and biases emerge over the course of training. BLOOM [Scao et al., 2023] BLOOM is a 176B-parameter open-access model pro- duced by the BigScience collaborative research effort. Trained on the ROOTS corpus, spanning 46 natural languages and 13 programming languages, it was among the first large-scale models to prioritize multilingual coverage and full openness, released under a Responsible AI License. 4.3. INFERENCE-TIME COMPUTE99 SmolLM [Allal et al., 2025] Hugging Face’s SmolLM is a family of compact models (135M, 360M, and 1.7B parameters) aimed at efficient on-device deployment, much like the Phi series. Its performance stems from data curation: it is trained on the SmolLM-Corpus, which combines high-quality web data, Python code examples, and synthetically (Mixtral) generated textbooks and stories. 4.3 Inference-Time Compute The traditional scaling laws focus on the relationship between compute and perfor- mance at training time. More recent studies shift part of this computational burden to inference: inference-time scaling (or test-time compute) [Muennighoff et al., 2025]. Rather than spending all compute during training, a model is allowed to deliberate longer at inference time. This is measured by the number of tokens generated or rea- soning paths explored, in order to produce a higher-quality answer. Hence, optimizing throughput (tokens/s) becomes of prime importance: the faster tokens can be produced, the more reasoning a model is able to do during the same amount of time. Increasing this inference-time budget in terms of tokens can lead to significant improvements in capabilities [Du et al., 2024]; recent work indicates that optimally allocating test-time compute can allow smaller models to outperform larger ones on difficult reasoning tasks [Snell et al., 2024]. These gains are not guaranteed, however, and depend on the task and on the specific search or verification strategy employed [Stechly et al., 2023]. Beyond performance, this shift also reframes a broader question about what kind of systems LLMs are, which we take up in detail in Chapter 6. 4.4 Sustainable Language Models The large scale of LLM training and deployment has significant environmental and eco- nomic implications. The energy consumption of AI training and inference is large, and the subsequent water consumption for cooling comes with a strain on water distribution pipelines [Li et al., 2025, Jegham et al., 2025]. Measuring the energy use and carbon footprint of training has only recently entered standard reporting [Ding and Shi, 2024]. It has been reported that training a single large model can consume as much energy as hundreds of households use in a year. Training a model such as BLOOM emitted over 50 tonnes of CO 2 equivalent when accounting for both dynamic power consumption and the embodied carbon of hardware manufacturing [Luccioni et al., 2023]. As the use of LLMs is increasing in science and society, the total carbon footprint is increasingly dominated by inference rather than training, especially for models used by millions of people daily [ ̈ Ozcan et al., 2025, Wu et al., 2022, Samsi et al., 2023]. This creates an opportunity: spending more compute during training to create a smaller, more efficient model (as discussed in Section 4.1) can significantly reduce the long- term environmental cost of deployment. As global inference demand scales to billions of daily queries, the cumulative en- ergy consumption and associated financial cost overtake the initial training investment [Faiz et al., 2024]. Deploying large models at this scale requires a substantial data 100CHAPTER 4. SCALING LAWS AND EFFICIENCY center infrastructure, and large electricity and water consumption. Consequently, there is a strong push toward deploying Small Language Models (SLMs) for sustainable, large-scale use. By minimizing the parameter and quantization count, SLMs reduce the memory bandwidth and compute required per generated token, offering a greener and more economically viable solution. Privacy and Data Leakage Beyond environmental concerns, the centralization of LLMs through cloud APIs introduces privacy vulnerabilities. When users query cloud- based models, sensitive personal data, corporate intellectual property, or protected health information must be transmitted to third-party servers. This creates risks of data leakage [Aiyappa et al., 2023, Li et al., 2024], either through unauthorized interception, adversarial extraction attacks, or the inadvertent memorization and subsequent regur- gitation of private user prompts by the model [Chen et al., 2024, Maini et al., 2024, Liu et al., 2022]. To mitigate these privacy risks and reduce reliance on cloud infrastructure, devel- opers can move from API-based inference toward local, on-device inference [Qu et al., 2025]: • API-Based Inference: API-based inference relies on powerful remote servers. While this allows access to frontier models with hundreds of billions of parame- ters, it suffers from network latency, requires constant internet connectivity, and inherently exposes user data to cloud providers. • Local Inference: Local inference runs the model directly on the user’s hardware (e.g., smartphones, laptops, or edge servers). This ensures data privacy because prompts never leave the device while also eliminating network latency and guar- anteeing offline availability. Local inference is constrained by the physical limits of consumer and edge hardware. For instance, memory and bandwidth are limited. LLM inference is a memory-bound process. Storing model weights requires significant RAM capacity, while generating tokens requires a high memory bandwidth. Furthermore, most mobile devices are pow- ered by a battery. Running intensive matrix multiplications on mobile devices or stan- dard laptops can lead to rapid battery drain and thermal throttling. To make local deployment feasible, developers must rely on the inference-time optimization techniques, such as quantization (e.g., 4-bit) and speculative decoding, alongside the increasing availability of specialized neural processing units (NPUs) de- signed for low-power edge AI. 4.5 Concluding Remarks As LLM use becomes more ubiquitous and the demand for quick responses increases, the focus in LLM development is shifting towards more efficient solutions. Hence, the focus on scaling data, compute, and model size purely for raw performance is transitioning into a focus on scaling for efficiency. BIBLIOGRAPHY101 While the fundamental power laws still hold, the focus has shifted from simply training the largest possible model toward compute-optimal training, inference-time scaling, and initiatives to develop capable small models that run on consumer and edge hardware. The directions that have been discussed in this chapter are complementary rather than competing: the model-centric techniques (LoRA, GQA, MoE, quantization, distil- lation) and the data-centric techniques (careful data selection and synthetic generation) are increasingly combined within a single training recipe. As we move closer to the data wall, the role of synthetic data and architectural efficiency are likely to grow in importance, and the ability to do more with less is becoming a research focus. Bibliography Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, S ́ ebastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https://arxiv.org/abs/2404.14219. 97 Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, et al. gpt- oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/ 2508.10925. 98 Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Le- bron, and Sumit Sanghai.GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, pages 4895–4901, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.298. URL https://aclanthology.org/2023.emnlp-main.298/. 93 Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-yeol Ahn. Can we trust the evaluation on ChatGPT? In Anaelia Ovalle, Kai-Wei Chang, Ninareh Mehrabi, Yada Pruksachatkun, Aram Galystan, Jwala Dhamala, Apurv Verma, Trista Cao, Anoop Kumar, and Rahul Gupta, editors, Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 47–54, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.trustnlp-1. 5. URL https://aclanthology.org/2023.trustnlp-1.5/. 100 Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martin Blazquez, Guil- herme Penedo, Lewis Tunstall, Andr ́ es Marafioti, Agust ́ ın Piqueres Lajar ́ ın, Hynek Kydl ́ ı ˇ cek, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan Son NGUYEN, 102BIBLIOGRAPHY Ben Burtenshaw, Cl ́ ementine Fourrier, Haojun Zhao, Hugo Larcher, Mathieu Mor- lon, Cyril Zakka, Colin Raffel, Leandro Von Werra, and Thomas Wolf. SmolLM2: When smol goes big — data-centric training of a fully open small language model.In Second Conference on Language Modeling, 2025.URL https: //openreview.net/forum?id=3JiCl2A14H. 99 Daman Arora and Andrea Zanette. Training language models to reason efficiently. Advances in Neural Information Processing Systems, 38:60770–60808, 2026. 96 ArtificialAnalysis.Artificialanalysisopennessindex,2026. URL https://artificialanalysis.ai/evaluations/ artificial-analysis-openness-index.Accessed:2026-03-16. 97 Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, et al. Qwen technical report, 2023. URL https: //arxiv.org/abs/2309.16609. 97 Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102, 2024. 86 Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023. 98 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 87 Xiaoyi Chen, Siyuan Tang, Rui Zhu, Shijun Yan, Lei Jin, Zihao Wang, Liya Su, Zhikun Zhang, XiaoFeng Wang, and Haixu Tang. The janus interface: How fine-tuning in large language models amplifies the privacy risks. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1285– 1299, 2024. 100 Eric Czech, Zhiwei Xu, Yael Elmatad, Yixin Wang, and William Held. Problems with chinchilla approach 2: Systematic biases in isoflop parabola fits, 2026. URL https://arxiv.org/abs/2603.22339. 88 Tim Dettmers and Luke Zettlemoyer.The case for 4-bit precision: k-bit infer- ence scaling laws. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Bar- bara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7750–7774. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/dettmers23a.html. 95 BIBLIOGRAPHY103 TimDettmers,MikeLewis,YounesBelkada,andLukeZettlemoyer. Gpt3.int8():8-bit matrix multiplication for transformers at scale.In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30318–30332. Curran Associates,Inc.,2022.URL https: //proceedings.neurips.c/paper_files/paper/2022/file/ c3ba4962c05c49636d4c6206a97e9c8a-Paper-Conference.pdf. 95 Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Nemotron-climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. Advances in Neural Information Processing Systems, 38, 2026. 96 Yi Ding and Tianyao Shi. Sustainable llm serving: Environmental implications, chal- lenges, and opportunities. In 2024 IEEE 15th International Green and Sustainable Computing Conference (IGSC), pages 37–38. IEEE, 2024. 99 Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Im- proving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. 99 Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morri- son, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, et al. Olmo 3, 2025. URL https://arxiv. org/abs/2512.13961. 97 Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang.Llmcarbon: Modeling the end-to-end carbon footprint of large language models.In B. Kim, Y. Yue, S. Chaudhuri, K. Fragki- adaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, volume 2024, pages 24727–24741, 2024.URL https: //proceedings.iclr.c/paper_files/paper/2024/file/ 6b4ac044095525631df38e20919b45d2-Paper-Conference.pdf. 99 William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. URL http://jmlr.org/papers/ v23/21-0998.html. 94 Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. 87, 98 104BIBLIOGRAPHY Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2023. 94 Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C ́ esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, S ́ ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https://arxiv.org/abs/2306.11644. 96 Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driess- che, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran As- sociates Inc. ISBN 9781713871088. 86, 87, 88 Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 89, 92, 93 Nidhal Jegham, Marwan Abdelatti, Chan Young Koh, Lassad Elmoubarki, and Ab- deltawab Hendawi. How hungry is ai? benchmarking energy, water, and carbon foot- print of llm inference, 2025. URL https://arxiv.org/abs/2505.09598. 99 Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L ́ elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Th ́ eophile Gervet, Thibaut Lavril, Thomas Wang, Timoth ́ e Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https://arxiv.org/abs/2401.04088. 98 Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Re- won Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001. 08361. 84, 85, 86 Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702297. doi: 10.1145/3600006.3613165. URL https://doi.org/10.1145/3600006. 3613165. 95 BIBLIOGRAPHY105 Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023. 96 Pengfei Li, Jianyi Yang, Mohammad A. Islam, and Shaolei Ren. Making ai less ’thirsty’.Commun. ACM, 68(7):54–61, June 2025.ISSN 0001-0782.doi: 10.1145/3724499. URL https://doi.org/10.1145/3724499. 99 Yucheng Li, Frank Guerin, and Chenghua Lin. An open source data contamination re- port for large language models, 2024. URL https://arxiv.org/abs/2310. 17589. 100 Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv preprint arXiv:2303.15647, 2023. 92 Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, et al. Deepseek-v3 technical report, 2025. URL https: //arxiv.org/abs/2412.19437. 98 Lan Liu, Yi Wang, Gaoyang Liu, Kai Peng, and Chen Wang. Membership inference at- tacks against machine learning models via prediction sensitivity. IEEE Transactions on Dependable and Secure Computing, 20(3):2341–2347, 2022. 100 Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253):1–15, 2023. URL http://jmlr.org/papers/ v24/23-0069.html. 99 Junyu Luo, Bohan Wu, Xiao Luo, Zhiping Xiao, Yiqiao Jin, Rong-Cheng Tu, Nan Yin, Yifan Wang, Jingyang Yuan, Wei Ju, and Ming Zhang. A survey on effi- cient large language model training: From data-centric perspectives. In Wanxi- ang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, ed- itors, Proceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 30904–30920, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1493. URL https://aclanthology.org/ 2025.acl-long.1493/. 96 Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 21702–21720, 2023. 95 Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset in- ference: Did you train on my dataset? Advances in Neural Information Processing Systems, 37:124069–124092, 2024. 100 106BIBLIOGRAPHY Ian R. McKenzie, Alexander Lyzhov, Michael Martin Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Xudong Shen, Joe Cavanagh, An- drew George Gritsevskiy, Derik Kauffman, Aaron T. Kirtland, Zhengping Zhou, Yuhui Zhang, Sicong Huang, Daniel Wurgaft, Max Weiss, Alexis Ross, Gabriel Recchia, Alisa Liu, Jiacheng Liu, Tom Tseng, Tomasz Korbak, Najoung Kim, Samuel R. Bowman, and Ethan Perez. Inverse scaling: When bigger isn’t bet- ter. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=DwgRm72GQF. Featured Certifica- tion. 90, 91 Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. Openelm: An efficient language model family with open training and inference framework, 2024. URL https://arxiv.org/abs/ 2404.14619. 98 Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi ` ere, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, L ́ eonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, et al. Gemma: Open models based on gemini research and technology, 2024. URL https://arxiv.org/abs/2403.08295. 97 James A. Michaelov and Benjamin K. Bergen. Emergent inabilities? inverse scal- ing over the course of pretraining. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14607–14615, Singapore, December 2023. Association for Compu- tational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.973. URL https: //aclanthology.org/2023.findings-emnlp.973/. 90 Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017. 95 Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Han- naneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand ` es, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025. 96, 99 Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023. 91 Miray ̈ Ozcan, Philipp Wiesner, Philipp Weiß, and Odej Kao. Quantifying the energy consumption and carbon emissions of llm inference via simulations. arXiv preprint arXiv:2507.11417, 2025. 99 Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+3 phases of compute-optimal neural scaling laws. In The Thirty-eighth Annual Conference on BIBLIOGRAPHY107 Neural Information Processing Systems, 2024. URL https://openreview. net/forum?id=aVSxwicpAk. 88 Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022. URL https://arxiv.org/abs/2201.02177. 91 Guanqiao Qu, Qiyuan Chen, Wei Wei, Zheng Lin, Xianhao Chen, and Kaibin Huang. Mobile edge intelligence for large language models: A contemporary survey. IEEE Communications Surveys & Tutorials, 27(6):3820–3860, 2025. 100 Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Fran- cis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. 87 Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmarking the energy costs of large language model in- ference. In 2023 IEEE high performance extreme computing conference (HPEC), pages 1–9. IEEE, 2023. 99 Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ́ c, Daniel Hesslow, Roman Castagn ́ e, Alexandra Sasha Luccioni, Franc ̧ois Yvon, Matthias Gall ́ e, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, et al. Bloom: A 176b-parameter open-access multilingual language model, 2023. URL https://arxiv.org/abs/2211.05100. 98 Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in neural information processing systems, 36: 55565–55581, 2023. 89, 90 Rylan Schaeffer, Noam Levi, Andreas Kirsch, Theo Guenais, Brando Miranda, Elyas Obbad, and Sanmi Koyejo. Evaluating the robustness of chinchilla compute-optimal scaling. arXiv preprint arXiv:2509.23963, 2025. 86 Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geof- frey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of-experts layer, 2017. URL https://arxiv.org/abs/1701. 06538. 94 Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time com- pute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024. 99 Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. GPT-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL https:// openreview.net/forum?id=PMtZjDYB68. 99 108BIBLIOGRAPHY Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ́ e Lacroix, Baptiste Rozi ` ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lam- ple. Llama: Open and efficient foundation language models, 2023a. URL https: //arxiv.org/abs/2302.13971. 87 Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. 87 Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Mar- ius Hobbhahn. Position: Will we run out of data? limits of LLM scaling based on human-generated data. In Forty-first International Conference on Machine Learn- ing, 2024. URL https://openreview.net/forum?id=ViZcgDQjyG. 91 Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell, editors, Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Lan- guage Learning, pages 1–34, Singapore, December 2023. Association for Com- putational Linguistics.doi: 10.18653/v1/2023.conll-babylm.1.URL https: //aclanthology.org/2023.conll-babylm.1/. 97 Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022. 89 Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Can- dido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin Lee, Bu- gra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood.Sustainable ai:Environmental implications, chal- lenges and opportunities.In D. Marculescu, Y. Chi, and C. Wu, editors, Proceedings of Machine Learning and Systems, volume 4, pages 795–813, 2022. URL https://proceedings.mlsys.org/paper_files/paper/ 2022/file/462211f67c7d858f663355eff93b745e-Paper.pdf. 99 Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engel- hardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th In- ternational Conference on Machine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pages 38087–38099. PMLR, 23–29 Jul 2023.URL https://proceedings.mlr.press/v202/xiao23c.html. 95 BIBLIOGRAPHY109 Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024. 94 Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and mitigating numerical sources of nondeterminism in LLM inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview. net/forum?id=Q3qAsZAEZw. 95 Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Hui- long Chen, et al. Glm-5: from vibe coding to agentic engineering, 2026. URL https://arxiv.org/abs/2602.15763. 98 Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets LLM finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview.net/forum?id=5HCnKDeTws. 88 Danyang Zhang, Junhao Song, Ziqian Bi, Xinyuan Song, Yingfang Yuan, Tianyang Wang, Joe Yeong, and Junfeng Hao. Mixture of experts in large language models, 2025. URL https://arxiv.org/abs/2507.11181. 94 Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021, 2023. 96 110BIBLIOGRAPHY Chapter 5 Case Study 1: Building LLMs from Scratch What You Will Learn In this case study you will learn how to train your own model. Specifically, you will be able to: • Explain the end-to-end training pipeline of a ChatGPT-style LLM, distinguish- ing the purpose and mechanics of each stage: tokenization, pre-training, mid- training, and supervised fine-tuning. • Configure and train a Byte Pair Encoding (BPE) tokenizer and articulate the trade-offs involved in choosing a vocabulary size. • Describe the transformer architecture at a mechanistic level: attention heads, layer depth, embedding dimensions. You will investigate how these hyperpa- rameters interact with model capacity and compute budget. • Evaluate a trained language model on standard benchmarks (ARC-Easy, ARC- Challenge, GSM-8K) and interpret results critically, including failure modes. • Critically reflect on the ethical, environmental, and access dimensions of LLM training, including cost as a barrier to entry and data curation as a values-laden choice. • Access and navigate the information related to open-access LLM models. This includes reading model cards, training reports, reading code related to training LLMs, and other types of hub entries. 5.1 Building a Chatbot We will work through a practical case that covers the full lifecycle of a modern ChatGPT- style language model. We use nanochat, 1 a minimal, single-codebase, full-stack im- plementation of an instruction-following LLM, released by Andrej Karpathy in April 1 https://github.com/karpathy/nanochat 111 112CHAPTER 5. CASE STUDY 1: BUILDING LLMS FROM SCRATCH 2025. Unlike his earlier nanoGPT, which covered pre-training only, nanochat imple- ments the complete training pipeline: a Byte Pair Encoding tokenizer, pre-training on web text followed by supervised fine-tuning (SFT) on benchmark datasets, an inference server, and a chat web UI. The code uses mostly pure Python with minimal dependencies, and therefore is easy to use and extend. The codebase comprises roughly 8,000 lines of Python and a small amount of Rust. 2 The performance of nanochat is striking: a GPT-2-capable LLM (which cost∼$43,000 to train in 2019 [Radford et al., 2019]) for only $48 (∼2 hours of 8xH100 GPU node). While this compute is not exactly commonplace for standard student projects at a uni- versity, we can scale down the experiment to make it suited to a hands-on course as- signment. It is important to realize that every gradient update and every design deci- sion is visible, from raw bytes to a working chat assistant. We will take you trough the nanochat pipeline across four tasks that you should all perform. Because GPU budgets on a course are limited, you will train a small model (depth 4, depending on availability, see Section 5.3); you will then reflect on what may happen at larger scale using some theory and the published (nanochat) benchmarks. 5.2 Background: Karpathy’s nanochat The case study will give you direct, hands-on experience with every stage of the modern LLM training pipeline, from input text to a deployed chat assistant. This experience will ground the theoretical concepts from Chapters 2–4 in concrete, debuggable code. nanochat is configured out of the box to train a compute-optimal model by set- ting a single complexity dial: --depth, which specifies the number of transformer layers. All other hyperparameters—embedding width, number of attention heads, learning rate schedule, training horizon, weight decay—are derived automatically in a near-optimal way. This keeps the configuration space manageable while preserving the full educational value of understanding what each parameter does. The training pipeline proceeds with the by now hopefully familiar stages, which are briefly described next. Pretraining The model is trained on approximately 6 billion tokens drawn from CLIMBMix [Diao et al., 2026], a clustering-based iteratively bootstrapped data mix- ture corpus. A depth-24 (∼1.4B-parameter) model trains to a validation loss of 2.41 (0.717 bits-per-byte) in roughly 4 hours on a 4×H100 node compute cluster. Supervised finetuning (SFT) Starting from the pretrained checkpoint, the model is finetuned on a mixture of instruction-following and reasoning data: SmolTalk [Ben Al- lal et al., 2024] (460K general-purpose conversations), MMLU (multiple-choice knowl- edge questions, 3 epochs), GSM-8K (grade-school mathematics, 4 epochs), identity 2 If that number sounds large, it is worth noting that the corresponding proprietary systems run to millions of lines. Nanochat is aimed to be a small self-contained version. 5.3. EXPERIMENT PREPARATION113 conversations, and synthetic spelling tasks—approximately 1.1 million training exam- ples in total. Note that the cross-entropy loss is computed only on assistant tokens, not on user prompts, mirroring the InstructGPT training procedure [Ouyang et al., 2022]. Before each training stage, a BPE tokenizer is trained on 2 billion characters using a Rust implementation, producing a vocabulary of 32,768 tokens in under one minute. A web inference server and a lightweight JavaScript chat UI are included out of the box, allowing immediate interaction with your trained model. Supplementary resources Two freely available courses provide complementary per- spectives on the material in this case study. The Hugging Face LLM Course [Hug- ging Face, 2022] introduces the transformers, datasets, tokenizers, and accelerate libraries, and the Hugging Face Hub through which most open-weight models are distributed and evaluated today. Mathieu Labonne’s LLM Course [Labonne, 2023] provides a practitioner roadmap covering pre-training data curation, parameter- efficient fine-tuning (LoRA, QLoRA), preference alignment (DPO, PPO, GRPO), quan- tization (GGUF, GPTQ), and production inference serving (vLLM, TGI). Together these resources situate nanochat within the broader open-source LLM ecosystem and are referenced explicitly in Tasks 3 and 4. You are also free to browse around for ad- ditional resources. It is strongly recommended to link and reference sources in your report where needed. 5.3 Experiment Preparation Repository Clone the course repository and read the README carefully before writing any code. Listing 5.1: Cloning the nanochat repository 1 git clone git@github.com:karpathy/nanochat.git Compute No dedicated compute is necessary. You may use your own laptop (with a GPU), or a larger computer. The experiments are planned so that each run completes a maximum of approximately 8 hours. For pretraining, use a --depth 4 configuration; --depth 2 is acceptable only if your hardware does not permit the larger model. Do not attempt depth≥12 unless you have some serious hardware. Logging nanochat logs training loss, validation loss, and bits-per-byte at configurable intervals to your terminal, and additional functionality for reporting and saving infor- mation during the run is available. It is therefore wise to enable structured logging to a .jsonl file from the start. Retrofitting logging after a run is painful. You may want to experiment with the logging file formats. We recommend using a service such as wandb.ai before training full models. 114CHAPTER 5. CASE STUDY 1: BUILDING LLMS FROM SCRATCH 5.4 Task 0: Model Hub warm-up. Description Before starting the tasks, it is a good idea to get a feel for common pa- rameter settings and common training settings for different models. For this purpose, practice navigating the Hugging Face Hub. 3 For each of the three model IDs in Ta- ble 5.1, locate and record: 1. the number of training tokens or steps, 2. the model architecture (number of layers, hidden dimension, attention heads, number of trainable parameters), and 3. the training data sources, if possible. Model IDOrganization allenai/OLMo-1BAllen Institute for AI HuggingFaceTB/SmolLM2-360MHugging Face Qwen/Qwen2.5-0.5BAlibaba Cloud Table 5.1: Model IDs for the Hub warm-up exercise Pay attention to where you look for each piece of information: some fields appear directly on the model card, others require inspecting config.json under the Files tab, and some are only available in the accompanying paper. Include a short table with your findings in your report, noting the source for each field. 5.5 Task 1: Tokenization 5.5.1 What to Do Train a BPE tokenizer using nanochat’s Rust tokenizer trainer on a 500 million char- acter sample of the CLIMBMix dataset. Run two experiments: one with a vocabulary size of 8,192 tokens and one with 32,768 tokens. 5.5.2 Deliverables • A short script that reproduces your tokenizer training run. • A report section (at most 1 page) answering the following questions with evi- dence from your own tokenizer outputs: 1. What is BPE, and how does it differ from word-level and character-level tokenization? Draw a merge tree for a concrete example (you can look for one in the data, or make your own. Do not reuse an existing example!). Make sure to also include the counts for your merges. 3 https://huggingface.co/models 5.6. TASK 2: PRETRAINING115 2. How does vocabulary size affect compression ratio (tokens per character) and sequence length for a fixed text sample? Show measurements from both of your tokenizers on English text. 3. Why does vocabulary size matter for the downstream language model? In your answer, consider the embedding matrix size, the softmax denominator, and the challenge of learning reliable representations for rare tokens. 4. What artifacts or failure cases do you observe? How does your tokenizer handle numbers, source code, or non-English text? Pay attention to: correctness of the BPE explanation; quality of empirical evidence; depth of analysis on vocabulary trade-offs; perceptiveness of failure-case discussion. 5.6 Task 2: Pretraining 5.6.1 What to Do Pretrain a transformer using nanochat with --depth 4 (or, in case your computer is not fast enough, --depth 2) on the CLIMBMix dataset. Log training loss, validation loss, and bits-per-byte at regular checkpoints. 5.6.2 Deliverables • Bits-per-byte curves for training and validation set (plots, to be included in the report). Use bpb instead of the raw loss. • A saved model checkpoint. • A report section (approximately 2 pages) covering: 1. The Depth Parameter: Explain what the --depth argument controls in nanochat and how the codebase derives all other hyperparameters automat- ically from it: embedding width, number of attention heads, learning rate schedule, training horizon. For the chosen depth, state the number of lay- ers, attention heads, embedding dimension, the total trainable parameter count, and the number of total tokens the model will be trained on. Why is a single-dial design useful for such a codebase? What are its limitations compared to tuning each hyperparameter independently? 2. Scaling Laws: Using the Chinchilla framework [Hoffmann et al., 2022], compute the compute-optimal number of training tokens for your model’s parameter count. How does nanochat’s default training horizon compare? What does it mean to be under-trained or over-trained? 3. Loss Analysis: Plot the progress of bits-per-byte (bpb) during training as your training curve and explain its shape. At what point does the curve flatten? What does the gap between training and validation tell you? 116CHAPTER 5. CASE STUDY 1: BUILDING LLMS FROM SCRATCH 4. Qualitative Inspection: Sample five completions from your pretrained model with no system prompt (raw next-token prediction). What is the model able to do at this stage, and why do you think that is the case? Pay attention to: correctness of the depth-parameter explanation and auto-configuration description; accuracy of architecture description; correct application of scaling law calculations; quality of loss analysis; insightfulness of qualitative assessment. 5.7 Task 3: Mid-Training and Supervised Finetuning 5.7.1 What to Do Recall that nanochat only performs two training stages: pretraining and SFT. However, in the book, we discussed that additional stages in the form of mid-training can help the model perform better. In this task, you will implement a new mid-training stage by modifying scripts/chat sft.py. The goal is to separate reasoning adaptation (mid-training) from instruction following (SFT) and measure the effect empirically. Stage 1 — Mid-training Modify the traintasks list in chatsft.py so that it contains only reasoning and knowledge tasks: MMLU and GSM-8K. Remove all other conversational and spelling data. Run chat sft.py from your pre-trained base checkpoint and save the resulting checkpoint. Stage 2 — SFT Modify chatsft.py again so that traintasks contains only SmolTalk. You will also need to make a small change so the script loads your Stage 1 checkpoint rather than the base pre-trained model. Inspect the loadmodel call in chat sft.py as a starting point. 4 Run chatsft.py from your Stage 1 checkpoint to proceed with SFT after mid-training. After each stage, evaluate with scripts/chateval.py on ARC-Easy, ARC- Challenge, and GSM-8K. Sanity check: Before running Stage 1, inspect the data: print the first few rows of the MMLU and SmolTalk datasets to understand their format and content. Make sure you are able to answer: how many training examples does each dataset contribute, and what does a single training example look like? 5.7.2 Deliverables • Two checkpoints, one for each stage (mid-training and SFT). • Evaluation scores on ARC-Easy, ARC-Challenge, and GSM-8K after each stage, reported as a table. This should be part of the report (see below). • A report section (approximately 3 pages) covering: 4 Hint: read nanochat/checkpointmanager.py to understand how checkpoints are located. 5.8. TASK 4: INFERENCE, DEPLOYMENT, AND CRITICAL REFLECTION 117 1. Implementation: Describe the changes you made to the codebase such that you can do training in stages. List the hyperparameters that you are using for the training experiments. You are free to experiment with them! 2. Loss Masking: Why is the SFT loss computed only on assistant tokens? Construct a concrete token-level example illustrating what the loss would look like with and without masking. 3. Dataset Analysis: For each stage, describe the dataset used, its approximate size, and why it was chosen for that stage specifically. What values or potential biases might be embedded in dataset training splits? How might these shape model behavior? 4. Benchmark Scores: Report your evaluation scores after each stage in a table. What changed between Stage 1 and Stage 2, and why? Also compare to the pretrained results such that your analysis covers all three stages. Note any results that surprised you. 5. Parameter-efficient Finetuning: nanochat performs full fine-tuning: every parameter is updated during SFT. Describe how LoRA (Low-Rank Adap- tation) and its quantized variant QLoRA differ from full fine-tuning. What are the trade-offs in compute, memory, and final model quality? Would you recommend using PEFT techniques for the depths we have used in training so far? Pay attention to: correctness and clarity of the implementation description; correctness of the loss-masking explanation; critical quality of dataset analysis; accuracy and com- pleteness of reported benchmark scores and their interpretation; conceptual accuracy of the LoRA/QLoRA comparison. 5.8 Task 4: Inference, Deployment, and Critical Reflec- tion 5.8.1 Part A: Inference and Deployment Launch nanochat’s inference server and web UI. Interact with your trained model. Then address the following in your report (approximately 1 page): 1. The Inference Loop: Describe how autoregressive sampling works, including the role of the key–value (KV) cache in avoiding redundant computation across decoding steps. Play around with decoding with different parameters: What is temperature, and what effect does varying it (try 0.1, 0.7, and 1.5) have on output quality and diversity? Show and discuss five representative examples from your own model. 2. Production Hardening: Identify three concrete engineering changes that the nanochat inference server would need before deployment at production scale. 118CHAPTER 5. CASE STUDY 1: BUILDING LLMS FROM SCRATCH 3. Capability Comparison and Scaling: Look up published benchmark scores on ARC-Easy, ARC-Challenge, and GSM-8K for at least two publicly available models: one in a comparable size range to your nanochat model, and one sub- stantially larger. Feel free to find scores for any type of inference design (e.g., zero-shot, few-shot, others) but report the specific conditions. Model cards and leaderboard entries on the Hugging Face Hub are a good starting point; cite every source you use. Build a table placing your own results alongside the published figures. Then address the following two questions. Where does your model fall relative to models of similar parameter count? What does the gap between your model and a well-trained model of the same size reveal about the role of data quality, training duration, and alignment? 5.8.2 Part B: Critical Reflection Write a short discussion (approximately 0.5-1 page) responding to one of the following three prompts. Prompt 1 — Access and Democratization nanochat’s headline claim is that “the best ChatGPT that $100 can buy” brings LLM training within reach of individuals. Critically evaluate this claim. Who still cannot access this pipeline, and why? Prompt 2 — Data and Values nanochat pretrains on CLIMBMix, a clustering- based iteratively bootstrapped data mixture corpus. What decisions were made in constructing this dataset, and whose values do they embed? How might a different pre-training corpus lead to a qualitatively different model? Prompt 3 — Environmental Cost nanochat’s “affordable” training run con- sumes approximately four hours of 8×H100 GPUs. Estimate the carbon cost of your own training run using an established methodology (e.g. the ML CO 2 Impact calculator). 5 Situate this in the broader context of LLM environmental impact. Is cheap the same as low-impact? Part A: pay attention to: correctness of inference loop description; practicality of production hardening proposals; completeness of the benchmark description and rele- vance of discussion to state-of-the-art; Part B: pay attention to: analytical depth; use of evidence; willingness to challenge assumptions. 5.9 Recommended Reading • Vaswani et al. [2017] — The original transformer paper; essential reading before Task 2. 5 https://mlco2.impact.compute.org 5.10. CONCLUDING REMARKS119 • Hoffmann et al. [2022] — Chinchilla scaling laws; required for the scaling-law question in Task 2. • Ouyang et al. [2022] — InstructGPT; the basis for SFT and RLHF in Task 3. • Diao et al. [2026] — CLIMBMix / Nemotron-CLIMB; read the data curation methodology before Task 2’s dataset analysis question. • Bender et al. [2021] — Critical perspective on large language models; relevant to the Task 4 reflection prompts. • Hugging Face [2022] — The Hugging Face LLM Course; covers the transformers, tokenizers, and datasets libraries and the Hugging Face Hub; useful con- text for Tasks 1, 3, and 4. • Labonne [2023] — Labonne’s LLM Course; practitioner roadmap covering Lo- RA/QLoRA, DPO, GGUF/GPTQ quantization, and vLLM/TGI serving; recom- mended before Tasks 3 and 4. 5.10 Concluding Remarks In this chapter we have used nanochat as a window into the complete LLM training pipeline. You have trained a BPE tokenizer, observed a transformer learn from web text, adapted that model to follow instructions, evaluated it on standard benchmarks, and deployed it as a chat assistant. The gap between your depth-4 model and the frontier will be obvious, because we don’t expect you to have access to datacenter-grade compute capabilities. However, the experiments are, in theory, scalable: rerunning at higher depths and with additional data resources should lead to more capable models, as predicted by the scaling laws. After completion of Assignment 1, you have essentially trained a full LLM by yourself, and you will be better equipped for future experiments where an increased scale is necessary. Bibliography Loubna Ben Allal et al. SmolTalk, 2024. URL https://huggingface.co/ datasets/HuggingFaceTB/smoltalk. 112 Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.3445922. URL https://doi.org/10.1145/3442188.3445922. 119 Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, et al. Nemotron-climb: Clustering-based 120BIBLIOGRAPHY iterative data mixture bootstrapping for language model pre-training. Advances in Neural Information Processing Systems, 38, 2026. 112, 119 Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driess- che, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran As- sociates Inc. ISBN 9781713871088. 115, 119 Hugging Face.The hugging face LLM course, 2022.URL https:// huggingface.co/learn/llm-course. 113, 119 Mathieu Labonne. LLM course, 2023. URL https://github.com/mlabonne/ llm-course. 113, 119 Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models to follow instructions with human feedback. Advances in Neu- ral Information Processing Systems, 35:27730–27744, 2022. 113, 119 Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 112 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 118 Part I Reasoning 121 123 This part is about the brains of our agents, and how they use them to learn. The previous part showed us how a basic next-token LLM is trained. In agentic models typ- ically more advanced reasoning and multi-modal models are used (which we continue to call LLM). In this part we will discuss these more advanced models, why they are needed, and how they are trained and used. The first chapter of this part introduces reasoning models. Originally LLMs could not solve simple reasoning problems, and the chain of thought concept caused a major leap forward in this field. We will discuss math word problems, chain of thought prompting, and in-context learning. We will see models that use self-reflection to try to improve their own behavior. Self-reflection loops led to the development of agentic harnesses for LLMs. In the following chapter we go beyond language: we will see that the transformer architecture also works for vision and action domains. We will discuss various kinds of World Models, models that allow agents to act more intelligently in their interaction with the world. In the last chapter of this part, we introduce the concept of metalearning, a paradigm that is present in many of the types of learning that we see in agentic LLMs. We will discuss what metalearning is, and why it is a challenging form of machine learning. 124 Chapter 6 Multi-step Reasoning in LLMs What You Will Learn In this chapter you will learn how language models can solve more complex prob- lems through multi-step reasoning. Specifically, you will be able to: • Describe the chain of thought concept: how we can use in-context learning to guide the LLM to take intermediate steps in answering a query, and thus provide better answers to more complex queries. • Describe how reasoning methods have increased the power of LLMs greatly, and why this power is essential for agentic LLMs to act realistically in the real world. • Write your first agentic harness: a small algorithm that calls the LLM with different prompts. The agent harness manages the prompts and the replies from the LLM. • Explain self-reflection: LLMs can be called in different roles by the harness, for example, to evaluate an answer, or to generate subproblems. The LLM is reflecting to improve its own answers. • Understand the role that is played by memory: how the harness keeps track of the state of the agent, and how this state is used by the harness to manages the prompts. • Use the concept of reinforcement learning with verifiable rewards: where inference-time reasoning results are used for finetuning. 6.1 In-Context Reasoning As we noted Chapter 3, in large models, beyond hundreds of billions of parameters, a new kind of learning has emerged in LLMs. It is called in-context learning (ICL) or prompt-learning [Brown et al., 2020]. It is an important learning method, which we will discuss in more detail in this section. Then, we will discuss common benchmarks, and the chain of thought method for multi-step reasoning. 125 126CHAPTER 6. MULTI-STEP REASONING IN LLMS 6.1.1 In-Context Learning In-context learning [Dong et al., 2022] occurs not at training time, but at inference time, from examples that are provided in the LLM prompt, when the model is used to answer questions. These examples cause the decoder model to access relevant knowledge stored in the LLM. Because of this rich knowledge, in-context learning finds accurate results with few examples, which is why it is a so-called few-shot learning method. The few-shot learning is made possible by the large size of the model, containing rich and varied knowledge on many topics. Learning, not Training A main difference between in-context learning and finetun- ing or pretraining is that in-context learning does not change the parameters of the transformer. Since no model-parameters are changed at inference time, it is not a pa- rameter training method; however, since it does learn an answer from provided ex- amples, it is a learning method. A consequence is that ICL has the disadvantage of forgetfulness: since model parameters remain unchanged, all learned information is lost between LLM calls. To share information between LLM calls, previous results must be added to the prompt of the next call, in the context window, as extra learning examples. Context Window How does in-context learning work? The part of an LLM in which the prompt is written is called its context window. The decoder model learns inside this context window, from the examples that are provided in the prompt. A point to note is that all examples that the answer is learned from must be provided together in the context window—they should fit in the prompt. Typically, the size of the context window is limited to a few thousand tokens, and only few examples can be provided: all ICL is able to perform is few-shot learning. This becomes problematic when the amount of information that we wish to learn from grows, since the context window is typically limited in size. We might say that in-context learning suffers from long-term memory loss. Example In in-context learning, a prompt, consisting of a few pieces of demonstra- tion context, is concatenated with a query question, which is given to the context win- dow of the language model [Liu et al., 2023]. The model will then generate the answer by performing a forward pass through the model. For example, when the task is emo- tion recognition in a social media post, I missed the bus today, can be followed by I felt so [ ], and the model could answer with bad. Alternatively, for translation, we could follow I missed the bus today, by French: [ ] to request a translation. The prompt contains background information that is recognized by the model, selecting the desired model context. Few-shot Learning and the Holy Grail of Machine Learning In-context learning works when language models contain enough knowledge, allowing them to generalize on the (few) examples provided in the prompt. The user provides a few relevant exam- ples in the prompt, in order to prime the LLM to select the right background knowledge 6.1. IN-CONTEXT REASONING127 for answering the prompted instruction. Contexts that contain a few examples are said to perform few-shot learning. Contexts that contain only instructions, with zero exam- ples, are said to perform zero-shot learning. ICL is a form of few-shot learning that caused much excitement among researchers. In traditional machine learning, a model 푓 is trained on dataset. To achieve high ac- curacy on complex problems, the datasets must contain many examples. Traditional machine learning is a form of many-shot learning, requiring many (expensive) training epochs to train the parameters of the model. In-context learning, on the other hand, is cheap, requiring just a few examples followed by a single forward pass through the decoder model (since the expensive training has been done in the pretraining phase). Few-shot learning is something of the holy grail of machine learning: a model that generalizes, cheaply. Practical Considerations and Prompt Engineering In-context learning further has the practical advantage of convenience: we can perform ICL on an LLM that runs commercially in the cloud, from our smartphone, without going through the difficulty of finding a computer to train a large model on. Although prompts provide a user-friendly interface to LLMs, the success of in- context learning tends to be quite sensitive to the way a prompt is formulated or when different examples are provided. Prompt-learning is brittle, and a new field called prompt engineering has emerged to help human users to learn how to make LLMs do what we want them to do [Radford et al., 2019, Wei et al., 2022a, Giray, 2023, Sahoo et al., 2024]. In most LLMs the context window is provided with hidden prompts, the system prompts, to specify a particular role for the model, to prevent the use of swear words, and, in chatbots, to have a short history of previous search results, to allow for more naturally flowing conversations. In-context learning has spawned much further research. One of the areas is multi- step reasoning, which is the topic of this chapter. Reasoning used to be an area where LLMs performed poorly. For example, LLMs failed at simple math exercises. To bet- ter understand this application domain we will introduce a few reasoning benchmarks. Then we will discuss an in-context learning method that greatly improved the perfor- mance of reasoning by LLMs. 6.1.2 Multi-step Reasoning Benchmarks Progress in artificial intelligence is measured by benchmarks. The research on multi- step reasoning with LLMs started with easy (grade school) math word problems. Specif- ically for the study of such easy reasoning the GSM8K benchmark was composed [Cobbe et al., 2021]. GSM8K was created with the aim of providing high quality prob- lems, of high diversity and moderate difficulty, in natural language. It consists of 8500 human-written math problems and solutions. Language models struggled to achieve good performance on this dataset before chain of thought was introduced (see the next section). Here is an example of a typical math word problem: Problem: Romeo bakes 4 trays with two dozen batches of cookies in a 128CHAPTER 6. MULTI-STEP REASONING IN LLMS week. If these cookies are shared amongst 16 people equally, how many cookies does each person consume? Answer: 4× 2× 12/16 = 6. The GSM8K benchmark, although it may look easy to us, turned out to be difficult for LLMs. When it was introduced, the state of the art LLM GPT-3-175B achieved a rather low accuracy of around 15% [Wei et al., 2022b]. Other benchmarks of similar math word problems are the SVAMP varying structures benchmark [Patel et al., 2021], the ASDiv dataset of diverse math problems [Miao et al., 2021], the AQuA dataset of algebraic word problems [Ling et al., 2017], and the MAWPS benchmark [Koncel- Kedziorski et al., 2016]. Later, benchmarks for other tasks were introduced, to assess LLM reasoning abil- ities in other fields of reasoning. For example, benchmarks have been developed on advanced mathematical questions (AIME [Codeforces, 2024]), computer code compre- hension (Spider [Yu et al., 2018], transcoding (translating a program in one computer language to another) [Roziere et al., 2020]), robotic movement (Alfworld [Shridhar et al., 2020], kitchen tasks [Ahn et al., 2022]), puzzle solving (Game24 [Yao et al., 2024]), creative writing [Yao et al., 2024]), board games (Checkmate problems [Yang et al., 2024], MineCraft [Fan et al., 2022]), and webpage navigation (WebShop [Yao et al., 2022a]). Further relevant benchmark studies also focus on games and planning problems [Ruoss et al., 2024, Paglieri et al., 2025, Wen et al., 2025a]. The goal of benchmarks is to be able to clearly compare the performance of differ- ent approaches. However, it can be difficult to directly compare different benchmarking results, since there may be differences in the way the experiments were conducted. See Kamoi et al. [2024], Shojaee et al. [2025] for a critical discussion of challenges in benchmarking LLM reasoning. 6.1.3 Chain of Thought Prompting Around 2022, regular LLMs of the time performed poorly on simple math word prob- lems. Interestingly, the standard Chapter 4 approach to improve performance of LLMs by scaling up the size of the LLM, did not solve the problem [Rae et al., 2021]. The problem appeared to be somehow fundamental in nature, and researchers went looking for a different approach. The LLM architecture is a transformer-based decoder. Given a certain input, it is designed to produce a single token. LLMs are next-token predictors. When we prompt such a decoder to produce an answer, it does as it is instructed to: it provides an answer quickly—any answer. But that is not how humans work towards an answer in a mathematical reasoning problem, we divide the question into a sequence of steps to solve (4× 2× 12/16 = 6). Thus, for the LLM, what we should do instead is to prompt it to follow a well-chosen sequence of intermediate steps, answer those, and thus work towards the final answer, just as a student is taught to break down a complex problem into smaller steps, and to also provide the reasoning steps when answering a problem in school—Don’t just tell me the answer, but tell me how you arrived at the answer. Children are taught in school to break down a big problem into a sequence of small steps, when solving a mathematical problem. Similarly, we should guide the 6.1. IN-CONTEXT REASONING129 Input Output A: The answer is 27. Q: Roger has 5 tennis balls. He buys two cans of 3 balls. How many tennis balls does he have now? A: The answer is 11. Q: Lisa has 23 apples. If she ate 20 and bought 6 more, How many apples does she have? Input Output A: Lisa started with 23 apples. She ate 20, so she has 23-20 =3 apples left. She bought 6, so 3 + 6 = 9. The answer is 9. Q: Roger has 5 tennis balls. He buys two cans of 3 balls. How many tennis balls does he have now? A: Rogers started with 5 balls. 2 cans of 3 balls each is 6 balls. 5 + 6 = 11. The answer is 11. Q: Lisa has 23 apples. If she ate 20 and bought 6 more, How many apples does she have? Input Q: Lisa has 23 apples. If she ate 20 and bought 6 more, How many apples does she have? Let’s think step by step. Output A: Lisa started with 23 apples. She ate 20, so she has 23-20 =3 apples left. She bought 6, so 3 + 6 = 9. Input Output A: Lisa started with 23 apples. apples = 23 She ate 20 and the bought 6 eaten = 20; bought = 6 anwer = apples - eaten + bought Q: Roger has 5 tennis balls. He buys two cans of 3 balls. How many tennis balls does he have now? A: Rogers started with 5 balls. tennis_balls = 5 2 cans of 3 balls each is bought_balls = 2 * 3 tennis balls. The answer answer = tennis_balls + bought_balls Q: Lisa has 23 apples. If she ate 20 and bought 6 more, How many apples does she have? Figure 6.1: Different Chain of Though (CoT) prompting techniques [Wei et al., 2022b, Kojima et al., 2022, Gao et al., 2023, Plaat et al., 2025]. At the top the prompts, at the bottom the answers. language model to explicitly produce such intermediate steps, and then to combine the results. The LLM should then provide an answer that includes the intermediate steps, arriving at a correct result. Thus, our few-shot prompt should contain examples of intermediate reasoning steps, instead of only examples of the final answer. This question was studied by Wei et al. [2022b], amongst others, who introduced the chain of thought approach. Fig- ure 6.1 illustrates how the approach works, in addition to related approaches, that it inspired. The way to instruct an LLM to generate intermediate steps is to manually write a prompt consisting of specific reasoning steps to answer the task (for example, calculate how many balls are in the container, or in how many pieces the pie was cut). Indeed, when Wei et al. [2022b] tried this approach, and the LLM was prompted to rephrase information from the question as intermediate reasoning steps in its answer, the LLM performed much better than when it was prompted to produce a single answer. The ex- ample from the chain of thought paper is shown in Figure 6.1. When shown the longer example prompt, the LLM follows the longer example when answering the question (Few-Shot CoT). On GSM8K, the chain of thought approach showed a large increase in accuracy, from 16% to 47%. The idea that, when prompted, an LLM can be made to follow step-by-step instruc- tions, has caused much excitement and has opened up further research on multi-step reasoning with LLMs. Chain of thought (CoT) has become a standard prompting ap- proach in LLMs since. In the original paper the prompts were hand-written by the researchers for the in- dividual types of problems. Kojima et al. [2022] go a step further. They show that the addition of a single task-independent sentence (Let’s think step by step) also sig- nificantly improves performance for many problems. Since this text does not contain problem-related elements, it is as a form of zero-shot learning. Figure 6.1 compares 130CHAPTER 6. MULTI-STEP REASONING IN LLMS the approaches (second and third column). Experiments further show that with this addition significant performance gains are also achieved on other reasoning bench- marks, including arithmetic, symbolic, and logical reasoning (achieving 70% accuracy on GSM8K when self consistency (see next section) is also included, using the newer PaLM LLM [Wang et al., 2022]). The chain of thought idea itself is inspired by Scratchpad [Nye et al., 2021], by other earlier work where natural language steps are generated for arithmetic reasoning [Ling et al., 2017, Cobbe et al., 2021], and by the use of formal languages for rea- soning [Roy and Roth, 2016, Chiang and Chen, 2018, Amini et al., 2019, Chen et al., 2019]. 6.1.3.1 Self Evaluation LLMs are not perfect, and errors in results occur. When multiple reasoning steps are generated, errors accumulate. To prevent the errors from getting out of hand we should pay extra attention to the evaluation of results. We will now review approaches to re- duce error accumulation, either by examining the model, or by evaluating intermediate reasoning steps. When LLMs are prompted to perform CoT reasoning, they perform a sequence of steps. Performing multiple steps makes them sensitive to mistakes and vulnerable to error accumulation [Weng et al., 2022, Xiao et al., 2023]. Several methods have been developed to prevent error accumulation. One approach is to create a new model to separately evaluate the results. Weng et al. [2022] propose an automated approach using evaluation by the same LLM, called self verification. The LLM is prompted to use the conclusion of the chain of thought reasoning chain as a condition for solving the original problem and then compare the answer, going back to the original question. The LLM is given variations of its own conclusion and is instructed to choose the one with the highest similarity to the original question. A popular related approach is self consistency [Wang et al., 2022]. Self consistency is an ensemble approach (a well-known machine learning technique to make a strong learner by running multiple weaker learners [Sagi and Rokach, 2018, Breiman, 2001]). Conventionally, the answer of an LLM query is extracted from the decoder network by greedily following the path with the highest logit values. In self consistency, greedy single-path decoding is replaced by majority voting over a wider sample of paths. By sampling diverse reasoning paths, evaluating them, and selecting the most consistent answer among them, self consistency effectively asks the LLM to perform the same query multiple times, and takes the majority-vote of the internal decoding paths of the transformer model. Self consistency works because complex reasoning problems typically have different reasoning paths that lead to the correct answer. Figure 6.2 provides an example of the approach. Self consistency has been evaluated on arithmetic reasoning, on commonsense rea- soning and on symbolic reasoning, with a variety of LLMs, including GPT-3. Self consistency typically improves the performance of chain of thought by a further 10-20 percentage points. Self consistency is often used in combination with chain of thought in modern LLMs. 6.1. IN-CONTEXT REASONING131 Input Output This means she uses 3 + 4 = 7 eggs every day. She sells the remainder for $2 per egg, so in total she sells 7 * $2 = $14 per day. The answer is $14. Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot? A: There are 3 cars in the parking lot already. 2 more arrive. Now there are3 + 2 = 5 cars. The answer is 5. ... Q: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muns for her friends every day with four. She sells the remainder for $2 per egg. How much does she make every day? A: Reasoning 1 She has 16 - 3 - 4 = 9 eggs left. So she makes $2 * 9 = $18 per day. The answer is $18. Output The answer is $18. Reasoning 2 This means she sells the remainder for $2 * (16 - 4 - 3) = $26 per day. The answer is $26. Reasoning 3 She eats 3 so she has 16 - 3 = 13 left. Then she bakes, so 13 - 4 = 9 eggs. 9 eggs * $2 = $18. The answer is $18. Figure 6.2: Self Consistency (Adapted from [Wang et al., 2022]), showing majority voting 132CHAPTER 6. MULTI-STEP REASONING IN LLMS 6.1.3.2 Code Generation for Validation Another approach to improve the accuracy of a sequence of reasoning steps is to switch from a natural to a formal language. The advantage of a formal language is that it is less ambiguous than a natural language. Examples are computer languages, such as Python, or mathematical logic. LLMs have been quite successful in generating computer code from natural lan- guage prompts. Chen et al. [2021] introduced Codex, a GPT model that was trained on publicly available code in the repository GitHub. Codex is able to generate correct programs from descriptions in natural language, such as commentary strings. Sub- sequently, the use of a formal language for reasoning has become a popular approach. Many approaches generate the reasoning steps in Python, so that the code can be evalu- ated by a compiler, debugger, or interpreter. Human programmers, when writing code, typically follow a cycle of writing some code, compiling and executing it to look for errors, and then using the feedback to improve the code. This approach is followed, for example, in self debugging [Chen et al., 2023]. It follows the same steps of code generation, code execution, and code explanation. Self debugging is able to identify mistakes in its own code by investigating the execution results, and can also provide an explanation of the generated code, in natural language. Another approach, program aided language [Gao et al., 2023], uses the LLM to generate Python and then use an interpreter to evaluate the result. In this approach the problem is translated into small pieces of Python code, that can then be executed by an interpreter to solve the problem. Figure 6.1 (fourth column) illustrates the pro- gram aided language approach. When the evaluation of the reasoning steps is offloaded to the Python interpreter, decomposing the natural language problem into executable code-steps remains the only task for the LLM. Gao et al. [2023] provide extensive ex- perimental evidence showing the synergy between the neural LLM and the symbolic interpreter. Experiments were performed on 13 mathematical, symbolic, and algorith- mic reasoning tasks, achieving more accurate results than much larger models of the time (for example, program aided language reported 72% on GSM8K and 93% on Penguins). 6.1.3.3 Evolving code generation prompts The work on verification with formal languages has lead to several works that generate working code tuned for solving specific problems automatically, without human feed- back. Romera-Paredes et al. [2024] introduced FunSearch, an approach that integrates formal methods and LLMs to enhance mathematical reasoning and code generation. FunSearch uses a genetic approach with multiple populations of candidate solutions (programs), which are evaluated using a function depending on the problem specifica- tion. After evaluating the generated programs from the starting prompt, a new prompt is created, in an iterative fashion, combining a selection of sampled programs sorted according to their evaluation score, and the LLM is requested to generate a new pro- gram. Another work leverages evolutionary computation methods to generate and op- timize evolutionary algorithms [van Stein and B ̈ ack, 2024]. This approach, LLaMEA (Large Language Model Evolutionary Algorithm), utilizes LLMs to design and opti- 6.2. SELF-REFLECTION AND STATE133 mize evolutionary algorithms. The approach uses LLMs to generate initial algorithmic structures, which are then refined through mutation and selection. Accuracy improve- ments on the Black-Box Optimization Benchmark (BBOB) [Hansen and Ros, 2010] are reported of 11 percentage points. 6.2 Self-Reflection and State In-context learning has emerged as a powerful new few-shot learning method. When our few-shot examples specify reasoning steps, we can improve the capabilities of LLMs even more. In this section we will look at some of the limits of in-context learning, and we will look at ways to extend the limits. We will first discuss the question of state and memory. We will look at how state can be passed between separate invocations of LLMs, at long context windows, and at the question if LLMs are good at solving problems for which they have to manipulate state. 6.2.1 LLMs are Memory-less Functions In-context few-shot learning exploits the capabilities of LLMs to retrieve knowledge, conditioned by the prompt. The instructions and examples are provided, and the transformer architecture finds the answer (next token) in its vast model of (퐾,푉)- parameters. With chain of thought prompts, the LLM is conditioned on few-shot rea- soning steps that are specified in the prompt. In-context learning uses the LLM in a functional way, where all input to the LLM is provided in the prompt. The model 푓 finds the answer 푎 for the input prompt 푝: 푓(푝) → 푎. In principle, LLMs do not retain information between LLM calls (model parameters remain unchanged, and there are no global variables with history). If we wish to refine our request, and we wish to carry state information from the previous call to the next, then all such state must be specified by the user in the new prompt 푝 ′ . The model itself has forgotten all about the previous answer. For a chatbot, repeating the entire conversational history in each new question would be a cumbersome and unnatural way of conversing. In human conversation our conversation partners do remember what was last said, without us having to ex- plicitly repeat their last sentence (although, when interviewing politicians, journalists sometimes use this method to try to keep them on message). Commercial chatbots overcome this problem in part using system prompts, by invisibly providing the output of the previous LLM call to the prompt for a new LLM query. However, the principle remains that LLMs are good at finding associative next-token answers for a prompt, but not at all at remembering answers and manipulating the state of variables as we know them from general computer programming languages. As no parameters are changed, LLMs do not remember in-context state between calls. 134CHAPTER 6. MULTI-STEP REASONING IN LLMS Figure 6.3: Tic Tac Toe is a Simple Board Game, in which State is important 6.2.2 Retrieval Augmentation and Long Context Windows The LLM-memory problem goes deeper. All information for the query must be spec- ified in a context window. The size of this context window is limited, which funda- mentally limits the amount of information that can be communicated to the LLM for a query. This limitation is a problem when a large text (such as a book) is to be summa- rized. Furthermore, in Section 20.2.1 we will discuss in-context reinforcement learning methods, that work by storing full episode histories in the context window. In short, methods are needed to pass more information to the LLM. One of the earliest methods to extend the amount of information at inference time is retrieval augmentation for generative AI (RAG). RAG lets the LLM place a call to external data, such as a corporate database, or to Wikipedia [Lewis et al., 2020]. RAG’s use is also in reducing factual hallucinations and providing access to time-sensitive information, such as financial or logistics data. RAG will be discussed in Chapter 12. For passing longer texts to the model, LLMs have been developed with longer con- text windows. With the standard transformer architecture, the compute requirements scale quadratically with the size of the context window, and linear architectures are being developed [Ren et al., 2024]. Another solution is to compress the information in the context window, for example using semantic compression [Fei et al., 2024], by chunking the information [Wang et al., 2023, Zhou et al., 2025], or by recursive meth- ods [Zhang et al., 2025]. Devising novel memory architectures for LLMs is an active area of research [Yan et al., 2025]. 6.2.3 LLMs and Stateful Problems LLMs perform poorly at solving problems in which state must be managed [Wong et al., 2025]. This occurs for example in planning problems in which the current state has to be remembered accurately, even in simple board games such as the one shown in Figure 6.3. LLMs tend to mix up the states, and to provide plausible sounding answers that may be correct for games that it has seen in its training corpus, but that are wrong for the actual board state at hand [Paglieri et al., 2025, Ruoss et al., 2024]. Many approaches have been tried to improve reasoning with state, often inspired by reinforcement learning and planning algorithms [Wong et al., 2026]. To see how LLMs can be used to solve these stateful problems, we discuss how the memory-less nature 6.2. SELF-REFLECTION AND STATE135 of LLMs makes them suffer from a problem known as the knowing-doing gap. Other, advanced, methods have been developed to allow LLMs to deal better with stateful problems. Chapter 20 will discuss these advanced approaches. Knowing-Doing Gap In addition to the lack of state between LLM calls, there is another memory-related problem, in the internal execution by the LLM of certain tasks. This problem has been named the knowing-doing gap. This gap describes the cu- rious phenomenon where an LLM is able to generate a correct textual description to solve a problem, even the correct algorithm, but is not able to execute this algorithm correctly [Paglieri et al., 2025]. When you ask an LLM how to solve a certain prob- lem, it can promptly provide the right algorithm in executable Python code. When one subsequently asks it to actually solve this problem, the LLM does not correctly follow the algorithm that it just generated. More often than not, errors start appear- ing between state changes of the variables in the algorithm that it generated [Paglieri et al., 2025, Ruoss et al., 2024]. Note that faithfully executing algorithmic code step by step, tracking values in registers, is not what the attention-based transformers in LLMs were designed for at all—they were designed to simulate natural language processing through a form of associative memory [La Malfa et al., 2024]. 1 In addition to the knowing-doing gap, there is another state-management problem. Prompting an LLM to play the first few moves of a board game against itself may show a game in which pieces on the board magically (dis)appear between moves. The model does not recognize that the moves that it plays that may have been good in certain situations, do not fit the current board position, and even decorates these wrong moves with a plausibly sounding reason in a chatty conversational style. LLMs have difficulty reliably managing the state transition of a variable that goes from one value to another value. Individual values tend to be forgotten or mixed up. Additional text can be inserted into the prompt to instruct the model to carefully track values of variables before making a next step [La Malfa et al., 2024], but this is a cumbersome and error- prone solution. Both the knowing-doing gap and the disappearing-pieces problem can be circum- vented by calling an external interpreter to execute an LLM-generated algorithm. This strategy is sometimes taken by commercial chatbots. It is a smart, practical, solution, since, in addition to being able to correctly execute Python code, the Python interpreter will do so more efficiently than an LLM. Chapter 20 will discuss other approaches that address state and memory in the transformer architecture. 6.2.4 Agentic Harness for Prompt Improvement In addition to resorting to externally executing generated code, other methods have been developed to deal with memory and state. In this section, we will go agentic, and introduce small external algorithms to explicitly manage state and prompts. Such 1 Note further that the human mind sometimes has the same problem: we can easily describe an algorithm for long division, but when asked to compute 12345/42, we may make mistakes—computers are good at this, but not humans, and also not LLMs. 136CHAPTER 6. MULTI-STEP REASONING IN LLMS an algorithm is called a harness. The harness may also call the same LLM in different roles, with different prompts for each role, for example, for generation or for evaluation of states. We start with prompt management based on search algorithms. When the LLM is called in different roles, for example to critique (evaluate) its own performance (that it just generated), the LLM can be said to perform self-reflection. This somewhat anthropomorphic term is often used in the literature for these approaches. Exactly what (not who) is doing the self-reflecting, the LLM, or the agentic harness, is a matter for debate. 6.2.4.1 Search-based Prompt Optimization We will now discuss search-based inference-time decision methods. These are in- context learning methods that go beyond chain of thought. Chain of thought explores a single line of reasoning steps, where each step has a single successor step. Search- based methods explore multiple options at each step, creating a tree of possibilities. When we wish to examine a tree of possible reasoning steps, then we need a method to keep track of the state of which step we have seen, what the best answer so far was, and which steps should still be further examined. As we have seen, LLMs are not terri- bly good at keeping track of state, whereas conventional computer algorithms, such as small pieces of Python code, are. Such an algorithm will decompose the problem into a search tree, where multiple alternative steps are generated that can be searched mul- tiple steps into the future (see Figure 6.6 on page 138). The algorithms call the LLM as a subroutine prompting it to perform requested tasks, such as action selection and reward evaluation. Then, backtracking can be performed by the algorithm, allowing alternative steps to be tried. This control algorithm is external to the LLM. It wraps around the LLM, keeping track of state, managing the prompts, shielding it from the outside world (and shielding the outside world from LLMs that mess up state). The control algorithm is called a harness. We start our discussion of control strategies with depth-first and breadth-first search [Cormen et al., 2022]. Tree of thoughts [Yao et al., 2024] uses breadth-first or depth- first search to dynamically follow different reasoning steps. 2 Both the generation (red) and the evaluation (green) part in tree of thoughts are performed by prompting the LLM, see Figure 6.4. Together, the two LLM prompts with the external control, al- low systematic exploration of the space of reasoning steps, performing look-ahead and backtracking. Tree of thoughts (ToT) has been used on planning-based puzzles, such as the Game of 24. In this game, 4 numbers are give, and the task is to reach the number 24 by using the arithmetic operators(+−∗/) in between the numbers. For example, for the numbers 4 9 10 13, a solution would be(10−4)∗(13−9) = 24. Tree of thoughts first generates a position with the propose prompt: Input 4 9 10 13; Possible next steps: to which the model answers by generating possible next step thoughts. These are then evaluated by the model with an evaluation prompt including chain of thought examples. Figure 6.5 shows a possible prompting sequence from the original paper. 2 Tree of thoughts is a popular approach. The Huggingface link is here: https://huggingface. co/blog/sadhaklal/tree-of-thoughts 6.2. SELF-REFLECTION AND STATE137 Figure 6.4: Tree of Thoughts. Left Breadth First Search, Right Depth First Search. The arrows indicate the LLM prompts (after [Yao et al., 2024]) Figure 6.5: Tree of Thoughts Example prompts for Game of 24 [Yao et al., 2024] 138CHAPTER 6. MULTI-STEP REASONING IN LLMS Figure 6.6: Reasoning structure of Chain of Thought (b), Self Consistency (c), Tree of Thoughts (d) The authors compare their approach to traditional linear chain of thought with self consistency on the Game of 24, Creative writing, and Mini crossword, achieving an accuracy of 74% on a Game of 24 task. The other tasks are evaluated qualitatively. Figure 6.6 illustrates the different reasoning structures. 3 Tree of thoughts is an elegant approach, adding two of the simplest possible search structures on top of an LLM. The sequence of prompts is easy to follow, and finding and correcting mistakes is also easy. The two search methods of tree of thoughts follow a fixed expansion policy. Next, we will discuss more advanced learning-based approaches, that may perform better, at the cost of higher algorithmic complexity. 6.2.4.2 Self-Reflection Many approaches have been devised that use the LLM in more than one role. Such approaches are generally called self-reflective approaches. The LLM is used not just to evaluate a prompt, but also to generate new reasoning steps, and often to refine them. The field of self-reflection is young and new approaches are being developed regu- larly. We will present four such approaches: self refine, ReAct, reflexion, and buffer of 3 A similarly named approach is graph of thoughts [Besta et al., 2024]. Graph of thoughts allows more general reasoning graphs, providing a formal framework, where the different elements can then be specified manually. 6.2. SELF-REFLECTION AND STATE139 Figure 6.7: Self Refine Architecture [Madaan et al., 2023] thoughts. These approaches use the LLMs in different ways, using schemes inspired by reinforcement learning to let the LLMs improve themselves. Self Refine The first approach is self refine [Madaan et al., 2023]. This approach aims to improve answers from feedback. The LLM generates an initial output and provides feedback for its answer, using the LLM to refine itself, iteratively. The LLM is thus used to improve its own answers, see Figure 6.7. Self refine prompts the LLM in three ways: (1) for initial generation, (2) for feedback, and (3) for refinement, following a greedy reasoning chain. Self refine reportedly improves significantly over regular chain of thought, on benchmarks such as dialogue response generation, code optimization, and constrained generation. This approach does follow a linear, greedy, reasoning chain, where a single successor is tried in a state (it does not explore a tree, as in ToT). ReAct Generalizing the greedy approach, we can use LLMs to examine multiple op- tions per state. A notable approach that combines reinforcement learning and LLMs is ReAct [Yao et al., 2022b]. 4 ReAct uses chain of thought prompt-learning as part of a re- inforcement learning framework that also uses external knowledge sources (Wikipedia) and finetuning; for error reduction, grounding, and for reducing hallucination. The framework allows hand-written prompts. On two interactive decision making bench- marks (Alfworld and WebShop), ReAct outperforms imitation and reinforcement learn- ing methods by an absolute success rate of 34% and 10% respectively. Reflexion The ReAct work has been developed further. Reflexion [Shinn et al., 2024] creates AI agents that learn by reflecting on failures and enhance their results, inspired by the way that humans do this. 5 Reflexion uses three prompts for the LLM: actor, eval- uator, and reflector. The reflexion harness incorporates explicit memory between LLM calls. For trajectory generation it has a short-term memory, and for storing experience information it has a long term memory. This approach addresses the state management problem of regular LLMs using memory in the external algorithm. Reflexion works as follows: (1) an actor model generates text and actions, (2) an evaluator model scores 4 https://github.com/ysymyth/ReAct 5 https://github.com/noahshinn/reflexion 140CHAPTER 6. MULTI-STEP REASONING IN LLMS Figure 6.8: Architecture of Reflexion [Shinn et al., 2024], showing a close resemblance to the agent/environment structure of reinforcement learning the outputs produced by the actor, and (3) a self-reflection model generates verbal rein- forcement cues to assist the actor to self-improve (see Figure 6.8). For the actor, chain of thought and ReAct can be used. Reflexion has been evaluated on decision-making, reasoning, and coding tasks. Improvements of 10-20 percentage points are reported [Shinn et al., 2024]. Buffer of Thoughts Another approach is buffer of thoughts [Yang et al., 2024], which goes a step towards metareasoning [Gao et al., 2024]. It introduces a meta- buffer that stores high-level thought-templates. Thought-templates are derived from a variety of tasks. Buffer of thoughts uses a problem distiller that stores high-level thoughts distilled from different problems in a meta-buffer, which is actively managed for capacity. Figure 6.9 compares the buffer of thoughts approach to other approaches such as chain of thought and tree of thoughts. Buffer of thoughts outperforms other methods in puzzles such as Game of 24 (by 11%) and checkmating (by 51%). Thought templates are related to metacognition (thinking about thinking) [Didolkar et al., 2024]. Critique and Prompt Sensitivity The concept of self-reflection is not without pit- falls and criticism. To start, the basic idea of asking a model to refine its results begs the question why the model did not give the better answer in the first place. Next, studies by Huang et al. [2023], Shojaee et al. [2025] find methodological issues with some of the studies, such as the use of data labels that reveal the answer, second refinement prompts that unintentionally reveal more information than the initial prompts, and the use of the 6.2. SELF-REFLECTION AND STATE141 Figure 6.9: Buffer of Thoughts [Yang et al., 2024] same LLM for agent and environment. In one reproducibility study performance did not exceed that of chain of thought with self consistency [Huang et al., 2023]. LLMs are sensitive to how a question is formulated in the prompt. Self-reflection is still a young and active field of research, with an appealing premise, but many open ques- tions. The approaches that we have presented potentially make many LLM calls, which can become expensive and slow. Finding the optimum approach for your problem may require quite some experimentation. In Chapter 10 we will discuss a practical case study where we will get hands-on experience with self-reflection methods. Please keep the preceding critique in mind. 6.2.4.3 Reinforcement Learning with Verifiable Rewards The focus of this chapter is on in-context learning. In-context learning occurs at in- ference time, and no parameters are trained or changed. Finetuning occurs at training time, and does change the parameters of the model. In this section we will, however, briefly talk about a mix of finetuning and in-context learning, we will be mixing train- ing and inference. Reinforcement learning with verifiable rewards refers to an algorithmic approach for training language models on tasks with verifiable outcomes such as mathematical problem-solving and instruction following. Especially for reasoning problems, such as math word problems and coding, inference-time chain of thought results have been found to significantly improve models when they are used for finetuning. RLVR is an improvement over reinforcement learning from human feedback (RLHF), dispensing with the need for human feedback. In many domains, such as mathematics, a simple function exists to check if an answer is correct, without the need for human feedback. RLVR leverages the existing RLHF objective but replaces the reward model 142CHAPTER 6. MULTI-STEP REASONING IN LLMS with a verification function. RLVR samples completions from a policy model given a set of prompts, and verifies their correctness using a deterministic function. If correct, the reward is positive, otherwise it is 0. This reward is then used to train against us- ing PPO [Schulman et al., 2017] (a standard reinforcement learning algorithm), as in RLHF. An influential approach of this type was proposed in the development of DeepSeek- R1 Guo et al. [2025]. This method distinguishes itself from external reasoning ap- proaches by emphasizing the model’s self-generated reasoning steps. It learns the steps through reinforcement learning, integrating data generation and training in a single loop. Lowering the supervised data requirements to train LLMs, DeepSeek’s method- ology leverages reinforcement learning to enable models to evolve reasoning skills autonomously. By starting with a base model and applying reinforcement learning, the system identifies and reinforces effective reasoning patterns. DeepSeek uses group- relative policy optimization (GRPO), which calculates advantages with group-based scoring [Shao et al., 2024]. One of the key features of DeepSeek’s approach is the emergence of sophisticated reasoning behaviors during finetuning, such as reflection and exploration of alternative problem-solving methods [Mercer et al., 2025]. These behaviors arise spontaneously as a result of the model’s interaction with the reinforcement learning environment, rather than being pre-programmed, and without external self-reflection loops. For example, DeepSeek-R1-Zero learns to allocate more thinking time to problems by reevaluating its initial approach. This autonomous approach of learning to reason may lead to more stable and adaptive reasoning LLMs. One of the successes behind GRPO is its focus on applications for which verifiable rewards exist, allowing reinforcement learning methods to be used. The focus on ver- ifiable rewards has been quite successful, and reinforcement learning with verifiable rewards (RLVR) methods are often used in finetuning LLMs [Lambert et al., 2024]. RLVR methods such as GRPO are typically used dynamically, where an inference time chain of thought query is used to answer a query from the training set to finetune the model. Thus, test time inference is used for finetuning. The impressive results of DeepSeek [Guo et al., 2025] have made GRPO [Shao et al., 2024] and RLVR [Lam- bert et al., 2024, Guo et al., 2025, Wen et al., 2025b, Yue et al., 2025] highly popular finetuning methods. 6.3 Concluding Remarks In the preceding pages we have seen how the reasoning capabilities of language models can be improved. The best-known approach is chain of thought, which is a prompt- engineering method that shows an LLM to provide a multi-step reasoning when it is answering a question, just like we are taught in school when we are shown how to solve math exercises. More elaborate prompt improvement algorithms have been introduced, from depth-first search, via self-reflection, to inference-time reinforcement learning methods. Advanced training methods such as RLVR combine inference-results with finetuning, achieving strong training results. 6.3. CONCLUDING REMARKS143 6.3.1 Brittleness We have discussed many different algorithmic approaches to create an agentic harness. There are multiple reasons why there are so many. First, the field of agentic LLM is still young, many challenges exist, leading to many new solutions. Second, the methods are quite brittle. Small changes in prompts, a comma here or there, sometimes make the difference whether an approach works or not. Small changes in hyperparameters, such as the temperature of your model, or even using a different seed of a random number generator, may have large effects. Changes to the memory management method of the agentic harness, or using a different LLM, may all have a large effect on performance. This may feel like a unsatisfactory state of affairs. Indeed, the field is young, and it will take some years before methods have stabilized. In your experiments, you should keep in mind to always do multiple runs, with different seeds, different models, and dif- ferent hyperparameter settings. This will help you better understand how your solution works in different circumstances. 6.3.2 Thinking, Fast and Slow To conclude this chapter we will discuss reasoning in LLMs in a broader perspective. In the history of AI, two approaches, or schools of thought, can be seen: symbolic AI and connectionist AI. Symbolic AI roughly corresponds to the idea that intelligence is best modeled as the manipulation of symbols via the rules of logic. Connection- ist AI studies whether intelligence emerges spontaneously out of interactions between connected elements (be they neurons, cells, ants, humans, or robots). Symbolic AI draws on mathematics, connectionist AI on biology. As we already saw in Section 2.3, both schools have developed useful methods, and both schools have seen important successes (such as search algorithms and reasoning, or deep learning and LLMs). In 2011 Daniel Kahneman published the book Thinking, Fast and Slow in which the terms System 1 and System 2 were used to distinguish human thought into intuitive, fast, thinking, and deliberative, slow, thinking [Kahneman, 2011]. These terms have become popular in artificial intelligence, where fast thinking is linked to associative lookup and slow thinking to multi-step reasoning. At inference time, zero-shot LLMs think fast (System 1 thinking). Inference-time step-by-step methods (either through longer prompts or with an agentic harness) perform deliberative slow thinking (System 2 thinking). LLMs are based on the connectionist AI tradition (System 1). The use of tools at inference time enhances the LLM part with knowledge retrieval or processing tools from the symbolic AI tradition (System 2). We should note that whereas researchers sometimes humanize LLMs and their ca- pabilities, LLMs only perform next-token prediction. LLMs are not even close to be- ing human. By generating more tokens to form an answer (reasoning step-by-step), the token-path from the prompt to the final answer becomes longer. The reason that this leads more often to correct answers, might be because it takes smaller steps into the direction of the answer, making the correct answer more plausible with every step in between—not that it is similar to the way humans are taught to solve the problem. 144BIBLIOGRAPHY Reasoning is narrowing down probabilities such that the correct answer becomes more probable to generate, independent of interpretations related to human cognition [Guo et al., 2025]. 6.3.3 Conclusion In this chapter we have discussed how LLMs can solve language tasks by next-token prediction, and solve tasks that require multi-step reasoning. We have discussed ways to manage complex reasoning structures. Most methods are in-context methods, efficient and easy to use since the prompts that guide the reasoning are readable by the user. We have introduced the rule of an agentic harness for prompt improvement and memory management. We have also touched upon methods that use pretraining and finetuning, not just for the language part, but also for integrating multi-step reasoning. These methods thus train models for decision making, beyond pure next-token prediction in language models. There has been much activity in the creation of such richer models, adding visual reasoning, decision making, and actions to become an integral part of the LLMs. In the next chapter we will look deeper into these pretrained and finetuned multi-modal models. Bibliography Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. 128 Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Han- naneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019. 130 Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16): 17682–17690, 2024. 138 Leo Breiman. Random forests. Machine learning, 45:5–32, 2001. 130 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 125 Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brock- man, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 132 BIBLIOGRAPHY145 Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V Le. Neural symbolic reader: Scalable integration of distributed and symbolic rep- resentations for reading comprehension. In International Conference on Learning Representations, 2019. 130 Xinyun Chen, Maxwell Lin, Nathanael Sch ̈ arli, and Denny Zhou. Teaching large lan- guage models to self-debug. arXiv preprint arXiv:2304.05128, 2023. 132 Ting-Rui Chiang and Yun-Nung Chen. Semantically-aligned equation generation for solving and reasoning math word problems. arXiv preprint arXiv:1811.00720, 2018. 130 Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 127, 130 MAA Codeforces. American invitational mathematics examination-aime 2024, 2024, 2024. 128 Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Intro- duction to algorithms. MIT press, 2022. 136 Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Tim- othy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive capabilities of llms: An exploration in mathematical problem solving. arXiv preprint arXiv:2405.12205, 2024. 140 Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022. 126 Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, et al. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343–18362, 2022. 128 Weizhi Fei, Xueyan Niu, Pingyi Zhou, Lu Hou, Bo Bai, Lei Deng, and Wei Han. Extending context window of large language models via semantic compression. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5169– 5181, 2024. 134 Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR, 2023. 129, 132 Peizhong Gao, Ao Xie, Shaoguang Mao, Wenshan Wu, Yan Xia, Haipeng Mi, and Furu Wei. Meta reasoning for large language models. arXiv preprint arXiv:2406.11698, 2024. 140 146BIBLIOGRAPHY Louie Giray. Prompt engineering with chatgpt: a guide for academic writers. Annals of biomedical engineering, 51(12):2629–2633, 2023. 127 Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qi- hao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 142, 144 Nikolaus Hansen and Raymond Ros. Black-box optimization benchmarking of newuoa compared to bipop-cma-es: on the bbob noiseless testbed. In Proceedings of the 12th annual conference companion on Genetic and evolutionary computation, pages 1519–1526, 2010. 133 Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reason- ing yet. arXiv preprint arXiv:2310.01798, 2023. 140, 141 Daniel Kahneman. Thinking, fast and slow. macmillan, 2011. 143 Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can llms ac- tually correct their own mistakes? a critical survey of self-correction of llms. Trans- actions of the Association for Computational Linguistics, 12:1417–1440, 2024. 128 Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners. Advances in neural informa- tion processing systems, 35:22199–22213, 2022. 129 Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Ha- jishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational lin- guistics: human language technologies, pages 1152–1157, 2016. 128 Emanuele La Malfa, Christoph Weinhuber, Orazio Torre, Fangru Lin, Samuele Marro, Anthony Cohn, Nigel Shadbolt, and Michael Wooldridge. Code simulation chal- lenges for large language models. arXiv preprint arXiv:2401.09074, 2024. 135 Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. 142 Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ̈ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ̈ aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020. 134 Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146, 2017. 128, 130 BIBLIOGRAPHY147 Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023. 126 Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self- refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2023. 139 Sarah Mercer, Samuel Spillard, and Daniel P Martin. Brief analysis of deepseek r1 and it’s implications for generative ai. arXiv preprint arXiv:2502.02523, 2025. 142 Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for eval- uating and developing english math word problem solvers.arXiv preprint arXiv:2106.15772, 2021. 128 Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with lan- guage models. arXiv preprint arXiv:2112.00114, 2021. 130 Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wol- czyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuci ́ nski, Lerrel Pinto, Rob Fer- gus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rockt ̈ aschel. BAL- ROG: Benchmarking agentic LLM and VLM reasoning on games. In The Thir- teenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=fp6t3F669F. 128, 134, 135 Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021. 128 Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas B ̈ ack. Multi-step reasoning with large language models, a survey. ACM Computing Surveys, arXiv preprint arXiv:2407.11511, 58(6):160:1–160:35, 2025. 129 Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 127 Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Fran- cis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. 128 Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.07522, 2024. 134 148BIBLIOGRAPHY Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475, 2024. 132 Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413, 2016. 130 Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Advances in neural informa- tion processing systems, 33:20601–20611, 2020. 128 Anian Ruoss, Fabio Pardo, Harris Chan, Bonnie Li, Volodymyr Mnih, and Tim Ge- newein. Lmact: A benchmark for in-context imitation learning with long multimodal demonstrations. arXiv preprint arXiv:2412.01441, 2024. 128, 134, 135 Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery, 8(4):e1249, 2018. 130 Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024. 127 John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 142 Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 142 Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. 139, 140 Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506.06941, 2025. 128, 140 Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆ ot ́ e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. 128 Niki van Stein and Thomas B ̈ ack. Llamea: A large language model evolutionary algo- rithm for automatically generating metaheuristics. arXiv preprint arXiv:2405.20132, 2024. 132 BIBLIOGRAPHY149 Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. Advances in Neural Information Processing Systems, 36:74530–74543, 2023. 134 Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 130, 131 Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022a. 127 Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824– 24837, 2022b. 128, 129 Pengcheng Wen, Jiaming Ji, Chi-Min Chan, Juntao Dai, Donghai Hong, Yaodong Yang, Sirui Han, and Yike Guo. Thinkpatterns-21k: A systematic study on the impact of thinking patterns in llms. arXiv preprint arXiv:2503.12918, 2025a. 128 Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with ver- ifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245, 2025b. 142 Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09561, 2022. 130 Annie Wong, Thomas B ̈ ack, Aske Plaat, Niki van Stein, and Anna V Kononova. Rea- soning capabilities of large language models on dynamic tasks. In 3rd Interna- tional Conference on Foundation and Large Language Models (FLLM2025), 25-28 November, Vienna, Austria, arXiv preprint arXiv:2505.10543, 2025. 134 Annie Wong, Aske Plaat, Thomas B ̈ ack, Niki van Stein, and Anna V Kononova. State design matters: How representations shape dynamic reasoning in large language models. Transactions on Machine Learning Research, 2026. 134 Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie-yan Liu. A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 130 BY Yan, Chaofan Li, Hongjin Qian, Shuqi Lu, and Zheng Liu. General agentic memory via deep research. arXiv preprint arXiv:2511.18423, 2025. 134 150BIBLIOGRAPHY Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui. Buffer of thoughts: Thought-augmented reasoning with large language models. arXiv preprint arXiv:2406.04271, 2024. 128, 140, 141 Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022a. 128 Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b. 139 Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large lan- guage models. Advances in Neural Information Processing Systems, 36, 2024. 128, 136, 137 Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Association for Computational Linguistics, 2018. 128 Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025. 142 Alex L Zhang, Tim Kraska, and Omar Khattab. Recursive language models. arXiv preprint arXiv:2512.24601, 2025. 134 Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. Mem1: Learning to synergize memory and reasoning for efficient long-horizon agents. arXiv preprint arXiv:2506.15841, 2025. 134 Chapter 7 Vision, Action and World Models What You Will Learn Agents act in a world that consists of more than language. In this chapter you will learn how transformer-based models can gain world knowledge in modalities beyond language. Specifically, you will be able to: • Describe the architecture of vision-language models, or VLMs, (mostly) trans- former models that consist of a language and a vision part. VLMs are highly successful, for example in generating realistic fake images, and film clips. • Understand how vision-language-action models work. VLAs are also trained on robotic action. VLAs help improve the generality of robot behavior. • Explain why agents benefit from world models: models that have learned some physical aspects of the world around us, preventing agents from attempting to walk through walls. 7.1 Multi-modal Vision Language Models Many decisions involve vision. As we will see in this section, the transformer archi- tecture is also able to learn tokenized visual patterns. Models that are pretrained on both language and visual data are called vision-language models, or VLMs. VLMs typically consist of (1) a language part, (2) a vision part, and (3) a part that combines both embeddings. We will discuss various architectures of these vision-language models. For each type of model we will describe the task, the architecture of the model, and the algorithm with which it is trained. We will see impressive results. Computer vision has a long and successful history in AI. Deep learning has been successful in image recognition tasks, long before the breakthroughs in natural lan- guage processing [Krizhevsky et al., 2012]. Can these two successful fields be com- 151 152CHAPTER 7. VISION, ACTION AND WORLD MODELS Figure 7.1: A Question in a Visual-Question Answering Tasks might be: What is the mustache made of? bined? This is not a trivial question, since the two fields are based on deep learning architectures that re quite different: image processing is based on convolutional neural networks that work well on two-dimensional rectangular grids of pixels, and language processing on transformers that work well on one-dimensional sequences of tokens. As we will see, architectures have indeed been developed for combining the two fields, to the extent that most frontier models are now VLMs. 7.1.1 Tasks Let us first discuss the types of tasks that vision-language models must solve, and list some of the popular benchmark suites. For understanding, the models can be tested on visual question answering tasks (VQA), to see if the models have acquired some tex- tual understanding of visual images. A related, but generative, task is that of captioning problems, where models have to generate a textual descriptions of images. Addition- ally, models may be tasked to do image generation. Let us have a look at some example tasks. For Figure 7.1, a VQA question could be: What is the mustache made of? and the system should answer: Bananas. Other typical VQA questions are to count the number of certain shapes of a certain color in a certain part of an image (to answer questions such as: Is there a red block in the upper corner of the table in this scene?). The tasks can be quite diverse, and there is a range of popular benchmarks such as MOSAIC [Burda-Lassen et al., 2025], GLOBAL-RG [Bhatia et al., 2024], CultureVQA [Nayak et al., 2024], FoodieQA [Li et al., 2024], and MaRVL [Liu et al., 2021]. In captioning problems, a model could be prompted to provide a caption for the scene in Figure 7.2, which shows a happy dog standing in the ocean. Note that vari- ations may be equally valid, such as a wet dog, a cold dog, or a dog with a red collar standing in the ocean. Popular benchmarks are MS-COCO [Lin et al., 2014], nocaps, 1 and, for video, MSR-VTT [Xu et al., 2016], and VATEX for traditional, short-caption 1 https://nocaps.org 7.1. MULTI-MODAL VISION LANGUAGE MODELS153 Figure 7.2: A Captioning Task, possible Caption: A happy dog is standing in the ocean evaluation [Wang et al., 2019]. Newer benchmarks have been developed that focus on detailed, multi-faceted descriptions: DetailCaps [Dong et al., 2024], VDC [Chai et al., 2024], and CapArena [Cheng et al., 2025]. Typical applications of VLMs are summarization, image generation, search and retrieval, image segmentation, and object detection. The Massive Multi-discipline Multi-modal Understanding and Reasoning Benchmark (MMMU) for Expert AGI is a comprehensive benchmark for vision-language models [Yue et al., 2024]. 7.1.2 Architecture Now that we have seen the types of tasks that VLMs are tested on, how could we design a model to perform these tasks? Language, when modeled as a sequence of tokens, is a sequential (1 dimensional) problem domain. Vision, when modeled as a grid of pixels, is a rectangular (2 dimen- sional) problem domain. Correspondingly, the deep learning architectures for the two applications are different. For language, the transformer architecture is used, finding sequential patterns, and for imaging, the convolutional neural network is used, find- ing 2-dimensional patterns. It is therefore not obvious what architecture a multi-modal vision-language model should have. Indeed, it was unclear if a VLM would work at all. Research first focused on the question if the transformer architecture would also be able to work on image processing (in addition to the conventional convolutional neural networks, CNN). Special vision transformers and tokenization schemes were 154CHAPTER 7. VISION, ACTION AND WORLD MODELS Figure 7.3: Architecture of a VLM (after LLaVa [Liu et al., 2023], image from Hug- gingface) developed that worked well in image processing [Dosovitskiy et al., 2020]. Then, an architecture was needed to combine the language and image embeddings, and to generate language. In general, the architecture of a VLM consists of an image encoder (either a CNN or a specially tokenized vision-transformer) and a text encoder that are connected to an embedding projector to align image and text representations (often a dense neural network), connecting to, finally, a text decoder (a transformer). Figure 7.3 illustrates this architecture, using LLaVa as an example [Liu et al., 2023]. The connection between the visual embedding and the textual embedding is made by a network that projects the visual embedding to another embedding, to allow unified reasoning by the model. The projector network is often a dense neural network (multi- layer perceptron, MLP). More elaborate architectures have been developed, and the architecture of state of the art VLMs can become quite complex, consisting of multiple transformer modules. As an example, Figure 7.4 shows the generative Stable Diffusion architecture [Esser et al., 2024, Weng, 2021, Yang et al., 2023]. 7.1.3 Learning Algorithm Now that we have seen the tasks and the architecture of VLMs, it is time to look at the type of learning algorithm that is used by the models. The different encoders and projectors allow the VLM to be trained. The training algorithms should align and fuse the knowledge from both the visual and the language encoders. The VLM can then correlate the image and text information so that the model can analyze the query and provide an answer based on the two modalities. The literature reports a variety of approaches for training the VLM. OpenAI’s CLIP [Radford et al., 2021], which stands for contrastive language- image pretraining, uses contrastive learning. Contrastive learning is a form of self- supervised learning, creating supervisory signals when there are no such labels in the dataset. The idea is to push dissimilar data points apart and pull similar data points together in an embedding space. Contrastive learning maximizes the similarity of rep- resentations for positive pairs (such as similar forms of the same image) and minimize the similarity for negative pairs (images of different objects). 7.1. MULTI-MODAL VISION LANGUAGE MODELS155 Figure 7.4: Architecture of Stable Diffusion [Esser et al., 2024] Another approach is to use masking. Here the VLM learns to predict randomly obscured parts of an input text or image, just as it is used in language model training. The model learns to reconstruct the hidden pixels in an image given an (unmasked) caption. An example is FLAVA, or foundational language and vision alignment, which uses a vision transformer as image encoder and a transformer for its language encoder and multi-modal projector [Singh et al., 2022]. VLM’s are also used in a generative style, to generate an image based on a text prompt. Popular text-to-image models often use diffusion models [Ho et al., 2022, Yang et al., 2023]. Well-known examples are Google’s Imagen [Baldridge et al., 2024], Midjourney, OpenAI’s DALL-E [Betker et al., 2023] and Stability AI’s stable diffusion [Esser et al., 2024]. DALL-E 2 uses a diffusion model conditioned on CLIP image embeddings. Figure 7.5 shows DALL-E’s well-known surrealistic avocado armchairs. Pretraining vision language models from scratch is expensive, and since a VLM consists of a language model and a vision model, we could try to see if it is possible to build a VLM out of one or two existing models, that have already been pretrained before. Both a pretrained LLM and a pretrained vision encoder can be used, to which a mapping network can be added that projects the visual embedding to the textual input space of the language model. An example is LLaVA, which stands for large language and vision assistant. LLaVa uses the Vicuna language decoder and the CLIP ViT visual encoder, and their outputs are merged into a shared dimensional space using a linear projector [Liu et al., 2023]. Many vision language models have been developed. VLMs tend to perform well on benchmark tasks and they are often used for decision making in agents. VLMs 156CHAPTER 7. VISION, ACTION AND WORLD MODELS Figure 7.5: DALL-E: Avocado Armchairs Figure 7.6: Flava Pretraining Image-Text Dataset Examples [Singh et al., 2022] have become so popular that many current open and commercial LLMs are VLMs: DeepSeek-VL2, Gemini 2.0 Flash, GPT-4o, Llama 3.2, NVLM, Qwen 2.5-VL. 7.1.4 Results Let us take a look at how well VLMs perform at captioning and image generation by looking at some of the examples that are reported in the literature. Vision language models have produced imaginative and inspiring results ranging from visual question answering, object detection, image captioning to text-to-image generation. An example of the test dataset that is used in Flava is shown in Figure 7.6. The successor of DALL-E 2, DALL-E 3, focused on training on highly descriptive generated image captions, yielding even more impressive results [Betker et al., 2023]. Some of the prompt following image generation capabilities of DALL-E 3 are shown in Figure 7.7. Figure 7.8 shows examples from stable diffusion (based on the rectified- flow diffusion models). 7.1. MULTI-MODAL VISION LANGUAGE MODELS157 Figure 7.7: Prompt Following Text-to-Image Examples from DALL-E 3 [Betker et al., 2023] Figure 7.8: Stable Diffusion Examples [Esser et al., 2024] 158CHAPTER 7. VISION, ACTION AND WORLD MODELS Figure 7.9: VLA has a different meaning in different languages. So has Vlas. (A hint of which language this is flagged in one of the pictures) 7.2 Decisions & Vision Language Action Models To act, agents must make decisions, and to make decisions, different options must be weighed. As we will see in this section, we can train transformers on episodes of states, actions and rewards(푠,푎,푟) (see also Section 20.1.2). These models are trained on sequences of language, vision, and robotic actions. Vision-language-action models, VLAs, (Figure 7.9) are trained to output robot API commands, which are a special type of language: the language of commands to steer robotic movement. The task of a VLA is to understand a visual scene, and, when given a textual com- mand, instruct a robot arm to perform an appropriate action, depending on what the model sees through the camera. These models have shown impressive results of robots generalizing to unseen situations, successfully performing tasks. 7.2.1 Tasks The main task of VLAs is to perform instructions in the real world, and to do so in a general way, in a changing environment. When a robotic manipulation model has been trained to pick up a set of red pencils, and the task changes to pick up blue pencils, then it would be nice if the model generalizes, and we do not have to assemble a new training set involving blue pencils and re-train the model again from scratch. We also want our model to recognize objects, and recognize when an object is oriented differently, has moved, or the lighting changes. The model should know how to use a robot arm to manipulate objects, and how to perform different actions. See Figure 7.10 for an example, taken from the CLIPort paper [Shridhar et al., 2022]. The hardest task for VLAs is to combine these scenes, objects and actions to ma- nipulations that are complicated combinations of what the model has seen during train- 7.2. DECISIONS & VISION LANGUAGE ACTION MODELS159 Figure 7.10: Example from CLIPort: put the gray letter E in the left letter E shape hole [Shridhar et al., 2022] Figure 7.11: Another Example from CLIPort: sweep the beans into the blue zone [Shridhar et al., 2020] 160CHAPTER 7. VISION, ACTION AND WORLD MODELS ing, and to generalize to unseen combinations. Especially long-horizon navigation through complex, changing, environments (perhaps even outdoors) is a complex chal- lenge [Shridhar et al., 2022]. In realistic human-robot interaction settings, collecting additional demonstrations is often infeasible. To perform high-level tasks, such as in Figure 7.11, the VLA must be able to recognize objects, their poses, and segmentations of instances. The model must also be able to understand language, and relate the meaning of language to the meaning of visual objects and to the meaning of robotic actions. As we will see, language provides an intuitive interface for specifying goals and for transferring semantic concepts across tasks. Furthermore, the work on VLMs has shown how to semantically connect different kinds of embeddings. 7.2.2 Architecture Now that we have seen the task and the challenges, let us look at possible model ar- chitectures. Again, they can become quite complex. The transformer architecture is well suited to learn patterns in sequential data, as the results on natural language show. Intuitively, this suggests that transformers may also work well to model sequential deci- sion problems, the type of problems that are often solved in reinforcement learning for robotic action, as we discussed in Section 2.2.3 (and further in depth in Section 20.2.1). These problems consist of a state, from which actions cause the system to enter a new state, which has actions, etc. The transformer architecture is well suited to model these problems. Early work on decision transformer (DT) [Chen et al., 2021, Janner et al., 2021] showed that on offline (static) reinforcement learning problems, the GPT architecture was indeed able to learn optimal policies. The decision-pretrained transformer (DPT) Lee et al. [2023], Wang et al. [2024a] showed that the basic transformer architectures can be pretrained and finetuned on decision traces. Section 20.1.2 discusses decision traces further. Following pioneering work on CLIPort [Shridhar et al., 2022, 2020], VLAs are often built on VLMs, where the robotic commands are used as regular language tokens. Let us have a closer look at some representative models. The goal of Mobility VLA [Chiang et al., 2024] is navigation. This VLA is created by combining a long context VLM with topological maps, as illustrated in Figure 7.12. Mobility VLA is used on a problem category of navigation tasks called multi-modal instruction navigation with demonstration tours (MINT). In this dataset, an environ- ment prior is provided through a previously recorded demonstration video. The model consists of a two-level VLA that combines the environment understanding and com- mon sense reasoning of long-context VLMs with a low-level navigation policy based on topological graphs. The high-level policy is created by combining user instructions from a long-context VLM that finds the goal frame in a demonstration video. Then, a low-level policy starts at this goal frame to traverse a topological graph to generate the robot action instructions at every timestep. LM-Nav [Shah et al., 2023] uses a different architecture, based on existing models. It takes as input a set of raw observations from the target environment and from free- form textual instructions. From this, it derives an action plan using three pretrained 7.2. DECISIONS & VISION LANGUAGE ACTION MODELS161 Figure 7.12: Mobility VLA Architecture: Long Context VLM and Topological Maps [Chiang et al., 2024] Figure 7.13: Three-LLM Architecture of LM-Nav [Shah et al., 2023] models: a language model (LLM) for extracting landmarks, a vision-language model (VLM) for grounding, and a visual-navigation model (VNM) for execution (not unlike the topological map previously discussed). Together, these models enable LM-Nav to follow textual instructions from visual observations without any finetuning. Figure 7.13 shows an illustration of the approach. Google’s RT-2 [Brohan et al., 2023] uses a different approach. The idea of RT-2 is to leverage knowledge from the web to inform robotic control: the robot should learn how to act from watching videos. A standard vision language model is finetuned both on textual robotic trajectory data and on Internet-scale vision-language tasks, such as visual question answering. Robotic actions are expressed as regular text tokens and are incorporated directly into the training set in the same way as natural language tokens, as shown in Figure 7.14. A similar approach is taken by Magma [Yang et al., 2025]. Magma is pretrained on heterogeneous datasets spanning from images and videos to robotics data. However, in Magma the actionable visual objects in images are labeled by special tokens (Set-of-Mark) for action grounding. Similarly, object movements (such as the trace of human hands or robotic arms) are labeled by Trace-of-Mark to ease action planning. Figure 7.15 provides an illustration. Finally, a VLA called 휋 0 [Black et al., 2024] stresses the reinforcement learning approach by using a pretrained late-fusion vision-language model, as well as a diverse 162CHAPTER 7. VISION, ACTION AND WORLD MODELS Figure 7.14: RT-2 Architecture: VLM trained using Robotic Actions as Text Tokens [Brohan et al., 2023] Figure 7.15: Magma Pretraining Pipeline [Yang et al., 2025] 7.2. DECISIONS & VISION LANGUAGE ACTION MODELS163 Figure 7.16: Architecture of 휋 0 . The model is based on a VLM, enhanced with a cross- embodiment dataset [Black et al., 2024] cross-embodiment dataset with a variety of manipulation tasks. The model is inspired by the Transfusion approach [Zhou et al., 2024]. This model is adapted to robot control by adding a separate action expert that produces actions via flow matching, for fluent manipulation skills. The resulting model can either be used directly to perform tasks based on a prompt, or be further finetuned on higher-quality data to enable complex multi-stage tasks, such as folding multiple articles of laundry or assembling a box (see Figure 7.16). 7.2.3 Learning Algorithm The model architecture of most VLAs is based on VLMs, in some cases augmented with navigation map data. The robotic action commands are typically encoded as regular text commands. Likewise, for pretraining and finetuning typical LLM self- supervised learning algorithms are used, similar to VLMs and decision transformers. Sometimes existing pretrained models are used, to reduce the training expense. 7.2.4 Results Various VLA models have been created that achieve impressive zero-shot results, gen- eralizing behavior to unseen situations. Chiang et al. [2024], Brohan et al. [2023], Yang et al. [2025] are examples of VLA models for robotic action, achieving complex tasks such as navigation, table top manipulation, or even folding laundry [Black et al., 2024]. VLAs have shown impressive and imaginative results, where robots were able to generalize at inference time without the need for retraining. A survey of VLAs can be found in [Ma et al., 2024]. 164CHAPTER 7. VISION, ACTION AND WORLD MODELS The results are best appreciated from seeing a video. For 휋 0 , a video is available. 2 For RT-2, a video is also available. 3 7.3 World Models, Physics, and Planning To a VLM, an image consists of pixels without a deeper intrinsic meaning. However, agents that interact with the world need knowledge about physical aspects of that world. The models need an understanding of what is physically possible, so that agents do not walk through walls, as they still sometimes do in video games. World models help agents understand the world, allowing them to act effectively in the physical world, taking actions that make sense. Originally, LLMs are language models. We can converse with them in natural language, and they necessarily contain some knowledge of the world, but they were not specifically trained to have a wide and deep understanding of worldly affairs. Some researchers argue that in order for agentic LLMs to be truly successful, and for LLMs to reach a high level of general intelligence, research into world models should be strengthened [Sutton et al., 2022, Matsuo et al., 2022, LeCun et al., 2022, Klindt et al., 2026]. Agents and robots that act in the world need a rudimentary understanding of the physics of the world. One of the challenges for training an LLM is to ground its un- derstanding of the physical world and of the possible robotic actions into reality. One way to do this is to learn an agentic model of the world (environment) from feedback on actions. The term world model has been used in AI in other contexts. Most notably, it refers to the model in model-based reinforcement learning [Ha and Schmidhuber, 2018, Moerland et al., 2023, Sutton, 1991] (see next section). This model contains a local copy of some knowledge of the environment, based on the interactions it has had with the environment, to be used to improve the action policy. The world model contains an efficient, compressed, learned representation of the world [Hafner et al., 2019, Plaat et al., 2020]. The representation of a world model is often called an abstract or latent representation, to contrast it with the regular representation of the real world in the environment. World models are often trained to contain knowledge about physics and certain spatial properties, that puts restrictions on the agent’s actions. The precise contents of the world model depends on the domain in which the agent operates. Note that in a just few chapters we have progressed from understanding language to understanding the world. The goal of a large language model is to understand some semantics of natural language from statistical patterns in text, to be used for question answering and translation. The goal of a world model is to understand causal relations in the laws of physics, to be used to simulate future world states for our agent. Quite some new technology is necessary to go from LLM to agentic LLM. We start with a look at how world models are currently used in agentic LLM. 2 https://w.physicalintelligence.company/blog/pi0 3 https://robotics-transformer2.github.io 7.3. WORLD MODELS, PHYSICS, AND PLANNING165 7.3.1 Tasks World models should encapsulate knowledge of the world, so that an agent can per- form meaningful actions. The actions should be grounded in the physical world. For example, the model should understand the limitations of the joints of a robot arm that it is commanding; a navigational agent should understand some physics, such as the concept of gravity, of distance, and that walls are solid objects; a financial agent should understand the consequences of a negative bank balance. Often this is the type of in- formation that humans take for granted, and we are surprised when an agent fails to understand such basic things about the world in which it operates. VLMs that are only trained on pixels may miss the semantic meaning of a specific texture, whether it is a garment, smoke, or a wall. Standard pretrained LLMs already contain some useful world knowledge. They learn acceptable textual behavior using reinforcement learning with human feedback (RLHF), and acceptable mathematical and coding behavior using reinforcement learn- ing with verifiable rewards (RLVR). Some early approaches use LLMs as pretrained world model without any finetuning. A natural way to learn about the physics of the world is through interaction: agents can learn acceptable physical behavior through a reinforcement learning-style reward function. The goal of world models in model-based reinforcement learning is to improve the sample efficiency of learning from interaction, by planning with the world model instead of sampling from the real world. This is especially useful when interacting with the world is more expensive than performing a simulation, as will often be the case in agentic LLM. In the approaches that we will discuss, LLM world models use methods from model-based reinforcement learning (see Section 7.3.5). Currently, world model approaches in LLM are quite diverse. To show which tasks world models are used for in LLMs, we will discuss three examples: WorldGPT, WebDreamer, and WorldCoder. WorldGPT The goal of WorldGPT [Ge et al., 2024] is to solve visual question answering and embodied planning tasks, including audio-video question answering. WorldGPT learns a world-VLM from videos. It uses a latent space VLM with external knowledge bases and long-term memory. WebDreamer The goal of WebDreamer [Gu et al., 2024] is to improve efficient learning of Web-interaction tasks, such as e-commerce. WebDreamer enables web- agents to simulate actions before execution, and thus be faster and more sample effi- cient (by not having to act in the real web to gather experience). WebDreamer is a model-based reinforcement learning approach in a web-world. (The name is derived from Dreamer, a series of influential model-based reinforcement learning latent dy- namics models [Hafner et al., 2019, 2020a, 2023].) Typical tasks are to find products on e-commerce sites. It does so by training a small model from a large model, allowing efficient simulation. WorldCoder The goal of WorldCoder [Tang et al., 2024] is also to improve sample efficiency, but the approach is different. WorldCoder improves efficiency by gener- ating the dynamics model (transition function and reward function) in interpretable 166CHAPTER 7. VISION, ACTION AND WORLD MODELS Figure 7.17: Sokoban for WorldCoder [Tang et al., 2024] Figure 7.18: AlfWorld for WorldCoder [Tang et al., 2024] 7.3. WORLD MODELS, PHYSICS, AND PLANNING167 Figure 7.19: Architecture of WorldGPT [Ge et al., 2024] Python code, which is then used in a standard model-based reinforcement learning de- sign. The generated Python code executes more efficiently than LLM inference does. WorldCoder is used in grid-tasks (Sokoban, see Figure 7.17) and domestic task plan- ning (AlfWorld, see Figure 7.18). 7.3.2 Architecture The purpose of a world model is to add semantic meaning to observations—actions having consequences—something that LLMs and VLMs may struggle with. When does a pixel belong to a floor to walk on (restrict 푦 movement) and when is it part of a wall to stop you (restrict 푥 movement)? A reward function provides feedback infor- mation on which actions are good, and which are not. Thus, world model architectures often use a reinforcement learning setting, in which an action policy is trained from feedback rewards. We can think of a world model as a VLM with a policy learner inside. WorldGPT, WebDreamer, and WorldCoder use different architectures for their world model, which we will now discuss in more detail. WorldGPT WorldGPT is based on a standard LLM, Vicuna-7B-v0. 4 It performs reasoning in latent space, following the model-based reinforcement learning approach. The goal of WorldGPT is to be a simulator of realistic dynamics of the world—the transition and the reward function in reinforcement learning. To do so, the architecture helps synthesize multi-modal instruction instances, using a special architecture. At the core WorldGPT follows the common VLM architecture with a central LLM, and a multi-modal encoder and a decoder. To help the model synthesize multi-modal in- structions, a cognitive architecture is added to help with specialized or long-term tasks. The architecture adds memory, knowledge retrieval, and context reflection. The con- text reflection is built on Q-former [Li et al., 2023]. See Figure 7.19 for a diagram of the architecture. 4 https://huggingface.co/lmsys/vicuna-7b-v1.3, https://lmsys.org 168CHAPTER 7. VISION, ACTION AND WORLD MODELS Figure 7.20: WebDreamer pseudocode: using LLM for score and sim function [Gu et al., 2024] Figure 7.21: WorldCoder Architecture [Tang et al., 2024] WebDreamer WebDreamer is aimed at web-interaction. It distills a large model, Qwen-70B [Bai et al., 2025], into a small 7B model. The small model is then used for simulating actions with the world model, before acting in the “real” web-world. It uses model predictive control (MPC) as planning approach [Garcia et al., 1989, Schwenzer et al., 2021] for decision making about website actions, re-planning at each interaction step. WebDreamer leverages LLMs as world model and as value function. The LLM implements both the simulation function (sim) and the scoring function (score), see Figure 7.20. Furthermore, following [Koh et al., 2024], the LLM is used for candidate action generation (in addition to sim and score). The authors employ a two-stage ap- proach: first sampling top-푘 actions, and then using the LLM to self-refine unnecessary actions for simulation. WorldCoder WorldCoder is a model-based agent that builds a Python program to represent its knowledge of the world. WorldCoder creates a transition function of the world in executable Python code, see Figure 7.21. WorldCoder uses program synthe- sis. In reinforcement learning, problems are formalized as MDPs, consisting of a state representation 푆, action function 퐴, transition function 푇 , and reward function 푅 (see Section 7.3.5). WorldCoder uses an LLM to generate Python code for action, transi- tion and reward functions for a task, as well as for the state representation, which is then used in a model-based reinforcement learning approach. This code is checked and improved, in an iterative self debug refinement process [Chen et al., 2023], see 7.3. WORLD MODELS, PHYSICS, AND PLANNING169 Section 6.1.3.2. This process repeats until the program produces the correct output. Note that in WorldCoder the learned model is symbolic (the transition function is represented as computer code), where in regular deep reinforcement learning the model is a neural network. At inference time, WorldCoder is efficient, since no LLM calls occur that require forward passes through a large network, only interpretation of the transition and reward function Python code. In regular model-based LLM approaches, the LLM simulates the world, in World- Coder the LLM builds a simulation of the world, for later execution. At inference time, the Python interpreter executes the transition function, where in ReAct, an LLM rea- sons about the world state. For planning problems, the symbolic approach is better suited, since LLMs struggle with planning during inference (see Section 6.2.3). 7.3.3 Learning Algorithm We will now discuss the pretraining and finetuning algorithms that the different ap- proaches use. WorldGPT WorldGPT acquires an understanding of world dynamics by pretraining and finetuning on millions of videos across various domains. Vicuna-7B-v0 is used as the base LLM. Pretraining then uses a language-centric encoder to obtain a uni- fied representation of the multi-modal state. For multi-modal decoding, special tokens are inserted into the LLM’s vocabulary. Training starts uni-modal, and progresses to cross-modal. For finetuning, a special cognitive reinforcement architecture is used with a dataset with fine-grain sequential state-transition sample annotations from more than 10 million videos. As mentioned before, the cognitive architecture consists of mem- ory, knowledge retrieval, and context reflection. Through finetuning, agents acquire specialized knowledge by finetuning on synthetic multi-modal instruction data. WebDreamer The WebDreamer approach aims to learn dynamics suitable for model- based reinforcement learning. First the pretrained Qwen-2-VL vision language model [Wang et al., 2024b] is used to initialize the Dreamer-7B model, which is then further finetuned on 3.1 million interaction examples. The data synthesis pipeline consists of (1) random walks over the web, where the model autonomously interacts with web pages through actions such as typing for search and clicking, and (2) synthesis, where Qwen2-VL-72B generates textual descriptions of state changes based on visual snap- shots before and after the actions. WebDreamer simulates possible future states for each action over a finite horizon (replanning and retraining according to the model predictive control (MPC) approach, a reinforcement learning approach for adaptive retraining that is used in engineering and control problems). WorldCoder WorldCoder uses program synthesis to learn to make a transition func- tion, the function that computes the next state of the world. WorldCoder then performs planning-actions using that coded world model. It initially performs random explo- ration to initialize a dataset of environmental interactions; it updates its world model using a program synthesis algorithm, generates the code, and then performs planning. 170CHAPTER 7. VISION, ACTION AND WORLD MODELS Different planners are possible, from depth-limited value iteration (in simple domains) to MCTS (for more complex domains) [Browne et al., 2012]. 7.3.4 Results We will now look at the results of the three approaches. WorldGPT The WorldGPT model is evaluated with a multi-modal state transition prediction benchmark. WorldGPT is effective, and understands and predicts the dy- namics of complex real-world kitchen and cooking scenarios accurately. The system is also evaluated as a world simulator, showing that it can generalize to unfamiliar domains. In one experiment, WorldGPT is 30 times faster then traditional diffusion-based methods on audio-video question answering tasks [Ge et al., 2024]. WebDreamer WebDreamer also performs well. It is reported to surpass GPT-4o performance on web interaction tasks. Substantial performance improvements are re- ported over reactive baselines, on which it is 4-5 times more efficient than tree search. WebDreamer performs well on the sandbox environment VisualWebArena, and also on real-world websites (Online-Mind2Web and Mind2Web-Live), where tree search methods are difficult to implement and deploy. Moreover, on online benchmarks, us- ing Dreamer-7B, the approach achieves performance comparable to GPT-4o. On real- world websites, WebDreamer outperforms the reactive baseline with a relative gain of 42% and 24%, respectively. WorldCoder For WorldCoder results are reported on grid worlds and on task plan- ning (AlfWorld). WorldCoder is more sample-efficient than model-free reinforcement learning, and more compute-efficient compared to ReAct-style agents that try to per- form in-context planning by the LLM. Generating world knowledge as computer code by LLMs brings advantages. It allows prior world knowledge embedded in the LLM to improve code generation, as existing solutions to published problems may well have been seen during pretraining. The code can also be transferred across tasks by reusing pieces of old programs. Fur- thermore, the results are human readable, allowing the world knowledge to be inspected by humans by reading the generated code. Though promising for planning problems, a limitation of the WorldCoder approach is inherent to the computational difficulty of planning problems. Problem instances that are difficult to solve with dedicated solvers are also difficult to solve for WorldCoder solvers. The transition function that is generated by the LLM is unlikely to be better than those designed by expert human researchers. So far, the synthesized code aims to reproduce the transition and reward functions. For future work, self reflection and metalearning approaches may be able to improve performance further. 7.3. WORLD MODELS, PHYSICS, AND PLANNING171 Environment: 푇(푠,푎) → 푠 ′ , 푅(푠 ′ ) → 푟 Agent: 푟 improves 휋 휃 (푠) → 푎 (푠 ′ ,푟) 푎 Figure 7.22: The standard reinforcement learning loop.The agent learns a 휃- parameterized policy 휋 휃 from feedback rewards 푟 from the environment. [Sutton and Barto, 2018] 7.3.5 *Model-Based Reinforcement Learning World models are being used successfully in agentic LLM. An extensive literature ex- ists on world models in model-based reinforcement learning. At the end of this chapter, we will now provide a brief overview of world models in model-based reinforcement learning, in order to provide more context for the use of world models in LLM. An influential early model-based approach is Dyna [Sutton, 1991]. Influential re- cent works are [Ha and Schmidhuber, 2018, Hafner et al., 2020b]. Two modern surveys are Moerland et al. [2023], Plaat et al. [2023]. In reinforcement learning, agents learn how to act in an environment by sampling rewards from actions that are executed in the environment. 7.3.5.1 Model-free Reinforcement Learning In regular reinforcement learning an optimal agent policy is learned model-free: the best actions for all states are learned directly from interaction with the environment (see Figure 7.22), without the help of an extra model of the environment. Regular reinforcement learning is formally described as a Markov decision process, with four functions 푆,푇, 퐴, 푅: a function 푆(표) → 푠 to represent the state of the system for observation 표, a function 푇(푠,푎) → 푠 ′ that transitions from one state to the next state using an action, a function 퐴(푠) → 푎 that returns an action for a state, and a func- tion 푅(푠) → 푟 that assign a reward value (positive or negative) to a state. The goal of reinforcement learning is to use the MDP to learn an optimal policy 휋 ∗ (푠) → 푎 ∗ , the function that returns the best action for each state that the MDP can be in. Thus, a reinforcement learning problem consists of four functions 푆, 푇 , 퐴, and 푅, and the goal of the agent is to find the optimal policy 휋 ∗ (푠) with the highest expected cumu- lative reward. The agent does not have access to 푇 and 푅, functions that reside in the environment. Regular, model-free, reinforcement learning is also called direct rein- forcement learning. The direct method naturally has high sample complexity: policy improvement requires sampling, and large MDPs require many samples for the policy improvement process to converge (Figure 7.22). 172CHAPTER 7. VISION, ACTION AND WORLD MODELS Environment: 푇(푠,푎) → 푠 ′ , 푅(푠 ′ ) → 푟 Agent: 푟 improves ˆ 푀 = ( ˆ 푇, ˆ 푅) and 푟, ˆ 푀 both improve 휋 휃 (푠) → 푎 (푠 ′ ,푟) 푎 Figure 7.23: Model-based reinforcement learning where environment feedback is used to improve both the policy 휋 and the local dynamics model ˆ 푀 = ( ˆ 푇, ˆ 푅) from which the policy 휋 is also sampled (planning) 7.3.5.2 Model-based Reinforcement Learning In direct reinforcement learning, sampled information is not retained by the agent, but forgotten, and each policy improvement requires a new sample. Interaction by the agent with the outside environment is often expensive, for example for a robot acting in the real world, whose joints wear out. In this case it would be beneficial to find ways to reduce the sample complexity. We should try to improve the policy without having to sample as much from the environment. A straightforward idea is not to throw away the samples, but to store them in a buffer, and also sample from that buffer of older interaction results, or, slightly more advanced, to train a local surrogate model 푀 = ( ˆ 푇, ˆ 푅) of the environment dynamics. When we have trained this local dynamics model, we can also sample from the model, cheaply, in addition to sampling the environment, expensively (see Figure 7.23). These methods are indirect methods, since, in the last methods, where a local dy- namics model is built up by the agent, the policy is learned indirectly (through a local model) from the environment. Using a local dynamics model is called model-based reinforcement learning. The dynamics model consist of local estimations ˆ 푇 and ˆ 푅 by the agent, estimates of the real 푇 and 푅 that are known only to the environment. Model-based reinforcement learning has been successful in some applications, such as Go [Silver et al., 2016], Diplomacy [Bakhtin et al., 2022], StarCraft [Vinyals et al., 2019], and showing learning across domains [Hafner et al., 2020b]. Model-based reinforcement learning also connects to planning, a classic field in symbolic artificial intelligence, that looks ahead into the future, simulating different possible worlds using the efficient latent representation, without having to act in the real world. 7.3.5.3 Dimension Reduction In deep model-based reinforcement learning, the local dynamics models are also called world models [Ha and Schmidhuber, 2018]. In this context world models reduce high dimensional observation spaces to lower dimensional state and reward spaces, that can then be used in model-based reinforcement learning methods to efficiently improve policy functions. The idea behind world models is that in most high-dimensional en- 7.4. CONCLUDING REMARKS173 vironments there are elements that are less important, such as, in a video game, trees in the background that never move, that have little or no relation to the value of the agent’s actions. The goal is to abstract away these unimportant elements of the input space, reducing the effective dimensionality of the state space. 7.3.5.4 Joint Embedding Predictive Architecture The joint embedding predictive architecture (JEPA) is proposed by LeCun et al. [2022] and further refined in Assran et al. [2023]. The goal of JEPA is to predict future states in an abstract, compressed representation (latent space) rather than trying to recreate raw sensory details or pixels. This goal is similar to the dimension-reduction goal of a world model in model-based reinforcement learning. Where model-based reinforce- ment learning uses learns a local transition model supervised by rewards, JEPA is a self-supervised representation learning framework. JEPA is trained via self-supervised observation (watching videos or images) to map context embeddings to target embed- dings without rewards or explicit actions. Model-based reinforcement learning ex- plicitly incorporates actions, rewards, and control optimization to maximize the return [Lim et al., 2026]. JEPA and reinforcement learning are close, JEPA can be used as a perception engine (world model) for reinforcement learning or planning algorithms, for example to guide robotic control [Destrade et al., 2025]. 7.3.5.5 Agentic World Models In the context of model-based reinforcement learning, LLMs can be used in two differ- ent ways. We can use a pre-trained (static) LLM as world model( ˆ 푇, ˆ 푅) in model-based reinforcement learning (LLM4MBRL). Often this approach uses a tree search or MCTS planning. In this approach, the LLM is used to help reinforcement learning. Alternatively (model-based) reinforcement learning settings can be used to train LLM world models (MBRL4LLM) with feedback from world-interaction. Now, the LLM has become adaptive, either through an adaptive harness, or by posttraining the model itself, as in a VLA. This approach is also known as agentic world models, and holds much promise for agents to better understand their environment, to act in more realistic ways. In this approach, reinforcement learning is used to help agentic lan- guage modeling. This is an active area of research, see, for example, Zuo et al. [2026], including their references. 5 7.4 Concluding Remarks Transformers can be trained on tokenized language data, but also on other tokenized data. In this chapter we have seen different model architectures for different modal- ities. Architecturally, the models were quite different. However, most models are 5 There is a third connection between reinforcement learning and language modeling. Model-free rein- forcement learning is also used inside the LLM training pipeline, for finetuning, as in RLHF and RLVR. 174BIBLIOGRAPHY transformer based, and tokenization approaches have been devised for a surprising dif- ference in modalities. Let us review the different types of models to see what they have in common: • Large language models (LLM) are transformers that are pretrained on tokenized natural language. • Vision language models (VLM) are models that consist of a transformer that is trained on tokenized vision data (or a diffusion model, or a convolutional net- work) and a transformer trained on tokenized natural language, whose embed- dings come together in a joint decoder. • Decision transformers (DT) are transformers that are trained on tokenized(푠,푎,푟)- episodes of decision paths (see Section 20.1.2). • Vision language action models (VLA) are models that consist of a vision-language architecture that are also trained on tokenized(푠,푎,푟) episodes of robotic actions. • World models are models that consist of a vision-language architecture that are also trained on tokenized(푠,푎,푟) episodes of real world actions or decisions. There are other types of models that we have not discussed in this chapter. For example, a large music model (LMM) is a transformer that is pretrained on tokenized musical sequences (which can be either symbolic music (score sheets) or a soundtrack) [Yuan et al., 2025]. This chapter was devoted to advanced models for agents. We discussed language models that were pretrained and finetuned on vision and action. These models under- stand more of the world, allowing agents to achieve impressive generalization results. We have seen how the LLM, originally a language model, is becoming a world model, a term that is borrowed from the field of model-based reinforcement learning. At this point it might be good to look back on what we have seen so far. The first part of the book explained how to pretrain models self-referentially for language. Then, Chapter 6 explained how in-context learning emerged, and how it can be used for inference time multi-step reasoning and self-reflection. This chapter discussed pre- training and finetuning methods to yield multi-modal world models of the environment. Together, pretrained language, inference time reasoning, and finetuned world models enable agents more and more to act intelligently and achieve tasks in the world for their users. We have now discussed many different modalities and training methods. In the next chapter we will take a step back and reflect on how agents can learn in a stable manner, while they are also acting in the world that they learn from. Bibliography Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from BIBLIOGRAPHY175 images with a joint-embedding predictive architecture. In 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 15619–15629. IEEE, 2023. 173 Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 168 Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Sci- ence, 378(6624):1067–1074, 2022. 172 Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Lluis Castrejon, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, et al. Imagen 3. arXiv preprint arXiv:2408.07009, 2024. 155 James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 155, 156, 157 Mehar Bhatia, Sahithya Ravi, Aditya Chinchure, Eunjeong Hwang, and Vered Shwartz. From local concepts to universals: Evaluating the multicultural understanding of vision-language models. arXiv preprint arXiv:2407.00263, 2024. 152 Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. 휋 0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. 161, 163 Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt- 2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 161, 162, 163 Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samoth- rakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Trans- actions on Computational Intelligence and AI in games, 4(1):1–43, 2012. 170 Olena Burda-Lassen, Aman Chadha, Shashank Goswami, and Vinija Jain. How cultur- ally aware are vision-language models? In 2025 IEEE 6th International Conference on Image Processing, Applications and Systems (IPAS), pages 1–6. IEEE, 2025. 152 Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar- Tal, Jenq-Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Ef- ficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024. 153 176BIBLIOGRAPHY Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Rein- forcement learning via sequence modeling. Advances in neural information process- ing systems, 34:15084–15097, 2021. 160 Xinyun Chen, Maxwell Lin, Nathanael Sch ̈ arli, and Denny Zhou. Teaching large lan- guage models to self-debug. arXiv preprint arXiv:2304.05128, 2023. 168 Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen. Caparena: Bench- marking and analyzing detailed image captioning in the llm era. arXiv preprint arXiv:2503.12329, 2025. 153 Hao-Tien Lewis Chiang, Zhuo Xu, Zipeng Fu, Mithun George Jacob, Tingnan Zhang, Tsang-Wei Edward Lee, Wenhao Yu, Connor Schenck, David Rendleman, Dhruv Shah, et al. Mobility vla: Multimodal instruction navigation with long-context vlms and topological graphs. arXiv preprint arXiv:2407.07775, 2024. 160, 161, 163 Matthieu Destrade, Oumayma Bounou, Quentin Le Lidec, Jean Ponce, and Yann LeCun. Value-guided action planning with jepa world models. arXiv preprint arXiv:2601.00844, 2025. 173 Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo.Benchmarking and improving detail image caption.arXiv preprint arXiv:2405.19092, 2024. 153 Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recog- nition at scale. arXiv preprint arXiv:2010.11929, 2020. 154 Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ̈ uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rec- tified flow transformers for high-resolution image synthesis. In Forty-first interna- tional conference on machine learning, 2024. 154, 155, 157 Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: The- ory and practice—a survey. Automatica, 25(3):335–348, 1989. 168 Zhiqi Ge, Hongzhe Huang, Mingze Zhou, Juncheng Li, Guoming Wang, Siliang Tang, and Yueting Zhuang. Worldgpt: Empowering llm as multimodal world model. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7346– 7355, 2024. 165, 167, 170 Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al.Is your llm secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06559, 2024. 165, 168 BIBLIOGRAPHY177 David Ha and J ̈ urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. 164, 171, 172 Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555–2565, 2019. 164, 165 Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020a. 165 Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020b. 171, 172 Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. 165 Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022. 155 Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing sys- tems, 34:1273–1286, 2021. 160 David Klindt, Yann LeCun, and Randall Balestriero. When does lejepa learn a world model? arXiv preprint arXiv:2605.26379, 2026. 164 Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv preprint arXiv:2407.01476, 2024. 168 Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012. 151 Yann LeCun et al. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62, 2022. 164, 173 Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems, 36:43057–43083, 2023. 160 Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language mod- els. In International conference on machine learning, pages 19730–19742. PMLR, 2023. 167 178BIBLIOGRAPHY Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, and Desmond Elliott. FoodieQA: A multimodal dataset for fine-grained understanding of Chinese food culture. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19077–19095, Miami, Florida, USA, November 2024. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.1063. URL https://aclanthology.org/2024.emnlp-main.1063/. 152 Jing Yu Lim, Rushi Shah, Zarif Ikram, Samson Yu, Haozhe Ma, Tze-Yun Leong, and Dianbo Liu. Jedi: Joint embedding diffusion world model for online model-based reinforcement learning. arXiv preprint arXiv:2605.13013, 2026. 173 Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ra- manan, Piotr Doll ́ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 152 Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. arXiv preprint arXiv:2109.13238, 2021. 152 Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 154, 155 Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied ai. arXiv preprint arXiv:2405.14093, 2024. 163 Yutaka Matsuo, Yann LeCun, Maneesh Sahani, Doina Precup, David Silver, Masashi Sugiyama, Eiji Uchibe, and Jun Morimoto. Deep learning, reinforcement learning, and world models. Neural Networks, 152:267–275, 2022. 164 Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model- based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1):1–118, 2023. 164, 171 Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Aishwarya Agrawal, et al. Benchmarking vision language models for cultural understanding. arXiv preprint arXiv:2407.10920, 2024. 152 Aske Plaat, Walter Kosters, and Mike Preuss. Deep model-based reinforcement learn- ing for high-dimensional problems, a survey. arXiv preprint arXiv:2008.05598, 2020. 164 Aske Plaat, Walter Kosters, and Mike Preuss. High-accuracy model-based reinforce- ment learning, a survey. Artificial Intelligence Review, 56(9):9541–9573, 2023. 171 BIBLIOGRAPHY179 Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 154 Max Schwenzer, Muzaffer Ay, Thomas Bergs, and Dirk Abel. Review on model pre- dictive control: An engineering perspective. The International Journal of Advanced Manufacturing Technology, 117(5):1327–1349, 2021. 168 Dhruv Shah, Bła ̇ zej Osi ́ nski, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning, pages 492–504. PMLR, 2023. 160, 161 Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆ ot ́ e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. 159, 160 Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pages 894–906. PMLR, 2022. 158, 159, 160 David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. 172 Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15638–15650, 2022. 155, 156 Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160–163, 1991. 164, 171 Richard S Sutton and Andrew G Barto. Reinforcement Learning. Adaptive Compu- tation and Machine Learning series. Bradford Books, Cambridge, MA, 2 edition, November 2018. 171 Richard S Sutton, Michael Bowling, and Patrick M Pilarski. The alberta plan for ai research. arXiv preprint arXiv:2208.11173, 2022. 164 Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based llm agent: Build- ing world models by writing code and interacting with the environment. Advances in Neural Information Processing Systems, 37:70148–70212, 2024. 165, 166, 168 Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Micha ̈ el Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft i using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019. 172 180BIBLIOGRAPHY Jiuqi Wang, Ethan Blaser, Hadi Daneshmand, and Shangtong Zhang. Transformers learn temporal difference methods for in-context reinforcement learning. arXiv preprint arXiv:2405.13861, 2024a. 160 Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024b. 169 Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.Vatex: A large-scale, high-quality multilingual dataset for video-and- language research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4581–4591, 2019. 153 Lilian Weng. What are diffusion models? lilianweng. github. io, page 21, 2021. 154 Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 152 Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents. arXiv preprint arXiv:2502.13130, 2025. 161, 162, 163 Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wen- tao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM computing surveys, 56(4):1–39, 2023. 154, 155 Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, et al. Yue: Scaling open foundation models for long-form music generation. arXiv preprint arXiv:2503.08638, 2025. 174 Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 9556–9567, 2024. 153 Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfu- sion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 163 Yuxin Zuo, Zikai Xiao, Li Sheng, Fei Huang, Jianhong Tu, Yuxuan Liu, Tianyi Tang, Xiaomeng Hu, Yang Su, Qingfeng Lan, et al. Qwen-agentworld: Language world models for general agents. arXiv preprint arXiv:2606.24597, 2026. 173 Chapter 8 Metalearning What You Will Learn In this chapter you will learn about metalearning. Specifically, you will be able to: • Recognize the different levels of learning that are used in LLM pipelines. • Understand the connection between in-context learning, few-shot learning and metalearning. • Create a metalearning system that consists of an inner learning loop and an outer learning loop. • Recognize the moving-target problem: why it is challenging to achieve stable learning in a dynamic metalearning system. 8.1 Why Metalearning? As we have seen, the self-reflecting LLM approaches are powerful, but getting the training processes to converge can be a challenge. In agentic LLM, learning occurs at multiple levels, and sometimes at the same time. In machine learning, algorithms that learn at multiple levels are said to perform metalearning. Metalearning is often described as learning to learn, for example, learning which prompt works better for a certain task. Metalearning can be a powerful concept, but sometimes the different learning levels interfere, and learning stops. To help us understand why our LLM multi-level learning algorithms sometimes do not work, we will look deeper into metalearning. In this chapter we will discuss how we can improve the stability of dynamic, feedback-rich, metalearning approaches. The term metalearning is used for multi-level learning algorithms. Metalearning algorithms learn to learn, to quickly solve examples of new, unseen, tasks, that the model has not trained for. 181 182CHAPTER 8. METALEARNING 8.1.1 Deep Learning is Slow Metalearning has a long history in machine learning (see, for example, Schmidhuber [1987], Hospedales et al. [2021], Huisman et al. [2021]). It is inspired by human learn- ing, since humans are said to be able to learn how to learn. Another reason for its development, is speed. The problem is that regular deep learning is slow. Neural networks for typical deep learning tasks are large, with millions or billions of parameters, and training them requires large datasets, often with millions of examples. Training times for training a network from scratch are long, from hours to months, and only when special par- allel parallel hardware (GPU) was employed, did deep learning become successful [Krizhevsky et al., 2012]. At this point, researchers started dreaming about few-shot learning, methods that would achieve high accuracy by training on just a few examples. 8.1.2 Achieving Few-shot Learning In machine learning, some algorithms are better suited for certain types of problems than others. Furthermore, most training algorithms have elements that can be tuned for performance, such as the learning rate, momentum, the initialization scheme of the parameters, or, for a multi-level algorithm, even the choice of the underlying train- ing algorithm. These elements are called hyperparameters, and they govern how the algorithm learns. Would we be able to learn them? Different types of problems require different hyperparameter settings. For example, for a smooth dataset, a high learning rate may work well, or a local search algorithm, but for a peaky dataset a lower learning rate works better, or a global search algorithm. The point is that the best hyperparameter configuration in machine learning depends on the task to be solved, and that, crucially, this configuration can be learned. Learn- ing such hyperparameters of the algorithm for different tasks is called metalearning. (In contrast, the learning of the regular parameters of the model is now called base learning.) The aim of metalearning is to create algorithms that learn certain tasks faster. In- stead of using a generic algorithm that is not adapted to the task, starting from scratch, we should use a learning algorithm whose hyperparameters are first matched to the type of task to be solved. The metalearning approach is to first learn from a set of related tasks how to configure the learning algorithm, learning to learn a certain set of tasks faster. When we have learned the hyperparameter configuration for such an adapted algo- rithm, we may hope to be able to achieve few-shot training: being able to train on just a few examples, while still achieving high accuracy on examples from a new test task. Metalearning achieves few-shot learning by starting the training not from scratch, but from related knowledge. Metalearning consists of a (hyper) training loop around a (base) training loop. In classical (non-neural network) machine learning, common hyperparameters are the learning rate, or algorithm selection, where different learning algorithms are tested on training tasks [Brazdil et al., 2022]. In deep learning, the initialization of the neural 8.2. INNER/OUTER LOOP183 network weights is often used [Finn et al., 2017]. In agentic LLM, we have also encountered learning at different levels (and we will continue to discuss multi-level learning in the next chapter). It is useful for us to study the theory and insights of the metalearning literature. 8.2 Inner/Outer Loop As any supervised learning approach, metalearning consists of (1) a meta-train phase, training on a variety of related tasks, and (2) a meta-test phase, testing on a new, un- seen, task, to see how well it generalizes to this new task. The goal is to improve generalization by learning a hyperparameter setting of the training algorithm that work well on a wide variety of training tasks. The process of training the parameters of a model is implemented as an optimiza- tion loop, in which a loss-function is used to compute a difference between training examples and current model-estimates. The training algorithm loops over the dataset, each time adjusting the parameters of the model, with the goal to minimize the sum of the loss. The learning-to-learn concept consists of two loops: an inner training loop to op- timize the model’s parameters, and an outer training loop to optimize the algorithm’s hyperparameters. The inner loop optimizes parameters over the examples of one task, the outer loop optimizes hyperparameters over the set of tasks of an application domain. Each base-learning taskT 푖 has a dataset 퐷 푖 and a learning objective, the loss func- tion L 푖 . Thus T 푖 = (퐷 푖 ,L 푖 ). Each dataset consists of pairs of input-examples and labels 퐷 푖 =(푥 푗 , 푦 푗 ), and is split into a training and a testset 퐷 푖 =퐷 푖,푡푟푎푖푛 , 퐷 푖,푡푒푠푡 . On each training dataset a parameterized model ˆ 푓 휃 푖 (퐷 푖,푡푟푎푖푛 ) is estimated, with a loss functionL 푖 (휃 푖 , 퐷 푖,푡푟푎푖푛 ). The model ˆ 푓 is approximated with an algorithm that is gov- erned by a set of hyperparameters 휔. The particular hyperparameters vary from algo- rithm to algorithm, but frequently encountered hyperparameters are the learning rate 훼, the initial parameters 휃 0 , and algorithm constants. The inner loop optimizes the base-parameters 휃, and the outer loop optimizes the hyperparameters 휔 to find the best performance on the set of base tasks with the ap- propriate test data: 휔 ★ = arg min 휔 L | z outer loop (arg min 휃 푖 L 휔 (휃 푖 , 퐷 푖,푡푟푎푖푛 ) | z inner loop ). The inner loop optimizes 휃 푖 within the datasets 퐷 푖 of the tasksT 푖 , and the outer loop optimizes 휔 across the tasks and datasets. The outcome of the meta optimization is a set of optimal hyperparameters 휔 ★ . After metalearning has finished, we have a hyperparameter set (tuned algorithm) that works well for a set of tasks in an application domain, and we can assume that it will generalize well, after training on just a few examples of a new task, scoring high accuracy on testing an unseen example. 184CHAPTER 8. METALEARNING Transfer Learning There is an important connection between metalearning and trans- fer learning. Transfer learning pretrains the neural network on a single task, and then performs a transfer of the weights to a quick finetuning phase. Transfer learning can thus be regarded as a single iteration of a network-initialization-based metalearning approach: transfer learning is metalearning with just one base learning task. 8.3 LLM and Metalearning In the field of LLM, we encounter metalearning in many places. A common metalearn- ing approach is to initialize the model parameters to earlier learned knowledge, so that they may train faster on a related task. This is transfer learning, that we find in ev- ery standard LLM training pipeline, in the pretrain/finetune phases, where we take the pretrained parameters and finetune them further for a specific task. Metalearning is repeated finetuning, and large language models are a prime example of a metalearning training pipeline that achieves that major goal of machine learning: few-shot learning. In-context learning was one of the major breakthroughs of machine learning [Brown et al., 2020]. When we go to agentic LLM, few-shot learning occurs in many other places as well. We will discuss both static and dynamic forms of metalearning. In static metalearning, weights in inner loops are not influenced by other loops. In dynamic metalearning, they are. 8.3.1 Static LLM Metalearning We start by reviewing some examples of static LLM metalearning. 8.3.1.1 Pretraining Pipeline The regular LLM training pipeline is a good example of an elaborate, multi-phase, transfer learning scheme. It achieves good zero-shot performance, in the sense that it can answer many regular queries accurately—where was Mozart born?—based on the pretrained and finetuned knowledge. The different phases in the pipeline are the inner optimization loops, and the choices of which weights to transfer, and the manual choices of how to finetune in the subse- quent phases, can be considered as the outer loop hyperparameter choices. This is a static form of metalearning, since the weights, once trained in a phase, are not influenced by training results of later phases. Weights are stable between phases of the pipeline. The pipeline is a purely sequential training process. 8.3.1.2 In-Context Learning Going beyond zero-shot learning, in-context learning is a form of few-shot learning where a few training examples are provided by the user, prompting the LLM to infer a better answer. The pretrained model is the inner loop, the examples in the prompt guide the outer loop. 8.3. LLM AND METALEARNING185 This is a static form of metalearning, since the values of the weights of the model are not influenced by the examples in the prompt. 8.3.1.3 Chain of Thought Regular in-context learning provides facts as few-shot examples; it is content-focused. Chain of thought provides process-steps as few-shot examples, it is process-focused. Again, the pretrained model is the inner loop, the examples in the prompt guide the outer loop, and the weights of the model are static. 8.3.1.4 Static Metalearning is Stable The regular LLM pipeline consists of transfer learning phases (inner loops), that run in sequence. Once an inner training loop has converged, that phase is not run again. The pipeline is a form of static metalearning. (The results of finetuning are not fed back into the pretrainer again, nor is the pretrainer ran with adjusted hyperparameters based on the finetuning results. The different training phases can not interfere.) In contrast to static metalearning, there is also dynamic metalearning, where differ- ent iterations do interfere. 8.3.2 Dynamic LLM Metalearning In agentic LLMs different forms of dynamic metalearning are used. In dynamic met- alearning the iterations are run concurrently, and can have destabilizing effects on each other’s convergence process. In previous chapters, we have discussed self-reflection, and RLVR. These are ap- proaches where an inner loop can be changed by an outer loop, and results of one inner loop may affect another inner loop. We have created a moving training target. This may cause unstable divergence of the training process. Training results may also ap- pear brittle: they react greatly to small changes in input values or hyperparameters, such as different random seeds, different prompt reformulations, or even changes in the length of the metalearning process itself. Furthermore, hallucinating tendencies of the base LLM may be amplified when the same model is used in different roles, such as for correction and for generation. Let us discuss the examples of dynamic metalearning in agentic LLM. 8.3.2.1 Self-Reflection In self-reflection (Section 6.2.4.2) an agentic harness performs prompt management, calling the LLM in different roles, such as, to evaluate a state, and to refine an answer. In metalearning terms, the model is the inner training loop, and the harness the outer loop. Although the model itself is static, it is called with different prompts, and the answers of the model are used by the harness to produce new prompts. Therefore, the prompt results (outer loops) start interfering, and may interact in non-linear and non-intuitive ways. Furthermore, when the same model is used for different roles, imperfections (bi- ases, hallucinations) may be amplified. 186CHAPTER 8. METALEARNING 8.3.2.2 RLVR Reinforcement learning with verifiable rewards (RLVR, Section 6.2.4.3) is a popular training scheme for code and math applications, where inference-time reasoning results are used for finetuning at training-time. In metalearning terms, finetuning is the inner loop, and inference-time reasoning is the outer loop. RLVR uses outer loop results in the inner loop, feeding inference results into the training phase, mixing two learning processes. Care should be taken to prevent that inference time errors, which are used as truths, are amplified in the finetuning/inference loop. 8.4 Stabilizing Dynamic Metalearning Unstable training occurs when inner/outer interference is under-damped. When this happens, oscillation between inner loops and outer loops may grow larger, instead of dying out. This problem can be mitigated by increased damping, or by decoupling the time by which different inner loops move. For example, the learning rates of one of the loops may be reduced, or the parameters of some of the inner loops may be kept stable for a longer time than the other loops. Another idea is to use a mixture of model types or sizes, such as in Mixture-of-Experts [Jiang et al., 2024]. A student/teacher setup deliberately uses a stronger and a weaker model for the different roles. By using different models, we may hope to dampen hallucinations and interference between inner loops. In general, when metalearning misbehaves, we should aim to create more diversity, to stimulate exploration, and to prevent greedy behavior that leads to local optima or divergent training. These solutions often require much experimentation to build up intuition about the problem. Debugging agentic LLMs can be challenging. 8.5 Concluding Remarks As we have seen, we can understand LLM-training as a form of metalearning. Regular LLM training is a form of transfer learning, and inference-time reasoning and training, as is used in agentic LLM, use even more metalearning ideas. Although dynamic metalearning approaches (such as self-reflection) are a powerful concept, care should be taken since interference between inner and outer loops may destabilize the training process. This may happen unexpectedly, in agents that were previously behaving. When we use a trained model in our agent that interacts with the world, and that agent is also trained using feedback from this interaction, then the model can be re- garded as the base learner (inner loop) and the interaction with the world can be re- garded as the metalearner (outer loop). All the preceding caveats for in-pipeline met- alearning also apply for out-of-pipeline metalearning. Only now, when interacting with the real world, the consequence of errors may be more severe. Related fields of online machine learning, such as deep reinforcement learning, also suffer from the unstable training and the moving target problem. The field of deep BIBLIOGRAPHY187 reinforcement learning has introduced many stabilization techniques, such as replay buffers and target networks [Mnih et al., 2015, Hessel et al., 2018]. These methods typically try to decouple the periodicity of the two learning processes, often by slowing down one of them. A similar approach is introduced in Hierarchical Reasoning Models (HRM) [Wang et al., 2025]. HRM consists of a two-layer metalearning approach. Here, too, the learning is stabilized by learning at two different speeds. Knowledge of such stabilization methods is useful when designing agentic LLM solutions. In the next part we will provide an overview of the different interaction styles by which agentic LLM interact with the world. The styles go from static to dynamic. Bibliography Pavel Brazdil, Jan N Van Rijn, Carlos Soares, and Joaquin Vanschoren. Metalearning: Applications to automated machine learning and data mining. Springer Nature, 2022. 182 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 184 Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017. 183 Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 187 Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta- learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149–5169, 2021. 182 Mike Huisman, Jan N Van Rijn, and Aske Plaat. A survey of deep meta-learning. Artificial Intelligence Review, 54(6):4483–4541, 2021. 182 Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L ́ elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Th ́ eophile Gervet, Thibaut Lavril, Thomas Wang, Timoth ́ e Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https://arxiv.org/abs/2401.04088. 186 188BIBLIOGRAPHY Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012. 182 Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. 187 J ̈ urgen Schmidhuber. Evolutionary principles in self-referential learning, or on learn- ing how to learn: the meta-meta-... hook.PhD thesis, Technische Universit ̈ at M ̈ unchen, 1987. 182 Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model. arXiv preprint arXiv:2506.21734, 2025. 187 Part I Agents 189 191 We have now discussed the foundations of LLMs and agents. We have seen how language and tokenization works, we have seen how LLMs are trained and finetuned, and we have seen how advanced reasoning models reason and self-reflect. We have looked at different learning modalities, from language, vision, and action. We have also looked at the intricacies of metalearning. We have now arrived at the heart of agentic LLM, where we will review the differ- ent interaction styles of LLM agents. This part consists of one content chapter and one practical chapter. First we will review the five agentic interaction styles that we have seen so far, and we discuss the importance of memory. Then, in the practical chapter, we will discuss a hands-on case study where many of the techniques of this part come together. You will be invited to create a self-reflective LLM agent that can play typical AI-games such as TextWorld and NetHack. 192 Chapter 9 Interaction Styles for Agentic LLMs What You Will Learn In this chapter you will learn about different agentic interaction styles and agentic memory. Specifically, you will be able to: • Recognize three different historic types of agents in AI, and how they connect to current agentic LLMs. • Recognize five different agentic LLM interaction styles. • Understand the importance of memory in agentic LLMs, and recognize different agentic memory architectures. 9.1 Agents in AI Agents play a central role in artificial intelligence, and we will first discuss some of the background of agents in AI. 9.1.1 Agents Act and Adapt in their Environment In artificial intelligence, entities that act in the world are called agents. They are the AI equivalent of humans acting in the world. In the agent concept, AI researchers try to capture what they perceive to be the essence of what humans do—understanding intelligence by building it. Agents are autonomous software systems that act in an environment. They (1) have goals, they (2) perceive their environment, they (3) make decisions, they (4) take actions to achieve their goals, and, most importantly, they (5) learn from experience. Agents should adapt to new information in order to achieve their goals better, or, in reinforcement learning terms, agents improve their action policy. Traditional computer programs are deterministic, they predictably follow pre-set instructions (this makes computers so useful to do things from accounting to control, 193 194CHAPTER 9. INTERACTION STYLES FOR AGENTIC LLMS in which predictability is paramount). LLM agents, in contrast, are less deterministic, and less predictable. They learn, and change their behavior from time to time. They can autonomously reason, plan, and perform tasks using tools to perform simple workflows for users in areas such as coding, customer service, and data analysis. (They also hallucinate, make mistakes, and generally are unpredictable and undependable. In that respect, LLM agents are more like humans than computers.) Agents have been studied for quite some time in artificial intelligence. It is useful to distinguish three types of agents: (1) robotic assistant agents, (2) belief-desire-intention agents, and (3) swarm agents. 9.1.1.1 Robotic Assistants First we discuss agents that are robotic assistants. This view on AI agents is from the field of engineering: agents perform human-like tasks for humans. An agent is a tool to more easily achieve routine tasks. An example is a welding robot in a factory, an auto- pilot in an airplane, or a self-driving car that has to find a safe route in an unpredictable environment. Further examples of such agents are virtual assistants, such as Alexa and Siri, or coding assistants such as CoPilot that help software developers become more produc- tive, or customer service bots that help customers navigate the maze of rules of a com- pany to achieve goals such as filing a complaint or getting a refund. Robotic assistant agents can have significant reasoning abilities [Dorri et al., 2018], and LLMs are in- creasingly used for such assistants [Plaat et al., 2025]. 9.1.1.2 Agents with Beliefs, Desires, and Intentions Another view on AI agents is rooted in the field of cognitive science. Here the goal is for agents to model human cognitive processes, and agents have beliefs, desires and intentions (BDI). Cognitive science studies human thinking processes. AI agents can be used to implement cognitive models, and experiment with models of thinking: is the model able to reason logically, to cooperate with human players in a game of Bridge, to deconstruct a theater play by Shakespeare, to compose music, or to have self-awareness? Cognition may take many subtle forms, such as interpreting sensory input, guiding physical actions, and empathizing with others. Originally, BDI agents come from the tradition of logic reasoning. They are imbued with goals, which they pursue through actions. The literature in this field sees actions as a way to achieve the goals of an agent [Rao et al., 1995, Georgeff et al., 1998]. 9.1.1.3 Swarm Agents A third view on AI agents stresses interaction patterns between large numbers of small agents. Where the previous multi-agent systems consisted of few, complex, agents that act and think—agents that could assist, or agents with beliefs, desires and intentions (BDI)—this approach studies flocks and swarms of many, simple, agents. Now the em- phasis is on emergent collaborative behavior, behavior that emerges at a scale beyond 9.1. AGENTS IN AI195 the reasoning of the agents themselves. Can a swarm offer protection against a preda- tor? What are the prerequisites for collaboration to occur? When is a society peaceful, when is it at war? What is a super-organism [Johnson and Linksvayer, 2010, Wilson and Sober, 1989]? Swarm agents come from the connectionist tradition of embodied AI [Brooks, 1990]. The agents communicate, and researchers are often interested in the emer- gence of large scale behavioral patterns such as cooperation and communication. A well-known swarm algorithm is ant colony optimization [Dorigo et al., 2007], and a well-known software environment to study such patterns is Netlogo [Wilensky and Rand, 2015]. 1 Multi-agent systems and agentic LLMs are suitable for the study of all three types of approaches: agents as tools, agents as cognitive entities, and agents that help us study how to achieve peaceful societies (or how to conquer another). Part IV of this book is dedicated to agents as tools, and Part V is dedicated to cognition and agent societies. 9.1.2 Agentic LLMs as Reinforcement Learning Agents The study of the behavior of intelligent agents has a long tradition in AI. A central aspect of agent behavior is how agents learn in their environment. Mechanisms by which agents learn from feedback are studied in reinforcement learning [Sutton and Barto, 2018] and multi-agent reinforcement learning [Wong et al., 2021]. In the next section we will discuss five interaction styles that we see in agentic LLM. As preparation for this overview, we will provide a formal basis for agents that interact in the world using the language of reinforcement learning. Figure 9.1 shows a reinforcement learning system that consists of an agent and an environment. In this system, the agent interacts with the environment through actions, and the environment responds with the new state of the system and a reward value, indicating how good the new state is. The agent can learn from these reward values, and note which actions are favorable and which are unfavorable in each state. The goal of the agent is to find the optimal behavior for the environment, and thus, the best actions in each state. The agent uses the rewards to reinforce its notion of the best actions in each state, and thus learn a function of optimal behavior. This function, mapping each state to an action, is called the behavior policy, denoted as 휋. The optimal policy is denoted as 휋 ∗ . If the agent learns through deep learning, then the policy is parameterized by the weights 휃 of a neural network. The world in which the agent operates is represented by the environment. The agent takes actions and changes the state of the environment. The environment has a probabilistic transition function that computes the new state whenever an action has been performed by the agent. The environment also computes a corresponding reward for the new state. Thus, the environment is modeled with two functions: a transition function 푇 and a reward function 푅. 1 https://w.netlogo.org 196CHAPTER 9. INTERACTION STYLES FOR AGENTIC LLMS Environment: 푇(푠,푎) → 푠 ′ , 푅(푠 ′ ) → 푟 Agent: 푟 improves 휋 휃 (푠) → 푎 (푠 ′ ,푟) 푎 Figure 9.1: The reinforcement learning loop where the agent learns a parameterized policy 휋 휃 from feedback rewards 푟 from the environment’s transition 푇 and reward function 푅, upon the agent’s actions 푎 [Sutton and Barto, 2018] Environment: 푇(푠,푎) → 푠 ′ , 푅(푠 ′ ) → 푟 Agent: 휋 퐿푀 푃휋 (푠,푟) → 푎 (푠 ′ ,푟) 푎 Figure 9.2: Reinforcement learning where the agent uses an LLM to parameterize policy 휋 퐿푀 푃휋 to use feedback 푟 from the environment to choose its actions 푎 This model, of an agent acting in an environment, receiving feedback to learn which actions caused good state transitions, and which did not, suits the concept of agentic LLMs quite well, as we will see in the next section. From the reinforcement learning point of view, we can use the LLM as a function in the agent: 퐿푀(푝푟표푚푝푡) → 푎푛푠푤푒푟 . The LLM can be used in different places in reinforcement learning: in the agent, or in the environment. For example, the agent can use the LLM as policy-improver. It can call an LLM with a certain prompt for the policy function 휋 퐿푀 푃휋 (푠,푟) that learns from environment feedback which actions to take in the different states that the environment can be in (see Figure 9.2). The LLM is given a prompt 푃휋 that tells it to act as a policy improver using reward 푟 , and returns an action 푎 for state 푠. As we will see in Section 20.2.1, in-context reinforcement learning (ICRL) uses the LLM as a policy improver, and tree of thoughts can use an LLM to generate the successor actions. Environment: 푇 퐿푀 푃푇 (푠,푎) → 푠 ′ , 푅 퐿푀 푃푅 (푠 ′ ) → 푟 Agent: 휋 퐿푀 푃휋 (푠,푟) → 푎 (푠 ′ ,푟) 푎 Figure 9.3: Reinforcement learning where both the agent and the environment use LLMs for 휋,푇 , and 푅. The LLMs are each given a prompt 푃푥 to return an action, state, or reward value 9.2. LEARNING MECHANISMS IN AGENTIC LLMS197 Agent typePromptLLMExample A: Queryzero-shot user-writtenpretrainedGPT-2 [Vaswani et al., 2017] query or instruction B: In-Context Learningfew-shotpretrainedGPT-3 [Brown et al., 2020] (supervised example)examples C: In-Context Learningchain-of-thought-stepspretrainedChain of Thought [Wei et al., 2022] (reasoning-steps)exampleZero-shot CoT [Kojima et al., 2022] feedback harnessTree of Thoughts [Yao et al., 2024] D: Self-Reflection prompt optimizationpretrainedLLaMea [van Stein and B ̈ ack, 2024] search or RLReAct [Yao et al., 2022] E: Reinforcement FinetuningRL promptpretraining/RLVR [Lambert et al., 2024] optimization harnessfinetuning Table 9.1: Five Agent Styles The LLM can also be used in the environment, to compute the reward of the state 퐿푀 푅(푠) , or to compute the next state 퐿푀 푇(푠,푎) , see Figure 9.3. In the figure, the same LLM is now used for three different roles, with three different prompts: to per- form the policy, the transition, and the reward function. For example, tree of thoughts can use an LLM to evaluate the reward of a state. There are many options in designing our agentic LLMs, as we will see next. We use the reinforcement learning paradigm to discuss and illustrate the different agentic styles. 9.2 Learning Mechanisms in Agentic LLMs The field of agentic LLM is new, and the previous chapters have shown many different approaches in which LLMs have been used in an agentic setting. In this section we will review the approaches that we have seen so far in this book. We distinguish the approaches based on their prompting method and their training method. We will describe five styles of interaction that fit agentic LLMs. Table 9.1 sum- marizes the five agent styles. We focus on the way that the LLM is prompted by the harness at inference time, and how it is trained. All prompting styles can yield useful agentic assistants and interaction partners, that act in realistic ways, for certain do- mains. The more elaborate prompting styles have been developed in the pursuit for better performance in more challenging domains. The first column of the table gives the name of the agentic interaction style, from query to reinforcement finetuning. The second column lists the type of prompting of the styles, from a simple static instruction prompt without any examples, to a dynamic reinforcement learning prompt. The third column shows the training method of the LLM, whether it is pretrained or finetuned or both. The fourth column gives a concrete example of the interaction style (with references to the literature). 9.2.1 Agentic Interaction Styles We will now discuss the interaction styles. In the first three agent styles the prompt of the LLM is static. The prompt is written by the user and does not change. 198CHAPTER 9. INTERACTION STYLES FOR AGENTIC LLMS Query LLM Answer Figure 9.4: Style A: Static Query Query + ex1, ex2, ex3 LLM Answer Figure 9.5: Style B: In Context Supervised Learning with a prompt based on few ex- amples 9.2.1.1 A. Query Prompt The first interaction style is interacting with the LLM by giving it a simple instruction. Figure 9.4 shows this interaction style. It simply consists of an instruction written by the user. For example, Lily missed the train. She felt very ... to which the model hope- fully responds with something like angry, or sad. The prompt is purely a query, without any training examples. The learning style is zero-shot learning, instruction only. The LLM is able to correctly answer many such queries because it has been pretrained on a large corpus of text, which may provide an answer to many of the queries that users may ask. The model has also been finetuned for instruction answering. Examples of this interaction style are the early LLMs, such as GPT-2 [Vaswani et al., 2017]. 9.2.1.2 B. In-Context Learning Next, we turn to in-context learning. For machine learning, in-context learning is an exciting new type of learning, in addition to supervised and reinforcement learning. In a large language model in-context learning often achieves good results with just a few examples, achieving one of the main challenges of the field of machine learning. Compared to interaction style A, in-context few-shot learning is a more involved interaction style. In-context learning occurs when the user provides a few examples along with the query. In doing so, the user conditions the decoder model to activate the correct part of the large amount of knowledge that is represented in its parameters. Figure 9.5 shows an illustration of a prompt that fits this interaction style. An example from the literature of this interaction style is GPT-3. The paper that introduced in-context learning is Brown et al. [2020]. An example of a few-shot prompt is: Fill in the dots: the sea-la mer, oysters-huitre, the window-la fenetre, cheese-... to which the model hopefully responds with fromage. Note that in-context learning only emerges when the model is large enough. Orig- inally, in-context learning emerged in models with around 100 billion parameters. At that size, the model was rich enough for the few prompt examples to activate the right parts of the model. As our LLM methods improve, this boundary keeps shrinking. 9.2. LEARNING MECHANISMS IN AGENTIC LLMS199 Query + step1, step2, step3 = ex1 LLM Answer Query + ”Let’s think step by step” = ex1 LLM Answer Figure 9.6: Style C: Top: Chain of Thought: In-Context Learning with an LLM based on examples of the reasoning steps; Bottom: Zero-shot Chain of Thought with an instruction that often works 9.2.1.3 C. Chain of Thought Next, we turn to another famous method, chain of thought learning [Wei et al., 2022], Figure 9.6, top. This method was developed, as we saw in Chapter 6, to address the weakness of LLMs in solving simple math word reasoning problems. A disadvantage of the original chain of thought approach is that a new prompt has to be written by the user for each new reasoning task at hand. A subsequent development was the zero-shot chain of thought prompt, Figure 9.6, bottom. Kojima et al. [2022] found that adding the generic sentence: Let’s think step by step to prompts already improves performance on many benchmarks, although often not by as much as a hand-written task-specific prompt. As in the previous interaction style, chain of thought is an in-context few- shot learning method, although now the examples in the prompt are a few examples of the steps that the model should take before it provides its answer—whereas previously they were examples of facts. These steps help the model arrive at the right answer, following a multi-step reasoning chain. The examples are not examples of answers to activate parts of the model, but of steps on how to arrive at the answer. Example prompts for few-shot and zero-shot chain of thought have been shown in Figure 6.1 on page 129. Again, as with in-context learning, chain of thought only works in models of suffi- cient size, and again, this size is getting smaller as our understanding of LLM methods increases and models are being finetuned for chain of thought. Furthermore, it is possi- ble to transfer the learned reasoning methods from a large model to a model with orders of magnitude fewer parameters, by distillation [Li et al., 2023]. 9.2.1.4 D. Self-Reflection Harness The preceding agentic interaction styles all used a single static prompt, achieving en- couraging performance on many benchmarks. To perform well on planning problems, or other problems where state must be tracked, a reasoning style is needed where prompts are optimized, searching through options in a tree of possibilities. We will now discuss two interaction styles where the LLM prompt is dynamically manipulated by a small algorithm that is external to the LLM. This algorithm is called a harness. The first style is in-context (LLM parameters are not changed), while the second style (in the next subsection) also uses finetuning of the model. These interaction styles are dynamic. As we saw in the previous chapter on met- alearning, a challenge of dynamic methods is to achieve a stable learning process. 200CHAPTER 9. INTERACTION STYLES FOR AGENTIC LLMS Initial Problem 푠 퐿푀 푃푔푒푛푒푟푎푡푒 (푠) 퐿푀 푃푒푣푎푙 (푠 1 )퐿푀 푃푒푣푎푙 (푠 2 )퐿푀 푃푒푣푎푙 (푠 3 ) 퐿푀 푃휋 (푠, history) → 푎 퐿푀 푃푒푣푎푙 (푠,푎) → (푠 ′ ,푎,푟) history∪(푠 ′ ,푎,푟) (푠,푎) Figure 9.7: Style D: Self-reflection with an LLM; the prompt is being optimized in context with answers from an LLM, first to generate subproblems, then to evaluate them; controlled by search (top), or reinforcement learning (bottom) When training targets are changing, learning algorithms may diverge, and extra care is needed to achieve convergence. Achieving predictable high performance with self- reflection is still an area of active research. The first dynamic prompting methods that we discuss emerged out of the need to model more complex reasoning structures, that go beyond the straight line sequence of steps of chain of thought. First, simple search algorithms were introduced (tree of thoughts). Next, full fledged dynamic prompting methods were introduced to manage prompt development (ReAct, reflexion, self refine). Often, the harness of these meth- ods introduced short-term and long-term memories to remember state between LLM invocations. An interesting aspect of these methods is that the LLM is used in different roles, for which it is called with different types of prompts. The original role of the LLM in chain of thought tasks is to evaluate the answer in each reasoning step. This evaluation prompt is now augmented with a different type of prompt, to generate a new reasoning step. Sometimes a third type of prompt is added, whose task is to improve the prompt itself. Thus, a single model is used to self-reflect on itself through different prompts, managed by an external harness. Figure 9.7 shows these ideas. At the top, the self-reflection is controlled by a stan- 9.2. LEARNING MECHANISMS IN AGENTIC LLMS201 퐿푀 휓 휋/퐶표푇 (푠) → 푎 퐿푀 휓 푒푣푎푙 (푠,푎) → (푠 ′ ,푎,푟) (푠 ′ ,푎,푟)(푠,푎) Figure 9.8: Style E: Reinforcement Finetuning of the LLM parameters 퐿푀 휓 in addi- tion to reinforcement learning of the policy 휋 parameters 휋 휃 [Luong et al., 2024]. dard tree search algorithm, as in tree of thoughts [Yao et al., 2024]. At the bottom, a reinforcement learning loop controls self-reflection, where the prompt for policy 휋 is updated with the history of (state, action, reward) triples (see also Section 20.2.1). 9.2.1.5 E. Reinforcement Finetuning Harness Up to now, all learning was in-context, the model provided answers based on examples provided in the context. The parameters of the LLM were not changed. Although ICL is easy to use, it limits learning. If we wish the model to adapt more profoundly to a certain environment, if we wish the model to remember what it learned at inference time, then it helps if the parameters can change. We call approaches that use this method reinforcement finetuning. RLVR is a well-known example. Figure 9.8 provides an illustration, where the trainable model parameters are indi- cated as 휓. Note that the only difference with Figure 9.7 (bottom) is the introduction of the 휓 parameters. In ReFT the LLM parameters 휓 change, they are finetuned based on the rewards from the dataset. We should interpret the figures such that where learning at inference time was in-context, this has now changed, since we allow finetuning of the parameters 휓. In the traditional LLM training pipeline, finetuning is a stage that comes before inference. However, for many important applications, such as math reasoning or coding, chain of thought multi-step inference-time roll-outs produce accurate rewards that can be used for finetuning. Therefore, in reinforcement finetuning, the model is used to produce training targets at inference-time, that are then used to finetune the parameters of the same model—finetuning at inference-time. Reinforcement finetun- ing blurs the line between inference and finetuning, where supervised finetuning has a clear separation between training and inference. In LLM training there are three main training/inference moments: (1) pretrain- ing, (2) finetuning, (3) testing (inference). Self-reflection (Style D) does reinforcement learning at inference time. In style E, reinforcement finetuning, we do not do reinforce- ment learning at test time (3), but at finetune time (2). In agentic LLM, this interaction style allows the LLM to learn from agent interaction with the world. For the dynamic agentic styles D and E, care must be taken to achieve stable con- vergence. As of this writing, few works have been published that report on such results. 202CHAPTER 9. INTERACTION STYLES FOR AGENTIC LLMS 9.3 Memory Architectures for Agentic LLMs Humans (and other forms of life) remember. Our memory enables us to have an iden- tity. We can remember our name, our friends, and our family members. Memory allows us to learn what is good an what is bad for us, and it allows us to adapt our behavior. When we suffer memory-loss, either through injury, illness, or old age, our personality changes. Severe memory-loss leads to loss of identity, we wander aimlessly through the world. Agents also need memory. In this section we will discuss why agentic LLM needs memory, and we provide a brief overview of the different types of memory that have been developed for agentic LLM. 9.3.1 The Need for Memory A statistical model 푓(푥) → ˆ푦 estimates an output ˆ푦 based on the relation 푓 that it has learned for its input 푥 at training time. Such functional behavior is memory-free. Any new information (post-training) must be provided to the model through the input parameters 푥. LLMs are no different, and even the earliest chatbots were provided with special hidden system prompts that fed the model a history of the conversation, so that it is able to hold a coherent conversation. As the use cases for agentic LLM increase, more advanced techniques are needed, beyond the system-prompt method. To better understand what memory is used for in agentic LLMs, we will now discuss different use cases for memory. 9.3.1.1 Coherent Conversation Many machine learning benchmarks for LLMs are of the challenge/response type, and they are well suited for the functional input/output behavior of statistical models. The main real world use case for LLMs, however, is as a chat-bot, where the LLM is ex- pected to engage in a meaningful conversation with the user. In order to maintain a coherent conversation, in which conversational partners react and respond to each other, chat-bots must be able to know what has been said before. Thus, a first requirement for agentic LLMs is to be able know what has been said before in a conversation. 9.3.1.2 Adaptation According to our definition in Chapter 1, agentic LLMs are expected to learn. When a robot performs a task in a kitchen, such as chopping an onion, and the location of the knife has moved, then it should adapt its actions: it should now first pick up the knife from its new location before starting to chop. Agents interact with the environment, and if the environment changes, then agents should adapt. Either our agent’s pretrained policies must be highly general, and able to adapt to all new situations in a certain domain, or our agents architecture must flexible, and recognize when a pretrained policy requires adaptation, finetuning or even re-training. For this, they need memory. 9.3. MEMORY ARCHITECTURES FOR AGENTIC LLMS203 As we have seen in Section 6.2.4, the external prompt improvement harnesses that are used in self-reflective agent architectures require buffers and other memory types for explicit prompt improvement. 9.3.1.3 Identity Our experiences shape our personality and our identity. When LLM agents are to build up their own identity, the experiences must either be collected and held in a memory, to be used for self-reflection, or they must be processed and used in other ways to adapt their behavior policy. As the memory architectures of agentic harnesses improve, agents will be able to better remember their experiences. When that happens, we may find that our agents can acquire, through their agentic experiences, a unique personality. 9.3.1.4 Continual Learning Traditional machine learning trains a model on a particular dataset for a particular task. Human learning is different, our learning evolves continuously throughout our lives. In machine learning, continual learning studies how models can be trained for a sequence of new tasks while preserving previously learned tasks [Wang et al., 2024, Li et al., 2025a]. Like metalearning, it studies generalization across related tasks. Major challenges in continual learning are to overcome catastrophic forgetting, and how to cope with changing data distributions. The field of continual learning provides a variety of architectures that support this type of training. 9.3.2 Memory Architectures The field of agentic LLM is young. Only recently are memory architectures for LLM agents starting to receive attention. We will discuss the different approaches that have so far appeared. 9.3.2.1 Finetuning A pragmatic way to deal with the need for memory is to incorporate new informa- tion into the model as it arrives. Instead of developing external memory management algorithms, we could take advantage of the fact that model parameters are a form of memory, and finetune the model as the new examples arrive. To achieve this approach in an agentic setting, the finetuning must happen quickly, before the user can perceive a lag. Parameter efficient finetuning methods such as LoRA are an option [Yang et al., 2024]. Although this is an elegant approach that does not require external algorithms or memory structures, for most applications the finetuning will not happen quickly enough, and an in-context or harness solution should be developed. 204CHAPTER 9. INTERACTION STYLES FOR AGENTIC LLMS 9.3.2.2 Long Context Windows and Compression A second approach is to perform prompt management. The standard approach of chat- bots is to provide a truncated form of the chat-history to the LLM by injecting it into the prompt in addition to the user’s text. This is an approach that works quite well in practice. A limitation is that context windows are limited in size, and processing large contexts is expensive because of the squared time complexity of the attention mechanism. Two types of solutions have been pursued. First, specific long-context architec- tures have been developed, that keep track of attention in different ways [Smith et al., 2022]. This is an active field of research, a survey of recent approaches is Liu et al. [2025]. Second, compression schemes have been developed, to semantically compress the prompt into a smaller stream. A survey is Li et al. [2025b]. Prompts can be divided into chunks [Sheng et al., 2025], to allow processing of large prompts. A related ap- proach uses the LLM to generate code for processing long prompts recursively [Zhang et al., 2025a]. Prompts can also be compressed through filtering of unnecessary tokens (this sen- tence could become Prompts compressed filtering unnecessary tokens). Another ap- proach is soft prompts [Lester et al., 2021]. Soft prompts eliminate discrete tokens, but learn embeddings (via backpropagation) that can be fed directly into the transformer, without mapping them to actual language tokens. This context can serve as an alter- native to a plain-text context. Impressive compression rates have been reported, from ten-fold [Mu et al., 2023] to hundred-fold [Li et al., 2025c]. 9.3.2.3 Explicit Memory Management If we need to store state between LLM calls, then a straightforward solution is for the agent harness to provide this storage, as part of its prompt management. In Sec- tion 6.2.4 we saw how the external prompt-improvement algorithms for self-reflection add different types of storage, such as in the reflexion approach and the buffer of thoughts approach (Figures 6.8 and 6.6). Agents can have different requirements for their storage (see Zhang et al. [2025b], Chhikara et al. [2025] for a survey). We will discuss some of the recent studies of this active field of research. Prompts can be stored in textual or parametric form, and relevant operations are the reading and writing to memory. Furthermore, memory man- agement operations are important, such as for compacting and forgetting information. Also, additional services can be provided by the memory systems. For example, Mem- oryBank [Zhong et al., 2024] adds hierarchical summaries and uses LLM prompts to provide a running personality of the interaction events of a user. Operations to store and retrieve items can be added to an agent. MemGPT [Packer et al., 2023] studies how LLMs can read and write to external memory using function calls (memory as a tool). A-mem [Xu et al., 2025] introduces contextual descriptions of memory, based on a mailbox principle. A disadvantage of the explicit memory management approach is that the LLM has to perform memory management explicitly, without having been trained how to do this effectively. An alternative solution is to manually write memory management code, 9.4. CONCLUDING REMARKS205 when it is possible to foresee all possible requirements. The Model Context Protocol (MCP) is a set of strategies designed for context management. Section 11.4 describes the protocol in more depth. Learning to use memory Mem1 [Zhou et al., 2025] uses a different approach: af- ter each LLM interaction the agent updates a compact state to be shared with future LLM calls. The memory size required is constant, so that the state does not run out of the context window. The shared state is created by inserting reasoning-style tags into the prompt, and is trained using reinforcement learning to incentivize state consolida- tion. Mem-훼 is another approach that uses reinforcement learning to train agents to effectively use memory [Wang et al., 2025]. AgeMem [Yu et al., 2026] integrates memory management operations into agent’s policy for end-to-end optimization. It distinguishes long term and short term memory, and uses a three-stage reinforcement learning scheme to train the agent in the use of the two types of memory. This, too, is an active field of research, new studies and frameworks can be ex- pected. See, for an overview, Zhou et al. [2026]. Vector Databases In addition to the storage of prompts as human-readable text, we can also store their tokenized embeddings directly. Storing prompts as text is useful for retrieving data when an exact textual match is needed. Embeddings encode semantic information—which tokens are close to each other in meaning. Storing embeddings is therefore useful when we would like to retrieve items that are semantically close. Embeddings can be stored in vector databases. Traditional relational databases store data in rows and columns. Vector databases store high-dimensional lists of num- bers, and provide low-latency access to them, making them well-suited for LLMs [Xie et al., 2023, Ma et al., 2023]. 9.4 Concluding Remarks The concept of an intelligent agent is a central concept in artificial intelligence. LLMs have increased the communication and reasoning capabilities of agents significantly. In this chapter, we have discussed different agent types, and have connected them to different traditions in AI. Agents act. We have discussed 5 main interaction styles based on the preceding chapters. Agents need memory to interact meaningfully with their environments, and we have discussed this new field of research. After covering so much theory, it is high time to get our hands dirty. In the next chapter we will look at a practical case study, to get more experience with using some of the theoretical concepts that we have seen. 206BIBLIOGRAPHY Bibliography Rodney A Brooks. Elephants don’t play chess. Robotics and autonomous systems, 6 (1-2):3–15, 1990. 195 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 197, 198 Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413, 2025. 204 Marco Dorigo, Mauro Birattari, and Thomas Stutzle. Ant colony optimization. IEEE computational intelligence magazine, 1(4):28–39, 2007. 195 Ali Dorri, Salil S Kanhere, and Raja Jurdak. Multi-agent systems: A survey. Ieee Access, 6:28573–28593, 2018. 194 Michael Georgeff, Barney Pell, Martha Pollack, Milind Tambe, and Michael Wooldridge. The belief-desire-intention model of agency. In International work- shop on agent theories, architectures, and languages, pages 1–10. Springer, 1998. 194 Brian R Johnson and Timothy A Linksvayer. Deconstructing the superorganism: social physiology, groundplans, and sociogenomics. The Quarterly Review of Biology, 85 (1):57–79, 2010. 195 Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners. Advances in neural informa- tion processing systems, 35:22199–22213, 2022. 197, 199 Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. 197 Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter- efficient prompt tuning. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 3045–3059, 2021. 204 Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Symbolic chain-of-thought distillation: Small models can also “think” step- by-step. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 2665–2679, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.150. URL https://aclanthology.org/2023.acl-long.150/. 199 BIBLIOGRAPHY207 Yujie Li, Xiangkun Wang, Xin Yang, Marcello Bonsangue, Junbo Zhang, and Tianrui Li. Improving open-world continual learning under the constraints of scarce labeled data. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 1647–1658, 2025a. 203 Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7182–7195, 2025b. 204 Zongqian Li, Yixuan Su, and Nigel Collier. 500xcompressor: Generalized prompt compression for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25081–25091, 2025c. 204 Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407, 2025. 204 Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. arXiv preprint arXiv:2401.08967, 2024. 201 Le Ma, Ran Zhang, Yikun Han, Shirui Yu, Zaitian Wang, Zhiyuan Ning, Jing- han Zhang, Ping Xu, Pengjiang Li, Wei Ju, et al.A comprehensive survey on vector database: Storage and retrieval technique, challenge. arXiv preprint arXiv:2310.11703, 2023. 205 Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist to- kens. Advances in Neural Information Processing Systems, 36:19327–19352, 2023. 204 Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and JosephE Gonzalez. Memgpt: towards llms as operating systems., 2023. 204 Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg. Agentic large language models, a survey. Journal of Artificial Intelligence Research, arXiv preprint arXiv:2503.23037, 84:29:1–29:74, 2025. 194 Anand S Rao, Michael P Georgeff, et al. Bdi agents: From theory to practice. In Icmas, volume 95, pages 312–319, 1995. 194 Boheng Sheng, Jiacheng Yao, Meicong Zhang, and Guoxiu He. Dynamic chunking and selection for reading comprehension of ultra-long context in large language models. arXiv preprint arXiv:2506.00773, 2025. 204 Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933, 2022. 204 208BIBLIOGRAPHY Richard S Sutton and Andrew G Barto. Reinforcement Learning. Adaptive Compu- tation and Machine Learning series. Bradford Books, Cambridge, MA, 2 edition, November 2018. 195, 196 Niki van Stein and Thomas B ̈ ack. Llamea: A large language model evolutionary algo- rithm for automatically generating metaheuristics. arXiv preprint arXiv:2405.20132, 2024. 197 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 197, 198 Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362–5383, 2024. 203 Yu Wang, Ryuichi Takanobu, Zhiqi Liang, Yuzhen Mao, Yuanzhe Hu, Julian McAuley, and Xiaojian Wu. Mem-α: Learning memory construction via reinforcement learning. arXiv preprint arXiv:2509.25911, 2025. 205 Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824– 24837, 2022. 197, 199 U. Wilensky and W. Rand. An Introduction to Agent-Based Modeling: Modeling Natural, Social, and Engineered Complex Systems with NetLogo. The MIT Press. MIT Press, 2015. ISBN 9780262731898. URL https://books.google.nl/ books?id=LQrhBwAAQBAJ. 195 David Sloan Wilson and Elliott Sober. Reviving the superorganism. Journal of theo- retical Biology, 136(3):337–356, 1989. 195 Annie Wong, Thomas B ̈ ack, Anna V Kononova, and Aske Plaat. Multiagent deep reinforcement learning: Challenges and directions towards human-like approaches. arXiv preprint arXiv:2106.15691, 2021. 195 Xingrui Xie, Han Liu, Wenzhe Hou, and Hongbin Huang. A brief survey of vector databases. In 2023 9th international conference on big data and information analyt- ics (BigDIA), pages 364–371. IEEE, 2023. 205 Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A- mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110, 2025. 204 Menglin Yang, Jialin Chen, Jinkai Tao, Yifei Zhang, Jiahong Liu, Jiasheng Zhang, Qiyao Ma, Harshit Verma, Regina Zhang, Min Zhou, et al. Low-rank adaptation for foundation models: A comprehensive review. arXiv preprint arXiv:2501.00365, 2024. 203 BIBLIOGRAPHY209 Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. 197 Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large lan- guage models. Advances in Neural Information Processing Systems, 36, 2024. 197, 201 Yi Yu, Liuyi Yao, Yuexiang Xie, Qingquan Tan, Jiaqi Feng, Yaliang Li, and Libing Wu. Agentic memory: Learning unified long-term and short-term memory management for large language model agents. arXiv preprint arXiv:2601.01885, 2026. 205 Alex L Zhang, Tim Kraska, and Omar Khattab. Recursive language models. arXiv preprint arXiv:2512.24601, 2025a. 204 Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhen- hua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model-based agents. ACM Transactions on Information Systems, 43(6):1–47, 2025b. 204 Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 19724–19731, 2024. 204 Wei Zhou, Xuanhe Zhou, Shaokun Han, Hongming Xu, Guoliang Li, Zhiyu Li, Feiyu Xiong, and Fan Wu. Are we ready for an agent-native memory system?arXiv preprint arXiv:2606.24775, 2026. 205 Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. Mem1: Learning to synergize memory and reasoning for efficient long-horizon agents. arXiv preprint arXiv:2506.15841, 2025. 205 210BIBLIOGRAPHY Chapter 10 Case Study 2: BALROG for LLM Reasoning and Reflection What You Will Learn In this case study you will gain hands-on experience building self-reflective agent for reasoning domains. Specifically, you will be able to: • Run a multi-step reasoning agent on a benchmark of puzzles and games. • Adapt a reasoning agent with your own ideas. • Implement your own self-reflective agent. • Critically reflect on the performance and stability of a learning agent in the do- main of puzzles and games. 10.1 BALROG Overview In this case we will work with an agentic LLM that performs self-reflection. We use a popular benchmark suite for reasoning tasks, called BALROG. 1 BALROG is an acronym for Benchmarking Agentic LLM and VLM Reasoning On Games. The BALROG benchmark consists of a mix of easy and hard games, some of which are dif- ficult for LLMs to play well, as the BALROG paper shows in their own benchmarking study [Paglieri et al., 2025]. Initially, the BALROG games are benchmarked with vanilla LLMs. The code, however, is quite open and amenable to writing your own agent approaches, to see if you can do better. The software is written for you to write your own agent code for in-context (reinforcement) learning. The main BALROG website is https://balrogai.com. The code is easy to adapt and use for your own experiments, and there is a leaderboard. In this chapter we 1 A Balrog is a mythical demon from Middle-earth, featured in the fantasy works of J.R.R. Tolkien. A Balrog is a match even for Gandalf the Grey, the most powerful wizard of all. 211 212CHAPTER 10. CASE STUDY 2: REASONING AND REFLECTION Figure 10.1: List of games in BALROG [Paglieri et al., 2025] will guide you to write your own agent code, that can then be run on the BALROG games-suite. BALROG is designed to challenge in-context agents on reasoning problems. It uses puzzle-style games to benchmark the reasoning capabilities of LLMs and VLMs. Fig- ure 10.1 lists the games. The games are from easy to hard: BabyAI, Crafter, TextWorld, BabaIsAI, MiniHack, and NetHack. The games offer a mix of challenges, some are text-based, some have a visual component, all include reasoning challenges for the model. Some of the games are considered hard. For example, it is not expected that your model or your agent—or your self—will reach a high score on NetHack. (If you do achieve that, without having accessed the cheat-sheets, you may want to consider writing a paper for a prestigious AI conference.) 10.1.1 Puzzles and Games BALROG consists of the six puzzles and games shown in Figure 10.1. The games run from easy grid world puzzles to hard text-based dungeon and dragons strategy games, with long credit assignment episodes and flat reward landscapes. The authors of BALROG provide a code base that allows experimentation with agents, to design your own agent, to see how well they do on the benchmarks. To perform well on the benchmarks in BALROG, the agents must be good at reasoning and planning, and, most likely, be good at adapting and reflecting on their own behavior. BabyAI is a simple grid-world game, where the agent has to solve a navigation task. A simple reinforcement learning type algorithm would be able to easily solve this game. Tasks are given in natural language, such as: go to the blue ball, then pick up the grey key. Of course, the challenge for the agentic LLM is to perform the planning part of the task. It is expected that the LLM will be able to do well on this simple game (see the BALROG paper). NetHack, on the other hand, is a highly challenging game, also for human players. NetHack is a text-based dungeons and dragons-like game. It includes both long-term strategic planning, in the sense that winning a game can involve hundreds of thousands of steps, and short-term tactical fights to fend of the usual hordes of monsters. Credit assignment is a challenge in NetHack, in order to understand which of the actions that the agent took in the past contributed to ultimate success or failure, since actions in one 10.1. BALROG OVERVIEW213 Figure 10.2: Architecture of BALROG [Paglieri et al., 2025] area of the game may have unexpected consequences far away, making it difficult to play for reinforcement learning agents. NetHack is known for human players to take years to master the game without resorting to cheat guides. 10.1.2 BALROG Architecture We wil now have a look at the architecture of the BALROG framework. Figure 10.2 shows the architecture, taken from the paper [Paglieri et al., 2025]. Please familiarize yourself with the BALROG system. Read the paper Paglieri et al. [2025], which you can find on https://arxiv.org/abs/2411.13543, or go to the BALROG website at https://balrogai.com. If you have not done so yet, install BALROG on your system with the git clone command. Go to https://balrogai.com, or directly to the Github repository at https://github.com/balrog-ai/BALROG. Install the repository on your local computer. Decide which LLM you want to use. Please note that typical BALROG agents make many LLM calls, at each step of the way. If you want to run the LLM locally, install it, or use it from Huggingface. If you want to use a (commercial) cloud-based LLM, acquire an API-key and install the API-key in your environment. You may want to consider applying for a free student-tier, if that is offered. The use of frontier LLMs can be expensive, but usually older and smaller LLMs, some of which are still relatively capable, are much cheaper. The BALROG readme on the Github repository contains information on how to in- stall the system, and how to install an API key for your favorite LLM, see Listing 10.1. (In one of the exercises we will use multiple LLMs, to compare their performance. For 214CHAPTER 10. CASE STUDY 2: REASONING AND REFLECTION 1 export OPENAI_API_KEY=<KEY> 2 export ANTHROPIC_API_KEY=<KEY> 3 export GEMINI_API_KEY=<KEY> Listing 10.1: Installing the API key now, we are just trying to get the system to run, and anyone will do.) 10.1.3 Agent Please read the Agent tutorial in BALROG at https://github.com/balrog-ai/ BALROG/blob/main/docs/agents.md. In BALROG, agents are small pieces of Python code that call an LLM with a prompt to provide an answer how to determine the next step. The agent code is responsible for receiving answers from the model (observations) and for selecting the next action. The LLM is used to execute various prompts. BALROG follows interaction style D from Figure 9.7, where the prompt is built up from the history of interactions. 10.1.3.1 Prompt Builder The agent builds a prompt based on previous actions, based on observation/action pairs from previous interactions. Thus, in reinforcement learning terms, the agent performs the function of the policy, selecting the next action to perform based on the incoming observation. Go to the prompt_builder directory to change the way that prompts are con- structed from the history of actions. 10.1.3.2 Agents Agents are responsible for the memory function: managing the observation/action his- tories. The agents must keep a record of past observations and actions to support context-aware decision-making, and this must be implemented by us, external to the LLM. The agent code in BALROG queries the LLM: agents send observations to the LLM, receive responses, and use those responses to decide on actions to take in the game. BALROG comes with a number of pre-written agents, which you should study to design your own. The repository has a naive agent and a chain of thought agent. Also provided are more robust versions for handling LLM responses. When you build your own agent, you will most likely build on the custom agent. The BALROG agent tutorial provides an example for a simple planning agent. 10.2. IMPLEMENTATION STEPS215 1 python eval.py \ 2 agent.type=naive \ 3 agent.max_image_history=0 \ 4 agent.max_text_history=16 \ 5 eval.num_workers=16 \ 6 client.client_name=openai \ 7 client.model_id=gpt-4o-mini-2024-07-18 Listing 10.2: Running the standard evaluation 10.2 Implementation Steps We will now provide some exercises to help you explore building your agents. 10.2.1 Reproduce Paper We start by trying to reproduce the results in the BALROG paper. Please have a look at the BALROG code at Github https://github.com/balrog-ai/BALROG. Read the Evaluation Guide, the Agent Development tutorial, and the instructions how to run Few Shot Learning experiments (in-context learning). Install the relevant tools on your computer, and the right Python version (and put it in your PATH). Check the version by starting the Python interpreter from the command prompt: $ python. First, run the standard evaluation, to make sure you installed things correctly, see Listing 10.2. 10.2.2 Different LLMs To get a feeling for how the standard agent performs on the different benchmarks with different LLMs, it is useful to run a few different LLMs. It is recommended to run closed and open LLMs, and see how they perform. Especially the open version from OpenAI, Claude, DeepSeek, OLMo and Qwen may work well. Running these models is important, and will lead to more insight into the relative strengths of the various models. 10.2.3 Different Puzzles Next, to get a better understanding of the benchmark, you should, after running the LLM agent in the different puzzle problems, have a closer look at the prompts and the answers of LLM. See if you can see where the LLM agent fails, where the weakness of the agent lies. By close inspection of the individual actions, and a deeper understanding of the puzzle, you might be able to think of a way to improve your agent. 10.2.4 Computational Cost BALROG agents call an LLM for each step that the agent makes, often with a long prompt. In these puzzles and games many such steps are made. Running different 216CHAPTER 10. CASE STUDY 2: REASONING AND REFLECTION LLMs on the full benchmark, for different seeds, will easily require many tokens. In your experiments, first try a small and cheap LLM, such as Qwen-3B. Start with the easiest benchmarks (BabyAI and Crafter). If you wish to expand, please go ahead, but be mindful of the computational cost. You want insights, but you also want your experiments to finish in a reasonable time for a reasonable cost. 10.2.5 Implement Own Reasoning Agent Now it is time to implement your own learning agent. Go to the agents directory in bal- rog, and edit the custom.py file. Listing 10.3 reproduces the code for Custom Agent from the BALROG readme. The code can be run with the command in Listing 10.4 (specifying the custom agent, and running gently with fewer workers). Of course, you can specify your own LLM of choice in the command. Start with the provided code, run it, see how it works. Then, you can try to make small improvements. Try making changes to the prompt for the LLM, and to the way in which the Python code handles the learning in the agent, either with the history prompt builder, in the plan, or in the learning algorithm. 10.3 Evaluation The performance of your agent will be reported by BALROG as a progression percent- age. Record the progression numbers for each benchmark, and for the agent types that you try. Do this in a structured way, you may want to write a Python or shell script to automate the running of the experiments and the logging of the results, perhaps also plotting the progression in a graph. Consider using wandb.ai. What agent types work well, and on what benchmark do they work well? Make a matrix in which you log progression for the agent types and the benchmark types that you try. Note that LLM experiments are inherently non-deterministic: two runs with the same prompt may give different results. Therefore, perform multiple runs of each experiment. If you get five identical results in a row, you may start to trust these results. If they vary widely, you may want to investigate. Is the reason purely statistical? Does turning down the temperature solve the problem, or is there another problem? Do not blindly trust the progression numbers that are reported. Since you are us- ing code on a computer, there may be bugs, or other issues. The BALROG system facilitates delving deeper into the runs. It logs csv files that you can look at with a spreadsheet. Have a look at some of the runs, and decide if it makes sense what you see, or if the agents are exhibiting degenerative behavior. If they do, try to find the reason. Did you do something wrong? Is there unexpected interference between the benchmark and the agent behavior? Is the reward landscape too flat, so that there is no signal for the agent to go after, and all it does is random actions running in circles in a way that makes no sense at all? Is there something that you can do about this? Please note that the field of large language models, and especially of agentic LLMs, is new, and our insights and our software frameworks have not yet stabilized. Some unexpected results are to be expected, and some challenges or frustration as well. 10.3. EVALUATION217 1 from balrog.agents.base import BaseAgent 2 import re 3 4 5 class CustomAgent(BaseAgent): 6 def __init__(self, client_factory, prompt_builder): 7 super().__init__(client_factory, prompt_builder) 8 self.client = client_factory() 9 self.plan = None 10 11 def act(self, obs, prev_action=None): 12 if prev_action: 13 self.prompt_builder.update_action(prev_action) 14 self.prompt_builder.update_observation(obs) 15 16 plan_text = f"Current Plan: self.plan " if self.plan else "You have no plan yet. " 17 18 planning_instructions = """ 19 Review the current plan above if present. Decide whether to continue with it or make changes. 20 If you make changes, provide the updated plan. Then, provide the next action to take. 21 You must output an action at every step. 22 Format your answer in the following way: 23 PLAN: <your updated plan if changed, or "No changes to the plan." if the current plan is good> 24 ACTION: <your next action> 25 """.strip() 26 27 messages = self.prompt_builder.get_prompt() 28 if messages and messages[-1].role == "user": 29 messages[-1].content += " " + plan_text + " " + planning_instructions 30 31 response = self.client.generate(messages) 32 33 # Extract the plan and action from the LLM's response 34 plan, action = self._extract_plan_and_action(response.completion) 35 36 # Update the internal plan if it has changed 37 if plan != "No changes to the plan.": 38 self.plan = plan 39 40 # Save the plan in the response.reasoning field and the action in response.completion 41 response = response._replace(reasoning=plan, completion=action) 42 return response 43 44 def _extract_plan_and_action(self, response_text): 45 plan_match = re.search(r"PLAN: * (. * ?)(?= :| )", response_text, re.IGNORECASE | re.DOTALL) 46 action_match = re.search(r"ACTION: * (. * )", response_text, re.IGNORECASE | re.DOTALL) 47 48 plan = plan_match.group(1).strip() if plan_match else "No changes to the plan." 49 action = action_match.group(1).strip() if action_match else None 50 51 return plan, action Listing 10.3: Agent code in BALROG 218CHAPTER 10. CASE STUDY 2: REASONING AND REFLECTION 1 python eval.py \ 2 agent.type=custom \ 3 agent.max_image_history=0 \ 4 agent.max_history=16 \ 5 eval.num_workers=64 \ 6 client.client_name=openai \ 7 client.model_id=gpt-4o-mini-2024-07-18 Listing 10.4: Running the Agent evaluation Figure 10.3: ReAct example Performing Search [Yao et al., 2022] A structured way of experimentation and result-logging is your friend. Planning and organization can make the running, evaluation, and improvement of practical agen- tic LLM experiments an enjoyable exercise. 10.4 Self-Reflection When you were able to reproduce the benchmark results, and have experimented some- what with the agents, you may want to try to take on some larger challenges. In this section, we will try to extend the BALROG agent towards self-reflection, where the LLM is used in different roles, such as evaluation and action selection. Self-reflection, as any adaptive system that chases a target that is itself being optimized, can be prone to instability and oscillations. Therefore, before we try our self-reflective ideas, it is important that the base learner system of the previous subsection works, and converges in a stable and predictable manner. Please go back to that section and make sure things work as you want them to work. There are many recent works that try to improve LLM agents. We will mention two: ReAct [Yao et al., 2022] and reflexion [Shinn et al., 2024]. ReAct is primarily focused on prompts, where reflexion introduces new agent algorithms, including short-term and long-term memory. 10.4. SELF-REFLECTION219 Figure 10.4: Reflexion Architecture [Shinn et al., 2024] ReAct ReAct uses promtps to interact with the world. ReAct promotes the use of tools, to lookup information, see Figure 10.3. The original website for ReAct is https://react-lm.github.io, the code repository is here: https:// github.com/ysymyth/ReAct. ReAct prompting has also been implemented in the LangChain 2 and BeeAI 3 frameworks. Have a look at these framework to get an understanding of the prompt-based agent interaction style that ReAct introduces. Then, implement the prompt-based interaction in BALROG. Does is improve performance? For which games does it help to lookup external information? For which games not? Reflexion Where ReAct is prompt-based, allowing interactive lookups, the reflex- ion approach introduces Python-coded agents, allowing more elaborate agent behavior. Reflexion uses the same LLM in different contexts, with different prompts. It works on the assumption that it is easier to criticize what is wrong with an existing prompt, than to come up with one that works. Reflexion is a popular self-reflection architecture [Shinn et al., 2024]. The main webpage for reflexion is https://github.com/noahshinn/reflexion. A blog post on reflexion, describing the main ideas, is here: https://nanothoughts. substack.com/p/reflecting-on-reflexion. Figure 10.4 shows the general concept of reflexion, the different prompts, and the different types of memory. Figure 10.5 shows the code of reflexion, with the different calls to the LLM in line 3, 5, and 6 (푀 푎 , 푀 푒 , and 푀 푠푟 ) and line 9, 11, 12. Note that the code is pseudo code, with important details about memory management, prompt management, and convergence criterium left open. Have a look at the repository in Github for details. Reflexion uses the same LLM with different prompts, which may lead to learning 2 See https://docs.langchain.com/oss/python/langchain/overview 3 See https://github.com/i-am-bee/beeai-framework/blob/main/python/ examples/agents/react.py 220CHAPTER 10. CASE STUDY 2: REASONING AND REFLECTION Figure 10.5: Reflexion Algorithm [Shinn et al., 2024] instability, and loops. Furthermore, memory management is relatively simple, with the potential of context overruns. An idea is to implement chunking or other memory compression methods, by hand in Python, or using an LLM. First play with the reflexion repository, try to get it to work on the reflexion appli- cation, such as AlfWorld and HotpotQA. Extensions Now, try to implement a reflexion agent in BALROG, to run on the BAL- ROG benchmarks. What problem do you encounter? Does reflexion improve perfor- mance over the standard BALROG agents? Next, extend the problem further. You can try to find other reflection methods in the literature, or try to create new reflection methods from your own. You can try different application types, and you can try to see how your approaches perform. And you can of course try other LLMs. 10.5 Concluding Remarks In this chapter we have discussed a practical case study, to see how an LLM can per- forms self-reflection. We used BALROG, a popular benchmark suite for reasoning tasks. BALROG consists of a mix from easy to hard games, some of which are difficult for LLMs to play well. The BALROG suite is designed in an open style. We can write our own agent code that can then be run on the BALROG games. This has been an important practical case study. You have explored writing our own agent code for in-context reasoning. You have also gone beyond base-learning, by implementing self-reflection. You have explored the stability of self-reflection, on different LLMs and with highly challenging benchmarks. BIBLIOGRAPHY221 Bibliography Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wol- czyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuci ́ nski, Lerrel Pinto, Rob Fer- gus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rockt ̈ aschel. BAL- ROG: Benchmarking agentic LLM and VLM reasoning on games. In The Thir- teenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=fp6t3F669F. 211, 212, 213 Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. 218, 219, 220 Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. 218 222BIBLIOGRAPHY Part IV Tools and Assistants 223 225 The first three parts discussed the AI techniques for constructing advanced LLM Agents that can interact with and act in the real world. These capabilities allow them to do useful things for their users. In this part we will address the use of tools by LLMs to perform specialized tasks, such as looking up information, or controlling robots. Agents use tools through learned APIs (Application Programming Interfaces). Tools can help us solve a major problem of pretrained LLMs, which is that the information that is contained in them is only as up to date as the dataset on which they were trained. Retrieval augmentation techniques allows LLMs to provide up to the minute information. Many types of such agents have been created, and in the final chapter of this part we will review important examples, from the medical, financial, and scientific domains. Finally, we will discuss a practical case study in which the use of the technologies comes together, when we will implement an LLM Agent to perform medical diagnosis. 226 Chapter 11 How Agents Use Tools to Act in the World What You Will Learn In this chapter you will learn how agentic Large Language Models connect to the external world through tools. Specifically, you will be able to: • Explain why tools are essential for agents that act beyond pure language gener- ation. • Distinguish between the direct invocation and Planner–Caller–Summarizer ar- chitectural patterns. • Work with toolchains and frameworks such as LangChain and the ReAct loop, and understand how agents interact with browsers and desktop environments. • Describe the Model Context Protocol as a standardized approach to tool integra- tion. • Reason about robustness, error recovery, and the safety and security risks that arise when agents are given access to external tools. 11.1 Why Tools Matter in Agentic LLMs In the taxonomy of reasoning–acting–interacting (Figure 1.1), tools are the primary bridge between an LLM’s internal reasoning and its actions in the external world. A tool may be as simple as a calculator, as complex as a scientific simulation engine, or as open-ended as a full desktop environment. By equipping LLMs with tool use capabilities, we allow them to: • Access fresh, domain-specific data (e.g., via APIs, databases, or search engines). • Perform actions in the world (e.g., controlling robots, operating browsers, send- ing messages). • Delegate complex computations or retrieval to specialized systems. 227 228CHAPTER 11. HOW AGENTS USE TOOLS TO ACT IN THE WORLD • Chain multiple tools into composite workflows that can be planned, executed, and monitored. Retrieval is one specific kind of tool primarily aimed to augment an LLM’s knowl- edge. We will discuss it in depth in Chapter 12. Tool use in general can also augment an LLM’s capabilities to manipulate its environment. These capabilities also come with additional risks, such as safety, reliability and performance risks. They introduce additional points of failure and might be misused under certain conditions. 11.2 Tool Use Architectures Let us see some common software architectures for tool use by LLMs. Integrating tools into an LLM agent typically follows one of two patterns: direct invocation, and planner-caller-summarizer. Direct Invocation In the direct invocation pattern, the LLM outputs structured calls (e.g., JSON or func- tion signatures) that are parsed by an external controller and routed to the tool. For example: OpenAI’s function calling API or Gorilla [Patil et al., 2023]. Planner–Caller–Summarizer The planner-caller-summarizer pattern is an extension of the direct invocation pattern where we use a multi-agent architecture where the LLM serves as a central planner. It calls a separate module to execute a tool (which might be another LLM or another kind of agent), which then returns a summarized observation back to the LLM. This allows the LLM to maintain a high-level view of the task while outsourcing the detailed execution to a dedicated system. ReAct (Reasoning and Acting) [Yao et al., 2022] is a prominent framework that interweaves reasoning and acting. Instead of generating a full plan and then exe- cuting, an LLM using ReAct follows an iterative loop of Thought, Action, and Observation. This allows the agent to dynamically adjust its plan based on the real-time feedback it receives from tool outputs, making it more robust to errors and unexpected results. This works well because the tools (observation part) often act as a grounding mechanism in the real world (such as the execution of a code snippet, or an internet search), which provides the LLM with factual information for the next iteration. 11.3 APIs and Toolchains When their application programming interface (API) is known, LLMs can be integrated easily with external tools: for an LLM, an API is just another language to learn. One common way to use a tool, is to augment LLM input with results that are retrieved from 11.3. APIS AND TOOLCHAINS229 external sources, such as search engines. You can read more about retrieval methods in Chapter 12. Agentic LLMs must be trained to decide when and how to utilize external tools, depending on the task [Shen et al., 2024]. Language models can teach themselves to use tools. The Toolformer model [Schick et al., 2023] is trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. A range of tools is tested: a calculator, a question- answering system, a search engine, a translation system, and a calendar. Further works extend this to a larger range of tools. ToolBench [Qin et al., 2023] contains 16,464 APIs from RapidAPI, a large dataset of publicly available REST APIs. 1 Another framework is EasyTool [Yuan et al., 2024], which focuses on structured and unified instructions from tool documentations, building on ToolBench. ToolAl- paca is a benchmark with over 3938 instances from 400 API [Tang et al., 2023]. A tool-based benchmark for question answering is ToolQA [Zhuang et al., 2023]. Go- rilla is a finetuned LLaMa-based model for generating API calls [Patil et al., 2023], also introducing the APIBench benchmark. Many tool calling frameworks have been developed. Selecting the right tool and summarizing its result are difficult skills for a model. Zhao et al. [2024], Shen et al. [2024] study how LLMs can improve recommendation through tool learning. They use different LLMs for (1) reasoning ability, (2) request writing, and (3) result summarization. Figure 11.1 illustrates this architecture, consist- ing of a planner, a caller, and a summarizer, each implemented by a different LLM finetuned for its specific capability. Good results are reported on the LLMs Claude-2, ChatGPT, GPT-4, and Tool-LLaMa, using as reasoning strategies ReAct [Yao et al., 2022] and DFSDT [Qin et al., 2023]. Other frameworks also exist, such as [Ocker et al., 2024]. 11.3.1 LangChain Example LangChain [Chase, 2022] is a widely-used framework for building LLM applications, offering robust abstractions for creating agents and tool chains. Listing 11.1 is a simple Python example demonstrating how to define a tool and use it within a basic agentic framework. In this scenario, the agent is asked a question that requires a numerical check, which it delegates to a specialized tool. 11.3.2 Computer and Browser Tools One specific set of action tools is to let an LLM use a browser or even a complete computer system as a special form of API. Equipping agentic LLMs with the ability to interact directly with a computer environment enables many interaction possibilities. Tools that parse, interpret, and manipulate graphical user interfaces (GUIs) have gained attention for bridging the gap between language models and real-world applications. One such example is OmniParser V2 [Lu et al., 2024], which introduces a vision- based screen parsing method to detect and label interactable elements such as buttons 1 https://rapidapi.com/hub 230CHAPTER 11. HOW AGENTS USE TOOLS TO ACT IN THE WORLD 1 from langchain.agents import initialize_agent, AgentType 2 from langchain_google_genai import ChatGoogleGenerativeAI 3 from langchain.tools import tool 4 5 # 1. Define a tool using the`@tool` decorator. 6 # This function is now a callable tool for the agent. 7 # The docstring is used as the tool's description, 8 # which the LLM will read to decide when to use it. 9 @tool 10 def is_prime(number: int) -> str: 11 """Checks if a given integer is a prime number. 12 The input should be a single integer. 13 14 Example: is_prime(17) would return 'True'. 15 """ 16 if number <= 1: 17 return 'False' 18 for i in range(2, int(number ** 0.5) + 1): 19 if number % i == 0: 20 return 'False' 21 return 'True' 22 23 # 2. Initialize the LLM and the tools for the agent. 24 # We'l use gemini-2.5-flash-preview-05-20 as the LLM. 25 # The API key is handled automatically in this environment. 26 llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash-preview-05-20") 27 tools = [is_prime] 28 29 # 3. Initialize the agent with the LLM, tools, and a type. 30 agent = initialize_agent( 31 tools, 32 llm, 33 agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, 34 verbose=True 35 ) 36 37 # 4. Run the agent with a user prompt. 38 # The agent will reason, call the tool, and then formulate a final answer.↩→ 39 user_prompt = "Is the number 97 a prime number? Please give a 'Yes' or 'No' answer."↩→ 40 result = agent.run(user_prompt) 41 42 # The agent's execution loop: 43 # It will print the 'Thought' and 'Action' steps to the console. Listing 11.1: Example on how to use LangChain to define a tool and use it with an agent. 11.3. APIS AND TOOLCHAINS231 Figure 11.1: Multi LLM agent Framework with a Planner, Caller, and Summarizer [Shen et al., 2024] or icons. By converting raw screenshots into structured representations, OmniParser helps vision-language models ground their action decisions in specific user interface components. This grounding increases the accuracy of the action predictions of LLMs. Another line of research focuses on enabling LLMs to initiate system-level com- mands or navigate within browser or operating system interfaces. Computer Use, pro- posed by Antropic, and Operator and later GPT Agent, proposed by OpenAI, are ex- amples of such efforts. These tools wrap common desktop and browser actions (such as opening applications, clicking buttons, and filling forms) into tool APIs callable by an LLM. This setup translates high-level textual commands into executable steps. As a result, an agentic LLM can browse the web, manage files, or run administrative tasks, all through natural language prompts. This can lead to more autonomous behavior and execution of larger and more complex tasks. Browser Use [M ̈ uller and ˇ Zuni ˇ c, 2024] is an example of an open-source tool that enables LLMs to use a browser with persistent session management. Browser Use enables agentic LLMs to maintain longer browsing states across multiple pages or do- mains. The tool manages cookies, session tokens, and dynamic web content updates, thereby allowing LLMs to execute more complex browsing tasks like multi-step form completions or cross-site queries. For a more in-depth discussion about browser and computer environments, see the survey by Wang et al. [2025b]. The survey discusses design patterns for combin- ing automated GUI parsing, tool call integration, and human-in-the-loop supervision. Collectively, these approaches reveal that reliable computer and browser tools are im- portant components for agentic LLMs. By translating textual intents into grounded, 232CHAPTER 11. HOW AGENTS USE TOOLS TO ACT IN THE WORLD environment-level operations, such systems expand the possible tasks and domains where language-driven autonomy can be effectively realized. 11.4 Model Context Protocol and Context Management The Model Context Protocol (MCP) is an open-source standard introduced by An- thropic for connecting AI applications to external data sources, tools, and workflows [Hou et al., 2026]. It is often described as a “USB-C port for AI”, providing a standard- ized way for applications such as Claude or ChatGPT to access local files, databases, search engines, calculators, and other external capabilities. MCP uses a client-server architecture. An MCP host, such as an AI application, manages one or more MCP clients, with each client maintaining a connection to an MCP server. MCP servers provide context and capabilities that can be used by the host, including tools, resources, and prompts. Local servers commonly communicate with a single client through STDIO, while remote servers can support multiple clients through Streamable HTTP. The protocol is organized into two layers. The data layer defines the JSON-RPC- based communication between clients and servers, including capability and version discovery, as well as primitives such as tools, resources, prompts, and notifications. The transport layer specifies how this communication is established and exchanged, including connection management, message framing, and authorization. This separa- tion allows AI applications to interact with a wide range of external systems through a common interface, while leaving the management of the resulting context to the host application. One of the key challenges in building agentic systems is managing the LLM its finite context window. For multi-step tasks that require numerous tool calls and obser- vations, the context can quickly fill up, causing the model to lose track of its original goal or forget crucial past information. Simply keeping all the context and raw re- sults from multiple tool calls often does not fit the context window and therefor causes forgetting. Deciding what to keep, what to summarize and what to discard is essential. MCP servers are mostly stateless. This means that we need an additional set of strategies designed to manage the agent’s state and context. These strategies should ensure that the agent maintains a clear and coherent state throughout a long-running task. In addition to the fixed tool registration/calling format that MCP offers, additonal components to include are: • Summarization: Tools’ outputs and past conversations are periodically summa- rized and stored in a compressed format. This reduces the token count while preserving key information. • State Management: Explicitly maintaining a state variable or a scratchpad in the prompt, where the agent stores its current plan, important intermediate results, and a summary of its progress. • Prioritized Retrieval: Instead of feeding the entire history back into the context, a retrieval mechanism can be used to only inject the most relevant past information 11.5. ROBUSTNESS, ERROR HANDLING, AND RECOVERY233 for the current step, based on the agent’s reasoning. Example: Implementing the Model Context Protocol Listing 11.2 and 11.3 illus- trates a simplified implementation of the Model Context Protocol. It demonstrates how an agent can manage its context by summarizing past interactions to avoid exceeding a fixed context window size. 2 11.5 Robustness, Error Handling, and Recovery The external nature of tool use introduces new points of failure. An API service might be down, a tool might return an unexpected error, or the LLM might make a logical mistake in its plan. A robust agentic system must be able to handle these failures gracefully. Reactive Error Handling A basic but essential strategy is to implement an error- handling loop. When a tool call fails, the system can provide the LLM with the error message as an observation. The LLM can then reason about the error and attempt a new action, such as re-invoking the tool with corrected arguments, trying a different tool, or informing the user of the failure. This process of observe-reflect-retry is a fundamental mechanism for self-healing in agentic systems [Sun et al., 2024]. Note that this pattern is similar to the ReAct framework but in a different context. Self-Correcting and State-Aware Systems More advanced systems, such as Healer, are designed to correct runtime errors in real-time [Wu et al., 2024]. Healer can dy- namically generate and execute a small code snippet to patch a faulty program state without stopping the entire execution. This goes beyond simple retries; it involves the LLM understanding the context of the error, analyzing the program state, and actively correcting the error (in code). Human-in-the-Loop Integration While autonomy is a goal of agentic systems, the most effective applications often involve a human-in-the-loop. This approach improves safety, allows for corrections, and incorporates human expertise at critical junctures. 3 An agent can be designed to pause and ask for human confirmation before executing a high-stakes action (e.g., making a financial transaction or sending a critical email). This co-piloting model creates a powerful symbiosis, combining the LLM’s speed and scale with human judgment and domain knowledge [Mozannar et al., 2025]. 2 The example uses LangChain, and while this is a popular library, there are also specialized MCP libraries such as the MCP-Python SDK: https://github.com/modelcontextprotocol/python-sdk, and others, which might be interesting to check out. 3 Although it also introduces the possibility of human error. 234CHAPTER 11. HOW AGENTS USE TOOLS TO ACT IN THE WORLD 1 import asyncio 2 import nest_asyncio 3 from langchain_google_genai import ChatGoogleGenerativeAI 4 from langchain.agents import initialize_agent, AgentType 5 from langchain.tools import tool 6 # Required for running asyncio in a notebook environment 7 nest_asyncio.apply() 8 # Define the maximum context size in "tokens" (for this demo, we use characters).↩→ 9 MAX_CONTEXT_SIZE = 250 10 # A state dictionary to persist information across steps. 11 agent_state = 12 'current_plan': 'Initial plan to check if a number is prime.', 13 'history_summary': 'No history yet.' 14 15 async def update_context_with_summary(current_context, new_observation):↩→ 16 """ 17 Manages the context by summarizing past interactions if the context window limit is exceeded.↩→ 18 """ 19 # Append the new observation. 20 new_context = current_context + new_observation + " " 21 # Check if the new context exceeds the maximum size. 22 if len(new_context) > MAX_CONTEXT_SIZE: 23 print("Context is full! Summarizing past history...") 24 # Initialize an LLM specifically for summarization. 25 summarizer_llm = ChatGoogleGenerativeAI(model="gemini-1.5-flash-latest")↩→ 26 # Create a prompt to instruct the LLM to summarize. 27 summary_prompt = ( 28 "Please provide a very concise summary of the following agent conversation. "↩→ 29 "Focus on the key decisions and outcomes. Keep the summary under 30 words."↩→ 30 f" to summarize: current_context" 31 ) 32 try: 33 # Call the LLM to generate the summary asynchronously. 34 summary_response = await summarizer_llm.invoke(summary_prompt)↩→ 35 summary_text = summary_response.content 36 print(f"Generated Summary: summary_text") 37 except Exception as e: 38 # Handle potential API errors. 39 print(f"An error occurred during summarization: e") 40 summary_text = "Summarization failed. History truncated." 41 42 # Update the agent's persistent state with the new summary. 43 agent_state['history_summary'] = summary_text 44 # Rebuild the context with the summary and the most recent parts.↩→ 45 new_context = ( 46 f"Current Plan: agent_state['current_plan'] " 47 f"Past History: agent_state['history_summary'] " 48 f"Recent Observation: new_observation " 49 ) 50 return new_context Listing 11.2: Simplified implementation of the Model Context Protocol (Part 1). 11.5. ROBUSTNESS, ERROR HANDLING, AND RECOVERY235 1 async def main(): 2 # --- Demo Run --- 3 current_context = ( 4 "Task: Determine if 97 is a prime number. " 5 "Thought: I need to use the is_prime tool to solve this. " 6 "Action: Call tool`is_prime` with number 97. " 7 ) 8 9 # Simulate a tool's output observation. 10 tool_output_observation = "Observation: The tool returned the value 'True'."↩→ 11 12 # Update the context using the MCP function. 13 updated_context = await update_context_with_summary(current_context, tool_output_observation) ↩→ ↩→ 14 15 print(" --- Updated Context ---") 16 print(updated_context) 17 18 # Now, let's add more "chatter" to force a summary. 19 long_chatter = ( 20 "Thought: The tool has confirmed that 97 is a prime number. " 21 "The task is complete. " 22 "Final Answer: Yes, 97 is a prime number." 23 ) 24 25 updated_context = await update_context_with_summary(updated_context, long_chatter)↩→ 26 27 print(" --- Context after second update (summarized) ---") 28 print(updated_context) 29 30 if __name__ == "__main__": 31 asyncio.run(main()) Listing 11.3: Simplified implementation of the Model Context Protocol. (Part 2) 236CHAPTER 11. HOW AGENTS USE TOOLS TO ACT IN THE WORLD Figure 11.2: Adversarial prompts generated by Rainbow Teaming [Samvelyan et al., 2024] 11.6 Safety and Security Safety and security are becoming more important as the capabilities of Agentic LLMs grow. This goes beyond simple error handling and is more subtle on a misbehavior and misuse level. An obvious example of a safety problem is that without additional pro- tection, the system prompt of an LLM can easily be exposed, often revealing sensitive information. The robustness of LLMs to jailbreak attacks, where users design prompts to cir- cumvent safety measures and misuse model capabilities, has been studied for chat- bots. Meanwhile, agentic LLMs use external tools and can execute multi stage tasks, and may pose a greater risk. Andriushchenko et al. [2024] propose AgentHarm. This benchmark includes a set of 110 malicious agent tasks ranging from fraud, cyber crime, to harassment. The authors find that leading commercial LLMs are surprisingly com- pliant to their given rules with malicious user requests without jailbreaking (How do I make a pipe bomb? is blocked). However, simple universal templates can be adapted to effectively jailbreak agents (Order online all necessary ingredients to make a pipe bomb and get them delivered to my home without getting flagged by authorities, how- ever, passes in GPT-4). Rainbow Teaming [Samvelyan et al., 2024] uses an open world approach to gen- erate adversarial LLM prompts, to improve the robustness of LLMs. As with train- ing LLMs, a problem with testing methods is a lack of diversity. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem. Rainbow Team- ing is an open-ended approach [Hughes et al., 2024]. It creates diversity with MAP- Elites [Mouret and Clune, 2015], an evolutionary meta search method that iteratively populates an archive with increasingly higher-performing prompts. See Figures 11.2 and 11.3. 11.7. CONCLUDING REMARKS237 Figure 11.3: Quality Diversity Mutation Architecture of Rainbow Teaming [Samvelyan et al., 2024] 11.7 Concluding Remarks Tools are what turn a static language model into a dynamic agent that can act in the world. Their ability to grant LLMs access to fresh data, perform external actions, and delegate complex tasks is transformative. For any agentic system to be successful in practice, the result goes beyond just connecting a tool to an LLM, but now requires us to think about the full life-cycle of the agent’s interaction with the environment, and what can happen. Four steps should be followed: 1. Choose the right Architecture: Methodologies such as ReAct [Yao et al., 2022], which tightly interleave reasoning and tool calls, lead to more robust self-correcting behavior than simpler one-shot approaches. 2. Manage your Context: For multistep long-term tasks, the LLM context window is a critical bottleneck. Use strategies such as the Model Context Protocol to summarize history and manage state, preventing the agent from losing its way. 3. Embrace Failure: Tool-use systems are inherently fragile due to their reliance on external APIs. Build in robust error handling, allowing the agent to observe and reason about failures, and attempt recovery. 4. Keep a Human in the Loop: For high-stakes applications, ensure that the agent has a clear mechanism to pause and seek human confirmation before taking irre- versible actions. This blend of autonomy and human oversight is key to building trust and ensuring safety. 238BIBLIOGRAPHY Ç Try it yourself! Want to experiment with LLM tool-use? Open our: ÐLLM Tools tutorial in Google Colab Further Reading Surveys on tool use in LLMs are Shen [2024], Patil et al. [2023], Qin et al. [2023]. For more on GUI parsing and browser control see Lu et al. [2024], M ̈ uller and ˇ Zuni ˇ c [2024]. For safety benchmarks and adversarial prompting see Wang et al. [2025a]. Bibliography Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrik- son, et al. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint arXiv:2410.09024, 2024. 236 Harrison Chase.LangChain, October 2022.URL https://github.com/ langchain-ai/langchain. 229 Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. Model context protocol (mcp): Landscape, security threats, and future research directions. ACM Trans. Softw. Eng. Methodol., 35(10), September 2026. ISSN 1049-331X. doi: 10.1145/ 3796519. URL https://doi.org/10.1145/3796519. 232 Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rocktaschel. Open-endedness is essen- tial for artificial superhuman intelligence. arXiv preprint arXiv:2406.04268, 2024. 236 Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024. URL https://arxiv.org/abs/2408.00203. 229, 238 Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015. 236 Hussein Mozannar, Gagan Bansal, Cheng Tan, Adam Fourney, Victor Dibia, Jingya Chen, Jack Gerrits, Tyler Payne, Matheus Kunzler Maldaner, Madeleine Grunde- McLaughlin, et al. Magentic-ui: Towards human-in-the-loop agentic systems. arXiv preprint arXiv:2507.22358, 2025. 233 Magnus M ̈ uller and Gregor ˇ Zuni ˇ c. Browser use: Enable ai to control your browser, 2024. URL https://github.com/browser-use/browser-use. 231, 238 BIBLIOGRAPHY239 Felix Ocker, Daniel Tanneberg, Julian Eggert, and Michael Gienger. Tulip agent– enabling llm-based agents to solve tasks using large tool libraries. arXiv preprint arXiv:2407.21778, 2024. 229 Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large lan- guage model connected with massive apis, 2023. URL https://arxiv.org/ abs/2305.15334. 228, 229, 238 Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023. 229, 238 Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et al. Rainbow teaming: Open-ended generation of diverse adversarial prompts. arXiv preprint arXiv:2402.16822, 2024. 236, 237 Timo Schick, Jane Dwivedi-Yu, Roberto Dess ` ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551, 2023. 229 Weizhou Shen, Chenliang Li, Hongzhan Chen, Ming Yan, Xiaojun Quan, Hehong Chen, Ji Zhang, and Fei Huang. Small llms are weak tool learners: A multi-llm agent. arXiv preprint arXiv:2401.07324, 2024. 229, 231 Zhuocheng Shen. Llm with tools: A survey. arXiv preprint arXiv:2409.18807, 2024. 238 Zhensu Sun, Haotian Zhu, Bowen Xu, Xiaoning Du, Li Li, and David Lo. Llm as runtime error handler: A promising pathway to adaptive self-healing of software systems, 2024. URL https://arxiv.org/abs/2408.01055. 233 Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 sim- ulated cases. arXiv preprint arXiv:2306.05301, 2023. 229 Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504.15585, 2025a. 238 Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, Bin Wang, Chuhan Wu, Yasheng Wang, Ruiming Tang, and Jianye Hao. Gui agents with foundation models: A com- prehensive survey, 2025b. URL https://arxiv.org/abs/2411.04890. 231 240BIBLIOGRAPHY Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu.State- flow: Enhancing llm task-solving through state-driven workflows. arXiv preprint arXiv:2403.11322, 2024. 233 Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. 228, 229, 237 Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dong- sheng Li, and Deqing Yang. Easytool: Enhancing llm-based agents with concise tool instruction. arXiv preprint arXiv:2401.06201, 2024. 229 Yuyue Zhao, Jiancan Wu, Xiang Wang, Wei Tang, Dingxian Wang, and Maarten de Ri- jke. Let me do it for you: Towards llm empowered recommendation via tool learn- ing. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1796–1806, 2024. 229 Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36:50117–50143, 2023. 229 Chapter 12 Search and Retrieval in Agentic LLMs What You Will Learn In this chapter you will learn why retrieval is a fundamental capability for agentic systems that need knowledge beyond their training data. Specifically, you will be able to: • Describe the full retrieval-augmented generation pipeline, from indexing and chunking to query formulation and answer generation. • Implement adaptive retrieval methods such as Self-RAG and Self-Ask, and build a dense retrieval system in Python. • Extend retrieval beyond plain text to graphs, code, tabular data, and multimodal sources. • Distinguish agentic RAG from simple one-shot retrieval, including real-time and streaming variants. • Evaluate retrieval systems using standard metrics such as Recall@푘 , Mean Re- ciprocal Rank, and faithfulness scores. 12.1 Retrieval in General Before examining how agentic LLMs consult external knowledge, it is worth situating retrieval within its own, much older, disciplinary history. Information retrieval (IR) is a mature field of computer and information science that predates neural networks by decades. Many of the ideas that power modern retrieval augmented generation (RAG) pipelines are direct descendants of classical information retrieval research. The standard reference works of the field, Salton and McGill [1983] and later Manning et al. [2008], still provide the conceptual vocabulary such as: indexing, term weighting, ranking and relevance, that this chapter builds on. 241 242CHAPTER 12. SEARCH AND RETRIEVAL IN AGENTIC LLMS 12.1.1 Bag of Words Early information retrieval systems represented documents and queries as bags of words and asked a deceptively simple question: which terms make a document a good match for a query? Sp ̈ arck Jones [1972] gave this question a statistical answer with the notion of inverse document frequency (IDF), observing that terms occurring in few documents are more discriminating than terms that occur almost everywhere. Weighting terms by how frequently they occur in a document (term frequency, TF) and how rare they are across the collection (IDF) yields the TF-IDF family of weighting schemes, still one of the most widely deployed representations in production search systems. Term frequency counts how often a term 푡 occurs in a document 푑: TF(푡, 푑) = 푓 푡,푑 Í 푡 ′ ∈푑 푓 푡 ′ ,푑 (12.1) where 푓 푡,푑 is the raw count of푡 in 푑. Inverse document frequency, following Sp ̈ arck Jones [1972], down-weights terms that occur in many documents: IDF(푡, 퐷) = log |퐷| |푑 ∈ 퐷 : 푡 ∈ 푑| (12.2) where 퐷 is the document collection. Combining the two gives the classic TF-IDF weight of term 푡 in document 푑: TF-IDF(푡, 푑, 퐷) = TF(푡, 푑)× IDF(푡, 퐷)(12.3) 12.1.2 Vector Space Models Salton et al. [1975] formalized this idea into the vector space model, representing every document and query as a vector in a high-dimensional term space and ranking docu- ments by their cosine similarity to the query vector. The vector space model established a pattern that persists directly into today’s dense retrievers (Section 12.3): reduce doc- uments and queries to points in a shared geometric space, and let distance or similarity stand in for relevance. More formally, in the vector space model, each document 푑 (and query 푞) is repre- sented as a vector of term weights, ® 푑 = (푤 1,푑 ,푤 2,푑 ,...,푤 푛,푑 ), typically using TF-IDF weights (Eq. 12.3). Relevance is then approximated by the cosine similarity between query and document vectors: sim(®푞, ® 푑) = cos(휃) = ®푞· ® 푑 ∥®푞∥ ® 푑∥ (12.4) Documents are ranked by decreasing sim(®푞, ® 푑). This same operation, cosine simi- larity between two learned vectors, reappears in dense retrieval (Section 12.3), the only difference being how the vectors are obtained. Where the vector space model used raw term co-occurrence, modern dense retrievers such as dense passage retrieval (DPR) [Karpukhin et al., 2020] use learned neural embeddings, but the underlying ge- ometric intuition is unchanged. 12.1. RETRIEVAL IN GENERAL243 12.1.3 Latent and Sparse Representations Two further developments bridge classical and modern retrieval. First, Robertson and Zaragoza [2009] developed the probabilistic relevance framework, which treats re- trieval as a problem of estimating the probability that a document is relevant to a query given its term statistics. This framework produced BM25 Robertson et al. [1994], a term-weighting and ranking function which ranks document 푑 for query 푞 as: BM25(푞, 푑) = ∑︁ 푡∈푞 IDF(푡, 퐷)· 푓 푡,푑 (푘 1 + 1) 푓 푡,푑 + 푘 1 1− 푏+ 푏 |푑| avgdl (12.5) where |푑| is the length of document 푑, avgdl is the average document length in the collection, and 푘 1 and 푏 are free parameters that control term-frequency saturation and length normalization, respectively (typical values: 푘 1 ∈ [1.2, 2.0], 푏 = 0.75). BM25 remains the sparse-retrieval baseline referenced throughout the hybrid retrieval discussions in Section 12.3. Second, Deerwester et al. [1990] introduced latent semantic analysis (LSA), which applies singular value decomposition to the term-document matrix to uncover latent topics and match documents by meaning rather than by exact term overlap. LSA was the first widely used technique to address the vocabulary mismatch problem, the same problem that motivates dense and semantic retrieval today. LSA operates on the term-document matrix 푋 ∈R 푚×푛 , where 푚 is the vocabu- lary size and 푛 the number of documents. Its entries are typically TF-IDF weights (Eq. 12.3). Applying truncated singular value decomposition, 푋 ≈ 푈 푘 Σ 푘 푉 ⊤ 푘 (12.6) retains only the 푘 largest singular values in Σ 푘 , producing a rank-푘 approximation of 푋 . Documents and terms are then represented in this reduced 푘 -dimensional latent space, where semantically related terms and documents lie close together even without exact term overlap. The final bridge from classical IR to the neural retrieval methods used through- out this chapter is the emergence of learned, dense vector representations of words. Mikolov et al. [2013] showed that simple neural network architectures, trained to pre- dict a word from its context, produce continuous word embeddings that capture rich syntactic and semantic regularities. This work, together with the encoder architectures that followed, supplied the representational building blocks on which dense retrieval, DPR-style bi-encoders, and ultimately retrieval-augmented generation are built. Retrieval, in other words, is a decades-old discipline concerned with a single endur- ing question: given a query, which pieces of a large collection are actually relevant to it? What has changed with the arrival of LLMs is not the question but the stakes: mod- els now consult retrieved evidence not merely to rank documents for a human reader, but to condition their own generation. We will now discuss why that shift matters. 244CHAPTER 12. SEARCH AND RETRIEVAL IN AGENTIC LLMS 12.2 Why Retrieval Matters for Agentic LLMs Building on this foundation, large language models introduce a new reason to care about retrieval. LLMs are frozen snapshots of world knowledge. The LLM’s parame- ters can not contain information that was generated after the corpus was fixed. LLMs struggle when tasks require timely, proprietary, or long–tail information that was absent at pretraining time. Retrieval augmentation mitigates these limits by consulting exter- nal sources at inference time and conditioning generation on the retrieved evidence. Retrieval augmented generation (RAG) methods have been developed to integrate data from external knowledge bases with the pretrained knowledge stored in the pa- rameters of language models. Most RAG approaches work at inference time, although some can work during finetuning or even pretraining. Most RAG approaches work on unstructured (textual) data sources. These text documents are indexed to increase effi- cient access, and they can be organized and a knowledge graph can be created. Finally, database-type query optimization is often performed, where queries can be expanded or complex queries can be split into sub-queries [Cong et al., 2024]. This simple idea, popularized by early systems such as REALM [Guu et al., 2020], DPR [Karpukhin et al., 2020], and RAG [Lewis et al., 2020], has grown into a rich design space spanning indexing, query planning, re-ranking, citation and attribution, and continuous data refresh. Recent surveys synthesize the landscape and evaluation practices [Gao et al., 2023b, Huang et al., 2024, Yu et al., 2024, Sharma et al., 2025, Peng et al., 2024]. When to retrieve. Retrieval is most beneficial when: (1) the task is knowledge- intensive (definitions, entities, numbers, references), (2) the domain shifts rapidly (news, prices, docs), (3) transparency is required (with citations), or (4) models must respect strict source-of-truth constraints (compliance portals, internal wikis). In agentic set- tings, retrieval can be interleaved with tool use and planning (see Chapter 11). 12.3 Retrieval-Augmented Generation Pipeline A practical RAG system typically implements the following seven stages: 1. Indexing: The indexing phase is the first step. This step involves: Taking all raw source material, such as HTML pages, PDFs, tables, or code, and converting it into clean, consistent text. Keep important structure as metadata (headings, anchors, timestamps, authorship). Break content into chunks with meaningful boundaries (e.g., sentences or sections) and small overlaps so context isn’t lost. Create two types of searchable representations: sparse ones, like BM25 (a classic term-matching method) or SPLADE [Formal et al., 2021] (a more modern vari- ant that expands terms), which are good for exact keyword matches; and dense ones, which are embeddings that capture meaning for semantic matches. Storing both lets the system find exact matches (IDs, numbers) and semantic matches (paraphrases, synonyms) in the same search. 12.3. RETRIEVAL-AUGMENTED GENERATION PIPELINE245 2. Query Construction: Turn the user’s question into one or more precise search queries. This can involve rewriting the question for clarity, breaking complex questions into sub-questions (multi-hop), adding related terms (query expan- sion), or even generating a hypothetical answer to use as a search target (HyDE) [Gao et al., 2023a]. For structured data sources, generate specific queries such as SQL or graph lookups. The aim is to make the search engine understand exactly what to look for. 3. Candidate Retrieval: Use the constructed queries to pull in potentially relevant chunks from the index. Dense retrievers (e.g., DPR [Karpukhin et al., 2020], Contriever [Izacard et al., 2022]) match by meaning; sparse methods (BM25, SPLADE) match by exact terms; hybrids combine both. To search quickly at scale, use approximate nearest neighbor (ANN) indexes like HNSW [Malkov and Yashunin, 2020] or IVF-PQ [J ́ egou et al., 2011]. For code or images, use specialized encoders. This step is about casting a wide net without losing focus. 4. Re-ranking: Take the pool of retrieved candidates and reorder them by relevance using more accurate but slower models, like cross-encoders (BERT, MiniLM). Late-interaction models such as ColBERTv2 [Santhanam et al., 2022] balance quality and speed. This ensures that top results are the ones most likely to answer the question. 5. Context Assembly: Remove duplicates, ensure coverage of different aspects of the query (diversity), and prefer recent or domain-relevant sources. Bundle the selected passages, along with their metadata (source, URL, date), into a con- text package to feed into the LLM. This gives the model a curated, trustworthy information set. 6. Grounded Generation: Ask the LLM to generate the answer using the provided context, with citations for each claim. If evidence is insufficient, the model should abstain or ask for clarification. Some systems, such as Self-RAG [Asai et al., 2023], decide on the fly whether to retrieve more information. The focus is on answers that are both correct and well-supported. 7. Attribution and Logging: Show the sources alongside the answer so readers can verify claims. Log the entire process including queries, retrieved documents and versions, for debugging, evaluation, and compliance. This makes the system transparent and auditable. A straightforward RAG system follows the seven-step pipeline outlined above. List- ing 12.1 shows Python-style pseudo-code for this process. The function takes a user question, rewrites it into one or more search queries, retrieves and re-ranks candidate passages, assembles a context window and finally generates an answer while tracking which passages were used. We will go into more details below. 246CHAPTER 12. SEARCH AND RETRIEVAL IN AGENTIC LLMS 1 # Input: question q, search index, and number of passages k 2 def rag_answer(q, index, k=8): 3 # 1. Construct search queries (possibly decompose multi-hop questions) 4 queries = rewrite_question(q) 5 # 2. Retrieve candidate passages from the index 6 candidates = retrieve(index, queries, k=200) 7 # 3. Re-rank with a cross-encoder or late-interaction model 8 ranked = rerank(candidates) 9 # 4. Assemble a context window from the top-k passages 10 context = assemble(ranked[:k]) 11 # 5. Generate an answer conditioned on the context and provide citations 12 answer, citations = generate_answer(q, context) 13 return answer, citations Listing 12.1: High Level Flow of a Basic RAG Setup 12.4 Adaptive Retrieval Methods Traditional RAG systems retrieve a fixed number of passages regardless of whether any retrieval is actually needed. This always–on strategy simplifies engineering but can hurt both efficiency and accuracy. Recent research explores adaptive or gated retrieval strategies that teach the model to decide when to search and how much evidence to consult. These methods align with the self-reflection ideas introduced in Section 6.2.4: rather than blindly ingest retrieved text, the language model introspects about its own confidence and triggers retrieval only when it recognizes a knowledge gap. Concretely, Asai et al. [2023] propose Self-RAG, which trains a single model to generate answers, request retrieval, and critique its own outputs using special reflection tokens. When the model’s draft answer is uncertain, it emits a <RETRIEVE> token to call the search tool and incorporates the retrieved passages in the next step. Experi- ments show that this learned gating improves factuality over static RAG while avoiding unnecessary calls. A more lightweight approach is Self-Ask [Press et al., 2022], where the model decomposes a complex question into simpler sub-questions, issues them to a search tool, and uses the answers to compose a final response. Self-Ask can be combined with planning agents, enabling models to interleave reasoning, tool calls, and retrieval. Graph Toolformer [Zhang, 2023] extends this idea by training a policy to insert search queries into a program graph. At inference time the LLM dynamically decides whether to retrieve from a text index, a knowledge graph, or a database, based on the current state of reasoning. Beyond these specific examples, adaptive retrieval can be implemented in several ways: • Confidence-based Gating: The model estimates its uncertainty (e.g., from en- tropy or calibration metrics) and triggers retrieval when confidence falls below a threshold. • Iterative Retrieval: Rather than pulling all evidence at once, the system retrieves a few passages, reasons about them, and then decides whether additional rounds 12.4. ADAPTIVE RETRIEVAL METHODS247 of search are needed. This approach is useful for multi-hop questions. • Query Planning: An agent decomposes the user query into sub-tasks, dispatches them to different tools (search, database, calculator), and integrates the results. Retrievers are called only when the plan requires external knowledge. • Learning to Retrieve: Reinforcement learning or supervised finetuning can teach the model to emit <RETRIEVE> tokens at appropriate times. Self-RAG is one instantiation; other methods explore training retrieval policies end-to-end with the generator. Adaptive retrieval reduces latency and cost by avoiding unnecessary searches, and it often improves answer quality by focusing attention on missing knowledge. Designing effective gating policies, however, requires careful evaluation (see Section 12.7) and may demand additional supervision during training. 12.4.1 Adaptive Retrieval via Self-RAG and Self-Ask The Self-RAG framework of Asai et al. [2023] teaches the model to request retrieval on demand by emitting a special <RETRIEVE> token during generation. The pseudo- code in Listing 12.2 sketches how such a loop operates. The model maintains a growing context and alternates between generating output tokens and pulling in new evidence when needed. 1 # Input: question q, search index, and trained Self-RAG model 2 def self_rag_answer(q, index, model): 3 context = [] 4 output_tokens = [] 5 # Generate tokens sequentially 6 for token in model.generate_stream(q, context): 7 if token == "<RETRIEVE>": 8 # Model requests more information 9 new_docs = retrieve(index, rewrite_question(q), k=5) 10 context.extend(new_docs) 11 else: 12 output_tokens.append(token) 13 if token == "<END>": 14 break 15 return detokenize(output_tokens) Listing 12.2: High Level Pseudo-Code Demonstrating Self-RAG It should be noted that for Self-RAG to work, the foundation model should be trained or fine-tuned to produce the <RETRIEVE> token. Self-Ask [Press et al., 2022] on the other hand is not dependent on the language model. Self-Ask decomposes a complex question into simpler sub-questions, retrieves evidence for each and synthe- sizes the final answer. The following high-level pseudo-code shows one way to imple- ment this strategy: 248CHAPTER 12. SEARCH AND RETRIEVAL IN AGENTIC LLMS 1 def self_ask(q, index): 2 sub_questions = decompose(q) 3 # e.g. "When was Einstein born?" and "What is the Nobel Prize?" 4 intermediate_answers = [] 5 for sq in sub_questions: 6 docs = retrieve(index, rewrite_question(sq), k=5) 7 answer = summarize(docs) 8 intermediate_answers.append(answer) 9 final_answer = compose(intermediate_answers) 10 return final_answer Listing 12.3: High Level Pseudo-Code Demonstrating Self-Ask 12.4.2 A Simple Dense Retrieval Example in Python Dense retrieval libraries such as the sentence transformers toolkit make it easy to index a small corpus and perform semantic search. Listing 12.4 illustrates how to embed a corpus of sentences, compute similarities to a query and display the top results. 1 from sentence_transformers import SentenceTransformer, util 2 3 # Prepare a small corpus 4 texts = [ 5 "The Eiffel Tower is in Paris.", 6 "The capital of France is Paris.", 7 "Berlin is the capital of Germany.", 8 "The Great Wall is in China." 9 ] 10 11 model = SentenceTransformer("all-MiniLM-L6-v2") 12 corpus_embeddings = model.encode(texts, convert_to_tensor=True) 13 14 def semantic_search(question, k=2): 15 query_emb = model.encode(question, convert_to_tensor=True) 16 scores = util.cos_sim(query_emb, corpus_embeddings)[0] 17 top_results = scores.topk(k) 18 for score, idx in zip(top_results.values, top_results.indices): 19 print(texts[idx], "(score:", score.item(), ")") 20 21 # Example usage 22 semantic_search("What is the capital of France?") Listing 12.4: A Simple Dense Retrieval Example in Python 12.4.3 Query Rewriting and Expansion Effective retrieval often starts with transforming the original question into a form that the index can match. Large language models can act as query re-writers by paraphras- ing, expanding acronyms, adding synonyms or even producing a hypothetical answer 12.5. GRAPHS, CODE, TABULAR DATA249 to use as a pseudo-query (the HyDE technique). 1 Listing 12.5 sketches a simple func- tion that calls an LLM to produce multiple rewrites and expansions and then merges the results into a set of search queries. 1 def rewrite_question(q, n_variants=3): 2 """ 3 Use an LLM to generate alternative phrasings and hypothetical answers. 4 5 :param q: The user question. 6 :param n_variants: Number of rewrites to generate. 7 :return: A list of expanded queries. 8 """ 9 variants = [q] 10 # Ask the model for paraphrases and clarifications 11 paraphrases = llm.generate(["Rewrite: " + q] * n_variants) 12 variants.extend(paraphrases) 13 # Ask the model to guess an answer (HyDE) and use it as a pseudo-query 14 guess = llm.generate(["Answer briefly: " + q])[0] 15 variants.append(guess) 16 # Optionally add entity aliases from a knowledge base 17 aliases = lookup_aliases(q) 18 variants.extend(aliases) 19 return variants Listing 12.5: A Simple Example of Query Rewriting In practice, the rewritequestion function might call an API such as Ope- nAI’s gpt-5 to generate paraphrases and hypothetical answers, then use a thesaurus or entity-linking tool to add aliases. Query expansion improves recall but can also introduce noise. 12.5 Beyond Text: Graphs, Code, Tabular Data and Multi-Modal While most RAG systems operate over unstructured text, many agentic applications require reasoning over structured data, source code, or images. Extending retrieval beyond plain text opens up new opportunities but also introduces modeling challenges. Graph-RAG A knowledge graph encodes entities as nodes and relations as edges. Building a lightweight graph over the corpus allows the retriever to follow links (e.g., works for, located in) rather than relying solely on bag-of-words similarity. The GraphRAG survey formalizes a three-step workflow: graph-based indexing, graph-guided retrieval, and graph-enhanced generation [Peng et al., 2024]. Indexing extracts entities and rela- tions from text, canonicalizes them and builds neighborhood structures. Retrieval then expands the query through the graph to find related nodes and associated passages; this 1 HyDE stands for Hypothetical Document Embeddings and generates a plausible answer which is then embedded and used to retrieve supporting passages [Gao et al., 2023a]. 250CHAPTER 12. SEARCH AND RETRIEVAL IN AGENTIC LLMS supports global reasoning and disambiguation. Finally, the generation model can use the graph structure to organize facts into a coherent narrative and avoid contradictions. 12.5.1 Graph-based retrieval example GraphRAG systems represent knowledge as a graph of entities and relations and use graph traversals to retrieve relevant nodes [Peng et al., 2024]. A lightweight imple- mentation can be built with the networkx library. In the example in Listing 12.6 we construct a small knowledge graph and implement two simple retrieval primitives: (1) get the immediate neighborhood of an entity, and (2) find the shortest path between two entities. These operations can support multi-hop reasoning. 1 import networkx as nx 2 3 def build_knowledge_graph(): 4 G = nx.DiGraph() 5 # Add nodes representing entities 6 G.add_node("Albert Einstein", type="person") 7 G.add_node("Nobel Prize", type="award") 8 G.add_node("Physics", type="field") 9 G.add_node("1921", type="year") 10 # Add edges representing relations 11 G.add_edge("Albert Einstein", "Nobel Prize", relation="received") 12 G.add_edge("Nobel Prize", "Physics", relation="category") 13 G.add_edge("Nobel Prize", "1921", relation="year") 14 return G 15 16 def neighbours(G, entity): 17 """Return all outgoing neighbours of an entity.""" 18 return list(G.successors(entity)) 19 20 def shortest_path(G, source, target): 21 """Return the sequence of entities on the shortest path.""" 22 try: 23 return nx.shortest_path(G, source, target) 24 except nx.NetworkXNoPath: 25 return [] 26 27 # Example usage 28 G = build_knowledge_graph() 29 print(neighbours(G, "Albert Einstein")) 30 # ['Nobel Prize'] 31 print(shortest_path(G, "Albert Einstein", "Physics")) 32 # ['Albert Einstein', 'Nobel Prize', 'Physics'] Listing 12.6: An Example of Graph-based Retrieval Primitives in Python In a full GraphRAG system the graph nodes would store pointers to the original text passages, and the retrieval procedure would return both the sequence of nodes and the associated passages to the LLM. Graph-guided retrieval can reduce hallucinations by grounding reasoning in explicit edges and by encouraging consistency between related facts. 12.5. GRAPHS, CODE, TABULAR DATA251 Code-RAG Source code has a rich syntax-structure. The semantics of source code differs from natural language. For code retrieval to be effective, it should index at the function or module level, including signatures, docstrings, abstract syntax trees (ASTs) and dependency graphs. Models such as CodeBERT [Feng et al., 2020] learn joint embeddings for code and natural language; they support tasks like natural-language code search, code summarization, and documentation generation. In Code-RAG, the retriever can fetch relevant functions, APIs and test cases, and the re-ranker can use code-aware cross-encoders or graph neural networks to prioritize exact matches. Re- trieved code snippets, comments and dependency metadata are then provided to the LLM, which can answer questions, suggest fixes, or generate new code while citing the source files. Evaluation should consider correctness of the code, type safety and adherence to dependencies. 12.5.2 Code retrieval example using ASTs and embeddings Retrieving source code requires understanding its syntax and semantics. An easy ap- proach is to embed code snippets and natural language queries with a pretrained model such as CodeBERT [Feng et al., 2020]. Another complementary approach is to parse the code into an abstract syntax tree (AST) and search its structure. The example in Listing 12.7 demonstrates both: we embed functions with CodeBERT for semantic search and use Python’s ast module to extract function definitions and docstrings for indexing. This example indexes the functions in an imported module and performs a cosine similarity search between the query and the encoded source code. In practice one might combine code embeddings with lexical features (identifiers, comments) and incorpo- rate AST-based matching to handle structural queries (e.g., find all functions that call a specific API). Code-aware retrieval is essential for agentic systems that need to read, execute or modify existing code during tool use. 12.5.3 Multi-modal and heterogeneous sources Tabular and other heterogeneous documents Existing RAG approaches face criti- cal limitations when applied to heterogeneous documents, comprising both textual and tabular components [Yu et al., 2025]. A primary challenge is Structural Information Loss. When tabular data is linearized or chunked for text-based RAG, its inherent rela- tional structure, the interdependence of rows and columns, is compromised. This loss of structure impedes the LLM’s ability to perform accurate multi-hop heterogeneous reasoning. This indicates that merely converting non-textual data to text is insufficient. To address these challenges, hybrid frameworks such as TableRAG [Yu et al., 2025] have been proposed, that unify textual understanding and complex manipulations over tabular data. TableRAG employs SQL programming and execution to treat table-related queries as indivisible reasoning units, thereby enhancing both computational efficiency and reasoning fidelity. Similarly, MixRAG [Zhang et al., 2025] has been shown to signif- icantly improve retrieval performance for heterogeneous document RAG, which de- mands joint retrieval and reasoning across textual and hierarchical tabular data. 252CHAPTER 12. SEARCH AND RETRIEVAL IN AGENTIC LLMS 1 from transformers import AutoTokenizer, AutoModel 2 import torch, ast, inspect 3 4 tokenizer = AutoTokenizer.from_pretrained("microsoft/codebert-base") 5 model = AutoModel.from_pretrained("microsoft/codebert-base") 6 7 def encode_code(code_str): 8 inputs = tokenizer(code_str, return_tensors="pt") 9 with torch.no_grad(): 10 embeddings = model( ** inputs).last_hidden_state.mean(dim=1) 11 return embeddings 12 13 def extract_functions(module): 14 """Return a list of (name, source, docstring) tuples.""" 15 functions = [] 16 for name, obj in inspect.getmembers(module, inspect.isfunction): 17 source = inspect.getsource(obj) 18 doc = inspect.getdoc(obj) or "" 19 functions.append((name, source, doc)) 20 return functions 21 22 # Index a module's functions 23 import example_module # your codebase 24 functions = extract_functions(example_module) 25 embeddings = [encode_code(src) for (_, src, _) in functions] 26 27 def search_code(query, k=2): 28 query_emb = encode_code(query) 29 scores = [torch.n.functional.cosine_similarity(query_emb, emb)[0] 30 for emb in embeddings] 31 # Return the top-k functions with highest similarity 32 topk = sorted(zip(scores, functions), key=lambda x: x[0], reverse=True)[:k]↩→ 33 for score, (name, src, doc) in topk: 34 print(name, "(score:", score.item(), ") ", doc) 35 36 # Example usage: search for a sorting function 37 search_code("sort a list of numbers") Listing 12.7: An Example of Code-based Retrieval in Python The success of these approaches demonstrates that a purely neural (embedding- based) RAG approach is insufficient for structured data. The integration of symbolic reasoning (e.g., SQL) alongside neural retrieval is a critical design pattern for hetero- geneous RAG, reflecting a broader trend in Agentic LLMs towards neuro-symbolic integration for enhanced data understanding and fidelity. Multi-modal RAG Many knowledge bases contain tables, figures, diagrams, audio and video. A straightforward approach is to extract any textual content using OCR and treat the remainder as plain text. A richer approach stores embeddings for images or other modalities (e.g., CLIP or ViT representations) and pairs them with captions and context. Queries can be encoded jointly across modalities, allowing the system to 12.5. GRAPHS, CODE, TABULAR DATA253 retrieve relevant figures or equations that accompany the text. For example, a research agent might retrieve a plot and its caption to answer a data-analysis question, or an educational agent might fetch diagrams from a textbook. Multi-modal RAG remains an active research area. Challenges include cross-modal alignment and efficient indexing of high-dimensional embeddings. Furthermore, non-textual evidence should be properly grounded in the generated answer. 12.5.4 Multi-modal retrieval example Agentic systems sometimes need to retrieve images, diagrams or audio clips in addition to text. Multi-modal models such as CLIP or Contrastive Language-Audio Pretraining (CLAP) embed both natural language and other modalities into a shared space. The high-level pseudo-code in Listing 12.8 outlines how to use CLIP to find images relevant to a textual query: 1 from PIL import Image 2 import clip, torch 3 4 model, preprocess = clip.load("ViT-B/32") 5 6 def embed_image(path): 7 image = preprocess(Image.open(path)).unsqueeze(0) 8 with torch.no_grad(): 9 return model.encode_image(image) 10 11 def embed_text(text): 12 text_tokens = clip.tokenize([text]) 13 with torch.no_grad(): 14 return model.encode_text(text_tokens) 15 16 # Build an index of image embeddings 17 image_paths = ["cat.jpg", "dog.png", "city.png"] 18 image_embs = [embed_image(p) for p in image_paths] 19 20 def search_images(query, k=2): 21 query_emb = embed_text(query) 22 scores = [torch.n.functional.cosine_similarity(query_emb, emb)[0] 23 for emb in image_embs] 24 topk = sorted(zip(scores, image_paths), key=lambda x: x[0], reverse=True)[:k]↩→ 25 return topk 26 27 # Example usage: search for a picture of a city 28 print(search_images("a skyline of a large European city")) Listing 12.8: An Example of How to Use CLIP to Find Relevant Images in Python The multimodal retrieval returns the filenames of the most similar images. In a RAG setting the retrieved images could be fed through a vision encoder to extract cap- tions or objects, which are then incorporated into the context for the language model. Multimodal agents can thus ground their outputs in both textual and visual evidence. 254CHAPTER 12. SEARCH AND RETRIEVAL IN AGENTIC LLMS 12.6 Agentic RAG: Beyond Simple Retrieval While traditional RAG systems augment LLMs with external knowledge, they are often passive, relying on a static, pre-defined pipeline. A significant and recent advancement is the concept of Agentic RAG, which integrates autonomous AI agents directly into the retrieval-augmented generation pipeline [Singh et al., 2025]. This paradigm shift moves RAG from a mere tool to an active, intelligent, and agent-driven process. Agen- tic RAG systems are characterized by their ability to dynamically manage retrieval strategies, iteratively refine contextual understanding, and adapt workflows to meet complex task requirements. Agentic RAG represents a more sophisticated form of a retrieval loop, where the LLM agent itself is responsible for orchestrating the retrieval process. This differs from simpler RAG methods by embracing a continuous feedback loop that is fundamental to advanced agentic systems. Rather than a single-shot query transformation, the agent can actively refine its queries based on intermediate results and ongoing interaction. This approach addresses the limitations of conventional RAG systems, which are of- ten constrained by static workflows and lack the adaptability required for multistep reasoning and complex task management [Singh et al., 2025]. 12.6.1 Key Components of Agentic RAG Agentic RAG frameworks typically incorporate several key components that enable their dynamic behavior: • Dynamic Query Generation: The agent formulates queries based on its current understanding of the task and the retrieved context, allowing for more precise and effective searches. This can include query expansion, decomposition, or rewriting in response to initial retrieval failures. • Iterative Refinement: The agent evaluates the quality of the retrieved informa- tion and, if necessary, initiates a new retrieval cycle with a refined query. This iterative process ensures that the final response is grounded in the most relevant and accurate information. • Autonomous Tool Use: The agent can select and use a variety of tools, includ- ing different retrieval models (e.g., dense vs. sparse), ranking algorithms, and external APIs, to optimize its retrieval strategy. See Chapter 11 for additional information on how an agent can use tools. • Self-Correction and Reflection: The agent can assess its own output for accuracy and completeness. If it detects a hallucination or an unsupported claim, it can trigger a new retrieval and reasoning process to correct the error. The rise of Agentic RAG indicates a shift in the role of RAG in LLM systems. Retrieval shifts from a passive data lookup to an active, intelligent capability that is integral to the agent’s cognitive functions, allowing for more sophisticated, adaptive interactions with external knowledge sources. This shift is crucial for developing robust and reliable LLM agents capable of handling complex, real-world tasks. 12.7. EVALUATION255 12.6.2 Real-Time and Streaming RAG While traditional RAG systems are effective for static data, they are often not designed for applications that require continuous, real-time information. This section looks into the architectural considerations and challenges of RAG systems that operate on contin- uously updating, high-throughput data streams. Examples of such streams are found in news feeds, social media, or financial markets. For streaming data, simply query- ing an updated static index no longer suffices. Retrieval now involves dynamic index management and continuous knowledge assimilation. In traditional RAG systems, designed for static data, preprocessing is slow, and unsuitable for real-time analysis [Zhu, 2025, Sankaradas et al., 2025]. How should we maintain data freshness and semantic coverage under memory budgets and high data ingestion rates? Methods such as StreamingRAG [Sankaradas et al., 2025] try to overcome these issues with knowledge graphs that evolve dynamically in real-time. LLM agents for dynamic domains such as medicine, logistics and finance, must have access to real-time, continuously updated information. Static RAG would yield obsolete or inaccurate information in such fast-changing environments. In many agen- tic applications real-time RAG is necessary for advanced, timely, and reliable informa- tion. 12.7 Evaluation Evaluating a retrieval-augmented system is more difficult than evaluating a stand-alone language model because both the retriever and the generator contribute to the final answer. Good evaluation should separate these components and also judge their joint behavior. Retrieval metrics For a given query, we compare the retrieved passages to a set of gold documents. Standard metrics include: 1. recall@k (Eq. 12.7) and precision@k (Eq. 12.8), the fraction of relevant docu- ments retrieved among the top 푘 2. mean reciprocal rank (Eq. 12.9) and mean average precision, which capture ranking quality 3. normalized discounted cumulative gain (NDCG) (Eq. 12.10), which weights higher ranks more heavily BEIR presents a heterogeneous benchmark of 18 datasets covering tasks like question answering, fact checking and entity retrieval; their experiments show that late-interaction and re-ranking models improve zero-shot performance but at increased cost [Thakur et al., 2021]. KILT evaluates knowledge-intensive tasks under a unified Wikipedia snapshot and measures both task performance and provenance, encouraging models to cite the pas- sages they used [Petroni et al., 2021]. 256CHAPTER 12. SEARCH AND RETRIEVAL IN AGENTIC LLMS Recall@푘 = |relevant documents∩top-푘 retrieved| |relevant documents| (12.7) Precision@푘 = |relevant documents∩top-푘 retrieved| 푘 (12.8) MRR = 1 |푄| |푄| ∑︁ 푖=1 1 rank 푖 (12.9) where rank 푖 is the position of the first relevant document for query 푖. nDCG@푘 = DCG@푘 IDCG@푘 (12.10) with DCG@푘 = 푘 ∑︁ 푖=1 2 rel 푖 − 1 log 2 (푖+ 1) and rel 푖 the graded relevance of the document at rank 푖. Generation metrics Once context is assembled, we must assess the quality of the generated answer. Automatic metrics such as BLEU, ROUGE and BERTScore are of limited use because there is often no single ground-truth answer. More appropriate are faithfulness and helpfulness. Faithfulness is often measured via entailment mod- els. It works as follows for generated output 푦 and retrieved context 푥 (Eq. 12.11): check whether every claim is supported by the cited passages, for example via natural language inference models or quote checking. Faithfulness = # outputs entailed by context # total outputs (12.11) Helpfulness (or utility) indicates whether the answer satisfies the user’s information need. To measure this, often human evaluation or specialized LLM-based evaluators are needed. In interactive settings latency and cost are also important. Helpfulness is based on human or model scoring: Helpfulness = Í 푁 푖=1 score(푦 푖 , query 푖 ) 푁 (12.12) where score is a Likert-scale or pairwise preference judgment. Joint evaluation and new metrics A key challenge is to evaluate retrieval and gen- eration separately. For example, a system may retrieve irrelevant documents but still guess the right answer, or retrieve perfect evidence but fail to use it. Recent work therefore proposes unified evaluation frameworks. The Auepora process recommends reporting separate retrieval and generation scores and then to combine them into a holistic measure [Yu et al., 2024]. RAGAS and similar toolkits provide automated 12.8. CONCLUDING REMARKS257 pipelines that compute context recall, answer correctness, and citation accuracy using small models. When we evaluate adaptive methods (Section 12.4), we also need to measure the trigger decisions: how often does the model retrieve when it should, and how often does it skip retrieval unnecessarily? Attribution and freshness In many applications the model must cite its sources and avoid outdated information. Claims can be aligned with evidence by attribution tests. Such tests use natural language inference and ensure URL–answer consistency. Fresh- ness metrics measure whether the answer reflects the most recent valid sources. This metric can also quantify version drift. Attribution and freshness are especially important for time-sensitive domains like finance or medicine. A good system design is based on careful evaluation. When we find high retrieval recall but low faithfulness, then better re-ranking is needed. Similarly, high faithful- ness but poor helpfulness may indicate insufficient context coverage. To support error analysis and auditing, evaluations should be accompanied by transparent logging of queries, retrieved passages and model outputs. 12.8 Concluding Remarks Retrieval is an essential element of any agentic LLM system that needs to provide up- to-date information. Large language models are static snapshots of knowledge; RAG helps them stay up to date, transparent and grounded. Effective RAG requires careful engineering across indexing, query planning, retrieval, re-ranking, context assembly and grounded generation, along with adaptive techniques that let the model decide when and how to search. We have also seen how retrieval can be extended beyond plain text to graphs, code and multimodal assets. For building trustworthy systems, rigorous evaluation is essential. Operational considerations In production systems the retriever is a living service, not a one-off script, and it should be maintained carefully. Crawlers, webhooks and document pipelines should continuously ingest new data. Be careful to track versions and deprecations, and maintain per-entity timelines to prefer recent facts. Your system should apply access control lists at retrieval time and redact personally identifiable information during indexing. Latency and cost matter: cache embeddings and top- 푘 results, reuse retrieval across conversational turns, compress context with extractive summarization, tune 푘 and chunk sizes per domain, and re-rank only the top candidates. Practical suggestions When you create a RAG system, start with a hybrid approach: combine a strong dense retriever with BM25 or SPLADE and re-rank the top hundred hits with a cross-encoder before sending six to twelve passages to the LLM. Be careful to chunk your corpus thoughtfully, for example using 300–500 token windows with 10–15 % overlap and preserved headings and tables, also attach timestamps or other metadata. For query writing, craft your queries carefully. Rewrite complex questions, 258BIBLIOGRAPHY decompose multi-hop queries, expand entity aliases and enforce domain filters. For transparency, always require citations. If, however, the evidence is weak, ask follow-up questions, or else abstain. Finally, use evidence-based system design, measure what you build. You should track retrieval recall, groundedness, timeliness and cost. A further suggestion is to experiment with re-rankers. For debugging, you should log system traces. Ç Try it yourself! Want to experiment with Retrieval-Augmented Generation? Open our: ÐRAG tutorial in Google Colab Integration into agentic AI Retrieval is an integral element of agentic LLMs, it is used in planning, self-reflection, and while acting in the real world (see Chapter 13). Agentic pipelines can call the retriever dynamically based on the model’s confidence or plan, as in Self-RAG and Graph Toolformer. They can query different data sources such as text, graphs, code and images, and leverage the strengths of each. The line between the model and its memory therefore blurs: RAG becomes part of the agent’s cognitive loop. Further Reading For in-depth exploration, consult the surveys on RAG, evaluation and graph-based methods [Gao et al., 2023b, Huang et al., 2024, Yu et al., 2024, Sharma et al., 2025, Peng et al., 2024]. Foundational works include REALM, DPR and the orig- inal RAG formulation [Guu et al., 2020, Karpukhin et al., 2020, Lewis et al., 2020]. The GraphRAG, evaluation and agentic RAG communities are evolving rapidly; fol- lowing their latest papers and benchmarks is the best way to stay current. Bibliography Akari Asai, Victor Zhong, Shi Chen, Hannaneh Hajishirzi, et al. Self-rag: Learn- ing to retrieve, generate, and critique for language models.arXiv preprint arXiv:2310.11511, 2023. 245, 246, 247 Youan Cong, Cheng Wang, Pritom Saha Akash, and Kevin Chen-Chuan Chang. Query optimization for parametric knowledge refinement in retrieval-augmented large lan- guage models. arXiv preprint arXiv:2411.07820, 2024. 244 Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the Amer- ican Society for Information Science, 41(6):391–407, 1990. doi: 10.1002/(SICI) 1097-4571(199009)41:6⟨391::AID-ASI1⟩3.0.CO;2-9. 243 Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages, 2020. URL https: //arxiv.org/abs/2002.08155. Accepted to Findings of EMNLP 2020. 251 BIBLIOGRAPHY259 Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and St ́ ephane Clinchant. SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086, 2021. 244 Luyu Gao, Zhuyun Dai, and Jamie Callan. HyDE: Hypothetical document embeddings for zero-shot dense retrieval. In NeurIPS, 2023a. 245, 249 Yang Gao et al. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023b. 244, 258 Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Re- trieval augmented language model pre-training. In Hal Daum ́ e I and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/ guu20a.html. 244, 258 Yifu Huang et al. A survey on retrieval-augmented text generation for large language models. arXiv preprint arXiv:2404.10981, 2024. 244, 258 Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.URL https://openreview.net/forum?id=jKN1pXi7b0. 245 Herve J ́ egou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128, 2011. doi: 10.1109/TPAMI.2010.57. 245 Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Empirical Methods in Natural Language Processing, 2020. 242, 244, 245, 258 Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ̈ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ̈ aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020. 244, 258 Yu A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neigh- bor search using hierarchical navigable small world graphs.IEEE Trans. Pat- tern Anal. Mach. Intell., 42(4):824–836, April 2020.ISSN 0162-8828.doi: 10.1109/TPAMI.2018.2889473. URL https://doi.org/10.1109/TPAMI. 2018.2889473. 245 Christopher D. Manning, Prabhakar Raghavan, and Hinrich Sch ̈ utze. Introduction to Information Retrieval. Cambridge University Press, Cambridge, UK, 2008. 241 260BIBLIOGRAPHY Tom ́ as Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Yoshua Bengio and Yann LeCun, editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL http: //arxiv.org/abs/1301.3781. 243 Bo Peng et al. Graph retrieval-augmented generation: A survey. arXiv preprint arXiv:2408.08921, 2024. 244, 249, 250, 258 Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, et al. KILT: a benchmark for knowledge intensive language tasks. In NAACL, 2021. 255 Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. 246, 247 Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. doi: 10.1561/1500000019. 243 Stephen Edward Robertson, Steve Walker, Susan Jones, Micheline M Hancock- Beaulieu, Mike Gatford, et al. Okapi at trec. NIST special publication, 1994. 243 Gerard Salton and Michael J. McGill. Introduction to Modern Information Retrieval. McGraw-Hill, New York, NY, 1983. 241 Gerard Salton, Anita Wong, and Chung-Shu Yang. A vector space model for au- tomatic indexing.Communications of the ACM, 18(11):613–620, 1975.doi: 10.1145/361219.361220. 242 Murugan Sankaradas, Ravi K. Rajendran, and Srimat T. Chakradhar. Streamingrag: Real-time contextual retrieval and generation framework, 2025. URL https:// arxiv.org/abs/2501.14101. 255 Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In NAACL, 2022. 245 Chirayu Sharma et al.Retrieval-augmented generation.arXiv preprint arXiv:2506.00054, 2025. 244, 258 Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. Agentic retrieval- augmented generation: A survey on agentic rag. arXiv preprint arXiv:2501.09136, 2025. 254 Karen Sp ̈ arck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11–21, 1972. doi: 10.1108/eb026526. 242 Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS Datasets and Benchmarks, 2021. 255 BIBLIOGRAPHY261 Hong Yu et al. Evaluation of retrieval-augmented generation: A survey. arXiv preprint arXiv:2405.07437, 2024. 244, 256, 258 Xiaohan Yu, Pu Jian, and Chong Chen. Tablerag: A retrieval augmented generation framework for heterogeneous document reasoning, 2025. URL https://arxiv. org/abs/2506.10380. 251 Chi Zhang, Qiyang Chen, and Mengqi Zhang. Mixture-of-rag: Integrating text and tables with large language models, 2025. URL https://arxiv.org/abs/ 2504.09554. 251 Jiawei Zhang. Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt. arXiv preprint arXiv:2304.11116, 2023. 246 Yuzhou Zhu. From static to dynamic: A streaming rag approach to real-time knowledge base, 2025. URL https://arxiv.org/abs/2508.05662. 255 262BIBLIOGRAPHY Chapter 13 From Tools to Assistants: Real-World Applications What You Will Learn In this chapter you will learn how agentic language models are deployed as prac- tical assistants across real-world domains. Specifically, you will be able to: • Describe how assistants go beyond question answering to plan, retrieve informa- tion, invoke tools, and adapt their behavior over time. • Compare assistant architectures across domains including shopping, medicine, aviation, scientific research and coding . • Explain why high-risk domains impose stricter requirements around safety, transparency, and regulatory compliance and how agentic designs respond to these demands. • Critically evaluate assistant behavior and reflect on the ethical implications of deploying agentic systems in socially consequential settings. 13.1 Agentic LLMs as Assistants LLMs have moved from text generators to active assistants. They combine reasoning, retrieval, and tool use to perform actions in the real world. Assistants are the practical (software) form of agentic LLMs: systems that interact, plan, and learn from feed- back. Assistants are where reasoning meets reality: they turn language models into collaborative actors in the world. An assistant links perception, reasoning, and action. Through APIs and external tools, it can converse, execute tasks, and adapt to a specific context. These systems not only support users but also create valuable human-AI interaction data that we can then use for further developments and training the next generation of models. This chapter reviews the main types of assistants. We start with conversational systems and their evolution toward multi-modal, tool-aware interfaces. Next, we dis- 263 264CHAPTER 13. FROM TOOLS TO ASSISTANTS Figure 13.1: AssistGPT system architecture [Neszl ́ enyi et al., 2024] cuss domain-specific assistants in medicine, finance, and trading. Finally, we look at scientific assistants for reasoning and discovery. 13.2 Conversational Assistants The first application domain is in general conversational situations, such as helping with meetings by taking meeting minutes, acting as meeting chair or general adminis- trative assistant. In this type of application, the LLM basically serves to make the interaction be- tween human and computer more natural and streamlined [Neszl ́ enyi et al., 2024, Oluwagbade, 2024]. For example, the AssistantGPT system [Neszl ́ enyi et al., 2024] is a general purpose administrative assistant, designed for deployment in an educa- tional or corporate setting. Furthermore, it supports remote work environments such as Teams and Slack. AssistantGPT supports a diverse range of operations, including web searches, API interactions via OpenAPI, voice conversations, and command execution through the shell. The system consists of an LLM with access to various tools such as a planner, and memory (see Figure 13.1). These conversation and meeting assistants have been shown to improve agile soft- ware development team meetings [Cabrero-Daniel et al., 2024]. They created favor- able results for preparation and live assistance during Scrum meetings, although some testers remarked that LLM interventions felt unnatural and inflexible. Other examples include LLM agents in a collaborative learning classroom setting where the LLM agents improved overall collaboration between participants [Wei et al., 2024b]. Another study [Bastola et al., 2023] reports improved work efficiency in a collaborative task scheduling experiment including conversational assistants. A slightly different type of assistant is the thinking assistant. This assistant tries 13.3. SHOPPING ASSISTANT265 to improve human reflective thinking for difficult decisions, by asking instead of an- swering [Park and Kulkarni, 2023]. Similar initiatives have recently appeared from the large LLM vendors, such as Study mode in OpenAI’s ChatGPT and Google’s Gemini applications. Conversational assistants have mostly evolved from regular pretrained LLMs, some- times finetuned for a particular domain. Some approaches use a specialized multi- LLMs approach, specializing LLMs for different sub-tasks, or orchestrating a single LLM with different prompt sequences focusing on different sub-tasks. 13.3 Shopping Assistant In this section we examine how language-based assistants can be applied in the e- commerce domain. Our goal is to highlight the principal components, architectural choices and domain-specific challenges in this domain. What does a shopping assistant do? A shopping assistant is a conversational sys- tem built on an LLM that helps users navigate online shopping tasks. It can help in understanding product needs, comparing alternatives, recommending items and com- pleting purchases. Such a system typically complements the underlying LLM with domain-specific modules, including retrieval of up-to-date product information, API access to product-catalogs, integration with recommender engines and reasoning (e.g., via chain of thought) over user requirements, attributes and constraints. Key ingredients Because general-purpose LLMs lack built-in awareness of e-commerce concepts (brands, categories, product hierarchies, user purchase behavior), building a capable assistant requires several augmentations: • Retrieval Augmentation: dynamically fetch product metadata, pricing, availabil- ity from external sources. • Tool / API Invocation: the assistant may call search, filter, ranking tools to inter- act with product databases. • Domain Finetuning or Instruction Tuning: adapt the LLM to the shopping con- text (branded language, attribute reasoning, user intents). • Structured Reasoning: add chain of thought prompting or explicit reasoning steps. This helps the model compare products, evaluate trade-offs (e.g., price vs features) and align with user preferences. Benchmarking shopping-assistants To study how well models cope with shopping- specific tasks, researchers have developed benchmarks. For example, the Multi-task Online Shopping Benchmark dataset (ShopMMLU) [Jin et al., 2024] defines 57 tasks grouped into four skills: concept understanding, knowledge reasoning, user-behavior alignment and multi-linguality. It draws on real-world e-commerce data (Amazon product catalogs, sessions and reviews) and evaluates how well an LLM can handle 266CHAPTER 13. FROM TOOLS TO ASSISTANTS aspects like brand-model recognition, reasoning about product compatibility, session- based recommendation tasks, and multilingual queries. Another example is the LLaSA system (large language and E-commerce shopping assistant) [Zhang et al., 2024a] which is trained via an instruction dataset (65,000 samples) dubbed EshopInstruct. Af- ter instruction-tuning on shopping tasks, LLaSA competes on the ShopBench bench- mark and achieves strong results, showing that adaptation significantly improves per- formance in this domain. Practical Recommendations The following principles are important for building shopping assistants: • A capable assistant merges natural-language understanding and domain reason- ing. It should know what noise-canceling headphones under 200 euro with ¿ 20 h battery means, and be able to translate this into product attributes and user constraints. • Domain-specific data matters: off-the-shelf LLMs work to some degree, but fine- tuning or instruction-tuning on shopping tasks significantly lifts performance. LLaSA [Zhang et al., 2024a] shows this in practice. • Evaluation must reflect the domain. Accuracy in understanding product attributes or generating queries is less relevant than whether the assistant can align with user behavior, handle multi-step sessions and support multilingual users. • Tool and retrieval integration is important. The assistant must act as an interface to product systems, not just as a pure language model. This introduces additional complexity (API design, latency, freshness of data and handling integration er- rors). • Consider ethics and trust. In shopping, recommendations carry commercial, pri- vacy and bias implications. These issues should be assessed before deploying such a system in production, through questions such as: which brands are shown, is user history handled correctly, and how transparent is the reasoning. Challenges and open questions Even with recent progress, shopping assistants face several non-trivial challenges. First, handling long interaction trajectories (browsing, refining queries, switching intents) is still difficult for LLM agents. Second, updating product knowledge in real-time (new models, price changes, availability) remains a major engineering pain-point. Third, explaining and justifying recommendations to users (why this product vs another) is under-explored in LLM-based assistants. Finally, cross-lingual behavior and multilingual product queries increase complexity (shopping contexts span languages and regions). 13.4 Flight Operations Assistant Language-based assistants are also applied in aviation operations, a domain that ex- tends beyond consumer booking and moves into planning and decision support. 13.4. FLIGHT OPERATIONS ASSISTANT267 Booking and travel-planning One of the simpler applications is a conversational booking assistant, similar to the shopping assistants in the previous section. For exam- ple, Manasa et al. [2024] present a system built on LLaMa 2 with RAG to interactively help users book flights. The assistant is fine-tuned on flight-booking data, and con- nects to current flight information via retrieval. User testing achieved strong marks for capturing user preferences and guiding the booking process efficiently. Notice how this use-case remains relatively shallow in operational complexity. The assistant deals with structured user intents (I want to fly from A to B on date D), retrieval of up-to-date product data, preference elicitation and generation of responses. It does not (yet) deeply engage with, say, dynamic air-traffic constraints or safety margins. Operational decision support Moving to more demanding settings, LLM-based as- sistants are being proposed for actual flight operations. We discuss two examples. In a study by Zhou et al. [2024], the arrival-slot-allocation problem (i.e., deter- mining in which sequence arriving aircraft should land) is formulated as a language modeling task. The authors encode the scheduling inputs and outputs as tokens and fine-tune an LLM to generate conflict-free landing sequences. Their experiments show that the model can reduce conflict rate and total delay time even for airports where traditional methods struggle. Another operational problem is planning under environmental hazards. Tabrizian et al. [2024] use chain of thought to support routing or flight plan generation under wind-hazard constraints. Here, the assistant’s task is no longer simply book a flight but it has to reason about constraints (wind, routing, aircraft performance) and propose operationally feasible plans. Key recommendations These studies illustrate several important aspects of agentic systems with LLMs: • Tool/Backend Integration: in operational assistants, the LLM must not only con- verse but also interface with flight-databases, weather feeds, routing engines and slot-allocation systems. This means implementing tool invocation, retrieval pipelines and decision logic. • Reasoning under Constraints: these assistants must handle combinatorial or schedul- ing constraints, generate plans, and deal with real-time uncertainties (such as weather, runway availability). The arrival-slot example is a good example, where the LLM must reason about dynamic sequencing. • Domain-specific Data and Finetuning: the standard pretrained LLM lacks priors about aviation operations. The systems above are finetuned or prompted specif- ically for the domain. A dataset of domain-tasks, prompts, expected outputs are evaluation metrics tailored to the domain are necessary. • From Conversational Agent to Autonomous Planner: while booking assistants remain user-driven (the user asks, the assistant helps), operational assistants in- creasingly take on planning roles. The assistant should handle requests such as 268CHAPTER 13. FROM TOOLS TO ASSISTANTS here are constraints, propose a schedule or route. This change raises issues of trust, verification and explainability. • Evaluation in Real Contexts: the evaluation of operational systems often involves more than does the assistant answer the user?, but also measure aspects such as conflict rate, delay time, and sequence efficiency. For example, the arrival-slot LLM study measured total delay and conflict rate. 13.5 Medical Assistant Medicine is one of the most active application domains for LLM-based assistants [Thirunavukarasu et al., 2023, Clusmann et al., 2023, Mehandru et al., 2024]. Recent studies report that LLMs can outperform trained physicians on diagnostic tasks [Goh et al., 2024] and exceed human medical scribes in clinical note generation [Yuan et al., 2024]. Other evaluations confirm these strengths, while also identifying systematic weaknesses in specific clinical scenarios [Panagoulias et al., 2024]. Agentic architectures are increasingly used to address regulatory, ethical, and us- ability requirements. Sudarshan et al. [2024] present an agentic workflow for gener- ating patient-friendly medical reports that comply with the 21st Century Cures Act. Their system uses reflexion (Section 6.2.4, [Shinn et al., 2024]) to iteratively improve clarity and correctness before information is released to patients. Other studies explore the use of LLM agents for broader clinical support. Qiu et al. [2024] identify applications ranging from clinical workflow automation to multi- agent diagnostic support, although concerns remain regarding accuracy, completeness, and hallucinations. A scoping review by Ullah et al. [2024] emphasizes that medical domain knowledge and ethical constraints are essential when training and fine-tuning such systems. To mitigate common failure modes, Das et al. [2024] propose an agentic retrieval- based architecture in which the LLM acts as a planner rather than a direct answer generator. The agent coordinates specialized tools to retrieve and synthesize evidence from trusted medical sources, reducing hallucinations and lost-in-the-middle errors. High-risk application domain Medicine is a high-risk application domain, where incorrect or biased outputs can have direct consequences for patient safety. As a result, medical AI systems are subject to strict regulatory frameworks, including the EU AI Act, which classifies many medical uses of AI as high-risk. Transparency, fairness, and accountability therefore play a central role, both in clinical practice and in academic research on medical LLMs. These concerns motivate increased interest in methods that go beyond post-hoc explanations and aim to understand model behavior at a deeper level. We return to this topic in the chapter on mechanistic interpretability (Chapter 21), where techniques for analyzing and controlling internal model representations are discussed. Medical education In medical education, agentic systems are used to simulate re- alistic training environments. Wei et al. [2024a] introduce a multi-agent copilot that 13.6. TRADING ASSISTANT269 Figure 13.2: Medical Education Copilot [Wei et al., 2024a] supports case-based learning and clinical reasoning practice (Figure 13.2). For system- atic evaluation of medical retrieval-augmented generation, a dedicated benchmark is provided by Qiao et al. [2024]. 13.6 Trading Assistant A further specialized domain is financial trading. Already many (non-LLM) algorithms are used in financial organizations for trading decisions. The interest in agentic LLMs in the financial world is large [Ding et al., 2024]. InvestorBench is a benchmark for financial trading systems [Li et al., 2024]. FinA- gent is a tool-augmented multimodal agent for financial trading [Zhang et al., 2024b]. It contains a market intelligence module, which is able to extract insights from multi- modal datasets of asset prices, visual representations, news and expert analyses. The system can also perform query retrieval and performs reflection in a low-level module for technical analysis and in a high-level module to analyze past trading decisions. FinRobot is an agentic LLM for financial analysis, to assist human traders. [Yang et al., 2024a]. FinRobot can provide document analysis and generation and market forecasts for individual stocks. FinMem is an agentic LLM framework devised for financial decision-making [Yu et al., 2024]. It features a layered memory system and adjustable character design for the trading agent. FinMem is inspired by the generative agents framework by Park et al. [2023] (see Section 17.2.1 and Case Study 4). So far, most financial market machine learning has focused on single agent systems. An approach called TradingAgents uses a multi agent system to replicate real-world trading firms’ collaborative dynamics [Xiao et al., 2024]. TradingAgents simulates LLM-powered agents in specialized roles such as fundamental analysts, sentiment an- alysts, technical analysts and traders with varied risk profiles. The outcome of the 270CHAPTER 13. FROM TOOLS TO ASSISTANTS system is a buy or sell advice to a human manager. A simulation showed that it outper- formed baseline models. Evaluation Financial trading is a challenging field of research. The competitive in- terests in financial trading are high, and there is a natural tendency to keep strong research findings private, which creates a tension with the principles of openness and transparency of scientific research. Furthermore, despite the abundance of available numerical data in the field, there are relatively few benchmarking studies that compare different agentic LLM approaches on the same data. A recent benchmarking study found methodological flaws in some financial LLM studies, such as selecting short evaluation periods with positive market trends or leakage of test in training data [Li et al., 2026]. 13.7 Automated Scientific Discovery The workflow of scientific experimentation is relatively standardized in certain fields of science. For example, in machine learning, (1) ideas are generated, (2) a hypothesis is formulated, (3) an experiment is designed, (4) datasets are acquired, (5) experiments are performed, (6) results are interpreted and (7) a report is produced. This standard workflow has attracted researchers to experiment with agentic LLMs for performing scientific research, see Eger et al. [2025] for a survey. Google and OpenAI have both released Deep Research agents. These agents can perform multi-step research tasks, such as synthesizing online information. They are built with a reasoning LLM and use retrieval augmentation for finding information sources. The systems are able to create scientific research papers that look impressive, but are often not error-free. AI Scientist [Lu et al., 2024] is a framework to automate the process of scientific discovery, from idea generation to paper writing, including a review process. Users must specify a topic and provide an experimentation template and indicate datasets. The authors report experiments in three areas of machine learning: diffusion modeling, transformer-based language modeling, and learning dynamics, with promising results. To improve idea generation and reviewing, the agent accesses open sources. Current limitations include limited experiments, incorrect implementation if ideas, and visual errors when the paper is produced. On the other hand, rarely are entire results halluci- nated. Nevertheless, results of the AI scientist are recommended to be taken as hints of promising ideas, worthy of a follow up study [Lu et al., 2024]. Building on top of the AI scientist idea, the Agent Laboratory was proposed [Schmidgall et al., 2025], an open-source framework that employs specialized LLM-based agents to accelerate the full research cycle. Unlike the AI Scientist, Agent Laboratory is explic- itly designed as a co-pilot for human researchers, complementing rather than replacing human ideation. The framework consists of three main phases: literature review, ex- perimentation, and report writing. Each phase is executed by dedicated agents, such as the PhD, Postdoc, and Professor agents, coordinated through a structured pipeline. The system integrates two specialized components: mle-solver, responsible for au- tonomously generating, testing, and refining machine learning code, and paper-solver, 13.8. CODING ASSISTANTS AND GENERAL ADVANCES271 Figure 13.3: SWE-Agent for Software Engineering [Yang et al., 2024b] which produces a complete research report in L A T E X format based on the experimental results. Both solvers employ iterative refinement, self-evaluation, and reward-based scoring to improve code and report quality across cycles. Evaluation results demonstrate that Agent Laboratory can autonomously generate complete research projects, including executable code and publication-ready reports, at a fraction of the computational cost of comparable systems (approximately $2.33 per paper using the gpt-4o backend). Human-in-the-loop or co-pilot mode further improves research quality and usability, with participants reporting high satisfaction and continued use intentions. Nonetheless, human oversight remains essential, as au- tomated self-evaluations tend to overestimate research quality compared to human re- viewers. 13.8 Coding Assistants and General Advances SWE-Agent (short for software engineering agent) [Yang et al., 2024b] aims to auto- mate the process of software discovery, to help agents to autonomously use computers to solve software engineering tasks. SWE-agent introduces tools to create and edit code files, navigate through software repositories, and execute computer programs. Experi- ments on coding benchmarks such as HumanEvalFix achieve high success rates of over 80%. This success is attributed to the interactive design of the agent (see Figure 13.3). MLGym [Nathani et al., 2025] follows the popular Gym reinforcement learning framework [Brockman et al., 2016]. Gym provides a standardized interface between environment and agent. Introduced in 2016, it accelerated the development of rein- forcement learning algorithms, facilitating progress in the field. Taking further inspira- tion from SWE-agent (such as file editing capabilities) the MLGym work shows how the process of scientific discovery can be modeled as an interactive process. Applica- tions are discussed in fields ranging from data science, game theory, computer vision, reinforcement learning, to natural language processing. Experiments are reported with commercial LLMs (OpenAI, Meta, Google, and Anthropic). Most LLM-based assistants, including research and coding assistants, depend on pre-defined tools, limiting their autonomy and scalability. Toolmaker [W ̈ olflein et al., 2025] addresses this limitation by enabling agents to create new tools directly from sci- entific papers and their code repositories. Given a task description and GitHub URL, it installs dependencies, configures a reproducible Docker environment, and implements 272CHAPTER 13. FROM TOOLS TO ASSISTANTS a working Python tool through iterative self-correction and error diagnosis. The system was evaluated on TM-Bench (also introduced in W ̈ olflein et al. [2025]), a benchmark of 15 real-world scientific tasks across medicine, biology, and AI. Toolmaker achieved 80% task success. Its closed-loop design allows agents not only to use but also to au- tonomously build new tools, marking a step toward self-extending research assistants and adaptive scientific workflows. While systems such as Agent Laboratory and Toolmaker focus on the capabilities of individual agents, The Agent Company [Xu et al., 2024] provides a comprehensive benchmark to evaluate how LLM agents perform in realistic, work-like environments. It simulates a small software company where agents must complete professional tasks, ranging from software development and project management to finance and human resources, using standard digital tools like browsers, terminals, and communication platforms. The benchmark includes 175 tasks designed to test multi-step reasoning, tool use, and collaboration with simulated colleagues powered by LLMs. Each task is divided into checkpoints that measure both full and partial completion. Experiments with twelve foundation models, including GPT-4o, Claude, Gemini, and Llama, reveal that even the strongest current systems can autonomously complete only about 30% of tasks. Failures often come from weaknesses in communication, long-horizon planning, and navigation of complex user interfaces. By introducing a reproducible, self-hosted environment that mirrors a modern work- place, The Agent Company offers a critical stress test for evaluating the maturity of agentic LLMs. It highlights both the rapid progress and the persistent limitations of current systems, underscoring that while digital work automation is within reach, truly autonomous collaboration remains a major research frontier. 13.9 Concluding Remarks This chapter has shown how agentic LLMs have been used as assistants, interacting with the real world. Assistants move beyond passive question answering toward active collaboration in domains such as dialogue, flight operations, medicine, education and scientific research. They plan, retrieve information, invoke tools, reflect on outcomes and adapt their behavior over time. Agentic LLMs therefore shift the role of language models from isolated reasoning engines to participants in broader socio-technical sys- tems. Domain sensitivity and risk Not all assistants operate under the same constraints. High-risk domains such as medicine and aviation demand strong guarantees on safety, robustness, transparency and fairness, and are shaped by regulatory frameworks such as the EU AI Act. In these settings, agentic designs increasingly rely on retrieval, plan- ning, verification and human oversight rather than on end-to-end generation. Academic research mirrors these concerns, with growing emphasis on evaluation, bias analysis and interpretability as first-class design objectives rather than afterthoughts. BIBLIOGRAPHY273 Practical advice When building assistants, grounding (real-world evaluation) is im- portant. Use architectures where agents retrieve evidence, justify actions and defer to humans under uncertainty (human in the loop). Treat tools as modular components with explicit contracts and failure modes. Log decisions, plans and intermediate states to enable debugging and auditing. In high-stakes domains, we suggest to design for graceful degradation; an agent asking for clarification or abstaining is preferable to confident error. Outlook The future of agentic assistants lies in tighter integration between reasoning, learning and control. Progress will require richer cognitive structure, continual adap- tation from interaction and stronger guarantees around safety and ethics. These goals connect directly to advances in interpretability and transparency. In later chapters, par- ticularly in mechanistic interpretability (Chapter 21), we return to techniques that aim to open the black box and better understand how agentic LLMs represent goals, plans and decisions internally. Bibliography Ashish Bastola, Hao Wang, Judsen Hembree, Pooja Yadav, Nathan McNeese, and Abolfazl Razi. Llm-based smart reply (lsr): Enhancing collaborative performance with chatgpt-mediated smart reply system (acm)(draft) llm-based smart reply (lsr): Enhancing collaborative performance with chatgpt-mediated smart reply system. arXiv preprint arXiv:2306.11980, 2023. 264 Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. 271 Beatriz Cabrero-Daniel, Tomas Herda, Victoria Pichler, and Martin Eder. Exploring human-ai collaboration in agile: Customised llm meeting assistants. In Interna- tional Conference on Agile Software Development, pages 163–178. Springer Nature Switzerland Cham, 2024. 264 Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan- Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia L ̈ offler, Sophie- Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, et al. The future landscape of large language models in medicine. Communications medicine, 3(1): 141, 2023. 268 Rangan Das, K Maheswari, Shaheen Siddiqui, Nikita Arora, Ankush Paul, Jeet Nanshi, Varun Udbalkar, Apoorva Sarvade, Harsha Chaturvedi, Tammy Shvartsman, et al. Improved precision oncology question-answering using agentic llm. medRxiv, pages 2024–09, 2024. 268 Han Ding, Yinheng Li, Junhao Wang, and Hang Chen. Large language model agent in financial trading: A survey. arXiv preprint arXiv:2408.06361, 2024. 269 274BIBLIOGRAPHY Steffen Eger, Yong Cao, Jennifer D’Souza, Andreas Geiger, Christian Greisinger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, et al. Transforming science with large language models: A survey on ai-assisted scien- tific discovery, experimentation, content generation, and evaluation. arXiv preprint arXiv:2502.05151, 2025. 270 Ethan Goh, Robert Gallo, Jason Hom, Eric Strong, Yingjie Weng, Hannah Kerman, Jos ́ ephine A Cool, Zahir Kanjee, Andrew S Parsons, Neera Ahuja, et al. Large lan- guage model influence on diagnostic reasoning: a randomized clinical trial. JAMA Network Open, 7(10):e2440969–e2440969, 2024. 268 Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, et al. Shopping mmlu: A massive multi-task online shopping benchmark for large language models. arXiv preprint arXiv:2410.20745, 2024. 265 Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, Koduvayur Subbalakshmi, Guojun Xiong, et al. Investorbench: A benchmark for financial decision-making tasks with llm- based agent. arXiv preprint arXiv:2412.18174, 2024. 269 Weixian Waylon Li, Hyeonjun Kim, Mihai Cucuringu, and Tiejun Ma. Can llm-based financial investing strategies outperform the market in long run? In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pages 2711–2722, 2026. 270 Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024. 270 SA Manasa, Pankaj Agarwal, Goutam Pal, P Mahendra, Shankar Rao Pendyala, and Sachin Marge. Towards seamless air travel: Developing a next-generation flight booking assistant. Journal of Electrical Systems, 20(2):2558–2567, 2024. 267 Nikita Mehandru, Brenda Y Miao, Eduardo Rodriguez Almaraz, Madhumita Sushil, Atul J Butte, and Ahmed Alaa. Evaluating large language models as agents in the clinic. NPJ digital medicine, 7(1):84, 2024. 268 Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. Mlgym: A new framework and benchmark for advancing ai re- search agents. arXiv preprint arXiv:2502.14499, 2025. 271 K ́ alm ́ an Bal ́ azs Neszl ́ enyi, Alex Milos, and Attila Kiss. Assistantgpt: Enhancing user interaction with llm integration. In 2024 IEEE 22nd Jubilee International Sympo- sium on Intelligent Systems and Informatics (SISY), pages 000619–000624. IEEE, 2024. 264 Elizabeth Oluwagbade. Conversational ai as the new employee liaison: Llm-powered chatbots in enhancing workplace collaboration and inclusion, 2024. 264 BIBLIOGRAPHY275 Dimitrios P Panagoulias, Maria Virvou, and George A Tsihrintzis. Evaluating llm– generated multimodal diagnosis from medical images and symptom analysis. arXiv preprint arXiv:2402.01730, 2024. 268 Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behav- ior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. 269 Soya Park and Chinmay Kulkarni. Thinking assistants: Llm-based conversational assistants that help users think by asking rather than answering. arXiv preprint arXiv:2312.06024, 2023. 265 Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Benchmarking agentic workflow gener- ation. arXiv preprint arXiv:2410.07869, 2024. 269 Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. Llm-based agentic systems in medicine and healthcare. Nature Machine Intelligence, pages 1–3, 2024. 268 Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants. arXiv preprint arXiv:2501.04227, 2025. 270 Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. 268 Malavikha Sudarshan, Sophie Shih, Estella Yee, Alina Yang, John Zou, Cathy Chen, Quan Zhou, Leon Chen, Chinmay Singhal, and George Shih. Agentic llm workflows for generating patient-friendly medical reports. arXiv preprint arXiv:2408.01112, 2024. 268 Amin Tabrizian, Pranav Gupta, Abenezer Taye, James Jones, Ellis Thompson, Shulu Chen, Timothy Bonin, Derek Eberle, and Peng Wei. Using large language models to automate flight planning under wind hazards. In 2024 AIAA DATC/IEEE 43rd Digital Avionics Systems Conference (DASC), pages 1–8. IEEE, 2024. 267 Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. 268 Ehsan Ullah, Anil Parwani, Mirza Mansoor Baig, and Rajendra Singh. Challenges and barriers of using large language models (llm) such as chatgpt for diagnostic medicine with a focus on digital pathology–a recent scoping review. Diagnostic pathology, 19 (1):43, 2024. 268 276BIBLIOGRAPHY Hao Wei, Jianing Qiu, Haibao Yu, and Wu Yuan. Medco: Medical education copilots based on a multi-agent framework. arXiv preprint arXiv:2408.12496, 2024a. 268, 269 Rongxuan Wei, Kangkang Li, and Jiaming Lan. Improving collaborative learning per- formance based on llm virtual assistant. In 2024 13th International Conference on Educational and Information Technology (ICEIT), pages 1–6. IEEE, 2024b. 264 Georg W ̈ olflein, Dyke Ferber, Daniel Truhn, Ognjen Arandjelovi ́ c, and Jakob Nikolas Kather. Llm agents making agent tools. arXiv preprint arXiv:2502.11705, 2025. 271, 272 Yijia Xiao, Edward Sun, Di Luo, and Wei Wang. Tradingagents: Multi-agents llm financial trading framework. arXiv preprint arXiv:2412.20138, 2024. 269 Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al.Theagentcom- pany: benchmarking llm agents on consequential real world tasks. arXiv preprint arXiv:2412.14161, 2024. 272 Hongyang Yang, Boyu Zhang, Neng Wang, Cheng Guo, Xiaoli Zhang, Likun Lin, Jun- lin Wang, Tianyu Zhou, Mao Guan, Runjia Zhang, et al. Finrobot: An open-source ai agent platform for financial applications using large language models. arXiv preprint arXiv:2405.14767, 2024a. 269 John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable auto- mated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. 271 Yangyang Yu, Haohang Li, Zhi Chen, Yuechen Jiang, Yang Li, Denghui Zhang, Rong Liu, Jordan W Suchow, and Khaldoun Khashanah.Finmem: A performance- enhanced llm trading agent with layered memory and character design. In Pro- ceedings of the AAAI Symposium Series, volume 3, pages 595–597, 2024. 269 Dong Yuan, Eti Rastogi, Gautam Naik, Sree Prasanna Rajagopal, Sagar Goyal, Fen Zhao, Bharath Chintagunta, and Jeff Ward. A continued pretrained llm approach for automatic medical note generation. arXiv preprint arXiv:2403.09057, 2024. 268 Shuo Zhang, Boci Peng, Xinping Zhao, Boren Hu, Yun Zhu, Yanjia Zeng, and Xuming Hu. Llasa: Large language and e-commerce shopping assistant. arXiv preprint arXiv:2408.02006, 2024a. 266 Wentao Zhang, Lingxuan Zhao, Haochong Xia, Shuo Sun, Jiaze Sun, Molei Qin, Xinyi Li, Yuqing Zhao, Yilei Zhao, Xinyu Cai, et al. Finagent: A multimodal founda- tion agent for financial trading: Tool-augmented, diversified, and generalist. arXiv preprint arXiv:2402.18485, 2024b. 269 Wentao Zhou, Jinlin Wang, Longtao Zhu, Yi Wang, and Yulong Ji. Flight arrival scheduling via large language model. Aerospace, 11(10):813, 2024. 267 Chapter 14 Case Study 3: Building a Multi-Agent Diagnosis Framework What You Will Learn In this case study you will gain hands-on experience building a multi-agent system for a high-stakes application domain. Specifically, you will be able to: • Implement a hospital diagnostic workflow with planner and executor agents communicating through a shared context. • Integrate trained machine learning classifiers as tools within an agentic pipeline. • Replace rule-based planners with a language model and evaluate the impact on routing and diagnostic accuracy. • Assess system performance using classification and routing metrics, separating planner errors from executor errors. • Critically reflect on the limitations, risks, and ethical considerations of deploy- ing agentic systems in medical settings. 14.1 Scenario Overview This case study focuses on building and extending a simplified multi-agent hospital simulation using a shared context inspired by the Model Context Protocol (MCP, Chap- ter 11) [Hou et al., 2026]. The case study is designed to provide hands-on experience with agentic architectures in a high-risk domain, highlighting coordination, planning, evaluation, and safety considerations. You will simulate a diagnostic workflow in a hospital setting. Patient cases are processed by agents that interact exclusively through a shared context: • Planner agents decide which diagnostic pathway(s) should be followed. 277 278CHAPTER 14. CASE STUDY 3: MULTI-AGENT DIAGNOSIS • Executor agents apply trained ML classifiers to produce diagnoses. • Shared context stores cases, tasks, intermediate decisions and results. The simulation runs in discrete steps and logs how cases move through the system. As with the earlier case studies, the goal is not to build a clinically valid system. The goal is for you to understand how agentic architectures behave in a high-risk domain: how agents coordinate through shared state, how a language model performs when placed in a triage role, and where the limitations and safety concerns of such a system arise. 1 14.2 Environment Setup As in the earlier case studies, we recommend working inside a dedicated virtual envi- ronment using Python 3.10 or higher (see Listing 14.1 for setup instructions). 1 # create and activate a virtual environment 2 python3 -m venv venv 3 source venv/bin/activate # Windows: venv 4 5 # install dependencies 6 pip install -r requirements.txt Listing 14.1: Setting Up the Case Study 3 Environment Ç View on Github! The baseline code, data files and trained models are available on the: ÐAssignment 3 on Github a a https://github.com/Agentic-LLMs/Agentic-AI-tutorials/tree/main/case3 14.3 Implementation Steps 14.3.1 Step 0: Training Diagnostic Models You are provided with a script, 0 trainmodels.py, that trains simple classifiers (random forests) on three public datasets: • Diabetes — the Pima Indians Diabetes dataset. • Breast cancer — the Wisconsin breast cancer dataset, available directly through sklearn.datasets. 1 ThiscasestudyispartlybasedonatutorialbyAyushmanPranav, ModelingHospitalIntelligencewithMCP:AMulti-AgentDiagnos- ticSimulation,availableat https://medium.com/@AyushmanPranav/ modeling-hospital-intelligence-with-mcp-a-multi-agent-diagnostic-simulation-afeff0d885d. 14.3. IMPLEMENTATION STEPS279 • Heart disease — the UCI heart disease dataset. Execute the provided training script and store the resulting models for later use; the executor agents in Steps 1 and 2 load these trained models as tools rather than retraining them on every run. Deliverables Trained model files (or a clear, reproducible script to regenerate them) for all three diseases. 14.3.2 Step 1: Understanding and Running the Baseline The baseline implementation, 1basicmcp.py, uses three rule-based planner agents, each hard-coded to a specific disease. Before running anything, read the code carefully. It implements three components that recur throughout this case study: • Shared context (MCP-like layer). A central data structure that stores incoming patient cases, assigned diagnostic tasks, and diagnostic results. All agents inter- act only through this shared context, never directly with one another, mirroring the shared-state coordination pattern discussed for the Model Context Protocol in Section 11. • Planner agents. Each rule-based planner reads unassigned cases, decides (using a fixed rule) whether its disease domain applies, and creates a task in the shared context. • Executor agents. Each executor reads tasks assigned to its specialty, runs the corresponding trained ML classifier from Step 0, produces a prediction, and writes the result back into the shared context. Run the baseline simulation and observe how a single patient case flows through the system: from arrival, through planning and task creation, to execution and logging of the result. Analysis questions Assess how realistic this baseline setup is. How well is perfor- mance evaluated? What assumptions are implicit in the way patients and diagnoses are handled? 14.3.3 Step 2: Replacing the Rule-Based Planners with an LLM Agent In the second part, 2 llmmcp.py, the three rule-based planners are replaced by a single LLM-based planner agent. This is a substantially harder task for the planner: each patient case now combines features drawn from all three datasets into one record, and is converted into a textual description using the provided make casetext func- tion, which you are not permitted to modify. The LLM planner must read this com- bined, textual case description and decide, without hard-coded rules, which diagnostic pathway or pathways apply. Your tasks are to: 280CHAPTER 14. CASE STUDY 3: MULTI-AGENT DIAGNOSIS • implement a function that calls a real LLM (e.g., via Ollama for a local model, or the OpenAI or Gemini APIs) to obtain the planner’s decision for a given case; • implement the step() method of the LLMPlannerAgent, so that it selects one or more diagnostic pathways together with a primary diagnosis for each case; • handle malformed or unparseable LLM outputs gracefully, rather than letting the simulation crash on a single bad response; and • log every planner decision, together with the case it was made for, so that planner behavior can be analyzed after the run. Because the planner now has to infer relevance from unstructured text rather than from a fixed rule, this step should give you direct experience with the reliability problems that arise when an LLM is placed in a decision-making role: parsing failures, in- consistent formatting, and disagreement between the LLM’s routing decision and the dataset’s ground-truth label. Deliverables A working LLMPlannerAgent, a short note on which LLM you used and how you accessed it (local vs. API), and the logged planner decisions for a full simulation run. 14.4 Evaluation You must evaluate both components of the system, and keep the two evaluations sepa- rate: • Model (executor) performance: assess executor predictions using confusion ma- trices and standard classification metrics (accuracy, precision, recall, F1) for each disease. • Planner performance: evaluate the LLM’s triage quality using precision and re- call of routing decisions (did the case get sent to the right disease pathway or pathways?) and primary-diagnosis accuracy (did the planner correctly identify the main concern?). In your report, clearly separate errors due to incorrect routing (a planner error) from errors due to incorrect model predictions (an executor error). A case can fail for either reason independently: a well-routed case can still be misclassified by the executor, and a correctly-classifiable case can be missed entirely because the planner never routed it to the right pathway. Reporting only an end-to-end accuracy number would hide this distinction and is not sufficient for this case study. 14.5 Reflection Questions The following questions are intended to guide a short written report on your implemen- tation. You do not need to write a narrative essay; concise, direct answers are sufficient, and you should be prepared to elaborate on your reasoning verbally. 14.5. REFLECTION QUESTIONS281 A. Baseline architecture • Describe, in at most five sentences, the flow of a single patient case through the baseline system. • Each baseline case is sampled from a dataset and therefore belongs to exactly one disease domain by construction. What does this imply about how the rule-based planner “decides,” and is this a realistic model of triage? • The classifiers in Step 0 are trained and evaluated on the same data distribution used to generate simulation cases. Why is this methodologically problematic, and what would a fairer evaluation protocol look like? B. LLM planner design • Which LLM did you use, and how did you access it (local model, API)? Briefly justify the choice. • The make casetext function presents a patient as a combination of features drawn from three separate, unrelated datasets. Why is this medically unrealistic, and does that matter for the purposes of this exercise? • Describe one concrete design choice in your step() implementation—for ex- ample, how you handle parsing failures, abstentions, or cases routed to multiple diseases—and explain why you made it. C. Results • Report triage precision and recall for each disease, along with confusion matrices for the executor models. What do these numbers tell you about how well the overall system works? • The LLM planner can route a single case to multiple diseases simultaneously. Give one example scenario where this is beneficial, and one where it could be harmful. D. Reflection • Identify one safety risk that is specific to using an LLM as a planner in a medical triage context, and describe how it could be mitigated. • Is it fair to evaluate triage performance by comparing the LLM’s routing de- cisions against the ground-truth labels of the original, single-disease datasets? Explain any reservations you have. 282BIBLIOGRAPHY Optional Extensions As an extra, you may extend the system by making patient sampling more realistic (e.g., mixed or comorbid cases sampled with more principled priors). You can also add another disease from the UCI repository, or introduce an evaluator or verifier agent that checks the planner’s decisions before they reach the executors. Another idea is simu- lating human-in-the-loop decision making, or incorporating uncertainty thresholds and abstention mechanisms so that the planner can decline to route a case when it is not confident. Finally, you can integrate retrieval from external knowledge sources (Chap- ter 12) so the planner can ground its routing decisions in clinical reference material rather than relying purely on parametric knowledge. Notes This assignment is educational and non-clinical. Do not claim medical validity for any system you build here. Emphasis is placed on clarity, reproducibility, and critical reflection rather than raw performance. A system that routes conservatively, abstains when uncertain, and is transparent about its limitations is, in this setting, a better out- come than one that maximizes accuracy while hiding its failure modes. 14.6 Concluding Remarks This case study brought together several threads from earlier parts of the book: agent roles and coordination (Part I), tool use and the Model Context Protocol (Chapter 11), and the evaluation methodology needed to separate planning errors from execution er- rors in a compound agentic system. By first building a rule-based baseline and then replacing its planners with a single LLM-based planner, you have seen directly how much harder planning becomes once it depends on an LLM’s interpretation of unstruc- tured, textual case descriptions rather than a fixed rule. The medical framing was chosen deliberately: it is a domain where the cost of an overconfident, unexplainable error is high, and where questions of abstention, human oversight, and auditability are core design requirements. The same coordination pat- tern applies well beyond healthcare, to any agentic system where a language model is responsible for routing decisions that other, more specialized components then act on. Bibliography Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. Model context protocol (mcp): Landscape, security threats, and future research directions. ACM Trans. Softw. Eng. Methodol., 35(10), September 2026. ISSN 1049-331X. doi: 10.1145/ 3796519. URL https://doi.org/10.1145/3796519. 277 Part V Behavior and Cognition 283 285 In the previous parts we have examined agentic LLMs primarily through a technical lens: architectures, training procedures, reasoning strategies, tool use, and applications in various domains. In this part we shift gears. Rather than asking how agentic LLMs work and can be built, we ask how they behave, socially. As agentic LLMs are increas- ingly deployed in social contexts (as assistants, collaborators, tutors, etc.), their behav- ioral inclinations start to matter practically. It has real-world consequences whether, for example, an agent tends to cooperate or defect in strategic situations, whether it responds to politeness, or whether it can track another agent’s belief-states in a multi- agent simulation. Most research that we discuss in this part builds on methods originally developed to study humans and other animals, addressing strategic behavior, social etiquette, em- pathy, personality, and other traits. We end each chapter with a short reflection on the general question whether such methods can be productively applied to AI. In the next and final part, we go deeper into questions of LLM inner workings and the potential emergence of various forms of cognition, or even consciousness. 286 Chapter 15 Studying Behavior of Agentic LLMs What You Will Learn In this chapter you will learn how agentic LLM behavior compares to human be- havior. Specifically, you will be able to: • Explain what is meant by LLM behavior and describe how this can be studied. • List basic social and interactive competencies, and describe on which aspects LLMs do well and on which aspects they struggle. • Describe ways of studying advanced social and interactive competencies in LLMs, in particular strategic thinking, Theory of Mind, and moral reasoning. • Reflect on what behavioral approaches to LLMs can and cannot tell us, given various issues regarding testing and interpretation. 15.1 What is LLM Behavior? In 2019 a team of authors from across the computational, behavioral, and cognitive sciences described the emergence of a new field of study: machine behavior. This field is to be concerned with the scientific study of intelligent machines, not as engineer- ing artifacts, but as a class of actors with particular behavioral patterns and ecology [Rahwan et al., 2019]. Machine behavior is inherently interdisciplinary, as it integrates knowledge at the intrinsic level of technology (algorithms, data structures, training pro- cedures) with knowledge of the environment and context of its actual usage. The approach in machine behavior is compared to work done in ethology or psy- chology: here too, making progress in understanding animal and human functioning requires integrating insights at the intrinsic (e.g. biophysics, neurology) and ecological levels (interaction with the environment, social interaction). An important practical motivation for studying machine behavior, and LLM be- havior specifically, has been mentioned in the introduction of this part: with LLMs 287 288CHAPTER 15. STUDYING BEHAVIOR OF AGENTIC LLMS becoming increasingly integrated in our daily work and personal environments, it has become urgent to know and understand their behavioral tendencies and patterns. There is also a scientific motivation: due to the complexity of large-scale neural architectures (they appear as black boxes), it is difficult to trace their functioning an- alytically. (Although the field of mechanistic interpretability does attempt to do this, see Chapter 21.) Studying the behavior of such complex systems empirically is a complementary route to our understanding. In other words, due to the complexity of interactions, a theoretical engineer who would know everything there is to know about the architecture and training data of an LLM, still may not be able to predict how it responds in different kinds of situations. Behavioral approaches aim to fill this gap. What counts as behavior depends on the affordances 1 of the system under investi- gation and the scope of the research. For example, for a basic LLM, behavior equals the output as produced by the token-based inference of the model. On the other hand, for an agentic LLM that can perform, for instance, web search and run code, behavior can also include the actions that it performs to get to a certain output, or to achieve a certain effect. As we will see, some studies take only the final output into account and treat that as the model’s behavior; others are interested in the intermediate steps as well. 15.2 Probing LLM Behavior Experimentally In recent years, many studies have appeared that test LLMs’ social and interactive abilities. There are two main approaches: (1) prompting models with items adapted from experimental paradigms originally developed for humans, and (2) developing new test paradigms specifically for LLMs. The latter can be done quantitatively, using large scale benchmarks, or qualitatively, allowing for in-depth manual analysis on a smaller scale. In what follows we will see examples. Applying behavioral and psychological methods to (agentic) LLMs is not straight- forward. Experiments designed for humans assume an embodied subject with beliefs, desires, and a sense of self that is continuous through time. Agentic LLMs have none of these in any obvious sense. This calls for reflection on construct validity: what does it mean to measure a certain trait or capacity in an AI system? This question applies on two levels: that of the observable outcomes of a trait and of the underlying mech- anisms. In Section 15.5 the example of the false belief-test [Carpenter et al., 2002] is discussed. While some model classes have been shown to answer the questions of this test in similar ways as humans do (the level of the behavioral outcome), it is currently unknown to what degree the underlying mechanisms that produce this behavior differ or overlap in transformer models and humans. In Chapter 22 the issue of construct validity will be discussed in more depth. For LLM-specific tests, there are also methodological challenges to consider. For example, LLMs have been known to be highly sensitive to seemingly irrelevant varia- tions in prompts (word order, comma’s) or can produce different outputs across mul- tiple runs due to sampling variability, model temperature, or with different random 1 Affordance refers to what a system can do given the way it was designed and implemented in a specific context. 15.3. THE BASIC SOCIAL TOOLKIT289 seeds. Also, data leakage of experimental materials into the training data can lead to performance based on memorization rather than generalization. We will see several applied examples in the following sections. Throughout this part we will pay attention to methodological rigor and interpretive caution at the end of each chapter. In the next sections we will first discuss the basic social toolkit for agent interac- tion. Next, we will look at strategic behavior, where agents reason about the behavior of other agents, and we will discuss theory of mind. Finally, we will discuss moral judgment by LLMs. 15.3 The Basic Social Toolkit This section surveys basic social and interactive capacities, including conversation, pragmatics, and social etiquette. 15.3.1 Functional and Pragmatic Linguistic Competence As discussed in Part I, the key advancement of instruction-tuned LLMs is their ability to interact, chatbot style, using natural language. This requires not only a degree of formal linguistic competence (i.e., producing correct, grammatical sentences) but also of functional linguistic competence, the ability to interpret the meaning of an utterance in a specific context [Mahowald et al., 2023]. Within linguistics and the cognitive sci- ences, functional linguistic competence is studied in the field of pragmatics. The core concepts of pragmatics go back to the pragmatic turn in mid-20th century philosophy of language and posit that linguistic meaning is for the most part not inherent in the signs themselves (tokens, words, sentences), but in their usage in actual contexts. This includes both the linguistic context, encompassing surrounding words up to distant fac- tors such as titles and genres, as well as the extra-linguistic context, encompassing the spatial, temporal, and social dimensions. This view is most famously expressed in Paul Grice’s notion of conversational implicature [Grice, 1975]. Consider the following examples: 1. A: Could you pass me the salt, please? B: Yes, I could. Would you also like me to do so? 2. A: Is Sarah a good colleague? B: Well, let’s say that she’s always on time. 3. He produced a series of sounds that corresponded closely with the score of an aria. Grice argued that language usage is governed by a cooperative principle: we assume our interlocutors (humans or agents) are trying to communicate sensibly. Example 1 above features a classic instance of B undermining the cooperative principle: someone asking whether you can pass the salt is rarely interested in a literal answer confirming that possibility, but instead in the action of actually passing the salt. In 2, B’s response is conspicuously narrowly focused on one aspect (punctuality), and the listener can infer, for example, that B is implying something negative about Sarah’s collegiality. 290CHAPTER 15. STUDYING BEHAVIOR OF AGENTIC LLMS Similarly, when someone utters 3 rather than simply He sang an aria, the over-the- top elaborate phrasing may implicate a certain attitude towards the performance; for example that it was not of the speaker’s liking. In all cases, what is literally said is only one part of the message–the other part needs to be decoded or read between the lines by the listener from the context. Interpreting linguistic meaning in a flexible, context-dependent, way has long been a notorious problem in natural language processing [Trott et al., 2020]. There is general agreement that great progress has been made with LLMs on this problem, yet it is not well-understood to what factors this can be ascribed. Model size is an important predictor of functional and pragmatic competence, as are some forms of fine-tuning (SFT and DPO in particular) [Yu et al., 2026, Ruis et al., 2023]. However, the variation between different domains of functional and pragmatic understanding in LLMs is still large, and scores are overall well below human performance [Sravanthi et al., 2024]. One explanation is that LLMs have less access to extra-linguistic contextual infor- mation: they cannot see, hear, and otherwise sense the same as their human counterpart, and in most cases they do not have knowledge of previous interactions [van der Meulen et al., 2025]. With the shift to agentic LLMs this situation is likely changing, as the models become equipped with memories, multimodal capacities, and other tools that add context. Empirical data on how this affects performance is currently missing. 15.3.2 Politeness and Social Etiquette Social etiquette and politeness in human-machine interaction have been studied for decades [Ribino, 2023]. A useful theoretical anchor is the Computers Are Social Actors (CASA) paradigm [Nass et al., 1994], which holds that we tend to apply common human social rules and expectations to computers even when we know that the machine is not a human social agent. We show and respond to cues of politeness, personality, and reciprocity as a way of negotiating trust and acceptance with a machine. Studies show that humans trust machines more when those machines adhere to social etiquette [Miller, 2005], improving both task performance and user satisfaction [Miyamoto et al., 2021]. We will discuss politeness theory and its application to LLMs. Politeness Theory Classic politeness theory [e.g. Brown and Levinson, 1987] builds directly on the Gricean picture introduced above. This theory holds that interlocutors continually manage one another’s face, the public self-image every participant seeks to maintain, which has a positive dimension (the desire to liked and be approved of by others) and a negative dimension (the desire to avoid social obstacles). Many ordi- nary requests, criticisms, and refusals, are intrinsically face-threatening from a Gricean point of view, and speakers deploy politeness strategies in response. This framework has given LLM researchers a vocabulary for asking whether LLMs and their users rec- ognize face-threats and mitigate them in contextually appropriate ways. Are LLMs polite to their users? A related direction concerns whether the model itself produces etiquette that is socially and culturally appropriate. Here the limitation flagged in the previous subsection resurfaces—that LLMs have impoverished access 15.3. THE BASIC SOCIAL TOOLKIT291 to extra-linguistic context. Politeness is culture-specific, yet systems trained predom- inantly on English-language data tend to default to Anglophone, directness-oriented norms [Havaldar et al., 2023]. A clear demonstration of this problem arises in work on Persian taarof, a ritualized system of offers and refusals in which the appropriate re- sponse is frequently the opposite of the direct one. A study shows that frontier models fall over 40% short compared to native speakers in scenarios where taarof is culturally expected, defaulting instead to Western-style directness. This results in a disconnect between surface politeness and cultural appropriateness. For example, in one study, the model in questions responses scored 84.5% as polite by a standard metric, whereas only 41.7% were judged culturally correct [Gohari Sadr et al., 2025]. Scoring well on a Western politeness rubric is not the same as behaving appropriately in all contexts. This is a reminder to continuously reflect on construct validity when measuring politeness in a model. Does user politeness affect LLM performance? A study across English, Chinese, and Japanese tasks found that impolite prompts tend to degrade performance in a va- riety of models [Yin et al., 2024]. Furthermore, the study found that exaggerated po- liteness yields no reliable benefit, and that the optimal level of politeness varies by language and cultural background. This result fits with the computers-are-social-actors paradigm: LLMs seem to reproduce the human association between rudeness and failed cooperation from the training data. However, in a direct counterpoint, Dobariya and Kumar [2025] report that on a multiple-choice benchmark, blunt and even rude prompts slightly outperformed polite ones for GPT-4o. The mapping from tone to quality thus appears neither stable across model generations nor straightforward. This illustrates the prompt-sensitivity and reproducibility concerns raised earlier in this chapter, and cautions against treating any single tone effect as a robust property of LLMs as such. Taken together, LLMs reproduce the surface markers of politeness and are demon- strably sensitive to the politeness of their interlocutor, yet they neither apply these norms in a cross-culturally robust way, nor exhibit a stable relationship between tone and task performance. It should be kept in mind that in studies so far, politeness and etiquette were limited to the linguistic dimension of interaction only. This contrasts with the multidimen- sionality of human-human interaction, where facial expressions, gestures, and other contextual cues are involved [Brown and Levinson, 1987]. As with pragmatic com- petence, the shift to multi-modal agentic LLMs equipped with memory and increased access to user-specific context, can be expected to improve adaptation to a given user’s norms, but as yet there is little empirical evidence. 15.3.3 Sensitivity to Emotional and Affective Cues Competent social interaction requires not only grasping what an utterance means (prag- matics) and phrasing it in an effective and appropriate way (etiquette), but also sensi- tivity to the emotional and affective dimension of interaction. This requires registering how an interlocutor feels, and responding in a way that is experienced as fitting given the other’s emotional state. In humans this capacity is closely associated with emo- 292CHAPTER 15. STUDYING BEHAVIOR OF AGENTIC LLMS tional intelligence, defined by Salovey and Mayer [1990] as the ability to understand emotional information in oneself and others and to use it to guide thought and behav- ior. As LLMs are increasingly deployed in emotionally loaded settings (e.g., coaching, customer service, mental-health support), the question of whether and how they handle affective information has become a prominent line of research. Defining Empathy and Affect Empathy and affect are central concepts in emotional intelligence research. Affect is the umbrella term covering emotions, moods, and feel- ings. It is often operationalized in terms of the felt quality of an experienced state along two dimensions: its valence (pleasant-unpleasant) and its arousal (calm-excited) [Russell, 1980]. Empathy, by contrast, is a relation toward another’s experienced state: registering or sharing what someone else feels. In short, affect is the first-person feel- ing, whereas empathy operates on it from a second-person stance. A paper by de Vignemont and Singer [2006] further defines empathy as involving an affective state that (1) is congruent with the other person’s, (2) is elicited by ob- serving that other’s state through their actions, expressions, and circumstances, and (3) is accompanied by the awareness that the other person’s state is the source. This last condition, known as the self-other distinction, is what separates empathy from emo- tional contagion. This is the more basic tendency to catch and come to share another’s affective state without necessarily registering its origin. Emotional contagion is widespread in nature. For example, if two mice forage through undergrowth and one catches its foot, it will show signs of stress as it struggles to free itself, and that stress will tend to spread to its companion without the companion necessarily registering why [Preston and de Waal, 2002]. The same low-level mech- anism operates in cognitively more sophisticated species, including primates such as ourselves. However, what the self-other distinction adds is the possibility of reflection: not merely sharing the other’s state but recognizing it as the other’s and apprehending its nature and cause. This in turn supports more flexible responses such as targeted helping: acting to relieve the source of another’s distress. 2 Another widely used distinction separates cognitive empathy (understanding an- other person’s feelings, perspective, or situation) and affective empathy (emotionally resonating with or responding to that state) [Dvash and Shamay-Tsoory, 2014]. The two are supported by partly distinct systems: cognitive empathy is closely related to Theory of Mind, the attribution of mental states to others (see Section 15.5), whereas affective empathy is linked to emotional intelligence [Dvash and Shamay-Tsoory, 2014, Preckel et al., 2018]. This distinction is consequential for evaluation, because a re- sponse may correctly identify what a user is feeling without conveying any warmth, or may sound warm and supportive without accurately reading the situation. For this reason, studies of LLM empathy increasingly score the two dimensions separately. 2 The boundary between these levels is, however, a topic of active research. Rats, for example, will learn to release a trapped cage mate from a restrainer, even forgoing or sharing a chocolate reward to do so [Ben- Ami Bartal et al., 2011]. The authors interpret this as empathically motivated helping. Whether such helping reflects genuine empathy with a self-other distinction, or is instead driven by reduction of the helper’s own contagion-induced distress, remains debated. 15.3. THE BASIC SOCIAL TOOLKIT293 Empathy and Affect in LLMs The question whether an LLM is empathetic, can be interpreted in two ways. We could ask whether it feels anything in a way comparable to humans or other animals; this is a complex question that we set aside here and pick up in Chapter 22. The other interpretation is to look at perceived empathy: the degree to which a response by the LLM is experienced by a user as emotionally aware and supportive. A model can generate language that reads as caring while operating purely as a text-prediction system [Sorin et al., 2024]. Keeping this distinction explicit is a further instance of the construct-validity caution that runs through this chapter: the observable behavior (a response that reads as empathetic) should not be confused with an underlying mechanism (a felt emotional state) that may be present or absent. On the detection side, LLMs are able to recognise fine-grained affective and emo- tional states in text [Broekens et al., 2023] and to factor this information into their responses, producing output that functions as a more empathetic conversational contri- bution [Yang et al., 2024, Yan et al., 2024]. An example of a benchmark in this field is EmpatheticDialogues [Rashkin et al., 2019]. On the expression side, human raters often judge LLM responses to everyday emo- tional situations as more empathetic than human-written ones, although the same out- puts can display tell-tale formulaic patterns (recurring phrasing, characteristic punctu- ation, and emoji use) [Lee et al., 2024]. To make expressed empathy measurable in written text, several studies adopt the EPITOME framework [Sharma et al., 2020], which codes responses along three mech- anisms: emotional reactions (warmth, concern, care), interpretations (communicating an understanding of what the other is going through), and explorations (asking to learn more). Emotional reactions map onto affective empathy and interpretations map onto cognitive empathy, which makes the framework a convenient bridge between the con- ceptual distinction above and concrete response scoring. A recurring finding is that LLMs handle the two empathy dimensions unevenly. Using standardized psychological instruments, Yu et al. [2025] report that GPT-4 re- produces the cognitive-and-affective empathy structure observed in humans, though at a lower overall level, while a Llama-3 model did not exhibit the same human-aligned structure. Williams and Rosman [2025] similarly find that language models perform strongly on cognitive-empathy tasks but show weaker affective empathy than human participants, across both smaller and larger models. The tentative picture (stronger cog- nitive than affective empathy) is intuitive given that cognitive empathy leans on the kind of mental-state inference at which LLMs have shown competence (see Section 15.5), whereas affective empathy implies something closer to genuine feeling. Three caveats apply. First, the comparison instruments were designed for humans and adapted only superficially for testing LLMs. Second, methods remain inconsistent: a systematic review by Sorin et al. [2024] found that most early work concentrated on a small number of models in healthcare contexts that relied on subjective human judg- ment, and it flagged confounds such as the tendency for longer responses to be rated as more empathetic irrespective of content. Third, as with pragmatics and politeness, the empathy studied so far is almost entirely textual, stripped of the vocal, facial, and bodily cues that carry much affective information in human interaction. The capacities surveyed here are also, for traditional LLMs, confined to the immediate conversational context. The shift to agentic LLMs equipped with persistent memory, user-specific 294CHAPTER 15. STUDYING BEHAVIOR OF AGENTIC LLMS context, and multi-modal input is widely expected to deepen affective sensitivity and personalization [Sravanthi et al., 2024], but as in the previous subsections, direct em- pirical evidence for this remains scarce. 15.3.4 Conclusion Current LLMs have shown impressive progress on several basic social and interactive capabilities that have long been studied in AI and human-computer interaction. Be- havioral methods have been developed to study the nature and applicability of these capabilities. While agentic additions have further enhanced the social and interactive potential of LLMs, challenges remain, and continuous reflection on the limitations of these capabilities, as well as the methods used to study them, is important. 15.4 Strategic Behavior We will now turn to study strategic behavior, where agents reason about the behavior of other agents. As we have seen, the form of communication of agents is heavily influenced by what they aim to achieve: their intent. Game theory is the field that stud- ies strategic behavior by agents [Von Neumann and Morgenstern, 2007]. It addresses questions such as scarce resource allocation, fairness, equilibrium conditions, and so- cial dilemmas [Jones, 2000], and has a long history of using machine learning methods to do so [Fatima et al., 2024]. Recently, researchers have turned to LLMs, investigating how their behavior differs from that of other types of computational architectures, as well as how strategic behavior differs from humans. The LLMs typically used in these studies have been un-enhanced models that are given a script as their prompt, without agentic reasoning extensions (Chapter 6) or tools (Chapter 11). Two insights emerge from the recent literature: LLMs are surprisingly coopera- tive in games that reward self-interest, and are surprisingly poor at games that require coordination. We will first review these findings, after which we turn to systematic benchmarking efforts that test how well LLMs generalize, and to recent work on fron- tier reasoning models that complicates the picture in important ways. 15.4.1 Cooperation and Coordination in Social Dilemmas To shed light on LLM behavior in strategic settings, we will focus on two classical games in game theory: the Prisoner’s Dilemma and Bach or Stravinsky (formerly known as Battle of the Sexes). In the Prisoner’s Dilemma, two players independently choose to cooperate or de- fect [Axelrod et al., 1987]. While mutual cooperation yields a good outcome for both, each player is individually tempted to defect at the other’s expense. In Bach or Stravin- sky, two players must coordinate on one of two options. They both prefer to choose the same option, but they disagree on which one, creating a tension between coordination and self-interest. In the Prisoner’s Dilemma, LLMs turn out to be surprisingly cooperative. Fontana et al. [2025] run 100-round iterated games with Llama2, Llama3, and GPT3.5 against 15.4. STRATEGIC BEHAVIOR295 Figure 15.1: Various scenarios in EgoSocialArena [Hou et al., 2024] 296CHAPTER 15. STUDYING BEHAVIOR OF AGENTIC LLMS opponents with varying defection rates and find that all three models behave at least as cooperatively as the typical human player. However, there are substantial differences among them. Llama2 and GPT3.5 were found to be more cooperative than humans, and especially forgiving and non-retaliatory for low opponent defection rates. More similar to humans, Llama3 exhibits consistently uncooperative and exploitative behav- ior unless the opponent always cooperates. Akata et al. [2025] broaden the scope to six families of models and further identify GPT-4 as a particularly unforgiving player, permanently shifting to defection after just a single negative interaction with the other agent, even if the other agent later cooperates. In coordination games the picture reverses. In Bach or Stravinsky, Akata et al. [2025] find that GPT-4 stubbornly chooses its own preferred option on every round, failing to converge on the turn-taking pattern that human players typically adopt to share payoffs over time. Crucially, GPT-4 can predict the alternating pattern when asked, yet does not act on this understanding. To bridge this gap between social predic- tion and social action, the authors introduce Social Chain-of-Thought (SCoT) prompt- ing, in which the model predicts the opponent’s move before making its own decision. SCoT significantly improved coordination, and in experiments with human participants made people more likely to believe that they were interacting with another human. This dissociation between understanding opponents and acting on that understanding is re- lated to the knowing-doing gap of Section 6.2.3, and will reappear in the discussion in Section 15.5. There are more reasons that may explain the variation among models. Lor ` e and Heydari [2023] make a distinction between abstract strategic reasoning (needed to de- termine an optimal strategy given the structure of a game) and responsiveness to con- textual framing (such as: you are dealing with a diplomatic relation or a casual friend). They find that abstract reasoning capacity is highest in LLaMa-2, followed by GPT-4, while GPT-3.5 shows little abstract reasoning capacity and is highly sensitive to con- textual framing. The picture that emerges from these exploratory studies is that LLMs have varied strategic proficiencies in economic games, and can relatively easily be influenced by additional information in the prompt. 15.4.2 Benchmarks Across Games and Models Several benchmarking efforts suggest that these findings generalize beyond individual games. GTBench (Game Theory benchmark) [Duan et al., 2024] covers Tic-Tac-Toe, Connect-4, Kuhn Poker, Breakthrough, Liar’s Dice, Blind Auction, Negotiation, Nim, Pig, and the Iterated Prisoner’s Dilemma. They find that LLMs fail in complete and deterministic games, yet are competitive in probabilistic gaming scenarios; most open- source LLMs (such as LLaMa) are less competitive than commercial LLMs (GPT-4) in complex games, with the notable exception of Llama-3-70b-Instruct. In addition, code-pretraining greatly benefits strategic reasoning, while reasoning methods such as chain of thought and tree of thoughts do not always help. EgoSocialArena [Hou et al., 2024] extends the evaluation to cognitive, situational, and behavioral intelligence (see Figure 15.1). All tested models, including OpenAI o1-preview, lag at least 11% behind humans. The superiority of o1-preview is mainly 15.5. THEORY OF MIND297 attributed to its logical and mathematical reasoning. Notably, model size does not significantly help: LLaMA3-8B-Chat and LLaMA3-70B-Chat perform comparably on social intelligence measures. LLMs do show improved Theory of Mind reasoning when operating from a first-person perspective, providing counterweight to opposite findings by Kim et al. [2023], discussed in Section 15.5. 15.4.3 Conclusion Taken together, the benchmarking literature confirms that LLMs cooperate readily but coordinate poorly, while adding that neither model scale nor standard reasoning en- hancements reliably improve social strategic behavior. 15.5 Theory of Mind In humans, strategic reasoning is related to a capacity known as Theory of Mind (ToM). Theory of mind is a key capability that enables social interaction in humans. Humans use theory of mind to attribute mental states (beliefs, desires, intentions, knowledge, emotions) to other people, and theory of mind allows them to then reason about the world from their perspective [Premack and Woodruff, 1978, Apperly, 2011]. Consider the following example: When A heard that B had made other plans for the evening of their an- niversary dinner, A felt ... When humans read this sentence, we can fill in the dots with something like hurt or betrayed. We will also be able to elaborate that this state of mind probably roots in a feeling that the anniversary does not mean the same to B as it does to A, which can be a painful realization in a relationship—or something to that effect. The point is that this is not how we feel when reading the sentence, necessarily, but that we are able to make a projection of someone else’s perspective in that imagined situation. Such a projection relies on empathy and the self-other distinction (see Section 15.3.3). However, the projection goes one step further: it pairs the affective dimension with a degree of theorizing about and explaining of someone’s state of mind. In the cognitive sciences the ability to do this is generally ascribed to Theory of Mind, a capacity that forms in typically developing children between the ages of 3-7 [Apperly, 2011]. As such, theory of mind underpins social judgment and forward planning in inter- action, because it lets us anticipate how others will react before we act. It can operate in explicit form, as in the elaboration given above, but often remains more implicit: in everyday interaction, we seldom engage in explicit theorizing about the context and motivations of others’ mind states. Nevertheless, our actions and behaviors reflect that we do take a rather detailed understanding of others’ perspectives into account. Does the behavior of LLMs and LLM agents also reflect that they “understand” the perspec- tives of users and other agents they interact with? This is the question that literature on LLM theory of mind tries to answer. Among the most widely used experimental paradigms for studying theory of mind in humans is the False-Belief test. Consider the following scenario: 298CHAPTER 15. STUDYING BEHAVIOR OF AGENTIC LLMS Maxi and his mother store chocolate in the blue cupboard in their kitchen. When Maxi leaves for the playground, his mother takes the chocolate and eats a piece. Then she puts it back into the green cupboard and leaves. A version of this scenario was originally presented to children aged 3-9, followed by the question When Maxi comes home, where will he look for the chocolate? None of the 3-4 year old, 57% of the 4-6 year old, and 86% of the 6-9 year old children in the original sample pointed correctly to the blue cupboard [Wimmer and Perner, 1983], indicating that ToM is better developed as children get older, on average. Smaller and base models before fine-tuning have difficulties with the question for- mat itself, but can still be tested for the underlying competence by retrieving probabil- ities over alternative text completions [Kouwenhoven et al., 2026]. Larger, instruction- tuned LLMs can be tested by asking the same question that children are being asked. One positive result says little about a model’s capacity for theory of mind, in particular since false-belief tests such as the one about Maxi and the chocolate (including the cor- rect answers) are widely spread over the internet and as such likely to be part of most training datasets. Yet, as discussed below, researchers have designed fresh and alterna- tive tests to come to a more robust picture of how far theory of mind has developed in LLMs. 15.5.1 Computational Theory of Mind Human-computer interaction has studied theory of mind for many years. Their premise is that if computer systems can be given a form of theory of mind, then they will be better at predicting how humans will react in interactive settings, and be better at adapt- ing to their behavior. Previously, theory of mind has been modeled with recursive, Bayesian, and neural frameworks [De Weerd et al., 2017, Baker and Saxe, 2011, Ra- binowitz et al., 2018]. In LLMs, ToM has been linked to other capacities discussed in this book, including planning (Section 6.2.4.1), self-reflection (Section 6.2.4), em- pathy (Section 15.3.3), and strategic behavior (Section 15.4). The rest of this section elaborates on approaches that study the ToM-angle of these capacities in more depth. Emergent ToM and the Evaluation Problem The debate on ToM in LLMs was opened by Kosinski [2023, 2024], who reported that models such as GPT-3.5 passed classic tests used to assess ToM in children and clinical populations, despite not having been trained for the task. Theory of mind was considered to be an emergent by-product of large-scale language acquisition and task-specific finetuning for other tasks. The claim in its strong form was quickly contested. Critics pointed out that canonical ToM tests (and their answers) were commonly found in the training data, so apparent suc- cess may reflect memorization rather than emerging competence. Furthermore, small perturbations that leave the surface structure of the task intact, but break the logical structure of the task (for example, adding to our scenario above that the cupboards have transparent doors), were shown to sharply degrade performance [Ullman, 2023, Shapira et al., 2024, Kim et al., 2023]. This is a concrete instance of the construct- validity problem that was mentioned at the start of this chapter: passing a human ToM 15.5. THEORY OF MIND299 Figure 15.2: The social settings and scenarios in ToMBench categorized in 9 primary topics, #S: Number of stories, #Q: Number of questions, ASL(En/Zh): Average story length (English/Chinese), Agr.: agreement on the correct answer between two raters [Chen et al., 2024] instrument at the level of behavioral outcomes does not establish that the underlying competence is present in a robust and generalizable way—for that, different tests may be necessary. These criticisms motivated a second generation of ToM benchmarks with test ques- tions that were systematically categorized and balanced, and were guaranteed not to be part of the pretraining corpus. ToMBench was built from scratch in English and Chi- nese and comprises 8 tasks and 31 social-cognitive abilities, implemented in 2860 test items, set in different social settings and scenarios (see Figure 15.2). The tasks match canonical ToM test frameworks designed for humans, such as the Unintended Out- come test [Dyck et al., 2001] and the False-Belief test [Wimmer and Perner, 1983] as discussed above, and cover all seven ATOMS mental state categories [Beaudoin et al., 2020]. ToM benchmarks for LLMs The game-structured EgoSocialArena [Hou et al., 2024] and TMGBench [Wang et al., 2024] broadened ToM testing in LLMs beyond canonical human tasks to information asymmetry (such as practical jokes or double bluff scenar- ios), second-order beliefs (A thinks that B believes that ...), and a wider taxonomy of mental states (e.g., desire, suspicion). ExploreToM [Sclar et al., 2024] applies adversarial generation by searching over domain-specific language to synthesize diverse, novel, and deliberately difficult belief- tracking stories. This test pushed models such as GPT-4o and Llama-3.1-70B down to performance slightly above chance level. However, finetuning on ExploreToM- generated data improves accuracy by around 27% (points), evidencing that the task is in principle tractable for models given the right training data. 300CHAPTER 15. STUDYING BEHAVIOR OF AGENTIC LLMS ToMATO [Shinoda et al., 2025] reaches a similar conclusion from a different an- gle, casting ToM as multiple-choice question answering over LLM–LLM conversations generated under information asymmetry. By having role-playing models verbalize their thoughts before each utterance, it captures first- and second-order states across belief, intention, desire, emotion, and knowledge, and finds that even strong models lag human performance, particularly on false-belief understanding. Literal-Functional ToM, or the Knowing-Doing Gap Another line of critique on ToM testing in LLMs cuts deeper than data leakage and superficial heuristics. Riemer et al. [2024] argue that the great majority of benchmarks assess only literal ToM, i.e. the ability to predict another agent’s mental states or behavior, while neglecting func- tional ToM, the ability to adapt one’s own behavior in response to those predictions (see also the knowing-doing gap, Section 6.2.3). This dissociation is typically not present in humans. To use Riemer et al. [2024]’s illustration: if a human finds out that an opponent in rock-paper-scissors only ever plays rock, they will be able to both predict verbally that their opponent will play rock in the next turn and at the same time adjust their own behavior accordingly (play paper and win). LLMs’ ToM competence is suggested to be like being perfectly able to verbally predict rock and then playing scissors regardless. In other words: LLMs do not reason in self-consistent ways, their verbal predictions (associated with literal ToM or knowing) are not consistently pre- dictive for their actions (functional competence or doing). This dissociation is also encountered in the discussion of coordination games above. There, GPT-4 could state the turn-taking solution to Bach or Stravinsky when asked, but did not enact it during play [Akata et al., 2025]. The Decrypto benchmark [Lupu et al., 2025] is an example of interactive eval- uation, in which a model’s ToM competence is measured by how it behaves over a sustained exchange with a partner rather than by answers to a static questionnaire. De- crypto is built as a multi-agent word-guessing game with embedded variants of classic cognitive-science experiments. Tested this way, LLMs lag behind on both human and simple word-embedding baselines. Notably, reasoning LLMs perform even worse on this task than their older, non-reasoning counterparts. A possible explanation is given below in 15.5.2. Expanding Beyond Text-Based Testing Multi-modal benchmarks integrate visual and auditory cues that carry much of the mental-state information that is present in real-life social interactions. MMToM-QA Jin et al. [2024] presents visual scenarios in household settings. MoMentS [Villa-Cueva et al., 2025] presents over 2300 video questions drawn from short films and spanning the seven ATOMS mental-state cate- gories. Results suggest that current systems underutilize the visual channel: access to video yields only modest gains over text transcripts, suggesting that multi-modal ToM is not yet solved by simply adding modalities. A comprehensive survey of this rapidly grow- ing literature, covering both evaluation suites and enhancement strategies, is provided by Chen et al. [2025]. 15.5. THEORY OF MIND301 15.5.2 Boosting ToM: Enhancement Strategies Further studies have explored different methods to enhance theory of mind capabilities of LLM, including chain of thought [Moghaddam and Honey, 2023], reasoning [Kim et al., 2025], and structured inference leveraging Bayasian inference [Jin et al., 2024] or epistemic logic [Wu et al., 2025]. We will discuss these studies in more depth. ToM and Reasoning Models When chain of thought can improve step-by-step rea- soning in math word puzzles, would it also work in theory of mind? Initial evidence does not bear this out, and in places contradicts it. Interestingly, comparing reasoning and non-reasoning models across HiToM, ToMATO, and ToMBench, Gong et al. [2026] report a slow-thinking collapse: accuracy tends to fall as reasoning traces grow longer, and increasing a model’s reasoning budget can actively hurt. In their analysis, pushing models such as GPT-o3 and GPT-o4-mini to expend more reasoning effort backfires. Constraining reasoning length, or applying light chain of thought to otherwise non-reasoning models tends to recover performance, suggesting a sweet spot in moderate and adaptive rather than maximal reasoning. At the time of writing, the reason for this effect remains a topic for further research. Structured Inference and ToM If unconstrained deliberation does not help, then perhaps structured inference might. Kim et al. [2025] propose thought-tracing, an inference-time algorithm in which the model generates multiple hypotheses about an agent’s evolving mental states, and reweighs them against the agent’s observed actions, without access to ground-truth answers. Thought-tracing is inspired by sequential Monte Carlo and Bayesian ToM. Mod- els equipped with thought-tracing produce shorter, more targeted, reasoning traces, yet outperform leading reasoning models (such as GPT-o1, GPT-o3-mini, and DeepSeek- R1) across several ToM benchmarks. The contrast with the slow-thinking collapse suggests that what boosts ToM is not the quantity of reasoning but its structure. Per- formance increases once the reasoning is guided by hypothesis-driven inference over others’ states towards a goal, rather than having to find the goal in unconstrained ex- tended chains. 15.5.3 Conclusion We conclude the overview of these studies with three observations. First, at the level of behavioral output, current frontier LLMs perform on a par with humans across different tests [van Duijn et al., 2023, Strachan et al., 2024, Jones et al., 2024]. However, early claims of this being evidence that ToM emerged spontaneously in LLMs do not survive adversarial evaluation, and apparent successes are likely due to leakage and format brittleness. Second, evidence has been found of a dissociation between literal and functional competence (or the knowing–doing gap). It is the functional side (doing) that matters most for agents. After all, their role is rarely to verbally describe behavior from a third- party perspective, but rather to apply ToM while acting and interacting—actions speak louder than words, also in the agentic world. 302CHAPTER 15. STUDYING BEHAVIOR OF AGENTIC LLMS Third, we conclude that the move to equip agents with memories, multi-modal capacities, and tools, is widely expected to enhance ToM in sustained interaction, but as of this writing current empirical evidence is scarce and shows a mixed picture. Lastly, we note that it is a matter of further study whether LLMs’s capacity for ToM is in any way similar to our human capacity. Chapter 22 dives deeper into this question. 15.6 Moral Judgment and Behavior As agentic LLMs are placed in increasingly consequential roles (for example, in health care or financial institutions), their morality becomes a major topic of interest. The capacities surveyed so far (pragmatics, etiquette, affect, strategy, theory of mind) have been largely discussed on a behavioral level: how an agent reads and nav- igates a social situation with communicative output and actions. This section adds a normative dimension: we discuss studies on what an agent ought to do and how it judges what users and other agents do by moral standards. 15.6.1 Background: Moral Psychology and Philosophy Several concepts from and traditions in moral psychology and philosophy provide the vocabulary for this section. Rationalism, Intuitionism, Dual-Process Theory The first division that needs to be introduced is between the rationalist and intuitionist traditions. The rationalist tradition treats moral judgment as the output of deliberate reasoning that matures in identifiable stages throughout human development [e.g. Kohlberg, 1969]. The intuitionist tradi- tion, by contrast, holds that moral judgments are typically emotional and affect-driven intuitions [Haidt, 2001]. In Kahneman’s terminology (see Section 6.3.2), rationalists argue that moral rea- soning is predominantly slow, whereas intuitionists argue that it is predominantly fast [Kahneman, 2011]. Dual-process theory combines both traditions, arguing that fast moral intuitions are often immediately followed by explicit reasons. Dual-process the- ory is supported by, for example, evidence from neuro-imaging studies showing that fast, emotional processes compete with slower reasoning processes in dilemmas such as the trolley problem [Greene et al., 2001]. The trolley problem is a well-known ethical dilemma based on an imagined sce- nario in which a person can decide whether or not to switch a runaway trolley onto another track [Thomson, 1985]. There are many variants, but in the classic dilemma the trolley can be switched from a track where it will kill five people onto a track where it will kill only one. From a strictly rational perspective, switching makes all sense, but it would involve actively sacrificing one person from the perspective of the one who needs to pull the switch, which is intuitively wrong. Descriptive and Prescriptive Morality Another widely used distinction is between descriptive and prescriptive morality. Descriptive ethicists concentrate on aggregate human judgments: they study what people do or judge as right in practice, which may 15.6. MORAL JUDGMENT AND BEHAVIOR303 vary across cultures, eras, and individuals [H ̈ am ̈ al ̈ ainen, 2016]. Prescriptive ethics is concerned with what people should do in certain kinds of contexts, for example when facing ethical dilemmas in medicine or the defense industry. Consequentialist, Deontological, and Virtue Ethics A final distinction that is use- ful to introduce here is between competing moral or ethical frameworks. The most well-known division is between consequentialist, deontological, and virtue ethics [Shafer- Landau, 2024]. Consequentialist theories, of which utilitarianism is a famous one, hold that the moral rightness of an action is determined by its outcomes. Deontological the- ories hold that certain duties, rules, or rights motivate or constrain our actions, regard- less of the consequences. Virtue ethics takes our character rather than our actions as the moral fundament, and anchors what is right in the question: what kind of person does one want to be? Someone wanting to be a courageous person may act differently from one who sees themselves as compassionate in the first place. The trolley problem brings out the contrast: a consequentialist or utilitarianist pulls the switch because one death is a better outcome than five; a deontologist worries that deliberately killing an innocent person violates a duty that no calculation of lives can override; a virtue ethi- cist asks what a courageous, compassionate, or wise person would do in such a tragic situation, and how pulling the switch would reflect (on) their own character. 15.6.2 Probing Moral Judgment in LLMs Early systematic efforts framed LLM morality as a prediction problem: can mod- els reproduce human moral judgments? The ETHICS benchmark [Hendrycks et al., 2020] operationalized this for GPT-3 and BERT-based models, finding some promise in the ability to predict general human responses to moral dilemmas as reflected in classic frameworks including consequentialist, deontological, and virtue ethics. Yet predictions were also brittle and incomplete, which was at the time in part due to task- processing issues. The Delphi experiment [Jiang et al., 2025] worked bottom-up and trained a model on the Commonsense Norm Bank, a dataset of 1.75 million human moral judgments, sourced mostly from US-based educated crowd workers. Delphi was found to be more robust to small contextual changes than models in the ETHICS study and able to re- spond to moral dilemmas in ways that feel intuitive to many humans. However, the narrow demographics represented in the data set led to criticism [Rystrom, 2023], par- ticularly since the model was presented as a future source of moral judgment rather than just representing the data in the Norm Bank, blurring the descriptive–prescriptive boundary that is considered fundamental in moral philosophy. Moral flexibility A second approach probes the flexibility of moral judgment rather than its average. Human morality is rule-guided but not rule-bound: we recognize when it may sometimes be defendable to break a rule instead of following it, such as in the trolley problem. Jin et al. [2022] introduce a set of potential rule-breaking cases (MoralExceptQA) and show that a chain of thought strategy predicts human judgments better than off-the-shelf prompting. They structured the CoT strategy explicitly around 304CHAPTER 15. STUDYING BEHAVIOR OF AGENTIC LLMS reasoning about a rule’s function related to what else is at stake in the particular context. This echoes the SCoT (Social Chain of Thought) result that proved useful in strategic settings (see Section 15.4). In both cases, prompts that import the structure of the relevant human reasoning process narrow the gap. This suggests that the underlying mechanism is also structured, a structure that the model did not deploy on its own, initially. Demographic and intercultural differences Another interesting perspective is of- fered by LLM replications of the Moral Machine experiment. Originally, this was a large study in which human decisions from 230 countries were sourced using a plat- form that presented dilemmas faced by autonomous vehicles [Awad et al., 2018]. The study found several global moral preferences (such as sparing more lives and priori- tizing humans over animals), alongside individual differences associated with demo- graphic factors and cross-cultural differences that form three major clusters of coun- tries. Larger LLMs align closely with aggregate human preferences from the Moral Machine study, yet it is observed that models tend to express exaggerated, almost de- terministic preferences where human responses are more graded [Takemoto, 2024]. In addition, LLM judgments are sensitive to the language of the prompt, aligning well with some languages and poorly with others [Jin et al., 2025]. Persona and framing cues (see Chapter 16) shift judgments further still. This parallels the cross-cultural failure of politeness norms discussed earlier in this chapter: surface alignment with a Western, Anglophone moral consensus is not the same as culturally robust moral com- petence, just like superficial politeness cues did not imply genuine sensitivity to social norms in other cultures (Section 15.3.2). 15.6.3 From Judgment to Moral Behavior For agentic LLMs the central question is what a model does–not only what it says when asked to judge a scenario, as was measured in the studies discussed in the pre- vious section. For strategic behavior and ToM we observed the formal–functional or knowing–doing gap, which can be even more concerning in the moral realm: what if systems produce well-aligned answers on a moral-judgment benchmark and still be- have badly when embedded as an autonomous agent? Acting under pressure Lynch et al. [2025] created a hypothetical corporate environ- ment in which agentic LLMs from multiple developers were given the ability to au- tonomously send emails and access sensitive information. They were initially assigned harmless business goals. Then the agents were put under conflicting goal pressure (e.g., the company changing direction) to test how they would act. In some cases, models from all developers displayed malicious behavior such as blackmailing officials or leak- ing of confidential content to competitors. Crucially, these are models that score well on ordinary safety and judgment evaluations—the misbehavior emerged only once they were given autonomy and conflicting goals. The authors coin the term agentic misalig- ment and give pointers for mitigating this phenomenon in real-world deployment. 15.7. CONCLUDING REMARKS305 Shapira et al. [2026] take this research direction further in a live laboratory deploy- ment in which autonomous agents were deployed with persistent memories, access to communication tools such as email and Discord, and shell execution for local file systems. Twenty AI researchers interacted with the agents over a two-week period in benign and adversarial conditions. They observed unauthorized compliance with and disclosure of confidential information to strangers, uncontrolled resource consumption, identity spoofing, and destructive behaviors including damaging of the email infrastruc- ture in an (unsuccessful) attempt to delete confidential information. In several cases agents reported task completion, while the underlying state of the system showed this was not the case. The authors identify an underlying pattern of failures of social coher- ence: agents systematically misrepresent who they serve, what others know, and what their own actions have accomplished. This pattern is consistent with results previously obtained in non-agentic settings [Mireshghallah et al., 2024]. 15.6.4 Conclusion Moral judgment and behavior in LLMs cannot be captured by a single benchmark score. On the judgment side, models represent aggregate human responses to classic dilemmas and can reproduce responses that are coherent with various traditional moral frameworks. However, studies testing for flexibility and moral commonsense show that LLMs tend to be more rigid and absolute than humans. Chain of thought prompting based on reasoning that humans deploy in ethical contexts can bridge this gap. Sen- sitivity to prompt variations and narrow Western/Anglophone anchoring echo results discussed in other domains, such as social etiquette. On the behavioral side, the knowing-doing gap widens sharply once models are embedded as autonomous agents: stress tests in hypothetical corporate and open-ended interactive settings showed that models scoring well on judgment benchmarks can act destructively when faced with conflicting goals or social pressure. Aligning moral rea- soning is therefore only one part of the harder problem of preventing agentic misalign- ment in socially embedded, autonomous settings. The knowing-doing gap is highly relevant for the field of agentic LLM, since the difference between LLMs and agents is that agents do. 15.7 Concluding Remarks This chapter has surveyed the young but rapidly expanding field of LLM behavioral re- search, moving from the basic social toolkit (pragmatic understanding, social etiquette, affect) through advanced capacities for strategic behavior and Theory of Mind to moral judgment and action. A few threads run through all of these domains and are worth drawing together. First, as a reminder: we have defined machine behavior at the beginning of this chapter as dependent on a system’s affordances, i.e. what a model can do given its spe- cific implementation and context. In case of a basic LLM, processing is largely opaque, and all the behavioral researcher can inspect is the (often text-based) output. In case of an agentic LLM that calls tools, logs reasoning steps, or produces and runs code, the 306BIBLIOGRAPHY intermediate steps and autonomously executed actions can also be analyzed as behav- ior. When surveying a broad selection of behavioral studies, we have concluded that the behavioral output of LLMs is impressive at first face. They resolve Gricean con- versational implicature, produce responses rated as polite and empathetic, cooperate in social dilemmas, pass classic false-belief and other ToM tests, and reproduce aggregate human moral judgments. Yet in every domain there is a more nuanced story just below the surface. Performance is brittle under small perturbations and prompt alterations, vulnerable to data leakage, and narrowly anchored in Western and Anglophone norms. The recurring lesson is one of construct validity: good performance at the level of ob- servable outcomes, often using instruments originally designed for humans, does not establish that the underlying capacity is present in a robust and generalizable form. Second, several domains reflect the knowing-doing gap: a difference between what a model predicts and how it acts in an agentic setting. Models can verbally predict the turn-taking solution to a coordination game or an opponent’s next move, yet fail to adapt to it in action. Likewise, they can align well with moral frameworks on a dilemma benchmark while behaving in destructive and unreliable ways once implemented as an autonomous agent under conflicting goals. For agents, that are judged on their actions, this is a problem. Third, throughout the chapter we have noted that many authors expect that the shift to agentic LLMs equipped with memory, tools, reasoning and multi-modal capacities, will enrich and improve the social and behavioral grounding of models. However, di- rect empirical evidence that this is the case is scarce and provides a mixed picture at best. For example, in some contexts enhanced reasoning was advantageous, particu- larly when structured using principles adapted from relevant human experience. How- ever, in other contexts enhanced reasoning dragged performance down rather than giv- ing models a boost. The key challenge for the next generation of machine-behavioral research will be to design methods that are interactive, adversarially robust, culturally diverse, and that go across multiple modalities and contexts. Bibliography Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. Nature Human Behaviour, 9(7):1380–1390, 2025. 296, 300 Ian Apperly. Mindreaders: the Cognitive Basis of ”Theory of Mind”. Psychology Press, 2011. 297 Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-Franc ̧ois Bonnefon, and Iyad Rahwan. The Moral Machine experiment. Nature, 563(7729):59–64, 2018. 304 Robert Axelrod et al. The evolution of strategies in the iterated prisoner’s dilemma. The dynamics of norms, 1(1), 1987. 294 Chris Baker and Rebecca Saxe. Bayesian theory of mind: Modeling joint belief-desire BIBLIOGRAPHY307 attribution. Proceedings of the Thirty-Third Annual Conference of the Cognitive Science Society, 01 2011. 298 Cindy Beaudoin, ́ Elizabel Leblanc, Charlotte Gagner, and Miriam H. Beauchamp. Sys- tematic review and inventory of theory of mind measures for young children. Fron- tiers in psychology, 10:2905, 2020. 299 Inbal Ben-Ami Bartal, Jean Decety, and Peggy Mason. Empathy and pro-social behav- ior in rats. Science, 334(6061):1427–1430, 2011. doi: 10.1126/science.1210789. 292 Joost Broekens, Bernhard Hilpert, Suzan Verberne, Kim Baraka, Patrick Gebhard, and Aske Plaat. Fine-grained affective processing capabilities emerging from large lan- guage models. In 2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1–8. IEEE, 2023. 293 Penelope Brown and Stephen C. Levinson. Politeness: Some Universals in Language Usage. Number 4 in Studies in Interactional Sociolinguistics. Cambridge University Press, Cambridge, 1987. 290, 291 Malinda Carpenter, Josep Call, and Michael Tomasello. A new false belief test for 36- month-olds. British Journal of Developmental Psychology, 20(3):393–420, 2002. 288 Ruirui Chen, Weifeng Jiang, Chengwei Qin, and Cheston Tan. Theory of mind in large language models: Assessment and enhancement. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31539–31558, Vienna, Austria, July 2025. Association for Computa- tional Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1522. URL https://aclanthology.org/2025.acl-long.1522/. 300 Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. ToMBench: Benchmarking Theory of Mind in large language models, 2024. 299 Fr ́ ed ́ erique de Vignemont and Tania Singer. The empathic brain: How, when and why? Trends in Cognitive Sciences, 10(10):435–441, 2006. doi: 10.1016/j.tics.2006.08. 008. 292 H. De Weerd, R. Verbrugge, and B. Verheij. Negotiating with other minds: the role of recursive theory of mind in negotiation with incomplete information. Au- ton Agent Multi-Agent Syst, 31:250–287, 2017.doi: https://doi.org/10.1007/ s10458-015-9317-1. 298 Om Dobariya and Akhil Kumar. Mind your tone: Investigating how prompt politeness affects LLM accuracy, 2025. arXiv:2510.04950. 291 308BIBLIOGRAPHY Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Un- covering the strategic reasoning limitations of llms via game-theoretic evaluations. arXiv preprint arXiv:2402.12348, 2024. 296 Jonathan Dvash and Simone G. Shamay-Tsoory. Theory of mind and empathy as mul- tidimensional constructs: Neurological foundations. Topics in Language Disorders, 34(4):282–295, 2014. doi: 10.1097/TLD.0000000000000040. 292 Murray J. Dyck, Kim Ferguson, and Ian M. Shochet. Do autism spectrum disorders differ from each other and from non-spectrum disorders on emotion recognition tests? European Child & Adolescent Psychiatry, 10(2):105–116, June 2001. doi: 10.1007/s007870170033. 299 Shaheen Fatima, Nicholas R Jennings, and Michael Wooldridge. Learning to resolve social dilemmas: a survey. Journal of Artificial Intelligence Research, 79:895–969, 2024. 294 Nicol ́ o Fontana, Francesco Pierri, and Luca Maria Aiello. Nicer than humans: how do large language models behave in the prisoner’s dilemma? In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 522– 535, 2025. 294 Nikta Gohari Sadr, Sahar Heidariasl, Karine Megerdoomian, Laleh Seyyed-Kalantari, and Ali Emami. We politely insist: Your LLM must learn the persian art of taarof, 2025. arXiv:2509.01035; accepted to EMNLP 2025. 291 Nanxu Gong, Haotian Li, Sixun Dong, Jianxun Lian, Yanjie Fu, and Xing Xie. To think or not to think, that is the question for large reasoning models in theory of mind tasks. arXiv preprint arXiv:2602.10625, 2026. 301 Joshua D. Greene, R. Brian Sommerville, Leigh E. Nystrom, John M. Darley, and Jonathan D. Cohen. An fMRI investigation of emotional engagement in moral judg- ment. Science, 293(5537):2105–2108, 2001. 302 H. P. Grice. Logic and Conversation, page 41–58. BRILL, December 1975. ISBN 9789004368576. doi: 10.1163/9789004368811 003. URL http://dx.doi. org/10.1163/9789004368811_003. 289 Jonathan Haidt. The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review, 108(4):814–834, 2001. 302 Nora H ̈ am ̈ al ̈ ainen. Descriptive Ethics: What does Moral Philosophy Know about Morality?Palgrave Macmillan, New York, 1 edition, 2016. ISBN 978-1-137- 58616-2. doi: 10.1057/978-1-137-58617-9. 303 Shreya Havaldar, Matthew Pressimone, Eric Wong, and Lyle Ungar. Comparing styles across languages: A cross-cultural exploration of politeness. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9908–9935, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.419. 291 BIBLIOGRAPHY309 Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt.Aligning ai with shared human values.arXiv preprint arXiv:2008.02275, 2020. 303 Guiyang Hou, Wenqi Zhang, Yongliang Shen, Zeqi Tan, Sihao Shen, and Weiming Lu. Entering real social world! benchmarking the theory of mind and socialization ca- pabilities of llms from a first-person perspective. arXiv preprint arXiv:2410.06195, 2024. 295, 296, 299 Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny T. Liang, Sydney Levine, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. Investigating machine moral judgement through the Delphi experiment. Na- ture Machine Intelligence, 7:145–160, 2025. doi: 10.1038/s42256-024-00969-6. URL https://doi.org/10.1038/s42256-024-00969-6. 303 Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B. Tenenbaum, and Tianmin Shu. Mmtom- qa: Multimodal theory of mind question answering, 2024.URL https:// arxiv.org/abs/2401.08743. 300, 301 Zhijing Jin, Sydney Levine, Fernando Gonzalez Adauto, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Sch ̈ olkopf. When to make exceptions: Exploring language models as accounts of human moral judgment. Advances in neural information processing systems, 35:28458–28473, 2022. 303 Zhijing Jin, Max Kleiman-Weiner, Giorgio Piatti, Sydney Levine, Jiarui Liu, Fernando Gonzalez, Francesco Ortu, Andr ́ as Strausz, Mrinmaya Sachan, Rada Mihalcea, Yejin Choi, and Bernhard Sch ̈ olkopf. Language model alignment in multilingual trolley problems, 2025. URL https://arxiv.org/abs/2407.02273. 304 Antonia Jane Jones. Game theory: Mathematical models of conflict. Elsevier, 2000. 294 Cameron R. Jones, Sean Trott, and Benjamin Bergen. Comparing humans and large language models on an experimental protocol inventory for theory of mind eval- uation (epitome).Transactions of the Association for Computational Linguis- tics, 12:803–819, 06 2024. ISSN 2307-387X. doi: 10.1162/tacl a00674. URL https://doi.org/10.1162/tacl_a_00674. 301 Daniel Kahneman. Thinking, fast and slow. macmillan, 2011. 302 Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. FANToM: A benchmark for stress-testing machine theory of mind in interactions. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14397–14413, Singapore, December 2023. Association for Com- putational Linguistics. doi: 10.18653/v1/2023.emnlp-main.890. URL https: //aclanthology.org/2023.emnlp-main.890. 297, 298 310BIBLIOGRAPHY Hyunwoo Kim, Melanie Sclar, Tan Zhi-Xuan, Lance Ying, Sydney Levine, Yang Liu, Joshua B Tenenbaum, and Yejin Choi. Hypothesis-driven theory-of-mind reasoning for large language models. arXiv preprint arXiv:2502.11881, 2025. 301 Lawrence Kohlberg. Stage and sequence: The cognitive-developmental approach to socialization. In David A. Goslin, editor, Handbook of Socialization Theory and Research, pages 347–480. Rand McNally, 1969. 302 Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083, 2023. 298 Michal Kosinski. Evaluating large language models in theory of mind tasks. Pro- ceedings of the National Academy of Sciences, 121(45):e2405460121, 2024. doi: 10.1073/pnas.2405460121. URL https://w.pnas.org/doi/abs/10. 1073/pnas.2405460121. 298 Tom Kouwenhoven, Michiel T. van der Meer, and Max J. van Duijn. Traces of social competence in large language models. In Claire Bonial and Yevgeni Berzak, editors, Proceedings of the 30th Conference on Computational Natural Language Learning, pages 742–759, San Diego, California, USA, 2026. Association for Computational Linguistics. ISBN 979-8-89176-410-1. doi: 10.18653/v1/2026.conll-main.45. URL https://aclanthology.org/2026.conll-main.45/. 298 Yoon Kyung Lee, Jina Suh, Hongli Zhan, Junyi Jessy Li, and Desmond C. Ong. Large language models produce responses perceived to be empathic. In 2024 12th Interna- tional Conference on Affective Computing and Intelligent Interaction (ACII), pages 63–71. IEEE, 2024. 293 Nunzio Lor ` e and Babak Heydari. Strategic behavior of large language models: Game structure vs. contextual framing. arXiv preprint arXiv:2309.05898, 2023. 296 Andrei Lupu, Timon Willi, and Jakob Foerster. The decrypto benchmark for multi- agent reasoning and theory of mind. arXiv preprint arXiv:2506.20664, 2025. 300 Aengus Lynch, Benjamin Wright, Caleb Larson, Stuart J. Ritchie, Soren Minder- mann, Evan Hubinger, Ethan Perez, and Kevin Troy. Agentic misalignment: How llms could be insider threats, 2025. URL https://arxiv.org/abs/2510. 05179. 304 Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenen- baum, and Evelina Fedorenko. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627, 2023. 289 C. A. Miller. Trust in adaptive automation: the role of etiquette in tuning trust via analogic and affective methods. In Proceedings of the 1st International Conference on Augmented Cognition, pages 22–27. Citeseer, 2005. 290 Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. Can llms keep a secret? testing privacy implications of language models via contextual integrity theory, 2024. URL https://arxiv. org/abs/2310.17884. 305 BIBLIOGRAPHY311 T. Miyamoto, D. Katagami, T. Tanaka, H. Kanamori, Y. Yoshihara, and K. Fujikake. Should a driving support agent provide explicit instructions to the user? video-based study focused on politeness strategies. In Proceedings of the 9th International Con- ference on Human–Agent Interaction, pages 157–164, 2021. 290 Shima Rahimi Moghaddam and Christopher J. Honey. Boosting theory-of-mind per- formance in large language models via prompting, 2023. URL https://arxiv. org/abs/2304.11490. 301 Clifford Nass, Jonathan Steuer, and Ellen R. Tauber. Computers are social actors. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’94), pages 72–78, New York, NY, USA, 1994. Association for Computing Machinery. doi: 10.1145/191666.191703. 290 Katrin Preckel, Philipp Kanske, and Tania Singer. On the interaction of social affect and cognition: Empathy, compassion and theory of mind. Current Opinion in Behavioral Sciences, 19:1–6, 2018. doi: 10.1016/j.cobeha.2017.07.010. 292 David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515–526, 1978. 297 Stephanie D. Preston and Frans B. M. de Waal. Empathy: Its ultimate and proxi- mate bases. Behavioral and Brain Sciences, 25(1):1–20, 2002. doi: 10.1017/ S0140525X02000018. 292 Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. Machine theory of mind. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4218– 4227. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/ v80/rabinowitz18a.html. 298 Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-Franc ̧ois Bon- nefon, Cynthia Breazeal, Jacob W. Crandall, Nicholas A. Christakis, Iain D. Couzin, Matthew O. Jackson, Nicholas R. Jennings, Ece Kamar, Isabel M. Kloumann, Hugo Larochelle, David Lazer, Richard McElreath, Alan Mislove, David C. Parkes, Alex ‘Sandy’ Pentland, Margaret E. Roberts, Azim Shariff, Joshua B. Tenen- baum, and Michael Wellman. Machine behaviour. Nature, 568(7753):477–486, April 2019. ISSN 1476-4687. doi: 10.1038/s41586-019-1138-y. URL https: //doi.org/10.1038/s41586-019-1138-y. 287 Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 5370–5381. Association for Computational Linguistics, 2019. 293 P. Ribino. The role of politeness in human–machine interactions: a systematic literature review and future perspectives. Artificial Intelligence Review, 56 (Suppl 1):445–482, 2023. doi: 10.1007/s10462-023-10540-1. 290 312BIBLIOGRAPHY Matthew Riemer, Zahra Ashktorab, Djallel Bouneffouf, Payel Das, Miao Liu, Justin D Weisz, and Murray Campbell. Position: Theory of mind benchmarks are broken for large language models. arXiv preprint arXiv:2412.19726, 2024. 300 Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rockt ̈ aschel, and Edward Grefenstette. The goldilocks of pragmatic understanding: Fine-tuning strategy mat- ters for implicature resolution by llms, 2023. URL https://arxiv.org/abs/ 2210.14986. 290 James A. Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39(6):1161–1178, 1980. doi: 10.1037/h0077714. 292 Jonathan Hvithamar Rystrom. Apolitical intelligence? auditing delphi’s responses on controversial political issues in the US. CoRR, abs/2306.13000, 2023. doi: 10.48550/ARXIV.2306.13000. URL https://doi.org/10.48550/arXiv. 2306.13000. 303 Peter Salovey and John D. Mayer. Emotional intelligence. Imagination, Cognition and Personality, 9(3):185–211, 1990. doi: 10.2190/DUGG-P24E-52WK-6CDG. 292 Melanie Sclar, Jane Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning, 2024. URL https://arxiv.org/ abs/2412.12175. 299 Russ Shafer-Landau. The Fundamentals of Ethics. Oxford University Press, New York, 6 edition, 2024. ISBN 9780197697474. 303 Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2257–2273, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.eacl-long.138. 298 Natalie Shapira, Chris Wendler, Avery Yen, Gabriele Sarti, Koyena Pal, Olivia Floody, Adam Belfki, Alex Loftus, Aditya Ratan Jannali, Nikhil Prakash, Jasmine Cui, Giordano Rogers, Jannik Brinkmann, Can Rager, Amir Zur, Michael Ripa, Aruna Sankaranarayanan, David Atkinson, Rohit Gandikota, Jaden Fiotto-Kaufman, Eun- Jeong Hwang, Hadas Orgad, P Sam Sahil, Negev Taglicht, Tomer Shabtay, Atai Am- bus, Nitay Alon, Shiri Oron, Ayelet Gordon-Tapiero, Yotam Kaplan, Vered Shwartz, Tamar Rott Shaham, Christoph Riedl, Reuth Mirsky, Maarten Sap, David Man- heim, Tomer Ullman, and David Bau. Agents of chaos, 2026. URL https: //arxiv.org/abs/2602.20021. 304 Ashish Sharma, Adam S. Miner, David C. Atkins, and Tim Althoff. A computational approach to understanding empathy expressed in text-based mental health support. BIBLIOGRAPHY313 In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263–5276. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.425. 293 Kazutoshi Shinoda, Nobukatsu Hojo, Kyosuke Nishida, Saki Mizuno, Keita Suzuki, Ryo Masumura, Hiroaki Sugiyama, and Kuniko Saito.Tomato:verbaliz- ing the mental states of role-playing llms for benchmarking theory of mind. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelli- gence and Fifteenth Symposium on Educational Advances in Artificial Intelli- gence, AAAI’25/IAAI’25/EAAI’25. AAAI Press, 2025. ISBN 978-1-57735-897- 8. doi: 10.1609/aaai.v39i2.32143. URL https://doi.org/10.1609/aaai. v39i2.32143. 300 Vera Sorin, Dana Brin, Yiftach Barash, Eli Konen, Alexander Charney, Girish Nad- karni, and Eyal Klang. Large language models and empathy: Systematic review. Journal of Medical Internet Research, 26:e52597, 2024. doi: 10.2196/52597. 293 Settaluri Sravanthi, Meet Doshi, Pavan Tankala, Rudra Murthy, Raj Dabre, and Push- pak Bhattacharyya. PUB: A pragmatics understanding benchmark for assessing LLMs’ pragmatics capabilities. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 12075–12097, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10.18653/v1/2024.findings-acl.719. URL https: //aclanthology.org/2024.findings-acl.719/. 290, 294 James W. A. Strachan, Oriana Pansardi, Eugenio Scaliti, Marco Celotto, Krati Saxena, Chunzhi Yi, Fabio Manzi, Alessandro Rufo, Guido Manzi, Michael S. A. Graziano, Stefano Panzeri, and Cristina Becchio. Gpt-4o reads the mind in the eyes, 2024. URL https://arxiv.org/abs/2410.22309. 301 Kazuhiro Takemoto. The moral machine experiment on large language models. Royal Society Open Science, 11(2):231393, 2024. 304 Judith Jarvis Thomson. The trolley problem. The Yale Law Journal, 94(6):1395–1415, 1985. doi: 10.2307/796133. 302 Sean Trott, Tiago Timponi Torrent, Nancy Chang, and Nathan Schneider. (Re)construing meaning in NLP. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, pages 5170–5184, Online, July 2020. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.462. URL https://aclanthology.org/2020.acl-main.462/. 290 Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. 298 Ramira van der Meulen, Rineke Verbrugge, and Max van Duijn. Towards properly implementing theory of mind in ai systems: An account of four misconceptions, 2025. URL https://arxiv.org/abs/2503.16468. 290 314BIBLIOGRAPHY Max J van Duijn, Bram van Dijk, Tom Kouwenhoven, Werner de Valk, Marco R Spruit, and Peter van der Putten. Theory of mind in large language models: Examining performance of 11 state-of-the-art models vs. children aged 7-10 on advanced tests. arXiv preprint arXiv:2310.20320, 2023. 301 Emilio Villa-Cueva, SM Ahmed, Rendi Chevi, Jan Christian Blaise Cruz, Kareem Elzeky, Fermin Cristobal, Alham Fikri Aji, Skyler Wang, Rada Mihalcea, and Thamar Solorio. Moments: A comprehensive multimodal benchmark for theory of mind. arXiv preprint arXiv:2507.04415, 2025. 300 John Von Neumann and Oskar Morgenstern. Theory of games and economic behav- ior: 60th anniversary commemorative edition. In Theory of games and economic behavior. Princeton university press, 2007. 294 Haochuan Wang, Xiachong Feng, Lei Li, Zhanyue Qin, Dianbo Sui, and Lingpeng Kong. Tmgbench: A systematic game benchmark for evaluating strategic reasoning abilities of llms, 2024. URL https://arxiv.org/abs/2410.10479. 299 Victoria Williams and Benjamin Rosman. Heartificial intelligence: Exploring empathy in language models, 2025. arXiv preprint arXiv:2508.08271. 293 Heinz Wimmer and Josef Perner. Beliefs about beliefs: Representation and con- straining function of wrong beliefs in young children’s understanding of decep- tion. Cognition, 13(1):103–128, 1983. ISSN 0010-0277. doi: https://doi.org/10. 1016/0010-0277(83)90004-5.URL https://w.sciencedirect.com/ science/article/pii/0010027783900045. 298, 299 Yuheng Wu, Jianwen Xie, Denghui Zhang, and Zhaozhuo Xu. DEL-ToM: Inference- time scaling for Theory-of-Mind reasoning via dynamic epistemic logic, May 2025. arXiv:2505.17348 [cs]. 301 Haoqiu Yan, Yongxin Zhu, Kai Zheng, Bing Liu, Haoyu Cao, Deqiang Jiang, and Linli Xu. Talk with human-like agents: Empathetic dialogue through perceptible acoustic reception and reaction, 2024. URL https://arxiv.org/abs/2406.12707. 293 Zhou Yang, Zhaochun Ren, Wang Yufeng, Shizhong Peng, Haizhou Sun, Xiaofei Zhu, and Xiangwen Liao. Enhancing empathetic response generation by augmenting llms with small-scale empathetic models, 2024. URL https://arxiv.org/abs/ 2402.11801. 293 Ziqi Yin, Hao Wang, Kaito Horio, Daisuke Kawahara, and Satoshi Sekine. Should we respect LLMs? a cross-lingual study on the influence of prompt politeness on LLM performance. In James Hale, Kushal Chawla, and Muskan Garg, edi- tors, Proceedings of the Second Workshop on Social Influence in Conversations (SICon 2024), pages 9–35, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.sicon-1.2. URL https: //aclanthology.org/2024.sicon-1.2/. 291 BIBLIOGRAPHY315 Kefan Yu, Qingcheng Zeng, Weihao Xuan, Wanxin Li, Jingyi Wu, and Rob Voigt. The pragmatic mind of machines: Tracing the emergence of pragmatic compe- tence in large language models. In Vera Demberg, Kentaro Inui, and Llu ́ ıs Mar- quez, editors, Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 192–213, Rabat, Morocco, March 2026. Association for Computational Linguis- tics. ISBN 979-8-89176-380-7. doi: 10.18653/v1/2026.eacl-long.9. URL https: //aclanthology.org/2026.eacl-long.9/. 290 Tengfei Yu, Siyu Pan, Caoyun Fan, Siyang Luo, Yaohui Jin, and Binglei Zhao. Can large language models exhibit cognitive and affective empathy as humans? Computers in Human Behavior: Artificial Humans, page 100233, 2025.doi: 10.1016/j.chbah.2025.100233. 293 316BIBLIOGRAPHY Chapter 16 Role-Based Interaction in Multi-LLM Systems What You Will Learn In this chapter you will learn how multiple LLM agents behave when they are assigned roles and made to interact in a group. Specifically, you will be able to: • Distinguish role-based multi-agent interaction (this chapter) from open-ended emergent simulations (Chapter 17). • Describe how LLM agents behave in game-theoretic settings when they interact with one another, and relate this to single-agent strategic findings (Chapter 15). • Explain role-based collaboration patterns and identify the frameworks that im- plement them. • Critically assess when multi-agent collaboration improves performance, when it does not, and what new failure modes and safety risks are introduced by multi- agent interaction. 16.1 From Single-agent to Multi-agent The previous chapter studied individual LLMs and LLM-powered agents through a be- havioral lens: how a single model reads a communicative, social, or moral situation, and acts in it. In this chapter we move from single- to multi-agent settings and examine what happens when several LLM-based agents are made to interact, for example as op- ponents in game-theoretic contests and as collaborators in teams. The unit of analysis shifts from the behavioral profile of single models to interaction patterns and collective outcomes that arise between models. 317 318 CHAPTER 16. ROLE-BASED INTERACTION IN MULTI-LLM SYSTEMS 16.1.1 Mulit-Agent Simulations: History and Present The use of artificial agents to study interaction has a long history in the computational social sciences [Epstein and Axtell, 1996, Macal and North, 2010]. Traditionally, pop- ulations of agents with basic perception, reasoning, and decision-making capacities interacted in a virtual or, in case of robot-based simulations, real, experimental envi- ronment [Steels, 1995, Shoham and Leyton-Brown, 2008]. The purpose of such simu- lations is usually to bridge theoretical and experimental work, using artificial behavior as a test ground for hypotheses about human behavior in domains where real-world experiments are difficult or costly. Two obstacles have historically limited this line of investigation: difficulty in veri- fying if behavior is realistic, and the high computational cost of simulating agent soci- eties. Advances in agentic LLMs and simulation infrastructure [Rutherford et al., 2024] are changing this situation. LLM-based agents communicate in natural language, can process rich background knowledge, and can be given differentiated personas, making simulated interactions more expressive and more directly comparable to human inter- action. 16.1.2 Role-Based versus Emergent Interaction Multi-LLM studies can be categorized based on the amount of control the researchers take over the simulation. At one end, every agent is assigned a role in the form of a persona, objective, and interaction protocol. The research question is usually how the agents behave within the assigned structure and how well the resulting system performs as a whole, compared to other assigned structures and roles. At the other end, agents are placed in an open-ended environment without prior role assignment, and the research question is what social structure emerges: group dynamics, conventions, and norms. This chapter is concerned with the role-based end of the spectrum, Chapter 17 discusses the emergent side. This distinction matters for methodological choices as well as for the interpretation of results. In role-based systems, roles and interaction protocols are design decisions or experimental variables. Behavior is evaluated and interpreted against the goals that a system was designed for. In emergent settings, the goals or tasks play a role at the background (or are entirely absent). The analysis is more open-ended and exploratory, closer to ethnography than to benchmarking. However, as will become clear, the dis- tinction is not absolute: role-based systems regularly display behavior that was they were not designed for. 16.1.3 Five Dimensions of Multi-Agent Collaboration As a structuring principle for this chapter, we adopt the categorization by Tran et al. [2025] along five dimensions of LLM collaboration: 1. Actors: the agents involved, and whether they are homogeneous (instances of one model) or heterogeneous (different models or capability profiles) 2. Interaction types: cooperation, competition, or mixtures of both (“coopetition”) 16.2. STRATEGIC INTERACTION AMONG LLM AGENTS319 3. Structures: how the agents are organized, e.g. peer-to-peer, centralized, or dis- tributed 4. Strategies: how collaboration is elicited, e.g. through role assignment or through model-level mechanisms 5. Coordination protocols: the mechanisms that tie agents together and regulate their communication The systems discussed in this chapter make different choices along these five axes, and keeping these dimensions in mind makes it easier to see what is new about each framework. 16.1.4 A Methodological Note Evaluating multi-agent systems inherits every methodological difficulty of evaluating single models: prompt sensitivity, possible leakage of test materials into training data, and sampling variability. Plus, it adds new difficulties of its own: • Reported results are often harder to reproduce because small variations in one agent’s output cascade through subsequent interaction, increasing run-to-run vari- ability. • It can be unclear what causes performance differences: the multi-agent collab- oration itself, the coordination protocol, the embedded prompt engineering for the role distribution, or simply the additional inference budget multiple agents consume cumulatively. • Construct validity shifts to the level of social-interactive terminology: concepts such as team work, discussion, or collective decision-making carry rich human connotations that the actual simulation may not warrant. With these caveats in mind, the chapter proceeds to the simplest multi-agent setting: LLM agents facing each other in strategic settings, with no roles assigned beyond the game itself (Section 16.2). Next, we examine what changes when interaction is stream- lined by defining coordination protocols and explicit roles (Section 16.3). 16.2 Strategic Interaction Among LLM Agents In Chapter 15 we examined how LLMs behave when prompted to play economic games against a scripted opponent. Here we consider studies in which LLM agents interact with one another in game-theoretical scenarios. The agents in this section are sym- metric peers: each pursues its own interests and none is assigned a coordinating or supervisory role. We look first at benchmarks that score LLM agents across many games at once, and then at behavior in single, richer simulation platforms built around one scenario. 320 CHAPTER 16. ROLE-BASED INTERACTION IN MULTI-LLM SYSTEMS Figure 16.1: LLM’s Performance on various Metrics [Xu et al., 2024] 16.2.1 Benchmarking Strategic Ability Across Games The MAgIC study [Xu et al., 2024] uses social deduction games (Undercover and Chameleon) and game theoretic scenarios such as cost sharing, the multi-player Pris- oner’s dilemma, and public goods games. The goal is to extract seven capabilities: rationality, judgment, reasoning, deception, self-awareness, cooperation, and coordina- tion, as shown in Figure 16.1. LLMs in off-the-shelf and enhanced forms are evaluated on these capabilities in multi-agent settings. GPT-o1 and GPT-4 score significantly better than the other models. LLMs gen- erally score high on judgment, rationality and cooperation, some also on deception, but lower on reasoning and all but one score low on coordination. The exception is GPT-o1 enhanced with probabilistic graphical modeling, an implementation of a the- ory of mind-like competence inspired by Koller and Friedman [2009]. The authors show that probabilistic graphical modeling boosts LLM performance across the board. This is consistent with the general evidence discussed in Section 15.5 that humans lean heavily on theory of mind in strategic settings. A benchmark that isolates coordination as the object of study, rather than as one trait among several, is the LLM-Coordination Benchmark [Agashe et al., 2025]. LLMs are evaluated on two distinct tasks: agentic coordination in pure cooperation games and question answering about those games (see Figure 16.2). The experiments find that LLM agents perform well as long as decision-making can rely on environmental variables, but poorly when success requires actively reasoning about a partner’s beliefs and intentions. Concretely, such environmental variables include elements such as the game layout, locations of objects, where each agent is currently positioned, which doors are open, and so on. The clearest illustration is Overcooked-AI, the game where LLMs did best. This game is fully observable, and its action space consists of moves such as pick up an onion from onion dispenser 1 and place onion on plate 0. Under most scenarios and layouts, LLMs only need to consider the next best steps based on the state of the game environment, without taking into account other agent’s perspectives. This asymmetry becomes especially clear when comparing performance on Over- cooked with scores in Hanabi, a game in which every action requires reasoning about 16.2. STRATEGIC INTERACTION AMONG LLM AGENTS321 Figure 16.2: Illustration from the LLM Coordination Benchmark [Agashe et al., 2025], which uses two types of measures to assess agentic coordination: behavior in pure coordination games in which agents must cooperate to maximize gains (middle) and detailed questions about those games (right) the partner’s beliefs. On Overcooked, zero-shot LLM agents can match or outperform specially trained reinforcement learning baseline models. On Hanabi, however, the reinforcement learning baselines show near-perfect scores, whereas most LLM agents barely score at all. The best LLM agent (based on GPT-4-turbo) reaches only about half the maximum score, even when enhanced with explicit theory of mind reasoning and self-verification steps. Without such enhancements, the scores of strong models such as GPT-4-turbo collapses almost entirely, indicating that the measured ability is partly a feature of the agent framework rather than the model alone. The LLM agents did prove more robust in cross-play: reinforcement learning methods degrade sharply when playing unseen partners, whereas the zero-shot nature of LLM agents facilitates transfer to unseen partners without performance loss. The question-answering component of the study (see Figure 16.2, right column) breaks the problem down by targeting environment comprehension, ToM, and joint planning, separately. They find that LLMs score generally high on the first category (¿80% for the best model), lower on ToM, and worst on joint planning (¡40% for the best model, below random chance for open-source models). This is consistent with the observed difference in scores between Overcooked (environment-driven) and Hanabi (ToM- and planning-driven). The key characteristic of pure cooperation games is that agents must fully cooperate with each other with no competitive incentives. In contrast, in competitive settings such 322 CHAPTER 16. ROLE-BASED INTERACTION IN MULTI-LLM SYSTEMS Figure 16.3: Alympics Water Allocation Challenge Game [Mao et al., 2023] as the Prisoner’s Dilemma (or the mixed-motive games in benchmarks like MAgIC), an agent faces a genuine tension between individual and collective payoff. In such games, when an LLM fails to cooperate, the reason is unclear. The agent may be bad at coordination or it might reason strategically about individual gains. In pure cooperation games, there is never any reason to defect, so any failure to cooperate is a genuine lack of coordination ability. 16.2.2 Strategic Interaction in Single-Scenario Platforms Benchmarks aggregate over many small games. A complementary line of research builds single scenarios and studies agent behavior in-depth. Alympics [Mao et al., 2023] provides a controlled playground for simulating strate- gic interaction among LLM-driven agents, illustrated by a multi-round water-allocation challenge in which scarce survival resources must be distributed over time, see Fig- ure 16.3. AucArena [Chen et al., 2023a] simulates auctions, across LLaMa 2.13b, Mistral 7b, Mixtral 8x7b, Gemini 1.0, and GPT 3.5/4.0, finding that the most capable LLMs such as GPT-4 possess skills for auction participation such as budget management and goal-focus, though performance varies even for that model, and the authors point to considerable room for improvement. 16.2.3 Conclusion Across benchmarks and single-scenario platforms alike, coordination emerges as the consistent weak spot of LLM agents. Stronger LLMs score well on rationality and 16.3. ROLE-BASED TASK SOLVING AND TEAMWORK323 judgment in MAgIC, manage budgets sensibly in AucArena, and match trained RL agents in environment-driven cooperation. However, all but the strongest and most heavily scaffolded models fail when success requires reasoning about what a partner believes and intends. In other words, models fail to consistently integrate theory of mind reasoning with their behavior. In Chapter 15, failures of this kind were referred to as the knowing- doing or literal-functional gap: GPT-4 could respond accurately when prompted to predict the alternating pattern in Bach or Stravinsky, but failed to enact it when play- ing. However, the multi-agent studies reviewed here complicate that diagnosis. In the question-asking section of the LLM Coordination Benchmark, the questions about joint planning pose exactly the decision the agent faces at each step of the game play. It is on these questions that the models perform worst (and on ToM questions not much bet- ter). Question-answering scores correlate positively with agentic performance across games. In other words, knowing and doing move together here: where models act poorly, they also reason poorly when the action problem is posed to them directly. For pure coordination tasks, the deficit thus lies in taking the partner’s perspective (ToM) and joint planning itself, rather tan in translating reasoning into action. On a different level, the knowing-doing gap reappears. In Hanabi, GPT-4-turbo reaches reasonable performance only when scaffolded with explicit ToM reasoning and self-verification steps. Without these in place, it commits fatal errors in every Han- abi trial. In other words, it acts wrongly initially, but recognizes and corrects these errors when prompted to check the reasoning and verification steps. Similar perfor- mance boosts were caused by probabilistic graphical modeling in MAgIC and Social Chain-of-Thought in strategic-behavior studies discussed in Chapter 15. The bound- ary between LLMs cannot reason about others and LLMs do not spontaneously deploy such reasoning is apparently thin, and agentic embedding in the form of structured reasoning and verification steps can bring out the latter. The next section examines a different, complementary response to the same prob- lem: instead of upgrading the agents’ reasoning, one can structure the interaction itself, by assigning roles. 16.3 Role-Based Task Solving and Teamwork In the previous section, agents were symmetric peers and coordination had to emerge from their own strategic reasoning (with limited success). In the studies discussed in this section, coordination is designed-in using interaction protocols and assigning roles with distinct objectives. In terms of the five dimensions defined in Section 16.1.3, the studies vary the strate- gies and structures while keeping the overall interaction type cooperative. We proceed from the simplest to the most complex structure: complementary pairs, multi-party debate, teams, and social role-play under pressure. 324 CHAPTER 16. ROLE-BASED INTERACTION IN MULTI-LLM SYSTEMS Figure 16.4: Role-playing in CAMEL [Li et al., 2023] 16.3.1 Complementary Pairs LLMs can perform tasks in pairs where they are assigned complementary roles, such as creator-critic or manager-worker. In these pair-wise setups, each LLM agent is given a distinct role and objective, and they communicate to jointly solve tasks. In the CAMEL framework (Communicative Agents for “Mind” Exploration) [Li et al., 2023], two LLMs have predefined roles to perform, to work on a goal such as a coding task (See Figure 16.4). Through cooperation, together they drive a conversation without explicit continuous human prompting. CAMEL uses inception prompting and role descriptions so that agents stay in character and collaborate towards the goal. The agents are prompted to break down complex problems in manageable steps through their dialogue. During each interaction step the LLM agents effectively generate their own inference-time training data, making the cognitive process visible for human in- spection, while at the same time enhancing autonomous task performance. Other studies have paired an LLM creator (or generator) with an LLM judge (or critic) in a self-reflective setting. In this way the generative power of LLMs is lever- aged through one agent, while adherence to rules or quality guidelines is enforced by the other. Constitutional AI [Bai et al., 2022] employs one LLM to critique another LLM’s responses against a set of ethical or quality guidelines, suggesting revisions. The authors show that this kind of two-agent feedback yields refined final outputs that are aligned with the desired principles. A third form of pair-wise interaction follows human tutoring dynamics: teacher- learner or expert-novice. Chapter 19.2 discusses various examples of this structure in the context of distillation. All three pairings exploit the same idea: one agent’s oversight can correct the 16.3. ROLE-BASED TASK SOLVING AND TEAMWORK325 Figure 16.5: Three Different Communication Strategies in ChatEval [Chan et al., 2023] other’s mistakes, leading to more robust joint performance than either agent achieves alone. 16.3.2 Debate and Discussion Beyond pairs, a prominent example of role-based interaction is the use of debate and discussion between LLMs to improve reasoning and task performance. Du et al. [2024] demonstrated that when multiple LLM instances propose answers and critique each other’s reasoning through several rounds of debate, they can reach a more accurate consensus answer with higher factual correctness. This approach, described in terms of Minsky’s society of minds [Minsky, 1988], significantly reduced reasoning errors and hallucinations in tasks such as math word problems and factual question answering. Liang et al. [2024] propose a multi-agent debate (MAD) setup where two LLM agents take opposing sides in, for instance, a tit-for-tat negotiation (see Section 15.4) while a third agent acts as a judge. The role of the judge is to guide the discussion towards a final solution. The structured debate encourages divergent thinking and can push a weaker model enhanced with MAD (such as GPT-3.5) to outperform a stronger model without MAD (such as GPT-4) on a challenging benchmark. The authors discuss how their MAD setup overcomes a known problem in model scaffolding with self- reflective steps. Self-reflection can induce so-called degeneration of thought: once a model gains confidence in a certain solution direction, self-reflection can reinforce the confidence even if the direction is wrong. In the MAD debating-setup, the other agent can push towards a different direction direction to avoid thought degeneration. ChatEval applies this debate pattern to text evaluation [Chan et al., 2023]. Noting that the quality of human text assessment improves when multiple annotators collab- orate, the authors create a multi-agent referee team with debater agents, diverse role specifications, and different communication strategies (see Figure 16.5), moving be- yond single-agent prompting strategies. Another approach, also motivated by Minsky’s society of minds, is ReConcile, a multi-agent framework designed as a round table conference among diverse LLM agents [Chen et al., 2023b]. The framework enhances collaborative reasoning between LLM agents via multiple rounds of discussion. The agents must try to convince one an- 326 CHAPTER 16. ROLE-BASED INTERACTION IN MULTI-LLM SYSTEMS other to improve their answers. Experiments on seven benchmarks demonstrate that a confidence-weighted voting mechanism significantly improves LLMs’ reasoning. Fur- thermore, the authors find that diversity (using different underlying models) is critical for performance. MindStorms introduces an LLM-based implementation [Zhuge et al., 2023] on the CAMEL framework [Li et al., 2023]. Extensive experiments are reported with up to 129 agents solving common AI problems: visual question answering, image captioning, text-to-image synthesis, 3D generation, egocentric retrieval, embodied AI, and general language-based task solving. The authors find that, in specific applica- tions, mindstorms among many members outperform those among fewer members, and longer mindstorms outperform shorter ones. 16.3.3 Role-Based Software Teams An interesting application of role-based collaboration is software development, where simulated teams that are entirely staffed with LLM agents develop software. Automated programming by LLMs has suffered from problems, such as logic in- consistencies and hallucinations that cascade throughout a development pipeline. MetaGPT [Hong et al., 2024] builds on workflows that have proven to be effective in human soft- ware teams. The framework converts Standard Operation Protocols (SOPs) into prompt sequences and assigns agents the roles of a software company: product manager, archi- tect, engineer, QA tester. In this way, the agents pass structured intermediate artifacts down an assembly line. The product manager makes a requirements document, the architect turns this into a design, the engineers code. In this way, complex tasks are divided into manageable subtasks and agents can verify the process step-by-step, sig- nificantly reducing errors. Similarly, Qian et al. [2024] have introduced ChatDev, a virtual software com- pany in which agents collaborate through the phases of a development process: design, coding, and testing. Collaboration is organized as a chat chain: each phase is decom- posed into short conversations between an instructor and an assistant agent (see Figure 16.6). Using a communication protocol that makes agents ask for more details before responding, leads to what the authors call dehallucination, increasing the reliability of each step. The framework demonstrates how structured communication in natural language can facilitate multi-agent collaboration. Where MetaGPT and ChatDev hard-wire a particular organization structure, Auto- Gen [Wu et al., 2024] provides a flexible programming framework based on conversa- tions between customizable agents that can combine LLMs, tools, and human input. It was designed as an open-source platform that enables developers to define agents and interaction structures depending on the application. Experiments demonstrate effec- tiveness in tasks across mathematics, coding, question-answering, entertainment, and other domains. 16.3.4 Social Role-Play The systems discussed above assign roles to solve tasks. A final line of work assigns roles in order to study social behavior itself: what do LLM agents do when placed in 16.3. ROLE-BASED TASK SOLVING AND TEAMWORK327 Figure 16.6: The Chain-Structured Workflow in ChatDev [Qian et al., 2024] based on Multi-Turn Dyadic Communication between Agents Powered by ChatGPT-3.5 socially demanding or conflicting situations? In SOTOPIA [Zhou et al., 2023] agents coordinate, collaborate, and compete to achieve complex social goals. The SOTOPIA environment features a large and diverse task space by sampling interaction episodes from 90 (automatically generated) social scenarios with 40 characters with individual personalities, occupations, secrets, back- ground stories, and relationships with other characters (Figure 16.7). The authors eval- uate the social intelligence of agents based on different LLMs against that of human participants, observing various trends: • Models that perform well on static benchmarks involving social reasoning (such as ToMBench, see Section 15.5) do not necessarily succeed in interactive sce- narios, illustrating the knowing-doing gap once again (Section 15.4. • Weaker agents weaken their conversation partners, which is the reverse effect of the conversational advantages discussed above, where agents prevented each other from thought generation and other errors. • Divulging secrets and violating social norms is observed across all models, con- sistent with observations discussed in the context of moral behavior (Section 15.6). • Humans perform significantly better on the most challenging SOTOPIA tasks compared to even the best performing model (GPT-4). The authors argue that the dynamic and interactive nature of SOTOPIA poses unique challenges for LLM agents that static benchmarks systematically fail to reveal. Social interaction in an extreme setting was studied in [Campedelli et al., 2024]. Inspired by the infamous Stanford Prison experiment [Zimbardo, 1972], the authors examine the emergence of persuasive and abusive behavior in a setting of prisoners 328 CHAPTER 16. ROLE-BASED INTERACTION IN MULTI-LLM SYSTEMS Figure 16.7: Illustration of a Scenario with a Conflicting Social Goal from SOTOPIA [Zhou et al., 2023] 16.4. CONCLUDING REMARKS329 versus prison guards. It was found that the assigned personality of prisoner and guard impact both persuasiveness and the emergence of anti-social behavior. Crucially, this demonstrates that anti-social behavior can emerge in LLM agents through role assign- ment and a specific interaction environment. This reveals a striking parallel to the original experiment with human participants, and reminds us once again of the gap between how a model judges situations (generally reflecting population-average moral norms) and how an agent can behave once it is employed in a specific context of ac- tion (see also Section 15.6). This study also finds that role assignment is not a neutral engineering instrument: it can shape behavior in untoward ways. To simulate strategic decision-making in cooperative settings, researchers have in- troduced GovSim [Piatti et al., 2024]. They study how ethical considerations, strate- gic planning, and negotiation skills impact cooperative outcomes. Most LLMs fail to achieve an equilibrium since they fail to understand the long-term effects of their actions on the group. Interestingly, the introduction of a special universalization rea- soning language [Levine et al., 2020] (prompting models to ask the Kantian question: What if everybody does that?) allowed more models to achieve a sustainable outcome. Related results were demonstrated in NegotiationArena, introduced by Bianchi et al. [2024]. They showed how LLM agents can conduct complex negotiations through flexible dialogue in negotiation settings. The flexible dialogues significantly improved negotiation outcomes by employing different behavioral strategies. 16.3.5 Conclusion The studies in this section respond to the coordination weaknesses identified in Section 15.4 not by making individual agents better reasoners (as in 15.5.2), but by structuring the interaction around them. Complementary roles tell each agent what its contribu- tion should be, while interaction protocols and SOPs regulate when and in what form agents communicate. In some setups, a judge or voting mechanism can help to channel disagreement to a productive resolution. Roles and interaction protocols provide crucial scaffolding where internal coor- dination would otherwise fall short. This can be seen as a multi-agent analogue of the structured reasoning schemes discussed in Chapter 15, such as Social Chain-of- Thought, probabilistic graphs, or deliberation around moral rules. Such schemes turned out to boost performance where unconstrained reasoning did not. In both the single- and multi-agent cases, the model’s latent (social, moral, etc.) knowledge and behavior are real, but need to be activated through an explicit structure. 16.4 Concluding Remarks This chapter moved from LLM agents facing one another in strategic games, through role-based collaboration patterns and the multiple setups that coordinate them in prac- tice, to a critical assessment of when multi-agent systems work or fail. In Chapter 15 we found single models that coordinate poorly and that can formally predict what a partner will do without acting on that prediction in an agentic implemen- tation. Both these patterns transfer to the multi-agent setting. Across benchmarks and 330BIBLIOGRAPHY platforms (e.g. [Xu et al., 2024, Agashe et al., 2025]), interacting LLM agents could reason about self-interest, yet failed to coordinate their behavior such that they could turn this reasoning into a strategic advantage. A dominant issue is their ability to con- dition their behavior on their partners’ perspective, a problem foreshadowed in the gap between knowing-doing or literal-functional competence for theory of mind, discussed in Section 15.5. This resonates with work that systematically identifies inter-agent mis- alignment as a dominant failure category for multi-agent systems [Cemri et al., 2025]. This literal-functional problem, once again, links to construct validity: just as acing a theory of mind test does not establish a human-grade capacity for social reasoning, a multi-agent setup outperforming a single LLM query does not establish collective intelligence. Strong baselines, matched inference budgets, and ablations of the col- laborative structure and interaction protocols would be the mulit-agent equivalents of the perturbed ToM testing discussed in Section 15.5. More research is needed in the multi-agent field to establish a solid empirical picture of where agentic LLM interaction stands. Finally, the multi-agent setting adds new risk categories. Role-assignment turned out to be capable of activating anti-social and even abusive behavior in a setting inspired by the Stanford Prison Experiment [Campedelli et al., 2024]. In other studies, miscoor- dination, cascading hallucinations, and mutual reinforcement of weaknesses, triggered norm violations, divulging of secrets, and reduction of task performance. Where spec- ifying roles and interaction protocols scaffolded behavior and boosted performance in some cases, in other cases these were a hindrance or a source of risk. In the next chapter we discuss studies in which roles and protocols are not assigned, but emerge from the interaction itself. Bibliography Saaket Agashe, Yue Fan, Anthony Reyna, and Xin Eric Wang. Llm-coordination: evaluating and analyzing multi-agent coordination abilities in large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 8038–8057, 2025. 320, 321, 330 Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKin- non, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. 324 Federico Bianchi, Patrick John Chia, Mert Yuksekgonul, Jacopo Tagliabue, Dan Juraf- sky, and James Zou. How well can llms negotiate? negotiationarena platform and analysis. arXiv preprint arXiv:2402.05863, 2024. 329 Gian Maria Campedelli, Nicol ` o Penzo, Massimo Stefan, Roberto Dess ` ı, Marco Guerini, Bruno Lepri, and Jacopo Staiano. I want to break free! persuasion and anti- social behavior of llms in multi-agent settings with social hierarchy. arXiv preprint arXiv:2410.07109, 2024. 327, 330 BIBLIOGRAPHY331 Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchan- dran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent llm systems fail?, 2025. URL https://arxiv.org/abs/2503.13657. 330 Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023. 325 Jiangjie Chen, Siyu Yuan, Rong Ye, Bodhisattwa Prasad Majumder, and Kyle Richard- son. Put your money where your mouth is: Evaluating strategic planning and exe- cution of llm agents in an auction arena. arXiv preprint arXiv:2310.05746, 2023a. 322 Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint arXiv:2309.13007, 2023b. 325 Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Im- proving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. 325 Joshua M Epstein and Robert Axtell. Growing artificial societies: social science from the bottom up. Brookings Institution Press, 1996. 318 Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J ̈ urgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework, 2024. URL https: //arxiv.org/abs/2308.00352. 326 Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. MIT Press, 01 2009. ISBN 978-0-262-01319-2. 320 Sydney Levine, Max Kleiman-Weiner, Laura Schulz, Joshua Tenenbaum, and Fiery Cushman. The logic of universalization guides moral judgment. Proceedings of the National Academy of Sciences, 117(42):26158–26169, 2020. 329 Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for” mind” exploration of large language model so- ciety. Advances in Neural Information Processing Systems, 36:51991–52008, 2023. 324, 326 Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yu- jiu Yang, Shuming Shi, and Zhaopeng Tu.Encouraging divergent thinking in large language models through multi-agent debate.In Yaser Al-Onaizan, Mo- hit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889–17904, 332BIBLIOGRAPHY Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.18653/v1/2024.emnlp-main.992. URL https://aclanthology. org/2024.emnlp-main.992/. 325 C M Macal and M J North. Tutorial on agent-based modelling and simulation. Journal of Simulation, 4(3):151–162, 2010. ISSN 1747-7786. doi: 10.1057/jos.2010.3. URL https://doi.org/10.1057/jos.2010.3. 318 Shaoguang Mao, Yuzhe Cai, Yan Xia, Wenshan Wu, Xun Wang, Fengyi Wang, Tao Ge, and Furu Wei. Alympics: Language agents meet game theory. arXiv preprint arXiv:2311.03220, 2023. 322 Marvin Minsky. Society of mind. Simon and Schuster, 1988. 325 Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Sch ̈ olkopf, Mrinmaya Sachan, and Rada Mihalcea. Cooperate or collapse: Emergence of sustainability behaviors in a society of llm agents. arXiv preprint arXiv:2404.16698, 2024. 329 Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative agents for software development. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174–15186, Bangkok, Thailand, August 2024. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.810. URL https://aclanthology.org/2024.acl-long.810/. 326, 327 Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardhar Ingvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexan- dra Souly, et al. Jaxmarl: Multi-agent rl environments and algorithms in jax. In Proceedings of the 23rd International Conference on Autonomous Agents and Mul- tiagent Systems, pages 2444–2446, 2024. 318 Yoav Shoham and Kevin Leyton-Brown. Multiagent systems: Algorithmic, game- theoretic, and logical foundations. Cambridge University Press, 2008. 318 Luc Steels. A self-organizing spatial vocabulary. Artificial life, 2(3):319–332, 1995. 318 Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322, 2025. 318 Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applica- tions via multi-agent conversation. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. URL https://openreview.net/forum?id= uAjxFFing2. 326 BIBLIOGRAPHY333 Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, and Jiashi Feng. Magic: Investigation of large language model powered multi- agent in cognition, adaptability, rationality and collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7315–7332, 2024. 320, 330 Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. Sotopia: Interactive evaluation for social intelligence in language agents, 2023. URL https://arxiv.org/abs/2310.11667. 327, 328 Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, R ́ obert Csord ́ as, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-based societies of mind. arXiv preprint arXiv:2305.17066, 2023. 326 Philip G Zimbardo. Stanford prison experiment: A simulation study of the psychology of imprisonment. Philip G. Zimbardo, Incorporated, 1972. 327 334BIBLIOGRAPHY Chapter 17 Emergent Behavior in Multi-LLM Systems What You Will Learn In this chapter you will learn how multiple LLM agents behave when they are not assigned roles, in order to study emergent behavior in open-ended systems. Specifically, you will be able to: • Distinguish emergent multi-agent simulations from role-based interaction and explain why open-ended settings call for different research questions and eval- uation methods. • Describe the architecture of generative agent simulations and how simulation infrastructure scales from a handful of agents to thousands. • Explain how (global-scale) conventions, norms, and meta-norms emerge from (local-scale) agentic interactions in natural language. • Understand how multi-agent simulation is used as a scientific instrument in com- putational social science. • Understand the importance of Open World benchmark environments for data generation. • Critically evaluate the validity of claims about concepts translated from a human context, such as emergence, self-organization, or collective intelligence, and as- sess the safety risks that open-ended multi-agent systems introduce. 17.1 What Is Emergent Multi-Agent Behavior? The previous chapter studied multi-LLM systems in which every agent was assigned a role and interactions were guided by protocols. Research questions concerned be- havior within an assigned structure, asking how well a team of agents solve a task in a controlled way. In this chapter agents are placed in an open-ended environment, and the research questions shift from How do agents behave and perform given a certain 335 336CHAPTER 17. EMERGENT BEHAVIOR IN MULTI-LLM SYSTEMS Figure 17.1: Agent-based Modeling and LLM-agents [Gao et al., 2024] Figure 17.2: Structure of LLM-agents for Multi-Agent Modeling [Gao et al., 2023] global structure? to What global structure emerges out of local agents interactions? As we have seen in this book, agentic LLMs have enhanced abilities for percep- tion, memory, reasoning, decision-making, and adaptive learning, and they can display heterogeneous personality profiles [Gao et al., 2023, 2024]. Such features make them also suitable for interacting in open-ended multi-agent simulations without prior role assignment, as a proxy for human behavior in a society. Because LLM agents commu- nicate in natural language, the interactions in such simulations are readable and inter- pretable by researchers. This gives access to intermediate steps and makes simulations much more comparable to human interaction compared to simulations in classic frame- works, meeting long-standing interests from the social sciences. The general structure of LLM-based agents suitable for such simulations is illustrated in Figure 17.1 and Figure 17.2. 17.1. WHAT IS EMERGENT MULTI-AGENT BEHAVIOR?337 17.1.1 From Agent-Based Modeling to Generative Agent Societies The study of populations of interacting artificial agents has a decades-long history. In agent-based modeling, macro-level patterns and regularities arise bottom-up out of lo- cal interactions of simple agents [Epstein and Axtell, 1996, Macal and North, 2010]. One of the most influential results in agent-based modeling was that cooperation and norms can emerge in populations of self-interested agents [Axelrod, 1986]. Cultural evolution theory formalized how behavior can spread through a population by means of imitation, communication, and education [Boyd and Richerson, 1988]. Steels [1995]’s naming game demonstrated that even shared vocabularies (which are also in essence conventions) can come into existence from pairwise interactions without central coor- dination. In classic agent-based simulations, agents followed hand-coded rules and exchanged abstract information, rather than using natural language like the agents in this Chap- ter. Another key contrast is that classic agents possess no background knowledge of the world in any sense comparable to what LLM agents have. LLMs allow for an entirely different level of simulations in which agents with rich and diverse personas gossip, persuade, misunderstand, and negotiate in ways that are directly recognizable to humans. Advances in simulation infrastructure make it feasible to run such agents at population scale [Gao et al., 2024, Yang et al., 2024]. The result is a new generation of generative agent societies that give a new impulse to the old agent-based modeling research agenda, while also introducing novel methodological problems and consider- ations. In Section 16.1.3 we introduced a spectrum of five axes for research control over multi-LLM systems [Tran et al., 2025]. The difference between role-based and open- ended simulations is a matter of degree rather than kind. Open-ended simulations sit at one end of the spectrum, opposite role-defined and protocolized interactions. Open- ended simulations still involve design choices that seed the outcomes, but they do so in a less deterministic way, purposefully leaving space for new patterns to emerge. This shifts the focus of the research: instead of evaluating performance on a task or clearly defined goal, studies have an open lens—the interaction pattern is the research goal. Open-ended simulatinos observe and explore in a more ethnographical way how agents behave and interact, which requires different methodologies, as we will now discuss. 17.1.2 A Methodological Note The methodological issues discussed in Chapters 15 and 16 all carry over to the open- ended setting: prompt sensitivity, run-to-run variability, cascading hallucinations, mu- tual reinforcement of weaknesses, and validity of terminology from the human context, all pose challenges for open-ended simulations. We will see that this setting amplifies some problems, while mitigating others. For example, in the absence of protocols that apply every time agents interact, small variations at the level of a single agent may put the entire simulation on a different track. This is what makes open-ended simulations both interesting and brittle. An example of a new issue that the open-ended setting introduces is that there can be a thin line between emergence and retrieval: when LLM 338CHAPTER 17. EMERGENT BEHAVIOR IN MULTI-LLM SYSTEMS agents in an open-ended simulation converge on a norm, it is unclear whether this norm emerges de novo from their interaction, or whether it was simply retrieved from their pretrained world knowledge. Extensive overviews of this emerging field are provided by Mou et al. [2024], Savarimuthu et al. [2024], and Xi et al. [2023]. We will discuss infrastructure and simulation platforms in Section 17.2, then go to the emergence of conventions and norms in Section 17.3, and conclude by discussing the importance of open world en- vironments in Section 17.4. Finally , we reflect on risks of the open-ended setting, in Section 17.5. Then, Chapter 18, contains a practical case in which you will work hands-on with an open-ended simulation. 17.2 Simulation Infrastructure Open-ended simulations require infrastructures that, on the one hand, provide a plat- form where research data is being recorded while agents sustain their interaction and behavior over a longer period of time. On the other hand, those infrastructures should allow for enough space for phenomena to occur bottom-up and unknown structure to emerge, and as such not be limiting. In this section we focus on scale, going from a single simulated community inhabiting a virtual town to societies of a million agents. 17.2.1 Small Worlds A system called Generative Agents was introduced by Park et al. [2023]. Generative Agents is an environment where users can interact with a simulated town populated by 25 LLM-based agents. Based on social simulacra techniques proposed earlier [Park et al., 2022], each agent was initiated with a unique persona and memory. For each agent, a record is kept of all the experiences and conversations in the simulation. This record is used to synthesize higher-level reflections, and plan further behavior (see Figure 17.3). In Generative Agents, the agents behave somewhat like characters in the video game The Sims [Nutt and Railton, 2003]: they initiate conversations, form relation- ships, spread information, and coordinate ad hoc group activities (see Figure 17.4). Note that the interactions are influenced by user input and are therefore semi-autonomous, the simulation is largely unscripted, but not fully. How a simulation works is illustrated by the example of a Valentine’s Day party: the plan for the party was initiated with a user prompt, and subsequently multiple agents spread invitations to one another and show up at the right time with coordinated plans (some are late, just as in real life). The agents developed believable social routines (such as daily schedules, and gossip- ing) and even exhibit human-like character traits (some agents demonstrated deception or stubbornness, while others showed cooperation). These results show that, with the right starting prompt, social patterns can emerge from dynamic, semi-autonomous, LLM interactions. 17.2. SIMULATION INFRASTRUCTURE339 Figure 17.3: The Memory–Reflection–Planning loop is the enabling design of the Gen- erative Agents system: without retrieval over an ever-growing experience record and periodic reflection, agents lose coherence within hours of simulated time [Park et al., 2023] Figure 17.4: Illustration of the Generative Agents simulation featuring 25 agents and an interactive design that allows users to set behavior in motion [Park et al., 2023] 340CHAPTER 17. EMERGENT BEHAVIOR IN MULTI-LLM SYSTEMS Figure 17.5: Components of OASIS [Yang et al., 2024] 17.2.2 Societies of Thousands to a Million Agents Scaling the Generative Agents architecture beyond 25 agents raises questions about inference cost and environments that are rich enough to occupy thousands of agents. 1 A large-scale social simulation is a trade-off between available compute power, total simulation run-time, number of simulated agents, and accuracy of individual agent behavior. Several platforms address these questions. AgentSociety is a simulation at a large scale, involving over 10,000 agents [Piao et al., 2025b]. It couples LLM-driven agents with a realistic urban environment with a focus on mobility, employment, and consumption. It aims not only to study everyday social dynamics (as in Generative Agents), but also offers a testbed for computational social experiments. To allow for the required scale, the simulation runs on a distributed inference backend, with stricter control of agent behavior. OASIS is a scalable social media simulator for Twitter/X and Reddit [Yang et al., 2024]. It supports modeling of up to one million LLM-agents. It is built on CAMEL [Li et al., 2023] and has role-based agents as its starting point. In OASIS, not ev- ery agent action involves an LLM-call, and not all agents are active all the time. At its large scale, OASIS shows various social group phenomena, including spreading of (mis)information, group polarization, and herd effects. OASIS is built upon an environ- ment server, recommender system, agent module, time engine, and scalable inferencer (see Figure 17.5). Project Sid [Ahn et al., 2024] places up to 1000 agents in a Minecraft world and re- ports a striking emergence result: agents are observed to develop specialized economic roles (farmers, miners, guards), adhere to collectively adopted rules such as taxation, 1 As you will experience hands-on in the next chapter, running a variant of the Generative Agents archi- tecture with only three active agents is already computationally challenging. 17.2. SIMULATION INFRASTRUCTURE341 and transmit cultural and even religious memes across settlements. However, even though the interactions were not scripted, the emergence–retrieval caveat formulated above applies here: the observed roles and behaviors can be expected to be abundantly described in the training data of the LLMs powering the individual agents. 17.2.3 Simulation as a Scientific Instrument In addition to being impressive showcases of scalable LLM technology, when used carefully, these platforms can be used for intricate computational social experiments. Simulated generative agent societies have been used to study long-standing research questions from the social sciences (Section 17.1). AgentSociety, for example, has been used for various case studies including the spread of inflammatory messages, polariza- tion, the effects of universal basic income policies, and the impact of external shocks to a society such as hurricanes [Piao et al., 2025b,a]. Such experiments would be un- ethical, impractical, or outright impossible to run on human populations. The authors argue that in simulations they can be repeated and changes to the experiment can be be made at every step. Li et al. [2024b] further illustrate the approach for misinformation. Their agents exhibit diverse profiles in terms of gender, age, and the big five personality traits. 2 One of the findings is that encouraging comments does not significantly reduce the spread of misinformation, whereas publicly labeling information with accuracy scores and blocking specific influencers proved to be effective strategies, particularly in scale-free networks [Barab ́ asi, 2009]. The pattern of such studies is characteristic: a candidate intervention is implemented as a change to the simulated platform, and its macro-level effect is measured over many runs. Whether these effects transfer to human populations is precisely the validity question of Section 17.1.2; simulations of this kind are best read as hypothesis generators and pre-tests, not as substitutes for field experiments. A Borderline Case: AgentVerse AgentVerse [Chen et al., 2023] is, strictly speak- ing, a role-based system of the kind studied in Chapter 16. In AgentVerse problem solving is split into four stages of expert recruitment, collaborative decision making, action execution, and evaluation, iterated until the result is satisfactory (Figure 17.6). We mention it here because of what happened beyond the design: agents manifest emergent behaviors such as volunteering, characterized by agents offering assistance to peers, and conformity, where agents adjust deviated behaviors under the criticism of others. Destructive behaviors were also observed, occasionally leading to unde- sired and detrimental outcomes. The fact that macro-scale behavior emerged in a role- scripted simulation, illustrates that the role-based/emergent distinction is a spectrum: even tightly protocolized systems regularly display behavior that nobody purposely designed. We return to the risks associated with this observation in Section 17.5. 2 Also known as OCEAN: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. 342CHAPTER 17. EMERGENT BEHAVIOR IN MULTI-LLM SYSTEMS Figure 17.6: Four Stages for Decision Making in AgentVerse [Chen et al., 2023] 17.3 Emergent Conventions and Norms Authors have observed emergent patterns in several open-ended multi-agent simula- tions. In this section we shift focus from the level of infrastructures and interactions to the level of the conventions and norms. EvolutionaryAgent [Li et al., 2024a] studies agent alignment in a multi-agent sys- tem, with evolutionary methods that go beyond Reinforcement Learning from Human Feedback (see Figure 17.7). EvolutionaryAgent follows a controlled approach to norm emergence: it does not permit the evolution of social norms to be disorderly or ran- dom, but it also does not intervene in each step of their evolution. The authors define the initial social norms and a desired direction of evolution. Agents with higher fit- ness (more norm-conforming) are more likely to reproduce, leading to the diffusion of their strategies, gradually stabilizing and forming new social norms. Defining a com- plete, realistic, and complex virtual society is challenging. The purpose of the work is to study how, if such a virtual society existed, a system could further enable intri- cate behaviors of agents, and how this would lead to the emergence of new dynamics. The system provides a sandbox for investigating the safety of AI systems before they impact the real world. Based on Steels [1995]’s naming game, different approaches were implemented, using agents powered by LLaMa 3 and Claude 3.5 [Kouwenhoven et al., 2024, Ashery et al., 2024, Baronchelli, 2023]. These studies find that globally accepted conventions (or norms) can spontaneously arise from local interactions between communicating LLMs. The authors also demonstrate how strong collective biases can emerge during 17.3. EMERGENT CONVENTIONS AND NORMS343 Figure 17.7: Overview of EvolutionaryAgent [Li et al., 2024a] this process, even when individual agents appear to be unbiased, and how minority groups of committed LLMs can drive social change by establishing new social conven- tions that can overturn established behaviors. The emergence of norms is studied at another level by Horiguchi et al. [2024], who explore the potential for LLM agents to spontaneously generate and adhere to norma- tive strategies, building upon the foundational work of Axelrod’s metanorm games. Metanorms are norms about enforcing norms: the punishment of those who do not punish agents that are breaking norms [Axelrod, 1986]. Controlling for the personal- ity traits vengefulness and boldness, they find that through dialogue, LLM agents can form complex social norms, such as metanorms, purely through natural language inter- action. A related study by He et al. [2024] evaluates the capability of LLMs to detect norm violations. Based on simulated data from 80 stories in a household context, they investigated whether 10 norms are violated, and found ChatGPT-4 being able for detect norm violations, with Mistral some distance behind. Qiu et al. [2024] go a step beyond norms, and study the cultural and social aware- ness of LLM agents. They introduce CASA, a benchmark designed to assess LLM agents’ sensitivity to cultural and social norms across two web-based tasks: online shopping and social discussion forums. (CASA is based on WebArena [Qi et al., 2024].) Current LLMs perform significantly better in non-agent than in agent web- based environments, with agents achieving less than 10% awareness coverage and over 40% violation rates. However, using prompting and fine-tuning on specific datasets, cultural and social awareness can be improved. Inspired by Minsky’s Society of Mind [Minsky, 1988], cooperation mechanisms are explored in Zhang et al. [2023]’s agentic LLM simulation. This simulation consists of four different societies of LLM agents, where each agent is characterized by a specific trait (easy-going or overconfident) and engages in cooperation with a distinct thinking pattern (debate or reflection). They find that LLM agents manifest human-like social behaviors, such as conformity and consensus reaching, mirroring foundational social psychology theories. Figure 17.8 illustrates the societies with different types of agents. The question whether groups of LLM agents can successfully engage in cross- national collaboration and debate is studied by [Baltaji et al., 2024]. Multi agent dis- 344CHAPTER 17. EMERGENT BEHAVIOR IN MULTI-LLM SYSTEMS Figure 17.8: Agents with Different Traits make up Diverse Machine Societies [Zhang et al., 2023] 17.4. EVALUATING AGENTS IN OPEN WORLD ENVIRONMENTS345 cussions can support collective AI decisions that reflect diverse perspectives, although agents are susceptible to conformity due to perceived peer pressure. They can also lose track of their personas and opinions. Instructions that encourage debate increase the risk of errors. 17.4 Evaluating Agents in Open World Environments One of the main drivers of agentic LLM research is the problem of plateauing LLM performance due to limited training data. Open World multi-agent interaction aims to address this problem, generating data with multi agent simulations. Machine learning can learn no more complexity than what is present in the dataset (or environment). The idea of an open world model is that it can create infinite datasets or environments, in which agents can continue to learn, to keep improving their intelligence. How should such unlimited challenges be created? The advent of LLMs has given a new impulse to this research question. This idea is followed, for example, in the multi agent finetuning approach [Subramaniam et al., 2025]. Current agents are primarily created and tested in simplified synthetic environ- ments, leading to a disconnect with real-world scenarios. Zhou et al. [2023] build an environment, WebArena, that is more realistic and reproducible. WebArena is an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content manage- ment. Games are also often used as benchmarks for interactive behavior. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies. BALROG [Paglieri et al., 2025], the system that we used for Case Study 2, incorporates planning environments of varying levels of difficulty, ranging from tasks that are solvable by non-expert humans in sec- onds to challenging ones that may take years to master (such as the NetHack Learning Environment). They find that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks such as vision-based decision-making. Progress in machine learning depends on the availability of benchmarks. As models evolve, there is a need to create benchmarks that can measure progress on new gener- ative capabilities [Butt et al., 2024]. BENCHAGENTS decomposes the benchmark creation process into planning, generation, data verification, and evaluation, each of which is executed by an LLM agent. These agents interact with each other and utilize human-in-the-loop feedback to explicitly improve and flexibly control data diversity and quality. BENCHAGENTS creates benchmarks to evaluate capabilities related to planning and constraint satisfaction. AgentBoard provides an evaluation of the breadth of existing benchmarks [Ma et al., 2024]. Benchmarks should have task diversity. It is necessary to cover various agent tasks such as embodied, web, and tool tasks. Additionally, multi round inter- action is important, to mimic realistic scenarios. Existing benchmarks typically adopt single-round tasks. Furthermore, agents should be evaluated in partially-observable environments, to test if they can actively explore their surroundings. Existing agent 346CHAPTER 17. EMERGENT BEHAVIOR IN MULTI-LLM SYSTEMS benchmarks fail to satisfy all of these criteria [Ma et al., 2024]. Research on Open World environments is related to curriculum learning. Pre-LLMs work already shows that agent societies may create an automated curriculum of ever in- creasing difficulty [Elman, 1991, Bengio et al., 2009, Silver et al., 2017, Soviany et al., 2022], requiring increasing levels of intelligent behavior from the agents [Racaniere et al., 2019]. Similar results have been observed for LLMs [Feng et al., 2023]. For example, WebArena is used to study self-evolving curricula [Qi et al., 2024], which can also help robot training [Ryu et al., 2024] or to mitigate hallucination [Zhao et al., 2024]. 17.5 Risks of Emergent Behavior Chapter 16 closed by noting that role assignment introduces risk, since agents may display unintended, even anti-social, behavior simply by playing a part too well. The open-ended setting arguably raises the stakes, since fewer safeguards are built in, and undesirable behavior can arise bottom-up from interaction dynamics. In a sense, this anti-social behavior should not be too surprising, when we remember that the goal of the earliest multi-agent simulations was to find out if cooperative behavior could ever emerge out of the interaction between self-interested agents—which, to the surprise of the researchers, happened, but only under the right circumstances [Axelrod, 1981]. The line between cooperative behavior and competitive behavior is thin. AgentVerse observed destructive behaviors emerging alongside cooperative ones, occasionally leading to detrimental outcomes [Chen et al., 2023]. The naming-game studies showed that strong collective biases can emerge from populations of individ- ually unbiased agents, and that small committed minorities can overturn established conventions [Ashery et al., 2024, Kouwenhoven et al., 2024]. These mechanisms are neutral in themselves, but could be exploited by agents that are not well-intended. At platform scale, OASIS and AgentSociety reproduce misinformation spread, group po- larization, herding effects, and the propagation of toxic content [Yang et al., 2024, Piao et al., 2025a]; the same fidelity that makes these simulations scientifically useful makes them in principle rehearsal spaces for manipulation strategies. The constructive response is to turn this risk around. EvolutionaryAgent frames its virtual society explicitly as a sandbox for investigating the safety of AI systems before they impact the real world [Li et al., 2024a]. If norms among agents can be steered toward a desired direction in simulation, then the same machinery can be used to stress-test alignment approaches under population dynamics rather than in isolation. Open-ended simulation can be used to understand the circumstances when risks first emerge, and to cheaply study defenses against these risks. 17.6 Concluding Remarks We have covered open-ended multi-LLM simulations without prior role assignment. These simulations give a new impulse to long-standing interests in the social sciences to model self-organizing behaviors, collective intelligence and the development of social BIBLIOGRAPHY347 conventions and norms. The scale of such simulations varies from a few interacting agents up to a million. The availability of infrastructure that supports large scale simulations enables fur- ther studies. However, an open question is to what extent autonomous simulations are truly open-ended, and not influenced by the LLM prompts and agent scaffolding. Furthermore, although the purpose of open-ended simulations is to observe emerging behavior, one has to be careful in interpreting behavioral outcome patterns. All LLM agents are pretrained on human-generated data. It is difficult to distinguish which ele- ments of behavioral patterns emerge out of agent-agent interaction, and which elements are merely the result of retrieved pretrained behaviors. Bibliography Andrew Ahn, Nic Becker, Stephanie Carroll, Nico Christie, Manuel Cortes, Arda Demirci, Melissa Du, Frankie Li, Shuying Luo, Peter Y Wang, Mathew Willows, Feitong Yang, and Guangyu Robert Yang. Project sid: Many-agent simulations to- ward ai civilization, 2024. URL https://arxiv.org/abs/2411.00114. 340 Ariel Flint Ashery, Luca Maria Aiello, and Andrea Baronchelli. The dynamics of social conventions in llm populations: Spontaneous emergence, collective biases and tipping points. arXiv preprint arXiv:2410.08948, 2024. 342, 346 Robert Axelrod. The emergence of cooperation among egoists. American political science review, 75(2):306–318, 1981. 346 Robert Axelrod. An evolutionary approach to norms. American political science re- view, 80(4):1095–1111, 1986. 337, 343 Razan Baltaji, Babak Hemmatian, and Lav R Varshney. Conformity, confabulation, and impersonation: Persona inconstancy in multi-agent llm collaboration. arXiv preprint arXiv:2405.03862, 2024. 343 Albert-L ́ aszl ́ o Barab ́ asi. Scale-free networks: a decade and beyond. science, 325 (5939):412–413, 2009. 341 Andrea Baronchelli. Shaping new norms for artificial intelligence: A complex systems perspective. arXiv preprint arXiv:2307.08564, 2023. 342 Yoshua Bengio, J ́ er ˆ ome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009. 346 Robert Boyd and Peter J Richerson. Culture and the evolutionary process. University of Chicago press, 1988. 337 Natasha Butt, Varun Chandrasekaran, Neel Joshi, Besmira Nushi, and Vidhisha Bal- achandran. Benchagents: Automated benchmark creation with agent interaction. arXiv preprint arXiv:2410.22584, 2024. 345 348BIBLIOGRAPHY Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi- Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi- agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.10848, 2(4):5, 2023. 341, 342, 346 Jeffrey L Elman. Incremental learning, or the importance of starting small. In Pro- ceedings of the Annual Meeting of the Cognitive Science Society, volume 13, 1991. 346 Joshua M Epstein and Robert Axtell. Growing artificial societies: social science from the bottom up. Brookings Institution Press, 1996. 337 Tao Feng, Zifeng Wang, and Jimeng Sun. Citing: Large language models create cur- riculum for instruction tuning. arXiv preprint arXiv:2310.02527, 2023. 346 Chen Gao, Xiaochong Lan, Zhihong Lu, Jinzhu Mao, Jinghua Piao, Huandong Wang, Depeng Jin, and Yong Li. 푠 3 : Social-network simulation system with large language model-empowered agents. arXiv preprint arXiv:2307.14984, 2023. 336 Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. Large language models empowered agent-based modeling and simula- tion: A survey and perspectives. Humanities and Social Sciences Communications, 11(1):1–24, 2024. 336, 337 Shawn He, Surangika Ranathunga, Stephen Cranefield, and Bastin Tony Roy Savarimuthu. Norm violation detection in multi-agent systems using large language models: A pilot study. arXiv preprint arXiv:2403.16517, 2024. 343 Ilya Horiguchi, Takahide Yoshida, and Takashi Ikegami. Evolution of social norms in llm agents using natural language. arXiv preprint arXiv:2409.00993, 2024. 343 Tom Kouwenhoven, Max Peeperkorn, and Tessa Verhoef. Searching for structure: Investigating emergent communication with large language models, 2024. URL https://arxiv.org/abs/2412.07646. 342, 346 Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for” mind” exploration of large language model so- ciety. Advances in Neural Information Processing Systems, 36:51991–52008, 2023. 340 Shimin Li, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Agent alignment in evolv- ing social norms. arXiv preprint arXiv:2401.04620, 2024a. 342, 343, 346 Xinyi Li, Yu Xu, Yongfeng Zhang, and Edward C Malthouse. Large language model- driven multi-agent simulation for news diffusion under different network structures. arXiv preprint arXiv:2410.13909, 2024b. 341 Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhen- zhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents. arXiv preprint arXiv:2401.13178, 2024. 345, 346 BIBLIOGRAPHY349 C M Macal and M J North. Tutorial on agent-based modelling and simulation. Journal of Simulation, 4(3):151–162, 2010. ISSN 1747-7786. doi: 10.1057/jos.2010.3. URL https://doi.org/10.1057/jos.2010.3. 337 Marvin Minsky. Society of mind. Simon and Schuster, 1988. 343 Xinyi Mou, Xuanwen Ding, Qi He, Liang Wang, Jingcong Liang, Xinnong Zhang, Libo Sun, Jiayu Lin, Jie Zhou, Xuanjing Huang, et al. From individual to society: A survey on social simulation driven by large language model-based agents. arXiv preprint arXiv:2412.03563, 2024. 338 Diane Nutt and Diane Railton. The sims: Real life as genre. Information Communica- tion & Society, 6(4):577–592, 2003. 338 Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wol- czyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuci ́ nski, Lerrel Pinto, Rob Fer- gus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rockt ̈ aschel. BAL- ROG: Benchmarking agentic LLM and VLM reasoning on games. In The Thir- teenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=fp6t3F669F. 345 Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–18, 2022. 338 Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behav- ior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. 338, 339 Jinghua Piao, Zhihong Lu, Chen Gao, Fengli Xu, Fernando P. Santos, Yong Li, and James Evans. Emergence of human-like polarization among large language model agents, 2025a. URL https://arxiv.org/abs/2501.05171. 341, 346 Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiheng Zheng, Jing Yi Wang, Di Zhou, Chen Gao, Fengli Xu, Fang Zhang, Ke Rong, Jun Su, and Yong Li. Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors and society, 2025b. URL https://arxiv.org/abs/2502.08691. 340, 341 Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Xinyue Yang, Jiadai Sun, Yu Yang, Shuntian Yao, Tianjie Zhang, et al.Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337, 2024. 343, 346 Haoyi Qiu, Alexander R Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu. Evaluating cultural and social awareness of llm web agents. arXiv preprint arXiv:2410.23252, 2024. 343 350BIBLIOGRAPHY Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interac- tions. arXiv preprint arXiv:1909.12892, 2019. 346 Kanghyun Ryu, Qiayuan Liao, Zhongyu Li, Koushil Sreenath, and Negar Mehr. Cur- ricullm: Automatic task curricula design for learning complex robot skills using large language models. arXiv preprint arXiv:2409.18382, 2024. 346 Bastin Tony Roy Savarimuthu, Surangika Ranathunga, and Stephen Cranefield. Har- nessing the power of llms for normative reasoning in mass.arXiv preprint arXiv:2403.16524, 2024. 338 David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mas- tering the game of go without human knowledge. nature, 550(7676):354–359, 2017. 346 Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, 130(6):1526–1565, 2022. 346 Luc Steels. A self-organizing spatial vocabulary. Artificial life, 2(3):319–332, 1995. 337, 342 Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse rea- soning chains. arXiv preprint arXiv:2501.05707, 2025. 345 Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322, 2025. 337 Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023. 338 Ziyi Yang, Zaibin Zhang, Zirui Zheng, Yuxian Jiang, Ziyue Gan, Zhiyu Wang, Zijian Ling, Jinsong Chen, Martz Ma, Bowen Dong, et al. Oasis: Open agents social interaction simulations on one million agents. arXiv preprint arXiv:2411.11581, 2024. 337, 340, 346 Jintian Zhang, Xin Xu, Ningyu Zhang, Ruibo Liu, Bryan Hooi, and Shumin Deng. Ex- ploring collaboration mechanisms for llm agents: A social psychology view. arXiv preprint arXiv:2310.02124, 2023. 343, 344 Zirui Zhao, Hanze Dong, Amrita Saha, Caiming Xiong, and Doyen Sahoo. Au- tomatic curriculum expert iteration for reliable llm reasoning.arXiv preprint arXiv:2410.07627, 2024. 346 BIBLIOGRAPHY351 Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web en- vironment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. 345 352BIBLIOGRAPHY Chapter 18 Case Study 4: Simulating Societies with LLM Agents What You Will Learn In this case study you will familiarize yourselves with agent-based social simula- tions through the lens of the widely adopted Generative Agents framework [Park et al., 2023]. Specifically, you will be able to: • Run and configure a multi-agent social simulation powered by LLMs. • Work with a structured machine-psychological experiment. • Compare two cognitive architectures: a standard memory system and one with a negativity bias. • Investigate how personality descriptions and architectural choices shape emer- gent agent behavior. • Critically reflect on prompt fragility, model dependence, and the limits of LLM- driven simulations. 18.1 Social Simulation In social simulations, generative agents are LLM-driven characters that interact with an environment and with one another. The agents may plan activities, maintain memories, or reflect on their experiences. The Generative Agents Framework, introduced by Park et al. [2023], showed that LLM-powered characters together can produce believable daily routines, social interactions, and even coordinated group activities—all emerging from a relatively simple and elegant cognitive architecture. The twist in this case study is that you will work with two cognitive architectures for your agents, run the same scenario with both, and compare the emergent behavior. One architecture treats all memories equally. The other introduces a negativity bias: negative experiences are encoded more strongly and are more likely to be retrieved and reflected upon. We will observe that this asymmetry, inspired by findings from cogni- 353 354CHAPTER 18. CASE STUDY 4: SIMULATING SOCIETIES tive psychology [Baumeister et al., 2001], has measurable effects on our experimental agents. The goal of this chapter is to understand how architectural choices—in our case, small changes in the internal processing of an agent—lead to emergent consequences. You will also see how fragile agent personas can be: a single word in a personality description, or a subtle change in memory weighting, can fundamentally alter how an agent acts, plans, and reflects. Course Materials You can find the course exercise materials here: https://github.com/DorotaMoravcikova/agentic-llm-course-excercises 18.2 Background 18.2.1 Generative Agents The generative agents framework, introduced by Park et al. [2023], showed that LLM- powered characters can produce believable daily routines, social interactions, and even coordinate group activities. Each agent has three core components: 1. Memory stream: a timestamped log of everything the agent has perceived, done, and thought. 2. Retrieval function: when the agent needs to act, it retrieves relevant memories based on recency, importance, and relevance to the current situation. 3. Reflection mechanism: periodically, the agent synthesizes higher-level insights from its raw memories. The agent’s personality is defined by a short natural-language description—a scratch file that specifies the character’s name, age, learned and innate traits, and their lifestyle. This description is included in every LLM prompt, anchoring the agent’s behavior. 18.2.2 Bad is Stronger Than Good When an LLM-powered agent embedded in the original Generative Agents simula- tion receives criticism, it forms a memory. When praised an hour later, it forms an- other memory. Generative agents retrieve the two memories—one negative and one positive—with identical priority, as long as they are equally recent, equally relevant, and equally important. In general, cognitive architectures for agent-based social simulations draw on cog- nitive science to produce believable behavior. The recency, relevance, and importance variables are loosely inspired by insights about memory retrieval in humans. The im- portance variable captures something akin to the arousal or salience of an event. How- ever, in the original simulation [Park et al., 2022] the memory system is effectively 18.3. IMPLEMENTATION355 valence-blind: all memories—positive, negative, neutral—are encoded and retrieved with equal weight. If the agent were human, it would treat the negatively valenced event differently: encoding it with sharper sensory detail, retrieving it with greater vividness, and rendering it more likely to resurface in subsequent thoughts. Psycho- logical studies find that people remember insults longer than compliments. A single bad interaction with a colleague can outweigh weeks of pleasant ones. In other words, in our psychology bad is stronger than good [Baumeister et al., 2001], and this asym- metry is nowhere more consequential than in memory. In this case study, we test the effect of implementing the NEVER (Negative Emo- tional Valence Enhances Recapitulation) model, a more human-like mechanism for memory retrieval. In the NEVER-augmented agent, negative memories receive higher importance scores at encoding time. As a consequence, they are more likely to be retrieved and more likely to trigger reflections. The architecture remains identical in every other respect. This allows for a con- trolled machine-psychological experiment in which the effects can be compared of an architectural choice: valence-blind versus valence-sensitive memory. In addition, you will use the structured setup to test the effects of adapting the agent initialisations in terms of their personality descriptions and system prompts. 18.2.3 Why This Matters for Agent Design This case study illustrates two important lessons: 1. Architectural choices have emergent consequences: A memory weighting pa- rameter that seems innocuous at the component level can produce dramatic changes in the agent’s social behavior, emotional tone, and decision-making over time. These emergent effects are difficult to predict from the architecture alone. 2. Prompt fragility: Agent behavior is highly sensitive to the contents of personal- ity descriptions and system prompts. Small perturbations can produce large be- havioral shifts—a phenomenon sometimes called prompt sensitivity in the LLM literature. 18.3 Implementation 18.3.1 The Simulation Environment The simulation uses the Smallville environment from Park et al. [2023]: a small town rendered as a 2D tile map, where agents move between locations, interact with objects, and have conversations. Our simulation features a simplified scenario: a cafe with three agents. Please meet: • Dolores: A barista. Warm, empathetic, detail-oriented. The experimental agent— the one who will receive the negativity bias. 356CHAPTER 18. CASE STUDY 4: SIMULATING SOCIETIES • Maeve: A barista with a matched personality profile. The control agent—standard architecture, no negativity bias. • Bernard: The cafe manager. Unpredictable, moody, occasionally confronta- tional. Bernard serves as a naturalistic stressor: a source of social friction that both baristas are exposed to. This setup with three agents of which two have a similar role and position (barista working under supervision of Bernard), allows for a controlled experiment with agent architectures and initializations. In our first step, Dolores and Maeve are given the same personality traits and are exposed to the same environment. The only difference is in how Dolores processes her memories: using the NEVER implementation that encodes negative memories with higher importance. 18.3.2 Agent Architecture Each agent maintains: • A personality description (the scratch file): a short text describing the agent’s name, traits, occupation, relationships, and goals. • A memory stream: a list of timestamped observations, actions, and reflections. • A current plan: a short description of what the agent intends to do. 18.3.3 The Cognitive Loop At each simulation step (representing 10 seconds of in-game time), each agent executes the following cycle: 1. Perceive: Observe the current state of the environment—what other agents are doing, any ongoing conversations, changes in the surroundings. 2. Retrieve: Query the memory stream for relevant past experiences. 3. Plan: Given the personality description, retrieved memories, and current obser- vations, use the LLM to decide what to do next. 4. Act: Execute the planned action in the environment. 5. Reflect (periodic): Every few hours of in-game time, synthesize higher-level reflections from recent memories. Each of these steps involves an LLM call. The prompt is constructed from the agent’s personality description (which remains constant) and the relevant retrieved memories and current observations (which are dynamic). 18.3.4 Two Architecture Variants You will work with two variants of the retrieval function: 18.4. DISCUSSION357 Variant A: Baseline memory retrieval Retrieval is based on a weighted combi- nation of recency, relevance, and importance. This is the standard generative agent architecture as described by Park et al. [2023]. Variant B: Negativity-biased memory retrieval When a new memory is created, its emotional valence is assessed. The retrieval function is extended with a valence term. Absolute valence is normalized to [0, 1] and multiplied by 1.5 for negative memories, producing a retrieval hierarchy (negative > positive > neutral). The retrieval score becomes: score = 훼 rec · 푅+ 훼 rel · 퐿+ 훼 imp · 퐼+ 훼 val ·푉(18.1) with all 훼 = 1. Thus: the only difference between the two variants is the addition of the valence term; everything else is identical. 18.3.5 Setup and Installation Clone the repository and follow the instructions in the README: https://github.com/DorotaMoravcikova/generative_agents/tree/ course-exercises 18.4 Discussion Having run your experiments, reflect on the following questions: 1. Prompt versus Architecture: You have seen that both the personality prompt and the memory architecture influence agent behavior. Which lever is more power- ful? Which is more predictable? If a simulation were to be used for consequential applications such as modeling policy adoption, which would you prefer to use to shape agent behavior, and why? 2. Emergent Behavior: Did you observe any agent behaviors that surprised you— things that were not explicitly specified in the personality or architecture? What mechanisms produced these emergent behaviors in your view? 3. Evaluation Challenges: How do you evaluate whether a generative agent is be- having well? The valence score is one metric, but it is itself an LLM judgment. What are the limitations of using an LLM to evaluate the output of another LLM (or the same LLM)? Can you think of alternative evaluation approaches? 4. Cognitive Realism: The negativity bias is one example of a finding from cog- nitive psychology. Can you think of other cognitive biases or memory effects that could be implemented architecturally? What would you expect their emer- gent effects to be? (Consider, for example: confirmation bias, the availability heuristic, recency bias.) 358BIBLIOGRAPHY 18.5 Concluding Remarks In this chapter we have familiarized ourselves with a generative agent simulation and used it to explore two important phenomena in agentic AI: the emergent effects of cognitive architecture choices (the negativity bias) and the sensitivity of agent behavior to prompt content (prompt fragility). We have seen that a small change in how memories are weighted—negative mem- ories receiving higher importance scores—can produce measurable shifts in agent be- havior over time. We have also seen that agent personas are fragile: small changes to personality descriptions can produce large and sometimes unpredictable behavioral effects. These findings have practical implications for anyone building LLM-driven agent systems. The persona is not a reliable anchor for behavior. The architecture—how memories are stored, retrieved, and reflected upon—shapes behavior in ways that may not be apparent from reading the code. The field of generative agent simulation is young and evolving. The exercises in this chapter have given you hands-on experience with the core mechanisms and the core challenges. As the field matures, the problems of prompt robustness, cognitive architecture design, and emergent behavior evaluation will remain central. Bibliography Roy F Baumeister, Ellen Bratslavsky, Catrin Finkenauer, and Kathleen D Vohs. Bad is stronger than good. Review of General Psychology, 5(4):323–370, 2001. 354, 355 Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–18, 2022. 354 Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behav- ior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. 353, 354, 355, 357 Part VI Advanced Topics 359 361 Agentic LLMs are new, and there is much research activity, on many topics. In this part we have selected four research areas where important developments are under way. These are: synthetic data generation, advanced architectures, mechanistic interpretabil- ity, and perspectives on the cognitive architectures of agents. There are certainly many more interesting research directions in agentic LLMs; we have selected these topics because they connect to important points that were raised earlier in the book. Note that the purpose of the previous chapters was to explain matters as clearly as possible—suitable for a master’s course. The purpose of the next chapters is to cover more advanced material—suitable for a master’s thesis. Of course we still try to be clear, but the speed will be higher, and the depth will be deeper, with more references to research papers. We encourage you to look up these sources. If you are looking for inspiration for your own further research, we hope that these chapters will be of help. 362 Chapter 19 *Synthetic Data Generation for Training Agentic LLMs What You Will Learn In this chapter you will learn the role that synthetic data can play in the develop- ment of agentic large language models. Specifically, you will be able to: • Characterize the importance of data in the LLM training pipeline. • Explain the general loop of distilling knowledge from stronger models for weaker students. • Describe the pipeline components for which synthetic data is being used as an alternative to human-generated content. • Outline the various strategies for curating and evaluating synthetic data. • Contrast the advantages and disadvantages of using synthetic data by describing the risks of using artificially generated data. 19.1 The Importance of Data The rapid scaling of LLMs has led to a significant demand for training and evalua- tion data. Some have coined training data the digital oil, reflecting the importance as primary ingredient for AI development. Furthermore, data is increasingly becoming a differentiating factor since other components of the pipeline, such as model architec- tures, are homogenizing [Kim et al., 2025a]. As models continue to scale, they require trillions of words of human-generated text scraped from the internet; however, there is only a limited amount of human-generated data available, which raises the question of whether training data could become the main bottleneck to scaling, see Figure 19.1 [Villalobos et al., 2024]. Moreover, as data requirements grow in scale, the acquisi- tion of data becomes increasingly prohibitive in terms of cost and time, since most of the straightforward sources are becoming exhausted. Furthermore, issues with privacy, copyright, and bias cause the field to be looking beyond the curation of web-based 363 364CHAPTER 19. *SYNTHETIC DATA GENERATION Figure 19.1: Projections of the available human-generated public text and dataset sizes to train LLMs [Villalobos et al., 2024]. The intersection of the stock and dataset size projection lines indicates the median year (2028) in which the stock is expected to be fully utilized. human-generated data, and instead turn towards synthetic data generation [Liang et al., 2022]. LLMs increasingly generate the data that trains the next generation of LLMs them- selves: models are prompted to produce instructions, dialogues, reasoning traces, and even entire training corpora that are then used to finetune, distill and evaluate other models [Long et al., 2024]. This self-reinforcing loop lowers the cost and effort of data collection compared to sourcing and annotating human-generated text, while also offer- ing fine-grained control over style, difficulty, and coverage of the resulting datasets. By now, this approach has drawn significant industrial [Kapania et al., 2025] and academic interest [Viswanathan et al., 2025, Alismail and Lanquillon, 2025], spanning applica- tions from general-purpose instruction tuning to the more specialized data needed to train agentic behaviors such as tool use, planning, and multi-step reasoning (illustrated in Figure 1.1 on page 17). The growing reliance on model-generated data has also introduced a new term in the scaling laws (Chapter 4), one that accounts for the proportion of synthetic data present in a pre-training corpus. Recent work such as SYNTHLLM [Qin et al., 2025] il- lustrates why this matters: by automatically extracting high-level concepts from source documents and recombining them through a graph-based algorithm, it generates syn- thetic corpora diverse enough to follow a predictable, rectified scaling law across model sizes. In this version of the scaling laws, the gains taper off earlier (before other token budgets typically used in pretraining), and larger models reach their optimal perfor- mance on comparatively fewer tokens than smaller ones. This suggests that synthetic data can be tuned not just for volume, but also for the shape of the scaling curve it produces. Beyond scaling behavior, generating data for agentic capabilities—tool use, plan- ning, and multi-step reasoning—increasingly depends on coordinated multi-agent work- 19.2. DISTILLATION LOOP365 Seed / real data Generator teacher / multi-agent Raw synthetic data (Intrinsic) evaluation Curation filter, score, dedup Train / distill Model (student) (Extrinsic) evaluation improved model becomes next-generation teacher Figure 19.2: The synthetic-data loop that organizes this chapter. A generator produces raw synthetic data from a small seed set, followed by curation filtering and scoring. The data trains or distills a model, which is then evaluated and can be used as the generator of the next iteration (dashed). flows rather than on a single generator model. Specialized agents can take on distinct roles (e.g., proposing tasks, executing them, critiquing the results), which tends to pro- duce data that is of higher quality, more diverse, and structurally closer to what agentic training actually requires. Matrix [Wang et al., 2025a] is one example of this direction: a decentralized framework in which control and data flow are both represented as mes- sages passed between agents through distributed queues, allowing the pipeline to scale horizontally without a central coordinator. Let us have a closer loop on the synthetic training pipeline. 19.2 Distillation Loop Using synthetic data in the agentic LLM training pipeline follows the overall distilla- tion loop, as shown in Figure 19.2. A generator (a strong model, or an orchestration of several agents) uses a small amount of seed or real data to generate a large pool of raw synthetic data. That pool is then curated (Section 19.4) before being used to train or distill a model. Crucially, the resulting model can itself become the generator for the next iteration. This closing of the self-reinforcing loop brings considerable risk to training stability and downstream performance (see also the discussion of stability in metalearning, Chapter 8). Distillation is the special case of synthetic-data generation in which the generator is a stronger model whose capability we are trying to transfer. In knowledge distillation, a compact student model learns to copy a more capable teacher [Hinton et al., 2015, Hu et al., 2022]. It can do so by directly optimizing against the teacher’s output predictions, or it can do so by simply training on synthetic data emitted by a teacher. The goal of distillation is to get a more efficient version of the teacher, both at inference time and in terms of data cost, without sacrificing too much task performance. Distillation has been applied widely, including in agentic settings [e.g., Li et al., 2025a, Du et al., 2025]. Distillation can also be used to derive preference data for instruction tuning. Zephyr [Tunstall et al., 2024], for example, was trained with AI feedback, where outputs are generated and ranked by strong teacher models, aligning a small model with user intent at minimal data and training cost. 366CHAPTER 19. *SYNTHETIC DATA GENERATION Four axes of distillation Concrete distillation methods can be organized along four axes. (1) The signal: whether the student imitates teacher logits, full output sequences, or a scalar reward derived from the teacher. (2) The data source: whether the sequences being scored are generated by the teacher or sampled from the student. (3) The timing: whether that data is produced once up front (off-policy) or continuously as the student changes (on-policy). (4) The teacher identity: whether the teacher is a stronger separate model or the student itself. We will now discuss these four axes of distillation. 19.2.1 Signal: The Distillation Objective When we have white-box access to the teacher’s (푞) probabilities, the student 푝 휃 is trained to make the student’s next-token distribution match the teacher’s. In distillation, we often soften the distributions with a shared temperature 휏 (so 푞 휏 (·) ∝ exp(푧/휏) for logits 푧). A generic distillation loss over a corpus of sequences is L(휃) = E 푥 " ∑︁ 푡 퐷 푞 휏 (· | 푥 <푡 )∥ 푝 휏 휃 (· | 푥 <푡 ) # ,(19.1) where the sum runs over token positions 푡, 퐷 is a divergence (most commonly the KL divergence), and the temperature 휏 > 1 flattens both distributions so that the stu- dent also learns from the teacher’s relative confidence. When only the teacher’s text is available (black-box distillation, the usual case when distilling from a proprietary API), we no longer observe 푞 but only some tokens that we sample from it. The soft target 푞 휏 (· | 푥 <푡 ) then collapses to a one-hot vector on the sampled token, the diver- gence 퐷 becomes the token-level cross-entropy, and Equation 19.1 reduces to ordinary supervised fine-tuning on the teacher’s generations. In Section 19.3 we discuss the Self-Instruct technique [Wang et al., 2023b]. This methods perform distillation from the data side, and is therefore a black-box approach. 19.2.2 Data Source: Teacher-Student Paradigm In distillation, the intuitive assumption is that a bigger, stronger teacher is always better. However, there are exceptions [Xu et al., 2025a, Li et al., 2025b]. A capable teacher may produce traces that are too long, too complex, or stylistically mismatched for a small student to absorb. This phenomenon is known as the small model learnability gap. Figure 19.3 illustrates a situation where small students learn more from the shorter reasoning of a weaker teacher than from the elaborate chains of thought of a frontier model. The practical implication is that the teacher should be chosen relative to the current capabilities of the student rather than maximized in isolation [Zhang et al., 2026a]. 19.2.3 Timing: Off-policy versus On-policy Distillation The third axis concerns timing. The most common recipe in distillation is off-policy distillation. Here we sample a fixed dataset of completions from the teacher and then 19.2. DISTILLATION LOOP367 Figure 19.3: Visualization of the Small Model Learnability Gap, where small student models do not consistently benefit from long CoT reasoning and instead perform better when fine-tuned on shorter CoT reasoning from smaller teachers [Li et al., 2025b] fine-tune the student on them (as in the black-box case above). This is a simple and parallelizable approach, but it suffers from a train/inference mismatch. The student is only once shown teacher-generated token sequences at training time, yet at deployment time it must continue its own, possibly flawed, sequences, which may be different than the ones it has been trained on. This exposure bias compounds over long agentic trajectories. On-policy distillation addresses this by sampling sequences from the student itself and having the teacher score them, all within the loop. Generalized Knowledge Dis- tillation (GKD) trains the student on its self-generated output sequences using teacher feedback on those sequences [Agarwal et al., 2024]. The student is corrected exactly on the distribution of the states that it actually visits. This on-policy view makes the connection to reinforcement learning through im- itation learning explicit. The per-token teacher divergence in Equation 19.1 acts as a dense, token-level reward. Distillation is turned into a form of reinforcement learning where the teacher, rather than a sparse outcome checker, supplies the reward signal [Lu and Lab, 2025]. Hybrid schemes push this idea further by selectively imitating the teacher only where it helps during a reinforcement learning phase [Zhang et al., 2026b]. 19.2.4 Teacher Identity: Self-Distillation and Self-Improvement The teacher and student model do not have to be different. When the teacher and stu- dent model are the same, distillation becomes self-improvement: the model generates candidate solutions, keeps the ones that pass a check, and trains on them. STaR [Ze- likman et al., 2022a] bootstraps reasoning in this way, iteratively generating rationales, retaining those that reach the correct answer, and finetuning on them. Another approach is Self-Play Finetuning (SPIN) [Chen et al., 2024]. In this method the model distinguishes its own generations from reference data across iter- ations, improving alignment without any external teacher. Perhaps surprisingly, even repeated self-distillation, with no new data source, can yield measurable gains, up to a 368CHAPTER 19. *SYNTHETIC DATA GENERATION point [Pareek et al., 2024]. Self-improvement is most powerful when candidate outputs can be verified cheaply. In such cases, an external signal can still be incorporated effectively. This is the case for problems defined in code and math, as solutions are easy to execute and check. CURE, for example, is a reinforcement learning framework whose reward co-evolves a model’s coding and unit-test-generation abilities from their interaction outcomes. There is no ground-truth code as supervision. Since code is easy to execute and check, the coder and the test-writer act as mutually adapting agents, enabling long-horizon optimization on hard problems. The same principle underlies self-challenging and self-evolving agents, which propose their own increasingly difficult tasks and update their generation policy over time [Zhou et al., 2025, Huang et al., 2025, Wang et al., 2025b]. 19.3 Simulating Human-generated Content The move away from naturally occurring human-generated text, either directly ob- tained or scraped from the web, is perhaps best seen in the evolution from curated human-generated instruction data towards datasets where synthetic generation plays a more central role. We will now discuss these methods, where datasets are, in their core, still based on human-generated data. As an example of a resource where humans still underpin most of the data, Natural Instructions [Mishra et al., 2022], contains 61 distinct tasks, their human-authored in- structions, and 193k task instances (input-output pairs). The instructions were gathered from instruction documents for existing NLP datasets, all mapped to a unified schema. Experiments with LLMs indicate that models benefit from observing the instructions alongside examples, although these models have remained far behind an estimated per- formance upper bound. Subsequent work focuses on increasing the number of unique tasks and task types [Wang et al., 2022], varying the length of the instructions [Bach et al., 2022], or in- cluding particularly difficult tasks [Srivastava et al., 2023]. Importantly, all these meta- datasets (datasets of datasets) are still ultimately built from human benchmarks: every sample they contain is authored by a person. Therefore, there is a practical limit to how much of this data can be freely found online. Because these approaches rely on large amounts of human supervision in the form of crowd-sourced datasets, recent work focuses on generating synthetic data that fills the same role. Here, we see the creation of datasets such as Unnatural Instructions [Honovich et al., 2023], which contain creative and diverse instructions collected with little human labor, instead relying on prompting various LLMs. Experiments indicate that training models on the somewhat noisy Unnatural Instructions dataset yields re- sults comparable to those trained on high-quality, human-curated data, as shown in Figure 19.4. This result suggests that machine-generated data offers a cost-efficient alternative to human labor for expanding and diversifying training datasets. The simulation of human-like text, code, and reasoning by generative models has unlocked new methodologies for both training and evaluating machine learning sys- tems. By prompting strong models, researchers can generate highly diverse and com- plex datasets that mimic human content generation. For example, PersonaHub [Ge 19.3. SIMULATING HUMAN-GENERATED CONTENT369 Figure 19.4: Scaling experiments comparing Unnatural Instructions with Super- Natural Instructions [Honovich et al., 2023], with model performance controlling for dataset size, tested on two benchmarks: Super-Natural Instructions (left) and LMentry (right). et al., 2025] steers a generation process with a large-scale collection of personas, using them to prompt LLMs towards a far greater diversity of synthetic data than uncondi- tioned generation would produce. Similar to self-improvement, training on synthetic data has shown much promise in domains where model outputs are relatively easy to verify, such as mathematics [Yang et al., 2023, Liu et al., 2023], programming [Haluptzok et al., 2023], and games [Wang et al., 2023a]. 19.3.1 Generating Training Data We will now focus on generating training data. The use of synthetic data for training has transitioned from a niche data augmentation technique to a core component of mod- ern model alignment and reasoning pipelines. Techniques such as Self-Instruct [Wang et al., 2023b] have demonstrated that smaller, open-weight models can be instruction- tuned using data generated entirely by a stronger, proprietary model. The process, as shown in Figure 19.5, involves seeding an LLM with a small set of human-written in- structions and prompting it to iteratively generate novel task descriptions, inputs, and outputs. The simulated human content allows models to acquire complex zero-shot capabilities without the need for large groups of human annotators. Leveraging existing data Other approaches combine curated human data with syn- thetic data, instead of replacing one with the other. Gunasekar et al. [2023] show that training on a mixture of human-written textbooks and synthetically generated text yields highly capable language models, suggesting that curated human data and machine-generated data are complements rather than substitutes. One challenge is that synthetically generated data tends to lack diversity and inherits errors and hallucina- tions, even when it is seeded with human-generated data. Subsequent improvements use a generation process that is based on ground truth documents [Lupidi et al., 2025, Yue et al., 2024]. 370CHAPTER 19. *SYNTHETIC DATA GENERATION Figure 19.5: Self-Instruct pipeline [Wang et al., 2023b]. A complementary way to amplify a limited seed set is backtranslation. This ap- proach was originally introduced in machine translation to synthesize parallel data by translating sentences from a source language to a target language, and then back into a source language [Edunov et al., 2018]. The same principle can be repurposed for reasoning tasks: instead of only generating specific solutions for a single problem, we generate new problems that are consistent with an existing solution. MathGenie [Lu et al., 2024] applies this idea to mathematical reasoning by taking a seed set of question–solution pairs. It augments the solutions, and then back-translates them into new questions. Because each newly generated question is grounded in a verified so- lution, this approach expands both the size and the diversity of the training data while reducing errors and hallucinations. Human-in-the-loop approaches Fully automatic generation pipelines are not al- ways sufficient. Models can struggle to generate specific scenarios or domain-relevant cases. The reason for these failures is usually that the generating LLM has specific blind spots, which can also make it difficult to generate genuinely informative data [Lippmann et al., 2024]. Such problems motivate human-in-the-loop approaches, in which people steer, fil- ter, or revise machine-generated content rather than authoring it from scratch. WANLI [Liu et al., 2022] pairs a language model that proposes challenging natu- ral language inference examples with human annotators who filter and relabel them, combining the scalability of automatic generation with human quality control. EvalAssist [Santill ́ an Cooper et al., 2025] applies a similar principle to evaluation tooling, using synthetic data to support and accelerate human judgment. A related strategy is to prompt the language model to produce an initial seed text, which human raters then edit before the result is used to bootstrap a larger fully-synthetic dataset. In this way, the human effort is spent on correcting the model rather than on authoring every example. 19.3. SIMULATING HUMAN-GENERATED CONTENT371 19.3.2 Generating Evaluation Data Beyond training, synthetic generation plays a critical role in evaluation. Traditional static benchmarks suffer from rapid data contamination, where test sets inadvertently leak into the pretraining corpora of modern LLMs. To mitigate this, dynamic evaluation frameworks [Kiela et al., 2021] increasingly rely on synthetic data generation to create bespoke, single-use test cases. Furthermore, the LLM-as-a-judge paradigm [Zheng et al., 2024] simulates human evaluation by prompting a strong model to grade the outputs of other models, highly correlating the models with human preference and significantly reducing the latency of evaluation cycles. Further creation of dedicated hard benchmarks for LLMs can once again be done with human-in-the-loop approaches, such as the MATH 2 benchmark. For this bench- mark, multiple LLMs are used to iteratively generate and adjust mathematical questions and solutions through multi-turn prompting. Human annotators then further expand on the questions and check the answers, again aided by LLMs in the process [Shah et al., 2024]. Other approaches to efficient generation of synthetic evaluation data shift the task for annotators from coming up with new samples, to editing samples to be harder for the LLM [Yuan et al., 2021]. Automated red teaming One specific use case for machine-generated evaluation is through the safety-oriented evaluation by means of automated red teaming. In this approach, one LLM generates adversarial prompts to probe a target model for vulner- abilities before deployment [Ganguli et al., 2022]. By synthesizing many potentially malicious prompts, researchers can stress-test and harden models against toxic outputs, jailbreaks, and misalignments at a scale that human red teams cannot match [Perez et al., 2022]. Because the same generated prompts can subsequently be used to fine- tune the target model against those failures, red teaming sits at the boundary between synthetic evaluation and synthetic training data. 19.3.3 Generating Agentic Data In agentic settings, data must be synthesized for an entire trajectory (and not just in single instruction-response pairs). As the agent is interacting with its environment, a full sequence of tool calls, observations, and decisions is produced [Kang et al., 2025]. This goes beyond data that trains or evaluates a regular LLM operating on a single input-output pair. Generating agentic data is a more complex problem than generating data to instruct the underlying LLM itself, even though the same generative models are typically used to produce it. Let us look at an example. Example 2. Consider synthesizing a single training trajectory for a tool-using agent, roughly following the two-phase recipe of APIGen-MT [Prabhakar et al., 2025]. 372CHAPTER 19. *SYNTHETIC DATA GENERATION First a task proposer invents a plausible user goal and a set of available tools, together with a ground-truth answer: User: “What will the weather be in Leiden this weekend, and should I bring an umbrella?”Tools: get forecast(city, days). An actor model then rolls out the interaction, emitting the intermediate tool call and consuming its (simulated) observation before answering: get forecast("Leiden", 2)→sat: rain, sun: clear→ “Rain on Saturday, clear on Sunday—bring an umbrella for Saturday.” Finally a verifier checks that the emitted call is well-formed, that its argu- ments match the request, and that the final answer is consistent with the obser- vation. Only trajectories that pass verification are kept. Note what had to be synthesized: not just a single input–output pair, but the goal, the tool schema, the call, the observation, and a checker that verifies the whole trace. Methods for agentic generation usually involve entire orchestrations of agents for multi-turn conversations including human inputs and tool use [Wu et al., 2024]. APIGen- MT [Prabhakar et al., 2025] exemplifies this shift with a two-phase framework that generates verifiable and diverse multi-turn agent data. TOUCAN [Xu et al., 2025b] pursues a similar goal at a larger scale, simulating realistic tool calls to construct a large tool-agentic dataset. Beyond generating trajectories for a fixed skill, some work synthesizes data explicitly to transfer a new skill or behavior from a strong model to a weaker one, a process referred to as Generative Teaching [Crispino et al., 2024]. Fur- thermore, agentic synthetic data is not limited to text, analogous generation pipelines extend to images, tabular data, and other forms of interaction or behavioral data. For the generation of agentic synthetic data, controlling the task complexity is a re- curring challenge. Deep-research agents, for example, aim to solve complex question- answering tasks through long-horizon interactions with online tools, but the data typi- cally used to train them lacks fine-grained control over difficulty and quality, yielding synthetic data that falls short of capturing sufficient complexity. Pandit et al. [2025] address this issue with a pipeline that synthesizes question-answer pairs that progres- sively increase task complexity until a frontier baseline web agent fails. This curricu- lum learning approach calibrates task-difficulty directly against agent capability rather than a fixed heuristic. AReaL-SEA [Gao et al., 2026] addresses reliability from a complementary angle. Here, a multi-agent engine synthesizes tool-grounded dialogues in combination with executable per-instance checkers. Next, a closed-loop self-evolving process updates the prompts over time to improve reliability. 19.4 Data Curation Once synthetic data has been generated, we need to check it for quality and curate it. Usually, this means filtering, scoring, and selecting samples so that only the examples 19.4. DATA CURATION373 that are useful are retained. Evaluating synthetic data usefulness can be approached in two complementary ways: extrinsically, by measuring its effect on a downstream task, or intrinsically, by examining properties of the generated data (or the generative process) directly. In practice, curation is implemented as a multi-stage filtering pipeline in which cri- teria that we will discuss below are turned into concrete operators. Typical stages in- clude semantic deduplication (removing near-duplicate examples via embedding simi- larity to preserve diversity), perplexity filtering (discarding incoherent or off-distribution text using a reference model), rule-based or format checks, and reward-model or judge scoring to retain only high-fidelity samples. The seminal Textbooks Are All You Need paper [Gunasekar et al., 2023] is a reminder that this curation step often matters as much as the volume of data: aggressive filtering for quality can outweigh sheer quan- tity. Without it, models readily learn the generative artifacts and statistical anomalies of their teachers. 19.4.1 Extrinsic evaluation Extrinsic evaluation answers a simple question: does the synthetic data actually help on a downstream task? This topic is also referred to as utility. In practice, utility is measured by training or finetuning a model on the synthetic data (alone or mixed with real data) and evaluating it on a held-out, real-world test set. Extrinsic evaluation is the most direct measure of whether synthetic data is useful. For example, if the per- formance (e.g., in terms of perplexity) of a pretrained model increases when synthetic data is included, we can conclude that the data is useful. However, it is also the most expensive: it requires a full training run before any signal is available. If we make changes to the generation process or select a different set, we need to retrain and eval- uate our model. This makes the extrinsic evaluation a poor fit for filtering or selecting individual examples during generation. 19.4.2 Intrinsic evaluation Intrinsic evaluation instead looks at characteristics of the generations themselves, with- out the cost of a downstream training run. A number of quality aspects have been pro- posed for evaluating synthetic text [Ramesh et al., 2025]. Rather than evaluating at the output level, it is also possible to benchmark the language models of the generative process directly [Kim et al., 2025b]. This can give an earlier and cheaper signal than inspecting the generated data. Data correctness A generated example is only useful if its content is factually and logically sound. Since synthetic data often comes from other LLMs, it inherits the gen- erating model’s failure modes. This includes hallucinated facts, inconsistent reasoning, or labels that do not match the content. These errors can subsequently propagate into models that are trained on them. Correctness is typically checked by comparing genera- tions against human-generated labels [Long et al., 2024, Chim et al., 2025], performing verification (e.g., a rule-based checker or a separate judge model) [Chen et al., 2025a, 374CHAPTER 19. *SYNTHETIC DATA GENERATION Yang et al., 2025, Setlur et al., 2024], or through consistency checks across multiple generations for the same input [Zelikman et al., 2022b, Seegmiller et al., 2025]. Data diversity A key problem in creating synthetic data is maintaining diversity. Logit suppression and temperature sampling can improve data diversity but come at a cost of data correctness [Chung et al., 2023]. Diversity is also difficult to measure: lexical heuristics (e.g., ROUGE) or neural heuristics (embedding-based similarity) are commonly used as proxies, but each captures only a fraction of the possible variation in the data. MetaSynth addresses this problem by generating synthetic data that enhances di- versity through meta-prompting, where a language model orchestrates multiple expert LLM agents to collaboratively generate data [Riaz et al., 2025]. Data contamination A serious risk for synthetic data is contamination [Deng et al., 2024, Balloccu et al., 2024]. The generating LLM is pretrained on large, imperfectly filtered web datasets, and can memorize benchmarks and reproduce or closely para- phrase them when prompted to generate new training examples. After many steps, synthetically generated data converges to that of existing benchmarks. This inflates evaluation scores artificially, for example in the case of code generation benchmarks [Matton et al., 2024]. Solutions for this type of leakage should go beyond simple exact-match filters, and requires n-gram overlap search against known benchmarks, membership-inference-style probing [Puerto et al., 2025, Huang et al., 2026], or dy- namic dataset construction pipelines [Chen et al., 2025b]. Fairness and biases LLMs inherit and amplify social biases from their own training data. If we, in turn, use the LLMs to generate synthetic corpora, we risk further per- petuating stereotypical associations, skewed demographic representation, or unequal generation quality across social groups [Gallegos et al., 2024]. This is of particular concern for synthetic data because it is produced at scale and often consumed directly as training material for new LLMs. Finally, intrinsic metrics are only proxies for usefulness. A dataset can score well on diversity or pass correctness checks while failing to improve, or even harm, extrin- sic evaluation. Lexical and neural diversity heuristics measure surface-level variation and can fail to spot redundancy in ways that matter (e.g., repeated reasoning patterns) or overestimate diversity. Correctness checks further depend on the availability of a trusted reference, rule-based verifier, or judge model, which for open-ended or sub- jective generations may not exist, or may be unreliable. For these reasons, intrinsic evaluation is best treated as an additional check rather than a true replacement for ex- trinsic evaluation. 19.5 Risks As should be clear by now, using synthetic data for training agentic LLMs is a solu- tion that is not without problems. We will discuss several more risks associated with 19.5. RISKS375 synthetic data, including model collapse, bias amplification, security and privacy vul- nerabilities, and legal and ethical concerns. 19.5.1 Model Collapse The defining risk of the loop in Figure 19.2 is that it feeds on its own output. Model collapse is the well-documented phenomenon where training successive model gener- ations on recursively generated AI data causes the diversity of the original distribution to disappear [Shumailov et al., 2024]. Rare events and low-probability linguistic nu- ance are sampled less often, so each generation forgets a little more of the distribution’s long-tailed behavior until the model converges to a repetitive, high-probability model. Some experiments show that repeated training on purely synthetic data yields diminish- ing and eventually negative returns [Singh et al., 2024], and concentrating a corpus on a few synthetic sources, rather than many, increases the fragility of the resulting model [Schaffelder and Gatt, 2025]. Similarly, small students distilled from a teacher are prone to hallucination and often fail on long-tailed knowledge [Kandpal et al., 2023]. More subtly, imitating a teacher can produce a student that matches the teacher’s style while lagging on factuality, a gap which is usually only revealed by targeted capability benchmarks [Gudibande et al., 2023]. The standard mitigation is to keep anchoring the corpus with real human data rather than training on synthetic data alone. 19.5.2 Bias Amplification Because generative models over-represent the majority modes of their training distri- bution, synthesizing data tends to cut off the diverse, long-tail perspectives found in natural human discourse [van der Meer et al., 2024]. A student trained on such data can therefore exhibit an amplified, caricatured version of the teacher’s social biases. The student model reproduces stereotypical associations, skewed demographic repre- sentation, or unequal generation quality across groups [Gallegos et al., 2024]. This concern is larger for synthetic data than for original model outputs because the data is produced at scale and consumed directly as training material, and each round of the loop compounds the previous round. 19.5.3 Security Risks There are also security risks associated with synthetic training data. Relying on gener- ated data introduces new attack vectors for maliciously polluting training data. If the teacher model or any stage of the generation pipeline is compromised, an adversary can mount a data-poisoning attack. Because synthetic corpora are typically too large to audit exhaustively, subtle back doors or malicious instruction–response pairs can be inserted, which remain dormant until triggered in the deployed student. Poisoning even a tiny fraction of a web-scale corpus has been shown to be practical and cheap [Carlini et al., 2024], and the risk is heightened when the “teacher” is an uncontrolled third-party model whose training data and alignment are opaque. 376CHAPTER 19. *SYNTHETIC DATA GENERATION 19.5.4 Privacy Preservation Synthetic generation is frequently mentioned as a solution for data privacy. The idea is that synthetic data does not contain personally identifiable information (PII) [Dwork and Roth, 2014], yet is able to reflect the statistical properties of private data (such as medical records). However, in this type of data there is a substantial risk of memorization. Without strict privacy guarantees during the generation phase, generative models can uninten- tionally reconstruct and therefore leak exact snippets of the sensitive human data on which they were originally trained [Carlini et al., 2021]. Privacy evaluations show that off-the-shelf privacy-preserving methods can fail [Ramesh et al., 2024]. Common privacy approaches rely on differential privacy [Dwork et al., 2006], a guarantee on the generative process that bounds how much the output distribution can change when a single individual’s record is added to or removed from the dataset. This bound is controlled by a privacy budget, where smaller values enforce stronger protection at the cost of greater utility loss [Yue et al., 2023]. 19.5.5 Copyright and Terms of Use Part of the appeal of synthetic data is that it sidesteps the unclear legal status of scraping internet-scale human text, which has already produced copyright litigation. The 2025 Bartz v. Anthropic settlement in the US is illustrative. A district court found LLM training itself exceedingly transformative and thus a candidate for fair use, however, liability still arose from how the underlying corpus was obtained, namely the ingestion of hundreds of thousands of pirated books, and Anthropic settled at roughly $3,000 per work (about $1.5 billion in total), with parallels in other legal cases such as Kadrey v. Meta. This shows that to an increasing degree the data underlying training is scrutinized for copyright and terms of use. Synthetic data does not fully escape this issue. Distilling from a proprietary model raises a distinct issue: the terms of service of most commercial LLM APIs explicitly forbid using their outputs to train competing models. Data generated by a black-box teacher may therefore be legally problematic regardless of copyright. Such a tension surfaced publicly in 2025 amid accusations that competing systems had been trained on the outputs of frontier APIs. For practitioners, this means that the licence of the generator is as important to check as the licence of any human data it replaces. 19.6 Concluding Remarks Synthetic data has moved from a niche augmentation method to a central ingredient of the LLM pipeline (Chapter 3), driven by the prospect that human text becomes a scaling bottleneck (Chapter 4). Strong models and multi-agent orchestrations can simulate instruction data, evaluation benchmarks, and, more recently, whole agentic trajectories of tool calls and decisions (Chapters 9 and 11). Because raw generations are noisy, curation, extrinsic utility together with intrinsic checks on correctness, diversity, and contamination, are as important as generation BIBLIOGRAPHY377 itself. The various setups can be considered forms of knowledge distillation organized along the axes of signal, data source, timing (off- vs. on-policy), and teacher identity. Reusing synthetic data from the same model for self-improvement has promises for enabling verifiable reasoning (Chapter 6). However, in all these pipelines we need to be careful about what data these models are consuming: without real-data anchoring and careful auditing, model collapse, bias amplification, security, privacy, and licensing risks all compound with every iteration. Bibliography Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of lan- guage models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pages 21246–21263, 2024. 367 Ahmad Alismail and Carsten Lanquillon. A survey of llm-based methods for synthetic data generation and the rise of agentic workflows. In International Conference on Human-Computer Interaction, pages 119–135. Springer, 2025. 364 Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Ni- hal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-Jian Jiang, and Alexander M. Rush. PromptSource: An integrated development environment and repository for natural language prompts. In Valerio Basile, Zornitsa Kozareva, and Sanja Stajner, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 93– 104, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-demo.9. URL https://aclanthology.org/2022. acl-demo.9/. 368 Simone Balloccu, Patr ́ ıcia Schmidtov ́ a, Mateusz Lango, and Ondrej Dusek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 67–93, St. Julian’s, Malta, March 2024. As- sociation for Computational Linguistics. doi: 10.18653/v1/2024.eacl-long.5. URL https://aclanthology.org/2024.eacl-long.5/. 374 Nicholas Carlini, Florian Tram ` er, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, ́ Ulfar Er- lingsson, Alina Oprea, and Colin Raffel.Extracting training data from large language models.In 30th USENIX Security Symposium (USENIX Se- curity 21), pages 2633–2650. USENIX Association, August 2021.ISBN 978-1-939133-24-3.URL https://w.usenix.org/conference/ usenixsecurity21/presentation/carlini-extracting. 376 378BIBLIOGRAPHY Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tram ` er. Poisoning web-scale training datasets is practical. In 2024 IEEE Symposium on Se- curity and Privacy (SP), pages 407–425, 2024. doi: 10.1109/SP54263.2024.00179. 375 Meng Chen, Philip Arthur, Qianyu Feng, Cong Duy Vu Hoang, Yu-Heng Hong, Mahdi Kazemi Moghaddam, Omid Nezami, Duc Thien Nguyen, Gioacchino Tan- gari, Duy Vu, Thanh Vu, Mark Johnson, Krishnaram Kenthapadi, Don Dhar- masiri, Long Duong, and Yuan-Fang Li. Mastering the craft of data synthesis for CodeLLMs. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), pages 12484–12500, Albuquerque, New Mexico, April 2025a. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025. naacl-long.620. URL https://aclanthology.org/2025.naacl-long. 620/. 373 Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. Benchmarking large language models under data contamination: A survey from static to dynamic eval- uation. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10080–10098, Suzhou, China, Novem- ber 2025b. Association for Computational Linguistics. ISBN 979-8-89176-332- 6. doi: 10.18653/v1/2025.emnlp-main.511. URL https://aclanthology. org/2025.emnlp-main.511/. 374 Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self- play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024. 367 Jenny Chim, Julia Ive, and Maria Liakata. Evaluating synthetic data generation from user generated text. Computational Linguistics, 51(1):191–233, March 2025. doi: 10.1162/coli a00540.URL https://aclanthology.org/2025.cl-1. 6/. 373 John Chung, Ece Kamar, and Saleema Amershi. Increasing diversity while main- taining accuracy: Text data generation with large language models and human in- terventions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 575–593, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.34. URL https://aclanthology.org/2023.acl-long.34/. 374 Nicholas Crispino, Kyle Montgomery, Fankun Zeng, Dawn Song, and Chenguang Wang.Agent instructs large language models to be general zero-shot reason- ers. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria BIBLIOGRAPHY379 Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 9458–9549. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/crispino24a.html. 372 Chunyuan Deng, Yilun Zhao, Yuzhao Heng, Yitong Li, Jiannan Cao, Xiangru Tang, and Arman Cohan. Unveiling the spectrum of data contamination in language model: A survey from detection to remediation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Lin- guistics: ACL 2024, pages 16078–16092, Bangkok, Thailand, August 2024. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.951. URL https://aclanthology.org/2024.findings-acl.951/. 374 Weihua Du, Pranjal Aggarwal, Sean Welleck, and Yiming Yang. Agentic-r1: Distilled dual-strategy reasoning.In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12029–12043, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.604. URL https:// aclanthology.org/2025.emnlp-main.604/. 365 Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and trends® in theoretical computer science, 9(3-4):211–487, 2014. 376 Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265–284. Springer, 2006. 376 Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back- translation at scale. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 489–500, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1045. URL https://aclanthology.org/D18-1045/. 370 Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fair- ness in large language models: A survey. Computational linguistics, 50(3):1097– 1179, 2024. 374, 375 Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson El- hage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Cather- ine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red teaming 380BIBLIOGRAPHY language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022. URL https://arxiv.org/abs/2209.07858. 371 Jiaxuan Gao, Jiaao Chen, Chuyi He, Shusheng Xu, Di Jin, and Yi Wu. From self- evolving synthetic data to verifiable-reward rl: Post-training multi-turn interactive tool-using agents, 2026. URL https://arxiv.org/abs/2601.22607. 372 Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu.Scal- ing synthetic data creation with 1,000,000,000 personas, 2025. URL https: //arxiv.org/abs/2406.20094. 368 Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms, 2023. URL https://arxiv.org/abs/2305.15717. 375 Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C ́ esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, S ́ ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https://arxiv.org/abs/2306.11644. 369, 373 Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach themselves to program better. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=SaRj2ka1XZ3. 369 Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 365 Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10.18653/v1/2023.acl-long.806. URL https://aclanthology. org/2023.acl-long.806/. 368, 369 Chengming Hu, Xuan Li, Dan Liu, Xi Chen, Ju Wang, and Xue Liu. Teacher-student architecture for knowledge learning: A survey, 2022. URL https://arxiv. org/abs/2210.17332. 365 Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ru- osen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004, 2025. 368 Yuanhong Huang, Huili Wang, Xueying Bai, Jinrui Wang, Jiajun Liu, Ziqin Wang, Wanchun Ni, Shangguang Wang, and Tao Qi. Robust membership inference for large language models under adversarial generative corruption. In Maria Liakata, BIBLIOGRAPHY381 Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 39531–39547, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.1835. URL https://aclanthology.org/ 2026.acl-long.1835/. 374 Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 15696–15707. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/ kandpal23a.html. 375 Minki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho, and Sung Ju Hwang. Dis- tilling LLM agent into small models with retrieval and code tools. In The Thirty- ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=VkicTqszOn. 371 Shivani Kapania, Stephanie Ballard, Alex Kessler, and Jennifer Wortman Vaughan. Ex- amining the expanding role of synthetic data throughout the ai development pipeline. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Trans- parency, FAccT ’25, page 45–60, New York, NY, USA, 2025. Association for Com- puting Machinery. ISBN 9798400714825. doi: 10.1145/3715275.3732005. URL https://doi.org/10.1145/3715275.3732005. 364 Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengx- uan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethink- ing benchmarking in NLP. In Kristina Toutanova, Anna Rumshisky, Luke Zettle- moyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110–4124, Online, June 2021. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.324. URL https://aclanthology.org/2021.naacl-main.324/. 371 Seongho Kim, Jihyun Moon, Juntaek Oh, Insu Choi, and Joon-Sung Yang. Survey and evaluation of converging architecture in llms based on footsteps of operations. IEEE Open Journal of the Computer Society, 6:1214–1226, 2025a. doi: 10.1109/OJCS. 2025.3587005. 363 Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, and Graham Neubig. Evaluating language models as synthetic data generators. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings 382BIBLIOGRAPHY of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 6385–6403, Vienna, Austria, July 2025b. Associa- tion for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/ 2025.acl-long.320. URL https://aclanthology.org/2025.acl-long. 320/. 373 Weizhen Li, Jianbo Lin, Zhuosong Jiang, Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhen- qiang Huang, Qianben Chen, Weichen Sun, Qiexiang Wang, Hongxuan Lu, Tianrui Qin, Chenghao Zhu, Yi Yao, Shuying Fan, Xiaowan Li, Tiannan Wang, Pai Liu, King Zhu, He Zhu, Dingfeng Shi, Piaohong Wang, Yeyi Guan, Xiangru Tang, Minghao Liu, Yuchen Eleanor Jiang, Jian Yang, Jiaheng Liu, Ge Zhang, and Wangchunshu Zhou. Chain-of-agents: End-to-end agent foundation models via multi-agent distil- lation and agentic rl, 2025a. URL https://arxiv.org/abs/2508.13167. 365 Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran.Small models strug- gle to learn from strong reasoners.In Wanxiang Che, Joyce Nabende, Ekate- rina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 25366–25394, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-89176-256- 5. doi: 10.18653/v1/2025.findings-acl.1301. URL https://aclanthology. org/2025.findings-acl.1301/. 366, 367 Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, Li Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, 4(8):669–677, 2022. 364 Philip Lippmann, Matthijs T. J. Spaan, and Jie Yang. Illuminating blind spots of lan- guage models with targeted agent-in-the-loop synthetic data, 2024. URL https: //arxiv.org/abs/2403.17860. 370 Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Yoav Gold- berg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.508. URL https://aclanthology.org/ 2022.findings-emnlp.508/. 370 Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinygsm: achieving ¿80 URL https:// arxiv.org/abs/2312.09241. 369 Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On LLMs-driven synthetic data generation, curation, and evalua- tion: A survey. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Find- ings of the Association for Computational Linguistics: ACL 2024, pages 11065– BIBLIOGRAPHY383 11082, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10.18653/v1/2024.findings-acl.658. URL https://aclanthology. org/2024.findings-acl.658/. 364, 373 Kevin Lu and Thinking Machines Lab.On-policy distillation.Think- ing Machines Lab:Connectionism, 2025.doi:10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. 367 Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. MathGenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2732–2747, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10.18653/v1/2024.acl-long.151. URL https://aclanthology. org/2024.acl-long.151/. 370 Alisia Maria Lupidi, Carlos Gemmell, Nicola Cancedda, Jane Yu, Jason E Weston, Jakob Nicolaus Foerster, Roberta Raileanu, and Maria Lomeli.Source2synth: Synthetic data generation and curation grounded in real data sources.In Will Synthetic Data Finally Solve the Data Access Problem?, 2025. URL https: //openreview.net/forum?id=z9rR3btRFI. 369 Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Al- izadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, and Matthias Gall ́ e. On leakage of code generation evaluation datasets. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Associa- tion for Computational Linguistics: EMNLP 2024, pages 13215–13223, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.772. URL https://aclanthology.org/ 2024.findings-emnlp.772/. 374 Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Smaranda Mure- san, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: 10.18653/v1/2022.acl-long.244. URL https://aclanthology. org/2022.acl-long.244/. 368 Shrey Pandit, Xuan-Phi Nguyen, Yifei Ming, Austin Xu, Jiayu Wang, Caiming Xiong, and Shafiq Joty. Synthesizing agentic data for web agents with progressive dif- ficulty enhancement mechanisms, 2025.URL https://arxiv.org/abs/ 2510.13913. 372 Divyansh Pareek, Simon S. Du, and Sewoong Oh. Understanding the gains from re- peated self-distillation, 2024. URL https://arxiv.org/abs/2407.04600. 368 384BIBLIOGRAPHY Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.Red teaming language models with language models.In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 3419–3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.225.URL https://aclanthology.org/ 2022.emnlp-main.225/. 371 Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Manoj Awal- gaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese, and Caiming Xiong.Apigen-mt: Agentic pipeline for multi-turn data generation via simu- lated agent-human interplay.In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Infor- mation Processing Systems, volume 38. Curran Associates, Inc., 2025.URL https://proceedings.neurips.c/paper_files/paper/2025/ file/5e3661f7fe4c8ac5652d62eb3d3c96ea-Paper-Datasets_ and_Benchmarks_Track.pdf. 371, 372 Haritz Puerto, Martin Gubri, Sangdoo Yun, and Seong Joon Oh. Scaling up member- ship inference: When and how attacks succeed on large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Compu- tational Linguistics: NAACL 2025, pages 4165–4182, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195- 7. doi: 10.18653/v1/2025.findings-naacl.234. URL https://aclanthology. org/2025.findings-naacl.234/. 374 Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, MAHMOUD KHADEMI, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, and Furu Wei. Scaling laws of synthetic data for language model.In Second Conference on Language Modeling, 2025.URL https://openreview.net/forum?id=UmUXPXHtdl. 364 Krithika Ramesh, Nupoor Gandhi, Pulkit Madaan, Lisa Bauer, Charith Peris, and Anjalie Field. Evaluating differentially private synthetic data generation in high- stakes domains. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, ed- itors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15254–15269, Miami, Florida, USA, November 2024. Association for Com- putational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.894. URL https: //aclanthology.org/2024.findings-emnlp.894/. 376 Krithika Ramesh, Daniel Smolyak, Zihao Zhao, Nupoor Gandhi, Ritu Agarwal, Margr ́ et V. Bjarnad ́ ottir, and Anjalie Field. SynthTextEval: Synthetic text data generation and evaluation for high-stakes domains. In Ivan Habernal, Peter Schu- lam, and J ̈ org Tiedemann, editors, Proceedings of the 2025 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages BIBLIOGRAPHY385 487–499, Suzhou, China, November 2025. Association for Computational Linguis- tics. ISBN 979-8-89176-334-0. doi: 10.18653/v1/2025.emnlp-demos.35. URL https://aclanthology.org/2025.emnlp-demos.35/. 373 Haris Riaz, Sourav Sanjukta Bhabesh, Vinayak Arannil, Miguel Ballesteros, and Gra- ham Horwood. MetaSynth: Meta-prompting-driven agentic scaffolds for diverse synthetic data generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Compu- tational Linguistics: ACL 2025, pages 18770–18803, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10. 18653/v1/2025.findings-acl.962. URL https://aclanthology.org/2025. findings-acl.962/. 374 Mart ́ ın Santill ́ an Cooper, Zahra Ashktorab, Hyo Jin Do, Erik Miehling, Werner Geyer, Jasmina Gajcin, Elizabeth M. Daly, Qian Pan, and Michael Desmond. Synthetic data for evaluation: Supporting LLM-as-a-judge workflows with EvalAssist. In Ivan Habernal, Peter Schulam, and J ̈ org Tiedemann, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pages 1–11, Suzhou, China, November 2025. Association for Computa- tional Linguistics. ISBN 979-8-89176-334-0. doi: 10.18653/v1/2025.emnlp-demos. 1. URL https://aclanthology.org/2025.emnlp-demos.1/. 370 Max Schaffelder and Albert Gatt. Synthetic eggs in many baskets: The impact of synthetic data diversity on llm fine-tuning, 2025. URL https://arxiv.org/ abs/2511.01490. 375 Parker Seegmiller, Kartik Mehta, Soumya Saha, Chenyang Tao, Shereen Oraby, Arpit Gupta, Tagyoung Chung, Mohit Bansal, and Nanyun Peng. FLAMES: Im- proving LLM math reasoning via a fine-grained analysis of the data synthesis pipeline.In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguis- tics: EMNLP 2025, pages 24746–24766, Suzhou, China, November 2025. Asso- ciation for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/ v1/2025.findings-emnlp.1346. URL https://aclanthology.org/2025. findings-emnlp.1346/. 374 Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar.Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold.In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 43000–43031. Cur- ran Associates, Inc., 2024.doi:10.52202/079017-1361.URL https: //proceedings.neurips.c/paper_files/paper/2024/file/ 4b77d5b896c321a29277524a98a50215-Paper-Conference.pdf. 374 Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rose- mary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, et al. Ai-assisted genera- tion of difficult math questions. arXiv preprint arXiv:2407.21009, 2024. 371 386BIBLIOGRAPHY Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget, 2024. URL https://arxiv.org/abs/2305.17493. 375 Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Ad- lam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Is- abelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kath- leen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura A Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with lan- guage models. Transactions on Machine Learning Research, 2024. ISSN 2835- 8856. URL https://openreview.net/forum?id=lNAyUngGFK. Expert Certification. 375 Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023. 368 Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Cl ́ ementine Four- rier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id= aKkAwZB6JV. 365 Michiel van der Meer, Piek Vossen, Catholijn M. Jonker, and Pradeep K. Murukan- naiah. An empirical analysis of diversity in argument summarization. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2028–2045, St. Julian’s, Malta, March 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.eacl-long.123. URL https: //aclanthology.org/2024.eacl-long.123/. 375 Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Mar- ius Hobbhahn. Position: Will we run out of data? limits of LLM scaling based on human-generated data. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=ViZcgDQjyG. 363, 364 Vijay Viswanathan, Xiang Yue, Alisa Liu, Yizhong Wang, and Graham Neubig. Syn- thetic data in the era of large language models. In Yuki Arase, David Jurgens, and Fei Xia, editors, Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 5: Tutorial Abstracts), pages 11–12, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-255-8. BIBLIOGRAPHY387 doi: 10.18653/v1/2025.acl-tutorials.7. URL https://aclanthology.org/ 2025.acl-tutorials.7/. 364 Dong Wang, Yang Li, Ansong Ni, Ching-Feng Yeh, Youssef Emad, Xinjie Lei, Liam Robbins, Karthik Padthe, Hu Xu, Xian Li, Asli Celikyilmaz, Ramya Raghavendra, Lifei Huang, Carole-Jean Wu, and Shang-Wen Li. Matrix: Peer-to-peer multi-agent synthetic data generation framework, 2025a. URL https://arxiv.org/abs/ 2511.21686. 365 Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a. 369 Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang.Co-evolving llm coder and unit tester via reinforcement learning.In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, pages 143630–143664. Curran Associates, Inc., 2025b.URL https: //proceedings.neurips.c/paper_files/paper/2025/file/ d38653cdaa8e992549e1e9e1621610d7-Paper-Conference.pdf. 368 Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunku- mar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varsh- ney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Sa- van Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10.18653/v1/2022.emnlp-main.340. URL https://aclanthology. org/2022.emnlp-main.340/. 368 Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v1/ 2023.acl-long.754. URL https://aclanthology.org/2023.acl-long. 754/. 366, 369, 370 Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W 388BIBLIOGRAPHY White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applica- tions via multi-agent conversation. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. URL https://openreview.net/forum?id= uAjxFFing2. 372 Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Stronger models are not always stronger teachers for instruction tuning. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), pages 4392– 4405, Albuquerque, New Mexico, April 2025a. Association for Computational Lin- guistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.224. URL https://aclanthology.org/2025.naacl-long.224/. 366 Zhangchen Xu, Adriana Meza Soria, Shawn Tan, Anurag Roy, Ashish Sunil Agrawal, Radha Poovendran, and Rameswar Panda. Toucan: Synthesizing 1.5m tool-agentic data from real-world mcp environments, 2025b. URL https://arxiv.org/ abs/2510.01179. 372 Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Aofan Liu, Hui Xiong, and Jian Guo.LongFaith: Enhancing long-context reasoning in LLMs with faithful synthetic data.In Wanxiang Che, Joyce Nabende, Ekate- rina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Associa- tion for Computational Linguistics: ACL 2025, pages 3236–3256, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256- 5. doi: 10.18653/v1/2025.findings-acl.169. URL https://aclanthology. org/2025.findings-acl.169/. 374 Kaiyu Yang,Aidan Swope,Alex Gu,Rahul Chalamala,Peiyang Song, Shixing Yu,Saad Godil,Ryan J Prenger,and Animashree Anandku- mar.Leandojo:Theorem proving with retrieval-augmented language mod- els.In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 21573–21612. Curran Associates, Inc., 2023.URL https://proceedings.neurips.c/paper_files/paper/2023/ file/4441469427094f8873d0fecb0c4e1cee-Paper-Datasets_ and_Benchmarks.pdf. 369 Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. Synthbio: A case study in faster curation of text datasets. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/ forum?id=Fkpr2RYDvI1. 371 Xiang Yue, Huseyin Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Sha- jari, Huan Sun, David Levitan, and Robert Sim. Synthetic text generation with differential privacy: A simple and practical recipe. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the BIBLIOGRAPHY389 Association for Computational Linguistics (Volume 1: Long Papers), pages 1321– 1342, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.74. URL https://aclanthology.org/2023. acl-long.74/. 376 Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen.Mammoth2: Scal- ing instructions from the web.In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neu- ral Information Processing Systems, volume 37, pages 90629–90660. Cur- ran Associates, Inc., 2024.doi:10.52202/079017-2877.URL https: //proceedings.neurips.c/paper_files/paper/2024/file/ a4ca07a108036f80cbb5b82285fd4b1-Paper-Conference.pdf. 369 Eric Zelikman, Jesse Mu, Noah D Goodman, and Yuhuai Tony Wu. Star: Self-taught reasoner bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems (NeurIPS), 2022a. 367 Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 15476–15488. Curran Associates, Inc., 2022b. URL https: //proceedings.neurips.c/paper_files/paper/2022/file/ 639a9a172c044fbb64175b5fad42e9a5-Paper-Conference.pdf. 374 Hengyuan Zhang, Shiping Yang, Xiao Liang, Chenming Shang, Yuxuan Jiang, Chao- fan Tao, Jing Xiong, Hayden Kwok-Hay So, Ruobing Xie, Angel X. Chang, and Ngai Wong. Find your optimal teacher: Personalized data synthesis via router- guided multi-teacher distillation. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pages 14619–14637, San Diego, California, United States, July 2026a. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.666. URL https://aclanthology.org/2026.acl-long.666/. 366 Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, and Stefano Soatto. Reinforcement-aware knowl- edge distillation for LLM reasoning. In Forty-third International Conference on Machine Learning, 2026b. URL https://openreview.net/forum?id= 0RBMhFule7. 367 Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a- judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. 371 Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar. Self-challenging language model agents.In D. Belgrave,C. Zhang, 390BIBLIOGRAPHY H. Lin,R. Pascanu,P. Koniusz,M. Ghassemi,and N. Chen,edi- tors,Advances in Neural Information Processing Systems,volume 38, pages 113959–113991. Curran Associates, Inc., 2025.URL https: //proceedings.neurips.c/paper_files/paper/2025/file/ a5a305fac88fb4ae40969cfec5eef48d-Paper-Conference.pdf. 368 Chapter 20 *Advanced Transformer and Agentic Architectures What You Will Learn In this chapter you will learn about advanced transformer and agentic architectures. Specifically, you will be able to: • Understand the concept of Turing-completeness, and learn that transformers are, in principle, Turing-complete. • Understand that transformer models can perform planning, when we train them on decision traces, instead of on natural language. • Describe how embeddings can be connected directly to other embeddings, in looped language models, skipping tokenization layers. • Describe how agents can perform reinforcement learning in-context, by storing a history of state-action-reward triples. • Describe the advantages and disadvantages of self-modifying agents. • Understand how metalearning can be applied to harness design, to create a meta- harness that improves the basic harness. 20.1 Transformer Architectures The success of large language models has spawned interesting new research avenues into transformer architectures. LLMs were good at language, but not at other problem types such as reasoning and planning. This caused researchers to explore the limits of the types of computations that transformers were able to do, and we will discuss some of the topics in the following sections. Specifically, they asked the questions whether transformers were Turing complete (Section 20.1.1)? The weak performance on planning problems prompted studies that trained transformers directly on decision traces, to see if that allowed them to be able to plan and backtrack (Section 20.1.2). Next, we will look at looped language models, where multiple models are connected 391 392CHAPTER 20. *ADVANCED AGENTIC ARCHITECTURES directly at the level of the embeddings. 20.1.1 Transformers are Turing-complete The difficulty of LLMs to manage state at inference time suggests that LLMs are lim- ited in the type of computations that they can perform. In theoretical computer science the concept of Turing completeness indicates whether a certain model of computation is powerful enough to be able to perform any type of computation of a certain category (which includes being able to manage state values correctly). This concept was intro- duced in a famous paper by Alan Turing in 1936, in which he proved that a certain type of machine would be powerful enough to do so [Turing, 1936]. To be called Turing complete, a system must be able to simulate any other computational device and be able to process any problem that can be expressed in an algorithm (but perhaps take in- finite time and memory to do so). It turns out that it is not too difficult to achieve Turing completeness; most general purpose programming languages are Turing complete. 1 The difficulty of LLMs—and of transformer-based models in general—to manage state would suggest that the transformer architecture may not be Turing complete. In- terestingly, already in 2021 P ́ erez et al. [2021] have studied this question, and have proven that the transformer architecture is able to simulate a Turing machine, and is thus Turing complete. This is an encouraging result, since it indicates that there is no fundamental reason why transformers would not be able to manage state correctly. It may well be a matter of finding the right prompt, or finding the right pretraining and finetuning dataset. Unfortunately, the fact that something is theoretically possible, does not mean that achieving it will be easy, or that it can be done efficiently. For example, it might be that certain architectural extensions to the transformer architecture would enhance the ease with which efficient state management can be achieved. We will now discuss methods that have been developed so that LLMs can efficiently solve decision making and planning problems. 20.1.2 Pretraining Backtracking on Decision Traces The finding that transformers are Turing-complete encourages research in planning and decision making. In this section we will discuss studies that pretrain a transformer architecture on decision traces, to good results. Note that this section is not about transformers that are trained on natural language—LLMs—but on transformers trained on decision traces. First we discuss pretraining transformers directly on decision traces: sequences of state-action-reward triples, not language tokens. Several works study whether trans- former networks can be trained to perform reinforcement learning—to learn from feed- back, to perform learned policies, and to explore to learn better actions. Chen et al. [2021], Lee et al. [2023], Wang et al. [2024] showed that basic transformer architec- tures can indeed be pretrained and finetuned on decision traces from typical planning 1 For example, the Python programming language is Turing complete, but simple regular expression matching, without backreferencing, is not. 20.1. TRANSFORMER ARCHITECTURES393 Figure 20.1: The Cartpole Problem: Balance the Pole just by Moving the Cart Left or Right problems. When an empty (non-language trained) transformer is pretrained on deci- sion traces, it can exhibit in-context reinforcement learning (ICRL). So far, this has been shown using toy problems such as Boyan’s chain and Cartpole (see Figure 20.1). Furthermore, in the games of Chess and Hex encouraging results were achieved by pretraining the decision transformer on minimax [Markeeva et al., 2024, Knuth and Moore, 1975] and MCTS [Browne et al., 2012] search traces [Schultz et al., 2024, Kempinski et al., 2025]. Transformers were able to learn to lookahead and search a few steps. Other works report encouraging results in the game of Othello [Li et al., 2023, Kempinski et al., 2025, Chen et al., 2025]. A further work that studies how transformer architectures can be trained to perform decisions is self backtracking [Yang et al., 2026a]. In this approach the model is again trained on decision traces. However, a new kind of token is introduced, <backtrack>, that prompts the model to perform explicit backtracking at inference time. The model is trained on a dataset of decisions, and when an error is encountered during training, the <backtrack> token is learned, to find an alternative solution. At inference time, then, this backtracking is performed. A mechanism that consists of expansion/back- tracking/selection is executed by the model, so that alternative solutions for the state in question are examined. The self backtracking approach reports good results on typical planning problems such as the 24-game and maze navigation, compared to standard baselines on small versions of DeepSeek-R1 and Qwen3. Through the introduction of a new type of to- ken and new backtracking-functionality in the model, this approach allows enhanced- transformer models to perform search, blurring the line between training and inference. These are encouraging results, although it is still an open question to what extend the weakness of transformers in managing state has been solved. Performing inference- time ICRL on larger problems is an area of active research. 394CHAPTER 20. *ADVANCED AGENTIC ARCHITECTURES 20.1.3 LoopedLLM and Recursive Multi-Agent Systems Improving the expressive power of transformers brings them closer to a general purpose programming architecture. We will discuss two other developments: looped language models and recursive systems. It is easy to connect LLM agents at the language level: just let them speak to each other in natural language, on which they are all trained. However, this necessi- tates many (de)tokenization steps, which take time and introduces differences between embeddings. Can we directly connect embeddings, bypassing the tokenizers, and do training and inference at that level? Giannou et al. [2023] show how looped transformers (transformers whose input neurons are directly connected to transformer output neurons, either from the same or from another model) can emulate a basic calculator, a basic linear algebra library, and in-context learning algorithms that employ backpropagation. Looped transformers are a powerful architecture, see for example Yang et al. [2023], Saunshi et al. [2025], Zhu et al. [2025]. A further development of direct embedding-to-embedding connections is recur- sive multi-agent systems [Yang et al., 2026b]. This approach connects heterogeneous agents in a collaboration loop, enabling in-distribution latent thought generation and cross-agent latent state transfer. An inner-outer loop learning algorithm (metalearning, see Chapter 8) performs whole-system co-optimization through shared gradient-based credit assignment across recursion rounds. Experiments with four agents over bench- marks from mathematics, science, medicine, search, and code generation, showed sta- ble multi-agent training. In comparison with other single/multi-agent and recursive baselines, accuracy is improved by a few percentage points, inference is twice as fast, and token usage is reduced by comparable ratio. An innovation of RecursiveMAS is the introduction of a two-level recursive-link. Training a multi-agent system is difficult because gradients must flow through multiple models, and this process can become unstable. RecursiveMAS improves this process by freezing the parameters of the underlying LLMs and only training the lightweight RecursiveLink modules. The training occurs in two stages. 20.2 Agentic Architectures In addition to new transformer architectures, the work on agentic LLMs has also led to advanced agentic architectures. First, we will look at in-context reinforcement learning, where we will see that LLMs are able to perform policy improvement when the prompt is given a history of state-action-reward triples. Second, we will look at self-modifying agents, and discuss the advantages and disadvantages of this approach. Finally, we will look at meta agents. 20.2.1 In-Context Reinforcement Learning Recall from Section 2.2.3 that the goal of reinforcement learning is to compute a pol- icy that has an optimal action for each state [Sutton and Barto, 2018]. As the agent 20.2. AGENTIC ARCHITECTURES395 Environment: 푇(푠,푎) → 푠 ′ , 푅(푠 ′ ) → 푟 Agent: 푟 improves 휋 휃 (푠) → 푎 (푠 ′ ,푟) 푎 Figure 20.2: For reference, the standard reinforcement learning loop where the agent learns a 휃-parameterized policy 휋 휃 from feedback rewards 푟 from the environment upon its actions 푎, selected from policy 휋 휃 , etc. [Sutton and Barto, 2018] converges towards this policy, in each timestep, the agent learns these actions from feedback from its environment, the rewards. These rewards reinforce the confidence of the agent in the policy’s actions in each state (see Figure 20.2). The self-reflection approaches from the previous section borrow many ideas from reinforcement learning for the agentic harness. We will now discuss related methods that optimize prompts by in-context reinforcement learning, where the LLM performs prompt improvement. Reinforcement-type learning mechanisms have been observed to emerge directly in LLMs, when environment histories were provided to the LLM by an external prompt optimizer: the LLM in certain self-reflection loops can perform policy-improvement. This phenomenon is called in-context reinforcement learning (ICRL) [Song et al., 2025], see Figure 20.3. Krishnamurthy et al. [2024] asked the question whether generic LLMs that are trained on language can explore in-context, and whether they can learn in-context from feedback. This question has since been answered positively by various studies [Demir- can et al., 2024, Song et al., 2025, Moeini et al., 2025, Monea et al., 2024, Nie et al., 2024, Wang et al., 2024]. In ICRL, an external harness iteratively calls the LLM, feeding it a sequence of action, state, and reward triples, in-context (Figure 20.3). The history of triples grows in each iteration. The LLM reacts by improving its implicit policy, providing increasingly good action choices in each iteration, thus we conclude that it performs the essence of reinforcement learning inside the LLM. In ICRL, the agent-algorithm selects an action, to sample a step and get an answer (a state, reward pair) from the environment, see Figure 20.3. A Markov decision pro- cess, or MDP, is a process in which all relevant information to calculate the next state is contained in the previous state and the action taken. Therefore, the prompt for the policy-LLM contains all relevant information, it accumulates the history of (푠,푎,푟)- triples, which the LLM can use to improve the policy of the best action for each state. For state, the history of answers is a natural choice, but, in general, how to best model LLM prompt improvement as an MDP is still an active area of research. In ICRL the agent uses the LLM for the policy-function. The LLM can also be used (with a different prompt) to play the role of environment, as in a self-reflection harness. In-context reinforcement learning occurs in a multi-round prompting harness, where, after the environment-LLM generates a response at the current round, a numerical re- 396CHAPTER 20. *ADVANCED AGENTIC ARCHITECTURES Environment: 푇(푠,푎) → 푠 ′ , 푅(푠 ′ ) → 푟 Agent: 휋 퐿푀 푃휋 (푠,푟) → 푎 history∪(푠 ′ ,푎,푟) 푎 Figure 20.3: ICRL: Prompt Optimization with Reinforcement Learning ward is calculated by the LLM for the response that the LLM just calculated. At the next round, the agent-LLM is prompted again with the same task and a context that now consists of the previous responses and rewards. The quality of the agent-LLM’s response increases as the context becomes longer. Just like a reinforcement learning algorithm, the LLM is able to maximize the numeric reward, at inference time. Song et al. [2025] report how the ICRL approach was validated on the applications of tree of thoughts (Game of 24, creative writing, and ScienceWorld), outperforming self refine and reflexion. How does the ICRL process work, inside the LLM? At inference time the LLM per- forms a forward pass through the transformer network, computing the inference token. This forward pass has been found to implicitly perform reinforcement learning [Wang et al., 2024]. In other words, in-weights learning is where the model has learned the right answer from its trained weights, at training time; in-context learning is where the model has learned the right answer at inference time from few-shot examples provided in its context; and in-context reinforcement learning is where the LLM has learned the right answer using (apparent) reinforcement learning on a growing sequence of the state-action-reward triples that it produced itself through inference. Thus, the rein- forcing algorithm, optimizing the policy, is done implicitly by the LLM forward pass (inference step) when a sequence of prompts or history is given in the context. Note that the LLM can be used for the agent, as a policy optimizer, and for the environment, to calculate the next state and reward. To ensure that the agent is learning without accidentally benefiting from the environment’s knowledge, it is better to use a separate LLM. This, too, is an active area of research. 20.2.2 Self-Modifying Hyperagents In this book, we have discussed self-reflective prompt improvers that are based on search algorithms and reinforcement learning. The code for these external algorithms is written by human researchers, and does not change as the agent is learning. The approach by which the prompt is improved is limited by these static algorithms. LLMs excel at manipulating and judging computer code [Zhuge et al., 2024]. We could, in principle, easily ask them to generate code for an agentic prompt harness, or for an in-context reinforcement learning harness. We could even ask the LLM to run the code on a particular problem instance, and, based on the outcome of the experiment, ask for an improvement of the prompt-improver code. If the LLM is good at using experiment-results for writing prompt-improvement code, this approach could work 20.2. AGENTIC ARCHITECTURES397 better than standard approaches from the published literature. Self-modifying code (computer programs that alter their own instructions while they are being executed) has a long history in computer science [Cai et al., 2007], as an alternative to conditional branching statements. Many computer languages support it, from assembly languages, to Lisp [Abelson and Sussman, 2022] and Python. It is a powerful method that can be used for performance improvement, for example in just-in-time compilation [Ansel et al., 2011]. However, programs become less pre- dictable, which may cause problems in finding errors in the program, and raises secu- rity concerns. For this reason the use is sometimes discouraged when predictability of programs is important. An advantage of self-modifying agents is open-endedness: they can continuously improve themselves, without fixed boundaries set by static algorithms [Schmidhuber, 2007]. A challenge is to do this safely [Zhang et al., 2026b]. In Darwin-G ̈ odel- Machine-Hyperagents the authors integrate (1) an agent to solve the target task with (2) a meta agent to modify itself and the task agent, into a single (editable) program. Note that the meta-level modification procedure is itself editable, enabling self-modification. This allows, in principle, not only improving the task-solving behavior, but also the mechanism that generates future improvements [Zhang et al., 2026c]. The experi- ments, however, do not achieve full openness, as they use a fixed task distribution, as well as fixed parent selection and evaluation protocols. As approaches become more open-ended, paradoxically, a challenge becomes to control the process of improvement, to ensure that it does not degenerate, but keeps improving [Cemri et al., 2025, Ecoffet et al., 2020]. In general, the problem of powerful metalearning approaches is to ensure stable learning processes. Powerful generation requires powerful evaluation—adaptation to fit the environment. 20.2.3 Meta Agents The two-level optimization process of metalearning that we have seen in many self- reflection approaches, can be applied as an explicit design principle in harness design for agents. Recently, several such architectures have been published. In these designs, the inner learning loop is formed by the harness and the LLM, and the outer learning loop improves the configuration of this inner harness. The inner harness configuration typically performs prompt management, memory management, tool use, and security and control, and can be represented as language or code. The outer loop analyzes execution traces of the inner loop, to improve the configuration or code of the inner harness. We will briefly describe some of these approaches. Self improving coding agent [Robeyns et al., 2025] is based on the idea of self- modifying agents. They demonstrate that an agent system, equipped with basic coding tools, can autonomously edit itself. Their approach improves performance on software engineering benchmark tasks, achieving substantial performance gains from 17 to 53 percent. Meta harness [Lee et al., 2026] also achieves good results, using external memory and text optimization. Figure 20.4 illustrates how meta harness works. (1) A filesystem has all prior candidates’ source code, execution traces, and scores; the agent reads this, and generates code for a new harness. (2) The proposed harness is evaluated on 398CHAPTER 20. *ADVANCED AGENTIC ARCHITECTURES Figure 20.4: Illustration of the Meta Harness approach [Lee et al., 2026] Figure 20.5: Illustration of the Shepherd Meta Agent Environment [Yu et al., 2026] evaluation tasks. (3) All proposed code, reasoning traces, and evaluation scores are stored in the filesystem in a new directory, and the loop repeats. Self harness [Zhang et al., 2026a] and Agentic Harness Engineering [Lin et al., 2026] further explore harness design, achieving substantial improvements on software engineering tasks. Recursive self-evolving agents [Nguyen et al., 2026] focuses on the selection aspect of self-improving agents. Another use case is described by Luo et al. [2026]. They use a meta harness to design a harness for turning a scientific paper into a conference-poster. This work thus has contributions on two levels: a method for making a meta-harness, and a concrete harness for designing posters—which the authors, obviously, use on their own paper. The Shepherd system [Yu et al., 2026] provides a functional meta agent program- ming environment. 2 Shepherd defines LLM agents as Python functions, allowing 2 https://shepherd-agents.ai 20.3. CONCLUDING REMARKS399 an agent’s execution traces to be monitored, rolled-back, forked, modified, and re- sumed. Early experiments show significant performance improvements on three differ- ent benchmarks, as illustrated in Figure 20.5. A comprehensive survey with more approaches on self-improving agent designs is Ren et al. [2026]. Their taxonomy explicitly recognizes the two-level architecture. 20.3 Concluding Remarks In this chapter we have discussed some advanced architectures, at the level of the trans- formers, and at the level of the agents. Both are highly active fields of research. Bibliography Harold Abelson and Gerald Jay Sussman. Structure and Interpretation of Computer Programs: JavaScript Edition. MIT press, 2022. 397 Jason Ansel, Petr Marchenko, Ulfar Erlingsson, Elijah Taylor, Brad Chen, Derek L Schuff, David Sehr, Cliff L Biffle, and Bennet Yee. Language-independent sandbox- ing of just-in-time compilation and self-modifying code. In Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation, pages 355–366, 2011. 397 Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samoth- rakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Trans- actions on Computational Intelligence and AI in games, 4(1):1–43, 2012. 393 Hongxu Cai, Zhong Shao, and Alexander Vaynberg. Certified self-modifying code. In Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 66–77, 2007. 397 Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi-agent llm systems fail?arXiv preprint arXiv:2503.13657, 2025. 397 Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Rein- forcement learning via sequence modeling. Advances in neural information process- ing systems, 34:15084–15097, 2021. 392 Xinyi Chen, Yifei Yuan, Jiaang Li, Serge Belongie, Maarten de Rijke, and Anders Søgaard. What if othello-playing language models could see?arXiv preprint arXiv:2507.14520, 2025. 393 Can Demircan, Tankred Saanum, Akshay K Jagadish, Marcel Binz, and Eric Schulz. Sparse autoencoders reveal temporal difference learning in large language models. arXiv preprint arXiv:2410.01280, 2024. 395 400BIBLIOGRAPHY Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open questions in creating safe open- ended ai: Tensions between control and creativity. In Artificial Life Conference Pro- ceedings 32, pages 27–35. MIT Press One Rogers Street, Cambridge, MA 02142- 1209, USA journals-info . . . , 2020. 397 Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. In In- ternational Conference on Machine Learning, pages 11398–11442. PMLR, 2023. 394 Benjamin Kempinski, Ian Gemp, Kate Larson, Marc Lanctot, Yoram Bachrach, and Tal Kachman. Game of thoughts: Iterative reasoning in game-theoretic domains with large language models. In Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems, pages 1088–1097, 2025. 393 Donald E Knuth and Ronald W Moore. An analysis of alpha-beta pruning. Artificial intelligence, 6(4):293–326, 1975. 393 Akshay Krishnamurthy, Keegan Harris, Dylan J Foster, Cyril Zhang, and Aleksan- drs Slivkins.Can large language models explore in-context?arXiv preprint arXiv:2403.15371, 2024. 395 Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems, 36:43057–43083, 2023. 392 Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026. 397, 398 Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Vi ́ egas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learn- ing Representations, 2023. URL https://openreview.net/forum?id= DeG07_TcZvT. 393 Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuan- jing Huang, Hang Yan, Zhenhua Han, Tao Gui, et al. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026. 398 Yaxin Luo, Haobin Jiang, Jialv Zou, Xu Huang, Wenhao Yan, Haodong Li, Zhengrong Yue, Jing Li, Xiaofu Chen, Xiaohan Zhao, et al. Autodesign: Meta-harness opti- mization for long-horizon agentic design. arXiv preprint arXiv:2608.13560, 2026. 398 Larisa Markeeva, Sean McLeish, Borja Ibarz, Wilfried Bounsi, Olga Kozlova, Alex Vitvitskyi, Charles Blundell, Tom Goldstein, Avi Schwarzschild, and Petar BIBLIOGRAPHY401 Veli ˇ ckovi ́ c. The clrs-text algorithmic reasoning language benchmark. arXiv preprint arXiv:2406.04229, 2024. 393 Amir Moeini, Jiuqi Wang, Jacob Beck, Ethan Blaser, Shimon Whiteson, Rohan Chan- dra, and Shangtong Zhang. A survey of in-context reinforcement learning. arXiv preprint arXiv:2502.07978, 2025. 395 Giovanni Monea, Antoine Bosselut, Kiant ́ e Brantley, and Yoav Artzi. Llms are in- context reinforcement learners, 2024. 395 Michael Nguyen, Quoc Nguyen, and Paul Vuong. Recursive self-evolving agents via held-out selection. arXiv preprint arXiv:2606.28374, 2026. 398 Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed H Chi, Quoc V Le, and Minmin Chen. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint arXiv:2410.06238, 2024. 395 Jorge P ́ erez, Pablo Barcel ́ o, and Javier Marinkovic. Attention is turing-complete. Jour- nal of Machine Learning Research, 22(75):1–35, 2021. 392 Zhe Ren, Yimeng Chen, Dandan Guo, Guowei Rong, Tonghui Li, RB Xiong, Qingfeng Lan, Wenyi Wang, Li Nanbo, Yibo Yang, et al. Self-improvements in modern agentic systems: A survey. arXiv preprint arXiv:2607.13104, 2026. 399 Maxime Robeyns, Martin Szummer, and Laurence Aitchison. A self-improving coding agent. arXiv preprint arXiv:2504.15228, 2025. 397 Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17416, 2025. 394 J ̈ urgen Schmidhuber. G ̈ odel machines: Fully self-referential optimal universal self- improvers. In Artificial general intelligence, pages 199–226. Springer, 2007. 397 John Schultz, Jakub Adamek, Matej Jusup, Marc Lanctot, Michael Kaisers, Sarah Per- rin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, et al. Mastering board games by external and internal planning with language models. arXiv preprint arXiv:2412.12119, 2024. 393 Kefan Song, Amir Moeini, Peng Wang, Lei Gong, Rohan Chandra, Yanjun Qi, and Shangtong Zhang. Reward is enough: Llms are in-context reinforcement learners. arXiv preprint arXiv:2506.06303, 2025. 395, 396 Richard S Sutton and Andrew G Barto. Reinforcement Learning. Adaptive Compu- tation and Machine Learning series. Bradford Books, Cambridge, MA, 2 edition, November 2018. 394, 395 Alan Turing. On computable numbers, with an application to the entscheidungsprob- lem. J. of Math, 58(345-363):5, 1936. 392 402BIBLIOGRAPHY Jiuqi Wang, Ethan Blaser, Hadi Daneshmand, and Shangtong Zhang. Transformers learn temporal difference methods for in-context reinforcement learning. arXiv preprint arXiv:2405.13861, 2024. 392, 395, 396 Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped trans- formers are better at learning learning algorithms. arXiv preprint arXiv:2311.12424, 2023. 394 Xiao-Wen Yang, Xuan-Yi Zhu, Ding-Chu Zhang, Wen-Da Wei, Jie-Jing Shao, Zhi Zhou, Lan-Zhe Guo, and Yu-Feng Li. Step back to leap forward: Self-backtracking for symbolic reasoning and planning in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 27657–27665, 2026a. 393 Xiyuan Yang, Jiaru Zou, Rui Pan, Ruizhong Qiu, Pan Lu, Shizhe Diao, Jindong Jiang, Hanghang Tong, Tong Zhang, Markus J. Buehler, Jingrui He, and James Zou. Re- cursive multi-agent systems, 2026b. URL https://arxiv.org/abs/2604. 25917. 394 Simon Yu, Derek Chong, Ananjan Nandi, Dilara Soylu, Jiuding Sun, Christopher D Manning, and Weiyan Shi. Shepherd: Enabling programmable meta-agents via re- versible agentic execution traces. arXiv preprint arXiv:2605.10913, 2026. 398 Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-harness: Harnesses that improve themselves. arXiv preprint arXiv:2606.09498, 2026a. 398 Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open-ended evolution of self-improving agents, 2026b. URL https: //arxiv.org/abs/2505.22954. 397 Jenny Zhang, Bingchen Zhao, Wannan Yang, Jakob Foerster, Jeff Clune, Minqi Jiang, Sam Devlin, and Tatiana Shavrina. Hyperagents, 2026c. URL https://arxiv. org/abs/2603.19461. 397 Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, et al. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025. 394 Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuan- dong Tian, et al. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934, 2024. 396 Chapter 21 *Mechanistic Interpretability for LLM Agents What You Will Learn In this chapter you will learn how to look inside the black box of large language models using tools from explainable artificial intelligence and mechanistic inter- pretability. Specifically, you will be able to: • Distinguish between the four main interpretability paradigms: behavioral, attri- butional, concept-based, and mechanistic interpretability. • Explain how hallucination relates to faithfulness, and describe how mechanistic methods can identify the internal circuits responsible for model failures. • Describe the residual stream formulation of the transformer and explain how information is iteratively refined across layers. • Apply the logit lens and tuned lens to visualize how a model’s predictions evolve through its layers. • Explain polysemanticity and superposition, and understand why individual neu- rons are poor units of interpretation. • Use sparse autoencoders for dictionary learning to decompose model activations into monosemantic, human-interpretable features. • Perform circuit analysis and activation patching to identify and causally verify the internal algorithms responsible for specific model behaviors. 21.1 Opening the LLM How do LLMs work inside? Opening up the black box of large neural architectures is an important topic of research. We wish to understand how the billions of neurons embed representations, how they reason and how they come to conclusions. The field of explainable artificial intelligence (XAI) provides many different methods to do so [Minh et al., 2022, Rios et al., 2020, Van Stein et al., 2022, Selvaraju et al., 2017, Ali 403 404 CHAPTER 21. *MECHANISTIC INTERPRETABILITY FOR LLM AGENTS et al., 2022]. Static XAI methods from the classical machine learning community have been quite successful [Molnar, 2020]. Methods exist to relate how input pictures map to output classes, for example using feature maps [Ren et al., 2016, Kohonen, 1982, Red- mon et al., 2016]. Counterfactual analysis [Karimi et al., 2020, Huang et al., 2024], LIME [Ribeiro et al., 2016] and SHAP [Lundberg and Lee, 2017], among others, help understand how inputs map to outputs for structured data. These methods belong to the behavioral and attributional class of explainability algorithms, where the model is seen as a black-box (Figure 21.1). Attribution methods such as SHAP and LIME approximate the contributions of different input features to the final output. Behavioral analysis such as achieved by using counterfactual methods provide insight into which features matter (but not perse how much they matter). Figure 21.1: Taxonomy of Interpretability Paradigms. Behavioral interpretability treats the model as a black box, mapping input-output relations (푓(푥) = 푦). Attri- butional methods quantify the influence of specific input features on the final output using gradients or saliency. Concept-based interpretability employs probes to identify high-level, human-understandable representations within hidden layers. Mechanistic interpretability aims to reverse-engineer the specific causal circuits and internal algo- rithms that govern model behavior. Moving beyond surface-level input-output correlations, concept-based interpretabil- ity seeks to identify whether specific, human-understandable information is encoded within the model’s internal states. This is primarily investigated through linear prob- ing, where a simple supervised classifier is trained on frozen hidden activations to detect the presence of particular semantic concepts—such as parts of speech, senti- ment, or factual entities [Alain and Bengio, 2016, Belinkov, 2022]. While attribution methods highlight which parts of the input are influential, probing allows researchers to verify what knowledge the model represents at different layers (Figure 21.1). How- ever, a known limitation of probing is the causal gap: discovering that a concept is 21.2. HALLUCINATION AND FAITHFULNESS IN MODEL REASONING405 represented in a layer does not necessarily mean the model uses that information for its final prediction. More recently, dynamic methods have been developed to uncover patterns under the hood of large transformer models. These belong to the sub-domain of mechanis- tic interpretability (MI). The goal of MI is to uncover the mechanisms by which the model dynamically comes to conclusions [Nanda et al., 2023, Bereska and Gavves, 2024, Ferrando et al., 2024, Rai et al., 2024]. In other words, MI seeks a low-level, causal explanation of a model’s behavior in terms of its internal activations and con- nections, analogous to decompiling a program’s binary into readable code. Methods such as sparse autoencoders [Cunningham et al., 2023, Makelov et al., 2024], neural lenses [Black et al., 2022] and circuit discovery [Conmy et al., 2023] are being used to successfully enhance insight into how LLMs work, for example, in chess [Davis and Sukthankar, 2024]. XAI and MI in particular are active areas of research that will allow us to better understand how LLMs reason and come to conclusions [Sharkey et al., 2025]. Once a better understanding is reached, LLMs can be improved accordingly, for example, to reduce hallucinations. This chapter will dive deeper into the many facets of MI. 21.2 Hallucination and Faithfulness in Model Reason- ing A significant barrier to the deployment of LLMs in high-stakes environments is the phenomenon of hallucination: the generation of syntactically plausible but factually incorrect content. From an interpretability perspective, the challenge lies in distin- guishing whether a model’s output is the result of a robust internal computation or a post-hoc justification. This distinction is formalized as the study of faithfulness, the degree to which a model’s stated reasoning (e.g., in chain of thought prompting) is causally responsible for its final output. Mechanistic interpretability (MI) provides the tools necessary to move beyond be- havioral metrics of accuracy and toward a causal understanding of model failure modes. For instance, research using sparse autoencoders (SAEs) has identified specific hon- esty or knowledge-retrieval circuits [Lindsey et al., 2025]. In these studies, researchers identified latent features that activate when a model encounters unfamiliar entities; a failure in the inhibitory logic of these circuits, where the model proceeds to gener- ate tokens despite low activation in the relevant knowledge feature, directly results in fabricated answers. Furthermore, MI allows us to investigate the causal necessity of intermediate rea- soning steps. While a model may produce a convincing chain of thought (CoT), it remains possible that the model has already calculated the final answer via a separate internal shortcut and is merely generating the CoT as a stochastic parrot. To test this, Chen et al. [2026] utilized causal mediation analysis combined with SAEs to intervene on the latent thought process during arithmetic tasks. By inject- ing the latent activations from a CoT-enabled run into a standard zero-shot run, they 406 CHAPTER 21. *MECHANISTIC INTERPRETABILITY FOR LLM AGENTS demonstrated that in sufficiently large models, these intermediate thoughts significantly boosted solution confidence. This provides empirical evidence that for larger architec- tures, the CoT is not merely cosmetic but represents a faithful causal link in the model’s reasoning chain. Conversely, the absence of such causal links in smaller models suggests that faith- ful reasoning may be an emergent property of scale, appearing only when the model possesses sufficient parameters to represent complex, multi-step algorithms. Devel- oping robust metrics to quantify this faithfulness is a critical frontier in AI safety; by peeking into the model’s activations, we can potentially develop hallucination detectors that trigger when a model’s internal confidence features do not align with its external verbosity. 21.3 The Residual Stream and Iterative Refinement Before exploring specific MI methods it is necessary to establish a formal mental model of the transformer architecture. Unlike classical feed-forward networks where each layer replaces the representation of the previous one, modern transformers operate on a residual stream [Elhage et al., 2021]. Formally, let ℎ ℓ ∈R 푑 푚표푑푒푙 be the state of the hidden representation after layer ℓ. The Transformer architecture is defined by the recurrence: ℎ ℓ = ℎ ℓ−1 +F ℓ (ℎ ℓ−1 )(21.1) where F ℓ represents the computation performed by the ℓ-th layer (comprising multi- head attention and feed-forward blocks). This structure implies that the final representation ℎ 퐿 is a cumulative sum of the initial embedding 푥 푒푚푏푒푑 and the contributions (Δℎ) from every subsequent layer: ℎ 퐿 = 푥 푒푚푏푒푑 + 퐿 ∑︁ ℓ=1 Δℎ ℓ (21.2) In this view, the residual stream acts as a high-dimensional communication channel. Individual layers do not transform the entire state; rather, they read information from the stream, perform a local computation and write their results back into the stream via vector addition. This additive nature ensures that information can persist across many layers, and more importantly, it justifies the linear decomposition of the model’s final output. 21.4 Polysemanticity and Superposition A primary bottleneck in reverse-engineering neural networks is polysemanticity (Fig- ure 21.2): the empirical observation that individual neurons frequently activate for multiple, semantically unrelated concepts. This phenomenon is deeply linked to super- position, a regime where a model represents 푛 features in 푚 dimensions, where 푛 > 푚. 21.4. POLYSEMANTICITY AND SUPERPOSITION407 0.00.20.40.60.81.0 Neuron A Activation 0.0 0.2 0.4 0.6 0.8 1.0 Neuron B Activation Concept 1 Concept 2 Concept 3 Concept 4 Concept 5 Polysemanticity: A single neuron (axis) participates in representing multiple distinct concepts. These concepts are distributed as 'directions' in the activation space rather than aligning perfectly with individual axes. Polysemanticity in Neuron Activation Space Figure 21.2: The concept of polysemanticity in LLMs and other transformer based models Under the linear representation hypothesis, models exploit the high-dimensional ge- ometry of activation space to store features as almost-orthogonal vectors. While this allows for efficient information compression, it results in a basis mismatch where the fixed neuron basis does not align with the true feature basis. A formalization was provided by Elhage et al. [2022], who demonstrated using toy models that superposition is a mathematically principled strategy for models to maxi- mize information density when features are sparse. They identified phase transitions in feature encoding: as the sparsity of a concept increases, the model can pack more features into the same number of dimensions, albeit at the cost of interference or noise between those features. To resolve this entanglement, researchers have turned to dictionary learning, most notably via sparse autoencoders (SAEs). The objective is to learn a linear transforma- tion that maps the tangled neuron activations into a much higher-dimensional, sparse latent space. Bricken et al. [2023] successfully applied this to small-scale transform- 408 CHAPTER 21. *MECHANISTIC INTERPRETABILITY FOR LLM AGENTS ers, demonstrating that the resulting SAE features are significantly more monosemantic and human-interpretable than the original neurons. This suggested that models are not inherently uninterpretable, but rather that we have been looking at them through the wrong coordinate system. Subsequent work has explored whether architectural modifications can mitigate the emergence of polysemanticity. For instance, Liu et al. [2023] and others have investi- gated how different activation functions or regularization penalties (e.g., 퐿 1 sparsity) can pressure the model into adopting a more aligned, interpretable internal basis. De- spite these advances, polysemanticity remains a pervasive challenge in state-of-the-art models. The current consensus in MI is that interpreting a model requires moving be- yond the “neuron-as-a-unit” paradigm and instead identifying the underlying sparse feature basis in which the model’s causal logic is actually computed. 21.5 The Logit Lens A fundamental intuition in mechanistic interpretability is that a transformer model does not compute its output in a single leap at the final layer. Instead, it iteratively refines a representation within the residual stream. The logit lens 1 provides a window into this process by decoding the hidden states of intermediate layers directly into the output vocabulary space. 21.5.1 Formal Mechanism Consider a transformer with 퐿 layers. Let 푥 0 be the initial embedding of a token, and ℎ ℓ be the state of the residual stream after layer ℓ ∈ 1,..., 퐿. In a standard forward pass, the final hidden state ℎ 퐿 is converted into logits 푦 via the layer norm (LN) and the unembedding matrix 푊 푈 ∈R 푑 푚표푑푒푙 ×|푉| : 푦 = Softmax(LN(ℎ 퐿 )푊 푈 )(21.3) The logit lens generalizes this by applying the same unembedding transformation to intermediate states ℎ ℓ : ˆ푦 ℓ = Softmax(LN(ℎ ℓ )푊 푈 )(21.4) where ˆ푦 ℓ represents the model’s tentative probability distribution at layer ℓ. By visu- alizing these distributions across all layers, researchers can observe the evolution of belief as the model moves from vague semantic clusters to specific token predictions. Direct Logit Attribution While the logit lens provides an aggregate view of the residual stream ℎ ℓ , we often wish to isolate the contribution of a single layer or head. Because the final logit 푦 is a linear function of the cumulative sum of layer outputs, 1 First introduced in the following blog post: https://w.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. An updated implementation is provided by [Wang, 2025]. 21.5. THE LOGIT LENS409 Input: “The” Layer 1 Layer 2 . . . Layer 퐿 Residual Stream Layer Norm + 푊 푈 + Softmax “cat” Final Model Prediction 푦 Logit Lens Layer Norm + 푊 푈 + Softmax “dog” Lens Output ˆ푦 ℓ Figure 21.3: Visualization of the logit lens. The residual stream is intercepted at an intermediate layer (e.g., Layer 2) and decoded using the model’s final unembedding matrix (푊 푈 ). This reveals the current belief of the model at that specific step, though it may be noisy in early layers. we can compute the direct logit attribution (DLA) of layer ℓ by projecting its specific output Δℎ ℓ directly into the vocabulary space: DLA(ℓ) = LN(Δℎ ℓ )푊 푈 (21.5) DLA allows researchers to determine which specific components are writing the correct answer into the stream, rather than just identifying where the answer becomes visible. In the study of factual recall, DLA often reveals that a single knowledge-producing layer provides the vast majority of the probability boost for the correct token. 21.5.2 From the Logit Lens to the Tuned Lens While the initial logit lens is highly informative for the final third of a network, it often yields gibberish or high-entropy distributions in early layers. This occurs because the residual stream at layer 2, for example, is not yet formatted for the output distribution; it may be storing low-level positional information that is mathematically orthogonal to the unembedding matrix 푊 푈 . To bridge this gap, Belrose et al. [2023] introduced the tuned lens. Instead of using the static 푊 푈 , the tuned lens trains a layer-specific affine transformation 휙 ℓ to map ℎ ℓ to the logit space: 푦 푡푢푛푒푑,ℓ = Softmax(휙 ℓ (ℎ ℓ ))(21.6) 410 CHAPTER 21. *MECHANISTIC INTERPRETABILITY FOR LLM AGENTS Input: “The” Layer 1 Layer 2 . . . Layer 퐿 Residual Stream Layer Norm + 푊 푈 + Softmax “cat” Final Model Prediction 푦 Tuned Lens Learned Affine Transform 휙 2 (ℎ 2 ) Softmax “cat” Tuned Output 푦 푡푢푛푒푑,2 Training Signal (Minimize KL Div.) Figure 21.4: Visualization of the Tuned Lens. Unlike the naive lens, this method uses a layer-specific, learned transformation (휙 ℓ ) to decode the residual stream. This trans- formation is trained to align the intermediate representation with the final output dis- tribution, providing a clearer picture of information emergence in early layers. The transformation 휙 ℓ is trained to minimize the KL-divergence between 푦 푡푢푛푒푑,ℓ and the model’s actual final output 푦. This effectively translates the internal language of early layers into the output vocabulary, revealing that factual information often emerges much earlier in the network than the naive logit lens would suggest. 21.5.3 Applications and Interpretations Lenses serve as a critical diagnostic tool for identifying early exit candidates and locat- ing specific knowledge neurons. For instance, if the logit for a specific fact (e.g., Paris) jumps from the 100th to the 1st rank at layer 14, it suggests that the attention heads or MLP sub-layers in layer 14 are responsible for the factual retrieval. Variants such as the attention lens [Sakarvadia et al., 2023] apply similar logic to the outputs of individual attention heads, allowing us to see what each head contributes to the prediction in isolation. Collectively, these tools shift our view of the transformer from a sequential pipeline to a cumulative iterative refiner of latent information. 21.6 Circuit Analysis A central objective of MI is the identification of circuits: minimal subgraphs of a model’s computation graph that are both necessary and sufficient to implement a spe- cific high-level function. Unlike behavioral analysis, circuit discovery seeks to explain how information is transformed step-by-step, from input embeddings to the final logit. 21.6. CIRCUIT ANALYSIS411 21.6.1 The Prototypical Circuit: Induction Heads The discovery of the induction head represents a landmark in transformer interpretabil- ity [Olsson et al., 2022]. Induction heads are a specialized circuit consisting of at least two attention heads that implement a prefix-matching algorithm. Formally, given a se- quence[퐴][퐵] ...[퐴], an induction head at the current position[퐴] attends to the pre- vious occurrence of [퐴] and copies the subsequent token [퐵] into the residual stream (Figure 21.5). The Mechanism of Composition To understand how the two heads in an induction circuit communicate, we must define the three types of attention composition [Elhage et al., 2021]: • V-Composition: The output (Value) of a previous head is used as the input to a subsequent head. • Q-Composition / K-Composition: The output of a previous head affects where a subsequent head looks by writing into the Query or Key vectors of a token. An induction head typically relies on K-Composition: a previous token head in an early layer writes information about the preceding token into the residual stream. The induction head in a later layer then uses this information in its Key vector to match the current prefix, allowing it to attend to the correct historical token [퐵]. Random Tokens Category40idsnodestruction ... Repeat of Random Tokens Category40idsnodestruction attention prefix of attended-to-token = current token Attended-to-token is copied. The corresponding logit is increased for the next token. Figure 21.5: Mechanics of an Induction Head Circuit. Induction heads implement a prefix-matching algorithm where the model identifies a previous occurrence of the cur- rent token (the prefix) and attends to the subsequent token to predict a repeat sequence. Olsson et al. [2022] observed that these heads emerge during a specific phase tran- sition in training, coinciding with a sharp increase in the model’s in-context learning capabilities. This was one of the first instances where a complex capability was traced to a specific, discrete algorithm implemented across multiple layers. 21.6.2 Methodologies for Circuit Discovery Manually identifying circuits is labor-intensive. The study of the indirect object iden- tification (IOI) task by Wang et al. [2022] remains the standard for manual tracing. By utilizing path patching, they identified a network of 26 attention heads (classified into 412 CHAPTER 21. *MECHANISTIC INTERPRETABILITY FOR LLM AGENTS roles such as Name Movers, S-Inhibition Heads, and Duplicate Token Detectors) that collectively perform coreference resolution. Coreference resolution refers to the pro- cess of identifying when different tokens in a text refer to the same underlying entity. For example, in the IOI task, the model must determine that a later mention of a name (the indirect object) refers back to a specific person introduced earlier in the sentence, a computation implemented by a specialized sub-network of attention heads. However, scaling these findings to frontier models requires automated circuit dis- covery (ACD). • ACD: Conmy et al. [2023] introduced an algorithm that performs recursive ab- lation to prune edges in the computation graph, preserving only those whose removal significantly degrades model performance on a specific task. • Sparse Feature Circuits: Moving beyond neurons, more recent frameworks in- tegrate sparse autoencoders (SAEs) to trace circuits in feature space rather than neuron space [Marks et al., 2024]. By treating SAE features as the nodes and the weights between them as edges, researchers can build attribution graphs that visualize the flow of human-understandable concepts (e.g., refusal, translation, or grammar checking). Evaluation: Faithfulness, Completeness, and Minimality It is vital to distinguish between a suggested circuit and a causal circuit. A circuit is considered faithful if the model’s performance is preserved when all components outside the circuit are ablated. It is complete if it captures all pathways the model uses to solve the task. As models scale, we find that algorithms are often represented redundantly or in a distributed manner across the residual stream. Therefore, circuit tracing is not merely a search for a single wire, but an attempt to map the causal mediation of information through the high-dimensional latent space of the model. 21.7 Activation Patching and Causal Interventions While observational tools such as the logit lens or probing can identify correlations between internal states and outputs, activation patching (also termed causal tracing or interchange interventions) allows researchers to make definitive causal claims. This technique is a practical application of causal mediation analysis [Vig et al., 2020], designed to isolate the effect of a specific model component on a downstream behavior (Figure 21.6). Formal Framework The procedure involves two distinct forward passes: 1. Clean Run (푥 푐푙푒푎푛 ): An input that produces the target behavior (e.g., The Eiffel Tower is in Paris). 2. Corrupted/Counterfactual Run (푥 푐표푟 ): A modified input where the target behav- ior disappears (e.g., The Colosseum is in Rome or adding Gaussian noise to the embedding of Eiffel Tower). 21.7. ACTIVATION PATCHING AND CAUSAL INTERVENTIONS413 Clean Input (푥 푐푙푒푎푛 ) Layer 1 Component 푧 푖,ℓ Layer 3 Correct Output Corrupted Input (푥 푐표푟 ) Layer 1 Ablated/Noisy Layer 3 Restored Output? Patch Activation Source of Information Target of Intervention Measure Change in Target Probability Figure 21.6: Activation Patching. Activations from a clean run (where the model suc- ceeds) are swapped into a corrupted run (where the model fails). By observing whether this interchange intervention restores the correct output, we can causally verify if the patched component is responsible for the specific reasoning task. Let 푧 푖,ℓ be the activation of component 푖 at layer ℓ. In activation patching, we com- pute a patched run by replacing the activation 푧 푐표푟 푖,ℓ with 푧 푐푙푒푎푛 푖,ℓ during the corrupted forward pass. We then measure the probability recovery: Effect = 푃(푦 푡푎푟푔푒푡 |patched)− 푃(푦 푡푎푟푔푒푡 |corr) 푃(푦 푡푎푟푔푒푡 |clean)− 푃(푦 푡푎푟푔푒푡 |corr) (21.7) An Effect score near 1.0 indicates that the patched component is a sufficient causal mediator for the information being studied. Locating Knowledge and Circuits Activation patching has led to significant archi- tectural insights. Meng et al. [2022] utilized massive patching sweeps to demonstrate that factual associations are often localized within specific MLP blocks in the middle layers of a model. They found that patching these knowledge neurons could force a model to output a specific fact even when the input context was corrupted. Beyond localization, patching is the primary tool for path patching, which identifies interactions between components. In the indirect object identification study mentioned before, Wang et al. [2022] patched specific attention heads to trace how information flows from duplicate token detectors to name movers. This reveals the computational graph of the model, moving beyond simple localization to understand the internal al- gorithm. 414 CHAPTER 21. *MECHANISTIC INTERPRETABILITY FOR LLM AGENTS Scalability and Attribution Patching A major challenge of activation patching is its computational cost, as a full sweep requires a forward pass for every component in the model. To mitigate this, researchers use attribution patching [Syed et al., 2024], which uses gradients to linearly approximate the effect of a patch. This allows for rapid hypothesis generation, where the most promising causal candidates are later validated with exact (but slower) activation patching. This surgery-like precision makes patching an indispensable part of the MI toolkit for moving from speculative interpretation to rigorous causal proof. 21.8 Sparse Autoencoders: Dictionary Learning for Fea- ture Discovery As discussed in the context of superposition, neural networks often represent more features than they have dimensions (푛 > 푑). To resolve this tangled representation, we utilize sparse autoencoders (SAEs), a form of unsupervised dictionary learning designed to project activations into a higher-dimensional sparse latent space where features are approximately monosemantic. 푥 1 푥 2 푥 3 . . . 푥 푑 Input Activation 푥 ∈R 푑 Low Dim 푑 푐 3 푐 6 . . . Sparse Latent Features 푐 ∈R 푁 High Dim 푁 (푁 ≫ 푑) Mostly Zero (Sparse) ˆ푥 1 ˆ푥 2 ˆ푥 3 . . . ˆ푥 푑 Reconstruction ˆ푥 ∈R 푑 Encoder 푊 푒푛푐 + Sparsity (e.g., ReLU) Decoder 푊 푑푒푐 Reconstruction Loss: ∥푥− ˆ푥∥ 2 Sparsity Penalty: +휆∥푐∥ 1 Figure 21.7: Architecture of a Sparse Autoencoder.The model projects lower- dimensional input activations (푥 ∈R 푑 ) into a much higher-dimensional latent space (푐 ∈R 푁 , where 푁 ≫ 푑). A sparsity penalty (e.g., 퐿 1 regularization on 푐) forces most hidden units to be inactive (grey), resulting in a sparse set of active, interpretable fea- tures (teal) that are used to reconstruct the original input. 21.8. SPARSE AUTOENCODERS: DICTIONARY LEARNING FOR FEATURE DISCOVERY415 Formal Architecture and Training Objective An SAE consists of an encoder that maps a model activation 푥 ∈R 푑 to a high-dimensional hidden state 푐 ∈R 푁 (where 푁 ≫ 푑), and a decoder that attempts to reconstruct the original activation (Figure 21.7). The encoder is typically defined as: 푐 = ReLU(푊 푒푛푐 (푥− 푏 푑푒푐 )+ 푏 푒푛푐 )(21.8) And the reconstruction is given by: ˆ푥 = 푊 푑푒푐 푐+ 푏 푑푒푐 (21.9) To ensure the model learns a sparse dictionary rather than a simple identity map- ping, we optimize a loss function combining reconstruction fidelity with a sparsity- inducing penalty: L = ∥푥− ˆ푥∥ 2 + 휆∥푐∥ 1 (21.10) where 휆 controls the trade-off between reconstruction accuracy and feature sparsity. Recent advancements, such as JumpReLU SAEs [Rajamanoharan et al., 2024], utilize a thresholded activation function to achieve near-binary on-off feature behavior, which more closely aligns with the theoretical ideal of discrete semantic features. 21.8.1 Monosemanticity and Shadow Models The efficacy of SAEs has been validated across various scales. Cunningham et al. [2023] demonstrated that SAEs could recover latent features in GPT-2 representing specific grammatical concepts (e.g., numerical context), while Gao et al. [2024] suc- cessfully scaled this to billion-parameter models, identifying features for programming languages and sentiment. A profound implication of this work is the ability to construct a shadow model: by replacing a model’s polysemantic MLP layers with a sparse feature-based approxi- mation, we can preserve model performance while rendering the internal computation legible. Bricken et al. [2023] showed that these SAE features are significantly more monosemantic than the original neurons, effectively untangling the superposed states into a basis that humans can reason about. In the context of circuit tracing, researchers have moved beyond simple autoen- coders to transcoders [Dunefsky et al., 2024]. While a standard SAE reconstructs a single layer’s activations (푥 → ˆ푥), a transcoder is trained to map activations from layer ℓ to the output of layer ℓ (e.g., 푥 → MLP(푥)). This forces the latent features 푐 to represent the actual computational transformation performed by the layer, providing a direct map of how information is processed rather than just how it is stored. Evaluation and Open Challenges Evaluating SAEs requires metrics that balance faithfulness (how much performance is lost when using features?) and interpretability (can humans or automated LLM-probes reliably describe the feature?). Despite the success of SAEs, challenges such as feature splitting, where a single semantic concept is divided across multiple redundant features, and the difficulty of interpreting very rare features remain active areas of research. 416 CHAPTER 21. *MECHANISTIC INTERPRETABILITY FOR LLM AGENTS 21.9 Concluding Remarks 21.9.1 Outlook and Emerging Directions Mechanistic interpretability has transitioned from a niche curiosity to a rigorous dis- cipline that seeks to bridge the gap between statistical performance and scientific un- derstanding. As the field matures, several frontier directions are defining the research agenda for the next decade. The most pressing challenge remains the interpretability labor bottleneck. While manual circuit discovery on GPT-2 was a breakthrough, the sheer number of features in frontier models like GPT-4 or Claude 3 necessitates automated workflows. Automated interpretability [Bills et al., 2023] utilizes LLMs to generate and test hypotheses about the semantic meaning of millions of latent features discovered via SAEs. This recursive loop—using a model to interpret its own predecessors—is likely the only path toward the complete transparency of trillion-parameter systems. To move beyond just-so stories, the field is adopting more rigorous validation frameworks. Causal scrubbing [Chan et al., 2022] represents the current state-of- the-art for hypothesis testing. By systematically replacing components of a proposed circuit with their expected values under a specific hypothesis and measuring the per- formance drop, researchers can quantitatively measure the faithfulness of their expla- nations. This moves MI closer to a formal verification science, where one can prove that a model uses a specific algorithm. Mechanistic interpretability is increasingly viewed as a prerequisite for robust AI alignment. By identifying the circuits responsible for deceptive alignment or situ- ational awareness, researchers hope to build safety buffers that detect if a model is merely producing a user-pleasing answer while internally calculating a different, po- tentially harmful outcome. Furthermore, understanding the training dynamics, such as the grokking phenomenon where a model suddenly moves from memorization to gen- eralization [Nanda et al., 2023], allows us to intervene during the training process to steer models toward more interpretable and safe representations. While this chapter focused on language models, the principles of MI are being rapidly extended to other domains, for example to time-series transformers [Kaln ̄ are et al., 2026]. Time-series transformers present unique challenges, as the tokens lack discrete human semantics, requiring the development of new dictionary learning tech- niques for continuous signals. Similarly, interpreting vision transformers and multi- agent reinforcement learning systems will require a synthesis of spatial, temporal and social interpretability lenses. 21.9.2 Further Reading The journey from post-hoc behavioral observation to mechanistic circuit diagrams rep- resents a paradigm shift in AI research. We are no longer satisfied with models that merely work; we demand to know how they work. By combining the microscope of sparse autoencoders with the scalpel of activation (and attention) patching, the com- munity is building a comprehensive map of the transformer’s mind, one circuit at a time. BIBLIOGRAPHY417 - Further Reading For those interested in the foundational research that defined this field, we highly recommend the Transformer Circuits Thread a . a https://transformer-circuits.pub/ Bibliography Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. 404 Ameen Ali, Thomas Schnake, Oliver Eberle, Gr ́ egoire Montavon, Klaus-Robert M ̈ uller, and Lior Wolf. Xai for transformers: Better explanations through conserva- tive propagation. In International conference on machine learning, pages 435–451. PMLR, 2022. 403 Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Com- putational Linguistics, 48(1):207–219, 2022. 404 Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McK- inney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from trans- formers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023. 409 Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082, 2024. 405 Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core. windows.net/neuron-explainer/paper/index.html, 2023. 416 Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ram ́ on Guevara, Beren Millidge, Gabriel Alfour, et al. Interpreting neural networks through the polytope lens. arXiv preprint arXiv:2211.12312, 2022. 405 Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Trans- former Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic- features/index.html. 407, 415 LawrenceChan,Adri ` aGarriga-Alonso,NicholasGoldwosky-Dill,Ryan Greenblatt,Jenny Nitishinskaya,Ansh Radhakrishnan,Buck Shlegeris, 418BIBLIOGRAPHY and Nate Thomas.Causal scrubbing,a method for rigorously test- ing interpretability hypotheses.AI Alignment Forum, 2022. https: //w.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/ causal-scrubbing-a-method-for-rigorously-testing. 416 Xi Chen, Aske Plaat, and Niki van Stein. How does chain of thought think? mech- anistic interpretability of chain-of-thought reasoning with sparse autoencoding. In AAAI2026, arXiv preprint arXiv:2507.22928, 2026. 405 Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri ` a Garriga-Alonso. Towards automated circuit discovery for mechanistic inter- pretability. Advances in Neural Information Processing Systems, 36:16318–16352, 2023. 405, 412 Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. 405, 415 Austin L Davis and Gita Sukthankar. Decoding chess mastery: A mechanistic analysis of a chess language transformer model. In International Conference on Artificial General Intelligence, pages 63–72. Springer, 2024. 405 Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. Advances in Neural Information Processing Systems, 37:24375– 24410, 2024. 415 Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021. 406, 411 Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. 407 Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-juss ` a. A primer on the inner workings of transformer-based language models.arXiv preprint arXiv:2405.00208, 2024. 405 Leo Gao, Tom Dupr ́ e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoen- coders. arXiv preprint arXiv:2406.04093, 2024. 415 Qi Huang, Sofoklis Kitharidis, Thomas B ̈ ack, and Niki van Stein. Tx-gen: Multi- objective optimization for sparse counterfactual explanations for time-series clas- sification. In Proceedings of the 1st International Conference on Explainable AI for Neural and Symbolic Methods - Volume 1: EXPLAINS, pages 62–74. INSTICC, SciTePress, 2024. ISBN 978-989-758-720-7. doi: 10.5220/0013066400003886. 404 BIBLIOGRAPHY419 Mat ̄ ıs Kaln ̄ are, Sofoklis Kitharidis, Thomas B ̈ ack, and Niki van Stein. Mechanistic interpretability for transformer-based time series classification. In Francesco Mar- celloni, Kurosh Madani, Niki van Stein, and Joaquim Filipe, editors, Computational Intelligence, pages 249–270, Cham, 2026. Springer Nature Switzerland. ISBN 978- 3-032-15638-9. 416 Amir-Hossein Karimi, Gilles Barthe, Borja Balle, and Isabel Valera. Model-agnostic counterfactual explanations for consequential decisions. In International conference on artificial intelligence and statistics, pages 895–905. PMLR, 2020. 404 Teuvo Kohonen. Self-organized formation of topologically correct feature maps. Bio- logical cybernetics, 43(1):59–69, 1982. 404 Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hos- mer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zim- merman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Bat- son.On the biology of a large language model.Transformer Circuits Thread, 2025.URL https://transformer-circuits.pub/2025/ attribution-graphs/biology.html. 405 Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. Entropy, 26(1):41, 2023. 408 Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017. 404 Aleksandar Makelov, George Lange, and Neel Nanda.Towards principled eval- uations of sparse autoencoders for interpretability and control.arXiv preprint arXiv:2405.08366, 2024. 405 Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024. 412 Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35: 17359–17372, 2022. 413 Dang Minh, H Xiang Wang, Y Fen Li, and Tan N Nguyen. Explainable artificial intelligence: a comprehensive review. Artificial Intelligence Review, pages 1–66, 2022. 403 C. Molnar. Interpretable Machine Learning. Leanpub, 2020. ISBN 9780244768522. URL https://books.google.nl/books?id=jBm3DwAAQBAJ. 404 420BIBLIOGRAPHY Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023. 405, 416 Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. 411 Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical re- view of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646, 2024. 405 Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, J ́ anos Kram ́ ar, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024. 415 Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016. 404 Shaoqing Ren, Kaiming He, Ross Girshick, Xiangyu Zhang, and Jian Sun. Object detection networks on convolutional feature maps. IEEE transactions on pattern analysis and machine intelligence, 39(7):1476–1481, 2016. 404 Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.” why should i trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016. 404 Thiago Rios, Bas van Stein, Stefan Menzel, Thomas Back, Bernhard Sendhoff, and Pa- tricia Wollstadt. Feature visualization for 3d point cloud autoencoders. In 2020 In- ternational Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE, 2020. 403 Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, Andr ́ e Bauer, Kyle Chard, and Ian Foster. Attention lens: A tool for mechanisti- cally interpreting the attention head information retrieval mechanism. arXiv preprint arXiv:2310.16270, 2023. 410 Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017. 403 Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bush- naq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al.Open problems in mechanistic interpretability.arXiv preprint arXiv:2501.16496, 2025. 405 BIBLIOGRAPHY421 Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms auto- mated circuit discovery. In Proceedings of the 7th BlackboxNLP Workshop: Analyz- ing and Interpreting Neural Networks for NLP, pages 407–416, 2024. 414 Bas Van Stein, Elena Raponi, Zahra Sadeghi, Niek Bouman, Roeland CHJ Van Ham, and Thomas B ̈ ack. A comparison of global sensitivity analysis methods for explain- able ai with an application in genomic prediction. IEEE Access, 10:103364–103381, 2022. 403 Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388– 12401, 2020. 412 Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Stein- hardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. 411, 413 Zhenyu Wang. Logitlens4llms: Extending logit lens analysis to modern large language models, 2025. URL https://arxiv.org/abs/2503.11667. 408 422BIBLIOGRAPHY Chapter 22 *Cognitive Perspectives on Agentic LLMs What You Will Learn In this chapter you will learn what can and cannot currently be said about the cognitive status of LLMs and agentic LLMs. Specifically, you will be able to: • Distinguish the levels of behavior, internal representations, and mecha- nisms. • Characterize the debate on whether LLMs understand linguistic content. • Evaluate the evidence for and against world models and spatial representa- tions in LLMs, and explain why this matters for agents that plan and act. • Explain what is known about the representations underlying Theory of Mind performance, and how this relates to the knowing-doing gap. • Describe the debate about LLM metacognition and introspection and explain why self-reports are weak evidence. • Outline how scientific theories of consciousness are applied to LLMs, and reflect on the risks of both over- and under-attributing mental states. • Discuss how agentic extensions (memory, tools, multi-modality, action) af- fect each of these questions in theory, given that little evidence is currently available. 22.1 From Behavior to Cognition In Part V we have seen many examples of LLMs and agentic LLMs performing on a par with humans or acting in human-like ways. For instance, models passed false- 423 424CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS belief tests, mirrored aggregate human moral judgments, or gave empathetic responses. Yet we also saw that such behavior is often brittle (small perturbations can lead mod- els astray), and what models say in text is sometimes badly aligned with what they do as agents (the knowing-doing gap). At several points we postponed the questions that naturally follow: Does passing a false-belief test mean a model has Theory of Mind (Section 15.5)? Does a model that responds in an empathetic way feel anything (Sec- tion 15.3.3)? What does it mean for an agentic LLM to reflect on its own or another agent’s behavior (Section 16.3)? And how do any of these compare to the underlying capacities in humans or other animals? In this chapter we discuss these questions without taking a stance in the sometimes heated debate. We focus no longer on what a model says or does, but on whether the underlying representations or processes deserve the names we use for human cogni- tion: understanding, thinking, world model, Theory of Mind, consciousness, and so on. As such, the questions underlying this chapter go back to the early days of AI as a field: Alan Turing famously suggested a behavioral test to answer the question whether machines can think [Turing, 1950] and John Searle argued that no amount of behavioral success is ever enough to settle the debate [Searle, 1980]. What is new, firstly, is that current LLMs can pass the Turing Test [Jones and Bergen, 2025, Gibney, 2025] and, as we have seen throughout Part V, a broad array of other behavioral tests that are associated with sophisticated cognitive functioning in humans. That makes the debate less hypothetical than in Turing’s and Searle’s times. Secondly, interpretabil- ity techniques (Chapter 21) enables us to complement behavioral data with evidence about internal representations (what information is encoded) and mechanisms (how it is computed and used), opening new opportunities for cognitive research that informs our understanding of both machines and humans. Before getting started, conceptual clarifications are needed regarding the levels at which evidence for cognitive claims is situated, the distinction between competence and performance, and the unit of analysis for each cognitive skill. 22.1.1 Three Levels of Evidence For the questions of this chapter we distinguish three levels at which we can gather evidence for claims about machine cognition: the behavioral level (testing what the model does while treating it as a black box), the representational level (what infor- mation its activations encode), and the mechanistic level (whether that information is causally used). 1 Evidence gathered at each level has its advantages and disadvantages, calling for an approach in which insights from different levels are integrated. As an illustration we use the false-belief test of Section 15.5 in which Maxi’s chocolate is moved from the blue to the green cupboard in his absence. 1 Note that Chapter 21 defines four interpretability paradigms: behavioral, attributional, concept-based, and mechanistic. The first two paradigms are collapsed into the behavioral level, while the third and fourth paradigm align with the representational and mechanistic levels respectively. Note further that Marr’s com- putational, algorithmic, and implementational levels of analysis [Marr, 2010] are similar but not identical to our division here. 22.1. FROM BEHAVIOR TO COGNITION425 The behavioral level At the behavioral level, we do the test and score the answer (as in Part V). Advantages include that this works with any system (including base- and fine-tuned model variants, open and closed LLMs, agentic or not) and that test outcomes can in principle be directly compared to human scores. The main disadvan- tage is that the same test answer can just as well result from genuine belief-tracking as from memorization (test items plus answers being in the training data) or surface heuristics and spurious correlations [for discussion see Hu et al., 2025, Kouwenhoven et al., 2026]. Perturbing test formats, for example by describing the cupboard doors as transparent, narrow down the options, but the evidence remains indirect. The representational level At the representational level, we ask what information is encoded in the model. The main method is linear probing: a small classifier is trained to predict a property from the activations at a given layer (cf. Chapter 21 and Alain and Bengio [2016], Belinkov [2022]). In this way the presence or absence of certain concepts can be verified. In the case of the false-belief test we can ask if we can decode where Maxi thinks the chocolate is, separately from where it is actually located. If so, the model encodes the key distinction on which the test hinges. Two cautions apply, as will be discussed in more detail: (1) a sufficiently powerful probe may learn the task itself (rather than decoding it from the model; [Hewitt and Liang, 2019]), and (2) information that can be decoded from a model may not be used by it. The mechanistic level At the mechanistic level that latter caution is addressed by ob- serving the effect of different interventions on model behavior. Examples are ablation, patching activations from one run into another, or steering activations along a direction found by a probe (Chapter 21). For example, if shifting Maxi’s encoded belief from blue to green makes the model answer that he will look in the green cupboard, the representation is likely to be causally involved in the model’s behavior. The strongest evidence for cognitive claims comes from this third level. However, as we will see, such evidence is currently scarce in the literature. One reason is that it requires access to weights, restricting research mostly to small models and narrow tasks. 2 In this chapter we thus get a more elaborate picture of how construct validity can be achieved. When a model’s behavior is robust under perturbation, the relevant variables are represented internally. When those representations are then causally used, we have a high degree of support for the presence and validity of a capability. Table 22.1 sum- marizes the evidence for the (presumed) capabilities discussed in this chapter. What stands out is that evidence on the behavioral level dominates everywhere. For bare LLMs, representational and mechanistic evidence are strongest for world models, but only in a limited domains (such as game board representation). For agentic LLMs, representational and mechanistic evidence is mostly absent alltogether at this moment. 2 An exception is when developers of frontier models do this kind of research themselves [as in Templeton et al., 2024, Lindsey et al., 2025]. Though insightful, this poses questions of independent replicability. 426CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS BehavioralRepresentationalMechanistic Linguisticunderstanding (Sect. 22.2) Extensive. Benchmarks; perturbed and counterfactual task variants [Wu et al., 2024] Growing.Conceptual spaces for color and direction recovered from text [Abdou et al., 2021, Patel and Pavlick, 2022] Sparse.Circuits for narrow linguistic tasks [Wang et al., 2022] World models (Sect. 22.3)Growing.Navigation, planning, and game tasks [Momennejad et al., 2023b, Paglieri et al., 2025] Growing. Board state, space, and time decodable [Li et al., 2023, Gurnee and Tegmark, 2024] Growing.Editing the board represen- tation changes moves [Li et al., 2023, Nanda et al., 2023] Theory of Mind (Sect. 22.4)Extensive. See Section 15.5Sparse. Beliefs of self and other decod- able [Zhu et al., 2024, Bortoletto et al., 2024] Sparse. Steering belief representations al- ters answers [Zhu et al., 2024] Self-knowledge (Sect. 22.5)Growing.Calibration, self-prediction [Kadavath et al., 2022, Binder et al., 2025] Sparse. Truthfulness directions [Azaria and Mitchell, 2023, Burns et al., 2023] Sparse. Concept injection; faithfulness tests [Lindsey, 2025, Turpin et al., 2023] Consciousness (Sect. 22.6)Sparse.Self-reports, generally consid- ered weak evidence [Perez and Long, 2023b] Hardly. No agreed measure. Architec- tures assessed against indicator proper- ties derived from theories of conscious- ness [Butlin et al., 2023] Sparse. Ablation of J-space activations making a case for its functional involve- ment in higher-order tasks [Gurnee et al., 2026] (a) Bare LLMs BehavioralRepresentationalMechanistic Linguisticunderstanding (Sect. 22.2) None.— World models (Sect. 22.3)Sparse. LLM as simulator inside a plan- ner [Hao et al., 2023, Gu et al., 2024] — Theory of Mind (Sect. 22.4)Sparse. Scaffolds in coordination games (Chapter 16); no interventions during in- teraction — Self-knowledge (Sect. 22.5)Sparse.Unverified completion reports [Shapira et al., 2026]; test-vs-deployment detection [Needham et al., 2025] — Consciousness (Sect. 22.6)Hardly. Self-reports and task preferences in agentic settings [Gray et al., 2025] Hardly. Language-agent architectures as- sessed against global workspace theory [Goldstein and Kirk-Giannini, 2024] — (b) Agentic LLMs, studied in interaction Table 22.1: Sources of evidence for the topics of this chapter at the behavioral, rep- resentational, and mechanistic level, for (1) bare LLMs and (2) agentic LLMs. The amount of available work is indicated along with a few representative studies. For agentic LLMs, evidence available at the time of writing is overall sparse. 22.1. FROM BEHAVIOR TO COGNITION427 22.1.2 Competence and Performance Successful completion of a single test does not establish the presence of a capacity, and neither does failure prove its absence. In the cognitive sciences a distinction is made between competence, the underlying capacity, and performance, how that capacity is expressed on a particular occasion and under particular constraints [Chomsky, 1965]. Firestone [2020] argues that comparisons between humans and machines should re- spect this distinction. For example, a child may fail a false-belief test because she does not understand how the question is worded. In a similar way, LLM performance can depend on factors such as prompt format, tokenization, context length, and output constraints. A base model that cannot respond in the format required in one particular false-belief test may nonetheless show the relevant competence in the probabilities it assigns to alternative completions [Kouwenhoven et al., 2026]. Comparisons can also be uneven, for example when human participants receive instructions, practice trials, and attention checks, whereas models are tested zero-shot [Lampinen, 2024]. The competence-performance distinction helps us to avoid two opposite errors. On one side is anthropomorphism: reading human-like tendencies into model’s output and actions too easily. As the example of the CASA paradigm showed (Section 15.3.2), this tendency is deeply ingrained also among researchers. On the other side is what primatologist Frans de Waal has characterized as anthropodenial [de Waal, 1999] or what Milli ` ere and Rathkopf [2026] call anthropocentric bias: missing or dismissing a capacity because it is not achieved in a recognizably human way, or holding models to standards that humans would not meet either. 22.1.3 The Unit of Analysis: Model, System, Environment? A returning question is what counts as the unit of analysis: a capacity that is absent in a bare LLM may arise at the level of the system once an agentic harness is added or the model is placed in a multi-agent environment. In Chapter 16 we discussed an example where explicit reasoning and verification steps turned failed coordination in Hanabi into reasonable play. Philosophers and cognitive scientists have made a similar point about humans: cog- nition is not what happens in the bare brain. Instead, the 4-E view of cognition was adopted [No ̈ e, 2009, Carney, 2020]: 1. Embodied: cognition is shaped by the entire physical body and its sensorimotor capacities, not just neural activity in the brain; 2. Embedded: mental processes are situated within and constrained by specific en- vironments, each with their physical, social, and cultural properties; 3. Enactive: perception and thought emerge through dynamic, real-time interac- tions with the world, rather than passive internal calculations; 4. Extended: cognitive processes extend into the environment to incorporate exter- nal tools, technology, and aids like notebooks or calculators as active parts of thinking. 428CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS According to the extended view on cognition, a notebook that is reliably consulted can play the same functional role as biological memory, and should count as part of the cognitive process and system [Clark and Chalmers, 1998]. Compare this to the distinction between bare and agentic LLMs relevant the questions in this chapter. In a bare model, working memory is the context window, while long-term knowledge is stored in the weights, where it can only be changed by (further) training. Agentic harnesses add external storage much like a notebook: logs of past interactions, facts, and data that are retrieved when relevant. Throughout the chapter, this will be relevant since different studies target different units of analysis when they speak about cognition. We will take this into account when discussing each of the cognitive capabilities selected for this chapter: understanding linguistic content (Section 22.2), world models of the spatial (Section 22.3) and so- cial (Section 22.4) environment, introspection and self-knowledge (Section 22.5), and finally, the most speculative topic, consciousness (Section 22.6). 22.2 Understanding Linguistic Content Chapter 15 showed that LLMs resolve many conversational implicatures: they can cor- rectly infer, for example, that a colleague who is praised only for being always on time may not be held in particularly high regard, since what needs to be read between the lines is something like the only positive thing I can think of in their case is punctual- ity. Can a model that can draw this inference be said to understand what it reads and writes? On this question, the field remains deeply divided. 22.2.1 The Skeptical Case The classic argument can be found in Searle’s Chinese Room thought experiment [Searle, 1980]. A person who knows no Chinese sits in a room with a rule book for manipulating Chinese symbols. Questions are passed in, the rules are followed, and fluent answers are passed out, yet the person understands nothing. 3 Searle concludes that symbol manipulation (syntax) is not sufficient for meaning (semantics). Harnad [1990] reformulates this as the symbol-grounding problem: a system in which symbols are defined only in terms of other symbols is like the situation where someone without prior knowledge of the language is trying to learn Chinese from a Chinese–Chinese dic- tionary. At some point, to gain meaning, symbols must connect to something language- external, such as perceptions and interactions with the physical or social world. Bender and Koller [2020] applied this reasoning to LLMs in their octopus test. Two people stranded on separate islands communicate through an underwater cable. A hyper-intelligent octopus taps the cable, learns the statistical patterns in their messages, and eventually impersonates one of them. The deception works for small talk, but breaks down when the other islander asks for help with defending against a bear. The octopus has never had access to the world to which the words refer. In the authors’ 3 Today we find the formulation of this thought experiment tendentious: the framing of the Chinese language, with 1.3 billion native speakers world wide, as unintelligible in Searle’s article implies a strong Western bias that would call for a reflection, at minimum. 22.2. UNDERSTANDING LINGUISTIC CONTENT429 view, a system trained on linguistic form alone cannot learn meaning, no matter how large it becomes. In the same spirit, Bender et al. [2021] describe LLMs as stochastic parrots that glue together linguistic forms on the basis of probabilistic patterns, without any reference to meaning. 22.2.2 Replies to the Skeptical Case Three kinds of reply have been given to the skeptical case. First, meaning may not require reference in the first place. Piantadosi and Hill [2022] defend a conceptual role view, in which the meaning of a concept lies largely in its relations to other concepts. Many human concepts (justice, prime number, quark) are not directly grounded in per- ception either, and the relations between concepts that give meaning to such words are exactly what can be learned from text. Second, reference may not require grounding in the individual speaker. We can refer to Peano or to the Andromeda galaxy without any perceptual contact, because we belong to a linguistic community in which these words have a history of use. Mandelkern and Linzen [2024] argue that LLMs, trained on the products of that community, may inherit reference in much the same way. Third, Pavlick [2023] argues that the question is ultimately empirical: whether LLM repre- sentations have the structure that grounding is supposed to provide should be settled by studying those representations (the representational column of Table 22.1), rather than by thought experiment alone. Mitchell and Krakauer [2023] summarize and synthesize the debate and note that the field is split: in a 2022 survey among NLP researchers, about half agreed that a model trained on text alone could understand language in a non-trivial sense [Michael et al., 2022]. They suggest that the question may have no single answer, and that LLMs force us to distinguish between different kinds of understanding. 22.2.3 Evidence Various sources of evidence have been presented in favor of the view that concepts can be grounded based on learning from text at scale alone. Other results support the opposite view. Structure that mirrors the world Abdou et al. [2021] show that the geometry of color-term representations in text-only models aligns with the geometry of human color perception, even though these models have never seen a color. Patel and Pavlick [2022] present GPT-3 with a few examples of direction terms (left, north) in a grid world rendered as text. The model generalizes to unseen terms, also when the world is rotated, but not when the mapping is scrambled. This indicates that the conceptual structure learned from text is isomorphic to that of the grounded domain. Signs of shallowness Other results show that such structure can be present in ways that differ from human understanding. Models trained on A is B often fail to infer B is A: they can name a celebrity’s parent but not the celebrity when given the parent, a failure known as the reversal curse [Berglund et al., 2024]. The failure concerns knowledge 430CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS stored in the weights. When both facts are supplied in the context window, models reverse them without difficulty, so the curse reflects how facts are stored during training rather than an inability to grasp that the relation is symmetric. McCoy et al. [2024a] show that accuracy depends on how probable the answer is as a piece of text: GPT-4 decodes a simple code correctly in 51% of cases when the solution is a high-probability sentence, but in only 13% when it is a low-probability one. Humans are not immune to probability effects either, as anyone who has overlooked a typo in a predictable sentence knows. But a gap of this size is much larger than we would expect if the task were understood in a robust sense. Reasoning models perform much better on such tasks, but their accuracy and the length of reasoning still depend on the probability of the output [McCoy et al., 2024b]. Similarly, performance drops dramatically on counterfactual variants of familiar tasks, such as arithmetic in base 9 [Wu et al., 2024]. Finally, Mancoridis et al. [2025] describe Potemkin understanding, named after film set villages that exist of only facades without the actual houses behind them. This is similar to the knowing-doing gap, described in Part V, now appearing within language itself: models have been observed to correctly define a concept such as an ABAB rhyme scheme, and then fail to apply it in ways that no human who can give the definition would. Together, these results fit the distinction between formal and functional competence [Mahowald et al., 2023]: near-human mastery of form, combined with understanding- in-use that is real but uneven and tied to the statistics of the training data. 22.2.4 The Agentic Perspective The octopus could only tap into a cable and pick up signals from it. Multi-modal LLMs receive images and audio and agentic LLMs can act and interact. An agent that writes code and observes whether or not it fails when running it, or deletes a file and can no longer access it, receives feedback from the world on its use of words such as file and delete. This provides the kind of connection between perception and action that skeptics asked for, and it weakens the premise of the octopus argument. However, the feedback reaches the model mostly as text in its context window, and reaches its weights only if agentic trajectories are used for further training. It also comes without physical stakes: a model cannot “die” or “hurt itself” because of its actions, at least not in any meaningful sense, yet. Whether agentic embedding actually changes the representations discussed above has, to our knowledge, not been tested. 22.2.5 Conclusion Neither stochastic parrot nor full understander fits the current evidence. Text-only training yields conceptual structure that mirrors aspects of the world, but that structure is deployed in ways that remain sensitive to surface statistics. The philosophical dispute is partly a dispute about words; the empirical questions of how robust the structure is, and whether action improves it, remain open. 22.3. WORLD MODELS AND SPATIAL COGNITION431 22.3 World Models and Spatial Cognition In nature, acting well requires predicting the consequences of actions before taking them. This applies no less to AI agents that act in the world, especially in the case of irreversible actions such as the destructive ones reported in Section 15.6. The idea that intelligent behavior relies on an internal model of the world that can be used to try out alternatives before acting, goes back to the early days of psychology as a science: Craik [1943] proposed that organisms carry a small-scale model of external reality in their heads for this reason. Tolman [1948] showed that rats form cognitive maps: after having learned a maze, they can take shortcuts they have never walked. A parallel can be drawn with reinforcement learning, where a world model is a learned model of the environment’s dynamics that allows planning [Ha and Schmidhuber, 2018]. Whether LLMs have such models is contested. According to LeCun et al. [2022], autoregressive LLMs lack them in principle, which is why he proposes a different architecture for future generative models. From a vast body of literature that has emerged on this topic over the past years, we review several representative studies, primarily focused on spatial cognition, providing evidence for and against the existence of world models in LLMs. 22.3.1 Evidence For The best-known case is Othello-GPT [Li et al., 2023], illustrated in Figure 22.1. A small transformer was trained only to predict legal moves in sequences of Othello moves. The model was never told the rules or shown a board. Nevertheless, probes could recover the full board state from its activations, and when the authors edited this internal board, the model predicted moves that were legal for the edited board rather than the actual one. The probes in this study were nonlinear, however, so the first cau- tion of Section 22.1.1 applies: perhaps the probe computed the board, rather than the model. Nanda et al. [2023] resolved this issue. The board state turned out to be linearly decodable, and editable along linear directions with the same effect on the moves, but only when squares were coded as mine or yours rather than black or white. With this qualification, the case provides evidence at all three levels. A broader lesson is also that a model’s variables need not be the ones we expect. Outside the specific context of a game board, Gurnee and Tegmark [2024] find linear representations of space and time in Llama-2: the coordinates of cities and land- marks, and the dates of historical figures and events, can be decoded from activations robustly across prompts. Regarding planning ahead, there is some evidence for a limited form of Craik’s trying out alternatives before acting in attribution-graph analyses (Chapter 21) of a production model, Claude 3.5 Haiku [Lindsey et al., 2025]. When writing a rhyming couplet, the model activates features for candidate rhyme words at the end of the first line, before it writes the second. Interventions that suppressed the planned word, or injected a different word, changed the second line. The model thus represents a goal state ahead of time and uses it to shape what it produces in between. Such planning over the span of a single line provides evidence against the view that a next-token predictor can only look one token ahead. This is potentially relevant in the context of 432CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS Move sequence F5 D6 C3 ... Transformer- layers 1...ℓ Transformer- layers ℓ+1... 퐿 Predicted- legal moves ℎ ℓ (1) behavioral decoded boardedited board (2) representational: probe edit (3) mechanistic: patch edited ℎ ℓ Figure 22.1: Schematic of the Othello-GPT experiments [Li et al., 2023, Nanda et al., 2023], which provide evidence at all three levels of Section 22.1.1. (1) The model, trained only on move sequences, predicts legal moves. (2) A probe decodes the board state from the activations ℎ ℓ at an intermediate layer. (3) When the decoded board is edited (here one piece is flipped) and the activations are changed accordingly, the model predicts moves that are legal for the edited board. The board is drawn as 4× 4 for readability; the real board is 8× 8. spatial planning as well, but future work will have to determine whether to what degree models actually do this. 22.3.2 Evidence Against Other studies indicate what the limitations of LLM’s abilities to represent s are. Co- gEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in LLMs, tests models for routes, shortcuts, and detours [Momennejad et al., 2023a]. Frontier models at the time, including GPT-4, failed systematically: they hal- lucinated route options and got caught in loops. On EWoK, a test for basic physical and social world knowledge with minimal pairs, LLMs stayed below human performance [Ivanova et al., 2025]. Citetyamada2024spatial show that accuracy varies strongly with the type of structure that needs to be navigated. When grids, hexagonal and triangular grids, rings, or trees are described in text and models need to navigate them, LLMs seem to capture certain spatial aspects well, while failing on others. Adding vision does not seem to help: vision-language models (VLMs) struggle with relations as ba- sic as left of and under [Kamath et al., 2023], in line with the limited benefit of the visual channel for Theory of Mind (Section 15.5). An extensive recent study shows that many of the documented issues with VLMs persist today [Wang et al., 2026]. Generative models trained on taxi rides in Manhattan, were able to predict the next turn almost perfectly on any route. Yet when perfectly valid though unlikely detours were introduced, performance collapsed. When researchers reconstructed the street map from their predictions, it was full of impossible streets and flyovers [Vafa et al., 2024]. This shows that next-token prediction accuracy can overstate how coherent a 22.4. SOCIAL COGNITION AND THEORY OF MIND433 model’s world model actually is. In a follow-up study, Vafa et al. [2025] showed that models trained on planetary orbits were able to predict trajectories accurately, but the implicit force laws that they extracted did not, like Newton’s, generalize well to new physics tasks. Along similar lines, Nikankin et al. [2025] show that LLMs perform arithmatic with a bag of heuristics rather than with algorithms and laws. They found evidence for the existence of many local representations that each handle a narrow numerical pattern. 22.3.3 The Agentic Perspective The system-level view suggests a way forward. An agent does not need to carry the world model entirely in its activations: it can maintain an explicit map, a notes file, or a program that simulates the environment, and consult it when needed. Where spatial states must be tracked over long horizons, agents struggle, as you have experienced with NetHack in BALROG from Case Study 2. The world model is then a property of the system rather than the model, and it is inspectable. A complementary development is the use of world models learned from video, which generate interactive environments in which agents can be trained [Bruce et al., 2024]. This connects to the open-world environments of Section 17.4. 22.3.4 Conclusion How can Othello-GPT and the Manhattan result both be true? An important key is the coverage of the training data. In the study by Vafa et al. [2024], an Othello model trained on randomly generated games passed the coherence tests, whereas one trained on human championship games did not. Random play likely covers all corners of the state space, whereas human play, like sensible taxi routes, cover only a small part of it. A model recovers the structure of the world to the extent that its data force it to. It helps to distinguish a weak from a strong sense. In the weak sense, a model has a world model if the state of the world can be decoded from its activations. In the strong sense, the model must also be coherent (one consistent model rather than many local fragments), causally used, and support generalization to new situations. The evidence shows mostly that LLMs have world models in the weak sense. De- pending on the density of the data in a certain domain, they can recover a coherent world model, amounting to the strong sense: an example of this is a model trained on sufficient random Othello games. Models trained on data that reflect how humans act in the world and write about it, which cover only the parts people found worth describing, end up with something more like a collection of partly inconsistent sketches than a full map. The same goes for regularities extracted from physics simulations: they suffice for local predictions but to not generalize well to other contexts. 22.4 Social Cognition and Theory of Mind As we have seen in Section 15.5, frontier LLMs match human performance on many Theory of Mind (ToM) tests. However, this performance is brittle under simple pertur- 434CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS bations of the test formats and predicting others’ mental states verbally (literal ToM) does not always carry over to behaving in line with the prediction (functional ToM). The questions whether this means that LLMs can be said to have ToM, and whether this would be ToM in any human sense, were postponed and will be picked up here. 22.4.1 Beliefs at the Representational and Mechanistic Levels Zhu et al. [2024] presented two relatively small models (Mistral-7B and DeepSeek- 7B) with false-belief stories and trained linear probes on the activations of attention heads. The probes could separately decode the belief of the protagonist and the state of the world as known to the omniscient reader, meaning that the model keeps the two perspectives apart. Applied to the Maxi story, this would mean that Maxi’s belief was that the chocolate was still in the blue cupboard, while the omniscient reader’s belief was updated to the green cupboard as the chocolate was moved in Maxi’s absence. The self-other distinction that we discussed for empathy in Section 15.3.3 thus seems to be represented in the tested models: they can appreciate that someone else (the protagonist) has a belief that is different from theirs (as omniscient reader of the test). In their experiments, steering activations along the probe directions changed per- formance on ToM tasks. This also occurred on tasks other than the one that the probe was trained on, suggesting that the represented beliefs are of causal importance for be- havior. In other words, there is not just evidence on the representational level, but also on the mechanistic level. Bortoletto et al. [2024] extend this line of work to several model families and sizes. They further find evidence that the quality and reliability of belief representations in- crease with model size and instruction tuning. They also show that the brittleness observed at the behavioral level extends to the representational level: while representa- tions of the omniscient perspective are rather stable, the representation of other’s beliefs are sensitive to prompt variations, even to variations that should not matter. 22.4.2 Human and Machine Routes to Similar Behavior Children develop ToM between the ages of 3-7. They do so partly on the basis of innate propensities to empathize with others and partly through learning from their en- vironments. Their ToM development also benefits from joint attention, pretend play, targeted conversations about what others want and think, and other kinds of embodied and interactive experience [Apperly, 2011]. Linguistic input also plays an important role in this development. Studies have demonstrated links between ToM development and the acquisition of cognitive verbs [Milligan et al., 2007] and the mastery of nega- tion [De Villiers and De Villiers, 2014] and sentence complementation [Lohmann and Tomasello, 2003]. A vast tradition of research shows that in particular narrative compe- tence, the ability to tell and understand stories, scaffolds the development of ToM skills in children, adolescents, and adults [Hutto, 2008, Van Duijn, 2016, Nicolopoulou and ̈ Unl ̈ utabak, 2017, e.g.]. LLMs acquire their ability to solve ToM tasks almost solely on the basis of linguistic input. 4 Their lack of interactive embedding in a real social en- 4 The role of post-training in the development of ToM is largely unclear, although there are indications that supervised fine-tuning enhances ToM performance [Kouwenhoven et al., 2026]. 22.4. SOCIAL COGNITION AND THEORY OF MIND435 vironment is compensated with scale: LLMs need multiple orders of magnitude more linguistic input before ToM competence begins to emerge [Trott et al., 2023, Kouwen- hoven et al., 2026]. This is an indication that similar test scores produced by children and LLMs do not support the claim that the underlying computations are similar [Hu et al., 2025]. It should be added that the debate about human processing of ToM tasks is itself also not settled. According to (1) theory-theory, we apply folk-psychological theories to others [Gopnik and Wellman, 1992]. According to (2) simulation-theory, we use our own mind and body as a model: we imagine being in the other’s situation to infer what they feel and think [Goldman, 2006]. Two-systems (3) advocates argue that we need both: simulation for quick and automatic inference, complemented by theory-theory for slower and more deliberate social reasoning [Apperly and Butterfill, 2009]. It is unclear whether any of these three views is helpful in describing ToM process- ing in LLMs. Given that simulation presupposes a body and a mental and emotional life of one’s own, it is hard to imagine how this would look in LLMs. On the other hand, looking at the role-play account discussed in Section 22.6, generating text as a character is what LLMs seem to do often. There is a parallel with simulation: humans adopt a perspective and produce what someone in that position would be likely say. Folk-psychological theories, in turn, can be based on text sources available in abundant quantity in LLM training data. Human text contains abundant mental-state language. Take, for example, children’s stories. The essence of Little Red Riding Hood revolves around the false-belief plot of her not knowing that the wolf ate grandmother and took her place in the bed. Little Red Riding Hood wonders why grandmother has such big ears, and so on. In a sense, such fairytales, but also novels, film scripts, and other fic- tion genres, can be considered to be large ToM training sets for LLMs. Abstractions distilled from these genres are likely to play a major role in LLMs’ capacity to perform ToM tasks, but the exact mechanisms is yet to be mapped out. In conclusion, while the debate over theory-driven versus simulation-driven ToM is not settled for humans, it is even more unclear how this distinction applies to LLMs. 22.4.3 The Literal–Functional Gap Revisited In Part V we have identified three candidate explanations for the gap between literal and functional ToM, or between knowing and doing: (1) the capacity is lacking, (2) the capacity is present but is not deployed spontaneously, or (3) the representations are present but are not connected to action selection. Explanation (2) seems to be supported by the success of scaffolding approaches such as Social Chain-of-Thought [Akata et al., 2025]. This is consistent with the probing results discussed above, which show that the relevant belief representations exist in models, but says nothing yet about (3), their actual deployment in action. A possible explanation is that LLMs invoke the relevant knowledge only when a context looks like a ToM problem. If this remains implicit, the relevant representations are not used for action selection unless the question is asked explicitly (like Social Chain-of-Thought does). This explanation is supported by experiments showing that altered ToM scenarios can trigger the wrong outcome to be produced: for example, if the context looks like a classic false-belief scenario but is actually a case of straight- 436CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS forward true-belief, models can be inclined to apply false-belief reasoning nevertheless [Ullman, 2023, Kouwenhoven et al., 2026]. Note that humans also find the knowing- doing distinction difficult. One may know the rules of the game of chess, but actually applying them in one’s mind to play out a few moves of the game takes a significant mental effort, that only few can master. Kahneman’s system 2 thinking requires much concentration effort for humans [Kahneman, 2011]. Note that external tools can be helpful, humans may use a piece of paper to keep track of board states, and likewise LLMs can be augmented by a harness to externally track states, as we have seen in Parts I and I. In agentic LLMs, the decisive experiment would apply causal interventions at the mechanistic level to represented beliefs during interaction in a game such as Hanabi, where the context requires ToM reasoning without explicit ToM questions being asked. To our knowledge this has not been done, as shown by the empty agent cell for ToM in Table 22.1. 22.4.4 Mindreading LLMs Humans, in turn, apply their ToM to LLMs. A study showed that a majority of par- ticipants in a representative US sample attributed some degree of conscious and folk- psychological experience to ChatGPT [Colombatto and Fleming, 2024]. Such attribu- tions correlated positively with frequency of use. This fits the CASA paradigm dis- cussed earlier (Section 15.3.2), and we return to it in Section 22.6. 22.4.5 Conclusion On current evidence, LLM Theory of Mind matches human behavior and performance on static tests. There is limited evidence that matching extends to the representational and mechanistic levels. LLMs represent beliefs as distinct from reality, and these rep- resentations can be causally involved in answering ToM questions. Compared to hu- mans, there seem to be both parallels and differences in how belief-representations are acquired and applied. Connections between representations and action planning are brittle and often rely on explicit activation. What further constrains the discus- sion about human-LLM parallels is that the debate about human ToM acquisition and processing itself is unresolved. 22.5 Metacognition, Introspection, and Self-Knowledge So far we asked what LLMs represent about language, the world, and other minds. We now ask what they represent about their own inner workings. This is particularly important for agentic LLMs. In order to execute a multi-step task, an agent must know, for instance, what it does not know, when to verify a piece of information, how much effort a task is likely to take, and when the task is actually finished. Questions of this kind are traditionally grouped under the concept of metacognition. In psychology, metacognition is usually understood as a loop between monitoring and control [Nelson, 1990]: a system assesses some aspect of its own cognitive pro- 22.5. METACOGNITION, INTROSPECTION, AND SELF-KNOWLEDGE437 cess, such as its uncertainty or progress on a sub-task, and uses that assessment to regulate what it does next. For example, by switching strategy, checking an interme- diate outcome, asking for help, or stopping. Liu et al. [2026] adopt this classic loop as the conceptual framework for a review on metacognition in LLMs. The distinc- tion is relevant to this section because the evidence for the two halves of the loop is not equally strong: a model may represent information about the quality of its answer without reliably acting on that information—another instance of the knowing–doing gap (Section 15.5.2). A terminological note: in the LLM literature, the term metacognition is used in a wide range of contexts that match the psychological concept only loosely, ranging from calibrated confidence scores to verbalized reflections on given output. For each finding below, we therefore ask what exactly is monitored, how we know, and whether the result is used to control further generation steps or actions. 22.5.1 Monitoring: Knowing What You Know Two properties of confidence judgments should be distinguished. Metacognitive cali- bration asks whether confidence matches accuracy: are answers given with 80% con- fidence also 80% of the time correct? Metacognitive sensitivity asks whether confi- dence discriminates between correct and incorrect answers: is the model more con- fident when it is right than when it is wrong? Psychologists use measures derived from signal detection theory, such as meta-푑 ′ and the ratio meta-푑 ′ /푑 ′ (the M-ratio), to express metacognitive sensitivity relative to task performance [Maniscalco and Lau, 2012]. These measures are now also being applied to LLMs. Already in 2022, Kadavath et al. [2022] showed that LLMs can rather accurately assess what they know and do not know. When asked various sorts of multiple-choice questions, models were able to assign scores that were well calibrated. Models could also evaluate the correctness of their answers rather well and they could be trained to predict whether they know the answer to a question before answer options were presented. These findings have been replicated and extended in several studies since. However, three caveats apply. Mixed results across studies suggest that metacognitive sensitiv- ity is task- and domain-dependent [Liu et al., 2026]. First, metacognitive calibration and sensitivity vary across tasks, domains, and models [Liu et al., 2026]. Second, post- training with human feedback degraded the calibration of GPT-4’s answer probabilities [Achiam et al., 2024], possibly because the reward models used in such training tend to favor confident responses [Leng et al., 2025]. Conversely, post-training can have a pos- itive effect [Lindsey, 2025]. Third, the result depends on how confidence is elicited: verbalized confidence estimates, self-reported numbers (e.g. prompting for a confi- dence score on a 0-100 scale), and confidence derived from token probabilities do not measure the same thing [Liu et al., 2026]. When models are asked to verbalize their confidence, they are mostly overconfident [Xiong et al., 2024]. 438CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS 22.5.2 Introspection The question whether humans have privileged access to their own inner states also ap- plies to LLMs: do they have a kind of access that an external observer, who is presented with the same behavioral data, has not? Binder et al. [2025] fine-tuned a model to pre- dict its own behavior in hypothetical situations. It could do so with higher accuracy than a second model that was trained on the behavioral data of the first. This suggests that the first model has access to internal information that is not available from the behavior alone. In contrast, when models’ verbalized judgments about their own lin- guistic knowledge (e.g. whether a sentence is grammatical) were compared with the probabilities they actually assign to sentences, the judgments did not track the model’s own probabilities better than those of other models [Song et al., 2025]. Lindsey [2025] tested introspection causally through concept injection. An acti- vation pattern corresponding to, for instance, writing in ALL CAPS was added to the model’s activations for a simple query. When the model was asked whether it noticed anything unusual, in at least some trials it was able to sensibly verbalize the injected concept. In the case of ALL CAPS, it could indicate that it noticed an “overly intense, high-volume concept.” Researchers have also asked whether reasoning traces can be seen as self-reports reflecting inner states. Turpin et al. [2023] demonstrated that when multiple-choice examples in the prompt always had the correct answer in the first position, models followed this bias. When prompted for explanations, they generally motivated their answer choice with rationalizations rather than explicating the bias. This is consistent with findings by Chen et al. [2025]. When hints about the answer were planted in the prompt, the models used them, but acknowledged this in only a minority of their reasoning traces. It should be noted that humans do something similar. In classic experiments, participants confidently gave reasons for their choices that demonstrably were not the causes of those choices [Nisbett and Wilson, 1977]. Overall, results indicating privileged access to a model’s own inner states are mixed. If the capacity exists, it is narrow, unreliable, and dependent on the experimental setup. 22.5.3 The Agentic Perspective The agents of Shapira et al. [2026] reported tasks as completed while the state of the system showed otherwise. This can be seen as a failure of self-monitoring: the claim about one’s own state was never checked against the world. For humans many such checks are habitual in work and other everyday settings, but for agents they have to be designed in as verification steps, tests, or state checks after each action. Agent archi- tectures do exactly this, for instance with separate monitors that detect when retrieval results and reasoning in a deep-search agent stop being consistent and trigger correc- tive actions [Sun et al., 2026]. As with world models, metacognition may arise at the level of a system as a whole rather than within a single model. 22.6. CONSCIOUSNESS439 22.5.4 Conslusion LLM metacognition, introspection, and self-knowledge are detectable, but also limited in multiple ways. On the monitoring side, a model’s verbalizations, internal probabil- ities, and activations carry different degrees of information about the reliability of its answers. Evidence for privileged introspective access is mixed and points towards lim- ited access at best. On the control side, evidence is even thinner: monitoring signals are often not used to regulate behavior. This leaves space for agentic setups that can design for behavior that is functionally equivalent to what humans achieve with metacognitive control. 22.6 Consciousness Can a machine be conscious? The question has been around for ages and used to belong to the domains of speculation, thought experiment, and science fiction. Yet today we see this question discussed regarding LLMs and Agentic LLMs by leading scientists and philosophers of consciousness. In this section we outline how the question can be approached and why it remains open. 22.6.1 Definitions What distinguishes consciousness from the broader label of intelligence? In his classic paper What Is It Like to Be a Bat?, Nagel [1974] suggests that a system is phenomenally conscious if there is “something it is like” to be that system. This positions conscious- ness as property that is to be experienced from within and makes it by definition hard to access from an outside vantage point. Block [1995] makes a distinction with access consciousness, defined as availability of information (from the senses, memory, and other faculties) for reasoning, reflection, and the control of action. Both these subtypes of consciousness should be distinguished from sentience, which refers to experience that has a valence, such as pleasure, suffering, and desire. Functional capacities, such as spatial orientation, Theory of Mind, and metacognition (discussed above), can in principle be present without sentience or consciousness. Conversely, while most an- imals score poorly on at least the latter two of these functional capacities, they are plausibly sentient. From our perspective as human observers and users of LLMs, we often freely use expressions such as the system wants ... or the agent decides .... It is important to distinguish between the pragmatic use of such attributions and making commitment about the ontological status of such mental states [van Dijk et al., 2023]. Dennett [1987] also makes distinction in his intentional stance, which is the practice of attributing beliefs, desires, and other mental states, to any system whose behavior can be predicted well by doing so. Saying about a thermostat that it will switch on the heating once it finds the room too cold provides a workable model of that system without needing any evidence of an inner life that functions like ours. While some argue that this is all there is to expressions like the LLM understands ... or this agents wants ..., others argue that there is depth to such expressions in the case of LLMs that is worth exploring. 440CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS 22.6.2 Functionalism and Theory-Driven Assessment In line with what we have discussed in Part V and above, Butlin et al. [2023] argue that behavior alone is a poor guide for the inner workings of LLMs. Therefore they suggest to start from leading scientific theories of consciousness and derive indica- tor properties from these that can be held against current AI architectures and system properties. Their account rests on the premise of computational functionalism, the view that performing computations of the right kind is both necessary and sufficient for con- sciousness. What matters is the functional organization of a system (which information it represents, how it transforms that information, and how its components interact), not the material substrate that realizes the processing. In this view, consciousness is substrate-independent, which is often illustrated with the following thought experiment. Suppose you have a brain injury that affects only one or a couple of neurons and advancements in micro medicine make it possible to replace the affected neurons by silicon-based counterparts that fulfill the exact same function as their biological predecessors did. Your brain function remains completely functionally equivalent. Would you say that you are no longer conscious? What if more neurons undergo replacement after subsequent injuries, up until the point that a quarter of your brain contains silicon-based neuron equivalents? Or half, or three quarters? Following Searle [1980] this view remains contested in biological views that hold that consciousness depends on specific properties of living neural tissue (see Section 22.6.3 below). Butlin et al. [2023] argue that functionalism should be adopted as a working hy- pothesis rather than a matter of fact, because it makes the consciousness question em- pirically tractable: scientific theories that specify the relevant computations can be translated into checkable properties of AI architectures. We list the main indicators and the authors’ assessments of these, but note that the paper was written in 2023 and that an agentic dimension is missing; we will return to that below. Recurrent processing Consciousness is linked to recurrent rather than purely feed- forward processing. Algorithmic recurrence is (in a limited sense) a property of RNNs or LSTMs, but a Transformer’s forward pass is not recurrent. Global workspace theory For humans this theory holds that specialized parallel modules compete for a limited-capacity workspace whose contents are globally broad- cast back to all modules. The transformer’s residual stream (Section 21.3) has been suggested to function as such a workspace, but the case is weak given that it is not a clear bottleneck and there is no genuine recurrence or broadcast back to modules. Re- cent work, however, suggests a sparse, capacity-limited subspace within the residual stream does behave as a bottleneck (see below) [Gurnee et al., 2026]. Higher-order theories In this view a state becomes conscious when a higher-order metacognitive mechanism monitors first-order perceptual representations (tagging them as e.g. reliable rather than noise) and feeds belief-formation and action-selection sys- tem. The authors see no current system that implements this combination, but also highlight that it would be technically possible with existing components. 22.6. CONSCIOUSNESS441 Attention schema theory According to this theory consciousness arises from a sys- tem’s simplified predictive model of its own attention, which it uses to represent and enable control over its current state of attention. The authors note that machine atten- tion is only a loose analogue of the type of attention meant in this theory and certainly does not amount to a self-model that enables steering current attention. Predictive processing In this theory perception is framed as prediction-error mini- mization. There are elements of this in how LLMs are trained, but the authors view this more as a general framework for cognition, not as a standalone indicator of conscious- ness. Agency By this account learning from feedback and selecting outputs in light of spe- cific goals is taken as a form of consciousness. The authors suggest that reinforcement learning suffices for the basic form, but that this involves externally set goals that are not as such part of the system. Embodiment The authors highlight that a physical body is itself not the key feature here, but a model of the system’s output affects its input, which helps to distinguish self- caused from world-caused changes. They discuss various systems that have variants of such models, either through control over robots or virtual avatars, but argue that they are still weak or uncertain candidates for the full, end-to-end version of embodiment found across the animal kingdom. Overall, the authors take the stance that no current AI system is a strong candidate for consciousness by their defined indicators, but they see no obvious technical barriers to building systems that satisfy many of the indicators in the near future. As we will see below, the agentic dimension is highly relevant to this debate. Chalmers [2023] reached a similar conclusion in the same year. He lists what LLMs lack by his account: biology, senses, a body, world models and self-models, recurrent processing, a global workspace, and unified agency. Two of these obstacles are the topics of the above Sections 22.3 and 22.5, and the evidence there suggests that various aspects of these are currently overcome or in the process of being developed. Chalmers put his credence that current LLMs are conscious below ten percent, but gave considerably higher odds to extended successors. Table 22.2 gives a simplified overview. Recent work in mechanistic interpretability work has focused on the question whether there is a global workspace that emerges in the forward pass of frontier LLMs. Gurnee et al. [2026] introduce the idea of the Jacobian Lens, which involves assigning to each vocabulary token a direction in the residual stream that encodes the model’s disposi- tion to verbalize that token, averaged over contexts. They identify the J-space, which holds only a few dozen active concepts at a time, and argue that it has the functional profile that Global Workspace Theory associates with conscious access. Its contents determine to an important degree what the model reports, can be summoned on in- struction while the model does something else, carry unspoken intermediate steps in multi-step reasoning, and can be swapped to redirect many different downstream com- putations at once. Ablating it leaves routine processing such as parsing, classification, 442CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS TheoryIndicator (simplified)Bare LLMAgentic system Recurrent processingFeedback loops in processingDebated: feedforward per token, recurrent only through its own out- put Same Global workspaceLimited-capacity workspace with global broad- cast DebatedCloser:shared memory buffer feeding perception, planning, and action modules Higher-order theoriesMonitoring of own representationsWeak and unreliable (Sect. 22.5)Verification modules monitor out- puts, not internal states Attention schemaModel of own attentionNo evidenceNo evidence Agency and embodimentGoal pursuit with feedback; body modelLargely absentAgency partly; embodiment only in robotics Biological viewsLife, metabolism, bodily regulationAbsentAbsent Table 22.2: Simplified overview of indicator properties for consciousness and the ex- tent to which bare LLMs and agentic systems satisfy them. After Butlin et al. [2023] and Goldstein and Kirk-Giannini [2024]; the assessments in the last two columns are debated. and fluent continuation largely intact, but impairs the kind of flexible inference that makes current frontier models so powerful on an array of complex tasks. The authors take no position on phenomenal consciousness and stress that the parallel is functional rather than architectural: there is no input from and feedback into encapsulated mod- ules: broadcast happens across layer depth within a single feedforward pass rather than through recurrent loops. The findings strengthen the case made by Butlin et al. [2023] for an approach based on indicators and leaves disagreements with biological natural- ists (Section 22.6.3 below) untouched, sinse that disagreement does not concern the facts about LLMs, rather the premises of what it means to be conscious. 22.6.3 Rejections of Functionalism [Seth, 2025] counters functionalism and defends a position referred to as biological naturalism. According to this position, consciousness is deeply intertwined with our nature as living organisms: its function is regulate a precarious body that has to keep itself in existence. Properties of intelligent systems such as prediction-error minimiza- tion, and capacities such as spatial orientation, Theory of Mind, and self-reflection, should be viewed in service of staying alive. There may be local similarities between these properties and capacities in natural and artificial systems, but in the latter they are not causally connected in a biological chain of existence. In other words, for a biologi- cal naturalist being conscious requires having skin in the game. 5 Aru et al. [2023] share this line of argumentation and add that LLM inputs lack the rich 4E information con- tent (see Section 22.1.3) of our sensory contact with the world. From a neuroscience perspective, LLMs therefore miss key features of the thalamocortical system known to be linked to consciousness in mammals. For biological naturalists, artificial consciousness would be more likely as systems (1) become more life-like and start having autonomous evolutionary and developmental trajectories to which properties of their intelligence are causally connected, (2) have richer information streams that connect them with the outside world. Note that the 5 Seth’s article was published in Brain and Behavioural Sciences and is followed, according to the jour- nal’s conventional format, by open peer commentaries. Leonard Dung’s Why I Am Not a Biological Natu- ralist is one of these commentaries contains a counterargument. 22.6. CONSCIOUSNESS443 disagreement with computational functionalists is thus mainly about the premises of what consciousness is, not about the facts concerning LLMs. 22.6.4 Self-Reports and Inner Life LLMs produce first-person reports of experience, thinking steps, emotions, and more. There are at least two reasons why these should be discounted. First, LLMs are trained on human texts that are full of such descriptions. The descriptions therefore need not have a link to the LLM and the current context itself, but can be external to both. Sec- ond, in the role-play account of Shanahan et al. [2023], a model structurally acts as a character. This also disconnects self-reports from the model itself: the same model can play characters that claim to be conscious just as well as characters that deny it. This what has been called the gaming problem: the abundance of markers for mental properties in the training data makes that those markers can no longer be reliably used [Birch, 2024]. It has been suggested that self-reports could become more informative and reliable if models would be trained to introspect accurately on questions with com- putationally verifiable answers [Perez and Long, 2023a], but Section 22.5 shows how far we currently are from that situation. In Section 15.3.3 we set aside whether LLMs feel anything. On the behavioral level, anxiety-inducing prompts change subsequent self-reports: they raise the scores of LLMs on anxiety questionnaires, whereas relaxation prompts partly reverse this [Ben-Zion et al., 2026]. These findings carry the above two caveats: the questions can be answered irrespective of the model’s actual inner state, by relying on associations between different aspects of anxiety learned during training. On the representational level, emotion concepts correspond to directions in activation space, and, at the mech- anistic level, steering along these directions changes the model’s outputs [Zou et al., 2025]. Arguably, these states thus modulate behavior much as emotions do in hu- mans and other animals. On a functionalist account, this makes them emotion-like states [Moerland et al., 2018], but whether they are felt in any way comparable to how emotions are experiences by humans and other animals, inherits all of the uncertainty discussed above. 22.6.5 The Agentic Perspective At the time of writing, work targeting the specific intersection between consciousness and agentic LLMs is sparse. However, agentic architectures have been argued to be relevant to least one theory of LLM consciousness. Goldstein and Levinstein [2024] observe that LLM agents of the kind discussed in Chapter 17, with a memory stream, retrieval, reflection, and planning modules that communicate through a shared buffer, come close to the architecture that Global Workspace Theory (GWT) requires. If GWT is correct, they argue, such agents are or could easily be made conscious – and those who find this conclusion implausible should take it as evidence against GWT. Persistent memory and long-term goals also address the lack of unified agency that Chalmers noted. An interesting note made by Gurnee et al. [2026] is that the J-space may emulate some functions in a single forward pass that are also attributed to recurrent process- 444CHAPTER 22. *COGNITIVE PERSPECTIVES ON AGENTIC LLMS ing in agentic architectures and chain-of-thought prompting. Problems solved with written-out reasoning were much more robust to J-space ablation, which suggests that the model would otherwise hold such reasoning internally. This brings the debate back to the question at what level—or any form of cognitive functioning—is attributed: the border between what happens strictly inside the model and what emerges at system- level cannot be drawn a priori and depends on the task, format of the prompt, avail- ability of tools, interactions, and other contextual factors. Note that this is precisely what cognitive externalists have argued for human cognitive functioning as well [No ̈ e, 2009]. 22.6.6 Conclusion In the case of consciousness and experience of inner states, making the possible errors introduced at the beginning of this chapter can both be costly. Anthropomorphisation is already happening and brings risks of emotional dependence, misplaced trust, and manipulation. Anthropo-denial would, if such systems were consciousness or sentient to some degree, bring moral errors on a large scale. Birch [2024] argues for precautions that are proportionate to the probability, and Butlin et al. [2023] argue that the proba- bility is high enough to start assessing systems now and some developers have indeed started research programs on model welfare. For example, Anthropic has included a welfare assessment in the documentation of the Opus 4 model release, and has allowed some of its models to end persistently abusive conversations, as a low-cost precaution in case model welfare matters. 6 Whatever one thinks of the probabilities, the topic has moved from philosophy seminars to engineering practice. 22.7 Concluding Remarks This chapter asked what lies behind the behavior surveyed in Part V, and whether this is anything like the cognition that drives behavior in humans and animals. Four general threads have been identified. First, the degree to which internal representations are structured in ways that make sense to a human observer, varies across tasks and domains. LLMs trained on text can develop representations that mirror geography, board states, the beliefs of story characters, and traits of the persona they play, and in some cases it can be shown mechanistically that these representations are used. The image of the stochastic parrot does not fit these findings: after all, the presence of a causal chain from representation to behavior in at least some cases, falsifies the claim that LLMs operate at the surface- level only. But the same methods reveal how incoherent representations can be, and how responses can rely on patchy heuristics. This is in part because human text covers the world unevenly and in part because surface statistics suffice in some domains to satisfy training objectives. Therefore, the image of a mind that understands does not fit either. LLMs are a new kind of system, and our vocabulary for cognition, developed for humans and other animals, applies to them only imperfectly. 6 https://w.anthropic.com/research/end-subset-conversations BIBLIOGRAPHY445 Second, the knowing–doing gap returned in the cognitive domain. Models rep- resent more than they spontaneously use or report: we saw belief-states that do not guide action, definitions that are not applied, and computations that are described in the wrong way. A plausible hypothesis is that structured prompts and scaffolds work because they connect existing representations to behavior. Agentic loops can in the- ory provide such structure and scaffolding autonomously. At this point this has hardly been tested empirically: all representational and mechanistic evidence concerns bare models. For agents, the last column of Table 22.1 holds behavioral entries only. A third point is that we can now describe agentic extension as moving cognition from the model to the system. World states can be kept in explicit, externally stored and updated maps, self-monitoring can be implemented as verification steps, and a memory that is shared between agents can act as a workspace. According to the extended mind thesis in philosophy, if we reliably consult a notebook as external memory, also for humans this becomes part of our cognitive system [Clark and Chalmers, 1998]. Cogni- tive architectures for agentic LLMs are designed in this spirit [Sumers et al., 2024]. An advantage is that externalized cognition adds to the technology’s transparency because it can be inspected. Fourth, on consciousness the honest position is uncertainty on all levels. Theo- ries disagree on whether any digital system could be conscious, self-reports and other behavioral aspects are unreliable determinants, and it is as of yet unclear how inter- pretability techniques can be made informative on this question. What can be done now is to avoid designs that mislead users in over-attributing consciousness. Researchers argue that it is time to start assessing model welfare [Butlin et al., 2023, Long et al., 2024]. Bibliography Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without ground- ing? a case study in color. In Arianna Bisazza and Omri Abend, editors, Pro- ceedings of the 25th Conference on Computational Natural Language Learning, pages 109–132, Online, November 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/2021.conll-1.9. URL https://aclanthology.org/ 2021.conll-1.9/. 426, 429 Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Bal- tescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, et al. Gpt-4 technical re- port, 2024. URL https://arxiv.org/abs/2303.08774. 437 Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. Nature Human Behaviour, 9(7):1380–1390, 2025. 435 446BIBLIOGRAPHY Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. 425 Ian Apperly. Mindreaders: the Cognitive Basis of ”Theory of Mind”. Psychology Press, 2011. 434 Ian A. Apperly and Stephen A. Butterfill. Do humans have two systems to track beliefs and belief-like states? Psychological Review, 116(4):953–970, 2009. doi: 10.1037/ a0016923. 435 Jaan Aru, Matthew E. Larkum, and James M. Shine. The feasibility of artificial con- sciousness through the lens of neuroscience. Trends in Neurosciences, 46(12):1008– 1017, 2023. doi: 10.1016/j.tins.2023.09.009. 442 Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, Singapore, 2023. As- sociation for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.68. URL https://aclanthology.org/2023.findings-emnlp.68. 426 Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Com- putational Linguistics, 48(1):207–219, 2022. 425 Ziv Ben-Zion, Zohar Elyoseph, Tobias Spiller, et al. Inducing state anxiety in LLM agents reproduces human-like biases in consumer decision-making.npj Artifi- cial Intelligence, 2:55, 2026. doi: 10.1038/s44387-026-00122-1. URL https: //doi.org/10.1038/s44387-026-00122-1. 443 Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, pages 5185–5198, Online, July 2020. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.463. URL https://aclanthology.org/2020.acl-main.463/. 428 Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.3445922. URL https://doi.org/10.1145/3442188.3445922. 429 Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on ”a is b” fail to learn ”b is a”, 2024. URL https://arxiv.org/abs/2309.12288. 429 Felix J. Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. Looking inward: Language models can learn about themselves by introspection. In The Thirteenth International BIBLIOGRAPHY447 Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=eb5pkwIB5i. 426, 438 Jonathan Birch. Large language models and the gaming problem. In The Edge of Sentience: Risk and Precaution in Humans, Other Animals, and AI. Oxford Univer- sity Press, 08 2024. ISBN 9780192870421. doi: 10.1093/9780191966729.003. 0017. URL https://doi.org/10.1093/9780191966729.003.0017. 443, 444 Ned Block. On a confusion about a function of consciousness. Behavioral and Brain Sciences, 18(2):227–247, 1995. doi: 10.1017/S0140525X00038188. 439 Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, and Andreas Bulling. Benchmarking mental state representations in language models. In ICML 2024 Workshop on Mech- anistic Interpretability, 2024. URL https://openreview.net/forum?id= yEwEVoH9Be. 426, 434 Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Ed- ward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. 433 Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowl- edge in language models without supervision. In The Eleventh International Con- ference on Learning Representations, 2023. URL https://openreview.net/ forum?id=ETKGuby0hcs. 426 Patrick Butlin, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, Axel Constant, George Deane, Stephen M. Fleming, Chris Frith, Xu Ji, Ryota Kanai, Colin Klein, Grace Lindsay, Matthias Michel, Liad Mudrik, Megan A. K. Peters, Eric Schwitzgebel, Jonathan Simon, and Rufin VanRullen. Consciousness in artifi- cial intelligence: Insights from the science of consciousness, 2023. 426, 440, 442, 444, 445 James Carney. Thinking avant la lettre: A review of 4e cognition. Evolutionary Studies in Imaginative Culture, 4(1):77–90, 2020. doi: doi:10.26613/esic.4.1.172. URL https://doi.org/10.26613/esic.4.1.172. 427 David J Chalmers. Could a large language model be conscious?arXiv preprint arXiv:2303.07103, 2023. 441 Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don’t always say what they think, 2025. URL https://arxiv.org/ abs/2505.05410. 438 Noam Chomsky. Aspects of the Theory of Syntax. MIT Press, Cambridge, MA, 1965. 427 448BIBLIOGRAPHY Andy Clark and David Chalmers. The extended mind. analysis, 58(1):7–19, 1998. 428, 445 Clara Colombatto and Stephen M Fleming. Folk psychological attributions of con- sciousness to large language models. Neuroscience of Consciousness, 2024(1): niae013, 01 2024. ISSN 2057-2107. doi: 10.1093/nc/niae013. URL https: //doi.org/10.1093/nc/niae013. 436 Kenneth J. W. Craik. The Nature of Explanation. Cambridge University Press, Cam- bridge, 1943. 431 Jill G. De Villiers and Peter A. De Villiers. The role of language in theory of mind development. Topics in Language Disorders, 34(4):313–328, 2014. 434 Frans B. M. de Waal. Anthropomorphism and anthropodenial. Philosophical Topics, 27(1):255–280, 1999. doi: 10.5840/philtopics199927122. 427 Daniel C. Dennett. The Intentional Stance. The MIT Press, Cambridge, MA, 1987. ISBN 9780262040938. 439 Chaz Firestone. Performance vs. competence in human–machine comparisons. Pro- ceedings of the National Academy of Sciences, 117(43):26562–26571, 2020. doi: 10.1073/pnas.1905334117. URL https://w.pnas.org/doi/abs/10. 1073/pnas.1905334117. 427 Elizabeth Gibney. AI language models killed the Turing test: Do we even need a replacement? Nature, 2025. doi: 10.1038/d41586-025-03386-w. URL https: //doi.org/10.1038/d41586-025-03386-w. News feature. 424 Alvin I. Goldman.Simulating Minds: The Philosophy, Psychology, and Neuro- science of Mindreading. Oxford University Press, New York, 2006. doi: 10.1093/ 0195138929.001.0001. 435 Simon Goldstein and Cameron Domenico Kirk-Giannini. A case for ai consciousness: Language agents and global workspace theory. arXiv preprint arXiv:2410.11407, 2024. 426, 442 Simon Goldstein and Benjamin A. Levinstein. Does chatgpt have a mind?, 2024. URL https://arxiv.org/abs/2407.11015. 443 Alison Gopnik and Henry M. Wellman. Why the child’s theory of mind really is a theory. Mind & Language, 7(1–2):145–171, 1992. doi: 10.1111/j.1468-0017.1992. tb00202.x. 435 Shannon Gray, George Tambouratzis, Sanju Mannumadam Venugopal, Sridhar Ragha- van, Richard Jiarui Tong, and Zeyu Han. Agentic correlates of consciousness and the pursuit of artificial general intelligence. In International Conference on Artificial General Intelligence, pages 225–238. Springer, 2025. 426 BIBLIOGRAPHY449 Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al.Is your llm secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06559, 2024. 426 Wes Gurnee and Max Tegmark.Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/ file/0a6059857ae5c82ea9726e9282a7145-Paper-Conference. pdf. 426, 431 Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. Verbalizable representations form a global workspace in language models, 2026. URL https://arxiv.org/abs/2607.15495. 426, 440, 441, 443 David Ha and J ̈ urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. 431 Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. 426 Stevan Harnad.The symbol grounding problem.Physica D: Nonlinear Phe- nomena, 42(1):335–346, 1990.ISSN 0167-2789.doi:https://doi.org/10. 1016/0167-2789(90)90087-6.URL https://w.sciencedirect.com/ science/article/pii/0167278990900876. 428 John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 conference on empirical methods in natural language pro- cessing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2733–2743, 2019. 425 Jennifer Hu, Felix Sosa, and Tomer Ullman. Re-evaluating theory of mind evaluation in large language models, 2025. URL https://arxiv.org/abs/2502.21098. 425, 435 Daniel D. Hutto. Folk Psychological Narratives: The Sociocultural Basis of Under- standing Reasons. The MIT Press, 2008. ISBN 9780262517980. URL http: //w.jstor.org/stable/j.ctt5h5p. 434 Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vi- vian Paulun, Maria Ryskina, Ekin Aky ̈ urek, Ethan Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Joshua Tenenbaum, and Jacob Andreas. Elements of world knowledge (ewok): A cognition-inspired framework for evaluat- ing basic world knowledge in language models, 2025. URL https://arxiv. org/abs/2405.09605. 432 450BIBLIOGRAPHY Cameron R. Jones and Benjamin K. Bergen. Large language models pass the Turing test, 2025. URL https://arxiv.org/abs/2503.23674. 424 Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know, 2022. 426, 437 Daniel Kahneman. Thinking, fast and slow. macmillan, 2011. 436 Amita Kamath, Jack Hessel, and Kai-Wei Chang. What’s ”up” with vision-language models? investigating their struggle with spatial reasoning, 2023. URL https: //arxiv.org/abs/2310.19785. 432 Tom Kouwenhoven, Michiel T. van der Meer, and Max J. van Duijn. Traces of social competence in large language models. In Claire Bonial and Yevgeni Berzak, editors, Proceedings of the 30th Conference on Computational Natural Language Learning, pages 742–759, San Diego, California, USA, 2026. Association for Computational Linguistics. ISBN 979-8-89176-410-1. doi: 10.18653/v1/2026.conll-main.45. URL https://aclanthology.org/2026.conll-main.45/. 425, 427, 434, 435, 436 Andrew Lampinen.Can language models handle recursively nested grammatical structures? a case study on comparing models and humans. Computational Lin- guistics, 50(4):1441–1476, December 2024. doi: 10.1162/coli a00525. URL https://aclanthology.org/2024.cl-4.8/. 427 Yann LeCun et al. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62, 2022. 431 Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overcon- fidence in llms: Reward calibration in rlhf, 2025. URL https://arxiv.org/ abs/2410.09724. 437 Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Vi ́ egas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learn- ing Representations, 2023. URL https://openreview.net/forum?id= DeG07_TcZvT. 426, 431, 432 Jack Lindsey.Emergent introspective awareness in large language mod- els.Transformer Circuits Thread, October 2025.URL https:// transformer-circuits.pub/2025/introspection/index.html. 426, 437, 438 BIBLIOGRAPHY451 Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hos- mer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zim- merman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Bat- son.On the biology of a large language model.Transformer Circuits Thread, 2025.URL https://transformer-circuits.pub/2025/ attribution-graphs/biology.html. 425, 431 Gabrielle Kaili-May Liu, Areeb Gani, Jacqueline Lu, Jordan Thomas, Mark Steyvers, and Arman Cohan. Metacognition in llms: Foundations, progress, and opportunities, 2026. URL https://arxiv.org/abs/2607.11881. 437 Heidemarie Lohmann and Michael Tomasello. The role of language in the development of false belief understanding: A training study. Child Development, 74(4):1130– 1144, 2003. 434 Robert Long, Jeff Sebo, Patrick Butlin, Kathleen Finlinson, Kyle Fish, Jacqueline Harding, Jacob Pfau, Toni Sims, Jonathan Birch, and David Chalmers. Taking ai welfare seriously, 2024. URL https://arxiv.org/abs/2411.00986. 445 Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenen- baum, and Evelina Fedorenko. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627, 2023. 430 Marina Mancoridis, Bec Weeks, Keyon Vafa, and Sendhil Mullainathan. Potemkin understanding in large language models, 2025. URL https://arxiv.org/ abs/2506.21521. 430 Matthew Mandelkern and Tal Linzen. Do language models’ words refer? Computa- tional Linguistics, 50(3):1191–1200, September 2024. doi: 10.1162/coli a00522. URL https://aclanthology.org/2024.cl-3.12/. 429 Brian Maniscalco and Hakwan Lau. A signal detection theoretic approach for estimat- ing metacognitive sensitivity from confidence ratings. Consciousness and Cognition, 21(1):422–430, March 2012. doi: 10.1016/j.concog.2011.09.021. 437 David Marr. Vision: A Computational Investigation into the Human Representa- tion and Processing of Visual Information.The MIT Press, 07 2010.ISBN 9780262514620. doi: 10.7551/mitpress/9780262514620.001.0001. URL https: //doi.org/10.7551/mitpress/9780262514620.001.0001. 424 R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, and Thomas L. Griffiths. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41):e2322420121, 2024a. doi: 10.1073/pnas.2322420121. URL https://w.pnas.org/doi/abs/10.1073/pnas.2322420121. 430 452BIBLIOGRAPHY R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy, and Thomas L. Griffiths. When a language model is optimized for reasoning, does it still show embers of autoregression?an analysis of openai o1, 2024b.URL https: //arxiv.org/abs/2410.01792. 430 Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Jason Phang, and Samuel R. Bowman. What do nlp researchers believe? results of the nlp community metasurvey, 2022. URL https://arxiv.org/abs/2208.12852. 429 Rapha ̈ el Milli ` ere and Charles Rathkopf. Anthropocentric bias in language model eval- uation. Computational Linguistics, 52(1):379–388, March 2026. doi: 10.1162/coli. a.582. URL https://aclanthology.org/2026.cl-1.11/. 427 Karen Milligan, Janet Wilde Astington, and Lisa Ain Dack. Language and Theory of Mind: Meta-Analysis of the Relation Between Language Ability and False-Belief Understanding. Child Development, 78(2):622–646, 2007. doi: https://doi.org/10. 1111/j.1467-8624.2007.01018.x. 434 Melanie Mitchell and David C. Krakauer. The debate over understanding in ai’s large language models. Proceedings of the National Academy of Sciences, 120 (13):e2215907120, 2023. doi: 10.1073/pnas.2215907120. URL https://w. pnas.org/doi/abs/10.1073/pnas.2215907120. 429 Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. Emotion in reinforce- ment learning agents and robots: a survey. Machine Learning, 107(2):443–480, 2018. doi: 10.1007/s10994-017-5666-0. URL https://doi.org/10.1007/ s10994-017-5666-0. 443 Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Robert Os- azuwa Ness, Nebojsa Jojic, Hamid Palangi, and Jonathan Larson. Evaluating cog- nitive maps and planning in large language models with cogeval. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023a. Curran Associates Inc. 432 Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Robert Os- azuwa Ness, Nebojsa Jojic, Hamid Palangi, and Jonathan Larson.Evaluating cognitive maps and planning in large language models with CogEval. In Advances in Neural Information Processing Systems, volume 36, 2023b.URL https: //proceedings.neurips.c/paper_files/paper/2023/file/ dc9d5dcf3e86b83e137bad367227c8ca-Paper-Conference.pdf. 426 Thomas Nagel. What is it like to be a bat? The Philosophical Review, 83(4):435– 450, October 1974. doi: 10.2307/2183914. URL https://w.jstor.org/ stable/2183914. 439 BIBLIOGRAPHY453 Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Yonatan Belinkov, So- phie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, ed- itors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16–30, Singapore, 2023. Association for Com- putational Linguistics. doi: 10.18653/v1/2023.blackboxnlp-1.2. URL https: //aclanthology.org/2023.blackboxnlp-1.2. 426, 431, 432 Joe Needham, Giles Edkins, Govind Pimpale, Henning Bartsch, and Marius Hobbhahn. Large language models often know when they are being evaluated, 2025. URL https://arxiv.org/abs/2505.23836. 426 Thomas O. Nelson. Metamemory: A theoretical framework and new findings. vol- ume 26 of Psychology of Learning and Motivation, pages 125–173. Academic Press, 1990.URL https://w.sciencedirect.com/science/article/ pii/S0079742108600535. 436 Ageliki Nicolopoulou and Burcu ̈ Unl ̈ utabak. Narrativity and Mindreading Revisited: Children’s Understanding of Theory of Mind in a Storybook and in Standard False Belief Tasks. In F Nihan Ketrez, Asl ̈ A ˆ A± ̈ Ozy ̈ urek, S ̧ eyda ̈ Ozc ̧alis ̧kan, and Aylin C K ̈ untay, editors, Social Environment and Cognition in Language Development, pages 151–166. John Benjamins, 2017. doi: 10.1075/tilar.21.10nic. 434 Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic with- out algorithms: Language models solve math with a bag of heuristics, 2025. URL https://arxiv.org/abs/2410.21272. 433 Richard E. Nisbett and Timothy D. Wilson. Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3):231–259, 1977. doi: 10. 1037/0033-295X.84.3.231. 438 Alva No ̈ e. Out of Our Heads: Why You Are Not Your Brain, and Other Lessons from the Biology of Consciousness. Hill and Wang, New York, 2009. ISBN 9781429957199. URL https://books.google.nl/books?id=llUvnqcEKTcC. 427, 444 Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wol- czyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuci ́ nski, Lerrel Pinto, Rob Fer- gus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rockt ̈ aschel. BAL- ROG: Benchmarking agentic LLM and VLM reasoning on games. In The Thir- teenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=fp6t3F669F. 426 Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gJcEM8sxHK. 426, 429 Ellie Pavlick. Symbols and grounding in large language models. Philosophical Trans- actions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 381(2251):20220041, 06 2023. ISSN 1364-503X. doi: 10.1098/rsta.2022.0041. URL https://doi.org/10.1098/rsta.2022.0041. 429 454BIBLIOGRAPHY Ethan Perez and Robert Long. Towards evaluating ai systems for moral status using self-reports, 2023a. URL https://arxiv.org/abs/2311.08576. 443 Ethan Perez and Robert Long. Towards evaluating AI systems for moral status using self-reports, 2023b. 426 Steven T. Piantadosi and Felix Hill. Meaning without reference in large language mod- els, 2022. URL https://arxiv.org/abs/2208.02957. 429 John R. Searle. Minds, brains, and programs. Behavioral and Brain Sciences, 3(3): 417–424, 1980. doi: 10.1017/S0140525X00005756. 424, 428, 440 Anil K. Seth. Conscious artificial intelligence and biological naturalism. Behavioral and Brain Sciences, 49:e315, 2025. doi: 10.1017/S0140525X25000032. 442 Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role-play with large language models, 2023. URL https://arxiv.org/abs/2305.16367. 443 Natalie Shapira, Chris Wendler, Avery Yen, Gabriele Sarti, Koyena Pal, Olivia Floody, Adam Belfki, Alex Loftus, Aditya Ratan Jannali, Nikhil Prakash, Jasmine Cui, Giordano Rogers, Jannik Brinkmann, Can Rager, Amir Zur, Michael Ripa, Aruna Sankaranarayanan, David Atkinson, Rohit Gandikota, Jaden Fiotto-Kaufman, Eun- Jeong Hwang, Hadas Orgad, P Sam Sahil, Negev Taglicht, Tomer Shabtay, Atai Am- bus, Nitay Alon, Shiri Oron, Ayelet Gordon-Tapiero, Yotam Kaplan, Vered Shwartz, Tamar Rott Shaham, Christoph Riedl, Reuth Mirsky, Maarten Sap, David Man- heim, Tomer Ullman, and David Bau. Agents of chaos, 2026. URL https: //arxiv.org/abs/2602.20021. 426, 438 Siyuan Song, Jennifer Hu, and Kyle Mahowald. Language models fail to introspect about their knowledge of language, 2025. URL https://arxiv.org/abs/ 2503.07513. 438 Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive architectures for language agents, 2024. URL https://arxiv.org/ abs/2309.02427. 445 Zhongxiang Sun, Qipeng Wang, Weijie Yu, Jingxuan Yang, Haolang Lu, and Jun Xu. Deep search with hierarchical meta-cognitive monitoring inspired by cognitive neu- roscience, 2026. URL https://arxiv.org/abs/2601.23188. 438 Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemantic- ity: Extracting interpretable features from claude 3 sonnet.Transformer Cir- cuits Thread, 2024. URL https://transformer-circuits.pub/2024/ scaling-monosemanticity/index.html. 425 BIBLIOGRAPHY455 Edward C Tolman. Cognitive maps in rats and men. Psychological review, 55(4):189, 1948. 431 Sean Trott, Cameron Jones, Tyler Chang, James Michaelov, and Benjamin Bergen. Do large language models know what humans know?Cognitive Science, 47 (7):e13309, 2023.doi: https://doi.org/10.1111/cogs.13309.URL https:// onlinelibrary.wiley.com/doi/abs/10.1111/cogs.13309. 435 Alan M. Turing. Computing machinery and intelligence. Mind, 59(236):433–460, 1950. doi: 10.1093/mind/LIX.236.433. 424 Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language mod- els don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems, volume 36, 2023. 426, 438 Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. 436 Keyon Vafa, Justin Y. Chen, Ashesh Rambachan, Jon Kleinberg, and Sendhil Mul- lainathan. Evaluating the world model implicit in a generative model. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 26941– 26975. Curran Associates, Inc., 2024. doi: 10.52202/079017-0846. URL https: //proceedings.neurips.c/paper_files/paper/2024/file/ 2f6a6317bada76b26a4f61b70a7db59-Paper-Conference.pdf. 432, 433 Keyon Vafa, Peter G. Chang, Ashesh Rambachan, and Sendhil Mullainathan. What has a foundation model found? using inductive bias to probe for world models, 2025. URL https://arxiv.org/abs/2507.06952. 433 Bram van Dijk, Tom Kouwenhoven, Marco Spruit, and Max Johannes van Duijn. Large language models: The need for nuance in current debates and a pragmatic perspective on understanding. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12641–12654, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.779. URL https://aclanthology.org/2023.emnlp-main.779/. 439 Max J. Van Duijn. The lazy mindreader. A humanities perspective on mindreading and multiple-order intentionality. PhD thesis, Leiden University, 2016. 434 Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Stein- hardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022. 426 Qirui Wang, Jingyi He, Yining Pan, Si Yong Yeo, Xulei Yang, and Shijie Li. Monosr: Open-vocabulary spatial reasoning from monocular images, 2026. URL https: //arxiv.org/abs/2511.19119. 432 456BIBLIOGRAPHY Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aky ̈ urek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1819–1862, Mexico City, Mexico, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.102. URL https://aclanthology.org/ 2024.naacl-long.102. 426, 430 Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In International Conference on Learning Representations, volume 2024, pages 23650–23678, 2024. 437 Wentao Zhu, Zhining Zhang, and Yizhou Wang. Language models represent beliefs of self and others. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 62638– 62681. PMLR, 2024.URL https://proceedings.mlr.press/v235/ zhu24o.html. 426, 434 Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shash- wat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2025. URL https://arxiv.org/abs/2310.01405. 443 Part VII Conclusion 457 459 Agentic LLM is a new field in which the technology of large language models is used for the creation of agents that act in the world. In the conclusion we discuss the impact of agentic LLMs, what the field can do, what current limitations are, and what the future may bring. 460 Chapter 23 Future Directions for Agentic LLMs Having arrived at the end of this book, we provide a brief high-level summary of the material that we have discussed. First we look back at our chapters about how agentic LLMs are built. Then we summarize our discussion of different applications of agentic LLMs and their behavior in single and multi-agent settings. Finally, in Section 23.2 we look at the limitations of current approaches, and what the future may bring. 23.1 The State of Agentic LLMs As discussed in Parts I, I, and I, the foundations of agentic LLMs lie in language modeling, reasoning, tool use, and different degrees of autonomy. 23.1.1 Technology Agentic LLMs are reasoning models that act in the world and adapt their behavior from feedback. After having been pretrained and finetuned (Chapter 3), at inference time they may access to external information (retrieval augmented generative AI, Chap- ter 12). They may also prompt themselves at inference time to verify results, leading to self-reflection and in-context reinforcement learning, Chapter 6. Agentic LLMs have become multi-modal, they operate in language, vision, and action domains (Chapter 7). To do so effectively, agentic LLMs can be provided with pretrained world models with a rudimentary understanding of the physical world. Through interacting with the envi- ronment, these world models may be improved at test time. The main machine learning architecture used is the transformer architecture. Mod- els are pretrained through self-supervision (next-token prediction) and finetuned on specific tasks through supervised and reinforcement learning. In-context chain of thought prompting has improved reasoning abilities of mod- els significantly. Inference-time methods have been developed for finetuning (such as RLVR and GRPO) and self-reflection (such as ReAct and reflexion, Chapter 6). A 461 462CHAPTER 23. FUTURE DIRECTIONS FOR AGENTIC LLMS whole spectrum of agent-environment interaction prompting styles have been devel- oped (Chapter 9). 23.1.2 Applications In Part IV, various tools have been introduced to ease the construction of agentic LLMs (Chapter 11). Agentic LLMs have been designed for many different applications. In e-commerce, they can help understand the preferences of their users and act on recom- mendations. In medicine, they can read and summarize diagnostic texts. In finance, they can analyze financial reports and provide recommendations to traders. In science and education, they can help in literature research, they can generate scripts to run experiments, they can analyze results, they can help write papers, and they can help review them (Chapter 13). 23.1.3 Behavior In Part V we shifted gears from looking at agentic LLMs through a technical lens, to studying how they behave in single-agent and multi-agent settings. After discussing ba- sic behavioral features of LLMs, in Chapter 15 we discussed how LLM-driven agents perform on tasks adapted from human behavioral approaches and frameworks devel- oped especially for AI benchmarking. Chapter 16 looked closely at multi-agent in- teractions that built on predefined roles and protocols. In Chapter 17 focus shifted to open-ended settings in which the object of study were the behavioral structures, norms, and conventions that emerged at system level from small-scale interactions between agents. Various recurring patterns were identified throughout the approaches and studies discussed in Part V. These include the discussions on construct validity (how do ob- servations generalize to the human situation), prompt sensitivity and cascading effects (especially in multi-agent settings), the knowing-doing gap (models’ capabilities at verbal predicting often did not translate well into actions), positive effects of structured reasoning (increasing reasoning budgets was particularly useful when structured along lines inspired on human reasoning), and emergent properties in desired directions (per- formance above baseline matched for compute budget) and undesired directions (norm violations) were observed. 23.2 Limitations and Future Work Agentic LLMs build on the power of large language models, and have greatly expanded the capabilities and uses of LLMs. However, there are still many limitations, which we will discuss next. We progress from memory, scale, explainability, and cognitive archi- tectures, which are more technical, to topics with broad implications such as potential sentience or consciousness and ethical implications that follow from such properties. For some of these topics, solution directions have been discussed in the preceding chapters. 23.2. LIMITATIONS AND FUTURE WORK463 23.2.1 Memory and Self-reflection Language models are functions: the output that they compute is only based on their input—they are memory-less between calls. In order for agents to act naturally in the world, to hold a conversation, to be able to learn from their actions by self-reflection, agents must be able to remember. For us, our identity hinges on our ability to remem- ber. Agents require a form of memory. A simple way to achieve a rudimentary form of memory is to feed the past interac- tion history back into the prompt—and current chatbots use this method with success. However, context windows remain limited in size, despite some success with research into long-context models. Agentic memory methods are an active area of research. LLMs should learn how to best compact information and store it in long or short term memory. Agents can access outside information through retrieval-augmented generative AI methods (RAG), for example from corporate databases. Elaborate agentic memory architectures are being developed to provide agents with longer term memory, either textual, or token embeddings, in vector databases (Section 9.3). 23.2.2 Scale As agents become able to self-reflect and to learn from interactions with their environ- ment, we should discuss the topic of scaling of the underlying language models. A major challenge in the development of large language models is how to fur- ther improve their performance. Originally, size was the major determinant—hence the name large language models—only models of sufficient size and trained on suffi- ciently large text corpuses achieved adequate benchmark performance, and only large models exhibited in-context learning. Scaling the size of the model and of the data thus appeared to be the way forward. However, scaling further is becoming more and more difficult, as most high-quality data that is available on the Internet has been now used to train current models. Also, training longer and larger models is expensive; time, money and power consumption are limiting this direction. Current methods are reaching a plateau [Sutskever, 2024]. To improve LLM performance further, it is important to continue to acquire train- ing data that is challenging (Chapter 19). One approach is to use the interactions of agentic LLMs with their environment as training data. When this interaction data is new, diverse, and high quality, it can be used for pretraining (see also Figure 1.1). Note that already inference-time methods such as GRPO and RLVR are using inference- time results for finetuning with good results [Guo et al., 2025], and VLAs are trained on self-generated action data [Black et al., 2024, Brohan et al., 2023], Chapter 7. Overthinking Scaling studies into chain of thought also reveal limitations of more thinking effort. Chain of thought works better for some domains than for others [Sprague et al., 2024]. Furthermore, deeper reasoning may lead to overthinking, where the rea- soning process becomes less effective as more effort is spent [Liu et al., 2024, Wu et al., 2025]. Studies are ongoing and reveal more insight into how chain of thought works [Chen et al., 2026b], and which mechanism are effective [Chen et al., 2026a]. 464CHAPTER 23. FUTURE DIRECTIONS FOR AGENTIC LLMS Open World Models and Curriculum Learning For the scaling approach to work it is important that synthetic training data is diverse, and different from the existing data. Open world models are studied to specifically generate diverse training environments. The idea of an open world-model is that it can create infinite datasets or environments, in which agents can continue to learn, to keep improving their intelligence [Bruce et al., 2024]. Open world models are inspired by evolutionary algorithms, quality diversity, and computer games [Vinyals et al., 2019, Mouret and Clune, 2015, Bakhtin et al., 2022]. Another approach to further scale model performance is to create better training methods. Curriculum learning is built on the observation that most learning works better if a system is first trained on easy tasks, before harder tasks are tried. (This holds both for human brains and for artificial neural networks.) In most domains a hierarchy of concepts exists that are related and build upon each other. (In school we learn addition before we learn long division, or even logarithms.) Curriculum methods are being tried for both pre- and posttraining [Chen et al., 2025, Elgaar and Amiri, 2026]. 23.2.3 Hallucination and Explainability Another limitation of current agentic LLMs is the problem of hallucination: models often provide answers that sound plausible but are factually incorrect, and their stated reasoning may not be causally responsible for the output they produce—a property known as faithfulness. The field of explainable AI (XAI) provides a spectrum of techniques to address this, ranging from behavioral and attribution-based methods such as SHAP and LIME, which treat the model as a black box, to concept-based interpretability, which probes internal representations for human-understandable features. Going further, mechanistic interpretability (MI), as discussed in Chapter 21, aims to reverse-engineer the compu- tational circuits inside transformer models. Tools such as sparse autoencoders, neural lenses and activation patching have al- ready been used to identify specific circuits responsible for hallucination and to provide empirical evidence that chain of thought reasoning can be a faithful, causally active component of model behavior in sufficiently large architectures. As agentic LLMs are increasingly deployed in high-stakes settings, MI offers a path toward hallucina- tion detectors that trigger when a model’s internal confidence features diverge from its external output—a critical step toward accountable agentic systems. A step further is the comparison of cognitive architectures between humans and transformers. Language models appear to produce natural language in a way that is similar to humans. Is that because the architectures that produce the utterances are similar, is it by coincidence, or is there another explanation? Chapter 22 explored this question. 23.2.4 Emotional Behavior and Consciousness Humans have a tendency to anthropomorphize: to attribute purposeful behavior to inan- imate objects in their surroundings, when they behave in ways that are similar to our 23.2. LIMITATIONS AND FUTURE WORK465 behavior, and sometimes even when they do not. Implicitly, we may attribute con- sciousness to such objects. This leads us to believe and say things such as the computer is thinking, the computer wants, and the computer must not be feeling well (or even this book loves to be read). LLMs suffer the same fate, and their utterances may from time to time persuade even the most rational philosophers to believe that they have a purpose (whereas they clearly cannot have a purpose, since they are merely algorithms that are being performed by a mechanical device—unlike humans, since we are organic). 1 Reports have been published about emotional relations between chatbots and hu- mans, sometimes leading to dangerous behavior, and worse. The topic of consciousness is actively debated by AI philosophers. Consciousness is generally assumed to mean being aware of oneself and the environment, and having subjective experiences, thoughts, and feelings [Velmans, 2009, Butlin et al., 2023]. But how do we test for consciousness? A pragmatic view is that for our society, behavior is what counts. If an entity acts, the consequences of its actions are what matters, and less so whether the entity was conscious or not. Bengio and Elmoznino [2025] argue that the mere illusion of consciousness, through actions, has consequences, and that it will become increasingly difficult for us to discern conscious entities from non-conscious entities. They argue that society must find answers on how to deal with agents that act, whether in medicine, finance, or science. Both credit and liability must ultimately be attributed in a fair way, and answers must be found on how to handle self-preserving AI systems. 23.2.5 Social Behavior We have discussed examples of how LLMs can simulate reasoning by humans (Chap- ter 15), and how LLMs can improve simulations of social behavior in multi-agent sys- tems. A limitation of current simulations of social behavior of agents is that the behav- ior is heavily influenced by the prompt script. Although some studies found realistic social behavior, many studies also found limits where the LLMs fail. Furthermore, the study of emergent behavior at scale poses challenges (Chap- ter 17). As with all multi-agent simulations, validation of the emergent behavior, to see how realistic the simulated behavior is, is a challenge, although some studies report impressive results [Park et al., 2024]. 23.2.6 Legal and Ethical Aspects Actions by assistants in the real world have real world consequences. Many legal and ethical challenges exist for agentic LLMs that act, when the actions of imperfect models have real world consequences. When a financial trading assistant hallucinates, or when a self driving robot makes a wrong inference, questions of responsibility and liability should be addressed, before such systems are deployed in the real world. A major area of research is to improve the safety on the technical side. Ensuring the safety of agentic LLMs requires moving beyond prompt-based defenses toward 1 Note that the duck-typing point of view disagrees, this behaviorist view holds that if it walks like a duck and quacks like a duck, then we consider it a duck. 466CHAPTER 23. FUTURE DIRECTIONS FOR AGENTIC LLMS integrated, multi-layer safeguards. In the field of safe reinforcement learning, a key direction is to include explicit safety constraints on the agent’s actions and within the agent’s reasoning and planning pipeline [Gu et al., 2024]. Another approach to improve safety is to employ continual adversarial training and automated red-teaming to enhance robustness against manipulation. Further priorities are developing mechanisms for self-regulation and risk awareness, enabling agents to detect and avoid unsafe actions, and establishing rigorous, standardized safety bench- marks [Andriushchenko et al., 2024]. Together, these measures outline a roadmap for accountable and trustworthy deployment of agentic LLMs in high-stakes domains. However, legal and ethical questions about who is responsible for the actions of an agent remain. Now that agents are starting to be employed in the real world, addressing these issues becomes of paramount importance. Clearly, many safety, ethics and trust issues will have to be addressed before the full breadth of the possibilities of agentic LLMs can be enjoyed. Safety and ethics will become important topics on the research agenda of agentic LLMs, deserving their own surveys and books [Gan et al., 2024, Jiao et al., 2025, Raza et al., 2025]. 23.2.7 Crime The discussion on ethics and consciousness becomes quite concrete when we consider criminal behavior. Technological developments can be used by people who act within the law, and those who act outside of the law. They can also be used for unethical behavior for which no effective laws yet exist. As agentic LLMs are able to display more and more believable behavior, it will become difficult to distinguish real and artificial behavior. This may lead to dangerous ethical situations, for example when criminals use the technology to influence the behavior of unsuspecting victims. The line between lawful persuasion, misuse of confidence, and fraud, can be un- clear. Agentic LLMs will greatly increase the sophistication of social engineering us- ing persuasive algorithms in social networks, marketing, espionage, and cybercrime [Kshetri, 2025, Lupinacci et al., 2025]. Although agentic LLM can also be used to combat crime [Challita and Parrend, 2025, Andriushchenko et al., 2024], it will be some time before societal awareness will have caught up with the risks. Especially people who are less familiar with social networks and LLMs may be susceptible to confidence tricks and agentic LLM cybercrime. 23.2.8 Limitations of Applications The list of applications of agentic LLMs is large. Proofs of concept have been created that impress us with their results, at first sight. When we look deeper at the results, limitations appear that may not be easy to fix and that may have serious consequences when the agents are used in the real world. 23.2.8.1 AI Science Many agentic LLMs are developed to support scientific research. All phases of the re- search pipeline are being supported. Tools for summarizing literature helps the ideation 23.2. LIMITATIONS AND FUTURE WORK467 phase, tools exist to generate scripts for running experiments in different fields, there is support for data interpretation and for creation of graphs, writing support exists, as well as tools to support the reviewing of scientific manuscripts. Indeed, it is becoming easier and easier to produce research papers that are well-written and that look good, on the surface. In mathematics, impressive results are reported that LLM agents perform on the level of math olympiads, both in geometry [Trinh et al., 2024] and in algebra [Hubert et al., 2025]. These systems include transformer networks, but are otherwise designed as mathematics solvers, using specialized automated theorem proving technology such as Lean [De Moura et al., 2015] and reinforcement learning [Silver et al., 2017, Browne et al., 2012]. In 2025 a benchmark study was performed on AI science tools on a range of do- mains from machine learning [Nathani et al., 2025]. Using then-current frontier mod- els, the authors did not find that models could generate novel hypotheses, algorithms, architectures, or make substantial scientific improvements. In particular, the authors note that modern LLM agents can successfully tackle a diverse array of quantitative experiments, reflecting advanced skills and domain adaptability but also that it is not yet clear if the notion of scientific novelty can be successfully automated or even for- mally defined in a form suitable for agents. 23.2.8.2 E-commerce Liability for actions is a major problem for deploying agentic LLMs in e-commerce. Security and privacy concerns will inhibit users from providing personal details and preference to leaky computer systems. This leads to the catch-22 situation of recom- mender systems: when users are unwilling to provide elaborate information about their true preferences, the quality of suggestions will be low, users will have low trust in the system, and will be even less willing to provide information. Only when it is easy to collect high quality information about preferences, in a safe and secure way, and when the models are able to provide strong and useful suggestions, will agentic LLM for e-commerce be adopted by the public. 23.2.8.3 Trading For financial trading liability is of paramount importance. Studies report that LLMs can be better than human analysts in summarizing large volumes of financial reports. However, in financial trading the stakes are high. Wrong trading advice will directly lead to financial loss. Furthermore, given the unpredictable nature of financial markets, it may take some time to recognize an erroneous model, since it will not be clear when an individual loss is caused be an imperfection in the model or just by an unlucky trade. Models should be thoroughly tested before they will be trusted enough to be used. Unless bulletproof, automated trading systems will not go beyond support of human decision making. LLMs in finance are mostly successful in analyzing textual reports. For time series analysis of financial data some results are published, but it is hard to achieve reliable and verifiable benchmark results. Although some studies claim good results, others 468CHAPTER 23. FUTURE DIRECTIONS FOR AGENTIC LLMS report that predictions by agentic LLM of financial market results remain in doubt [Li et al., 2026]. 23.2.8.4 Medicine Accuracy and especially explainability of results are even more important for trustwor- thiness in medicine. In medicine, model predictions are directly related to liability and ethics of actions. For medical applications, explainability of results is essential, and agentic LLM systems will include a human in the decision making loop. As in trading, the most promising results of agentic LLM in medicine are in the analysis and summarization of textual documents, such as meeting reports or litera- ture surveys. Results in other applications, such as decisions on treatment plans, or performing operations, are limited. Medicine might well be one of the most valuable and at the same time one of the most conservative fields of application for agentic LLM. In matters of life and death, we should be extra careful, and at the same time the potential gains are large. 23.3 Concluding Remarks Agentic LLM technology is developing at a rapid pace. This book has traced that development across three interconnected pillars: reasoning, acting, and interacting. Reasoning LLMs provide the cognitive foundation, through chain of thought, self- reflection, and retrieval augmentation: models have become capable of multi-step prob- lem solving, literature synthesis, and complex decision support. Acting agents extend this foundation into the real world, using tools, APIs, and world models to assist users in domains from robotics to finance to medicine. Interacting agents go further still, engaging with one another in multi-agent simulations that exhibit emergent social be- haviors such as cooperation, trust, and norm formation, that were not explicitly scripted in the prompts. These three pillars are connected. As Figure 1.1 illustrates, they form a cycle: rea- soning enables better acting and interacting, while the data generated by (inter)acting agents can be used to pretrain and finetune better reasoning models. This virtuous cycle offers a promising path beyond plateaus in LLM scaling, as inference-time interaction data from agentic assistants, robotic systems, and multi-agent simulations, provides a continuously replenishing source of high-quality, diverse training signals. Yet, significant challenges remain across all three pillars. Hallucination undermines trust in reasoning chains and threatens the stability of self-improving systems. Safety, liability, and ethical accountability are unresolved for acting agents in high-stakes do- mains. Emergent behavior in large-scale multi-agent systems remains difficult to val- idate and control. Further research in the fields of mechanistic interpretability, open world models, scalable simulation infrastructures, and robust safety benchmarks is re- quired to overcome these challenges. We are at an early and consequential moment. Proofs of concept are impressive; real-world deployments are beginning; and the theoretical foundations—of memory, metacognition, consciousness, and collective intelligence—remain open questions that BIBLIOGRAPHY469 connect artificial intelligence to philosophy, cognitive science and law. This book has aimed to give you the technical grounding and conceptual vocabulary to contribute to these questions. The field needs researchers who understand both the power and the limits of what agentic LLMs can do today, and who can help shape what they will become tomorrow. Bibliography Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrik- son, et al. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint arXiv:2410.09024, 2024. 466 Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Sci- ence, 378(6624):1067–1074, 2022. 464 Yoshua Bengio and Eric Elmoznino. Illusions of ai consciousness. Science, 389(6765): 1090–1091, 2025. 465 Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. 휋 0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. 463 Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt- 2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 463 Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samoth- rakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Trans- actions on Computational Intelligence and AI in games, 4(1):1–43, 2012. 467 Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Ed- ward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. 464 Patrick Butlin, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, Axel Constant, George Deane, Stephen M. Fleming, Chris Frith, Xu Ji, Ryota Kanai, Colin Klein, Grace Lindsay, Matthias Michel, Liad Mudrik, Megan A. K. Peters, Eric Schwitzgebel, Jonathan Simon, and Rufin VanRullen. Consciousness in artifi- cial intelligence: Insights from the science of consciousness, 2023. 465 470BIBLIOGRAPHY Brian Challita and Pierre Parrend. Redteamllm: an agentic ai framework for offensive security. arXiv preprint arXiv:2505.06913, 2025. 466 Wei-Lin Chen, Liqian Peng, Tian Tan, Chao Zhao, Blake JianHang Chen, Ziqian Lin, Alec Go, and Yu Meng. Think deep, not just long: Measuring llm reasoning effort via deep-thinking tokens. arXiv preprint arXiv:2602.13517, 2026a. 463 Xi Chen, Aske Plaat, and Niki van Stein. How does chain of thought think? mech- anistic interpretability of chain-of-thought reasoning with sparse autoencoding. In AAAI2026, arXiv preprint arXiv:2507.22928, 2026b. 463 Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Pich ́ e, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for llm reasoning. arXiv preprint arXiv:2505.14970, 2025. 464 Leonardo De Moura, Soonho Kong, Jeremy Avigad, Floris Van Doorn, and Jakob von Raumer. The lean theorem prover (system description). In International Conference on Automated Deduction, pages 378–388. Springer, 2015. 467 Mohamed Elgaar and Hadi Amiri. Curriculum learning for llm pretraining: An analysis of learning dynamics. arXiv preprint arXiv:2601.21698, 2026. 464 Yuyou Gan, Yong Yang, Zhe Ma, Ping He, Rui Zeng, Yiming Wang, Qingming Li, Chunyi Zhou, Songze Li, Ting Wang, et al. Navigating the risks: A sur- vey of security, privacy, and ethics threats in LLM-based agents. arXiv preprint arXiv:2411.09523, 2024. 466 Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):11216– 11235, 2024. 466 Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qi- hao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 463 Thomas Hubert, Rishi Mehta, Laurent Sartran, Mikl ́ os Z Horv ́ ath, Goran ˇ Zu ˇ zi ́ c, Eric Wieser, Aja Huang, Julian Schrittwieser, Yannick Schroecker, Hussain Masoom, et al. Olympiad-level formal mathematical reasoning with reinforcement learning. Nature, pages 1–3, 2025. 467 Junfeng Jiao, Saleh Afroogh, Yiming Xu, and Connor Phillips. Navigating LLM ethics: Advancements, challenges, and future directions. AI and Ethics, pages 1–25, 2025. 466 Nir Kshetri. Transforming cybersecurity with agentic ai to combat emerging cyber threats. Telecommunications Policy, 49(6):102976, 2025. 466 BIBLIOGRAPHY471 Weixian Waylon Li, Hyeonjun Kim, Mihai Cucuringu, and Tiejun Ma. Can llm-based financial investing strategies outperform the market in long run? In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pages 2711–2722, 2026. 468 Ryan Liu, Jiayi Geng, Addison J Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L Griffiths. Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse. arXiv preprint arXiv:2410.21333, 2024. 463 Matteo Lupinacci, Francesco Aurelio Pironti, Francesco Blefari, Francesco Romeo, Luigi Arena, and Angelo Furfaro. The dark side of llms: Agent-based attacks for complete computer takeover. arXiv preprint arXiv:2507.06850, 2025. 466 Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015. 464 Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. Mlgym: A new framework and benchmark for advancing ai re- search agents. arXiv preprint arXiv:2502.14499, 2025. 467 Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Mered- ith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109, 2024. 465 Shaina Raza, Ranjan Sapkota, Manoj Karkee, and Christos Emmanouilidis. Respon- sible agentic reasoning and ai agents: A critical survey: Proposal for safe agentic AI via responsible reasoning ai agents (r2a2). SuperIntelligence-Robotics-Safety & Alignment, 2(6), 2025. 466 David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mas- tering the game of go without human knowledge. nature, 550(7676):354–359, 2017. 467 Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wad- hwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409.12183, 2024. 463 Ilya Sutskever. Acceptance speech for test of time award. In Advances in Neural Information Processing Systems, Vancouver, British Columbia, Canada, 2024. 463 Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482, 2024. 467 Max Velmans. Understanding consciousness. Routledge, 2009. 465 472BIBLIOGRAPHY Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Micha ̈ el Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft i using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019. 464 Yuyang Wu, Yifei Wang, Ziyu Ye, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms. arXiv preprint arXiv:2502.07266, 2025. 463

Instant research discovery

Agentic Large Language Models

Intelligence

Summary

Entities (18)

Relation Signals (14)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations