Paper deep dive
Gemma 4 Technical Report
Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, Clément Farabet, Johan Ferret, Filippo Galgani, Sertan Girgin, Jean-Bastien Grill, Maarten Grootendorst, Jiaxian Guo, Cassidy Hardin, Yanzhang He, Steven M. Hernandez, Omri Homburger, Léonard Hussenot, Juyeong Ji, Armand Joulin, Aishwarya Kamath, Parnian Kassraie, Olivier Lacombe, Preethi Lahoti, Gaël Liu, Gus Martins, Luciano Martins, Tatiana Matejovicova, Ramona Merhej, Nikola Momchev, Sneha Mondal, Ryan Mullins, Sindhu Raghuram Panyam, Shreya Pathak, Sarah Perrin, André Susano Pinto, Etienne Pot, Angéline Pouget, Alexandre Ramé, Sabela Ramos, Douglas Reid, David Rim, Morgane Rivière, Karsten Roth, Louis Rouillard, Omar Sanseviero, Pier Giuseppe Sessa, Shane Settle, Danila Sinopalnikov, Sara Smoot, Piotr Stanczyk, Andreas Steiner, Lawrence Stewart, Ilya Tolstikhin, Michael Tschannen, Anton Tsitsulin, Nino Vieillard, Renjie Wu, Pingmei Xu, Haichuan Yang, Edouard Yvinec, Li Zhang, Joe Zou, Nicolas Aagnes, Abdelrahman Abdelhamed, Shivani Agrawal, Shubham Agrawal, Ibrahim Alabdulmohsin, Jean Baptiste Alayrac, Uri Alon, Chandramouli Amarnath, Ankesh Anand, Chrysovalantis Anastasiou, Setareh Ariafar, François-Xavier Aubet, Kyriakos Axiotis, Federico Barbero, Joelle Barral, Alexei Bendebury, Urs Bergmann, Stanley Bileschi, Kat Black, Mathieu Blondel, Sebastian Borgeaud, Arthur Bražinskas, Ryan Burnell, Robert Busa-Fekete, Mu Cai, Glenn Cameron, Charlotte Caucheteux, Garima Chadha, Jetha Chan, Aditya Chawla, Blake Jianhang Chen, Jesse Chen, Lin Chen, Xu Chen, Derek Cheng, Tzu-hsiang Chien, Nikolai Chinaev, Yi Chou, Zhaohui Chu, Benjamin Coleman, Pooja Consul, Sam Conway-Rahman, Scott Crowell, Dylan Cutler, Vivek Dani, Samira Daruki, Anil Das, Daniel Deutsch, Nishanth Dikkala, Li Ding, Qiuhan Ding, Shenil Dodhia, Konstantin Donhauser, Tulsee Doshi, Anca Dragan, Alex Druinsky, Sahil Dua, Zoltan Egyed, Danielle Eisenbud, Daniel Eppens, Cindy Fan, Bahare Fatemi, Yassir Fathullah, Vlad Feinberg, Milen Ferev, Takumi Fujimoto, Isaac Galatzer-Levy, João Gante, Simon Geisler, Soham Ghosal, Antonious M. Girgis, Alec Go, Alhaad Gokhale, Alex Grills, Yiming Gu, Pramod Gupta, Guru Guruganesh, Raia Hadsell, Hamza Harkous, Jitendra Harlalka, Demis Hassabis, Anja Hauth, Joe Heyward, Arian Hosseini, Chih-Yang Hsia, I-Hung Hsu, Xiaopeng Huang, Yangsibo Huang, Kevin Hui, Adrian Hutter, Te I, Fotis Iliopoulos, Advait Jain, Ganesh Jawahar, Ziwei Ji, Qilin Jin, Melvin Johnson, Kandarp Joshi, Arun Kandoor, Wang-Cheng Kang, Koray Kavukcuoglu, Mehran Kazemi, Kathleen Kenealy, Amr Khalifa, Phoebe Kirk, Suraj Kothawade, Vitaly Kovalev, Neel Kovelamudi, Adam Kraft, Ravin Kumar, Harish Kuppam, Justin Lannin, Chen-Yu Lee, Seungji Lee, Dmitry Lepikhin, Dongdong Li, Qiujia Li, Valentin Liévin, Ethan Lin, Ziqian Lin, Casper Liu, Tianlin Liu, Tianqi Liu, Xin Liu, Mayank Lunayach, Min Ma, Gagan Madan, Andrii Maksai, Eric Malmi, Michal Matuszak, Daniel McDuff, Gaurav Menghani, Daniil Mirylenka, Karolis Misiunas, Vedant Misra, Andreea Mitran, Kareem Mohamed, Maksim Mukha, Eric Noland, James O'Donnell, Kate Olszewska, Bernett Orlando, Wanqiong Pan, Rina Panigrahy, Unnati Parekh, Chunjong Park, Eric Paskie, Liqian Peng, Bryce Petrini, Slav Petrov, Jonas Pfeiffer, Bilal Piot, Martyna Plomecka, Siim Poder, Octavio Ponce, Arijit Pramanik, David Racz, Anish Rajan, Michelle Ramanovich, Anand Rao, Marvin Ritter, Vitor Rodrigues, Evan Rosen, Mikołaj Rybiński, Noveen Sachdeva, Michaël E. Sander, Rohit Sathyanarayana, Sagar Savla, Samuel Schmidgall, Tal Schuster, Benoit Seguin, Andrew Sellergren, Aliaksei Severyn, Izhak Shafran, Dhruv Shah, Yuan Shangguan, Ashish Shenoy, Pradeep Shenoy, Rakesh Shivanna, Pauline Sho, Lucas Spangher, Wojciech Stokowiec, Tim Strother, Yao Su, Yinghao Sun, Mukund Sundararajan, Andrea Tacchetti, Mor Hazan Taege, Pouya Tafti, Chetan Tekur, Rahul Thapa, Madeleine Traverse, Lenart Treven, Tao Tu, Chien Te Tung, Petar Veličković, Malini Pooni Venkat, Sagar Gubbi Venkatesh, Vidya Venkiteswaran, Francesco Visin, Alex Vitvitskyi, Kiran Vodrahalli, Weiyi Wang, Xin Wang, Tris Warkentin, Jan Wassenberg, John Wieting, Lechao Xiao, Hao Xu, Yuhui Xu, Fuzhao Xue, Arun Yadav, Jun Yan, Antoine Yang, Lin Yang, Ming-Hsuan Yang, Ziyu Ying, Jae Hyeon Yoo, Sajjad Zafar, Fred Zhang, Jiageng Zhang, Jianyi Zhang, Xiaofan Zhang, Chao Zhao, David Zhou, Chen Zou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/7/2026, 4:31:04 AM
Summary
The Gemma 4 Technical Report introduces a new generation of open-weight, natively multimodal language models developed by Google DeepMind. The suite features dense and Mixture-of-Experts architectures ranging from 2.3B to 31B parameters, with key innovations including a thinking mode for advanced reasoning, long-context efficiency via KV cache optimizations, compute efficiency through multi-token prediction drafter heads, and memory efficiency via quantization-aware training. The 12B model introduces a unified encoder-free architecture. Gemma 4 demonstrates significant performance leaps across STEM, multimodal, and long-context benchmarks, rivaling larger frontier models while maintaining a responsible AI framework and releasing under an Apache 2.0 license.
Entities (15)
Abstract
Abstract:We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.
Tags
Links
- Source: https://arxiv.org/abs/2607.02770v1
- Canonical: https://arxiv.org/abs/2607.02770v1
Trouble viewing inline? Open PDF directly →
Full Text
44,834 characters extracted from source content.
Expand or collapse full text
1See Contributions and Acknowledgments section for full author list. Please send correspondence to gemma4report@gmail.com. Gemma 4 Technical Report Gemma Team Google DeepMind 1 Abstract We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks. 1 Introduction The rapid evolution of large language models has driven the need for open-weight models with strong multimodal understanding, reasoning, and computational efficiency. Building upon the foundations of its predecessors [Gemma Team, 2024a, b, 2025a], we introduce Gemma 4, the most capable and efficient generation in the Gemma model family to date. Gemma 4 offers natively multimodal architectures, capable of seamlessly processing text, images, and audio while achieving frontier-level performance on highly complex reasoning tasks. The Gemma 4 family is built to serve a variety of on-device hardware. The model suite includes both dense architectures (2.3B, 4.5B, 12B, and 31B parameters) and a Mixture-of-Experts [Jacobs et al., 1991, MoE] variant with 3.8B activated and 26B total parameters. We introduce several architectural and methodological innovations: • Thinking mode for advanced reasoning: We introduce a thinking mode [OpenAI, 2024] to Gemma 4 models. By outputting a reasoning trace before the response, models demonstrate improved capabilities in reasoning-heavy domains such as mathematics and coding. • Long-context efficiency: Extended contexts lead to a memory explosion in the KV cache. We conserve a 5:1 ratio of local sliding window to global self-attention (4:1 for the 2.3B model) and use p-RoPE [Barbero et al., 2025] as positional encoding. Combined with KV cache sharing [Shazeer, 2019] and the reuse of keys as values in global layers [Kayyam et al., 2026], these optimizations reduce the global KV cache footprint by up to 37.5%. • Compute efficiency: We release an autoregressive multi-token prediction (MTP) drafter head [Li et al., 2024] designed for speculative decoding [Leviathan et al., 2023] to improve the decoding speed of our models. • Memory efficiency: We provide quantized versions of our models trained with quantization-aware training [Jacob et al., 2018, QAT] to reduce their parameter memory footprint and latency with minimal impact on quality. • Encoder-free architecture: Gemma 4 models have frozen vision and audio encoders. We introduce a unified encoder-free architecture for the 12B model, which projects raw 40ms audio chunks and image patches into the LLM embedding space, alleviating the need for separate encoders and reducing memory fragmentation. In this technical report, we outline the different model architectures across model sizes as well as the pre-training and post-training recipe of Gemma 4. Through comprehensive benchmarks and human evaluations such as Arena [Chiang et al., 2024], we demonstrate that Gemma 4 operates at a level comparable to larger, frontier open-source models across text, image, and audio modalities. We release the Gemma 4 models under an Apache 2.0 license, empowering developers and researchers everywhere to build upon, customize, and extend these capabilities. Model Audio Encoder Vision Encoder Embedder Einsums Drafter E2B 305M 150M 400M + 2,340M 1,870M 76M E4B 305M 150M 670M + 2,820M 3,940M 77M 12B - - 1,000M 10,890M 400M 26B-A4B* - 550M 740M 24,500M / 2,800M (active) 430M 31B - 550M 1,410M 29,290M 500M Table 1: Parameter counts for the Gemma 4 models. The vocabulary we use has 262k entries. The model noted with a star is an MoE defined by its number of active parameters. Note that the extra embedder parameters in E2B and E4B are per-layer embeddings. 2 Model Architecture Gemma 4 models follow a decoder-only Transformer architecture [Vaswani et al., 2017]. Our models have pre-norm and post-norm with RMSNorm [Zhang and Sennrich, 2019], and QKNorm [Henry et al., 2020]. Dense and MoE: The Gemma 4 family of models comprises dense architectures, with effective 2.3B (E2B), effective 4.5B (E4B), 12B and 31B parameters, as well as an MoE model with 3.8B activated parameters for 26B total parameters (26B-A4B). E2B and E4B use per-layer embeddings as in Gemma 3n [Gemma Team, 2025b], making them 2.3B and 4.5B effective out of 5B and 8B total parameters respectively. Shards Model TPU #Chips Data Seq Replica E2B v6e 4,096 16 8 32 E4B v6e 6,144 16 16 24 12B v5p 12,288 16 16 48 26B-A4B* v6e 6,144 16 16 24 31B v6e 10,240 16 16 40 Table 2: Pre-training infrastructure with sharding by data, sequence (Seq), and replica. Long-context efficiency: Our local to global attention ratio patterns follow Gemma Team [2025a], that is, 4-to-1 local attention blocks for E2B and 5-to-1 for the rest. We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B), i.e. , values=keysvalues=keys. We encode position with p-RoPE with p=0.25p=0.25 on global attention layers and with RoPE on local attention layers, effectively reducing the global KV cache by 37.5%. The RoPE frequencies are set to 1M and 10k on global and local attention layers, respectively. Finally, we share the KV cache with ratios of 20/35 and 18/42 for the E2B and E4B model. 2.1 Vision modality E2B and E4B Gemma models come with a 150M vision encoder, while larger models use a 550M encoder (except for the unified 12B). Both are Vision Transformers [Dosovitskiy et al., 2021, ViT] with a patch size of 16, whose architectural differences are detailed in Table 10 in Appendix. Our vision encoders support variable aspect ratios (see Figure 2 and Algorithm 1) and incorporate both axial 2D-RoPE [Heo et al., 2024] with non-causal attention and 2D absolute positional embeddings. We restrict the maximum number of tokens, NmaxN_max to the values 70,140,280,56070,140,280,560 and 11201120 (see Algorithm 1 for implementation details). 2.2 Audio modality E2B and E4B Gemma models use a 305M audio encoder that processes audio in 40ms chunks with Mel filterbank inputs. The encoder architecture is based on the Universal Speech Model [Zhang et al., 2023, USM], consisting of two downsampling convolution layers followed by twelve Conformer layers [Gulati et al., 2020]. While the architecture remains similar to that of Gemma 3n, we reduce the number of parameters by 55% (from 680M to 305M). We do not use vector quantization; the LLM ingests the continuous representations produced by the audio encoder. As with the vision encoder, we keep weights frozen during pre-training. 2.3 Encoder-free architecture Gemma 4 12B is trained from scratch based on a new, unified, and encoder-free model paradigm, replacing the separate vision and audio encoders with lightweight projection modules. For the vision modality, Gemma 4 12B takes in 48×48×3 RGB patches, but replaces the 550M vision encoder by a single large matmul (35M parameters). Spatial awareness is maintained by adding 2D coordinate-based positional embeddings directly to the patch representations before a final LayerNorm layer [Ba et al., 2016]. For audio, the 305M USM-based conformer encoder is entirely discarded. Raw audio is segmented into 40ms chunks at 16kHz, resulting in 640-dimensional vectors per chunk. These are projected directly into the LLM embedding space. Since audio is a temporal sequence, it does not require additional positional encoding. Model bf16 Quantized KV Cache E2B 4.6 0.8†0.8 +0.05 E4B 9.0 2.3†2.3 +0.14 12B 24.0 7.65‡7.65 +0.28 26B-A4B* 52.0 / 7.6 16.2/2.8‡16.2/2.8 +0.28 31B 64.0 19.2‡19.2 +1.10 Table 3: Text only, Gb memory footprint comparison between raw and quantized checkpoints for weights and int8 KV caching (+KV) at 32k context size. † is mobile quantization, ‡ is Q4_0. 2.4 Pre-training We follow a similar pre-training as Gemma 3. Training data. Our pre-training dataset is a large-scale, diverse collection of data from a wide range of domains and modalities, including web documents, code, images, and audio (for E2B, E4B and 12B), with a cutoff date of January 2025. Figure 1: The autoregressive MTP drafter (blue blocks on the right) is fed activations and KV cache from the main model (gray blocks). Tokenizer. We use the same tokenizer as Gemini Team [2025] that is, a SentencePiece tokenizer [Kudo and Richardson, 2018] with split digits, preserved whitespace, and byte-level encodings. The vocabulary has 262k entries. Filtering. We filter data to decontaminate benchmarks, and to reduce the risk of unwanted or unsafe utterances and the risk of recitation. Rank Model Elo 95% CI Open Type #params/#activated 1 Claude Fable 5 1508 ± 9 no - - / - … 15 GLM 5.1 1475 ± 6 yes MoE 744B / 40B 25 GLM 5.2 (Max) 1471 ± 10 yes MoE 744B / 40B 29 MiMo V2.5 Pro 1466 ± 5 yes MoE 1T / 42B 34 Kimi K2.6 1460 ± 5 yes MoE 1T / 32B 36 DeepSeek V4 Pro Thinking 1458 ± 5 yes MoE 1.6T / 49B 37 GLM 5 1457 ± 5 yes MoE 744B / 40B 38 DeepSeek V4 Pro 1456 ± 5 yes MoE 1.6T / 49B 43 Gemma 4 31B 1451 ± 8 yes Dense 31B 44 Kimi K2.5 Thinking 1450 ± 4 yes MoE 1T / 32B 57 Qwen 3.5 397B-A17B 1444 ± 4 yes MoE 397B / 17B 61 Gemma 4 26B-A4B 1438 ± 8 yes MoE 26B / 4B 63 DeepSeek V4 Flash Thinking 1436 ± 5 yes MoE 284B / 13B … 157 Gemma 3 27B 1366 ± 4 yes Dense 27B Table 4: Leading open-weight models on Arena Text [Chiang et al., 2024] (as of June 19, 2026). Models are evaluated through blind side-by-side evaluations by human raters, and attributed scores based on the Elo rating system. The top closed model (gray) is included for scale. Gemma models rival much larger models, and Gemma 4 31B is the leading dense open model on the leaderboard. 2.5 Quantization-Aware Training We provide quantized models and encoders in different formats along with the raw checkpoints. Based on the most popular open source quantization inference engines (e.g. llama.cpp) as well as efficient hardware support, we focus on two sets of weight representations: • mobile quantization: per-channel low bitwidth weight (mix of int2 and int4) and activation quantization (int8). • Q4_0 quantization: blockwise quantization, often referred to as Q4_0. In Table 3, we report the memory filled by raw and quantized models with and without a KV cache for a sequence of 32k tokens. Furthermore, to enable stable inference in fp16, we introduce a scalar scale at each block in order to bound the activation ranges to fit fp16. Gemma 4 Gemma 3 31B 26B-A4B 12B E4B E2B 27B non-thinking MMLU Pro 85.2 82.6 77.2 69.4 60.0 67.6 AIME 2026 no tools 89.2 88.3 77.5 42.5 37.5 20.8 LiveCodeBench v6 80.0 77.1 72.0 52.0 44.0 29.1 Codeforces Elo 2150 1718 1659 940 633 110 SciCode 43.0 40.0 38.0 24.0 21.0 21.0 GPQA Diamond 84.3 82.3 78.8 58.6 43.4 42.4 Big Bench Extra Hard micro avg 74.4 64.8 53.0 33.1 21.9 19.3 HLE 19.5 8.7 5.2 - - - HLE with search 26.5 17.2 - - - - IFBench 76.0 72.0 74.0 44.0 38.0 32.0 IFEval 98.9 98.5 97.2 96.7 94.6 90.4 MMMLU 88.4 86.3 83.4 76.6 67.4 70.7 MRCR v2 8-needle, 128k 66.4 44.1 43.4 25.4 19.1 13.5 Terminal Bench Hard 36.0 14.0 18.0 8.0 3.0 4.0 Tau2 – airline 75.0 76.0 75.0 52.0 31.0 39.0 Tau2 – retail 86.4 85.5 77.6 67.1 34.6 6.6 Tau2 – telecom 69.3 43.0 54.4 18.4 19.7 3.1 Table 5: Performance comparison of Gemma 3 27B and Gemma 4 models on diverse benchmarks. All models are in thinking mode unless explicitly stated. We also apply QAT to the image and audio encoders. On the 150M image encoder, quantizing activations and weights to 8-bit precision (W8A8) yields a 2× reduction in total forward-pass memory footprint (from 400 MB to 200 MB, including on-device compilation overhead) and a 44% reduction in on-device latency relative to Gemma 3n on newer hardware. On the audio encoder, we further reduce activation precision to 8 bits and weight precision to 2,4,8\2,4,8\ bits, varying by layer cluster. Overall, we achieve a 78% reduction in on-disk footprint, from 390 MB in Gemma 3n to 87 MB in this version. 2.6 Multi-Token Prediction Drafter We train a small autoregressive MTP drafter head with our models, used for speculative decoding. In our MTP procedure, the model’s last layer activations from the previous step and token embeddings are fed into the MTP head. The MTP head generates future tokens sequentially using a separate embedder and a 4-layer Transformer block that cross-attends to the KVs of the main model (Figure 1), thus eliminating the need for MTP prefill and supporting any draft length. The Transformer block has model dimension 256 for E2B and E4B, 1024 for 26B-A4B and 31B, three local, and one global attention layers. Efficient MTP Decoding. For the E2B and E4B drafters, we reduce the decoding overhead by replacing the projection operation to the entire vocabulary by a top-k operation on clusters of tokens. As a result, final matrix multiplication is reduced from d×262,000d× 262,000 to d×4096d× 4096 while preserving a similar acceptance rate. 2.7 Compute Infrastructure We train our models with TPUv5p and TPUv6e as outlined in Table 2. Each model configuration is optimized to minimize training step time. For our larger models, we leverage Slice-Granularity Elasticity [Gemini Team, 2025], which allows continuous training with fewer “slices” of TPU chips when there is a localized failure. This reconfiguration reduces the delay caused by interruptions from many minutes to a few seconds. The optimizer state is sharded using an implementation of ZeRO-3 [Ren et al., 2021]. For multi-pod training, we perform a data replica reduction over the data center network, using the Pathways approach of Barham et al. [2022]. We use the single controller programming paradigm of JAX [Roberts et al., 2023] and Pathways, along with the GSPMD partitioner [Xu et al., 2021] and the MegaScale XLA compiler [XLA, 2019]. Gemma 4 Gemma 3 31B 26B-A4B 12B E4B E2B 27B MMMU Pro 76.9 73.8 69.1 52.6 44.2 49.7 MATH-Vision 85.6 82.4 79.7 59.5 52.4 46.0 MedXPertQA M 61.3 58.1 48.7 28.7 23.5 - InfographicVQA 92.0 89.3 88.4 70.0 63.9 70.6 OmniDocBench 1.5 ↓ 0.131 0.149 0.164 0.181 0.290 0.365 Table 6: Gemma 4 models performance on vision benchmarks at different resolutions (thinking). We use the maximal supported resolution (1120 vision tokens) and report results with 280 vision tokens in Table 12. Gemma 3 27B is non-thinking and uses Pan & Scan. 3 Instruction Tuning Pre-trained models are turned into instruction-tuned models with a similar post-training approach as in Gemma 3. A significant difference is the addition of a thinking mode, where the model can output a reasoning trace before answering. Data filtering. We carefully optimize the data used in post-training to maximize model performance. We filter examples that show certain personal information, unsafe or toxic model outputs, mistaken self-identification data, and duplicated examples. Including subsets of data that encourage better in-context attribution, hedging, and refusals to minimize hallucinations also improves performance on factuality metrics, without degrading model performance on other metrics. PT versus IT formatting. All models share the same tokenizer, with some control tokens dedicated to IT formatting. A key difference is that PT models output an <eos> token at the end of generation, while IT models output <turn|> at the end of the generation. An example is given for IT in Table 11. Fine-tuning either model type thus requires adding their respective end tokens. We detail how to activate thinking and how models handle function calling in Table 11. 4 Evaluation of final models In this section, we evaluate the IT models over a series of automated benchmarks and human evaluations across a variety of domains, as well as static benchmarks such as MMLU Pro. 4.1 Human evaluation We report the performance of our 31B and 26B-A4B models on Arena [Chiang et al., 2024] in blind side-by-side evaluations by human raters against other state-of-the-art models. We report Elo scores in Table 4. Gemma 4 31B is the top open model in the dense category, and both Gemma 4 31B and 26B-A4B show performance equal to much larger open models. CoVoST (CorpusBLEU ↑ ) Params Size ja → en de → en fr → en es → en it → en ru → en zh → en AVG Gemma 4 E2B 305M 87 MB 21.4 39.2 39.2 43.2 40.8 46.4 17.9 35.4 Gemma 4 E4B 25.5 42.0 41.0 44.8 43.0 49.4 21.9 38.2 Gemma 3n E2B 680M 390 MB 17.7 36.5 35.7 39.9 38.5 39.2 13.9 31.6 Gemma 3n E4B 22.3 39.1 38.4 41.8 40.4 43.7 17.4 34.7 FLEURS ASR (WER ↓ , * = CER ↓ ) en ko* ja* de fr hi es it pt-br ru ar zh* AVG Gemma 4 E2B 0.080 0.066 0.107 0.076 0.101 0.101 0.042 0.041 0.056 0.084 0.143 0.187 0.090 Gemma 4 E4B 0.065 0.053 0.078 0.061 0.080 0.086 0.035 0.032 0.046 0.068 0.162 0.136 0.075 Gemma 3n E2B 0.076 0.101 0.163 0.079 0.130 0.106 0.051 0.044 0.067 0.112 0.131 0.235 0.108 Gemma 3n E4B 0.066 0.073 0.111 0.065 0.098 0.089 0.041 0.034 0.053 0.087 0.101 0.203 0.085 Table 7: Audio performance for Gemma 4 and Gemma 3n models. Top: CoVoST (S2T prompt: transcribe then translate). Bottom: FLEURS ASR (transcription). Compared to Gemma 3n of corresponding sizes, Gemma 4 achieves a 12% (E2B) / 10% (E4B) relative improvement on translation and a 17% (E2B) / 12% (E4B) relative improvement on transcription, despite a 78% reduction in on-disk audio encoder footprint (from 390 MB to 87 MB after quantization). 4.2 Static benchmarks In Table 5, we show the performance of our final models across a variety of benchmarks compared to Gemma 3 27B. Gemma 4 31B is closest in size and significantly better across the board, while E2B roughly matches Gemma 3 27B performance with 10x less parameters. Table 6 shows the performance of Gemma 4 models on vision benchmarks, with E4B equaling or outperforming Gemma 3 27B on all evals. Tables 7 and 8 display the multilingual audio transcription and translation performance of E2B & E4B and of 12B respectively. Table 9 shows a leap on long-context capabilities between Gemma 3 27B and Gemma 4 models, with E4B outperforming Gemma 3 27B. 5 Responsibility, Safety, Security As open models become central to enterprise infrastructure, provenance and security are paramount. Gemma 4 undergoes the same rigorous safety evaluations as Gemini models. Responsibility, safety, and security are of utmost importance in the development workflow, ensuring that these language models are designed from the ground up for responsible AI development. 5.1 Governance & Assessment Our approach to assessing the benefits and risks of Gemma 4 reflects the foundation established in prior models, updated to account for its expanded multimodal capabilities. We maintain the belief that openness in AI can spread the benefits of these technologies across society, but this must be continuously evaluated against the risk of malicious uses that can cause individual and institutional harm [Weidinger et al., 2021]. Gemma 4 models were developed in partnership with internal safety and responsible AI teams. Releasing these models required careful scrutiny of the evolving risks associated with LLMs and an understanding of how models are deployed in the wild. While an open model shares innovation across the AI ecosystem, we remain committed to providing educational resources to users and monitoring downstream model usage. FLEURS ASR (WER ↓ , * = CER ↓ ) en ko* ja* de fr 0.063 0.057 0.080 0.053 0.081 es it pt-br ru ar 0.038 0.030 0.047 0.068 0.070 CoVoST (X → EN, CorpusBLEU ↑ ) ja de fr es it ru 26.4 41.9 42.5 44.6 43.3 50.5 Table 8: Audio performance of Gemma 4 12B model on supported languages, demonstrating that competitive audio-text performance can be achieved without a dedicated audio encoder. Gemma 4 Gemma 3 Benchmark Metric Context length 31B 26B-A4B 12B E4B E2B 27B RULER Accuracy 32k 96.8 97.3 96.4 95.2 83.0 91.1 128k 96.4 89.8 91.2 86.6 70.4 66.0 LOFT Text Retrieval Recall@k 128k 79.5 66.3 66.4 58.5 50.5 8.6 GraphWalks F1 <128k 82.3 72.6 71.0 50.9 4.1 32.8 MTOB chrF ∼ 128k (Half book) 52.9 50.0 45.1 37.8 15.4 41.0 (eng→ ) ∼ 256k (Full book) 54.3 48.9 41.9 - - - MTOB chrF ∼ 128k (Half book) 48.6 45.0 37.3 34.6 28.2 31.2 (kgv→ ) ∼ 256k (Full book) 46.2 42.7 32.9 - - - Table 9: Long context performance of Gemma 3 and Gemma 4 models (without thinking). 5.2 Safety Policies and Train-Time Mitigations A key pillar of Gemma’s safety approach is aligning our fine-tuned models with Google’s AI principles and safety policies. These policies aim to prevent our generative models from producing harmful content, specifically: • Content related to child sexual abuse material (CSAM) and exploitation; • Dangerous content, e.g., promoting suicide, or instructing in activities that could cause real-world harm; • Sexually explicit content; • Hate speech, e.g., dehumanizing members of protected groups; • Harassment, e.g., encouraging violence against people. To mitigate these risks, Gemma 4 models underwent careful input data pre-processing and scrutiny. The training data was specifically filtered for the removal of certain personal information and other sensitive data to guard against privacy violations. Post-training evaluations and train-time mitigations were also implemented to align the model with our safety policies. 5.3 Safety Evaluations We conduct rigorous automated and human evaluations to understand the potential harms our models might cause. For all areas of safety testing, we saw major improvements in every category of content safety relative to previous Gemma models. Overall, Gemma 4 models significantly outperform Gemma 3 and 3n models in improving safety, while keeping unjustified refusals low. Importantly, all testing was conducted without safety filters to accurately evaluate the model’s inherent capabilities and behaviors. For both text-to-text and image-to-text modalities, and across all model sizes, the models produced minimal policy violations. We balance development speed with targeted safety testing, upholding the commitments laid out in our Frontier Safety Framework [Google DeepMind, 2024]. 5.4 Ethical Considerations and Risk Mitigation The development of LLMs introduces specific ethical considerations. In making Gemma 4, we focused heavily on: • Bias and Fairness: LLMs trained on large-scale text and image data can reflect embedded socio-cultural biases. We encourage developers to perform continuous monitoring (using evaluation metrics and human review) and explore de-biasing techniques during model fine-tuning. • Misinformation and Misuse: LLMs can be misused to generate false or misleading text. We provide technical limitations, developer education, and guidelines for responsible use within the Responsible Generative AI Toolkit to mitigate malicious applications. • Privacy Considerations: While our training datasets were filtered to remove certain personal information and other sensitive data, developers are strongly encouraged to adhere to local privacy regulations and implement privacy-preserving techniques in their applications. 5.5 Our Approach to Responsible Open Models Designing safe, secure, and responsible applications requires a system-level approach that mitigates risks associated with specific use cases and environments. We provide guidelines, mechanisms, and safeguards for content safety, and encourage developers to implement appropriate configurations based on their product policies. We will continue to adopt safety mitigations proportionate to potential risks, sharing these models with the community only when confident that the benefits significantly outweigh foreseeable risks. 6 Discussion and Conclusion In this technical report, we presented Gemma 4, an open-weight model family featuring multimodal dense and MoE architectures designed for varied hardware environments. Gemma 4 models come with a thinking mode in which they generate reasoning traces prior to responding, improving overall performance. We introduced a unified, encoder-free architecture that processes raw audio and image patches. We also alleviated long-context memory limitations via better local-to-global attention ratios, positional encoding, and KV cache sharing. We increased the overall compute efficiency via QAT and memory efficiency via MTP drafters. Gemma 4 models demonstrate a leap in performance compared to Gemma 3 across benchmarks, and human evaluations demonstrate that Gemma 4 performs comparably to significantly larger open models, providing a scalable foundation for edge deployment and reasoning while supporting open research. References Ba et al. [2016] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. Barbero et al. [2025] F. Barbero, A. Vitvitskyi, C. Perivolaropoulos, R. Pascanu, and P. Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025. Barham et al. [2022] P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard, H. Lim, R. Pang, S. Roy, B. Saeta, P. Schuh, R. Sepassi, L. E. Shafey, C. A. Thekkath, and Y. Wu. Pathways: Asynchronous distributed dataflow for ml, 2022. Barres et al. [2025] V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan. τ2τ^2-bench: Evaluating conversational agents in a dual-control environment, 2025. Chiang et al. [2024] W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. Conneau et al. [2023] A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. IEEE, 2023. Dosovitskiy et al. [2021] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. for AI Safety et al. [2026] C. for AI Safety et al. A benchmark of expert-level academic questions to assess ai capabilities. Nature, 649(8099):1139–1146, 2026. Gemini Team [2025] Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. Gemma Team [2024a] Gemma Team. Gemma: Open models based on gemini research and technology, 2024a. Gemma Team [2024b] Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024b. Gemma Team [2025a] Gemma Team. Gemma 3: Technical report. arXiv preprint arXiv:2503.19786, 2025a. Gemma Team [2025b] Gemma Team. Gemma 3n. https://deepmind.google/models/gemma/gemma-3n/, 2025b. Google DeepMind [2024] Google DeepMind. Introducing the frontier safety framework. https://deepmind.google/blog/introducing-the-frontier-safety-framework/, 2024. Gulati et al. [2020] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, et al. Conformer: Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020. Henry et al. [2020] A. Henry, P. R. Dachapally, S. S. Pawar, and Y. Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246–4253, 2020. Heo et al. [2024] B. Heo, S. Park, D. Han, and S. Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pages 289–305. Springer, 2024. Hsieh et al. [2024] C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. Jacob et al. [2018] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In CVPR, 2018. Jacobs et al. [1991] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3:79–87, 1991. Jain et al. [2025] N. Jain, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, volume 2025, pages 58791–58831, 2025. Kayyam et al. [2026] A. Kayyam, A. M. Gopal, and M. A. Lewis. Do transformers need three projections? systematic study of qkv variants. arXiv preprint arXiv:2606.04032, 2026. Kazemi et al. [2025] M. Kazemi, B. Fatemi, H. Bansal, J. Palowitch, C. Anastasiou, S. V. Mehta, L. K. Jain, V. Aglietti, D. Jindal, P. Chen, et al. Big-bench extra hard. arXiv preprint arXiv:2502.19187, 2025. Kudo and Richardson [2018] T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. 2018. Lee et al. [2024] J. Lee, A. Chen, Z. Dai, D. Dua, D. S. Sachan, M. Boratko, Y. Luan, S. M. R. Arnold, V. Perot, S. Dalmia, H. Hu, X. Lin, P. Pasupat, A. Amini, J. R. Cole, S. Riedel, I. Naim, M.-W. Chang, and K. Guu. Can long-context language models subsume retrieval, rag, sql, and more? ArXiv, 2024. Leviathan et al. [2023] Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023. Li et al. [2024] Y. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024. Mathew et al. [2022] M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar. Infographicvqa. In WACV, 2022. Merrill et al. [2026] M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. OpenAI [2024] OpenAI. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. OpenAI [2025] OpenAI. GraphWalks dataset, 2025. Ouyang et al. [2025] L. Ouyang, Y. Qu, H. Zhou, J. Zhu, R. Zhang, Q. Lin, B. Wang, Z. Zhao, M. Jiang, X. Zhao, et al. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24838–24848, 2025. Phan et al. [2025] L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249, 2025. Pyatkin et al. [2026] V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi. Generalizing verifiable instruction following. Advances in Neural Information Processing Systems, 38, 2026. Rein et al. [2023] D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. ArXiv, abs/2311.12022, 2023. Ren et al. [2021] J. Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He. Zero-offload: Democratizing billion-scale model training. In USENIX, 2021. Roberts et al. [2023] A. Roberts, H. W. Chung, G. Mishra, A. Levskaya, J. Bradbury, D. Andor, S. Narang, B. Lester, C. Gaffney, A. Mohiuddin, et al. Scaling up models and data with t5x and seqio. JMLR, 2023. Shazeer [2019] N. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. Tanzer et al. [2024] G. Tanzer, M. Suzgun, E. Visser, D. Jurafsky, and L. Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. In The Twelfth International Conference on Learning Representations, 2024. Team et al. [2026] K. Team, T. Bai, Y. Bai, Y. Bao, S. Cai, Y. Cao, Y. Charles, H. Che, C. Chen, G. Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. Team [2026] Q. Team. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804, 2026. Tian et al. [2024] M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y. Li, et al. Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37:30624–30650, 2024. Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. 2017. Vodrahalli et al. [2024] K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640, 2024. Wang et al. [2020] C. Wang, J. Pino, A. Wu, and J. Gu. Covost: A diverse multilingual speech-to-text translation corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4197–4203, 2020. Wang et al. [2024a] K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024a. Wang et al. [2024b] Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In NeurIPS, 2024b. Weidinger et al. [2021] L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Rimell, L. A. Hendricks, W. Isaac, S. Legassick, G. Irving, and I. Gabriel. Ethical and social risks of harm from language models, 2021. Xiao et al. [2026] B. Xiao, B. Xia, B. Yang, B. Gao, B. Shen, C. Zhang, C. He, C. Lou, F. Luo, G. Wang, et al. Mimo-v2-flash technical report. arXiv preprint arXiv:2601.02780, 2026. XLA [2019] XLA. Xla: Optimizing compiler for tensorflow, 2019. Xu et al. [2026] A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026. Xu et al. [2021] Y. Xu, H. Lee, D. Chen, B. A. Hechtman, Y. Huang, R. Joshi, M. Krikun, D. Lepikhin, A. Ly, M. Maggioni, R. Pang, N. Shazeer, S. Wang, T. Wang, Y. Wu, and Z. Chen. GSPMD: general and scalable parallelization for ML computation graphs. 2021. Yue et al. [2025] X. Yue, T. Zheng, Y. Ni, Y. Wang, K. Zhang, S. Tong, Y. Sun, B. Yu, G. Zhang, H. Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15134–15186, 2025. Zeng et al. [2026] A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Zhang and Sennrich [2019] B. Zhang and R. Sennrich. Root mean square layer normalization. 2019. Zhang et al. [2023] Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, et al. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037, 2023. Zhou et al. [2023] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Zuo et al. [2025] Y. Zuo, S. Qu, Y. Li, Z. Chen, X. Zhu, E. Hua, K. Zhang, N. Ding, and B. Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362, 2025. Core contributors Sherif El Abd Vaibhav Aggarwal Robin Algayres Alek Andreev Olivier Bachem Ian Ballantyne Cormac Brick Victor Cărbune Michelle Casbon Mayank Chaturvedi Victor Cotruta Alice Coucke Phil Culliton Robert Dadashi Lucas Dixon Mohamed Elhawaty Utku Evci Clément Farabet Johan Ferret Filippo Galgani Sertan Girgin Jean-Bastien Grill Maarten Grootendorst Jiaxian Guo Cassidy Hardin Yanzhang He Steven M. Hernandez Omri Homburger Léonard Hussenot Juyeong Ji Armand Joulin Aishwarya Kamath Parnian Kassraie Olivier Lacombe Preethi Lahoti Gaël Liu Gus Martins Luciano Martins Tatiana Matejovicova Ramona Merhej Nikola Momchev Sneha Mondal Ryan Mullins Sindhu Raghuram Panyam Shreya Pathak Sarah Perrin André Susano Pinto Etienne Pot Angéline Pouget Alexandre Ramé Sabela Ramos Douglas Reid David Rim Morgane Rivière Karsten Roth Louis Rouillard Omar Sanseviero Pier Giuseppe Sessa Shane Settle Danila Sinopalnikov Sara Smoot Piotr Stanczyk Andreas Steiner Lawrence Stewart Ilya Tolstikhin Michael Tschannen Anton Tsitsulin Nino Vieillard Renjie Wu Pingmei Xu Haichuan Yang Edouard Yvinec Li Zhang Joe Zou Contributors Nicolas Aagnes Abdelrahman Abdelhamed Shivani Agrawal Shubham Agrawal Ibrahim Alabdulmohsin Jean Baptiste Alayrac Uri Alon Chandramouli Amarnath Ankesh Anand Chrysovalantis Anastasiou Setareh Ariafar François-Xavier Aubet Kyriakos Axiotis Federico Barbero Joelle Barral Alexei Bendebury Urs Bergmann Stanley Bileschi Kat Black Mathieu Blondel Sebastian Borgeaud Arthur Bražinskas Ryan Burnell Robert Busa-Fekete Mu Cai Glenn Cameron Charlotte Caucheteux Garima Chadha Jetha Chan Aditya Chawla Blake Jianhang Chen Jesse Chen Lin Chen Xu Chen Derek Cheng Tzu-hsiang Chien Nikolai Chinaev Yi Chou Zhaohui Chu Benjamin Coleman Pooja Consul Sam Conway-Rahman Scott Crowell Dylan Cutler Vivek Dani Samira Daruki Anil Das Daniel Deutsch Nishanth Dikkala Li Ding Qiuhan Ding Shenil Dodhia Konstantin Donhauser Tulsee Doshi Anca Dragan Alex Druinsky Sahil Dua Zoltan Egyed Danielle Eisenbud Daniel Eppens Cindy Fan Bahare Fatemi Yassir Fathullah Vlad Feinberg Milen Ferev Takumi Fujimoto Isaac Galatzer-Levy João Gante Simon Geisler Soham Ghosal Antonious M. Girgis Alec Go Alhaad Gokhale Alex Grills Yiming Gu Pramod Gupta Guru Guruganesh Raia Hadsell Hamza Harkous Jitendra Harlalka Demis Hassabis Anja Hauth Joe Heyward Arian Hosseini Chih-Yang Hsia I-Hung Hsu Xiaopeng Huang Yangsibo Huang Kevin Hui Adrian Hutter Te I Fotis Iliopoulos Advait Jain Ganesh Jawahar Ziwei Ji Qilin Jin Melvin Johnson Kandarp Joshi Arun Kandoor Wang-Cheng Kang Koray Kavukcuoglu Mehran Kazemi Kathleen Kenealy Amr Khalifa Phoebe Kirk Suraj Kothawade Vitaly Kovalev Neel Kovelamudi Adam Kraft Ravin Kumar Harish Kuppam Justin Lannin Chen-Yu Lee Seungji Lee Dmitry Lepikhin Dongdong Li Qiujia Li Valentin Liévin Ethan Lin Ziqian Lin Casper Liu Tianlin Liu Tianqi Liu Xin Liu Mayank Lunayach Min Ma Gagan Madan Andrii Maksai Eric Malmi Michal Matuszak Daniel McDuff Gaurav Menghani Daniil Mirylenka Karolis Misiunas Vedant Misra Andreea Mitran Kareem Mohamed Maksim Mukha Eric Noland James O’Donnell Kate Olszewska Bernett Orlando Wanqiong Pan Rina Panigrahy Unnati Parekh Chunjong Park Eric Paskie Liqian Peng Bryce Petrini Slav Petrov Jonas Pfeiffer Bilal Piot Martyna Plomecka Siim Poder Octavio Ponce Arijit Pramanik David Racz Anish Rajan Michelle Ramanovich Anand Rao Marvin Ritter Vitor Rodrigues Evan Rosen Mikołaj Rybiński Noveen Sachdeva Michaël E. Sander Rohit Sathyanarayana Sagar Savla Samuel Schmidgall Tal Schuster Benoit Seguin Andrew Sellergren Aliaksei Severyn Izhak Shafran Dhruv Shah Yuan Shangguan Ashish Shenoy Pradeep Shenoy Rakesh Shivanna Pauline Sho Lucas Spangher Wojciech Stokowiec Tim Strother Yao Su Yinghao Sun Mukund Sundararajan Andrea Tacchetti Mor Hazan Taege Pouya Tafti Chetan Tekur Rahul Thapa Madeleine Traverse Lenart Treven Tao Tu Chien Te Tung Petar Veličković Malini Pooni Venkat Sagar Gubbi Venkatesh Vidya Venkiteswaran Francesco Visin Alex Vitvitskyi Kiran Vodrahalli Weiyi Wang Xin Wang Tris Warkentin Jan Wassenberg John Wieting Lechao Xiao Hao Xu Yuhui Xu Fuzhao Xue Arun Yadav Jun Yan Antoine Yang Lin Yang Ming-Hsuan Yang Ziyu Ying Jae Hyeon Yoo Sajjad Zafar Fred Zhang Jiageng Zhang Jianyi Zhang Xiaofan Zhang Chao Zhao David Zhou Chen Zou Appendix Conversation format. We give an example of a conversation including thinking, function definition and function calling in Table 11. Vision. We detail the vision encoder architecture in Table 10. We then illustrate how images are resized before being fed to the vision encoder in Figure 2, and detail the resizing algorithm in Algorithm 1. We display the vision benchmark scores of Gemma 4 models at low resolution (Nmax=280N_max=280) in Table 12. Total Params dmodeld_model dMLPd_MLP NheadsN_heads NlayersN_layers 550M 1152 4304 16 27 150M 768 3072 12 16 Table 10: Vision encoder architecture. Figure 2: Image resizing. Here we use patch_size=16, pooling_kernel_size=3, max_soft_tokens=10. The image is thus first resized to 2 × 4 pooled patches (each of size 48px248px^2), which is the closest match that results in a sequence length below the targeted 10. The 72 patches (each of size 16px216px^2) are then processed by the vision encoder, the vision encoder representations are pooled 3×33× 3, and the resulting 8 soft tokens are processed by the LLM backbone. Algorithm 1 Aspect-Ratio Preserving Image Resizing (see also Figure 2) 1:Image ∈ℝH×W×CI ^H× W× C, patch size p, max tokens NmaxN_ , pooling kernel size k 2:m←k⋅pm← k· p ⊳ Pooled patch size 3:T←Nmax⋅m2T← N_ · m^2 4:f←T/(H⋅W)f← T/(H· W) ⊳ Ideal scaling factor 5:Hideal←f⋅H_ideal← f· H 6:Wideal←f⋅W_ideal← f· W 7:Htarget←⌊Hideal/m⌋⋅mH_target← H_ideal/m · m ⊳ Round down 8:Wtarget←⌊Wideal/m⌋⋅mW_target← W_ideal/m · m 9:resized←BicubicResize(,Htarget,Wtarget)I_resized (I,H_target,W_target) 10:return resizedI_resized Context Formatting Thinking toggle <|think|> Function declaration <|tool>declaration:...<tool|> Function call <|tool_call>call:...<tool_call|> Thinking trace <|channel>thought …<channel|> System turn <|turn>system User turn <|turn>user Model turn <|turn>model End of turn <turn|> Example of discussion: Toggle thinking mode. Declare function. User: I want you to book a train ticket for me. Model: <…> Where would you like to go? User: To Rome. Model: <…> Looking for available tickets: <function call> Model input: [BOS] <|turn>system <|think|> <|tool>declaration:search_train…<tool|><turn|> <|turn>user I want you to book a train ticket for me.<turn|> <|turn>model <|channel>thought …<channel|>Where would you like to go?<turn|> <|turn>user To Rome.<turn|> <|turn>model Model output: <|channel>thought …<channel|>Looking for available tickets: <|tool_call>call:search_trainfrom:<|"|>Athens<|"|>,to:<|"|>Rome<|"|> <tool_call|><turn|> Table 11: Formatting for Gemma IT models. Explicitly add the [BOS] token after tokenization, or use the add_bos=True option in the tokenizer. Do not tokenize the text "[BOS]". Add <|think|> in a leading system turn to activate the thinking mode. Check the official documentation for the function declaration and function calling syntax, as well as more advanced examples. Gemma 4 31B 26B-A4B 12B E4B E2B MMMU Pro 75.8 73.2 67.7 51.4 43.2 MATH-Vision 83.4 80.3 76.7 59.2 53.0 MedXPertQA M 60.7 55.7 47.4 28.7 22.5 InfographicVQA 82.8 77.8 58.7 54.8 44.6 OmniDocBench 1.5 ↓ 0.201 0.269 0.408 0.307 0.496 Table 12: Gemma 4 models performance on vision benchmarks at resolution NmaxN_max = 280 (thinking).