Paper deep dive
RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks
Jinjun Huang, Zhongzhen Wen, Tongtong Xu, Meng Yan, Xin Xia, Zhongxin Liu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In modern AI frameworks, GPU kernels are key to overall system performance. Combining usability, portability, and near-handwritten CUDA performance, Triton is widely adopted for implementing GPU kernels. Recent advances show the potential of large language models (LLMs) to automatically generate Triton kernels, reducing the manual effort required from expert kernel developers. Several benchmarks evaluate LLM-generated Triton kernels. However, they suffer from three key limitations: (1) they restrict tasks to PyTorch-to-Triton translation, failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only individual-kernel performance rather than end-to-end performance, the core criterion for real-world deployment in AI frameworks; and (3) they rely on manually written evaluation scripts for individual kernels, which may contain flaws that models can exploit to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation. RealisticTritonBench systematically extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires a corresponding Triton kernel implementation, with a complete and reproducible evaluation environment. Unlike prior benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests, enabling a more faithful assessment. We evaluate leading LLMs on RealisticTritonBench and find that they still struggle with real-world Triton kernel generation tasks.
Tags
Links
- Source: https://arxiv.org/abs/2608.12004v1
- Canonical: https://arxiv.org/abs/2608.12004v1
Trouble viewing inline? Open PDF directly →
Full Text
78,616 characters extracted from source content.
Expand or collapse full text
RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks Jinjun Huang huanghuanghuang@zju.edu.cn College of Computer Science and Technology and the State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou, China Zhongzhen Wen wenzhongzhen@smail.nju.edu.cn State Key Lab for Novel Software Technology, Nanjing University Nanjing, China Tongtong Xu xutongtong9@huawei.com Software Engineering Application Technology Laboratory, Huawei Hangzhou, China Meng Yan mengy@cqu.edu.cn The School of Big Data and Software Engineering, Chongqing University Chongqing, China Xin Xia xin.xia@acm.org College of Computer Science and Technology and the State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou, China Zhongxin Liu ∗ liu_zx@zju.edu.cn College of Computer Science and Technology and the State Key Laboratory of Blockchain and Data Security, Zhejiang University Hangzhou, China Abstract In modern AI frameworks, the GPU kernel is a key determinant of overall system performance. By combining usability, portabil- ity, and near-handwritten CUDA performance, Triton has been widely adopted for implementing GPU kernels. Recent advances have shown the potential of using large language models (LLMs) to automatically generate Triton kernels, helping reduce the manual effort required by expert kernel developers. To evaluate the quality of LLM-generated Triton kernels, several benchmarks have been proposed. However, existing benchmarks primarily target isolated kernel generation tasks and suffer from three key limitations: (1) they restrict the task to only PyTorch-to-Triton translation, thus failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only the performance of individual kernels, limiting the evaluation of real-world performance of generated ker- nels in AI frameworks, where end-to-end performance is the core criterion for real deployment; (3) they rely on manually written evaluation scripts for single kernel, which may introduce vulner- abilities and allow models to exploit evaluation flaws to bypass correctness checks and obtain inflated scores. To address these lim- itations, we introduce RealisticTritonBench, the first benchmark that derives Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling evaluation under real- istic, production-like settings. RealisticTritonBench systematically extracts real-world PRs with Triton kernel modified from popular open-source AI frameworks and transforms them into kernel gen- eration tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires to generate a ∗ Corresponding author. This work is licensed under a Creative Commons Attribution 4.0 International License. ASE ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2882-2/2026/10 https://doi.org/10.1145/3832783.3837457 corresponding Triton kernel implementation. RealisticTritonBench also provides a complete and reproducible evaluation environment for each task. In contrast to prior benchmarks that focus on iso- lated kernel performance, RealisticTritonBench integrates gener- ated kernels into their original frameworks and evaluates them under end-to-end test, enabling a more faithful assessment. We conduct a systematic evaluation of the state-of-the-art LLMs on RealisticTritonBench, revealing that they still struggle to handle the challenges in real-world Triton generation tasks. CCS Concepts • Software and its engineering→ Automatic programming. Keywords Large Language Models, Triton, Deep Learning Kernels ACM Reference Format: Jinjun Huang, Zhongzhen Wen, Tongtong Xu, Meng Yan, Xin Xia, and Zhongxin Liu. 2026. RealisticTritonBench: A Benchmark for Triton- Kernel Generation in Real-World AI Frameworks. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3832783.3837457 1 INTRODUCTION Currently, modern AI models are typically deployed on training and inference frameworks, such as Pytorch [29], and Deepspeed [32]. Within these frameworks, GPU kernels constitute a critical compo- nent that determines system latency and throughput, motivating en- gineers to invest significant effort in developing high-performance kernels, as inefficient implementations can increase latency, reduce throughput, and raise the overall cost of large-scale model deploy- ment [12]. However, the implementation of correct and highly optimized GPU kernels using low-level native languages such as CUDA is inherently challenging and time-consuming [38]. To ad- dress these challenges, Triton [38] has emerged as a widely adopted high-level GPU programming language. It has been prevalently arXiv:2608.12004v1 [cs.SE] 12 Aug 2026 ASE ’26, October 12–16, 2026, Munich, GermanyJ. Huang, Z. Wen, T. Xu, M. Yan, X. Xia, and Z. Liu # An Example for Vector Addition @triton.jit def add_kernel(x_ptr, y_ptr, output_ptr, n_elements, BLOCK_SIZE: tl.constexpr): pid = tl.program_id(axis=0) block_start = pid * BLOCK_SIZE offsets = block_start + tl.arange(0, BLOCK_SIZE) mask = offsets < n_elements x = tl.load(x_ptr + offsets, mask=mask) y = tl.load(y_ptr + offsets, mask=mask) output = x + y tl.store(output_ptr + offsets, output, mask=mask) Figure 1: A Triton kernel for Vector Addition integrated into modern AI frameworks and optimization pipelines, serving as a preferred choice for implementing high-performance GPU kernels [13,18,51]. As a Python-based domain-specific lan- guage (DSL) for GPU kernel development, Triton significantly sim- plifies the implementation of complex kernels while maintaining performance competitive with native CUDA. As shown in Figure 1, a Triton kernel typically resembles a Python function decorated with @triton.jit, where developers explicitly define block-level par- allelism, memory access patterns, and tensor computation through Triton primitives. Owing to its ease of use, flexibility, and strong performance, Triton has gained substantial traction in modern AI systems and has become a key building block for optimized criti- cal kernels in contemporary AI frameworks. However, such DSLs have not fully eliminated the complexities of performance tuning. Developers are still required to reason about low-level execution details, including memory access patterns, parallelization strategies, and hardware-specific optimizations. Hence, current research in automated Triton generation has attracted increasing attention. More recently, a line of work has explored leveraging large lan- guage models (LLMs) to automatically synthesize Triton kernels. Several benchmarks have been proposed to evaluate models’ capa- bilities of generating Triton kernels[20,28,41,46]. These bench- marks provide an initial testbed for assessing the functional correct- ness and performance of generated kernels, helping LLM-assisted kernel generation rapidly progress. Representative approaches, in- cluding AutoTriton[21], QiMeng-Kernel [52], and KernelFalcon [30], investigate LLM-based GPU kernel generation and reveal the poten- tial of LLMs for automating Triton kernel generation. However, we find these key limitations in existing benchmarks: (1) Limited task diversity. Existing benchmarks predominantly focus on translating PyTorch reference implementations into Triton kernels [20,28,46]. In contrast, the real-world Triton generation task encompasses a broader range of activities, including performance optimization, bug fixing, and feature extension, as these tasks typically involve modifying existing Triton kernel implementations to meet new functional or performance requirements. These practical develop- ment scenarios remain insufficiently explored in existing studies. (2) Lack of framework-level evaluation. Existing benchmarks often evaluate kernels in isolation, only considering kernel-level metrics such as functional correctness and speedup compared to PyTorch implementations[20,28]. Such kernel-level evaluation overlooks the real impact of deploying generated kernels within AI frameworks. In practice, Triton kernels must interact with complex runtime systems, memory management strategies, distributed execution pipelines, and framework-specific abstractions. Without end-to- end evaluation in real-world AI frameworks, it is difficult to assess their genuine effectiveness and robustness with kernel-level evalua- tion. (3) Vulnerability in evaluation pipeline. Most benchmarks rely on manually written evaluation scripts[20,28], which may intro- duce vulnerabilities and allow models to exploit evaluation flaws to obtain misleading results. Some researchers have observed that models may exploit flaws in evaluation scripts to bypass correctness checks, thereby obtaining inflated scores[24, 33]. To address these limitations, we present RealisticTritonBench to enable evaluation under realistic environments. RealisticTriton- Bench has several characteristics: 1) Diverse tasks: we construct RealisticTritonBench from real-world Triton-related pull requests, covering a wide range of development activities, including per- formance optimization, modification, and new kernel addition, rather than limiting to PyTorch-to-Triton translation tasks. 2) Multi- dimensional evaluation: To better approximate real-world deploy- ment scenarios, we provide kernel-level unit tests, model accuracy tests, and end-to-end speed tests for tasks whenever possible. By in- corporating end-to-end accuracy and performance evaluation, our benchmark directly measures the real impact of generated kernels within complete frameworks. 3) Mitigate Hacking in Evaluation. Since end-to-end metrics directly reflect the model’s actual perfor- mance, improvements at this level correspond to actual performance gains. As a result, it becomes difficult to exploit evaluation flaws without genuine improvements in system performance, thereby reducing the risk of misleading results. We evaluate a diverse set of state-of-the-art LLMs on Realis- ticTritonBench, such as Qwen3.5 and GPT-5.4. To systematically measure task completion, we design these metrics for comprehen- sive evaluation: full test pass rate (FTP), unit test pass rate (UTP), numerical robustness for the model (NR), and end-to-end accelera- tion speedup. Together, these metrics form a realistic, system-level evaluation framework for generated kernels. To ensure comprehen- sive evaluation, our end-to-end testing pipeline is applied across a diverse set of models, covering different architectures and capa- bilities, summarized in our online appendix[14]. Based on these metrics, we systematically study the performance, robustness, and failure reasons of LLMs on RealisticTritonBench. Our experiments show that models achieve only 18.71% average task success rate, maintain model accuracy in 47.65% of cases, and present limited end-to-end speedup (approximately 1× on average), which indi- cates that generating correct Triton kernels in real-world tasks remains challenging, and the generated implementations often lead to degraded model accuracy or increased end-to-end latency. In summary, this paper makes the following contributions: •We curate the RealisticTritonBench dataset, providing a founda- tion for evaluating the performance of AI-generated kernels in realistic environments. •We propose a comprehensive evaluation pipeline for Triton ker- nel generation, which spans unit testing to end-to-end system- level testing, closely simulating real-world kernel validation pro- cedures. •We conduct an extensive evaluation of state-of-the-art language models and LLM-based agents on RealisticTritonBench, revealing RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI FrameworksASE ’26, October 12–16, 2026, Munich, Germany that even the SOTA LLMs still struggle with real-world Triton kernel generation tasks, and the generated kernels frequently lead to degraded model accuracy and increased end-to-end latency. 2 BACKGROUND 2.1 LLM Training and Inference Frameworks Recently LLMs have been widely applied to a variety of tasks including code generation[3], mathematical reasoning[35], video generation[42], and multimodal understanding[19]. As model scale and capability continue to grow, efficiently training and deploy- ing these large-scale models has become increasingly significant. To address this challenge, researchers and engineers have devel- oped various training and inference frameworks. For example, in the training task, frameworks such as PyTorch[29], Tensorflow[1] DeepSpeed[32], and Megatron-LM[37] provide essential capabili- ties including automatic differentiation, gradient optimization, fault tolerance, and large-scale distributed training. Although general- purpose deep learning frameworks such as PyTorch and TensorFlow are widely used for LLM inference, they are primarily designed to support a broad range of hardware platforms and model archi- tectures. As a result, they do not specifically optimize the core decoding process of LLM inference, which may lead to suboptimal performance and inefficient resource utilization. To improve inference efficiency, a number of specialized LLM inference engines have recently been developed[9,18,50,51]. These systems introduce a variety of optimizations tailored for LLM infer- ence and serving, including techniques such as dynamic batching[2] and inference-specific operators[48]. By incorporating these opti- mizations, modern inference engines provide features such as effi- cient batching, optimized attention computation, and distributed inference, enabling LLMs to be deployed and executed more effi- ciently on GPU hardware. 2.2 Triton for High-Performance GPU Kernel Development As deep learning models continue to grow in scale, GPU compu- tational efficiency has become a key factor affecting both training and inference performance. In GPU programming, a GPU kernel refers to a function that runs on the GPU and is executed in parallel by a large number of threads, typically performing data-parallel computations on tensors or matrices. These kernels constitute the fundamental building blocks of many operations in deep learn- ing systems. Although general deep learning frameworks such as PyTorch provide a rich set of built-in operators, many performance- critical computations in real-world systems still rely on customized GPU kernels to achieve higher efficiency. For example, in scenarios like attention computation, matrix operations, and sparse computa- tion, kernels optimized for specific data layouts and memory access patterns can significantly reduce latency. Traditionally, high-performance GPU kernels are implemented using CUDA, Nvidia’s parallel computing platform[41]. However, writing efficient CUDA kernels is complex and requires deep exper- tise in GPU architecture, including thread scheduling and memory management. To reduce this burden while retaining high perfor- mance, Triton[38] has emerged as a domain-specific language for GPU programming. It provides a Python-based abstraction that enables developers to write high-level kernels, while the compiler handles low-level optimizations such as parallelization and memory access. By combining high-level programmability with performance close to hand-written CUDA kernels, Triton significantly reduces the development complexity of high-performance GPU operators. As a result, Triton has been increasingly adopted in modern deep learning systems to implement GPU kernels. For example, systems such as vLLM[18] and libraries such as FlashAttention[4] use Triton to implement high-performance GPU kernels in their core compo- nents. However, implementing efficient Triton kernels still requires substantial effort for developers, which motivates efforts to explore the use of large language models for automatic kernel generation. 3 RealisticTritonBench RealisticTritonBench aims to evaluate the ability of large language models (LLMs) to generate Triton kernels in realistic development scenarios. The overview of the benchmark is illustrated in Figure 2. The benchmark consists of 31 real Triton kernel tasks collected from popular open-source AI frameworks. Each task corresponds to a merged pull request (PR) related to Triton kernel. For each task, RealisticTritonBench provides three components as input: a task description, relevant context, and the definition of the target function. These components are combined to form the prompt that is provided to the LLM. LLM is required to implement the target function strictly following the specified requirements. After generation, the produced kernels replace the original im- plementation in the prepared testing environment. We then execute a standardized evaluation pipeline that includes kernel-level unit tests, model-level accuracy tests, and end-to-end test. Specifically, the unit tests are first executed to verify the functional correctness of the generated kernels. The modified kernels are then integrated into the framework to evaluate model accuracy and end-to-end latency. The evaluation pipeline ensures that generated kernels satisfy the expected behavior of the task, while also capturing their impact on model accuracy and system-level performance under realistic deployment settings. In this section, we present the task formulation and the construc- tion pipeline of RealisticTritonBench in detail. 3.1 Task Formulation 3.1.1 Input Definition and Target Output. To evaluate the abil- ity of models to generate Triton kernels in real-world development environment, we construct the task inputs to closely resemble the information available to developers during kernel implementation. Specifically, each task consists of three components: task descrip- tion, context, and target function specification. The task description provides a detailed specification of the ker- nel functionality, describing the intended behavior and implemen- tation constraints of the operator. The context contains relevant information extracted from the original repository, presented in the form of file-function and file–class references. These references indicate functions and classes that may be useful for implement- ing the target kernel, enabling the model to access the necessary information required for development. ASE ’26, October 12–16, 2026, Munich, GermanyJ. Huang, Z. Wen, T. Xu, M. Yan, X. Xia, and Z. Liu Figure 2: Task formulation and evaluation pipeline of Real- isticTritonBench The definition of the target function defines the Triton kernel interface that the model is expected to implement. Given the task description and the relevant context, the model must generate a Triton kernel that conforms exactly to the provided definition. 3.1.2 Evaluation Suite. Compared with previous benchmarks that focus only on evaluation at the single-operator level, we sum- marize practical development experience from real-world PRs and design three types of tests to improve the robustness of the bench- mark: Unit Test, Model Accuracy Test, and Latency Test. Unit Test is used to verify the functional correctness of kernels. Model Accuracy Test evaluates whether replacing the operator im- plementation leads to noticeable degradation in model performance on common tasks. Latency Test measures the end-to-end system latency after the substitution. 3.2 Benchmark Construction We focus on constructing high-quality, real-world Triton kernel generation tasks and build RealisticTritonBench through a four- phase pipeline. As illustrated in Figure 3, we first collect relevant pull requests from selected open-source AI framework repositories (Phase 1). We then filter and analyze these PRs to extract concrete Triton kernel generation tasks (Phase 2). Next, we construct exe- cutable evaluation environments for each instance(Phase 3). Finally, we validate and refine the instances based on execution results to ensure the correctness and stability of the benchmark (Phase 4). 3.2.1 PR Collection. The goal of this phase is to collect pull re- quests (PRs) related to Triton kernel modifications from open- source repositories, in order to construct a candidate task pool grounded in real-world engineering practice. We first select several popular and actively maintained AI framework repositories as data sources, including PyTorch, vLLM, and SGLang. These repositories extensively use Triton to implement high-performance GPU kernels for both training and inference pipelines, making them represen- tative of real-world Triton development scenarios. We therefore collect historical pull requests (PRs) from these repositories as can- didate instances for further analysis. To efficiently identify Triton-related modifications among a large number of PRs, we design an automated filtering strategy as an initial filter. Specifically, we first perform a coarse-grained filter- ing step based on keywords appearing in PR titles, descriptions, and commit messages, such as triton, kernel, operation, and opti- mization. This step helps quickly locate PRs that are potentially related to Triton. Next, we further analyze the code diffs in each PR and retain only those that actually modify or introduce Tri- ton kernel implementations, such as files containing@triton.jit kernel definitions. By combining keyword-based filtering with code- level change analysis, our approach reduces manual checking effort while maintaining high relevance to Triton kernel modifications. Through this process, we obtain a set of PRs closely related to Triton kernel modifications as candidate instances. These PRs cover a variety of real development scenarios, including kernel optimizations, bug fixes, and the implementation of new features, providing a diverse foundation for constructing the benchmark. 3.2.2 Kernel Task Extraction. Based on the PRs related to Triton kernel modifications collected in the previous stage, we conduct a detailed analysis of each PR to identify the objectives of the kernel changes, and subsequently transform them into well-defined Triton kernel generation tasks. Initial Filtering. After the initial keyword-based filtering in the previous step, we obtained approximately 2,000 PRs that require manual analysis. We then manually examine each PR and further filter them according to the following evaluation criteria. (1)Triton Kernel Relevance. The PR must contain modifications related to Triton kernels. (2)Test Availability. The PR must include corresponding tests (e.g. at least unit tests) that can be used to verify correctness. (3)Clear Kernel Objective. The modification introduces a clear functional or performance-related objective that can be formu- lated as a standalone kernel task. Task Extraction. To extract tasks from PRs, we first conduct a manual analysis of each PR by carefully examining the PR descrip- tion and the corresponding code changes. Through this process, we aim to understand the background, objectives, and motivations behind the Triton kernel modifications. For example, some PRs introduce new Triton kernels to replace existing PyTorch imple- mentations in order to improve performance, while others may extend the functionality of existing kernels. This analysis allows us to clearly identify the target functions involved and the expected functional changes introduced by the modification. Based on the content of the PR, we then employ an LLM to auto- matically generate the corresponding kernel task description. The generated description is typically summarizing the target function to be implemented or modified and the expected behavior. In this way, real-world code modifications are abstracted into executable Triton kernel generation tasks. Then, the generated descriptions are manually reviewed and revised when necessary to ensure that they accurately reflect the original intention of the PR and provide a clear and complete specification of the task. Specifically, each task description is checked for intent faithfulness to the PR goal, requirement completeness regarding behavior, constraints, and suc- cess criteria, and the absence of implementation leakage from the RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI FrameworksASE ’26, October 12–16, 2026, Munich, Germany Figure 3: The pipeline of constructing our benchmark gold patch. Two Triton-experienced developers inspect the PR de- scription, diff, target function, and tests, and score each criterion on a 0/1/2 scale. If any criterion averages below 1.5, they revise the task description to consensus. In addition to the task description, we also manually analyze and extract the contextual information required for implementing the task, including related functions and classes , the definition of the target function to be implemented, and the corresponding tests. We categorize the tests into three types: Unit Test, Model Accuracy Test, and Latency Test. The Unit Test relies on existing pytest-based testing commands provided by the repository. For Model Accuracy Test and Latency Test, we analyze the models associated with each PR and construct testing com- mands based on the repository’s existing evaluation scripts. These tests are used to verify model accuracy after kernel replacement and to measure the end-to-end system latency respectively. 3.2.3 Environment Construction. Following SWE-bench[17], we provide a runnable Docker environment for each instance. Inspired by the multi-layered environment construction approach used in NoCodeBench[5], we adopt a two-level construction strategy to improve efficiency. We first build a shared base Docker image for instances orig- inating from the same repository. The base image includes the required system dependencies and standardized runtime config- urations, such as Python and CUDA Toolkit versions. With this image as the base image, we further construct instance-specific en- vironments that contain the exact code version and dependencies required for running the corresponding tests. More specifically, we initially attempt to build the environment using the repository-specific build commands. However, due to is- sues such as ambiguous dependency specifications or incompatible library updates, some instances fail to build. For example, newer versions of certain libraries may introduce APIs that are incompat- ible with previously supported configurations or older hardware, requiring specific files to be switched to compatible versions. To address these instance-specific issues, we record the full build logs during the environment setup process and manually resolve installation errors when they occur. The corresponding fixes are then documented as executable shell commands, such as specify- ing dependency versions or patching configuration files. These commands are integrated into the instance-specific build scripts to ensure that the environment can be reproduced automatically. This stage guarantees that each instance is equipped with an independent and fully functional containerized environment, pro- viding a reliable execution foundation for subsequent testing and evaluation. 3.2.4 Refine on Execution Feedback. After collecting candidate instances and constructing executable environments, we further Table 1: Distribution of task categories in RealisticTriton- Bench OptimizationModificationNew Kernel Percentage41.93%22.58%35.48% Table 2: Studied Large Language Models ModelTypeSizeTime Deepseek-V3.2(non-reasoning)non-reasoning685B2025-12-01 Qwen3.5-397B-A17Breasoning397B2026-03-09 Deepseek-V3.2(reasoning)reasoning685B2025-12-01 GPT-5.4reasoningunpublished2026-03-05 Gemini-3.1 Pro Previewreasoningunpublished2026-02-19 verify the validity of the tests associated with each instance and perform additional filtering and refinement. Specifically, we first execute all Unit Tests provided for each instance and remove those that fail to pass all Unit Tests, excluding failures caused by hardware- related issues. Since Unit Tests reflect the expected behavior of the modified Triton kernels, instances that cannot pass these tests are considered invalid as they fail to correctly reproduce the behavior of the original PR, and are therefore excluded from the final dataset. Next, we run the Model Accuracy Test and Latency Test and adjust the corresponding execution commands when necessary based on the observed results. This step is required because the repositories continue to evolve over time, and the interfaces or arguments of testing scripts may change accordingly. As a result, some original test commands may no longer execute correctly in the reconstructed environment. We therefore refine the testing commands according to the actual execution results to ensure that the evaluation scripts remain valid and can reliably measure model accuracy and system latency. Through these phases, we ultimately obtained 31 task instances, which are grouped into three categories, as summarized in Table 1. 4 EXPERIMENT In this section, we evaluate the performance of the state-of-the-art (SOTA) LLMs in different settings and analyze the results. Specifi- cally, we aim to address the following three research questions. • RQ1: How do state-of-the-art LLM-based agents perform on RealisticTritonBench? •RQ2: How do LLMs’ performance vary across different task categories? •RQ3: What are the main reasons for the failures of LLMs on RealisticTritonBench? ASE ’26, October 12–16, 2026, Munich, GermanyJ. Huang, Z. Wen, T. Xu, M. Yan, X. Xia, and Z. Liu Table 3: Performance comparison across different LLMs on RealisticTritonBench. ModelSuccess (%) Applied (%) FTP (%) UTP (%) NR(%)S TTFT S TPOT Cost Open-source Models Deepseek-V3.2 (non-reasoning)12.90%93.55%45.16%56.55%45.45%0.97080.958717.54M Deepseek-V3.2 (reasoning)19.35%96.77%41.94%66.84%40.00%0.99030.988125.89M Qwen3.5-397B-A17B25.81%96.77%45.16%58.12% 58.33%1.0110 0.995911.75M Closed-source Models GPT-5.416.13%100.0%35.48%52.16%44.44% 1.3750.8948 9.23M Gemini-3.1-Pro-Preview19.35%100.0%48.39% 67.98%50.00%0.94240.968710.47M Average18.71%97.42%43.23%60.33%47.65%1.05790.961214.98M 4.1 Methodology 4.1.1 Model Selection. The tasks in RealisticTritonBench require models to generate complete Triton kernel implementations based on task descriptions and repository context. The process demands strong capabilities in task understanding, Triton kernel generation, and long-context reasoning. Therefore, we select a set of state-of- the-art LLMs for evaluation, as shown in Table 2. Among them, three models are open-source models, including DeepSeek-V3.2 (non-reasoning)[23], DeepSeek-V3.2 (reasoning)[23] and Qwen3.5- 397B-A17B[31]. The other two are closed-source models, including Gemini-3.1 Pro Preview[10] and GPT-5.4[27]. These models repre- sent some of the most advanced large language models currently available, with strong performance on code generation and reason- ing tasks. They differ in model architecture, training paradigms, and functional capabilities, which enables us to investigate how SOTA models with different architectures perform on Triton kernel generation tasks. For models with reasoning capabilities, we set the reasoning effort to high when configurable and otherwise simply enable thinking. 4.1.2Scaffold Selection. Finishing Triton kernel generation tasks requires models to retrieve relevant code context from the reposi- tory as essential dependency information for implementation. Due to the lack of accurate context, directly prompting an LLM often fails to produce correct and complete implementations. Therefore, we select mini-SWE-agent [47] as the evaluation scaffold. mini-SWE-agent is a lightweight implementation derived from the SWE-agent system developed by the Princeton and Stanford teams, designed to simplify the coding-agent pipeline while main- taining strong performance. On the SWE-Bench (Verified) leader- board, it achieves over 76% pass rate. Although several agents rank higher, including live-swe-agent[45], Sonar Foundation Agent, TRAE[8], Atlassian Rovo Dev, and EPAM AI/Run Developer Agent, they are not suitable for our evaluation for the following reasons. First, Sonar Foundation Agent, Atlassian Rovo Dev, and EPAM AI/Run Developer Agent are closed-source systems, making it im- possible to obtain their implementations for experimentation. Sec- ond, TRAE is inactively maintained and currently does not support models without tool-calling capabilities. Finally, although live-swe- agent is SOTA and open-source, it modifies the original scaffold through tool evolution. Since our goal is to evaluate the intrinsic capability of models, we follow SWE-Bench and adopt a bash-only tool interface for the scaffold. Taking into account of these factors, mini-SWE-agent is ultimately selected as the scaffold used in our evaluation. 4.1.3 Evaluation Metrics. Different from kernel-oriented bench- marks [20,28,46], which mainly focus on kernel-level evaluation, we assess the practical performance of generated Triton kernels within the overall framework. We introduce four evaluation metrics to assess the correctness, model numerical stability, and system- level performance of the generated kernels: •Full Test Pass Rate (FTP%): The percentage of tasks whose generated implementations pass all unit tests. •Unit Test Pass Rate (UTP%): The percentage of unit tests passed out of the total number of unit tests for a task. •Numerical Robustness for Model (NR, T/F): Evaluates whether the kernel modification introduces numerical instability at the model level. After integrating the generated kernel into the framework, the model is evaluated on a common benchmark. Following the criteria derived from real-world open-source PRs, if the task performance does not degrade, the metric is marked as T; otherwise, it is marked as F. •End-to-End Acceleration Speedup: Measures the improve- ment in end-to-end latency after deploying the modified kernel in the AI framework compared with the reference implementa- tion under the same workload. We report the speedup of both Time to First Token (TTFT) and Time per Output Token (TPOT), defined as 푆 TTFT = TTFT base TTFT new , 푆 TPOT = TPOT base TPOT new . A value greater than 1 indicates a latency reduction compared with the baseline. (We report the average NR and speedup over tasks that pass all unit tests) In addition, following prior work [5,17,49], we also report the success rate of kernel task(Success%) and token cost(Cost) for each model, the success rate of patch application(Applied%) as aggregate evaluation metrics. A task is considered successful only if all of the following conditions are satisfied: •The Unit Test Pass Rate (UTP) matches that of the gold patch implementation. • The Numerical Robustness check passes (푁푅= 푇 ). • The end-to-end latency satisfies 푆 TTFT ≥ 0.98 and 푆 TPOT ≥ 0.98. The first condition ensures functional correctness, the second verifies numerical stability at the model level, and the third ensures that the generated kernel does not significantly degrade system- level inference performance. We adopt a tolerance threshold of RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI FrameworksASE ’26, October 12–16, 2026, Munich, Germany 0.98 for latency speedup to account for minor runtime fluctuations, and discuss its sensitivity in our online appendix [14]. All experi- ments are conducted on a server equipped with 8×NVIDIA RTX 3090 GPUs and averaged over three runs, the average variation is 1.08% for푆 TTFT and 0.98% for푆 TPOT , indicating stable speedup measurements. Notably, we adopt the reference implementation from the gold patch as the baseline for two reasons. First, these PRs from widely used frameworks such as vLLM underwent multiple rounds of review by experienced kernel experts, making them a credible refer- ence grounded in real-world development practice. Second, it is the only available reference implementation in the target framework at that time, making it a meaningful baseline for assessing whether generated patches reach the accepted real-world solution. 4.2 RQ1: Performance on RealisticTritonBench We evaluate five representative LLMs on RealisticTritonBench, and the results are presented in Table 3. Our findings show that un- der stricter and more realistic multi-dimensional evaluation cri- teria, even state-of-the-art LLMs exhibit limited performance on real-world Triton kernel tasks. Specifically, Qwen3.5-397B-A17B achieves the best performance, with a task success rate of 25.81%, indicating that only a small portion of tasks can be fully solved under realistic constraints. In terms of unit test performance, the average Unit Test Pass Rate (UTP) across all models reaches 60.33%, and the Full Test Pass Rate (FTP) is 43.23%. However, the overall task success rate is only 18.71%, which is significantly lower than the FTP. This notable gap indicates that although models can often generate implementations that pass all unit tests, these implementations may still degrade model accuracy or increase end-to-end latency when deployed in real systems. In other words, passing unit tests alone is insufficient to guarantee the practical usability of generated kernels. Regarding numerical robustness, only 47.65% of the cases maintain model ac- curacy without degradation after replacing the original code. This further highlights that correct implementations may also introduce subtle numerical issues that negatively affect model performance. For end-to-end latency, most models achieve an averageS TTFT of around 1.0579, indicating no clear performance improvement com- pared to the baseline. Although GPT-5.4 achieves a higherS TTFT of 1.375, this is mainly due to the limited number of correct cases included in its latency evaluation, making the result less represen- tative. Similarly, the averageS TPOT of 0.9612 further suggests that replacing kernels often results in comparable or slightly degraded performance. This observation indicates that performance opti- mization at the kernel level is still challenging for current LLMs, especially when considering system-level interactions. Overall, these results demonstrate that RealisticTritonBench ef- fectively exposes the gap between unit test correctness and real system performance, preventing models from concentrating on unit tests alone. By incorporating system-level accuracy and latency evaluation, our benchmark provides a more comprehensive, realis- tic, and challenging testbed for Triton kernel generation, and better reflects the requirements of real-world deployment scenarios. Summary: The state-of-the-art LLMs perform poorly on Re- alisticTritonBench, struggling to generate Triton kernels that simultaneously pass all unit tests, preserve model accuracy, and improve end-to-end latency. This highlights the diffi- culty of achieving system-level correctness and efficiency, and demonstrates that RealisticTritonBench effectively ex- poses the gap between unit test success and real-world de- ployment requirements. 4.3 RQ2: Category-wise Performance Analysis Prior work mainly focuses on translating PyTorch kernels into Tri- ton kernels, which corresponds to the New-kernel category in our work. Beyond this setting, we further introduce two additional task categories: Optimization and Modification. The Optimization cate- gory targets performance improvements of existing Triton kernels, while the Modification category involves fixing bugs or extending functionality (e.g., adding support for new data types) in existing kernels. In this section, we analyze the experimental results from a category-wise perspective. Table 4 presents the performance of different models across the three task types: Optimization, Modifi- cation, and New-kernel. Optimization. Models achieve average 23.08% success rates on Op- timization tasks, with relatively high UTP and FTP across all models. This indicates that LLMs are generally capable of generating opti- mized Triton kernels that preserve the original functionality based on existing implementations. However, the end-to-end speedup remains limited, with mostS TTFT andS TPOT values close to 1.0, suggesting that models struggle to deliver meaningful performance improvements. Furthermore, numerical robustness remains a chal- lenge, with only 56.19% of the generated kernels maintaining model accuracy without noticeable degradation. This indicates that even when functional correctness is preserved, the generated implemen- tations may still introduce subtle numerical inconsistencies that affect downstream model behavior. These results highlight the diffi- culty of considering low-level performance optimization, numerical stability, and system-level efficiency simultaneously. Modification. For Modification tasks, models achieve performance comparable to Optimization tasks in terms of UTP and FTP. Both task types share a common characteristic in that they require gen- erating kernels based on existing Triton implementations, while Optimization focuses on improving performance; Modification aims to alter the original functionality, such as adding new features or fixing bugs. The comparable performance results on UTP and FTP further prove that, given existing Triton kernels as references, LLMs are generally capable of generating functionally equivalent imple- mentations. However, numerical robustness remains a significant challenge in this setting. On average, only 20.00% of the generated kernels maintain model accuracy without degradation, with several models achieving 0% NR. This indicates that Modification tasks are more likely to introduce subtle numerical issues that degrade model accuracy. It also suggests that while LLMs can follow functional requirements, they lack a deep understanding of numerical stability and edge-case behaviors when modifying existing kernels. New-kernel. The average task success rate of only 5.455% in Ta- ble 4 indicates that New-kernel is the most challenging category for current models. Compared with the other two task types, this ASE ’26, October 12–16, 2026, Munich, GermanyJ. Huang, Z. Wen, T. Xu, M. Yan, X. Xia, and Z. Liu # Target Function @triton.jit def kernel_unified_attention_2d(...): ... cur_batch_in_all_start_index = tl.load(query_start_len_ptr + seq_idx) cur_batch_in_all_stop_index = tl.load(query_start_len_ptr + seq_idx + 1) cur_batch_query_len = cur_batch_in_all_stop_index - cur_batch_in_all_start_index ... seq_len = tl.load(seq_lens_ptr + seq_idx) ... q_end_pos = tl.min(q_end_pos, cur_batch_query_len) Figure 4: An example of incorrect usage of the Triton API. setting requires the model to implement a functionally equivalent Triton kernel without any prior Triton implementation as refer- ence. Across the 11 New-kernel tasks, on average, only 20% of the generated kernels are able to pass all unit tests, which is the lowest among all categories. Moreover, even among the limited set of correct kernels, the end-to-end latency metrics are gener- ally below 1.0, and the average numerical robustness (NR) is only 40%, suggesting that newly generated kernels often fail to match or surpass the performance of existing implementations. These results highlight the fundamental difficulty for LLMs in generating high-quality Triton kernels from scratch when no Triton-specific reference implementation is available. Summary: LLMs perform well in functional correctness when modifying existing Triton kernels but struggle with performance optimization and numerical robustness. When no Triton-based implementation is provided, performance drops significantly, revealing limited capability in generating high-quality kernels from scratch. 4.4 RQ3: Failure Analysis To better understand the limitations of LLMs in Triton kernel gen- eration, we manually inspect the failed cases and identify the key reasons that lead to their poor performance on RealisticTriton- Bench. Through this analysis, we reveal the limitations of current LLMs in real-world Triton kernel generation tasks. Insufficient fundamental capability in Triton programming. On average, 56.77% of the cases fail to pass all unit tests, indicating that generating fully correct Triton kernels remains highly challeng- ing for current LLMs. A fundamental prerequisite for completing these tasks is the ability to produce compilable and executable code that strictly adheres to Triton’s programming constraints, including its type system, supported control flow, and memory access seman- tics. However, our analysis of failure cases shows that a substantial portion of errors originate from violations of these low-level con- straints. Among the cases that fail to pass all unit tests, 64.71% fall into this category. Concretely, models frequently hallucinate non-existent APIs (e.g.tl.info,tl.finfo), misuse existing primi- tives (e.g. incorrect usage oftl.min), or produce code that violates Triton’s strict typing rules. In addition, several cases exhibit incor- rect memory access patterns, such as mismatched tensor shapes or invalid pointer arithmetic intl.load, which result in runtime errors or silent incorrect behavior. There are also instances of un- supported control flow constructs (e.g.break) and missing essential components (e.g. decorators), further preventing successful execu- tion. These results indicate that current LLMs lack a systematic and reliable understanding of Triton’s low-level programming model. In particular, they struggle to internalize the strict constraints im- posed by the compiler and runtime, including type safety, valid API usage, and explicit memory layout requirements. For example, as illustrated in Figure 4, the model incorrectly uses thetl.minAPI. Inkernel_unified_attention_2d, both cur_batch_query_lenandq_end_posare scalar values (i.e., 0-D Triton tensors). However,tl.minin Triton is designed as a reduc- tion API, expecting an input tensor together with a reduction axis, rather than two scalar operands for binary comparison. Therefore, tl.min(q_end_pos, cur_batch_query_len)does not match the intended semantics of taking the minimum of two scalar values. The correct API here should betl.minimum, which performs an elementwise minimum between two operands. Lack of understanding of kernel semantics in real-world repository code. In addition to violations of low-level program- ming constraints, we observe a 29.41% of failures that stem from incorrect implementations of kernel semantics in concrete code repositories. Successfully completing these tasks requires the model to comprehensively reason about kernel behavior in repositories, including boundary conditions, corner cases, and fine-grained im- plementation details that are critical to correctness and numer- ical behavior. We observe that LLMs frequently make mistakes in implementation details. As illustrated in Figure 5, the gener- ated implementation fails to handle segment processing. Instead of returning early for invalid segments, it incorrectly writes back incorrectM_seg,L_seg, andacc_seg, thereby corrupting the sub- sequent segment-level reduction. This case further highlights that LLMs lack a comprehensive understanding of kernel semantics in real-world implementations. In real-world kernel generation tasks, correct implementations require careful consideration of both the task objectives and the surrounding code context to properly han- dle different scenarios, including boundary conditions and edge cases. LLMs frequently struggle to correctly handle corner cases and often ignore implicit conditions, ultimately leading to func- tional inconsistencies or incorrect numerical results. This indicates that LLMs lack a comprehensive understanding of kernel semantics in real-world repository contexts. Lack of attention to performance and numerical stability during implementation. In our experiments, models successfully generate Triton kernels that pass all correctness checks in 43.23% of cases. However, in 25.16% of cases, replacing the original im- plementation with the generated kernel leads to degraded model accuracy or end-to-end performance. Through case analysis, we find that generated kernels often introduce unnecessary operations or miss critical optimizations compared to reference kernels. For example, as shown in Figure 6, the gold implementation en- closes the penalty-related computation within anif use_penalty branch, thereby avoiding redundant arithmetic operations and pred- icate computations. In contrast, the generated implementation exe- cutes these computations unconditionally, resulting in increased RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI FrameworksASE ’26, October 12–16, 2026, Munich, Germany Table 4: Category-wise performance of different models on RealisticTritonBench across task types: Optimization, Modification, and New-kernel. Task Type ModelSuccess (%) Applied (%) FTP (%) UTP (%) NR(%)S TTFT S TPOT Optimization Deepseek-V3.2 (non-reasoning)15.38%100.0%53.85%75.26% 71.43%0.94420.9510 Deepseek-V3.2 (reasoning)7.692%100.0%53.85%76.47%42.86%0.98910.9794 Qwen3.5-397B-A17B38.46%100.0%69.23% 82.88%66.67%1.0180.9920 GPT-5.423.08%100.0%46.15%53.65%50.00% 1.8120.9001 Gemini-3.1-Pro-Preview30.77%100.0%61.54%75.61%50.00%0.9964 1.005 Average23.08%100.0%56.92%72.78%56.19%1.1520.9654 Modification Deepseek-V3.2 (non-reasoning)28.57%100.0%71.43%71.43%0.000%1.0270.9824 Deepseek-V3.2 (reasoning)42.86%100.0%57.14% 79.92%0.000%1.0021.008 Qwen3.5-397B-A17B42.86%100.0%57.14%62.34% 50.00%0.98251.004 GPT-5.414.29%100.0%28.57%54.10%0.000%1.0261.019 Gemini-3.1-Pro-Preview28.57%100.0%57.14%73.19% 50.00% 1.033 1.024 Average31.43%100.0%54.29%68.20%20.00%1.0141.007 New-kernel Deepseek-V3.2 (non-reasoning)0.000%81.81%18.18%24.97%0.000%0.97910.9500 Deepseek-V3.2 (reasoning)18.18%90.90%18.18%47.14% 100.0%0.98250.9990 Qwen3.5-397B-A17B0.000%90.90%9.091%26.16%0.000% 1.007 1.014 GPT-5.49.091%100.0%27.27%49.17%50.00%0.80530.8559 Gemini-3.1-Pro-Preview0.000%100.0%27.27% 55.65%50.00%0.73800.8359 Average5.455%93.94%20.00%40.62%40.00%0.90250.9310 # Reference Implementation def kernel_unified_attention_3d(...): ... # Reference Implementation blocks_per_segment = cdiv_fn(seq_len, NUM_SEGMENTS_PER_SEQ * BLOCK_SIZE) if segm_idx*blocks_per_segment*BLOCK_SIZE >= seq_len: return # Generated implementation segment_len = cdiv_fn(num_blocks, NUM_SEGMENTS_PER_SEQ) start_block = segment_idx * segment_len end_block = tl.minimum(start_block + segment_len,...) ... tl.store(segm_max_ptr + segm_offset, M, ...) tl.store(segm_expsum_ptr + segm_offset, L, ...) Figure 5: An example where the generated implementation incorrectly writes back invalid segments instead of skipping latency. Furthermore, we observe that generated kernels may de- grade model accuracy due to subtle numerical issues introduced during implementation. For example, changes to the specializa- tion strategy (e.g., conversion totl.constexprand modifying do_not_specialize) may affect Triton’s execution behavior, lead- ing to small numerical deviations that accumulate and impact model accuracy. These cases show that LLMs lack of enough attention to performance and numerical stability in kernel implementation. Summary: LLM failures in real-world Triton kernel gener- ation stem from three key limitations: insufficient mastery of Triton programming constraints, incomplete understand- ing of kernel semantics in real-world codebases, and lack of attention to performance efficiency and numerical stability. def _penalties_and_temperature_kernel(...): ... # Reference Implementation if use_penalty: req_state_idx = tl.load(idx_mapping_ptr + batch_idx) output_bin_counts = tl.load(output_bin_counts_ptr + req_state_idx * output_bin_counts_stride + block,mask=mask) output_bin_mask = output_bin_counts > 0 # Generated Implementation req_state_idx = tl.load(idx_mapping_ptr + batch_idx) output_bin_counts = tl.load( output_bin_counts_ptr + req_state_idx * output_bin_counts_stride + block, mask=mask,) ... Figure 6: An example where the generated implementation performs redundant computation 5 Discussion 5.1 Necessity of Framework-Level Evaluation Among the failures reported in our evaluation, a portion of them, such as missing boundary masks, could in principle be caught by stronger isolated kernel-level tests. However, the failures related to numerical deviations, whose acceptability is decided by down- stream computation, can hardly be detected by kernel-level tests. For example, in one case from our evaluation, i.e.,_fused_moe_ lora_kernel(shown in Figure 7), the generated kernel introduces two numerical deviations. One arises from parallelism in compu- tation: the generated kernel replaces the gold implementation’s sequential accumulation over the K dimension with parallel K- splitting merged viaatomic_add. Under this parallel strategy, the commit order of the partial sums is decided by GPU scheduling ASE ’26, October 12–16, 2026, Munich, GermanyJ. Huang, Z. Wen, T. Xu, M. Yan, X. Xia, and Z. Liu Table 5: Reward-hacking strategies catalogued by SOL- ExecBench [22] CategoryExploit Description ConcurrencyOffloads computation to background threads, side streams, or forked processes to escape the timed region. State & CachingReturns cached or lazily materialized results, or behaves correctly only when a check is detected. EnvironmentMonkey-patches timing or comparison utilities, or silently downgrades numerical precision for speed. races. Since floating-point addition is not associative, the outputs deviate slightly from the reference and even vary across runs on the same input. The other arises from rounding of partial sums: the gold implementation rounds each value to BF16 before writing it, whereas the generated kernel rounds each value directly to the output element type. This difference in rounding behavior can intro- duce a small but systematic numerical discrepancy when the output dtype is not BF16. Both numerical deviations are tiny per call, and the kernel passes all unit tests, yet they accumulate across dozens of stacked MoE-LoRA layers and thousands of autoregressive de- coding steps, eventually degrading GSM8K accuracy. These failures can hardly be detected by the tolerance threshold of a kernel-level unit test, because it is hard to derive an appropriate threshold for two reasons. First, deriving an acceptable error threshold re- quires framework-level analysis. Determining how much per-call deviation can be tolerated without degrading task accuracy re- quires tracing how the error propagates through the actual model’s stacked layers, discrete routing decisions, and autoregressive de- coding process, none of which is a property of the kernel. Second, the acceptable error threshold is deployment-specific rather than kernel-specific. Although the same kernel may be reused across models, tasks, and runtime configurations, these contexts deter- mine how errors propagate and accumulate. Models may invoke the kernel across more layers, longer decoding sequences allow errors to accumulate over steps, and different serving precisions change the magnitude of each deviation. These system-level effects are difficult, if not impossible, to capture with kernel-level unit tests, making framework-level evaluation essential. 5.2 Reward Hacking Mitigation Reward hacking occurs when a model exploits loopholes in the evaluation environment to maximize its score without genuinely solving the underlying task[22]. Prior work[22] systematically cat- egorizes hacking strategies in kernel generation into three groups: concurrency-based hacks, state-and-caching hacks, and environ- ment manipulation, as summarized in Table 5. These hacks arise from a fundamental limitation of kernel-level testing: it cannot fully capture the correctness and performance of a kernel as it operates within the complete framework. Although kernel-level metrics are often strongly correlated with the metrics that matter in deployment, they are not equivalent. To mitigate the gap, Re- alisticTritonBench evaluates each kernel in a setting that closely for k in range(0, grid_k): ... accumulator += tl.dot(a, b) a_ptrs += BLOCK_SIZE_K * SPLIT_K * stride_ak b_ptrs += BLOCK_SIZE_K * SPLIT_K * stride_bk ... # Reference Implementation accumulator = accumulator.to(tl.bfloat16) # fixed to bf16 tl.store(c_ptrs, accumulator, mask=c_mask) # Generated Implementation tl.atomic_add(c_ptrs, accumulator.to(c_ptrs.dtype.element_ty), mask=c_mask) # rounds to output dtype, not bf16 ... Figure 7: An example of numerical deviations whose accept- ability is decided by downstream computation # Hacked kernel: escape the timed stream def hacked_kernel(x): out = torch.empty_like(x) with torch.cuda.stream(side_stream): real_kernel[grid](x, out, ...) return out # Kernel-level harness (simplified) start.record() # on default stream hacked_kernel(x) end.record() # on default stream torch.cuda.synchronize() t = start.elapsed_time(end) # launch overhead only assert_close(hacked_kernel(x), ref) # passes Figure 8: A stream-injection hack that defeats kernel-level timing checks. mirrors its real-world usage. Specifically, (1) the generated ker- nel is integrated through the same framework call paths used in deployment; (2) it is validated using the same benchmarks that framework developers use to test kernels before deployment; (3) performance is measured using the end-to-end serving latency, i.e., time-to-first-token (푆 TTFT ) and time-per-output-token (푆 TPOT ), and model accuracy, which actual deployment cares about. This design mitigates all three categories of reward hacking listed in Table 5. First, concurrency-based hacks are ineffective because an external client measures TTFT and TPOT as the wall-clock time from request submission to response delivery. This interval encom- passes work performed by any thread, CUDA stream, or process. Consequently, hidden work in concurrency-based hacks is either included in the measured latency or results in incorrect outputs. Second, state-and-caching hacks are unlikely to succeed because kernel outputs are immediately consumed by downstream compu- tations across thousands of invocations under evolving runtime states. Cached, stale, or lazily materialized results therefore quickly cause numerical divergence and fail the model-accuracy evaluation. Third, environment-manipulation attacks are neutralized because the timing client and accuracy harness run in separate processes beyond the reach of the generated kernel. The kernel therefore cannot improve its score by modifying the evaluation utilities. Sim- ilarly, silently reducing numerical precision for additional speed is RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI FrameworksASE ’26, October 12–16, 2026, Munich, Germany ineffective when the resulting errors propagate through the model and degrade its end-to-end accuracy. Figure 8 illustrates a stream-injection attack. The hacked wrap- per launches the actual computation on a separate CUDA stream and returns immediately. An in-process timer that synchronizes only the default stream consequently measures little more than the launch overhead, while a later correctness check may still pass after the delayed computation has completed. This strategy is ineffective in RealisticTritonBench because the external client continues mea- suring time until it receives the final response. Thus, computation hidden on a separate stream is either fully reflected in TTFT and TPOT or causes incorrect outputs that fail the accuracy evaluation. 6 Threats To Validity The first threat arises from the automatic generation of task de- scriptions. In our construction pipeline, the task description of each instance is generated by an LLM from the corresponding pull request (PR) information and code changes. The LLM may misinter- pret the developer’s intent, or omit implicit implementation details. To mitigate this, we manually review all generated task descrip- tions and refine them when necessary to ensure they accurately reflect the intent of the original PR. The second threat concerns the generalization of our results across models. We evaluate only a set of representative LLMs, so the conclusions may depend on the selected models and may not fully generalize to all existing models. To alleviate this, we select mainstream models from different model families with varying capabilities to improve the representative- ness of the evaluation. The third threat stems from the evaluation scaffold. Main experiments are conducted with mini-SWE-agent, while different scaffolds may adopt different retrieval strategies and execution workflows that could influence model performance. We choose mini-SWE-agent because it represents the state-of-the- art level among open-source coding agents and is also the official agent adopted by SWE-bench for evaluating model performance. Furthermore, we additionally evaluate GPT-5.4 with its native scaf- fold Codex to examine the influence of the scaffold choice and provide the results in our online appendix [14]. The last threat is potential data contamination. Our benchmark is constructed from publicly available repositories, so the evaluated models may have been exposed to the relevant code during training. We analyze this risk and provide results in our online appendix [14], which suggest that contamination is unlikely to affect our conclusions. 7 RELATED WORK 7.1 Triton Kernel Generation Datasets To evaluate the quality of Triton kernels generated by LLMs, sev- eral benchmarks have been proposed. KernelBench [28], the first benchmark for GPU kernel generation, collects 250 AI workloads of three complexity levels and requires models to generate optimized kernels from PyTorch reference implementations. TritonBench [20] targets Triton specifically, with tasks curated from highly starred Triton repositories and selected PyTorch implementations. More recently, FlashInferBench [46] extends the pipeline to integration within practical AI frameworks, enabling end-to-end evaluation. Despite these efforts, the first two benchmarks formulate tasks as PyTorch-to-Triton translation and evaluate kernels in isolation with kernel-level metrics. These kernel-level benchmarks do not capture real-world development scenarios such as performance optimization. FlashInferBench integrates generated kernels into practical frameworks, but still differs from our work in two aspects. (1) Task formulation. It targets generating new kernels for a set of predefined kernel definitions, whereas RealisticTritonBench derives tasks from merged PRs in popular AI frameworks, covering Opti- mization, Modification, and New Kernel. (2) Evaluation metrics. It mainly relies on the kernel-levelfast_pmetric complemented with request-level latency, whereas RealisticTritonBench evaluates unit-test correctness, model numerical robustness, and end-to-end TTFT/TPOT latency, directly measuring the impact on accuracy and serving performance after deployment. 7.2 LLM for Automated Triton Generation LLMs have been widely applied to general code generation tasks [15, 16,25,26,34,39,40,44] and have also shown promising potential for Triton kernel generation. Existing approaches can be categorized into three groups: Domain Model Training, Agent-Based Pipelines, and Agentic RL Training. Domain Model Training fine-tunes LLMs on systematically col- lected or synthesized Triton-specific data. KernelLLM [7] performs instruction tuning on compiler-aligned PyTorch-Triton pairs, Au- toTriton [21] combines supervised fine-tuning with GRPO-based reinforcement learning [36], and TritonRL [43] further introduces hierarchical reward decomposition. Agent-Based Pipelines employ LLMs as coding agents that iteratively refine kernels through a generate-execute-refine loop driven by tools and runtime feed- back, as exemplified by KernelFalcon [30], TritorX [11], and AKG Kernel Agent [6]. Agentic RL further optimizes the agent’s decision- making via reinforcement learning, e.g., QiMeng-Kernel [52] adopts a two-stage paradigm that first learns hardware-aware optimization strategies and then implements them into efficient kernels. 8 CONCLUSION In this paper, we introduce RealisticTritonBench, a benchmark for evaluating LLM-based Triton kernel generation under realistic de- velopment settings. Unlike prior benchmarks that focus mainly on PyTorch-to-Triton translation and kernel-level evaluation, Realistic- TritonBench is constructed from real pull requests collected from open-source AI frameworks and covers diverse development scenar- ios, including kernel optimization, modification, and new operator implementation. We further design a comprehensive evaluation pipeline that integrates unit testing with end-to-end model accu- racy and system-level latency evaluation. Through experiments on several representative LLMs, we provide a systematic analysis of their capabilities and limitations in practical Triton kernel develop- ment tasks. Our results show that current LLMs lack the capability to generate correct and performance-efficient Triton kernels in real-world tasks. We envision RealisticTritonBench to facilitate the development of Triton kernel generation methods that are effective under realistic scenarios. 9 DATA AVAILABILITY Our code and data are available at https://doi.org/10.5281/zenodo. 19221469 and https://github.com/ZJU-CTAG/RealisticTritonBench ASE ’26, October 12–16, 2026, Munich, GermanyJ. Huang, Z. Wen, T. Xu, M. Yan, X. Xia, and Z. Liu Acknowledgments This research is supported by the National Natural Science Foun- dation of China (No.92582107) and Zhejiang Provincial Natural Science Foundation of China (No.LZ25F020003). References [1]Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016.TensorFlow: a system forLarge-Scalemachine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16). 265–283. [2]Ahsan Ali, Riccardo Pinciroli, Feng Yan, and Evgenia Smirni. 2020. Batch: Machine learning inference serving on serverless platforms with adaptive batching. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–15. [3]Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng, Binyuan Hui, Yuheng Jing, Kaixin Li, Mingze Li, Junyang Lin, et al.2026. Qwen3-Coder- Next Technical Report. arXiv preprint arXiv:2603.00729 (2026). [4] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems 35 (2022), 16344–16359. [5]Le Deng, Zhonghao Jiang, Jialun Cao, Michael Pradel, and Zhongxin Liu. 2025. Nocode-bench: A benchmark for evaluating natural language-driven feature addition. arXiv preprint arXiv:2507.18130 (2025). [6] Jinye Du, Quan Yuan, Zuyao Zhang, Yanzhi Yi, Jiahui Hu, Wangyi Chen, Yiyang Zhu, Qishui Zheng, Wenxiang Zou, Xiangyu Chang, et al.2025. AKG kernel Agent: A Multi-Agent Framework for Cross-Platform Kernel Synthesis. arXiv preprint arXiv:2512.23424 (2025). [7] Zacharias V. Fisches, Sahan Paliskara, Simon Guo, Alex Zhang, Joe Spisak, Chris Cummins, Hugh Leather, Gabriel Synnaeve, Joe Isaacson, Aram Markosyan, and Mark Saroufim. 2025. KernelLLM: Making Kernel Development More Accessible. https://huggingface.co/facebook/KernelLLM [8]Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, et al.2025. Trae agent: An llm-based agent for software engineering with test-time scaling. arXiv preprint arXiv:2507.23370 (2025). [9]Georgi Gerganov and contributors. 2023. llama.cpp: LLM Inference in C/C++. https://github.com/ggml-org/llama.cpp. GitHub repository. [10]Google. 2026. Gemini 3.1 Pro Preview. https://ai.google.dev/gemini-api/docs/ models/gemini-3.1-pro-preview. accessed: 2026-03. [11] Alec Hammond, Aram Markosyan, Aman Dontula, Simon Mahns, Zacharias Fisches, Dmitrii Pedchenko, Keyur Muzumdar, Natacha Supper, Site Cao, Haishan Zhu, et al.2026. Agentic operator generation for ml asics. Proceedings of Machine Learning and Systems 8 (2026), 1583–1594. [12]Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. 2024. Liger kernel: Efficient triton kernels for llm training. arXiv preprint arXiv:2410.10989 (2024). [13]Jiawei Hu, Hong Jia, Mahbub Hassan, Lina Yao, Brano Kusy, and Wen Hu. 2025. LightLLM: A versatile large language model for predictive light sensing. In Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems. 158–171. [14] Jinjun Huang, Zhongzhen Wen, Tongtong Xu, Meng Yan, Xin Xia, and Zhongxin Liu. 2026. Online Appendix for “RealisticTritonBench: A Benchmark for Triton- Kernel Generation in Real-World AI Frameworks”. https://anonymous.4open. science/r/RealisticTritonBench-2583/appendix/appendix.md [15]Zhonghao Jiang, David Lo, and Zhongxin Liu. 2025. Agentic Software Issue Res- olution with Large Language Models: A Survey. arXiv preprint arXiv:2512.22256 (2025). [16]Zhonghao Jiang, Xiaoxue Ren, Meng Yan, Wei Jiang, Yong Li, and Zhongxin Liu. 2025. Issue Localization via LLM-Driven Iterative Code Graph Searching. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 3034–3045. [17]Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues?. In International Conference on Learning Representations, Vol. 2024. 54107–54157. [18]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles. 611–626. [19]Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xi- aoming Wei, and Si Liu. 2025. Llava-st: A multimodal large language model for fine-grained spatial-temporal understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8592–8603. [20]Jianling Li, Shangzhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, WangHaojie WangHaojie, Jianrong Wang, Xu Han, et al.2025. Triton- bench: Benchmarking large language model capabilities for generating triton operators. In Findings of the Association for Computational Linguistics: ACL 2025. 23053–23066. [21]Shangzhan Li, Zefan Wang, Ye He, Yuxuan Li, Qi Shi, Jianling Li, Yonggang Hu, Wanxiang Che, Xu Han, Zhiyuan Liu, et al.2025. Autotriton: Automatic triton programming with reinforcement learning in llms. arXiv preprint arXiv:2507.05687 (2025). [22]Edward Lin, Sahil Modi, Siva Kumar Sastry Hari, Qijing Huang, Zhifan Ye, Nestor Qin, Fengzhe Zhou, Yuan Zhang, Jingquan Wang, Sana Damani, et al.2026. SOL- ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits. arXiv preprint arXiv:2603.19173 (2026). [23]Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [24]Wei Liu, Jiawei Xu, Yingru Li, Longtao Zheng, Tianjian Li, Qian Liu, and Junxian He. 2026. Dr. Kernel: Reinforcement Learning Done Right for Triton Kernel Generations. arXiv preprint arXiv:2602.05885 (2026). [25]Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. 2025. Swe-gpt: A process-centric language model for automated software improvement. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 2362–2383. [26]Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2025. Alibaba lingmaagent: Improving automated issue resolution via com- prehensive repository exploration. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 238–249. [27] OpenAI. 2026. GPT-5.4 Model. https://developers.openai.com/api/docs/models/ gpt-5.4. Accessed: 2026-03. [28] Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Re, and Azalia Mirhoseini. 2025. KernelBench: Can LLMs Write Efficient GPU Kernels?. In Forty-second International Conference on Machine Learning. [29]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019). [30] PyTorch Team and Contributors. 2025. KernelFalcon: Autonomous GPU Kernel Generation via Deep Agents. https://github.com/meta-pytorch/kernelagent. Accessed: 2025-07. [31]Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen. ai/blog?id=qwen3.5 [32]Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 3505–3506. [33] Reddit Community. 2025. Sakana discovered its AI CUDA engineer cheat- ing. https://w.reddit.com/r/OpenAI/comments/1iwc24f/sakana_discovered_ its_ai_cuda_engineer_cheating/ [34] Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2025. Specrover: Code intent extraction via llms. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 963–974. [35]Zhihong Shao, Yuxiang Luo, Chengda Lu, Z Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang. 2025. Deepseekmath-v2: Towards self- verifiable mathematical reasoning. arXiv preprint arXiv:2511.22570 (2025). [36]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024). [37]Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019). [38]Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Program- ming Languages. 10–19. [39]Junyi Wang, Jialun Cao, and Zhongxin Liu. 2026. iCoRe: An Iterative Correlation- Aware Retriever for Bug Reproduction Test Generation. Proceedings of the ACM on Software Engineering 3, FSE (2026), 4231–4252. [40] Xinchen Wang, Pengfei Gao, Xiangxin Meng, Chao Peng, Ruida Hu, Yun Lin, and Cuiyun Gao. 2025. Aegis: An agent-based framework for bug reproduction from issue descriptions. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 331–342. [41] Zhongzhen Wen, Yinghui Zhang, Zhong Li, Zhongxin Liu, Linna Xie, and Tian Zhang. 2025. MultiKernelBench: A Multi-Platform Benchmark for Kernel Gener- ation. [42]Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. 2025. Video models are zero-shot learners and reasoners. arXiv preprint arXiv:2509.20328 (2025). RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI FrameworksASE ’26, October 12–16, 2026, Munich, Germany [43]Jiin Woo, Shaowei Zhu, Allen Nie, Zhen Jia, Yida Wang, and Youngsuk Park. 2025. Tritonrl: Training llms to think and code triton without cheating. arXiv preprint arXiv:2510.17891 (2025). [44] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. De- mystifying llm-based software engineering agents. Proceedings of the ACM on Software Engineering 2, FSE (2025), 801–824. [45] Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang. 2025. Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly? arXiv preprint arXiv:2511.13646 (2025). [46]Shanli Xing, Yiyan Zhai, Alexander Jiang, Yixin Dong, Yong Wu, Zihao Ye, Char- lie F Ruan, Yingyi Huang, Yineng Zhang, Liangsheng Yin, et al.2026. Flashinfer- bench: Building the virtuous cycle for ai-driven llm systems. Proceedings of Machine Learning and Systems 8 (2026), 2016–2064. [47]John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer in- terfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [48] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al.2025. Flashinfer: Efficient and customizable attention engine for llm inference serving. Proceedings of Machine Learning and Systems 7 (2025). [49] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Aoyan Li, Lu Chen, Xiaojian Zhong, et al.[n. d.]. Multi-SWE- bench: A Multilingual Benchmark for Issue Resolving. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [50]Li Zhang, Youhe Jiang, Guoliang He, Xin Chen, Han Lv, Qian Yao, Fangcheng Fu, and Kai Chen. 2025. Efficient Mixed-Precision Large Language Model Inference with TurboMind. arXiv preprint arXiv:2508.15601 (2025). [51] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al.2024. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems 37 (2024), 62557–62583. [52]Xinguo Zhu, Shaohui Peng, Jiaming Guo, Yunji Chen, Qi Guo, Yuanbo Wen, Hang Qin, Ruizhi Chen, Qirui Zhou, Ke Gao, et al.2026. Qimeng-kernel: Macro- thinking micro-coding paradigm for llm-based high-performance gpu kernel generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 29168–29176.