Paper deep dive
Exploring LLM-based Verilog Code Generation with Data-Efficient Fine-Tuning and Testbench Automation
Mu-Chi Chen, Po-Hsuan Huang, Yu-Hung Kao, Yen-Fu Liu, Yu-Kai Hung, Cheng Liang, Shao-Chun Ho, Chia-Heng Tu, Shih-Hao Hung
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 6/21/2026, 12:04:08 PM
Summary
This paper proposes a novel workflow for improving LLM-based Verilog code generation through two main strategies: generating reasoning traces from a filtered PyraNet dataset using DeepSeek-R1, and a multi-agent framework for automated testbench generation. The multi-agent framework uses specialized agents for specification refinement and testbench creation to improve data quality and efficiency. The resulting fine-tuned model, MA-tb-7B, achieved a 68% pass@1 rate on a refined VerilogEval v2 benchmark, demonstrating that automated testbench generation can effectively produce high-quality training data for hardware description languages.
Entities (8)
Relation Signals (4)
MA-tb-7B → evaluatedon → VerilogEval v2
confidence 100% · Our fine-tuned model, MA-tb-7B... achieved a pass@1 rate of 68% [on VerilogEval v2]
Multi-Agent Framework → generatestestbenchesfor → PyraNet
confidence 100% · We applied our multi-agent framework to generate testbenches for Verilog code from PyraNet [11] and DeepCircuitX [12].
DeepSeek R1 → generatesreasoningtracesfor → PyraNet
confidence 90% · Using DeepSeek-R1, reasoning traces were generated for each sample [from PyraNet]
Qwen-Coder-7B-Instruct → isbasefor → MA-tb-7B
confidence 80% · A base model, Qwen-Coder-7B-Instruct [13], was evaluated... Our fine-tuned model, MA-tb-7B...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in large language models have improved code generation, but their use in hardware description languages is still limited. Moreover, training data and testbenches for these models are often scarce. This paper presents a workflow that uses multi-agent models to generate testbenches for high-quality fine-tuning data. By automating testbench creation, the fine-tuned model for the specification-to-Verilog task achieves performance comparable to state-of-the-art methods on the refined VerilogEval v2 benchmark while using less training data. This study provides a basis for future work on LLM-based HDL generation and automated verification.
Tags
Links
- Source: https://arxiv.org/abs/2604.15388v1
- Canonical: https://arxiv.org/abs/2604.15388v1
Trouble viewing inline? Open PDF directly →
Full Text
21,433 characters extracted from source content.
Expand or collapse full text
Exploring LLM-based Verilog Code Generation with Data-Efficient Fine-Tuning and Testbench Automation Mu-Chi Chen ∗ , Po-Hsuan Huang † , Yu-Hung Kao † , Yen-Fu Liu † , Yu-Kai Hung † , Cheng Liang † , Shao-Chun Ho † , Chia-Heng Tu ‡ , and Shih-Hao Hung † ∗ Academia Sinica, Taipei, Taiwan † National Taiwan University, Taipei, Taiwan ‡ National Cheng Kung University, Tainan, Taiwan Abstract—Recent advances in large language models have improved code generation, but their use in hardware description languages is still limited. Moreover, training data and testbenches for these models are often scarce. This paper presents a workflow that uses multi-agent models to generate testbenches for high- quality fine-tuning data. By automating testbench creation, the fine-tuned model for the specification-to-Verilog task achieves performance comparable to state-of-the-art methods on the refined VerilogEval v2 benchmark while using less training data. This study provides a basis for future work on LLM-based HDL generation and automated verification. Index Terms—Large language models, verilog code generation, testbench generation I. INTRODUCTION The design and verification of digital circuits rely on hard- ware description languages (HDLs) such as Verilog, which allow engineers to describe and simulate hardware behavior at multiple abstraction levels [1]. As circuit complexity in- creases, manual coding and verification have become time- consuming and error-prone, creating a demand for automated tools that support efficient module generation and testing. Recent progress in large language models has shown strong capabilities in code understanding and generation [2]–[4], indicating potential for HDL-related applications. Despite these advances, the use of large language models (LLMs) in hardware design remains limited. HDLs have struc- tural and behavioral properties that differ from conventional programming languages, which makes them more difficult for language models to handle. Limited access to advanced proprietary LLMs further restricts research, customization, and reproducibility, especially in settings where design information must remain confidential. Dataset and its quality are also significant concerns, as the lack of diverse and representative training data can hinder model performance. These constraints highlight the need for accessible and domain-adapted solutions in hardware design automation. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. We acknowledge the financial support from Academia Sinica’s SiliconMind Project (AS-IAIA-114-M11). This work was also supported in part by National Science and Technology Council, Taiwan, under Grants 112-2221- E-002 -159 -MY3 and 114-2221-E-006 -165 -MY3. We also thank National Center for High-performance Computing (NCHC) and Taipei-1 for providing computational and storage resources. This study introduces a workflow that uses LLMs to gen- erate testbenches and create high-quality data for fine-tuning Verilog generation models. The workflow adopts a multi-agent structure in which one agent produces Verilog modules based on design specifications, and another generates testbenches for verification. This division improves code accuracy and enables automatic production of training data by integrating LLM agents with existing verification tools. Our experimental results demonstrate that the proposed LLM-based multi-agent workflow can generate syntactically correct and functionally relevant Verilog code as training data. The approach also accelerates testbench creation, improving verification coverage and reducing manual effort. With sig- nificantly reduced training data compared to existing state-of- the-art methods, we show that LLMs can effectively contribute to HDL design and verification tasks through systematic eval- uation on benchmarks. This research lays the foundation for future work in applying LLMs to electronic design automation (EDA), opening new opportunities for AI-assisted hardware development and intelligent verification systems. The main contributions of this paper are summarized as follows: • We propose a novel workflow to evaluate two data gen- eration strategies for fine-tuning LLMs on specification- to-Verilog code tasks. • We present a workflow that uses multi-agent LLMs for Verilog generation and automated testbench creation, improving data quality and efficiency. • We evaluate the approach on the refined VerilogEval v2 benchmark and show that it produces functionally accurate Verilog code while achieving performance com- parable to state-of-the-art methods. I. BACKGROUND This section presents an overview of Verilog, testbenches, and LLMs. It also reviews related research in the field of LLM-based code generation and verification to establish the foundation for this study. A. Verilog and Testbenches Verilog is a foundational hardware description language that supports combinational and sequential logic across multiple abstraction levels. It is widely used to model circuit behavior arXiv:2604.15388v1 [cs.AR] 16 Apr 2026 from register-transfer level descriptions to gate-level imple- mentations, and its flexibility supports efficient prototyping with standard simulation and synthesis tools [5]. Verifying Ver- ilog modules is essential because design errors can propagate through later development stages. Tools such as AVERT [6] automate testbench generation and reduce manual effort for combinational and sequential designs. For complex systems, the Universal Verification Methodology with coverage-driven verification remains a practical and effective approach [5]. B. LLMs for Hardware Design LLMs represent a major advancement in artificial intelli- gence and can process text, perform reasoning, and generate code across many programming languages [2], [3]. Although these models handle syntax and semantics well in languages such as Python and C++, applying them to hardware design remains challenging. HDLs differ from software languages because they describe concurrent operations, timing behav- ior, and signal interactions under strict design rules. These characteristics require domain-specific training and specialized workflows for effective LLM use in hardware design. Leveraging LLMs to assist in or fully automate testbench generation offers a promising pathway toward intelligent verification workflows. Recent work has extended verifica- tion methodologies to machine learning-based approaches. For instance, Wang et al. [7] integrated testbench feedback into the reinforcement learning process to improve the func- tional correctness of LLMs generating Verilog code. Similarly, VeriReason [8] enhances reasoning in Verilog generation using reinforcement learning with testbench feedback. Zhu et al. [9] introduced CodeV-R1, which combines rule- based testbench generation and round-trip data synthesis for reliable Verilog generation from natural language descriptions. They employ both SFT and RL (DAPO [10]) to enhance Verilog coding capabilities. The key part is that CodeV-R1 enhances base LLMs with reasoning capabilities by distilling DeepSeek-R1 [4]’s reasoning traces and outputs during its supervised fine-tuning (SFT) stage. CodeV-R1’s methodology significantly increases the accuracy of 7B models, enabling them to exhibit R1-style reasoning capabilities for solving challenging Verilog problems. From CodeV-R1’s results, we found that 7B models can also achieve good RTL coding capability after being equipped with reasoning capabilities via distillation from DeepSeek-R1 in the SFT stage. I. METHODOLOGY Our proposed workflow consists of several key components that leverage LLMs for Verilog code generation and testing. As shown in Fig. 1, the workflow begins with generating rea- soning data from a filtered PyraNet dataset using DeepSeek- R1. This reasoning data serves as high-quality training in- put for SFT of base LLMs, enhancing their Verilog coding capabilities. Another strategy in our workflow is the multi- agent framework for automated testbench generation. This framework employs multiple LLM agents, each specializing in different aspects of testbench creation, to collaboratively produce comprehensive verification environments. After data generation, an SFT process is applied to fine-tune base LLMs to evaluate their performance on Verilog code generation tasks. Strategy 2: Multi-Agent Framework for Testbench Generation Strategy 1: Generating Reasoning Data from Filtered PyraNet Dataset Supervised Fine-Tuning Eval with refined VerilogEval v2 Non-reasoning Base Model samples include the testbench Reasoning Traces Generation samples exclude the testbench Testbench Generation Pyra-tb-r-all- 6.7k Pyra-tb-r- functional-1.3k Pyra- DeepCircuitX- MA-tb-r-38k Data generationData evaluation Fig. 1. Proposed workflow for Verilog code training data generation. A. Generating Reasoning Data from Filtered PyraNet Dataset To train reasoning models, we first needed reasoning traces. However, datasets typically provide only design specifications and golden solutions (code). To address this, we used vLLM to deploy DeepSeek-R1 on 16x H100 GPUs to generate rea- soning traces. We selected the Pyra-tb dataset (6.7k samples), filtered from 692k [11], for this purpose. This dataset is considered the highest-quality subset produced by VeriPre- fer [7], following the three filtering process. Notably, the fourth block includes testbenches, which can be used to verify the functional correctness of the generated Verilog code. B. Multi-Agent Framework for Testbench Generation To automate the creation of testbenches for the Verilog code, we propose a multi-agent framework that leverages the capabilities of LLMs. In this framework, multiple agents are assigned specific roles in the testbench generation process, al- lowing for a collaborative approach to creating comprehensive reasoning traces and corresponding testbenches. Each agent is responsible for a distinct aspect of tasks, such as speci- fication quality check and testbench generation. The agents communicate and collaborate to ensure that the generated testbenches are thorough and cover a wide range of scenarios. This multi-agent approach not only enhances the quality of the testbenches but also improves the efficiency of the generation process, as each agent can focus on its area of expertise. As shown in Fig. 2, our multi-agent framework consists of the following key components: 1) well written check and refine; 2) testbench generation; 3) verilog verification tools (compiler and simulator). In the pipeline, we use two datasets, PyraNet [11] and DeepCircuitX [12] as the input of the multi- agent framework. The input code is first sent to the quality check agent, which tries to refine the question description if the initial description is unclear or incomplete. This description is then passed to the testbench generation agent, which creates a comprehensive testbench that includes the necessary com- ponents for simulation and verification. Finally, the generated testbench is reviewed and revised to ensure its correctness. To enhance the efficiency of the testbench generation pro- cess, we propose a variant of the multi-agent framework that incorporates pre-generated testbenches, as illustrated in Fig. 3. In this approach, testbenches are generated prior to the application of the well written check and refine agent. The workflow includes two stages of testbench revision (one before and one after question refinement) which leads to a higher pass rate compared to the original framework. This modification not only improves the overall quality of the generated testbenches but also reduces the time and manual effort required to produce reliable verification environments. make prompt = i, question, answer well written check & refine Testbench Generation yes no, drop no, a refined version is provide collect data save as a valid sample simulation drop can compile? Yes No Data Prepare Parallel Refine Collect Valid Data all pass? #attempt <6? No Yes No trigger the regeneration Legend: LLM Response Verilog Verification Tools Yes Fig. 2. Multi-agent framework for testbench generation. make prompt = i, question, answer well written check & refine Testbench Generation yes no, drop no, a refined version is provide collect data save as a valid sample simulation drop can compile? Yes No Data Prepare Parallel Refine Collect Valid Data all pass? #attempt <3? No Yes No trigger the regeneration Testbench Generation simulationall pass? #attempt <3? Yes No trigger the regeneration Testbench Pregen Yes Yes No save as a valid sample Fig. 3. Multi-agent framework with pregenerating testbench. IV. EVALUATION To evaluate the effectiveness of our proposed workflow, we conducted a series of experiments using a set of benchmark Verilog modules. A base model, Qwen-Coder-7B-Instruct [13], was evaluated. We measured the performance of the LLM in generating syntactically correct and functionally relevant Verilog code. Additionally, we evaluated the quality of the automatically generated testbenches by assessing their ability to detect functional errors in the generated code. A. Experimental Setup Almost all experiments were conducted on Nvidia DGX H100 clusters, except for agents (GPT-4.1), which ran on remote cloud instances. The hardware and software specifi- cations are listed in Table I. TABLE I EXPERIMENTAL SETUP. ComponentSpecification HardwareNvidia DGX H100 Cluster (2-node) GPU8x Nvidia H100 GPUs per node CPU2x Intel Xeon Platinum 8480+ per node SoftwareRHEL 8.10 Python VersionPython v3.12.9 Packages vLLM v0.10.1.1, PyTorch v2.7.1, trl v0.21.0, transformers v4.55.4 Verilog CompilerIcarus Verilog v11.0 B. Refinement of VerilogEval v2 Our experiments use a refined version of VerilogEval v2, a benchmark for evaluating LLMs’ Verilog code generation. VerilogEval v2 [14] contains coding problems with specifi- cations and test cases to validate generated code. We refined the benchmark by updating test cases to cover more scenarios and edge cases, enhancing specifications with detailed require- ments, and addressing ambiguities from initial experiments. These improvements provide a clearer and more comprehen- sive assessment of LLM performance. The refinements led to notable performance gains; for example, CodeV-R1-7B-Distill rose from 65% to 70%, and CodeV-R1-7B (w/ DAPO) from 69% to 74%. C. Generating Reasoning Data from Filtered PyraNet Dataset A total of 6,704 samples from the Pyra-tb dataset (each including a corresponding testbench generated by VeriPrefer [7]’s data processing flow) were processed on the DGX H100 cluster. Using DeepSeek-R1, reasoning traces were generated for each sample, resulting in 1,386 functionally correct sam- ples, 4,421 syntactically correct samples, and 897 compilation errors. The entire run took about 55 hours. In total, 6,073,695 input tokens and 54,778,748 output tokens were processed, achieving an average throughput of 305.225 tokens per second. D. Generating Testbenches with Multi-Agent Framework We applied our multi-agent framework to generate test- benches for Verilog code from PyraNet [11] and DeepCir- cuitX [12]. The framework successfully produced testbenches capable of validating the functionality of the generated Verilog code. We evaluated their quality by measuring coverage and effectiveness in detecting functional errors. The results demon- strate that the multi-agent framework can generate high-quality testbenches that improve the verification process. Tables I and I present statistics for the testbenches generated for the PyraNet and DeepCircuitX datasets, respectively. For each dataset, we tested 50 questions using GPT-4.1 and averaged the results over three runs. In a run, stages were executed from top to bottom; for example, an average of 4.33 questions passed at the first stage with pregeneration, and all stages yielded 38 passes out of 50. The findings show that incorporating pregenerated testbenches increases the total number of passes while reducing API call counts, highlighting the efficiency and effectiveness of our framework, particularly on DeepCircuitX. TABLE I MULTI-AGENT TESTBENCH GENERATION STATISTICS ON PYRANET (AVERAGE OVER 3 RUNS, EACH RUN PROCESS 50 QUESTIONS). Stage w/o pregenerating testbenches w/ pregenerating testbenches Init testbench pass13.334.33 1st retry pass13.0013.00 2nd retry pass3.3310.33 3rd retry pass1.332.67 Revision init–2.00 4th retry pass0.003.67 5th retry pass0.001.33 6th retry pass0.330.67 Total #Pass ↑31.3338.00 API Count ↓237.67234.33 TABLE I MULTI-AGENT TESTBENCH GENERATION STATISTICS ON DEEPCIRCUITX (AVERAGE OVER 3 RUNS, EACH RUN PROCESSES 50 QUESTIONS). Stage w/o pregenerating testbenches w/ pregenerating testbenches Init testbench pass6.676.33 1st retry pass9.3314.00 2nd retry pass3.334.67 3rd retry pass1.333.33 Revision init–0.67 4th retry pass0.332.33 5th retry pass0.332.33 6th retry pass0.000.33 Total #Pass ↑21.3334.00 API Count ↓295.00247.67 E. Data Evaluation Results Table IV presents the performance of several models on the generated dataset. The results show that dataset size is critical for domain-specific tasks. The 1.3k high-quality reasoning samples did not improve performance, while the 6.7k samples with syntax errors increased accuracy to 47%. Our fine-tuned model, MA-tb-7B, produced with the multi-agent generation pipeline, achieved a pass@1 rate of 68%, which is competitive with state-of-the-art models while requiring less training data. This confirms the effectiveness of the multi-agent testbench generation framework and the supervised fine-tuning process for improving LLM performance in Verilog code generation. V. CONCLUSION This paper presents a workflow for AI-assisted hardware development and intelligent verification systems, where multi- agent LLMs generate training data for a non-reasoning base TABLE IV PERFORMANCE OF VARIOUS MODELS ON THE REFINED VERILOGEVAL V2 BENCHMARK. CategoryModel#Params #Training data pass@1 (%) Foundation Model Gemini2.5-Pron/an/a91 DeepSeek-V3.1671Bn/a74 DeepSeek-R1-0528685Bn/a81 Base Model Qwen-Coder-7B-Instruct7Bn/a40 Related Work CodeV-R1-7B-Distill7B87k70 CodeV-R1-7B (w/ DAPO)7B87k+3.1k74 Ours Pyra-tb-r-functional-7B7B1.3k40 Pyra-tb-r-all-7B7B6.7k47 MA-tb-7B7B38k68 model performing spec-to-Verilog generation. By integrating automated testbench generation, the approach improves data productivity. Experiments show that the fine-tuned model produces correct and relevant Verilog code. This work lays a foundation for LLM-based HDL generation and verification, with potential impact on digital circuit design automation. REFERENCES [1] A. A. Vivekananda and E. Enoiu, “Automated test case generation for digital system designs: A mapping study on vhdl, verilog, and systemverilog description languages,” Designs, vol. 4, no. 3, 2020. [2] T. B. Brown, B. Mann, N. Ryder et al., “Language models are few-shot learners,” in Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS). Red Hook, NY, USA: Curran Associates Inc., 2020. [3] OpenAI, J. Achiam, S. Adler, S. Agarwal et al., “GPT-4 technical report,” 2024. [4] D. Guo, D. Yang, H. Zhang et al., “DeepSeek-R1 incentivizes reasoning in llms through reinforcement learning,” Nature, vol. 645, no. 8081, p. 633–638, 9 2025. [5] C. Liu, X. Xu, Z. Chen, and B. Wang, “A universal-verification- methodology-based testbench for the coverage-driven functional veri- fication of an instruction cache controller,” Electronics, no. 18, 2023. [6] J. McEllin, R. Conway, and C. Ryan, “AVERT: An automatic verilog testbench generation tool for grammatical evolution,” in 2022 33rd Irish Signals and Systems Conference (ISSC), 2022, p. 1–8. [7] N. Wang, B. Yao, J. Zhou, Y. Hu, X. Wang, N. Guan, and Z. Jiang, “Insights from verification: Training a verilog generation llm with reinforcement learning with testbench feedback,” 2025. [8] Y. Wang, G. Sun, W. Ye, G. Qu, and A. Li, “VeriReason: Reinforce- ment learning with testbench feedback for reasoning-enhanced verilog generation,” 2025. [9] Y. Zhu, D. Huang, H. Lyu, X. Zhang, C. Li, W. Shi, Y. Wu, J. Mu, J. Wang, Y. Zhao, P. Jin, S. Cheng, S. Liang, X. Zhang, R. Zhang, Z. Du, Q. Guo, X. Hu, and Y. Chen, “QiMeng-CodeV-R1: Reasoning-enhanced verilog generation,” in Proceedings of the 39th Annual Conference on Neural Information Processing Systems (NeurIPS), 2025. [10] Q. Yu, Z. Zhang, R. Zhu et al., “DAPO: An open-source llm reinforce- ment learning system at scale,” 2025. [11] B. Nadimi, G. O. Boutaib, and H. Zheng, “PyraNet: A multi-layered hierarchical dataset for verilog,” in 2025 62nd ACM/IEEE Design Automation Conference (DAC). IEEE, Jun. 2025, p. 1–7. [12] Z. Li, C. Xu, Z. Shi, Z. Peng, Y. Liu, Y. Zhou, L. Zhou, C. Ma, J. Zhong, X. Wang, J. Zhao, Z. Chu, X. Yang, and Q. Xu, “DeepCircuitX: A comprehensive repository-level dataset for rtl code understanding, generation, and ppa analysis,” in 2025 IEEE International Conference on LLM-Aided Design (ICLAD), 2025, p. 204–211. [13] B. Hui, J. Yang, Z. Cui et al., “Qwen2.5-Coder technical report,” 2024. [14] N. Pinckney, C. Batten, M. Liu, H. Ren, and B. Khailany, “Revisiting VerilogEval: A year of improvements in large-language models for hardware code generation,” 2025.