Paper deep dive
Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation
Minghao Shao, Zeng Wang, Weimin Fu, Xiaolong Guo, Johann Knechtel, Ozgur Sinanoglu, Ramesh Karri, Muhammad Shafique
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 99%
Last extracted: 4/27/2026, 4:06:56 AM
Summary
This research demonstrates that for open-source Large Language Models (LLMs) used in Register-Transfer Level (RTL) generation, inference-time hyperparameter configuration (e.g., temperature, top_p) is a more significant determinant of performance than the choice of model architecture itself. By benchmarking 26 LLMs on VerilogEval and RTLLM using synthesis-in-the-loop evaluation, the authors found that hyperparameter-induced performance gaps (up to 25.5%) can exceed the differences between entire model families. The study highlights that default configurations are often suboptimal or even detrimental, and that optimal settings are both architecture-specific and benchmark-specific, necessitating task-aware hyperparameter selection for hardware design automation.
Entities (12)
Relation Signals (5)
Temperature → affects → LLM_Performance
confidence 100% · The sweep reveals absolute pass-rate gaps of up to 25.5% between the best and worst settings for the same LLM
VerilogEval → evaluates → LLM
confidence 100% · Benchmarking 26 open-source LLMs on VerilogEval and RTLLM
RTLLM → evaluates → LLM
confidence 100% · Benchmarking 26 open-source LLMs on VerilogEval and RTLLM
GPT-OSS 120B → partof → GPT-OSS
confidence 100% · GPT-OSS 120B, Qwen-3.5 397B, and GLM-5.
Yosys → performssynthesis → RTL
confidence 100% · Yosys provides a widely used framework for RTL synthesis
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Benchmarking of open-source LLMs for hardware design focuses on which LLMs to use, while treating inference-time decoding configuration as a secondary concern. This work shows that it matters more how an LLM is configured than which model is selected. Benchmarking 26 open-source LLMs on VerilogEval and RTLLM with synthesis-in-the-loop evaluation, the study first maps the current capability landscape and then conducts an extensive 108-configuration hyperparameter sweep on three prominent models. The sweep reveals absolute pass-rate gaps of up to 25.5% between the best and worst settings for the same LLM, which is 5x larger than the average spread observed across various model families under their respective default configurations. Ranking all configurations by Spearman's $\rho$ across the two benchmark suites yields near-zero correlation, demonstrating that optimal configurations do not transfer. These results show that benchmarking conducted under default hyperparameters confounds model capabilities with configuration effects. Realizing the full potential of open-source LLMs for RTL generation requires architecture and benchmark aware hyperparameter selection, as enabled by the proposed methodology.
Tags
Links
- Source: https://arxiv.org/abs/2604.17102v1
- Canonical: https://arxiv.org/abs/2604.17102v1
Trouble viewing inline? Open PDF directly →
Full Text
35,355 characters extracted from source content.
Expand or collapse full text
Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation Minghao Shao †‡ Zeng Wang † , Weimin Fu ¶ , Xiaolong Guo ¶ , Johann Knechtel ‡ , Ozgur Sinanoglu ‡ , Ramesh Karri † , Muhammad Shafique ‡ † NYU Tandon School of Engineering, USA ‡ NYU Abu Dhabi, UAE ¶ Kansas State University, USA Email: shao.minghao, zw3464, johann, ozgursin, rkarri, muhammad.shafique@nyu.edu weiminf, guoxiaolong@ksu.edu Abstract—Benchmarking of open-source LLMs for hardware design focuses on which LLMs to use, while treating inference- time decoding configuration as a secondary concern. This work shows that it matters more how an LLM is configured than which model is selected. Benchmarking 26 open-source LLMs on VerilogEval and RTLLM with synthesis-in-the-loop evaluation, the study first maps the current capability landscape and then conducts an extensive 108-configuration hyperparameter sweep on three prominent models. The sweep reveals absolute pass- rate gaps of up to 25.5% between the best and worst settings for the same LLM, which is 5x larger than the average spread observed across various model families under their respective default configurations. Ranking all configurations by Spearman’s ρ across the two benchmark suites yields near-zero correlation, demonstrating that optimal configurations do not transfer. These results show that benchmarking conducted under default hy- perparameters confounds model capabilities with configuration effects. Realizing the full potential of open-source LLMs for RTL generation requires architecture and benchmark aware hyperpa- rameter selection, as enabled by the proposed methodology. I. INTRODUCTION Large language models (LLMs) have shown significant capability in automated hardware description generation [1], [2]. Register-transfer level (RTL) design remains a bottleneck in semiconductor manufacturing, requiring engineers to bal- ance functional accuracy with timing and area constraints. Although AI approaches can accelerate chip development, passing software simulations alone is insufficient: for machine- generated hardware to replace golden modules, it must survive the physical synthesis pipeline, making synthesis-backed val- idation a rigorous measure of design quality [3]. Recent evaluations of LLM-aided RTL generation indicate that commercial LLMs maintain higher accuracy than their open-source counterparts [4], yet open-source LLMs offer distinct practical advantages. Open-weights LLMs enable cost- free local deployment without recurring API costs and allow domain-specific fine-tuning that is unavailable with commer- cial equivalents [5]. For the semiconductor industry, where intellectual property protection is a concern [6], open-source LLMs enable isolated on-premise inference, removing the risk of sensitive design data reaching third-party servers. 0 50 100 VerilogEval (%) Pass@1 (solid) vs Pass@5 (faded) GLM 4.5 DeepSeek V3 GLM 4.6 DeepSeek V3.1 Qwen3.5 122B GLM 4.7 Qwen3.5 35B GLM 5 MiniMax M2.5MiniMax M2.1 Kimi K2.5 Qwen3.5 397B Step 3.5 Flash DeepSeek V3.2 GPT-OSS 120B Seed 1.6 Kimi K2 Qwen3 235B Qwen3 Coder LLaMA 4 Maverick Qwen3 30B GPT-OSS 20B Mistral Large LLaMA 4 Scout Devstral Mistral Nemo 0 50 100 RTLLM (%) Fig. 1: Pass rates of 26 open-source LLMs on VerilogEval and RTLLM. Despite these advantages, the performance ceilings and optimization strategies for open-source models remain un- derexplored [7]. A critical but largely overlooked factor is inference-time decoding configuration. Existing evaluations compare models under default or unspecified hyperparameters, implicitly treating configuration as a minor implementation detail. Studies on general code generation show that decoding choices such as temperature and top-p can materially change functional success rates [8]. This raises a key question: when model A outperforms model B on an RTL benchmark, how much of that gap reflects genuine model capability, and how much is an artifact of the hyperparameters used? To answer this question, we evaluate 26 open-source mod- els across VerilogEval and RTLLM with synthesis-in-the- loop assessment, first establishing a performance landscape to quantify inter-model differences, and then conducting a 108-configuration hyperparameter sweep on three architec- tures. The sweep shows that the same model’s pass rate varies by up to 25.5% depending solely on decoding settings, a configuration-induced variation that exceeds the pass-rate spread between entire model families observed in the land- scape evaluation. Spearman’s rank correlation of configuration rankings across the two benchmarks is near zero for all three models, indicating that a configuration optimized on one benchmark has no predictive value for performance on another. In summary, our primary contributions are: arXiv:2604.17102v1 [cs.AR] 18 Apr 2026 2030405060708090100 Verification Coverage (Best-of-5, complexity-weighted) Fewer Tasks Passed More Complex Tasks Passed 20 40 60 80 100 Global HQI (Best-of-5, reference-backed) Low Quality High Quality A B C D F Models (by Coverage ) GLM 4.5 DeepSeek V3 DeepSeek V3.1 GLM 4.6 Qwen3.5 122B GLM 4.7 Qwen3.5 35B GLM 5 Kimi K2.5 MiniMax M2.1 MiniMax M2.5 DeepSeek V3.2 Step 3.5 Flash GPT-OSS 120B Qwen3 235B Qwen3.5 397B Kimi K2 Qwen3 Coder Seed 1.6 LLaMA 4 Maverick GPT-OSS 20B Qwen3 30B Mistral Large LLaMA 4 Scout Devstral Mistral Nemo 384042444648505254 20 25 30 35 40 Qwen3 Coder Kimi K2 GPT-OSS 20B Qwen3 30B Mistral Large LLaMA 4 Scout Devstral 54555657585960616263 56 58 60 62 64 MiniMax M2.5 GLM 5 Kimi K2.5 DeepSeek V3.2 GPT-OSS 120B Qwen3.5 397B GLM 4.5 DeepSeek V3 GLM 4.6 DeepSeek V3.1 Qwen3.5 122B GLM 4.7 Qwen3.5 35B GLM 5 MiniMax M2.5MiniMax M2.1 Kimi K2.5 Qwen3.5 397B Step 3.5 Flash DeepSeek V3.2 GPT-OSS 120B Seed 1.6 Kimi K2 Qwen3 235B Qwen3 Coder LLaMA 4 Maverick Qwen3 30B GPT-OSS 20B Mistral Large LLaMA 4 Scout Devstral Mistral Nemo 0 25 50 75 100 Pass Rate (%) Pass@1 (solid) vs Pass@5 (faded) AI Organization Golden Baseline Alibaba (Qwen) ByteDance (Seed) DeepSeek Meta (LLaMA) MiniMax Mistral MoonshotAI (Kimi) OpenAI (GPT-OSS) StepFun Zhipu (GLM) GLM 4.5 DeepSeek V3 Seed 1.6 DeepSeek V3.1 GLM 4.6 Qwen3.5 122B GLM 4.7 LLaMA 4 Maverick Step 3.5 Flash Qwen3.5 35B MiniMax M2.1 Qwen3 235B Mistral Nemo Fig. 2: Pass-rate and HQI landscape for 26 language models. Top: pass@1 (solid) and pass@5 (faded) sorted by descending pass@5. Bottom: Complexity-weighted verification coverage versus Global HQI under synthesis-in-the-loop evaluation. • We benchmark 26 open-source LLMs with synthesis-in-the- loop evaluation to establish the magnitude of inter-model performance differences, providing a baseline against which the impact of hyperparameters can be measured. • We demonstrate through a 108-configuration sweep that hyperparameter-induced performance variation (up to 25.5%) can exceed inter-model variation, establishing inference-time configuration as a primary determinant of open-source LLM performance for RTL generation. • We reveal that optimal hyperparameter settings are architecture-specific and benchmark-specific (Spearman’s ρ ≈ 0 across benchmarks), ruling out universal “recom- mended configurations” and motivating task-aware tuning. I. BACKGROUND Large language models have attracted increasing interest in EDA, spanning hardware generation, design optimization, hardware security, and logic synthesis [9]. Their success in natural-language-to-code tasks [10], reinforced by execution- based benchmarks such as HumanEval and MBPP [11], [12], encouraged researchers to target synthesizable RTL [1]. RTL generation is harder to evaluate than software synthesis: a candidate design must be syntactically valid, functionally correct under simulation, and useful for downstream imple- mentation [13]. VerilogEval introduced automated simulation- based checking with pass@k as a central metric [14]; RTLLM extended the scope to natural-language-to-RTL generation with syntax, functionality, and design-quality assessment [13]; and OpenLLM-RTL expanded the benchmarks with updated tasks and larger training corpora [7]. Model-focused efforts including VeriGen and RTLCoder showed that domain specialization can substantially strengthen HDL generation, with even relatively compact models achiev- ing competitive benchmark results [5], [15]. Agentic ap- proaches now extend RTL generation beyond one-shot prompt- ing by incorporating iterative refinement, tool use, and multi- step reasoning into the design loop [16]. Current evaluations still follow early code benchmark logic, ending at simulation pass rates. This leaves a significant gap, as RTL is only an intermediate representation whose value ultimately depends on post-synthesis implementation quality [17]. This limitation has motivated re-evaluations of hardware code benchmarks, which emphasize how strongly reported performance depends on benchmark design and evaluation protocol [18]. Open- source EDA infrastructure now enables richer assessment: Yosys provides a widely used framework for RTL synthesis, and the Nangate45 open cell library offers a reproducible tech- nology context for academic quality-of-results studies [19], [20], enabling the synthesis-in-the-loop evaluation. Open-source models therefore deserve attention, not simply as cheaper stand-ins for proprietary systems. Recent work suggests that their performance ceiling can change substan- tially with model family, training data, and domain spe- cialization [4], [5]. Studies on code generation show that inference-time decoding choices such as temperature and top- p sampling can materially change functional success rates [8], but existing RTL evaluations typically report results under default or unspecified hyperparameters, making it impossible to determine whether observed model rankings reflect intrinsic capability differences or incidental configuration choices. For RTL generation, this means a fair comparison of open-weight models must distinguish the capability of the pretrained model from gains or losses introduced by runtime configuration. Basic Sequential Arithmetic & Datapath Waveform Reverse Eng. Combinational Logic Counters & Sequences Bug Fixing Memory & Buffers FSM & Protocols 8084756570758465806563657070566665555560595960626533 8077767881777772737173697064635765585858575349474334 92100926883839254794963545975655462454947465555453017 8989874884908542844441464081444645443941394137353954 6770655765596855575851515057555456505253464345394014 7575734074757438753838383875383838383838382238393875 7878805378785749584039285960334350502859431989040 3127285424262949264446545224204313101814131781083 Best-of-5 HQI MiniMax M2.5 GLM 5 Kimi K2.5 GLM 4.5 DeepSeek V3.2 GPT-OSS 120B Qwen3.5 397B DeepSeek V3 Seed 1.6 DeepSeek V3.1 GLM 4.6 Qwen3.5 122B GLM 4.7 LLaMA 4 Maverick Step 3.5 Flash Qwen3.5 35B MiniMax M2.1 Qwen3 235B Qwen3 Coder Kimi K2 GPT-OSS 20B Qwen3 30B Mistral Large LLaMA 4 Scout Devstral Mistral Nemo Basic Sequential Combinational Logic Arithmetic & Datapath Bug Fixing Waveform Reverse Eng. Counters & Sequences Memory & Buffers FSM & Protocols 6578685864586762725358525864485249475052454560385519 8479794177757938783439443774394039333733343335313538 6564665162595959513854545357494050364340382941303617 7574683174757438742838273475333538293726361838343839 7365784766608042543445404152514140284327273539202410 526057455648555150384840465142394237414429314328385 4356713044354639281630244532232621181227111083016 25212435201824341925293433131322104107676341 Expected HQI per Attempt 0 20 40 60 80 100 Best-of-5 HQI (0 100) 0 20 40 60 80 100 Expected HQI per Attempt (0 100) Fig. 3: HQI across eight hardware categories and 26 models. Models are ordered left-to-right by Global HQI; categories top- to-bottom by top models’ average scores. Top: best-of-five capability ceiling. Bottom: single-attempt deployment quality. I. EXPERIMENT SETUP a) Model Selection: To ensure broad model coverage, we evaluate 26 open-source models spanning eight families: Qwen (Qwen-3.5 397B/122B/35B, Qwen-3 235B/30B/Coder), GLM(GLM5/4.7/4.6/4.5),DeepSeek(V3.2/V3.1/V3), Mistral (Large/Nemo/Devstral), Kimi (K2.5/K2), MiniMax (M2.5/M2.1), LLaMA (4 Maverick/4 Scout), and GPT-OSS (120B/20B), as well as Seed 1.6 and Step 3.5 Flash. b) Benchmarks: Two established RTL generation bench- marks, VerilogEval [14] and RTLLM [13], are used, which provide complementary evaluation settings and reduce the risk that conclusions are driven by a single task style. c) Hyperparameter Setup: Three representative models are selected for the hyperparameter sweep: GPT-OSS 120B, Qwen-3.5 397B, and GLM-5. These are high-performing members of distinct families whose pilot experiments (default vs. modified temperature) revealed different levels of sensi- tivity, enabling a test of whether hyperparameter effects are architecture-dependent. We vary four decoding parameters: temperature (distribution entropy), top_p (probability- mass truncation), repetition_penalty (token reuse sup- pression), and presence_penalty (novelty bias). Together they cover the principal dimensions of decoding behavior: determinism, search-space restriction, repetition, and diversity. d) Metrics: We report pass@1 through pass@5 on both benchmarks as primary quality metrics, alongside HQI, ex- pected HQI, and complexity-weighted coverage for post- synthesis design quality. The Hardware Quality Index (HQI) follows [21]: a design that fails any evaluation gate receives HQI = 0. For a passing design on task t, let ˆa, ˆ d, ˆw denote its post-synthesis area, delay, and warning count, and a ∗ t , d ∗ t , w ∗ t the golden-reference values. The normalized cost is: cost = 0.5 ˆa a ∗ t + 0.5 ˆ d d ∗ t + 0.1 max 0, ˆw− w ∗ t ,(1) and the attempt is scored as HQI = min(100/cost, 100), where 100 indicates parity with the golden reference. Per-task scores are aggregated using complexity weights C t derived from AST dependency-edge counts: Global HQI takes the best-of-five ceiling per task, while Expected HQI averages over all five attempts. We also report deployment efficiency metrics (cost per task, throughput, time-to-first-token, and completion tokens). For the hyperparameter study, we report pass@1 and pass@5 on both benchmarks; the gap between the best and worst settings directly quantifies how much configuration choice affects performance relative to model choice. e) Framework Implementation: The evaluation is a uni- fied Python pipeline that merges VerilogEval and RTLLM, removes duplicated tasks by RTL matching, queries each prompt through an OpenAI-compatible API, and extract the first fenced Verilog block as the candidate design. Generated RTL is evaluated with Icarus Verilog for syntax checking and simulation and Yosys with ABC for synthesis and structural analysis. The pipeline records syntax validity, synthesizability, simulation pass, and synthesis-derived statistics for each task. IV. EVALUATION RESULTS A. Performance Landscape 1) Pass Rate vs. HQI: Figure 2 ranks all 26 models by pass@5 alongside pass@1, showing a consistent gap between the two metrics across all evaluated models. Even models with high synthesis quality such as GLM 5 exhibit a notable decline from pass@5 to pass@1, indicating that single-attempt deployment remains unreliable across the model pool. At the model-family level, DeepSeek, GLM, and Qwen dominate both pass-rate rankings and Global HQI, while the Mistral family clusters toward the lower end of both axes. The bottom panel shows that pass rate and synthesis quality do not always GLM 4.5 Qwen3.5 122B MiniMax M2.5MiniMax M2.1 Qwen3.5 35B Qwen3.5 397B Kimi K2.5 Step 3.5 Flash Seed 1.6 Qwen3 30B GLM 5 Kimi K2 GLM 4.6GLM 4.7 Devstral Mistral Large Qwen3 Coder Qwen3 235B DeepSeek V3.1 DeepSeek V3 LLaMA 4 Maverick GPT-OSS 120B DeepSeek V3.2LLaMA 4 Scout GPT-OSS 20B Mistral Nemo $0.000 $0.001 $0.002 $0.003 $0.004 $0.005 $0.006 $0.007 $0.008 Avg Cost / Task (USD) Cost varies >10×, yet does not correlate with model scale GPT-OSS: competitive HQI at <1/5 the cost of GLM 4.5 (a) Cost GPT-OSS 20B Step 3.5 Flash Qwen3.5 122B Qwen3.5 35B Devstral GPT-OSS 120B LLaMA 4 Scout Qwen3 Coder Qwen3.5 397B GLM 4.6 Mistral Large MiniMax M2.5MiniMax M2.1 LLaMA 4 Maverick GLM 4.5 GLM 5 Qwen3 30B Mistral Nemo GLM 4.7 Kimi K2.5 Qwen3 235B Seed 1.6 DeepSeek V3.2 DeepSeek V3 Kimi K2 DeepSeek V3.1 0 100 200 Median Output tok/s GPT-OSS 20B: 3x faster than median ideal for batch DeepSeek family: high quality but lowest throughput (b) Throughput Step 3.5 Flash DeepSeek V3.1 GLM 5 DeepSeek V3 DeepSeek V3.2 MiniMax M2.1MiniMax M2.5 Kimi K2 GLM 4.7 Kimi K2.5 GLM 4.6 Qwen3 Coder GLM 4.5 Qwen3 235B Qwen3.5 397B Qwen3 30B Mistral Nemo GPT-OSS 20B GPT-OSS 120B Seed 1.6 LLaMA 4 Scout LLaMA 4 Maverick Mistral Large Devstral Qwen3.5 35B Qwen3.5 122B 0.0s 0.5s 1.0s 1.5s 2.0s 2.5s 3.0s 3.5s 4.0s 4.5s 5.0s Median TTFT (sec) Step 3.5 Flash: TTFT outlier 4x slower than median Most models <1s TTFT viable for interactive use (c) TTFT Qwen3 30B Step 3.5 Flash MiniMax M2.5MiniMax M2.1 GLM 4.5 Qwen3.5 35B GPT-OSS 120B Seed 1.6 Qwen3.5 122BQwen3.5 397B GPT-OSS 20B Qwen3 235B DeepSeek V3.2 LLaMA 4 Maverick LLaMA 4 Scout Mistral Large Kimi K2.5 GLM 4.6 DeepSeek V3.1 Devstral Qwen3 Coder DeepSeek V3 GLM 5 GLM 4.7 Kimi K2 Mistral Nemo 0 10000 20000 Comp. Tokens / Task Qwen3 30B: excessive commentary: same as Step 3.5 Flash alongside RTL Concise output correlates with structural accuracy (d) Verbosity Fig. 4: Inference performance metrics. align: DeepSeek V3.2 and GPT-OSS 120B achieve higher Global HQI relative to their coverage, indicating strong post- synthesis quality on the tasks they do solve. Generational progression does not uniformly improve RTL capability either; within the GLM family, newer releases trade coverage breadth for design depth, a distinction that pass-rate metrics alone do not capture. Under default configurations, the pass@5 differ- ence between the top-ranked and mid-ranked model families (e.g., DeepSeek V3.2 vs. LLaMA 4 Maverick) is approxi- mately 15–20 percentage points. Section IV-B shows that a single model’s performance can swing by up to 25.5% under different hyperparameter settings alone, placing configuration- induced variation on the same scale as inter-family differences. 2) Category-Level Analysis: Figure 3 decomposes synthe- sis quality across eight hardware categories. Basic Sequential, Combinational Logic, and Waveform Reverse Engineering score highest, with top models reaching near-golden levels, whereas FSM & Protocols and Memory & Buffers remain difficult even for the strongest models. Both panels reveal a systematic reliability drop from best-of-five to per-attempt evaluation, widening most in these already-difficult categories. This compounding effect suggests the hardest categories are most exposed to hyperparameter misconfiguration, though confirming requires per-category analysis not conducted here. 3) Efficiency Evaluation: Figure 4 profiles four operational metrics. Cost per task (Figure 4a) varies by over an order of magnitude and does not correlate with model scale: GPT-OSS family achieves competitive HQI at substantially lower cost than GLM 4.5 or Qwen-3.5 397B. Throughput and latency (Figures 4b and 4c) show that inter- active and batch workloads may favor different architectures, but these metrics are primarily determined by the serving infrastructure and are orthogonal to the hyperparameter effects studied in Section IV-B. Verbosity (Figure 4d) is more directly related to decoding configuration: models like Qwen3 30B and Step 3.5 Flash produce high token counts with large variance, indicating excessive natural-language commentary alongside the generated RTL. GLM 5 and Kimi K2.5 maintain high synthesis quality with minimal verbosity, confirming that concise output correlates with structural accuracy. Because decoding hyperparameters such as temperature and penalty terms influence output length, verbosity may interact with the configuration effects examined in Section IV-B, though the present study does not isolate this relationship. B. Hyperparameter Sweeping 1) Performance Gap Under Hyperparameter Sweeping: Table I quantifies the central finding: across the three models, the absolute quality gap between the best and worst hyper- parameter settings reaches 25.5%, matching or exceeding the 15–20 percentage-point spread between top-tier and mid-tier model families observed in Section IV-A. A well-tuned GPT- OSS 120B on RTLLM (pass@1 = 0.575) outperforms a poorly configured Qwen-3.5 397B (pass@1 = 0.362) despite having less than one-third the parameters. The sensitivity concentrates on first-attempt correctness: pass@1 gaps exceed pass@5 counterparts for all three models, indicating that suboptimal configurations impair immediate solution accuracy rather than erasing underlying hardware knowledge. Model sensitivity varies: Qwen-3.5 397B and GPT- OSS 120B show double-digit fluctuations across most bench- mark metrics, while GLM-5 maintains narrower gaps. This TABLE I: Performance gap (∆) between the best and worst hyperparameter settings for three open-source models on Ver- ilogEval and RTLLM. Each setting is a tuple of (temperature, top p, repetitionpenalty, presencepenalty). ModelBenchmark Metric Best Score (HP) Worst Score (HP) Gap (∆) GPT-OSS (120B) VerilogEval pass@10.626 (0.4,1,1,-1)0.503 (1.2,1,1.1,0)0.123 pass@5 0.710 (0.4,1,1.1,-1)0.658 (1.2,1,1.2,1)0.052 RTLLM pass@10.575 (0,0.7,1,-1)0.319 (0.8,1,1,0)0.255 pass@5 0.681 (0.8,1,1.1,-1)0.553 (0.4,0.4,1,0)0.128 Qwen-3.5 (397B) VerilogEval pass@1 0.845 (0,0.4,1.1,-1)0.600 (1.2,1,1.2,0)0.245 pass@5 0.942 (1.2,0.7,1.1,1)0.858 (1.2,1,1.2,1)0.084 RTLLM pass@1 0.596 (1.2,0.4,1,0)0.362 (0.4,1,1.1,0)0.234 pass@5 0.681 (0,0.7,1.2,1)0.575 (0.8,1,1.2,-1)0.106 GLM-5 VerilogEval pass@1 0.671 (0,0.4,1.1,0)0.574 (1.2,1,1.2,0)0.097 pass@5 0.736 (0.8,0.7,1.2,0) 0.684 (0.8,0.4,1.1,1)0.052 RTLLM pass@1 0.617 (0.8,0.7,1,0)0.447 (1.2,1,1,0)0.170 pass@5 0.702 (1.2,1,1.2,1)0.596 (0,0.7,1.2,1)0.106 TABLE I: Default (1.0, 1.0, 1.0, 0) configuration rank among 109 hyperparameter settings. Rank indicates the position of the default setting (lower is better); color encodes percentile: blue (≥90 th ), green (61–89 th ), orange (26–60 th ), red (≤25 th ). ModelMetricDefault Best Worst∆ gap w ∆ gap d Rank GPT-OSS 120B VE pass@1.574.626.503.123.05246/109 VE pass@5.677.710.658.052.03247/109 RTL pass@1.383.575.319.255.19296/109 RTL pass@5.596.681.553.128.08551/109 Qwen-3.5 397B VE pass@1.619.845.600.245.226106/109 VE pass@5.710.942.710.232.232109/109 RTL pass@1.468.596.362.234.12861/109 RTL pass@5.575.681.575.106.106100/109 GLM-5 VE pass@1.626.671.574.097.04561/109 VE pass@5.723.735.684.052.0134/109 RTL pass@1.489.617.447.170.12882/109 RTL pass@5.660.702.596.106.04328/109 implies a robustness–ceiling trade-off: GLM-5 offers safer defaults but a lower peak, whereas the others reach higher ceilings at the cost of greater configuration dependence. 2) Default Configuration Positioning: Table I reveals that default configurations are frequently far from optimal, in the worst cases occupying the very bottom of the 109-setting rank- ing (108 sweeping + default). Qwen-3.5 397B is the clearest case: its default ranks last on VerilogEval pass@5 and near- last on pass@1, leaving over 22% of achievable performance unrealized. GPT-OSS 120B tells a similar story on RTLLM, where its default ranks 96th on pass@1 with nearly 19% left on the table. GLM-5 is the most robust, reaching the top five on VerilogEval pass@5, though it still drops to rank 82 on RTLLM pass@1. These results concretely indict current evaluation practice: the majority of default configurations are not merely suboptimal but actively detrimental for RTL generation, in multiple documented cases placing models at their empirical floor and obscuring the intrinsic capability that these models otherwise possess. 50.052.555.057.560.062.5 39 42 45 48 51 VerilogEval Global HQI GPT-OSS 120B 606570758085 48 52 56 60 64 Qwen-3.5 397B 5860626466 44 46 48 50 52 54 GLM-5 303540455055 Pass Rate 39 42 45 48 51 RTLLM Global HQI 354045505560 Pass Rate 48 52 56 60 64 4448525660 Pass Rate 44 46 48 50 52 54 Pareto FrontBest PassBest HQIWorst PassWorst HQIDefault Fig. 5: Scatter plots of Pass Rate vs. HQI across differ- ent LLMs and datasets. The Pareto frontier (dashed line) highlights the trade-off between functional correctness and comprehensive hardware quality. 3) Pass Rate–HQI Trade-off: Figure 5 plots pass rate against Global HQI for all 109 configurations. The overall trend is positive: configurations that achieve higher pass rates generally yield higher HQI, confirming that functional correct- ness and synthesis quality are broadly aligned. However, the configuration that maximizes pass rate never exactly coincides with the one that maximizes HQI; the two optima occupy distinct positions along a Pareto frontier, and a subset of configurations deviate from the main trend, achieving strong pass rates with mediocre HQI or vice versa. This divergence is most visible for Qwen-3.5 397B on VerilogEval, where the best-pass and best-HQI settings sit at opposite ends of the trade-off curve. In practice, while pass rate remains a reasonable proxy for synthesis quality in most configura- tions, practitioners who require Pareto-optimal designs must treat hyperparameter selection as a multi-objective problem, calibrating not only for the model and benchmark but also for whether the deployment objective prioritizes functional correctness, post-synthesis quality, or a balance of both. The default configurations tend to cluster in the lower-left region of the scatter, far from the Pareto frontier, consistent with the ranking analysis in Table I and reinforcing that out-of-the-box settings underperform on both objectives simultaneously. 4) Hyperparameter Correlation: Figure 6 maps the correla- tion between each decoding parameter and generation quality across the three models. No single parameter universally gov- erns RTL output. Temperature and top p correlate negatively with pass@1 for GPT-OSS 120B and Qwen-3.5 397B, indi- cating that broader sampling introduces more noise than im- provement under Verilog’s strict syntactic constraints, whereas GLM-5 shows weaker and occasionally positive responses, suggesting greater tolerance to randomness. The penalty pa- rameters reveal further architecture-specific patterns: Qwen- 3.5 397B’s strong negative correlation with presence penalty on VerilogEval suggests that forcing unseen tokens causes de- viation from synthesizable logic, while GLM-5’s mild positive response to repetition penalty may help suppress redundant logic blocks. GPT-OSS 120B exhibits intermediate behavior. These profiles caution against a “best practice” configuration; TempTopPRepPPresP VE@1 VE@5 RTL@1 RTL@5 -0.17-0.19-0.130.00 -0.150.11-0.08-0.05 -0.26-0.14-0.41-0.02 -0.060.010.100.07 GPT OSS 120B TempTopPRepPPresP -0.33-0.27-0.35-0.23 -0.020.02-0.04-0.23 -0.05-0.04-0.17-0.22 -0.01-0.03-0.15-0.07 Qwen 3.5 397B TempTopPRepPPresP -0.08-0.23-0.080.05 0.200.160.100.06 -0.240.06-0.040.10 0.11-0.100.080.05 GLM 5 0.4 0.2 0.0 0.2 0.4 Corr Fig. 6: Hyperparameter–performance correlation across the three models. the next section reveals a fundamental limitation. 5) Cross-Benchmark Non-Transferability: The correlation patterns in Figure 6 often weaken or reverse between Verilo- gEval and RTLLM, raising the question of whether a config- uration optimized on one benchmark retains its advantage on the other. To test this directly, all 108 configurations are ranked by pass@1 on each benchmark independently and Spearman’s rank correlation ρ between the two rankings. As shown in Table I, the correlations are weak to negligible. GPT-OSS 120B shows a weak positive correlation that, while statistically significant, explains less than 6% of the ranking variance. For Qwen-3.5 397B and GLM-5, the corre- lations are not statistically significant, meaning that a hyperpa- rameter setting that ranks among the best on VerilogEval has essentially no predictive value for its ranking on RTLLM. This non-transferability is the strongest evidence against a universal “recommended configuration” for open-source LLMs in RTL generation. Practitioners must treat hyperparameter selection as a task-specific calibration, rather than a one-time decision. TABLE I: Spearman’s rank correlation (ρ) of hyperparam- eter configuration rankings between VerilogEval and RTLLM (pass@1, 108 configurations per model). GPT-OSS 120BQwen-3.5 397BGLM-5 ρ p-value ρ p-valueρ p-value 0.230.0160.150.121 −0.050.590 6) Model Sensitivity on Hyperparameters: Figure 7 shows the distributional spread of pass rates under varying con- figurations. Pass@5 distributions remain compact across all three models, indicating stable underlying hardware knowl- edge, whereas pass@1 distributions exhibit wider spreads and downward outliers, confirming that first-attempt correctness is more sensitive to decoding choices. The effect is stronger on RTLLM, whose multi-module and protocol-level tasks [13] amplify configuration impact: both GPT-OSS 120B and Qwen- 3.5 397B show outliers approaching near-zero pass rates, demonstrating that a poorly configured top-tier model can underperform a well-configured smaller one. GLM-5 maintains the tightest boxes on both benchmarks, safest when per-task tuning is infeasible. GPT-OSS 120B and Qwen-3.5 397B reach higher peaks but require careful con- figuration to avoid degradation. These distributions reinforce 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Pass Rate VerilogEval (VE): Robustness across HP Metric Pass@1 Pass@5 GPT OSS 120BQwen 3.5 397BGLM 5 Model 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Pass Rate RTLLM (RTL): Robustness across HP Fig. 7: Distribution of pass rates under 108 hyperparameter configurations for each model. the central point: the same model can appear mediocre or competitive depending on its decoding configuration. V. CONCLUSION This work establishes that for open-source LLMs in RTL generation, how a model is configured can matter more than which model is chosen. Three findings support this con- clusion at increasing levels of specificity. First, the 25.5% pass-rate gap induced by hyperparameter variation within a single model exceeds the gap between many model families evaluated under default settings, and a well-tuned smaller model can surpass a poorly configured model with over 3× the parameters. Second, sensitivity to decoding configura- tion is architecture-specific: GLM-5 is robust across settings while GPT-OSS 120B and Qwen-3.5 397B exhibit double- digit swings, creating a robustness–ceiling trade-off. Third, optimal configurations are benchmark-specific: Spearman’s rank correlation of configuration rankings across VerilogEval and RTLLM is near zero, ruling out “recommended settings” and requiring task-aware calibration. These results have a direct methodological implication: benchmark comparisons of open-source LLMs conducted un- der default or unspecified hyperparameters conflate model quality with configuration quality. Reporting RTL generation results without specifying decoding parameters is incomplete, in the same way that reporting training results without speci- fying the learning rate would be. For practitioners deploying open-source LLMs in secure, on-premise EDA workflows, these findings indicate that a modest investment in per-task hyperparameter calibration can yield performance that rivals or exceeds much larger models, making open-source LLMs a viable solution when properly configured. REFERENCES [1] S. Thakur, B. Ahmad et al., “Benchmarking large language models for automated verilog rtl code generation,” 2022. [Online]. Available: https://arxiv.org/abs/2212.11140 [2] Z. Wang, L. Alrahis et al., “Llms and the future of chip design: Unveiling security risks and building trust,” in 2024 IEEE Computer Society Annual Symposium on VLSI (ISVLSI). IEEE, 2024, p. 385–390. [3] M. Abdelatty, M. Nouh et al., “Pluto: A benchmark for eval- uating efficiency of llm-generated hardware code,” arXiv preprint arXiv:2510.14756, 2025. [4] F. Cui, C. Yin et al., “Origen: Enhancing rtl code generation with code- to-code augmentation and self-reflection,” in Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, 2024, p. 1–9. [5] S. Liu, W. Fang et al., “Rtlcoder: Fully open-source and efficient llm- assisted rtl code generation technique,” IEEE Transactions on Computer- Aided Design of Integrated Circuits and Systems, vol. 44, no. 4, p. 1448–1461, 2024. [6] Z. Wang, M. Shao et al., “Verileaky: Navigating ip protection vs utility in fine-tuning for llm-driven verilog coding,” in 2025 IEEE International Conference on LLM-Aided Design (ICLAD). IEEE, 2025, p. 100–107. [7] Y. Lu et al., “Openllm-rtl: Open dataset and benchmark for llm-aided design rtl generation,” arXiv preprint arXiv:2404.06117, 2024. [8] Y. Zhu, J. Li et al., “Hot or cold? adaptive temperature sampling for code generation with large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 1, 2024, p. 437–445. [9] Z. Wang, M. Shao et al., “Netdetox: Adversarial and efficient evasion of hardware-security gnns via rl-llm orchestration,” arXiv preprint arXiv:2512.00119, 2025. [10] M. Shao, A. Basit et al., “Survey of different large language model ar- chitectures: Trends, benchmarks, and challenges,” IEEE access, vol. 12, p. 188 664–188 706, 2024. [11] M. Chen, J. Tworek et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [12] J. Austin, A. Odena et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [13] Y. Lu, S. Liu et al., “Rtllm: An open-source benchmark for design rtl generation with large language model,” in 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC).IEEE, 2024, p. 722–727. [14] M. Liu, N. Pinckney et al., “Verilogeval: Evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 2023, p. 1–8. [15] S. Thakur, B. Ahmad et al., “Verigen: A large language model for verilog code generation,” ACM Transactions on Design Automation of Electronic Systems, vol. 29, no. 3, p. 1–31, 2024. [16] Y. Zhao, H. Zhang et al., “Mage: A multi-agent engine for automated rtl code generation,” in 2025 62nd ACM/IEEE Design Automation Conference (DAC). IEEE, 2025, p. 1–7. [17] D. Garcia-Gasulla, G. Kestor et al., “Turtle: A unified evaluation of llms for rtl generation,” in 2025 ACM/IEEE 7th Symposium on Machine Learning for CAD (MLCAD). IEEE, 2025, p. 1–12. [18] N. Pinckney, C. Batten et al., “Revisiting verilogeval: A year of improvements in large-language models for hardware code generation,” ACM Transactions on Design Automation of Electronic Systems, vol. 30, no. 6, p. 1–20, 2025. [19] C. Wolf, “Yosys open SYnthesis suite,” https://yosyshq.net/yosys/, 2013. [20] Nangate Inc., “The NanGate 45nm open cell library,” https://si2.org/, 2008. [21] W. Fu, Z. Wang et al., “Synthesis-in-the-loop evaluation of llms for rtl generation: Quality, reliability, and failure modes,” 2026. [Online]. Available: https://arxiv.org/abs/2603.11287