Paper deep dive
NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation
Jiarui Ma, Jianghan Wang, Yuheng Ma, Ziyi Zhuang, Xiaoguang Liu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) are increasingly used in circuit design workflows, yet their reliability on simulator-facing SPICE netlist recognition and manipulation remains poorly understood and is rarely separated from high-level design reasoning. Although netlists are textual, they encode structured circuit objects through topology and parameters. We present \textbf{NetlistBench}, a structure-verified benchmark for SPICE netlist recognition and manipulation. NetlistBench contains 2,342 cases across 24 task families, covering parameter and connectivity recognition and edits, hierarchical operations, equivalence judgment, and long-horizon compound editing. Model outputs are evaluated by a deterministic structure-aware oracle. Across six non-thinking LLMs, performance varies substantially with operation-level structural complexity. Simple local edits reach $96\%$--$100\%$ accuracy, while device addition drops to $41\%$--$83\%$ and equivalence judgment to $49\%$--$90\%$. Enabling reasoning substantially improves weaker models but does not eliminate structure-preservation failures, with performance still degrading sharply as the edit horizon increases. NetlistBench identifies netlist reliability as a distinct bottleneck for trustworthy LLM-based circuit design automation.
Tags
Links
- Source: https://arxiv.org/abs/2608.12197v1
- Canonical: https://arxiv.org/abs/2608.12197v1
Trouble viewing inline? Open PDF directly â
Full Text
40,932 characters extracted from source content.
Expand or collapse full text
NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and ManipulationConference: ; ; CCS: Hardware Electronic design automationCCS: Computing methodologies Natural language processing Jiarui Ma Affiliation: Southern University of Science and Technology , School of Microelectronics , Shenzhen , China email: 12312626@mail.sustech.edu.cn , Jianghan Wang Affiliation: Southern University of Science and Technology , School of Microelectronics , Shenzhen , China email: 12311107@mail.sustech.edu.cn , Yuheng Ma Affiliation: Southern University of Science and Technology , School of Microelectronics , Shenzhen , China email: 12412108@mail.sustech.edu.cn , Ziyi Zhuang Affiliation: Southern University of Science and Technology , School of Microelectronics , Shenzhen , China email: 12412728@mail.sustech.edu.cn and Xiaoguang Liu Affiliation: Southern University of Science and Technology , School of Microelectronics , Shenzhen , China email: liuxg@sustech.edu.cn Abstract. Large Language Models (LLMs) are increasingly used in circuit design workflows, yet their reliability on simulator-facing SPICE netlist recognition and manipulation remains poorly understood and is rarely separated from high-level design reasoning. Although netlists are textual, they encode structured circuit objects through topology and parameters. We present NetlistBench, a structure-verified benchmark for SPICE netlist recognition and manipulation. NetlistBench contains 2,342 cases across 24 task families, covering parameter and connectivity recognition and edits, hierarchical operations, equivalence judgment, and long-horizon compound editing. Model outputs are evaluated by a deterministic structure-aware oracle. Across six non-thinking LLMs, performance varies substantially with operation-level structural complexity. Simple local edits reach 96%96\%â100%100\% accuracy, while device addition drops to 41%41\%â83%83\% and equivalence judgment to 49%49\%â90%90\%. Enabling reasoning substantially improves weaker models but does not eliminate structure-preservation failures, with performance still degrading sharply as the edit horizon increases. NetlistBench identifies netlist reliability as a distinct bottleneck for trustworthy LLM-based circuit design automation. Keywords: Large language models, circuit representation, SPICE netlists, netlist recognition and manipulation 1. Introduction Large language models (LLMs) are increasingly explored across the lifecycle of integrated circuit (IC) design, including domain-adapted chip-design assistance, analog circuit generation, simulation-driven optimization, and multimodal netlist extraction (21; 16; 15; 29; 20; 10; 31). By treating hardware artifacts as structured code, LLM-based systems hold the potential to accelerate electronic design automation (EDA) by translating specifications, editing topologies, and driving simulator- or layout-facing tools. Despite differences in their inputs, objectives, and tool interfaces, many of these workflows share SPICE netlists as a recurring representation layer (Figure 1). In generation-oriented settings, LLMs may synthesize netlists from specifications, schematics, or circuit images (15; 10; 31; 1). In simulation-driven optimization loops, they often revise existing netlists according to performance or simulator feedback (20; 26; 29). In netlist-to-schematic or netlist-to-layout workflows, they may interpret connectivity, hierarchy, and device relationships to guide downstream visualization or physical design (17; 5; 13; 14; 8). Across these settings, key model actions frequently take the form of reading, generating, editing, or reasoning over netlist text. Errors at this representation layer can directly corrupt downstream simulation, optimization, or layout reasoning. Consequently, failures in LLM-based circuit workflows may originate either from high-level design reasoning or from low-level netlist corruption, yet existing evaluations rarely separate these two sources. Reliable netlist operation is therefore a prerequisite for trustworthy LLM-based circuit design workflows. However, this prerequisite has not been directly quantified. General LLM-for-code benchmarks commonly evaluate executable functional correctness through unit tests or repository test suites (3; 12), but do not capture the device-specific terminal semantics, shared-node connectivity, and ordered subcircuit interfaces of SPICE netlists. Existing evaluations of LLM-based circuit design typically focus on end-to-end outcomes, such as syntactic validity, simulation success, specification improvement, or downstream task completion (15; 26; 20). Recent circuit-oriented benchmarks mainly assess domain-level capabilities, including circuit interpretation, topology reasoning, schematic understanding, AMS-domain multimodal reasoning, or graph-structured reasoning (25; 23; 27). While these evaluations reveal important limitations, they do not isolate the elementary operations required to interpret and modify SPICE netlists correctly. Accordingly, the reliability of LLMs in performing core SPICE netlist operations remains unclear. Figure 1. SPICE netlists as a common representation layer in LLM-based circuit design workflows.Diagram showing SPICE netlists as an intermediate representation connecting LLM-based circuit generation, editing, optimization, simulation, schematic processing, and physical-design workflows. To address this gap, we introduce NetlistBench, a structure-verified benchmark for evaluating whether LLMs can reliably recognize and manipulate analog SPICE netlists as structured circuit representations. NetlistBench focuses on structure-level operations that test a modelâs ability to recover circuit structure or apply explicit modifications without introducing unintended changes. Model outputs are evaluated through a deterministic canonical circuit representation that captures devices, ordered terminal bindings, node identities, parameters, directives, and hierarchy. The main contributions of this work are: ⢠We formulate SPICE netlist reliability as a representation-level evaluation problem, focusing on whether LLMs can correctly recognize and manipulate netlists as structured circuit artifacts. ⢠We introduce NetlistBench, a structure-verified benchmark covering netlist structural-property recognition and natural-language-guided netlist manipulation. Using NetlistBench, we evaluate representative frontier, flash-class, and open-weight LLMs and show that reliability varies sharply across operation type and task horizon. ⢠We develop a structure-aware evaluation pipeline based on canonical circuit representations, enabling manipulation outputs to be verified beyond raw text matching or final simulation outcomes. 2. Background 2.1. SPICE Netlists as Structured Circuit Representations SPICE netlists are simulator-facing circuit descriptions that encode devices, terminals, nodes, parameters, models, ports, and subcircuit hierarchies in a compact, positional textual format (18; 19). Although a netlist appears as a sequence of text lines, its underlying semantics correspond to a structured circuit object. Each device statement typically begins with an instance prefix, followed by an ordered sequence of node names connected to specific device terminals, a model reference, and optional parameter assignments. For hierarchical circuits, subcircuit definitions (.subckt) establish ordered port interfaces, and each subcircuit instance binds external nodes to internal ports strictly according to their positional order in the instance statement. A defining characteristic of this representation is that electrical connectivity is encoded implicitly through node-name sharing rather than explicit terminal-to-terminal links. For circuit simulation, node labels are sufficient because devices contribute equations to the modified nodal analysis (MNA) system according to the nodes attached to their terminals (9). Terminals sharing the same node name are treated as electrically connected. For structural analysis and manipulation, however, these connections are not represented as explicit terminal-to-terminal links; the circuit topology must be reconstructed from terminalânode bindings across the entire netlist. These properties make netlist operations different from ordinary text editing. A correct edit must preserve terminal-role bindings, maintain consistent node identities, and avoid unintended changes to unrelated devices or subcircuit interfaces. NetlistBench therefore evaluates netlist outputs through structural equivalence to a canonical circuit representation rather than through surface-string similarity. 2.2. Reported Limitations of LLMs on SPICE Netlists Existing studies have reported several limitations of LLMs in processing circuit representations and SPICE-like netlists. At the circuit-reasoning level, benchmarks such as CIRCUIT and AMSbench show that LLMs can struggle with topology-heavy circuit interpretation and multi-step circuit reasoning (25; 23). At the generation and adaptation level, systems such as SPICEPilot, SPICEAssistant, AnalogCoder, and Spice Wizard rely on simulation feedback, syntax checks, or tool-assisted repair loops to improve SPICE code or netlist generation, indicating that unvalidated LLM outputs may not provide sufficient reliability for direct downstream use (26; 20; 15; 6). At the netlist-analysis level, SPICED studies LLM-aided detection and localization of syntactical bugs and analog Trojans in A/MS netlists, further showing that node, parameter, subcircuit, and connectivity errors are meaningful failure classes in SPICE-like representations (2). Complementary representation-oriented work suggests why these failures are difficult to avoid with ordinary text modeling alone. CircuitFormer highlights the mismatch between standard language tokenization and the graph-structured semantics of circuits, while Image2Net evaluates diagram-to-netlist conversion using graph-structured netlist comparison rather than raw string matching (11; 31). Taken together, these findings suggest that LLM failures on SPICE netlists are not merely surface-level syntax issues, but are closely related to the difficulty of preserving structured circuit semantics in a positional textual representation. However, these studies evaluate netlists within broader generation, simulation, conversion, or reasoning pipelines rather than isolating representation-level netlist operations. 3. Benchmark Design NetlistBench evaluates representation-level netlist reliability through two modalities: recognition, which extracts or compares circuit structure, and manipulation, which applies explicit natural-language edits to SPICE netlists. This design separates structure interpretation and structure-preserving transformation from high-level design reasoning, simulator behavior, and optimization. Table 1. Composition of the NetlistBench source corpus. Source subset Count Structural summary AnalogGenieâSimple 492 Median: 5 devices AnalogGenieâMedium 1,752 Median: 20 devices AnalogGenieâComplex 594 Median: 36 devices; maximum: 69 Flat subtotal 2,838 Approximately 58,000 device instances ALIGN hierarchical 931 21 topology families; 3â4 subcircuits per netlist Total 3,769 3.1. Source Corpus NetlistBench uses two complementary SPICE netlist sources, summarized in Table 1. AnalogGenie provides flat CMOS analog netlists originally developed for topology discovery (7); after normalization, these circuits are used to construct flat recognition and manipulation tasks. ALIGN provides hierarchical analog netlists from a layout automation flow (5); its circuits contain multiple .subckt definitions and top-level instance calls and are used for hierarchical tasks. 3.2. Instance Construction Pipeline NetlistBench constructs evaluation instances through a deterministic, template-driven pipeline rather than stochastic or unconstrained generation. Starting from source SPICE netlists, the pipeline applies family-specific transformation rules, syntax perturbations, and prompt templates to produce reproducible benchmark cases. Each instance is represented as a self-contained triplet: (1) âi=(src(i),inst(i),target(i)),I_i= (N^(i)_src,T^(i)_inst,Y^(i)_target ), where src(i)N^(i)_src denotes the source SPICE netlist, inst(i)T^(i)_inst denotes the explicit task instruction, and target(i)Y^(i)_target denotes the task-specific target used for evaluation. Both inst(i)T^(i)_inst and target(i)Y^(i)_target are produced by deterministic, family-specific templates, ensuring that each instance has an unambiguous instruction and reproducible ground truth. The form of target(i)Y^(i)_target depends on the task type. For manipulation tasks, it is the uniquely determined target SPICE netlist that realizes the requested structural transformation. For recognition tasks, it is the canonical JSON answer derived from the source netlist. For equivalence judgment tasks, it is the binary structural-equivalence verdict. All instances undergo automated construction-time validation before inclusion. The checker ensures that manipulation targets implement exactly the specified structural changes without unintended edits, and that recognition and equivalence targets are consistent with the canonical IR of the corresponding input netlist or netlist pair. Figure 2. NetlistBench pipeline for generating benchmark cases and evaluating model outputs with a structure-aware oracle.Pipeline diagram showing source netlists, deterministic benchmark-instance construction, model inference, canonical circuit representation, and structure-aware evaluation. Figure 3. Overview of the NetlistBench benchmark, showing the distribution of cases across task families and representative task examples.Overview of the recognition and manipulation task families in NetlistBench, including their case distribution and representative netlist tasks. 3.3. Structure-Aware Evaluation Oracle This structure-aware evaluation follows the broader principle that circuit artifacts should be compared through their underlying connectivity and device structure rather than by surface text. Similar graph-structured evaluation ideas have been used in diagram-to-netlist conversion, where generated and reference netlists are compared through heterogeneous circuit graphs rather than raw strings (31). Concretely, each model output and the reference target are parsed into the canonical IR âa normalized structure that lists every device by instance name with its device kind, ordered terminal nodes, and parameters, together with top-level directives and, for hierarchical circuits, each subcircuitâs port interface and internal devices. An output passes only if its IR matches the reference IR under a fixed set of semantics-preserving normalizations: the two must contain the same set of named devices, with no missing or extra device, identical terminal-node bindings, parameters equal up to numeric normalization (e.g., 1k equals 1000), and identical top-level directives; symmetric two-terminal passives (R/C/L) are compared with unordered terminals, and subcircuit definitions must agree on their port interface (port order treated as semantic except for extraction tasks), internal devices, and directives. This exact-match-up-to-normalization rule directly encodes the name-preservation and locality constraints of edit tasks: renaming an untouched node, dropping or duplicating a device, or perturbing an unrelated parameter each surfaces as an IR mismatch and fails the case. The same canonical-IR comparator scores SPICE and PySpice (22) outputs through a uniform interface. For the equivalence-judgment family, the task is instead to decide whether two netlists denote the same circuit up to consistent node and instance renaming. To validate the generated ground-truth labels for this family, we projected the IR into a labeled bipartite deviceânet graphâdevice nodes labeled by type, model, and normalized parameter signatures, and deviceânet edges labeled by terminal rolesâand ran a VF2 graph-isomorphism check (4) confirming that positive pairs are isomorphic and negative pairs are not. This isomorphism check audits equivalence-case labels only and is not part of scoring model outputs: scoring must instead preserve device and node names, whereas VF2 equates circuits up to renaming and would therefore mask the very name- and locality-violations that the edit tasks are designed to detect. 3.4. Task Families NetlistBench contains 24 task families across recognition and manipulation, as summarized in Figure 3. The recognition modality contains 800 cases across eight families. Seven evaluate structured extraction of device parameters, terminal connectivity, node incidence, subcircuit interfaces, and instance mappings, while the eighth evaluates structural equivalence between netlist pairs. The manipulation modality contains 1,542 cases across 16 families. Six single-edit families cover connectivity editing, device addition, removal and replacement, parameter editing, and rename propagation. Five flat compound families combine 3, 6, 9, 12, or 15 dependent operations, and five hierarchical families evaluate subcircuit expansion, interface modification, and multi-step internal editing. 4. Evaluation 4.1. Experimental Setup and Protocol We evaluate six single-shot non-thinking models spanning frontier, flash-class, and open-weight tiers: Claude Sonnet 4.6, GPT-4.1, Gemini 2.5 Flash, DeepSeek-V4-Flash, Qwen3.6-Flash, and Qwen3-30B-A3B. All are queried through official provider APIs with explicit reasoning modes disabled, so the main comparison measures base netlist-operation reliability rather than reasoning elicitation. As a reasoning reference, we additionally evaluate the same DeepSeek-V4-Flash with its native thinking mode enabled; this column is reported separately and excluded from the non-thinking comparison. We also run two controlled secondary analyses on a paired stratified subset: SPICE versus PySpice output representation, and direct prompting versus native thinking and CoT prompting (28). Each model is queried once per case with deterministic decoding, and retries are used only for transport failures. Responses are graded by the structure-aware oracle in Section 3.3 and reduced to binary pass/fail outcomes: manipulation outputs must match the reference structure, recognition outputs must match the canonical JSON answer, and equivalence judgments must match the reference verdict. The parser tolerates incidental code fences, but empty, unparsable, or structurally invalid outputs fail. We report pass rates with Wilson 95% confidence intervals (30); Table 2 gives the per-family case count n, and aggregate ablation intervals are stated explicitly. Subtotals and overall scores are case-weighted. Reasoning-mode results are single samples and may carry run-to-run variance. Availability. NetlistBench is publicly available at https://github.com/WoshiMayou/NetlistBench. The repository contains all 2,342 benchmark cases across the 24 task families, the deterministic structure-aware oracle, seeded case-generation scripts, evaluation runners, and the per-family prompt templates required to reproduce the benchmark evaluation. The code is released under the Apache-2.0 license, while the benchmark cases and prompts are released under C BY 4.0. Table 2. Per-family NetlistBench pass rates (%). C/G/Ge/QF/Q30/DS denote Claude-S4.6, GPT-4.1, Gemini-2.5-F, Qwen3.6-F, Qwen3-30B, and DeepSeek-V4-F. DS+Râ is the reasoning reference, excluded from bolding; underlines exceed all non-thinking models. Task n C G Ge QF Q30 DS DS+Râ Conn. edit 100 97 91 93 59 47 85 99 Dev. add 100 83 72 57 54 43 41 55 Dev. remove 100 100 100 100 98 97 100 100 Dev. replace 100 95 93 86 75 61 78 91 Param. edit 100 100 99 98 99 96 98 99 Rename prop. 100 99 98 97 85 89 92 100 Comp. 3 100 80 71 70 34 28 44 74 Comp. 6 100 57 58 33 6 1 18 63 Comp. 9 100 56 51 21 2 1 6 50 Comp. 12 100 41 39 17 0 0 1 33 Comp. 15 97 26 34 6 0 0 0 31 Subckt inline 45 56 62 33 16 2 42 98 Port swap 100 88 80 83 33 19 65 97 Subckt comp. 3 100 79 69 55 39 21 32 66 Subckt comp. 6 100 67 55 36 5 4 8 54 Subckt comp. 9 100 42 43 41 1 1 3 47 Edit subtotal 1542 74 70 59 39 33 45 71 Dev. param. 100 99 100 100 100 99 100 100 Sem. term. conn. 100 99 73 74 25 6 82 100 Ord. term. conn. 100 100 100 99 99 100 100 100 Node inc. 100 98 42 59 21 15 53 96 Subckt ports 100 100 100 100 100 99 100 100 Inst. port map 100 100 96 87 89 65 96 100 Term. neigh. inc. 100 97 13 20 4 2 12 93 Equiv. judge 100 90 67 61 55 56 49 97 Recog. subtotal 800 98 74 75 62 55 74 98 Overall 2342 82 71 64 47 41 55 81 4.2. Performance Across Task Families Table 2 shows substantial variation across operation types. Local operations that primarily modify explicit text are the most reliable: device removal and parameter editing reach 96%96\%â100%100\% across models. Reliability decreases for operations that require maintaining connectivity or introducing new structure, including connectivity editing, device replacement, device addition, subcircuit port swapping, and inline expansion. Recognition exhibits a similar distinction. Explicit attributes such as device parameters, ordered terminal lists, and subcircuit ports are extracted with high accuracy, whereas relational queries vary substantially across models. Node incidence ranges from 15%15\% to 98%98\%, semantic terminal connectivity from 6%6\% to 99%99\%, and terminal-neighbor incidence from 2%2\% to 97%97\%. Structural equivalence judgment also remains challenging, with pass rates from 49%49\% to 90%90\%. Overall non-thinking pass rates range from 41%41\% to 82%82\%. The results indicate that current models are considerably more reliable on localized attribute extraction and substitution than on recovering or preserving the implicit connectivity graph. Enabling reasoning raises DeepSeek-V4-Flash from 55%55\% to 81%81\%, but does not consistently surpass the strongest non-thinking model. Figure 4. Pass rates under compound editing with increasing numbers of dependent edits for (a) flat netlists and (b) hierarchical subcircuit netlists. Lines show the mean pass rates, and shaded regions indicate the corresponding variation.Two compound-editing plots showing pass rate versus the number of dependent edits for flat and hierarchical netlists. All model curves decrease as the edit horizon grows. 4.3. Long-Horizon Compound Editing The compound editing families chain 33, 66, 99, 1212, and 1515 mutually dependent edits into a single instruction, revealing a substantial reliability degradation in NetlistBench. Accuracy declines consistently as the edit horizon increases (Figure 4). Even models with strong short-horizon performance degrade substantially: Claude drops from 80%80\% at 3 steps to 26%26\% at 15 steps, while GPT-4.1 drops from 71%71\% to 34%34\%. The remaining models decline to near-zero accuracy at longer horizons, with Gemini decreasing from 70%70\% to 6%6\%, DeepSeek-V4-Flash from 44%44\% to 0%0\%, and both Qwen3.6-Flash and the open-weight Qwen3-30B from 34%34\%/28%28\% to 0%0\%. Crucially, reasoning does not eliminate this trend: DeepSeek-V4-Flash with reasoning enabled, although far stronger at short horizons (74%74\% at 3 steps), still falls to 31%31\% at 15 steps. The degradation is not simply a consequence of weak atomic editing. Long-horizon compound tasks require models to track multiple dependent edit intents, update intermediate circuit state, and preserve edit locality across an extended instruction sequence. Because the edits are mutually dependent, errors compound across the sequence: even a high per-edit success rate yields a low joint success probability once many edits must all be correct. The same downward trend appears in the hierarchical compound family, indicating that this effect is not limited to flat netlists. These results show that high reliability on isolated edits does not translate into reliable multi-step netlist transformation. This pattern is consistent with broader observations that small per-step error rates can compound sharply over long execution horizons (24). 4.4. Representation and Reasoning We study two mitigations on DeepSeek-V4-Flash and Qwen3.6-Flash: changing the circuit representation (SPICEâ ) and enabling explicit reasoning (native thinking and CoT prompting (28)). Both analyses use the same paired, stratified subset (3030 cases per family, 28 for Comp. 15, n=718n=718); Table 3 reports overall pass rates with Wilson 95% intervals, and paired arms are compared with McNemarâs test. On this subset, reasoning gives the larger aggregate gains: native thinking raises both models by roughly 3030â4040 points (p<10â6p<10^-6), while CoT also improves performance but less strongly. The task-level results in Table 5 and Figure 6 show similar gains across several structure-heavy families. By contrast, PySpice has a smaller and less consistent effect: it improves DeepSeek-V4-Flash overall (p<0.001p<0.001), but not Qwen3.6-Flash (p=0.51p=0.51), with task-level trends shown in Table 4 and Figure 5. Because each per-family cell has only 3030 cases, we treat task-level patterns as descriptive. Overall, reasoning is the stronger mitigation here, but errors still concentrate on long-horizon compound edits, hierarchical operations, and relational structural queries; neither mitigation makes current LLMs sufficiently reliable for unverified netlist editing. Table 3. Mitigation results on the paired subset (n=718n=718), with Wilson 95% confidence intervals. Model Direct Native CoT PySpice DeepSeek-V4-F 5252 8181 7575 5858 [48,55][48,55] [78,83][78,83] [72,78][72,78] [55,62][55,62] Qwen3.6-F 4545 8585 7878 4545 [41,48][41,48] [82,88][82,88] [75,81][75,81] [42,49][42,49] 5. Discussion 5.1. Implications for LLM-Based Netlist Editing NetlistBench shows that netlist reliability cannot be reduced to general circuit knowledge or output-format compliance. Models tend to perform better on localized edits, such as parameter changes and device removal, while showing reduced reliability on tasks involving structural attachment, ordered port handling, equivalence judgment, or multi-step edits. The observed failures are frequently structural rather than purely procedural: recognition outputs usually follow the required JSON schema but contain incorrect circuit facts, while manipulation failures involve omitted edits, duplicated edits, loss of locality, unintended terminal rebinding, and topology drift. This error pattern follows directly from the representation properties described in Section 2.1. Local substitutions and deletions often require only limited changes to already explicit text, while attachment, hierarchy, equivalence, and compound editing require the model to maintain an implicit circuit graph across terminal roles, node identities, and subcircuit interfaces. The observed failures therefore indicate a structure-preservation bottleneck: models can often produce syntactically plausible netlists, but still lose edit locality, perturb unrelated bindings, or fail to maintain consistent topology across multiple dependent operations. These results suggest that current LLMs should not be treated as standalone, unverified netlist editors. Reasoning modes and alternative surface representations can mitigate some failures, but neither fully resolves the structure-preservation problem. More robust workflows may need to decompose complex edits, verify each intermediate netlist structurally, and provide feedback when unintended changes are detected. Future work may also explore graph-based or other structured circuit representations that expose connectivity more directly than raw SPICE text. 5.2. Limitations NetlistBench evaluates bounded, circuit-block-level netlists rather than industrial-scale post-layout decks. Although the compound tasks increase the number of dependent edits, all evaluated netlists fit within the tested modelsâ context windows. The benchmark therefore does not assess long-context retrieval, hierarchical partitioning, or direct processing of extracted netlists containing millions of device and parasitic statements. The current release also covers a restricted circuit and syntax domain, primarily flat and hierarchical analog CMOS blocks from AnalogGenie and ALIGN. It does not comprehensively cover symbolic .param expressions, complex .model cards, behavioral or controlled sources, include hierarchies, extracted parasitics, or simulator- and PDK-specific syntax. The reported results therefore should not be assumed to transfer unchanged to RF, power, digital, or mixed-signal netlists. Finally, task instructions are generated from deterministic templates to isolate structural capabilities and enable unambiguous grading. They do not capture the full linguistic variability or design intent of real designerâassistant interactions. Each modelâcase pair is evaluated once, so the results characterize the tested API snapshots rather than complete output distributions. 6. Conclusion NetlistBench shows that netlist reliability is a distinct bottleneck for LLM-based circuit design. Current models often handle local substitutions and simple extraction, but remain fragile on connectivity-sensitive edits, hierarchy manipulation, structural equivalence, and long-horizon compound transformations. Reasoning improves performance, yet does not make LLMs reliable unverified editors of simulator-facing netlists. These findings motivate decomposed editing workflows, structure-aware verification after each edit, and circuit representations that expose topology more directly than raw SPICE text. References Bhandari et al. (2024) J. Bhandari, V. Bhat, Y. He, S. Garg, H. Rahmani, and R. Karri Masala-CHAI: a large-scale SPICE netlist dataset for analog circuits by harnessing AI. External Links: 2411.14299 Cited by: §1. Chaudhuri et al. (2024) J. Chaudhuri, D. Thapar, A. Chaudhuri, F. Firouzi, and K. Chakrabarty SPICED: syntactical bug and trojan pattern identification in A/MS circuits using LLM-enhanced detection. External Links: 2408.16018 Cited by: §2.2. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §1. Cordella et al. (2004) L. P. Cordella, P. Foggia, C. Sansone, and M. Vento A (sub)graph isomorphism algorithm for matching large graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence 26 (10), p. 1367â1372. External Links: Document Cited by: §3.3. Dhar et al. (2021) T. Dhar, K. Kunal, Y. Li, M. Madhusudan, J. Poojary, A. K. Sharma, W. Xu, S. M. Burns, R. Harjani, J. Hu, D. A. Kirkpatrick, P. Mukherjee, S. Yaldiz, and S. S. Sapatnekar ALIGN: a system for automating analog layout. IEEE Design & Test 38 (2), p. 8â18. External Links: Document Cited by: §1, §3.1. Divakar et al. (2026) A. Divakar, A. Anekar, and M. Kulkarni Spice wizard: a unified AI agent for netlist generation. TechRxiv. Note: Preprint External Links: Document Cited by: §2.2. Gao et al. (2025) J. Gao, W. Cao, J. Yang, and X. Zhang AnalogGenie: a generative engine for automatic discovery of analog circuit topologies. In The Thirteenth International Conference on Learning Representations, Cited by: §3.1. Graeb and Leibl (2023) H. Graeb and M. Leibl Learning from the implicit functional hierarchy in an analog netlist. In Proceedings of the 2023 ACM International Symposium on Physical Design (ISPD), p. 93â100. External Links: Document Cited by: §1. Ho et al. (1975) C. Ho, A. E. Ruehli, and P. A. Brennan The modified nodal approach to network analysis. IEEE Transactions on Circuits and Systems 22 (6), p. 504â509. External Links: Document Cited by: §2.1. Huang et al. (2025) C. Huang, H. Chen, H. Ho, P. Kang, M. P. Lin, W. Liu, and H. Ren Netlistify: transforming circuit schematics into netlists with deep learning. In Proceedings of the 2025 ACM/IEEE 7th Symposium on Machine Learning for CAD (MLCAD), p. 1â8. External Links: Document Cited by: §1, §1. Islam et al. (2026) M. T. Islam, S. K. Saha, F. Farahmandi, and M. Tehranipoor CircuitFormer: a circuit language model for analog topology design from natural language prompt. External Links: 2605.05773 Cited by: §2.2. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan SWE-bench: can language models resolve real-world GitHub issues?. In International Conference on Learning Representations, Cited by: §1. Kunal et al. (2020) K. Kunal, T. Dhar, M. Madhusudan, J. Poojary, A. K. Sharma, W. Xu, S. M. Burns, J. Hu, R. Harjani, and S. S. Sapatnekar GANA: graph convolutional network based automated netlist annotation for analog circuits. In Proceedings of the 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE), p. 55â60. External Links: Document Cited by: §1. Kunal et al. (2023) K. Kunal, T. Dhar, M. Madhusudan, J. Poojary, A. K. Sharma, W. Xu, S. M. Burns, J. Hu, R. Harjani, and S. S. Sapatnekar GNN-based hierarchical annotation for analog circuits. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 42 (9), p. 2801â2814. External Links: Document Cited by: §1. Lai et al. (2025) Y. Lai, S. Lee, G. Chen, S. Poddar, M. Hu, D. Z. Pan, and P. Luo AnalogCoder: analog circuit design via training-free code generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 379â387. External Links: Document Cited by: §1, §1, §1, §2.2. Liu et al. (2023) M. Liu, T. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Alben, H. Anand, S. Banerjee, I. Bayraktaroglu, et al. ChipNeMo: domain-adapted LLMs for chip design. External Links: 2311.00176 Cited by: §1. Matsuo et al. (2024) R. Matsuo, S. Uhlich, A. Venkitaraman, A. Bonetti, C. Hsieh, A. Momeni, L. Mauch, A. Capone, E. Ohbuchi, and L. Servadei Schemato: an LLM for netlist-to-schematic conversion. External Links: 2411.13899 Cited by: §1. Nagel and Pederson (1973) L. W. Nagel and D. O. Pederson SPICE (simulation program with integrated circuit emphasis). Technical report Technical Report UCB/ERL M382, Electronics Research Laboratory, University of California, Berkeley. External Links: Link Cited by: §2.1. Nagel (1975) L. W. Nagel SPICE2: a computer program to simulate semiconductor circuits. Technical report Technical Report UCB/ERL M520, Electronics Research Laboratory, University of California, Berkeley. External Links: Link Cited by: §2.1. Nau et al. (2025) S. Nau, J. Krummenauer, and A. Zimmermann Evaluating LLM-based workflows for switched-mode power supply design. External Links: 2507.10639 Cited by: §1, §1, §1, §2.2. Pan et al. (2025) J. Pan, G. Zhou, C. Chang, I. Jacobson, J. Hu, and Y. Chen A survey of research in large language models for electronic design automation. ACM Transactions on Design Automation of Electronic Systems 30 (3). External Links: Document Cited by: §1. Salvaire (2021) F. Salvaire PySpice: simulate electronic circuit using python and the Ngspice/Xyce simulators. Note: SoftwareAccessed 2026-07-28 External Links: Link Cited by: §3.3. Shi et al. (2025) Y. Shi, Z. Zhang, H. Wang, Z. Tao, Z. Li, B. Chen, Y. Wang, Z. Yu, T. Lin, and L. He AMSbench: a comprehensive benchmark for evaluating MLLM capabilities in AMS circuits. External Links: 2505.24138 Cited by: §1, §2.2. Sinha et al. (2025) A. Sinha, A. Arun, S. Goel, S. Staab, and J. Geiping The illusion of diminishing returns: measuring long horizon execution in LLMs. External Links: 2509.09677 Cited by: §4.3. Skelic et al. (2025) L. Skelic, Y. Xu, M. Cox, W. Lu, T. Yu, and R. Han CIRCUIT: a benchmark for circuit interpretation and reasoning capabilities of LLMs. External Links: 2502.07980 Cited by: §1, §2.2. Vungarala et al. (2024) D. Vungarala, S. Alam, A. Ghosh, and S. Angizi SPICEPilot: navigating SPICE code generation and simulation with AI guidance. External Links: 2410.20553 Cited by: §1, §1, §2.2. Wang et al. (2023) H. Wang, S. Feng, T. He, Z. Tan, X. Han, and Y. Tsvetkov Can language models solve graph problems in natural language?. In Advances in Neural Information Processing Systems, Vol. 36, p. 30840â30861. Cited by: §1. Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Vol. 35, p. 24824â24837. Cited by: §4.1, §4.4. Wei et al. (2025) Z. Wei, Z. Kong, Y. Wang, D. Z. Pan, and X. Tang TopoSizing: an LLM-aided framework of topology-based understanding and sizing for AMS circuits. External Links: 2509.14169 Cited by: §1, §1. Wilson (1927) E. B. Wilson Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22 (158), p. 209â212. External Links: Document Cited by: §4.1. Xu et al. (2025) H. Xu, C. Liu, Q. Wang, W. Huang, Y. Xu, W. Chen, A. Peng, Z. Li, B. Li, L. Qi, J. Yang, Y. Du, and L. Du Image2Net: datasets, benchmark and hybrid framework to convert analog circuit diagrams into netlists. External Links: 2508.13157 Cited by: §1, §1, §2.2, §3.3. Appendix A Additional Results Table 4. Task-level pass rates for SPICE and PySpice. Task DS-S DS-Py Qwen-S Qwen-Py Connectivity edit 0.80 1.00 0.60 0.57 Device add 0.50 0.80 0.50 0.50 Device remove 1.00 1.00 1.00 0.97 Device replace 0.70 0.80 0.80 0.73 Parameter edit 0.93 0.97 0.97 0.93 Rename propagation 0.90 0.93 0.93 0.77 Compound 3-step 0.13 0.50 0.20 0.23 Compound 6-step 0.13 0.23 0.07 0.20 Compound 9-step 0.03 0.23 0.07 0.03 Compound 12-step 0.00 0.10 0.00 0.03 Compound 15-step 0.00 0.07 0.00 0.00 Subckt inline-expand 0.37 0.47 0.13 0.37 Subckt port-swap 0.60 0.73 0.27 0.33 Subckt compound 3-step 0.33 0.10 0.37 0.10 Subckt compound 6-step 0.07 0.03 0.00 0.03 Subckt compound 9-step 0.03 0.00 0.03 0.00 Edit subtotal 0.41 0.50 0.37 0.36 Equivalence judgment 0.50 0.53 0.50 0.60 Device parameter 1.00 1.00 1.00 0.90 Semantic terminal conn. 0.80 0.83 0.23 0.43 Ordered terminal conn. 1.00 1.00 1.00 0.97 Node incidence 0.53 0.60 0.23 0.23 Subckt port list 1.00 1.00 1.00 1.00 Instance port map 0.90 0.90 0.80 0.93 Terminal neighbor inc. 0.07 0.13 0.03 0.00 Recognition subtotal 0.76 0.78 0.61 0.64 Overall 0.52 0.58 0.45 0.45 DS-S: DeepSeek-V4 SPICE; DS-Py: DeepSeek-V4 PySpice; Qwen-S: Qwen-3.6 SPICE; Qwen-Py: Qwen-3.6 PySpice. Table 5. Task-level pass rates under different prompting modes. Task DeepSeek-V4 Qwen-3.6 Dir. Think CoT Dir. Think CoT Connectivity edit 0.80 1.00 1.00 0.60 0.93 0.87 Device add 0.50 0.47 0.57 0.50 0.83 0.67 Device remove 1.00 1.00 1.00 1.00 1.00 1.00 Device replace 0.70 0.90 0.83 0.80 0.93 0.93 Parameter edit 0.93 0.97 1.00 0.97 0.97 0.97 Rename propagation 0.90 1.00 1.00 0.93 0.97 1.00 Compound 3-step 0.13 0.57 0.50 0.20 0.73 0.73 Compound 6-step 0.13 0.50 0.53 0.07 0.67 0.63 Compound 9-step 0.03 0.47 0.57 0.07 0.73 0.70 Compound 12-step 0.00 0.37 0.27 0.00 0.63 0.33 Compound 15-step 0.00 0.18 0.14 0.00 0.54 0.32 Subckt inline-expand 0.37 0.97 1.00 0.13 0.83 0.77 Subckt port-swap 0.60 1.00 1.00 0.27 0.93 0.93 Subckt compound 3-step 0.33 0.73 0.63 0.37 0.90 0.63 Subckt compound 6-step 0.07 0.60 0.53 0.00 0.77 0.70 Subckt compound 9-step 0.03 0.63 0.40 0.03 0.63 0.40 Edit subtotal 0.41 0.71 0.69 0.37 0.81 0.73 Equivalence judgment 0.50 0.97 0.93 0.50 0.73 0.70 Device parameter 1.00 1.00 1.00 1.00 1.00 1.00 Semantic terminal conn. 0.80 1.00 0.93 0.23 0.90 0.87 Ordered terminal conn. 1.00 1.00 1.00 1.00 1.00 1.00 Node incidence 0.53 1.00 0.73 0.23 0.93 0.83 Subckt port list 1.00 1.00 1.00 1.00 1.00 0.97 Instance port map 0.90 1.00 0.97 0.80 1.00 1.00 Terminal neighbor inc. 0.07 1.00 0.47 0.03 0.83 0.73 Recognition subtotal 0.76 1.00 0.87 0.61 0.95 0.91 Overall 0.52 0.81 0.75 0.45 0.85 0.78 Dir.: direct prompting; Think: think-mode prompting; CoT: chain-of-thought prompting. (a) DeepSeek (b) Qwen (c) DeepSeek, Compound (d) Qwen, Compound Figure 5. Task-level performance under SPICE and PySpice representations.Radar plots compare task-level pass rates for SPICE and PySpice representations for DeepSeek and Qwen. The lower plots show pass rates as compound-edit sequence length increases. (a) DeepSeek (b) Qwen (c) DeepSeek, Compound (d) Qwen, Compound Figure 6. Task-level performance under non-thinking, thinking, and CoT prompting.Radar plots compare task-level pass rates under non-thinking, thinking, and chain-of-thought prompting for DeepSeek and Qwen. The lower plots show performance as compound-edit sequence length increases.