Paper deep dive
GPlaceRL: An Open-Source Graph Reinforcement Learning Framework for Detailed Placement
Pavlos Stoikos, Foteini Oikonomou, Christos Poulos, Maria Pantazi-Kypriou, Athanasios Tziouvaras, Christos Anagnostopoulos, Georgios Karakonstantis, George Floros
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) has emerged as a promising approach for placement optimization, particularly when combined with graph neural networks (GNNs) that capture circuit connectivity. However, most learning-based placement approaches focus on floorplanning, macro placement, or global placement, while detailed placement refinement remains relatively unexplored. In this paper, we present GPlaceRL, an open-source graph reinforcement learning framework for detailed placement refinement. GPlaceRL represents legalized placements as graphs and provides a modular environment for studying graph encoders, policy architectures, reward formulations, and local placement actions. To demonstrate the capabilities of GPlaceRL, we conduct a systematic evaluation of proximal policy optimization (PPO) policies with graph attention network (GAT) encoders in a per-design optimization setting. Across five placement benchmarks, the best greedy evaluation results achieve HPWL improvements ranging from $3.27\%$ to $32.87\%$. The results highlight the importance of compact GAT architectures and flexible local action spaces for placement optimization. Overall, GPlaceRL provides a reproducible and extensible framework for systematic research on RL-based detailed placement refinement.
Tags
Links
- Source: https://arxiv.org/abs/2610.06489v1
- Canonical: https://arxiv.org/abs/2610.06489v1
Trouble viewing inline? Open PDF directly →
Full Text
21,259 characters extracted from source content.
Expand or collapse full text
GPlaceRL: An Open-Source Graph Reinforcement Learning Framework for Detailed Placement Pavlos Stoikos†, Foteini Oikonomou†, Christos Poulos†, Maria Pantazi-Kypraiou†, Athanasios Tziouvaras†, Christos Anagnostopoulos*, Georgios Karakonstantis† and George Floros‡ Affiliation: Affiliation: †Department of Electrical and Computer Engineering, University of Thessaly, Volos, Greece pastoikos, fotoikon, cpoulos, mpantazi-, attziouv, gkarakon@e-ce.uth.gr *School of Computing Science, University of Glasgow, UK, christos.anagnostopoulos@glasgow.ac.uk ‡Department of Electronic and Electrical Engineering, Trinity College Dublin, Ireland, florosg@tcd.ie Abstract Reinforcement learning (RL) has emerged as a promising approach for placement optimization, particularly when combined with graph neural networks (GNNs) that capture circuit connectivity. However, most learning-based placement approaches focus on floorplanning, macro placement, or global placement, while detailed placement refinement remains relatively unexplored. In this paper, we present GPlaceRL, an open-source graph reinforcement learning framework for detailed placement refinement. GPlaceRL represents legalized placements as graphs and provides a modular environment for studying graph encoders, policy architectures, reward formulations, and local placement actions. To demonstrate the capabilities of GPlaceRL, we conduct a systematic evaluation of proximal policy optimization (PPO) policies with graph attention network (GAT) encoders in a per-design optimization setting. Across five placement benchmarks, the best greedy evaluation results achieve HPWL improvements ranging from 3.27%3.27\% to 32.87%32.87\%. The results highlight the importance of compact GAT architectures and flexible local action spaces for placement optimization. Overall, GPlaceRL provides a reproducible and extensible framework for systematic research on RL-based detailed placement refinement. Index Terms: Open-Source Frameworks, Reinforcement Learning, Graph Neural Networks, Detailed Placement I Introduction Placement is a central stage of the physical design flow, directly affecting wirelength, timing, routability, and power. After global placement and legalization, detailed placement locally refines a legal placement through operations such as cell shifting, swapping, and reordering. Although these local changes may appear small, they can significantly affect half-perimeter wirelength (HPWL) and downstream routing quality [1]. †footnotetext: *This work was funded under the COIN-3D project, which has received funding from the European Union’s Horizon Europe research and innovation program under grant agreement No. 101159667. Traditional detailed placement algorithms rely on carefully engineered local-search and optimization techniques, including dynamic programming, network flow, and min-cost matching formulations [2]. These methods are effective, but they often require evaluating many candidate local moves and may become costly as modern netlists grow in size and connectivity complexity [3]. Recent learning-based placement methods have shown that graph neural networks (GNNs) and reinforcement learning (RL) can capture structural information from netlists and optimize placement-related objectives [4, 5, 6, 10137100]. However, most RL-based works focus on floorplanning, macro placement, or global placement. In contrast, detailed placement refinement remains less explored, and open, reproducible frameworks for systematically studying RL-based detailed placement are still limited. This setting differs from other placement problems because the input is already legal, actions must preserve legality, and improvements arise from local refinements. This work presents GPlaceRL [7], an open-source graph reinforcement learning framework for detailed placement refinement. To demonstrate the capabilities of GPlaceRL, we study policies trained with proximal policy optimization (PPO) and graph attention network (GAT) encoders through a design-space exploration of architecture and action-space choices. This setup enables systematic evaluation of RL design choices for HPWL reduction. Rather than proposing a universal placer, we provide a reproducible framework for detailed placement research. The main contributions of this paper are summarized hereafter. First, we introduce GPlaceRL, an open-source graph reinforcement learning framework for detailed placement refinement. Second, we formulate detailed placement refinement as a per-design RL problem using graph-based placement states and legal local actions. Third, we demonstrate the capabilities of GPlaceRL through a systematic design-space exploration of PPO policies with GAT encoders, investigating the impact of encoder architecture and action-space design. Finally, we evaluate the framework across five placement benchmarks and analyze the resulting HPWL improvements. The remainder of the paper presents the GPlaceRL framework, evaluates the explored PPO-GAT configurations, and summarizes the main findings. I GPlaceRL Framework Figure 1: Overview of the GPlaceRL framework for detailed placement refinement. A legal placement is converted into a graph state and processed by a graph-based actor–critic policy. In the current implementation, a PPO-GAT policy applies local refinement actions, while periodic greedy evaluation selects the best checkpoint for generating the final optimized placement. GPlaceRL formulates detailed placement refinement as a sequential decision-making problem. Given an initial legal placement, the agent repeatedly applies local modifications and receives feedback according to the resulting change in HPWL. Rather than training a universal policy to generalize across unrelated circuits, GPlaceRL adopts a per-design optimization setting: for each benchmark, the policy is trained on the corresponding placement graph and used to produce a refined placement for that design. This allows the agent to exploit design-specific connectivity and spatial structure. Figure 1 illustrates the complete flow. The legal placement and netlist are parsed into a graph state that captures both connectivity and physical location. A PPO actor–critic policy with a shared GAT encoder processes this state and predicts a factorized action. The placement environment applies the selected local operation, updates the graph features, and returns an HPWL-based reward. During training, PPO updates the policy from sampled rollouts, while periodic greedy evaluation selects the checkpoint that produces the best placement. I-A Graph representation of placement To expose circuit connectivity to the policy, the placement is represented as a graph G=(V,E)G=(V,E). Nodes correspond to movable standard cells and fixed I/O pins, while edges approximate net connectivity. Since physical nets are hyperedges, each net is converted into pairwise graph edges by connecting the components that belong to the same net. The resulting graph is processed using PyTorch Geometric. Each node v is assigned the feature vector: v=[isio,degnorm,xnorm,ynorm,costproxy,iscand]⊤x_v=[\,is_io,\,deg_norm,\,x_norm,\,y_norm,\,cost_proxy,\,is_cand\,] The first two features are static. The binary feature isiois_io identifies fixed I/O pins, and degnormdeg_norm is the graph degree normalized by the maximum node degree. The remaining features are recomputed after each placement action. The normalized coordinates (xnorm,ynorm)(x_norm,y_norm) encode physical position inside the placement core. The feature costproxycost_proxy is a local wirelength proxy, computed as the average Manhattan distance from a node to its graph neighbors and normalized by the core dimensions. Finally, iscandis_cand marks whether a movable cell belongs to the current source-cell candidate set. Together, these features provide the policy with connectivity, geometry, and local placement-quality information. I-B Placement environment and action space The environment operates on a placement grid with discrete legal sites. Each episode starts from the benchmark legal placement. The environment maintains legality throughout the trajectory by masking invalid choices and applying only actions that preserve placement constraints. The objective is to reduce HPWL, which is widely used as a proxy for routing cost during placement optimization. Because the full placement decision space is extremely large, we use the factorized action representation shown in Fig. 1. At each step, the policy makes the following conditional decisions: 1. Source cell selection: A candidate set C is constructed from movable cells with the highest costproxycost_proxy values. The agent first selects one source cell from this set. 2. Action type selection: Given the selected source cell, the agent selects one operation from the experiment-specific action set: MOVE, SWAP, or NO-OP. In the ablation study, this set is restricted to move-only, swap-only, or swap & move variants. 3. Target selection: Depending on the selected operation, the agent chooses either another cell for swapping or an empty site for relocation. Candidate targets are restricted to legal positions within a local spatial window around the source cell. If NO-OP is selected, the placement state is left unchanged. This formulation reduces the action-space size while retaining the two local transformations needed for detailed refinement: small relocations and pairwise exchanges. I-C Policy network The policy follows an actor–critic architecture with a shared GAT encoder. The encoder processes node features and graph connectivity using stacked GAT layers, producing node embeddings and a global graph embedding obtained through mean pooling. The actor is divided into three heads that mirror the factorized action. The source head scores candidate cells, the action head predicts the refinement operation, and the target head scores valid target cells or sites conditioned on the selected source and operation. The critic estimates the state value V(st)V(s_t) from the global graph embedding. This architecture is well suited for placement refinement because attention allows the encoder to weight neighboring cells and pins according to their relevance, rather than treating all graph connections uniformly. I-D Training and reward The policy is trained using PPO, which stabilizes learning by constraining policy updates through a clipped objective. During rollouts, the agent samples actions from the factorized policy distributions and collects transitions (st,at,rt,st+1)(s_t,a_t,r_t,s_t+1). The critic estimates V(st)V(s_t), and advantages are computed using generalized advantage estimation (GAE) to reduce variance in the policy-gradient estimates. The reward is defined using the incremental improvement in HPWL: rt=HPWLt−1−HPWLtHPWLstartr_t= HPWL_t-1-HPWL_tHPWL_start (1) , where HPWLstartHPWL_start is the wirelength of the initial legal placement. A positive reward indicates that the current action reduces HPWL, while a negative reward penalizes degradations. Normalizing by the initial HPWL makes rewards comparable across designs with different wirelength scales. This dense reward provides immediate feedback after each local refinement step and encourages the agent to discover beneficial action sequences. I-E Evaluation During evaluation, the learned policy is applied greedily. At each step, the highest-scoring legal action is selected instead of sampling from the policy distribution. Since a later action may partially undo an earlier improvement, we report the best HPWL reached during the evaluation trajectory: Δ(%)=100⋅HPWLstart−HPWLbestHPWLstart (\%)=100· HPWL_start-HPWL_bestHPWL_start (2) The checkpoint achieving the best median Δ(%) (\%) across evaluation rollouts is selected as the final optimized policy. I Experimental evaluation I-A Experimental setup We evaluate GPlaceRL using PPO-GAT policies on five internally generated placement benchmarks representative of modern digital designs, named as B1–B5. Each benchmark corresponds to a legal placement snapshot produced by an EDA flow and contains the placement core, standard cells, fixed I/O pins, and net connectivity. Therefore, the problem considered in this work is not initial placement, but placement refinement: the agent starts from an already legal placement and applies local actions to reduce wirelength while preserving legality. Table I summarizes the main characteristics of the evaluated benchmarks. Table I: Statistics of the evaluated placement benchmarks. Design Core size Cells Nets IOs Initial HPWL B1 9.27×9.2169.27× 9.216 85 1494 26 1006 B2 21.78×21.88821.78× 21.888 437 12128 55 13028 B3 54.0×54.14454.0× 54.144 2976 135314 632 206641 B4 76.14×76.03276.14× 76.032 6848 372952 332 723181 B5 129.24×129.024129.24× 129.024 19463 1010888 674 3469054 To study the effect of the model configuration, we conduct both an ablation study and a design-space exploration. More specifically, we vary the number of GAT layers (L∈2,4L∈\2,4\), the number of attention heads (H∈4,8H∈\4,8\), and the available action space of the PPO agent. Three action settings are examined: move-only, swap-only, and swap & move. All experiments are trained for 1500 episodes with batch size 10 and learning rate 10−410^-4. As performance metric, we report the best greedy HPWL improvement Δ(%) (\%) defined in Eq. (2). I-B Experimental results Table I summarizes the best evaluation result obtained for each benchmark and action space, together with the corresponding GAT configuration. The reported value corresponds to the best greedy evaluation checkpoint observed during training, rather than necessarily the final training episode. Across the five benchmarks, the best HPWL improvement ranges from 3.27%3.27\% on B5 to 32.87%32.87\% on B1. The combined swap & move action space achieves the best result for every evaluated design, indicating that combining displacement and pairwise exchange operations gives the policy the most effective refinement capability. Although performance varies across benchmarks and architectures, the results reveal three consistent trends. Table I: The top-performing RL architecture for each benchmark and action space. Design Action space Best eval Δ (%) Best config Best episode B1 Move only 28.26 L2-H4 1400 B1 Swap only 23.10 L2-H4 60 B1 Swap + Move 32.87 L2-H4 1360 B2 Move only 12.65 L2-H8 1380 B2 Swap only 8.86 L4-H8 70 B2 Swap + Move 13.35 L2-H8 1280 B3 Move only 10.58 L4-H8 820 B3 Swap only 7.94 L4-H8 360 B3 Swap + Move 12.11 L2-H4 870 B4 Move only 3.96 L2-H8 770 B4 Swap only 1.74 L2-H8 1260 B4 Swap + Move 5.25 L2-H4 420 B5 Move only 2.91 L2-H8 1170 B5 Swap only 0.18 L4-H8 570 B5 Swap + Move 3.27 L2-H4 980 GNN layers. Table I shows that the best overall results are consistently obtained with two-layer GAT models. Four-layer configurations appear only in a few action-specific cases, mainly when the policy is restricted to simpler action spaces. This suggests that, for placement refinement, shallow message passing provides sufficient local structural context while avoiding the instability that deeper GNNs may introduce through oversmoothing and over-squashing. Attention heads. Increasing the number of attention heads does not necessarily translate into better results. Both H4 and H8 configurations appear among the best entries, depending on the benchmark and action space. For example, B4 selects an H8 configuration under swap-only, whereas its best overall result is obtained with an H4 configuration under swap & move. This suggests that attention-head count should be treated as a design parameter rather than a monotonic source of improvement, which is consistent with prior observations on diminishing returns from additional heads [8]. Figure 2: Qualitative placement and wire-density analysis for the best B3 result in Table I, obtained with the swap & move action space and yielding a 12.11%12.11\% HPWL improvement. (a) Initial placement with selected local windows. (b) Refined placement after optimization. (c) Before-optimization wire-density maps for the selected windows. (d) After-optimization wire-density maps. (e) Density-change maps computed as before minus after, where red indicates reduced wire density and blue indicates increased wire density. RL action space. The action space has the clearest impact on the final evaluation score. For all five benchmarks, swap & move outperforms both move-only and swap-only, with the largest absolute gains observed on B1 and B3. This behavior indicates that cell displacement and pairwise exchange are complementary: move actions provide fine-grained local adjustment, while swap actions can escape local ordering constraints. The benefit is especially visible on larger designs such as B3 and B4, although the corresponding convergence behavior remains noisier, as discussed below. Figure 2 provides a qualitative view of the placement refinement achieved by the best B3 policy. This run corresponds to the swap & move entry in Table I and achieves a 12.11%12.11\% HPWL improvement. The full-placement panels compare the design before and after optimization, while the highlighted windows focus on local regions where wire-demand changes are visually apparent. The corresponding density maps indicate that the refinement reduces wire concentration in selected regions, which is consistent with the measured HPWL reduction. Finally, Figure 3 illustrates the convergence behavior observed for the B4 swap & move policy reported in Table I. The best greedy evaluation reaches a 5.25%5.25\% HPWL improvement at episode 420420, matching the selected B4 result in the table. The training improvement increases rapidly during the early episodes and then reaches a plateau, while the greedy evaluation curve peaks and subsequently remains close to, but below, its best value. This trend indicates that the policy progressively identifies useful local refinements, with the PPO loss decreasing as training stabilizes. Figure 3: Convergence behavior for the B4 swap & move experiment. The best evaluation reaches a 5.25%5.25\% HPWL improvement at episode 420420, corresponding to the selected B4 result in Table I. IV Conclusion In this paper, we presented GPlaceRL, an open-source graph reinforcement learning framework for detailed placement refinement, and demonstrated its capabilities through a systematic evaluation of PPO-GAT policies. The proposed formulation operates directly on legal placements and applies local refinement actions while preserving legality. Across five benchmarks, the best greedy evaluation results achieve HPWL improvements ranging from 3.27% to 32.87%. The results show that compact two-layer GAT policies are generally the most effective and that combined swap-and-move action spaces outperform individual action variants. Overall, GPlaceRL provides a reproducible framework for systematic research on RL-based detailed placement refinement. Future work will focus on larger benchmarks and multi-objective optimization incorporating congestion and timing awareness. References [1] Y. Lin, B. Yu, and D. Z. Pan (2016) Detailed placement in advanced technology nodes: a survey. In Proc. of the International Conference on Solid-State and Integrated Circuit Technology (ICSICT), p. 836–839. Cited by: §I. [2] M. Pan, N. Viswanathan, and C. Chu (2005) An efficient and effective detailed placement algorithm. In Proc. of the International Conference on Computer-Aided Design (ICCD), p. 48–55. Cited by: §I. [3] J. Cong and M. Xie (2006) A robust detailed placement for mixed-size ic designs. In Proc. of the Asia and South Pacific Design Automation Conference (ASP-DAC), p. 188–194. Cited by: §I. [4] A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. M. Songhori, S. Wang, Y. Lee, E. Johnson, O. Pathak, A. Nazi, J. Pak, A. Tong, K. Srinivasa, W. Hang, E. Tuncer, Q. V. Le, J. Laudon, R. Ho, R. Carpenter, and J. Dean (2021) A graph placement methodology for fast chip design. Nature 594, p. 207 – 212. Cited by: §I. [5] Y. Lu, S. S. Kiran Pentapati, L. Zhu, K. Samadi, and S. K. Lim (2020) TP-gnn: a graph neural network framework for tier partitioning in monolithic 3d ics. In Proc. of the Design Automation Conference (DAC), p. 1–6. Cited by: §I. [6] Q. Xu, H. Geng, S. Chen, B. Yuan, C. Zhuo, Y. Kang, and X. Wen (2022) GoodFloorplan: graph convolutional network and reinforcement learning-based floorplanning. IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems (TCAD) 41 (10), p. 3492–3502. Cited by: §I. [7] GPlaceRL. Note: https://github.com/pastoikos/Graph-RL-Placement-Optimizer Cited by: §I. [8] P. Michel, O. Levy, and G. Neubig (2019) Are sixteen heads really better than one?. In Proc. of the International Conference on Neural Information Processing Systems (ICNIPS), p. 14037–14047. Cited by: §I-B.