Paper deep dive
DGPO: RL-Steered Graph Diffusion for Neural Architecture Generation
Aleksei Liuliakov, Luca Hermes, Barbara Hammer
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/20/2026, 8:18:12 PM
Summary
The paper introduces Directed Graph Policy Optimization (DGPO), a method extending Reinforcement Learning (RL) fine-tuning of discrete graph diffusion models to Directed Acyclic Graphs (DAGs). By incorporating topological node ordering and positional encoding, DGPO preserves edge directionality often discarded by undirected graph diffusion methods. Validated on NAS-Bench-101 and NAS-Bench-201, DGPO matches benchmark optima and demonstrates the ability to learn transferable structural priors, generating high-accuracy architectures even when pretrained on only 7% of the search space. Bidirectional control experiments confirm that the model is genuinely steered by reward signals.
Entities (11)
Relation Signals (9)
DGPO → handlesstructure → Directed Acyclic Graphs
confidence 96% · DGPO extends reinforcement learning fine-tuning of discrete graph diffusion models to DAGs
DGPO → extends → GDPO
confidence 95% · DGPO extends GDPO [12], a graph diffusion RL fine-tuning framework, to directed acyclic graphs
DGPO → validateson → NAS-Bench-101
confidence 95% · Validated on NAS-Bench-101 and NAS-Bench-201
DGPO → validateson → NAS-Bench-201
confidence 95% · Validated on NAS-Bench-101 and NAS-Bench-201
Reinforcement Learning → steers → Diffusion Models
confidence 94% · Reinforcement learning fine-tuning has proven effective for steering generative diffusion models toward desired properties
DGPO → appliedto → Neural Architecture Search
confidence 93% · Graph diffusion models have similarly been applied to combinatorial structure generation, including neural architecture search (NAS).
DGPO → usesbackbone → DiGress
confidence 92% · DGPO builds on DiGress as its discrete diffusion backbone.
DGPO → usestechnique → Topological Node Ordering
confidence 90% · DGPO extends GDPO to directed acyclic graphs via topological node ordering and positional encoding.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning fine-tuning has proven effective for steering generative diffusion models toward desired properties in image and molecular domains. Graph diffusion models have similarly been applied to combinatorial structure generation, including neural architecture search (NAS). However, neural architectures are directed acyclic graphs (DAGs) where edge direction encodes functional semantics such as data flow-information that existing graph diffusion methods, designed for undirected structures, discard. We propose Directed Graph Policy Optimization (DGPO), which extends reinforcement learning fine-tuning of discrete graph diffusion models to DAGs via topological node ordering and positional encoding. Validated on NAS-Bench-101 and NAS-Bench-201, DGPO matches the benchmark optimum on all three NAS-Bench-201 tasks (91.61%, 73.49%, 46.77%). The central finding is that the model learns transferable structural priors: pretrained on only 7% of the search space, it generates near-oracle architectures after fine-tuning, within 0.32 percentage points of the full-data model and extrapolating 7.3 percentage points beyond its training ceiling. Bidirectional control experiments confirm genuine reward-driven steering, with inverse optimization reaching near random-chance accuracy (9.5%). These results demonstrate that reinforcement learning-steered discrete diffusion, once extended to handle directionality, provides a controllable generative framework for directed combinatorial structures.
Tags
Links
- Source: https://arxiv.org/abs/2602.19261v2
- Canonical: https://arxiv.org/abs/2602.19261v2
Trouble viewing inline? Open PDF directly →
Full Text
33,063 characters extracted from source content.
Expand or collapse full text
DGPO: RL-Steered Graph Diffusion for Neural Architecture Generation Aleksei Liuliakov , Luca Hermes , Barbara Hammer Abstract Reinforcement learning fine-tuning has proven effective for steering generative diffusion models toward desired properties in image and molecular domains. Graph diffusion models have similarly been applied to combinatorial structure generation, including neural architecture search (NAS). However, neural architectures are directed acyclic graphs (DAGs) where edge direction encodes functional semantics such as data flow - information that existing graph diffusion methods, designed for undirected structures, discard. We propose Directed Graph Policy Optimization (DGPO), which extends reinforcement learning fine-tuning of discrete graph diffusion models to DAGs via topological node ordering and positional encoding. Validated on NAS-Bench-101 and NAS-Bench-201, DGPO matches the benchmark optimum on all three NAS-Bench-201 tasks (91.61%, 73.49%, 46.77%). The central finding is that the model learns transferable structural priors: pretrained on only 7% of the search space, it generates near-oracle architectures after fine-tuning, within 0.32 percentage points of the full-data model and extrapolating 7.3 percentage points beyond its training ceiling. Bidirectional control experiments confirm genuine reward-driven steering, with inverse optimization reaching near random-chance accuracy (9.5%). These results demonstrate that reinforcement learning-steered discrete diffusion, once extended to handle directionality, provides a controllable generative framework for directed combinatorial structures. Published at IEEE IJCNN 2026 (WCCI). © 2026 IEEE. I Introduction Diffusion models have emerged as powerful generative frameworks for learning complex distributions, with recent extensions to graph-structured data [16] enabling direct generation of molecular graphs and other combinatorial objects. Reinforcement learning (RL) fine-tuning steers pretrained diffusion models toward desired properties: Denoising Diffusion Policy Optimization (DDPO) [4] introduced this for image generation, and Graph Diffusion Policy Optimization (GDPO) [12] extended it to molecular graphs. In the graph domain, however, these methods have been limited to undirected structures. Many important domains involve directed acyclic graphs (DAGs): neural architectures encode data flow through directed edges, causal networks represent asymmetric dependencies, and computational workflows specify ordered execution. In such graphs, edge direction encodes functional semantics that is destroyed by symmetric treatment. Neural architecture search (NAS) [21, 18] - the problem of automatically discovering high-performing neural network architectures from a structured search space -exemplifies this challenge: existing generative approaches either condition generation on target accuracy [2] or build on graph diffusion models designed for undirected structures [16], forgoing directional information. Extending RL-steered discrete diffusion to DAGs requires respecting and reconstructing this directional structure. We propose Directed Graph Policy Optimization (DGPO), extending GDPO [12], a graph diffusion RL fine-tuning framework, to directed acyclic graphs via topological node ordering and positional encoding. This enables the discrete diffusion process to respect directionality while preserving the reward-driven steering mechanism of the underlying framework. We validate DGPO on two public NAS benchmarks, NAS-Bench-101 [19] and NAS-Bench-201 [5], achieving competitive architecture generation that reaches the NB201 benchmark optimum on all three tasks (91.61%, 73.49%, and 46.77%). We find that the model can acquire transferable structural priors: pretrained on only 7% of the search space, it generates near-oracle architectures after RL fine-tuning, within 0.32 percentage points of the full-data model and extrapolating up to 7.3p beyond its training ceiling. We provide additional evidence for reward-driven steering via bidirectional control, where inverting the reward signal drives generation toward near random-chance accuracy (9.5%), and we also show how the framework extends to multi-objective reward formulations. Our contributions are: 1. Methodological: DGPO extends GDPO to directed acyclic graphs via topological node ordering and positional encoding, enabling controllable generation of DAG-structured objects. 2. Empirical (primary): We show evidence that the resulting model learns transferable structural priors: a model trained on only 7% of the NAS search space generates near-oracle architectures after RL fine-tuning (within 0.32p), with up to ++7.3p extrapolation beyond its training ceiling. 3. Empirical (supporting): We provide additional evidence for the steering mechanism through bidirectional control (inverse reaches near random chance at 9.5%) and show extension to multi-objective reward formulations. Code and pretrained models are available at: https://github.com/AlekseiLiu/DGPO. I Related Work Graph diffusion models. Diffusion models have been extended to graph-structured data through both discrete and continuous formulations [7]. DiGress [16] introduced discrete denoising diffusion for graphs, applying categorical noise transitions to node and edge attributes with a graph transformer denoiser. Autoregressive variants [9] generate graphs sequentially but sacrifice parallelism. DGPO builds on DiGress as its discrete diffusion backbone. RL fine-tuning of diffusion models. Reinforcement learning can steer pretrained diffusion models toward desired properties by framing denoising as a Markov decision process [15]. DDPO [4] introduced this approach for image generation, optimizing human preference scores via policy gradients. DPOK [8] adds KL regularization for text-to-image alignment. GDPO [12] adapts the framework to molecular graph generation using reward-weighted denoising trajectories. These methods operate on undirected structures (images, molecules); DGPO extends the GDPO framework to directed acyclic graphs. Neural architecture search. NAS has been addressed through RL controllers [21], evolutionary algorithms [14], differentiable relaxations such as DARTS [11], and predictor-based methods including BANANAS [17]. Generative approaches learn to directly produce architectures: D-VAE [20] uses variational autoencoders for directed graph generation, DiffusionNAG [1] applies diffusion models for task-guided generation, AG-Net [13] uses autoregressive generation with a learned surrogate, and GraphPNAS [10] learns distributions over high-performing architectures. DiNAS [2] conditions a graph diffusion model on target accuracy to generate architectures directly. DGPO differs from conditioning-based approaches by steering the generation distribution via RL fine-tuning, enabling capabilities beyond accuracy targeting: bidirectional control, out-of-distribution discovery from filtered data, and multi-objective optimization. We extend an existing RL fine-tuning framework (GDPO) to a structurally different graph class (DAGs), rather than proposing a new RL algorithm or diffusion architecture. The contribution lies in enabling RL-steered discrete diffusion for directed combinatorial structures and providing empirical evidence for transferable structural priors in the resulting generative model. I Method We first review the discrete diffusion and RL fine-tuning foundations that DGPO builds on, then present our extension to directed acyclic graphs. I-A Preliminaries Discrete graph diffusion. DiGress [16] defines a discrete diffusion process over graphs =(,)G=(X,E) with n nodes. Node attributes ∈1,…,anX∈\1,…,a\^n take one of a categorical types; edge attributes ∈0,1,…,bn×nE∈\0,1,…,b\^n× n take one of b types plus an absent state. Attributes are categorical because the diffusion process operates via discrete Markov transition matrices over a finite state set at each step, rather than adding continuous noise. The forward process independently corrupts each attribute of the clean graph 0G_0: q(t∣t−1)=∏iq(xit∣xit−1)∏i<jq(eijt∣eijt−1),q(G_t _t-1)=\! _iq(x_i^t x_i^t-1)\! _i<jq(e_ij^t e_ij^t-1), (1) where the edge product runs over unique pairs i<ji<j since the diffusion operates on a symmetric (undirected) intermediate representation. Each factor is a categorical draw: xit∼Cat((it−1)⊤tX)x_i^t \! ((x_i^t-1) Q_t^X ) and eijt∼Cat((ijt−1)⊤tE)e_ij^t \! ((e_ij^t-1) Q_t^E ), with tX∈ℝa×aQ_t^X ^a× a and tE∈ℝ(b+1)×(b+1)Q_t^E ^(b+1)×(b+1) the per-step transition matrices for node and edge attributes, respectively. Over T steps this produces 0,1,…,TG_0,G_1,…,G_T, with TG_T approaching the limit distribution of the transition model (uniform or dataset marginals; following DiGress, we use marginals in our implementation [16]). A denoising network φθ _θ, parameterized by θ, learns to invert this corruption by predicting 0G_0 from the noisy input tG_t at each step [3]. Graphs are generated by sampling TG_T from the prior and iteratively applying the learned reverse transitions. Reinforcement learning fine-tuning of diffusion models. GDPO [12] formulates the denoising process as a T-step Markov decision process: the state is t=(T−t,T−t)s_t=(G_T-t,\,T-t), the action is t=T−t−1a_t=G_T-t-1, the policy πθ _θ is the denoising model, and a sparse reward r(0)r(G_0) is assigned only to the final generated graph. The objective is to maximize expected reward ∼πθ[r(0)]E_ τ _θ[r(G_0)] over denoising trajectories =(T,…,0) τ=(G_T,…,G_0). DDPO [4] introduced this MDP framing for image diffusion; GDPO extends it to molecular graphs using an eager policy gradient that directly optimizes ∇θlogpθ(0∣t) _θ p_θ(G_0 _t) at each timestep, reducing gradient variance compared to the standard REINFORCE estimator [12]. I-B DGPO: Extending GDPO to Directed Acyclic Graphs GDPO was developed for undirected molecular graphs where edge semantics are symmetric. In directed acyclic graphs (DAGs) - neural architectures, causal networks, computational workflows - edge direction encodes functional semantics such as data flow or causation. Treating directed edges as symmetric destroys this information. We propose Directed Graph Policy Optimization (DGPO), which extends GDPO to DAGs via three components. Topological node ordering. Before diffusion, we apply a topological ordering σ to the DAG nodes such that ∀(u,v)∈ℰ:σ(u)<σ(v)∀(u,v) :\;σ(u)<σ(v), where ℰE denotes the edge set of the DAG. Under this ordering the adjacency matrix A becomes strictly upper-triangular: ij≠0⇒i<jA_ij≠ 0 i<j, so all directional information is encoded in node indices. NAS benchmarks store architectures in a consistent DAG order (input node first, output node last); for general DAGs, any standard topological sort (e.g., Kahn’s algorithm) can be applied in preprocessing. Positional encoding. We augment each node’s feature vector with a sinusoidal positional encoding i∈ℝdp_i ^d derived from its topological position i [6]: i,2k=sin(i/100002k/d),i,2k+1=cos(i/100002k/d),p_i,2k= \! (i/10000^2k/d ), _i,2k+1= \! (i/10000^2k/d ), (2) for k=0,…,d/2−1k=0,…,d/2-1, where d is the hidden dimension of the graph transformer. The encoding is added to node features after the input projection and before the first transformer layer: i←i+ih_i _i+p_i. This provides φθ _θ with explicit positional information, enabling it to distinguish node roles (e.g., early- vs. late-layer operations) and reconstruct directed edges from the noisy intermediate graph. DAG recovery. During diffusion, intermediate graphs are treated as undirected to remain compatible with the DiGress backbone, which symmetrizes edge noise. After the final denoising step produces a full adjacency matrix ∈0,1,…,bn×nA∈\0,1,…,b\^n× n, we recover a valid DAG by retaining only upper-triangular entries: ijdag=ijA^dag_ij=A_ij if i<ji<j, and 0 otherwise (diagonal masked, as self-loops are invalid in the NAS benchmarks). This projection guarantees acyclicity by construction and requires no learned components. Together, topological node ordering and positional encoding enable the DiGress backbone and GDPO fine-tuning framework - designed for undirected graphs - to handle DAGs without modifying the diffusion or RL architectures. The extension is domain-agnostic: any DAG-structured combinatorial problem can be addressed by providing an appropriate reward signal. I-C Training Objective Following GDPO [12], the DGPO training objective is a reward-weighted cross-entropy loss over denoising trajectories with advantage normalization: ℒDGPO(θ)=1K∑k=1KT|k|∑t∈kA^k⋅ℒCE(φθ(t(k)),0(k)),L_DGPO(θ)= 1K _k=1^K T|T_k| _t _k A_k·L_CE\! ( _θ(G_t^(k)),\,G_0^(k) ), (3) where K is the number of trajectories per batch, kT_k is a uniformly sampled subset of timesteps for trajectory k, and the cross-entropy decomposes over nodes and edges: ℒCE=∑iCE(xi,x^i)+λ∑i,jCE(eij,e^ij).L_CE= _iCE(x_i,\, x_i)+λ _i,jCE(e_ij,\, e_ij). (4) The advantage estimate normalizes the reward signal: A^k=clip(rk−r¯σr,−5, 5), A_k=clip\! ( r_k- r _r,\,-5,\,5 ), (5) where r¯ r and σr _r are the running mean and standard deviation of the reward. The cross-entropy provides per-timestep ∇θlogpθ(0∣t) _θ p_θ(G_0 _t) gradients, while the advantage reweights each trajectory by its relative quality, yielding a REINFORCE-style policy gradient with variance reduction. The reward signal admits several formulations: single-objective (r(0)r(G_0) = normalized validation accuracy), inverse (−r(0)-r(G_0) for mechanism validation), and multi-objective (∑iwi⋅ri(0) _iw_i· r_i(G_0) for compound tasks). I-D Two-Phase Training Phase 1: Pretraining. The diffusion model is trained on the benchmark architecture distribution using the denoising cross-entropy loss ℒCEL_CE, which trains φθ _θ to predict 0G_0 from noisy inputs tG_t at uniformly sampled timesteps t∈1,…,Tt∈\1,…,T\. No reward signal or advantage term is used; the model learns the structural grammar of the search space - valid node types, connectivity patterns, and size distributions - purely from the data distribution. Phase 2: RL fine-tuning. Starting from the pretrained checkpoint, the model is fine-tuned using the DGPO objective (3). Layer freezing (75% of parameters) stabilizes training and prevents catastrophic forgetting; gradient accumulation compensates for the small batch sizes required by online reward evaluation. This phase shifts the generation distribution toward the reward signal while preserving the structural validity learned in Phase 1. IV Experiments IV-A Experimental Setup Benchmarks. We evaluate on NAS-Bench-101 (NB101) [19] and NAS-Bench-201 (NB201) [5]. NB101 contains 423,624 unique architectures represented as directed acyclic graphs with up to seven nodes and 21 possible edges, evaluated on CIFAR-10. NB201 defines a cell-based search space of 15,625 architectures evaluated on three tasks: CIFAR-10, CIFAR-100, and ImageNet-16-120. Model architecture. DGPO uses a DiGress backbone [16] with eight Transformer layers, T=800T=800 diffusion steps, and a cosine noise schedule. We augment each node with a topological positional encoding that captures its position in the topological ordering (Section I-B). At generation time, valid DAGs are recovered via upper-triangular projection of the generated adjacency matrix. Training protocol. Training follows a two-phase procedure. Phase 1 pretrains the diffusion model on the full benchmark architecture distribution using cross-entropy loss. Phase 2 applies RL fine-tuning (RL-FT) for 60 epochs with batch size K=15K=15, learning rates of 7×10−77×10^-7 (NB101) and 5×10−75×10^-7 (NB201), 75% layer freezing, and AdamW optimization. The training objective is a REINFORCE-style policy gradient with advantage normalization (Section I-C). Evaluation protocol. All experiments use three seeds (42, 123, 456); results are reported as mean ± std. Each evaluation draws 300 architectures from the current model. NB101 and NB201 results are obtained via benchmark API lookup. Baselines. We compare against two internal baselines: random search (uniform sampling from the search space) and pretrained-only sampling (generation from the pretrained diffusion model without RL-FT). External comparisons with prior NAS methods are presented in Section IV-B. IV-B Baseline Results DGPO steers DAG generation toward high-accuracy architectures on both NB101 and NB201. Fig. 1 shows training dynamics over RL-FT epochs: validation accuracy increases over training on both benchmarks while mean reward rises in tandem, with narrow ±1σ±1σ bands across three seeds suggesting stable convergence. Figure 1: Training dynamics of DGPO on (a) NB101 and (b) NB201 (CIFAR-10): validation accuracy (solid, left axis) and mean reward (dashed, right axis) over RL-FT epochs, with ±1σ±1σ bands across 3 seeds. Horizontal lines: random search and pretrained-only baselines. Both metrics converge reliably, confirming that RL fine-tuning steers the generation distribution toward higher-quality architectures. Fig. 2 illustrates the underlying mechanism: RL-FT progressively shifts the generated architecture distribution toward higher accuracy, concentrating probability mass in the high-performing region. Figure 2: Distribution of generated architectures over RL-FT epochs on (a) NB101 and (b) NB201 (CIFAR-10). Each strip shows 300 sampled architectures (dots) at a given epoch, with mean accuracy (diamond) and top-5 architectures (stars). As training progresses (bottom to top), the mean and overall sample density shift markedly toward higher accuracy, demonstrating that RL fine-tuning reshapes the generative distribution rather than merely selecting isolated high performers. Single seed (42); n=300n=300 samples per epoch. Table I compares DGPO against published NAS methods. On NB201, DGPO matches the benchmark optimum on all three datasets: 91.61% (CIFAR-10), 73.49% (CIFAR-100), and 46.77% (ImageNet-16-120), surpassing DiNAS [2] on ImageNet-16-120 (46.77% vs 46.66%). On NB101, DGPO achieves 94.50%, competitive with established methods but below DiNAS (94.98%). On NB101, several established methods achieve similar accuracy levels within a narrow range; DGPO’s distinctive value on this benchmark lies in the controllable distribution steering demonstrated in Sections IV-C and IV-D. TABLE I: Comparison with SOTA NAS methods. Max validation accuracy (%) on NB101 and NB201. Prior results from [2] (mean over 10 runs); DGPO: mean± over 3 seeds (NB101) / deterministic API lookup (NB201). * = benchmark optimum. DGPO uses ∼ 2k queries; other methods use 150–192. Method NB101 NB201 Val (%) Val C-10 C-100 IN-16 Optimum 95.06 91.61 73.49 46.77 Random Search 94.31 91.12 72.08 45.97 Reg. Evol. 94.47 – – – Bayesian Opt. 94.57 91.54 73.26 46.43 BANANAS 94.73 91.56 73.49* 46.65 AG-Net 94.90 91.60 73.49* 46.64 DiNAS 94.98 91.61* 73.49* 46.66 DGPO 94.50±0.0294.50 ± 0.02 91.61* 73.49* 46.77* The additional query cost (∼ 2k vs 150–192) reflects the online RL paradigm: rather than a fixed-budget search, DGPO fine-tunes the generative distribution itself. This distribution-level steering enables the capabilities demonstrated in the following sections - bidirectional control, out-of-distribution discovery, and multi-objective optimization - which are not available to conditioning-based methods. IV-C Transferable Structural Priors Does the generative model learn compositional structural priors, or does it merely memorize the training distribution? To answer this, we filter the pretraining data to architectures below a quality threshold T, removing all high-accuracy instances, and then apply RL-FT to the filtered model. Specifically, we retain only architectures with validation accuracy below =0.87T=0.87 on NB101 (7% of data; ∼ 30k of 423k) and below =0.85T=0.85 on NB201 (29%; ∼ 4,500 of 15,625). Out-of-distribution (OOD) architectures - those with validation accuracy above T - are entirely absent from the filtered pretraining set. Filtered vs full pretraining. Table I compares RL-FT results starting from the filtered versus full pretrained model. On NB101, the filtered model achieves 94.18%, only 0.32 percentage points below the full model’s 94.50%. On NB201, the filtered model reaches 91.71%, within noise of the full model (91.70%). Removing 93% of training data costs only 0.32p in generation quality on NB101 and is negligible on NB201, indicating that weak-region data suffices for acquiring structural priors. TABLE I: RL-FT on full vs filtered pretraining: max validation accuracy (mean ± std, 3 seeds). Filtered models are pretrained exclusively on architectures below threshold T (NB101: =0.87T=0.87, 7% of data; NB201: =0.85T=0.85, 29%). Filtered pretraining loses only 0.32p on NB101 and is within noise on NB201. Method NB101 NB201 (C-10) Full Pretrain 94.50 ± 0.02 91.70 ± 0.02 Filtered Pretrain 94.18 ± 0.33 91.71 ± 0.02 OOD architecture discovery. The filtered model generates architectures far above its training ceiling. Fig. 3 (top row) visualizes this: the filtered pretrained distribution (middle strip) clusters below T, but after RL-FT (top strip) the distribution recovers to resemble the full pretrained reference (bottom strip) - despite never having seen above-T architectures during pretraining. The bottom row tracks the threshold crossing rate (percentage of generated samples with accuracy ≥T) over RL-FT epochs. On NB101, the filtered model starts at ∼ 21% above T and climbs to ∼ 99% after RL-FT. On NB201, it starts at ∼ 9% and reaches ∼ 99%. Figure 3: OOD architecture discovery on (a, c) NB101 (=0.87T=0.87) and (b, d) NB201 (=0.85T=0.85). Top: distribution comparison - full pretrain (reference), filtered pretrain (epoch 0), and RL-FT (final epoch). Shaded region marks OOD architectures above T. Bottom: threshold crossing rate over RL-FT epochs. After pretraining on only sub-threshold architectures, RL-FT recovers above-threshold generation, demonstrating transferable structural priors. Single seed (42) for distributions; crossing rates are 3-seed aggregates. Per-seed consistency. Across all three seeds, the OOD discovery rate is 100% on both benchmarks: every seed generates architectures above T after RL-FT. On NB101, the best generated architecture reaches 94.27±0.11%94.27± 0.11\%, extrapolating ++7.27p above the training ceiling, with a mean crossing rate of 67.7±3.2%67.7± 3.2\%. On NB201, the best architecture reaches 91.68±0.06%91.68± 0.06\%, matching the benchmark oracle, with crossing rate 79.2±1.7%79.2± 1.7\% and an OOD lift of 0.77±0.020.77± 0.02 (OOD architectures are 77% more likely to appear among the top-performing generated samples). Interpretation. These results provide evidence for compositional generalization in the diffusion model: pretraining on weak architectures captures local structural motifs - connectivity patterns, operation combinations - that, when recombined under RL-FT guidance, produce high-quality architectures absent from the training set. The model extrapolates beyond its training support by recombining learned primitives, not by memorizing successful configurations. We use filtering as a controlled stress test; the threshold T is fixed before RL-FT and not tuned post hoc. IV-D Steering Versatility Bidirectional control. The same RL-FT framework steers generation in either direction, validating that the reward signal drives genuine structure–performance learning rather than exploitation of the pretraining distribution. Fig. 4 shows forward (maximize accuracy) and inverse (minimize accuracy) DGPO trajectories. The forward trajectory rises above the expected best-of-batch under random sampling ([max(15)]E[ (15)], computed via bootstrap with K=10,000K=10,000 resamples). Negating the reward signal causes the model to generate progressively worse architectures: the inverse trajectory converges toward near random-chance accuracy (9.51% on NB101, 9.71% on NB201; chance is 10% for 10-class classification). This provides evidence against explanations where the model merely exploits the pretraining distribution: with a negated reward, the same model is driven toward architectures it was not encouraged to generate during pretraining. Figure 4: Bidirectional steering on (a) NB101 and (b) NB201 (CIFAR-10): forward (maximize, ↑ ) and inverse (minimize, ↓ ) DGPO trajectories over RL-FT epochs, with ±1σ±1σ bands (3 seeds). Dashed/dotted lines: expected max/min of random batches (N=15N=15, bootstrap K=10,000K=10,000). The inverse trajectory converges to near random-chance accuracy (∼ 9.5%), supporting reward-driven distribution steering. On NB201, one of three seeds temporarily lingers near a secondary basin (∼ 59%), while others reach the floor, suggesting bimodal structure in the inverse optimization landscape. All seeds reach the global minimum during training; the wider error band reflects basin transitions rather than method failure. Multi-objective steering. We extend DGPO to multi-objective (MO) optimization on NB201 by using a compound reward: the weighted sum of normalized validation accuracies on CIFAR-10, CIFAR-100, and ImageNet-16-120. MO-DGPO improves all three tasks simultaneously, with the composite reward increasing by 26%. The Pareto hypervolume reaches 0.9901, matching the union of three separate single-task runs. MO-DGPO achieves comparable front quality in a single joint run, demonstrating that the RL-FT framework generalizes naturally from single-task to multi-objective formulations without architectural changes. An adversarial weight configuration (negative weight on CIFAR-10) reveals strong task correlation in the NB201 search space: penalizing one task simultaneously degrades others. This confirms the known correlation structure in NB201 and illustrates that DGPO can probe task relationships via reward engineering. V Conclusion We presented DGPO, extending RL-steered discrete graph diffusion to directed acyclic graphs via topological node ordering and positional encoding. Validated on NAS-Bench-101 and NAS-Bench-201, DGPO achieves competitive architecture generation and provides evidence of transferable structural priors: a model pretrained on only weak-region data can generate near-oracle architectures after RL fine-tuning. Bidirectional control and multi-objective steering further support the view that the reward signal drives structure–performance learning. The approach incurs higher query cost than conditioning-based methods, reflecting the online RL paradigm, and has been validated on NAS benchmarks only; generalization to other DAG domains remains to be demonstrated. Promising directions include reducing the query budget through offline RL or sample reuse strategies, scaling to open-ended search spaces such as NAS-Bench-301, and applying DGPO to other directed combinatorial domains including circuit design and causal discovery. References [1] S. An, H. Lee, J. Jo, S. Lee, and S. J. Hwang (2023) DiffusionNAG: task-guided neural architecture generation with diffusion models. arXiv preprint arXiv:2305.16943. Cited by: §I. [2] S. Asthana, J. Conrad, C. Debus, and M. Götz (2024) Multi-conditioned graph diffusion for neural architecture search. Transactions on Machine Learning Research (TMLR). Cited by: §I, §I, §IV-B, TABLE I. [3] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. van den Berg (2021) Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems (NeurIPS). Cited by: §I-A. [4] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine (2024) Training diffusion models with reinforcement learning. International Conference on Learning Representations (ICLR). Cited by: §I, §I, §I-A. [5] X. Dong and Y. Yang (2020) NAS-Bench-201: extending the scope of reproducible neural architecture search. International Conference on Learning Representations (ICLR). Cited by: §I, §IV-A. [6] V. P. Dwivedi and X. Bresson (2021) A generalization of transformer networks to graphs. AAAI Workshop on Deep Learning on Graphs. Cited by: §I-B. [7] W. Fan, C. K. Liu, Y. Liu, J. Li, H. Li, H. Liu, J. Tang, and Q. Li (2023) Generative diffusion models on graphs: methods and applications. arXiv preprint arXiv:2302.02591. Cited by: §I. [8] Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee (2023) DPOK: reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems (NeurIPS). Cited by: §I. [9] L. Kong, J. Cui, H. Sun, Y. Zhuang, B. A. Prakash, and C. Zhang (2023) Autoregressive diffusion model for graph generation. In International Conference on Machine Learning (ICML), Cited by: §I. [10] M. Li, J. Y. Liu, L. Sigal, and R. Liao (2022) GraphPNAS: learning distribution of good neural architectures via deep graph generative models. arXiv preprint arXiv:2211.15155. Cited by: §I. [11] H. Liu, K. Simonyan, and Y. Yang (2019) DARTS: differentiable architecture search. International Conference on Learning Representations (ICLR). Cited by: §I. [12] Y. Liu, C. Du, T. Pang, C. Li, M. Lin, and W. Chen (2024) Graph diffusion policy optimization. Advances in Neural Information Processing Systems 37, p. 9585–9611. Cited by: §I, §I, §I, §I-A, §I-C. [13] J. Lukasik, S. Jung, and M. Keuper (2022) Learning where to look – generative NAS is surprisingly efficient. In European Conference on Computer Vision (ECCV), Cited by: §I. [14] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le (2019) Regularized evolution for image classifier architecture search. AAAI Conference on Artificial Intelligence. Cited by: §I. [15] M. Uehara, Y. Zhao, T. Biancalani, and S. Levine (2024) Understanding reinforcement learning-based fine-tuning of diffusion models: a tutorial and review. arXiv preprint arXiv:2407.13734. Cited by: §I. [16] C. Vignac, I. Krawczuk, A. Siraudin, B. Wang, V. Cevher, and P. Frossard (2023) DiGress: discrete denoising diffusion for graph generation. International Conference on Learning Representations (ICLR). Cited by: §I, §I, §I, §I-A, §I-A, §IV-A. [17] C. White, W. Neiswanger, and Y. Savani (2021) BANANAS: bayesian optimization with neural architectures for neural architecture search. AAAI Conference on Artificial Intelligence. Cited by: §I. [18] C. White, M. Safari, R. Sukthanker, B. Ru, T. Elsken, A. Zela, D. Dey, and F. Hutter (2023) Neural architecture search: insights from 1000 papers. Note: arXiv preprint arXiv:2301.08727 Cited by: §I. [19] C. Ying, A. Klein, E. Real, E. Christiansen, K. Murphy, and F. Hutter (2019) NAS-Bench-101: towards reproducible neural architecture search. Proceedings of the 36th International Conference on Machine Learning (ICML). Cited by: §I, §IV-A. [20] M. Zhang, S. Jiang, Z. Cui, R. Garnett, and Y. Chen (2019) D-VAE: a variational autoencoder for directed acyclic graphs. Advances in Neural Information Processing Systems (NeurIPS). Cited by: §I. [21] B. Zoph and Q. V. Le (2017) Neural architecture search with reinforcement learning. International Conference on Learning Representations (ICLR). Cited by: §I, §I.