Paper deep dive
Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks
Taha Shieenavaz, Shabnam Zareshahraki, Loris Nanni
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms. Notably, the Parallelized Q-Network (PQN) algorithm achieves stable off-policy learning without relying on computationally expensive replay buffers or target networks. However, the representational capacity and parameter efficiency of visual encoders operating in these buffer-free settings remain underexplored. In this work, we systematically investigate the architectural design space of Convolutional Neural Networks for PQN. We design and rigorously evaluate eight distinct CNN topologies, optimizing for sample efficiency under strict parameter constraints. Furthermore, we study the impact of representation and value estimation enhancements by integrating the Hadamax encoding paradigm and advanced Q-learning extensions, including distributional, ensemble, and dueling heads. Extensive experiments on the Atari-57 benchmark demonstrate that our proposed composite architecture, Aftab, achieves an Interquartile Mean (IQM) Human-Normalized Score of 6.479, establishing a 0.86 Probability of Improvement over the standard PQN baseline. Additionally, structural resilience evaluations on the highly non-stationary Procgen Hard benchmark confirm out-of-distribution generalization, with Aftab yielding an IQM Procgen Normalized Score of 0.418 compared to the baseline's 0.382. Ultimately, this work establishes an efficient, probabilistically superior structural reference for model-free reinforcement learning, all while preserving the simplicity and memory efficiency of unbuffered, parallelized optimization. The complete Aftab framework, including all model definitions, training configurations, and raw experimental logs, is open-sourced and available on our GitHub repository: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2608.07335v1
- Canonical: https://arxiv.org/abs/2608.07335v1
Trouble viewing inline? Open PDF directly →
Full Text
128,235 characters extracted from source content.
Expand or collapse full text
Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks Taha Shieenavaz 1 (tahashieenavaz@gmail.com), Shabnam Zareshahraki 1 (shabnamzaresh@gmail.com), Loris Nanni 1 (loris.nanni@unipd.it) 1 Department of Information Engineering, University of Padua, Italy Corresponding author at: Department of Information Engineering, University of Padua, Italy. Taha Shieenavaz Email: tahashieenavaz@gmail.com Shabnam Zareshahraki Email: shabnamzaresh@gmail.com Department of Information Engineering, University of Padua, Italy arXiv:2608.07335v1 [cs.LG] 7 Aug 2026 Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks Taha Shieenavaz a,∗ , Shabnam Zareshahraki a , Loris Nanni a a Department of Information Engineering, University of Padua, Italy Abstract Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms. Notably, the Parallelized Q-Network (PQN) algorithm achieves stable off-policy learning without relying on compu- tationally expensive replay buffers or target networks. However, the represen- tational capacity and parameter efficiency of visual encoders operating in these buffer-free settings remain underexplored. In this work, we systematically in- vestigate the architectural design space of Convolutional Neural Networks for PQN. We design and rigorously evaluate eight distinct CNN topologies, optimiz- ing for sample efficiency under strict parameter constraints. Furthermore, we study the impact of representation and value estimation enhancements by inte- grating the Hadamax encoding paradigm and advanced Q-learning extensions, including distributional, ensemble, and dueling heads. Extensive experiments on the Atari-57 benchmark demonstrate that our proposed composite architecture, Aftab, achieves an Interquartile Mean (IQM) Human-Normalized Score of 6.479, establishing a 0.86 Probability of Improvement over the standard PQN base- line. Additionally, structural resilience evaluations on the highly non-stationary Procgen Hard benchmark confirm out-of-distribution generalization, with Aftab yielding an IQM Procgen Normalized Score of 0.418 compared to the baseline’s ∗ Corresponding author. Email addresses: tahashieenavaz@gmail.com (Taha Shieenavaz), shabnamzaresh@gmail.com (Shabnam Zareshahraki), loris.nanni@unipd.it (Loris Nanni) Preprint submitted to Expert Systems with ApplicationsAugust 10, 2026 0.382. Ultimately, this work establishes an efficient, probabilistically superior structural reference for model-free reinforcement learning, all while preserving the simplicity and memory efficiency of unbuffered, parallelized optimization. The complete Aftab framework, including all model definitions, training con- figurations, and raw experimental logs, is open-sourced and available on our GitHub repository: https://github.com/tahashieenavaz/aftab. Keywords: Deep Reinforcement Learning, Convolutional Neural Networks, Parallelized Q-Network (PQN), Buffer-Free Learning, Hadamax Representation, Distributional Reinforcement Learning, Deep Ensembles, Dueling Architecture, Out-of-Distribution Generalization. 1. Introduction Since the introduction of the Deep Q-Network (DQN) (Mnih et al., 2013), (Mnih et al., 2015), Deep Reinforcement Learning (DRL) has demonstrated proficiency in solving tasks within high-dimensional visual environments. This is achieved by mapping raw pixel intensities into the latent representations re- quired for value estimation. Central to this performance is the feature extractor, typically a Convolutional Neural Network (CNN) (LeCun et al., 1998), which is crucial for the overall quality of policy optimization. The field of Computer Vision (CV) has experienced a boom in recent years, with the emergence of advanced systems, e.g., deep Residual Networks (ResNets) (He et al., 2016), automated Neural Architecture Search (NASNet) (Zoph et al., 2017), and Vision Transformers (ViTs) (Dosovitskiy et al., 2021). Although re- cent agents have integrated deeper encoders, such as the 15-layer deep Residual Networks (ResNets) used in IMPALA for multi-task environments (Espeholt et al., 2018), or the deep residual architectures in AlphaGo Zero (Silver et al., 2017), many model-free DRL efforts continue to rely on the three-layer CNN architecture established in 2013 (Mnih et al., 2013). Furthermore, the transition to continuous control tasks, exemplified by (Lil- licrap et al., 2015), highlights the necessity for robust feature extraction across 2 diverse action spaces. Similarly, the Rainbow agent (Hessel et al., 2018) im- proved performance by combining multiple algorithmic enhancements, including distributional RL (Bellemare et al., 2017) and multi-step returns, into a single framework, though largely retaining standard convolutional principles. Other recent advancements have also automated the discovery of RL rules and stabilized learning objectives in buffer-free regimes. DiscoRL introduced an autonomous method for discovering RL rules represented by meta-networks (Oh et al., 2025). Similarly, the Parallelized Q-Network (PQN) framework addressed off-policy optimization instabilities by leveraging Layer Normalization and vec- torized sampling (Gallici et al., 2024). PQN and subsequent representational adaptations like Hadamax (Kooi et al., 2025) have modernized the learning objective; the underlying convolutional backbone remains largely unexamined. This suggests further potential for architectural improvement within parallelized model-free RL. We address this gap through an empirical evaluation within the PQN frame- work, utilizing the Atari-57 benchmark suite (Bellemare et al., 2013) and vector- ized simulation via EnvPool (Weng et al., 2022). We structured our investiga- tion in three progressive phases to isolate the impact of topological design from algorithmic capacity. First, we evaluated eight parameter-constrained CNN ar- chitectures (denoted Alpha through Theta). By controlling model capacity, we isolated the structural inductive biases that affect agent generalization. While the Alpha variant achieved an Interquartile Mean (IQM) human-normalized score (HNS) of 3.536, our analysis of performance relative to model complexity identified the Gamma model (3.481 IQM HNS) as a balanced foundation, out- performing PQN (Gallici et al., 2024), which is our baseline (2.692 IQM HNS), while maintaining an efficient parameter footprint. The Gamma model inves- tigates the effect of increased hierarchical depth by introducing two additional layers compared to the PQN architecture. To control the parameter count in the encoder, channels are expanded gradually across the first three layers. This results in a model slightly larger than PQN but strictly smaller than Alpha. Based on the performance of the Gamma architecture, the second phase of 3 our study evaluated its integration with non-linear feature interaction mecha- nisms. We integrated Hadamax (Hadamard Max-Pooling) (Kooi et al., 2025) principles, evaluating Hadamax against the Gamma-Hadamax-Valid, which in- corporates the same parameter setting as the PQN-Hadamax and sets the ‘pool_padding‘ parameter to 0 to act as a ’valid’ padding, and Gamma-Hadamax- Same, which only sets the ‘pool_padding‘ parameter to 1 in two layers, thus retaining the ’same’ dimensions. Finally, we extend the Gamma-Hadamax- Valid encoder with advanced regression heads. We evaluate the integration of Distributional RL (Bellemare et al., 2017), Deep Ensembles adapted from Boot- strapped exploration (Osband et al., 2016), (Nauman et al., 2024), and Dueling architectures (Wang et al., 2016). By evaluating Gamma-Hadamax-Same with a Distributional Dueling head, an Ensemble Dueling head, and a unified Dis- tributional Ensemble Dueling head, we present an empirical analysis of sample efficiency and performance in off-policy learning. Finally, we report the results of running our final system on the Procgen Benchmark, which provides a direct measure of how quickly a reinforcement learning agent learns generalizable skills (Cobbe et al., 2019). Procgen Bench- mark consists of 16 unique environments designed to measure both sample effi- ciency and generalization in reinforcement learning. All environments support two well-calibrated difficulty settings: easy and hard. This difficulty refers to the level distribution and not to individual levels; in both settings, the difficulty of individual levels has high variance (Cobbe et al., 2019). We report results using the hard difficulty setting. While our approach achieves a median HNS of 4.42, which is inherently lower than the raw state-of-the-art scores produced by massive, replay-buffer- dependent methods like GDI (Fan & Xiao, 2022) or MuZero (Schrittwieser et al., 2020), competing on unconstrained raw performance is not the objective of this work. We position Aftab as an empirically strong buffer-free alternative for memory-constrained and high-throughput environments. Notably, while the original PQN framework utilized ℓ 2 regularization to prevent value overestima- tion, we empirically investigate whether architectural topology and normaliza- 4 tion can support stable training when weight decay is removed under a fixed Atari-57 protocol. In summary, our primary contributions are: • Systematic Encoder Evaluation: A benchmarking of eight parameter- constrained CNN topologies within PQN, identifying Gamma as a favor- able balance of computational efficiency and performance. • Hadamax Synergies: The architectural integration of the Gamma topol- ogy with Hadamard max-pooling mechanisms improves baseline represen- tational capacity. • Advanced Value-Head Ablation: An empirical evaluation of Distribu- tional, Ensemble, and Dueling paradigms applied atop a Hadamax-optimized backbone in a parallelized, replay-free setting. We name our final, composite model Aftab, Persian for sunshine, as a the- matic nod to the Rainbow agent, reflecting our strategy of unifying advanced reinforcement learning paradigms into a single entity. 2. Related Work Different neural network architectures have been applied in reinforcement learning (RL) across online settings (Mnih et al., 2013; Bellemare et al., 2017; Espeholt et al., 2018; Hessel et al., 2018; Gallici et al., 2024; Lee et al., 2024) and offline limited-data scenarios (Bhatt et al., 2019; Chen et al., 2021; Schwarzer et al., 2023). In this work, we focus on model-free agents in the Atari-57 domain, a benchmark requiring discrete action selection from raw pixel inputs. 2.1. Architectural Evolution of Convolutional Encoders The work of (Mnih et al., 2013) introduced the Deep Q-Network (DQN), an agent that learns policies directly from high-dimensional sensory inputs. By utilizing experience replay and target networks, DQN was evaluated across a 5 variety of Atari 2600 games. This architecture relied on a three-layer Convolu- tional Neural Network (CNN) feature extractor (Mnih et al., 2015). This backbone remained the standard for many subsequent algorithmic mod- ifications, including Double DQN (Van Hasselt et al., 2016), Dueling DQN (Wang et al., 2016), Categorical DQN (C51) (Bellemare et al., 2017), and the composite Rainbow agent (Hessel et al., 2018). Recent RL agents have inte- grated deeper encoders, such as the 15-layer ResNet structure utilized in IM- PALA for multi-task environments (Espeholt et al., 2018), or the deep residual architectures in AlphaGo Zero (Silver et al., 2017). However, these deeper struc- tures often introduce large parameter counts, reaching into the tens of millions. This scaling makes it difficult to determine whether performance changes are derived from specific structural choices or an increase in raw capacity. Thus, we propose to evaluate the impact of the various CNN encoders by constraining the parameter count to remain comparable to the original DQN, which is the same as PQN. 2.2. Buffer-Free Optimization and Vectorized Frameworks Historically, stabilizing value-based RL required large experience replay buffers to break the temporal correlations of sequential data and anchor moving targets. Efforts to reduce this computational overhead led to the Parallelized Q-Network (PQN) (Gallici et al., 2024). PQN stabilizes training without a target network or a large replay buffer by integrating Layer Normalization (Ba et al., 2016) and targeted ℓ 2 regularization, bounding the non-linear updates of off-policy Temporal Difference (TD) learning. Concurrently, advancements in vectorized simulation engines, such as En- vPool (Weng et al., 2022), have drastically accelerated environment sampling throughput. By leveraging these synchronous, highly parallelized environments alongside hardware-accelerated frameworks like JAX (Bradbury et al., 2018), al- gorithms such as PQN can evaluate multi-step λ-returns with computational ef- ficiency. While subsequent representational modifications, such as the Hadamax encoder (Kooi et al., 2025), have adapted PQN by replacing standard convo- 6 lutional down-sampling with Hadamard products and max-pooling, they have inherited the base CNN topological hierarchy without systematically evaluating the underlying structure. In this study, we not only assess the performance of various CNN archi- tectures against the one implemented in DQN by substituting the CNN feature extractor in the PQN framework, but also we integrate the optimal architecture, Gamma, with the Hadamax architecture to obtain Gamma-Hadamax. 2.3. Advanced Regression and Representation Synergies Beyond the visual feature extractor, the design of the regression head affects sample efficiency. Dueling networks decouple state-value estimation from action- advantage (Wang et al., 2016), while Distributional RL predicts the categorical distribution of returns rather than a single scalar expectation (Bellemare et al., 2017). Furthermore, Bootstrapped DQN utilizes multi-head architectures to approximate Thompson sampling, driving exploration via epistemic uncertainty (Osband et al., 2016). While the Rainbow agent (Hessel et al., 2018) demonstrated that these al- gorithmic extensions are complementary in a replay-buffer-driven setting, their application atop non-linear encoders (like Hadamax) within a buffer-free, paral- lelized regime remains largely unexplored. By evaluating the integration of these advanced heads, adapting bootstrapping into deep ensembles on the Gamma- Hadamax backbone, we aim to provide an empirical reference point for online, off-policy learning. 3. Preliminaries We briefly formalize the general value-based reinforcement learning paradigm, the PQN framework, and the advanced representational and regression mecha- nisms that lay the groundwork for our architectural ablations. 7 3.1. Value-Based RL and DQN We consider tasks where an agent interacts with an environment E (such as an Atari emulator (Bellemare et al., 2013)) through a sequence of actions, observations, and rewards. This interaction is modeled as a Markov Decision Process (MDP), defined by the tuple ⟨S,A,P,R,γ⟩ (Sutton & Barto, 2018). At each time step t, the agent in state s t ∈ S selects an action a t from the discrete set of legal actionsA =1,...,K. The environment transitions to the next state s t+1 ∼P(·| s t ,a t ) and emits a scalar reward r t ∼R(s t ,a t ). In pixel- based environments, the true state is partially observable; thus, the agent’s state s t is typically constructed by stacking recent raw pixel observations x t ∈ R d to capture temporal dependencies (Mnih et al., 2013). The goal of the agent is to maximize the expected discounted return R t : R t = ∞ X k=0 γ k r t+k where γ ∈ [0, 1) is the discount factor. In value-based methods, we aim to learn the optimal action-value function Q ∗ (s,a), which represents the maximum ex- pected return achievable by taking action a in state s and following the optimal policy π thereafter: Q ∗ (s,a) = max π E[R t | s t = s,a t = a,π] To handle high-dimensional visual spaces, DQN (Mnih et al., 2013) approx- imates the action-value function using a Convolutional Neural Network (CNN) parameterized by θ. The network minimizes the Mean Squared Error (MSE) between its predictions and the Temporal Difference (TD) target y i : L i (θ i ) = E s,a,r,s ′ h (y i − Q(s,a;θ i )) 2 i y i = r + γ max a ′ ∈A Q(s ′ ,a ′ ;θ i−1 ) To prevent divergence caused by the "deadly triad" of off-policy learning, function approximation, and bootstrapping (Sutton & Barto, 2018), DQN relies on two primary stabilizing mechanisms: an Experience Replay buffer to decor- relate temporal transitions, and a periodically updated Target Network (θ i−1 ) 8 to stabilize the TD targets. Following the convention in deep reinforcement learning literature, we refer to the 2015 formulation of Deep Q-Networks (Mnih et al., 2015) as ’Nature DQN’, representing the canonical baseline equipped with target networks and the standard Nature CNN architecture. 3.2. Parallelized Q-Network (PQN) PQN (Gallici et al., 2024) is an online RL algorithm that modernizes DQN by eliminating the need for target networks and large replay buffers. By leverag- ing vectorized environments, PQN parallelizes interaction and learning in syn- chronous batches, altering the stability dynamics of off-policy TD learning. Because TD updates are not exact gradients of a global objective function, they must be analyzed as stochastic approximation algorithms. The stability of the parameter vector φ is determined via the TD Jacobian: J(φ) :=∇ φ δ(φ) where δ(φ) is the expected TD update vector. The system remains stable if the Jacobian is negative definite, meaning the update acts as a strict contraction mapping: v ⊤ J(φ)v < 0, ∀v ̸= 0 where v is an arbitrary non-zero test vector. This stability criterion decomposes into Off-policy Instability (C OffPolicy ) and Nonlinear Instability (C Nonlinear ). PQN establishes stability by demonstrating that Layer Normalization (Ba et al., 2016) upper-bounds the expected norm of the activations, mitigating C OffPolicy , while increasing network width and applying targeted ℓ 2 regularization controls the curvature of C Nonlinear . To improve sample efficiency and handle credit assignment over parallel tra- jectories of length T, PQN utilizes λ-returns computed recursively backward in time: R λ t = r t + γ h λR λ t+1 + (1− λ) max a ′ Q φ (s t+1 ,a ′ ) i This target interpolates between one-step TD and Monte Carlo returns, provid- ing stable, multi-step gradient flows without a replay buffer (see Figure 1). 9 (a) Nature DQN Architecture (b) PQN Architecture Figure 1: (a) Nature DQN highlights the standard workflow: a single actor populates a Replay Buffer, and a frozen Target Network is required to calculate stable learning targets. (b) PQN demonstrates the streamlined approach: multiple Vectorized Environments provide synchronous batches directly to the Learner Q-Network, eliminating the Target Network and Replay Buffer. 3.3. Hadamax Encoder The Hadamax (Hadamard Max-Pooling) encoder (Kooi et al., 2025) aug- ments standard CNN architectures by replacing spatial convolutional strides with max-pooling and introducing multiplicative non-linear interactions (see Figure 2). It alters representational capacity without expanding hidden layer dimensionality. A Hadamax hidden layer z j is formally defined as the element-wise Hadamard product (⊙) of two parallel, normalized projections: z j = MP f LN(z j−1 W j−1 1 ) ⊙ f LN(z j−1 W j−1 2 ) where W j−1 1 and W j−1 2 are the learned weight kernels for the two parallel paths, LN denotes Layer Normalization, and MP is the Max-Pooling operator used for spatial down-sampling. Crucially, Hadamax replaces ReLU with the Gaussian Error Linear Unit (GELU) (Hendrycks & Gimpel, 2016): GELU(x) = x· 1 2 1 + erf x √ 2 10 By permitting small negative values, GELU prevents zero-saturation during the element-wise multiplication, supporting stable gradient flow within the PQN framework. (a) Nature DQN Block (b) PQN Block (c) Hadamax Block Figure 2: Evolution of Convolutional Processing Blocks. (a) The standard unnormalized DQN block relying on spatial striding. (b) The PQN block stabilized via Layer Normalization to bound off-policy activations. (c) The Hadamax block replaces spatial strides with max- pooling and utilizes parallel Hadamard multiplicative interactions with zero-saturating GELU activations. 3.4. Dueling Network Architectures The dueling architecture (Wang et al., 2016) explicitly decomposes the action- value function into a state-value function V (s) and a state-dependent action- advantage function A(s,a). The network represents two separate estimators: one for the state value function, parameterized by β, and one for the state- dependent action advantage function, parameterized by α. To ensure parameter identifiability and zero-mean advantage centering, the streams are aggregated via: Q(s,a;θ,α,β) = V (s;θ,β) + A(s,a;θ,α)− 1 |A| X a ′ ∈A A(s,a ′ ;θ,α) ! This baseline subtraction isolates the relative utility of actions, improving policy evaluation in environments with many redundant actions. Figure 3 illustrates this architecture. 11 Figure 3: Dueling Network Architecture. The shared convolutional feature extractor bifurcates into separate scalar state-value and vector action-advantage streams before aggregating into the final Q-value. This topology forms the structural backbone for our Phase 3 evaluations. 3.5. Distributional Reinforcement Learning Standard value-based RL models the expected return. Distributional RL (Bellemare et al., 2017), conversely, models the entire probability distribution of the random return, denoted by the random variable Z(s,a). The distributional Bellman equation establishes that the distribution of this return is characterized by the interaction of the immediate reward, the transition dynamics, and the next-state value distribution. This is formally defined as: Z(s,a) D = R(s,a) + γZ(s ′ ,a ) where D = denotes equality in distribution, meaning that the random variables on both sides of the equation share the same probability distribution.In the C51 algorithmic formulation, the value distribution is approximated by a dis- crete categorical distribution. This distribution is parameterized by N discrete atoms (supports) z i uniformly spaced across a defined range [V min ,V max ], with corresponding probabilities p i (s,a). The network outputs these probabilities via a softmax layer, and learning is achieved by minimizing the Kullback-Leibler (KL) divergence between the predicted distribution and the Bellman-projected target distribution, which is formed by shrinking, shifting, and projecting the target back onto the fixed support.Building upon this foundation, recent ad- vancements, most notably the "Stop Regressing" framework (Farebrother et al., 2024), have demonstrated that reframing value estimation entirely as a classifi- cation task significantly enhances scalability and mitigates non-stationarity in deep reinforcement learning. Rather than explicitly modeling the distributional 12 return via the categorical projection of C51, this architectural paradigm casts the scalar expected return directly as a classification problem. As illustrated in Figure X, our model adopts this approach by utilizing the Histogram Loss (HL-Gauss) projection. This projection models the continuous scalar Bellman target as the mean of a Gaussian distribution, which is subsequently integrated across the discrete bin widths to distribute probability mass smoothly across neighboring categorical bins. This approach replaces standard mean-squared error regression with categorical cross-entropy classification, yielding stable gra- dients, mitigating overfitting through label smoothing, and providing superior robustness against the noisy temporal-difference targets inherent to buffer-free regimes. 3.6. Bootstrapped Exploration To drive directed exploration without relying solely on ε-greedy heuristics, Bootstrapped DQN (Osband et al., 2016) approximates Thompson sampling via an ensemble of K distinct Q-network heads, sharing a common feature extractor. Each head Q k (s,a;θ,θ k ) is trained on a distinct subset of the data stream, traditionally achieved via a Bernoulli masking distribution m t ∼ Bernoulli(p). During an episode, a single head is sampled uniformly to dictate the policy, allowing the agent to execute temporally extended exploratory behaviors based on the epistemic uncertainty of the ensemble. 4. Methodology 4.1. Phase 1: Convolutional Architectures In the first phase of our study, we evaluate eight distinct Convolutional Neural Network (CNN) architectures. The objective is to systematically ex- plore how structural characteristics, such as hierarchical depth, receptive field (kernel size), and spatial downsampling (stride), affect the performance of deep reinforcement learning agents. To ensure a rigorous and controlled evaluation, all architectures are im- plemented within the PQN framework (Gallici et al., 2024). Furthermore, all 13 Figure 4: The "Regression as Classification" Optimization Pipeline. The diagram illustrates the parallel transformation of network predictions and scalar Temporal-Difference (TD) tar- gets into categorical probability distributions. On the left pathway, the active Value Network, parameterized by θ, maps a state-action pair (s,a) to continuous logits, which are subsequently converted into predicted categorical probabilities p i (s,a;θ) via a softmax operator. On the right pathway, a scalar Bellman target y, derived using the frozen target network parameters θ − , is projected onto a set of discrete support bins z i . This transformation utilizes either the Two-Hot projection (which allocates probability mass strictly to the two immediately adjacent bins) or the HL-Gauss projection (which applies a Gaussian smoothing function to distribute mass continuously across multiple neighboring bins) to construct the target prob- abilities p ∗ i (s,a;θ − ). The network is optimized iteratively by minimizing the Cross-Entropy loss between the predicted and target distributions, passing the resulting gradient updates back to θ. models process a consistent input state: a tensor of four consecutive grayscale environmental frames, stacked to form an input shape of 4× 84× 84. 4.1.1. Architectural Terminology and Baseline Definitions To facilitate clarity across varied disciplines, we explicitly define the architec- tural parameters utilized in our study. As detailed in Table 1, each convolutional layer is specified by a five-element tuple: (Input Channels, Output Channels, 14 Kernel Size, Stride, Padding). • Channels: The depth of the feature maps, dictating the network’s ca- pacity to learn independent spatial features. • Kernel Size: The spatial dimensions of the sliding filter used to extract features. • Stride: The step size of the filter across the spatial dimensions, which governs the rate of spatial dimensionality reduction. • Padding: The addition of zero-values to the spatial borders of the input tensor, utilized to control the spatial dimensions of the output feature map. Additionally, all architectures utilize two-dimensional Layer Normalization (to stabilize the learning process by normalizing inputs across the feature dimen- sion) and Rectified Linear Unit (ReLU) activations, defined mathematically as f(x) = max(0,x), to introduce non-linearity. To measure computational com- plexity, we report the number of Floating Point Operations (FLOPs) required for a single forward pass. 4.1.2. Controlling Model Capacity Our experimental design isolates structural differences from overall capacity scaling by monitoring model complexity across two interdependent dimensions: • Encoder Parameters: The weights contained strictly within the con- volutional feature extractor. To establish a fair baseline, we artificially constrain the encoder parameter count in the Beta, Delta, Epsilon, Eta, Zeta, and Theta variants to closely match that of the baseline PQN model (≈ 78k parameters). • Regression Head Parameters: The weights in the final fully-connected layers responsible for value estimation. The footprint of this head is inex- tricably linked to the encoder’s output spatial resolution. Variants utiliz- ing fewer strides or smaller kernels output larger flattened feature maps, 15 consequently increasing the number of trainable weights in the regression head. As summarized in Table 2, the total parameter count is largely governed by this final CNN feature map size. By partitioning the parameter counts and FLOPs between the encoder and the regression head, we can effectively discern whether empirical performance changes are driven by the network’s total capacity or by its specific spatial configuration. 4.1.3. Evaluation Suite Specifications We design eight variants (Alpha through Theta) to test specific structural hypotheses against the PQN baseline. The baseline features three convolu- tional layers, culminating in a flattened feature vector of size 3,136 and a total parameter count of ≈ 1.76M. Alpha closely mirrors the baseline’s initial channel counts but adds a fourth layer with 64 channels. This nearly doubles the encoder’s parameter count to 174,752, testing the effect of increased representational capacity while yielding a smaller flattened feature vector of size 2,304. Conversely, Beta maintains a four-layer depth but introduces a representational bottleneck by decreasing the channel dimensions in the third and fourth layers. This bottleneck reduces the encoder size relative to Alpha, outputting a vector size of 3,136. To investigate the effects of hierarchical depth, Gamma extends the archi- tecture to five layers. To strictly control the encoder parameter count (≈ 117k), the channel dimensions are expanded gradually. The network topology for this variant is visualized in Figure 5. Exploring depth under tighter constraints, Epsilon utilizes a similar structure but is truncated to four layers, matching the ≈ 78k baseline encoder constraint. However, this configuration results in a larger output vector (size 4,096), pushing the total parameter count to≈ 2.26M. To assess spatial receptive fields, Delta utilizes a massive 9× 9 kernel in the initial layer. Constrained to three layers, this model maintains the baseline encoder footprint while producing a feature vector of size 3,456. Zeta further explores variations within a strict three-layer constraint; it is the second smallest 16 variant in terms of encoder parameters (≈ 77k) but yields the second largest total parameter count (≈ 2.61M) due to a flattened feature vector of size 4,800. Finally, we test structural extremes with Eta and Theta. Eta is the shal- lowest model, employing only two layers but massively expanding the channel width. This spatial preservation results in a vast feature vector of size 46,208, intentionally driving the total parameter count to ≈ 23.8M to test whether a shallow, high-capacity network can outcompete deeper, constrained architec- tures. In contrast, Theta represents the most compact holistic model (≈ 1.2M total parameters). By halving the output channel dimensionality in the final layer relative to the PQN baseline, it tests the network’s resilience to extreme feature compression (vector size 2,048). Table 1: Architectural specifications of the CNN feature extractor variants. Each layer is defined by the tuple (Input Channels, Output Channels, Kernel Size, Stride, Padding). All architectures apply two-dimensional Layer Normalization and ReLU activations after each convolution. VariantL1L2L3L4L5 PQN(4, 32, 8, 4, 0) (32, 64, 4, 2, 0) (64, 64, 3, 1, 0)– Alpha(4, 32, 4, 2, 1) (32, 64, 4, 2, 1) (64, 64, 3, 2, 1) (64, 64, 5, 1, 0)– Beta(4, 32, 6, 2, 2) (32, 64, 3, 1, 1) (64, 32, 4, 2, 1) (32, 16, 8, 1, 0)– Gamma (4, 32, 3, 2, 1) (32, 48, 3, 2, 1) (48, 64, 3, 1, 0) (64, 64, 3, 2, 0) (64, 64, 3, 1, 0) Delta(4, 24, 9, 4, 0) (24, 48, 5, 2, 0) (48, 96, 3, 1, 0)– Epsilon (4, 32, 3, 2, 1) (32, 48, 3, 2, 1) (48, 64, 3, 2, 0) (64, 64, 3, 1, 0)– Zeta(4, 48, 4, 2, 1) (48, 48, 4, 2, 1) (48, 48, 4, 2, 1)– Eta(4, 64, 4, 4, 0) (64, 128, 3, 1, 0)– Theta(4, 32, 7, 4, 2) (32, 64, 5, 2, 1) (64, 32, 3, 1, 0)– 17 Table 2: Model complexity and computational footprint of Phase 1 architectures. Floating Point Operations (FLOPs) are reported in millions. The Eta variant exhibits a substantial parameter and computation increase driven by the massive spatial dimensions of the encoder’s output. Variant Enc. Params Head Params Total Params Enc. FLOPs Head FLOPs Total FLOPs PQN78,3041,686,5001,764,8047.7341.6109.347 Alpha174,7521,782,9481,957,70027.5411.61029.151 Beta89,0081,782,9481,871,95661.5151.61063.126 Gamma117,1681,725,3641,842,53222.9011.61024.512 Delta78,5521,850,5881,929,1406.1431.7747.917 Epsilon80,1122,179,8282,259,94013.2522.10115.354 Zeta77,2322,537,3962,614,62825.3622.46227.824 Eta78,40023,739,46023,817,86028.42223.66352.085 Theta76,2881,127,4281,203,7169.0651.05310.118 18 (a) DQN Baseline (b) PQN Baseline (c) Gamma (Ours) (d) Gamma-Hadamax (Ours) Figure 5: Topological Evolution of Evaluated Encoders. A structural comparison of the core architectures evaluated in this study. The sequence illustrates the progression from the legacy, unnormalized 3-layer DQN (a), to the stabilized PQN baseline (b), to our 5-layer Gamma topology (c), and finally the multiplicative Gamma-Hadamax V1 architecture (d). 19 4.2. Phase 2: Hadamax Integration Following the evaluation of the base topologies in Phase 1, we extend our in- vestigation by integrating the Hadamax representation mechanism (Kooi et al., 2025) into our optimal model, the Gamma architecture. The objective of this phase is twofold: first, to evaluate the relationship between deep hierarchi- cal topologies and multiplicative feature interactions; and second, to determine whether the representational advantages of the Hadamax paradigm can be pre- served while reducing its parameter footprint through the new backbone. A detailed review of the theoretical foundations of Hadamax can be found in Section 3. To ensure clarity within the context of our architectures, we briefly define the core operations introduced in this phase: • Hadamard Product: An element-wise multiplication of two tensors of the same dimensions, used here to foster non-linear feature interactions. • Max-Pooling: A spatial down-sampling operation that extracts the max- imum value within a defined moving window (kernel), enhancing the net- work’s translational invariance. • GELU (Gaussian Error Linear Unit): An activation function that weights inputs by their cumulative Gaussian distribution. Unlike ReLU, which sharply truncates negative values, GELU provides a smoother, non- monotonic nonlinearity. Instead of standard spatial down-sampling via strided convolutions, the Hadamax block relies on a stride of 1 for its convolutional layers, subsequently delegating spatial compression to the max-pooling operation. We formulate three specific test configurations (detailed in Table 3) to rigorously analyze this mechanism: 1. Hadamax (Baseline): This variant serves as the control model for this phase. It mimics the classic three-layer DQN topology but utilizes Hadamax blocks. It employs large initial convolutional kernels (8 × 8) and aggressive max-pooling (4× 4 kernel with a stride of 4) in the first 20 layer to rapidly down-sample the input space. Note that Hadamax is also seamlessly integrated into the PQN framework. 2. Gamma-Hadamax-Valid: This model modifies the core five-layer Gamma architecture evaluated in Phase 1 (Figure 5). It substitutes standard con- volutional down-sampling with Hadamax blocks. To maintain the deep topology without prematurely collapsing the spatial dimensions, layers 3 and 5 utilize a pooling kernel of 3, a stride of 1, and no padding (0). This "valid" padding configuration slightly reduces the spatial dimensions while avoiding aggressive compression. 3. Gamma-Hadamax-Same: This variant acts as an architectural abla- tion of the previous model. It is identical in its convolutional parameters but alters the pooling hierarchy. Specifically, it introduces a padding of 1 to the pooling operations in layers 3 and 5. This "same" padding pre- serves the spatial resolution across these specific layers, resulting in a larger flattened feature map. This allows us to analyze the Floating Point Oper- ations (FLOPs)-to-performance ratio and determine if preserving spatial dimensions through padding yields a proportional increase in predictive accuracy. 4.3. Phase 3: Advanced Regression Head Topologies To further evaluate the robustness and expressivity of the latent representa- tions generated by the Gamma-Hadamax-Valid encoder, the final phase of our methodology examines three advanced regression paradigms. While these algo- rithmic enhancements traditionally rely on experience replay buffers to stabilize training (Hessel et al., 2018), we reconstruct them to operate entirely on the synchronous, buffer-free data stream provided by the PQN framework. A detailed review of the theoretical foundations of these topologies can be found in Section 3. To ensure clarity within this context, we briefly introduce them in this section. All regression heads process the flattened, normalized feature vectors output by the encoder. To ensure stable gradient flow, the fully-connected streams 21 Table 3: Architectural specifications of the Hadamax integration variants. To detail both operations within a single block, each variant is split into a Convolutional (Conv) row and a Max-Pooling (Pool) row. Conv is defined by (Input Channels, Output Channels, Kernel Size, Stride, Padding), and Pool is defined by [Kernel Size, Stride, Padding]. All blocks utilize GELU activations. VariantBlock 1Block 2Block 3Block 4Block 5 Hadamax (Baseline) Convolutinal(4, 32, 8, 1, 4) (32, 64, 4, 1, 2) (64, 64, 3, 1, 1)– Max Pool[4, 4, 0][2, 2, 0][3, 1, 1]– Gamma-Hadamax-Valid Convolutinal(4, 32, 3, 1, 1) (32, 48, 3, 1, 1) (48, 64, 3, 1, 1) (64, 64, 3, 1, 1) (64, 64, 3, 1, 1) Max Pool[2, 2, 0][2, 2, 0][3, 1, 0][2, 2, 0][3, 1, 0] Gamma-Hadamax-Same Convolutinal(4, 32, 3, 1, 1) (32, 48, 3, 1, 1) (48, 64, 3, 1, 1) (64, 64, 3, 1, 1) (64, 64, 3, 1, 1) Max Pool[2, 2, 0][2, 2, 0][3, 1, 1][2, 2, 0][3, 1, 1] within these heads consistently apply Layer Normalization prior to their final outputs. We evaluate the following three configurations: • Distributional Dueling Head: This configuration integrates the bifur- cated Dueling architecture (Wang et al., 2016) with Distributional value estimation (Bellemare et al., 2017). – Dueling Mechanism: Instead of directly estimating the action-value function (Q), the network splits into two independent streams: one estimating the scalar state-value (V ) and the other estimating the state-dependent action advantages (A). To resolve the issue of uniden- tifiability, the mean of the advantage stream is subtracted from its outputs before aggregating it with the value stream. – Distributional Mechanism: Rather than predicting a scalar expected return, the value stream outputs a probability distribution over N = 51 discrete atoms, spaced uniformly between V min = −10.0 and V max = 10.0. Concurrently, the advantage stream outputs |A| dis- tinct 51-atom distributions. The dueling mean-subtraction is per- formed strictly at the logit level. Temporal Difference (TD) tar- 22 gets are computed using λ-returns (with rewards clipped to [−1, 1]) and mapped onto the discrete support using Two-Hot Gaussian (HL- Gauss) projection to smooth the target distributions. For action selection, the greedy policy evaluates the expected value of the re- sulting state-action distribution. • Ensemble Dueling Head: To evaluate directed epistemic exploration (exploration driven by model uncertainty) without a replay buffer, we adapted the multi-head architecture of Bootstrapped DQN (Osband et al., 2016) into a Deep Ensemble (Nauman et al., 2024). The shared Gamma- Hadamax-Valid feature extractor feeds into an ensemble of K = 10 in- dependent dueling networks (heads). Crucially, the primary objective of this ensemble architecture is not to yield direct performance gains through predictive model averaging; rather, it is explicitly designed to drive deep exploration. Unlike traditional bootstrapped approaches that require strict Bernoulli data masking (e.g., dropping transitions with prob- ability p = 0.5), our framework utilizes a mask probability of p = 1.0. Con- sequently, every head observes the entirety of the synchronous PQN batch. Ensemble diversity is strictly maintained through independent random weight initializations and the natural trajectory variance generated across the 128 parallel EnvPool environments. Within a batch, all 10 heads are updated simultaneously using a single global optimizer state. This explo- ration mechanism works by approximating Thompson sampling. During environmental interaction, each parallel environment uniformly samples its own active head index at the start of an episode to dictate the policy. Because each independently initialized head, comprising separate value and advantage streams, holds slightly divergent Q-value estimates, the agent can commit to diverse, temporally extended exploratory behaviors for the duration of the episode. This enables the agent to explore based on the epistemic uncertainty of the ensemble, circumventing the limitations of relying solely on random, step-by-step ε-greedy heuristics. Finally, when 23 exploration is disabled during final policy evaluation, actions are selected via expected value aggregation by computing the arithmetic mean of the Q-values across all K heads. • Distributional Ensemble Dueling Head: This unified architecture represents our most complex value-estimation topology, combining Dis- tributional RL with the Deep Ensemble formulation. The single shared encoder feeds into an ensemble of K = 10 independent heads, where each individual head is a complete Distributional Dueling module outputting |A| categorical distributions of size N = 51. To manage the high dimen- sionality of the parameter space, this variant specifically employs GELU non-linearities within its dense streams. This configuration allows us to rigorously evaluate whether these composite, heavily parameterized al- gorithms remain stable and sample-efficient when constrained strictly to online, parallelized TD-learning. We designate this final, composite ar- chitecture as Aftab, Persian for sunshine, serving as a thematic nod to the Rainbow agent, reflecting our overarching strategy of unifying these advanced reinforcement learning paradigms into a single, cohesive entity. 4.4. Training Details and Optimization Stability To ensure a consistent comparative analysis, all architectural variants across the three phases were trained using a unified set of hyperparameters, adapted from the original PQN framework and summarized in Table 4. We utilized the Rectified Adam (RAdam) optimizer (Liu et al., 2020) with a learning rate of 2.5 × 10 −4 . Each variant was trained for 50 million environmental steps per parallel environment; with a standard frame-skip of 4, this equates to 200 million observed frames. To address the non-stationary data distributions inherent in buffer-free re- inforcement learning, we apply LayerNorm2d (Gallici et al., 2024) after every convolutional operation and prior to the non-linear activation function (ReLU in Phase 1, GELU in Phases 2 and 3). This explicitly normalizes the expected 24 norm of the activations, thereby stabilizing the network against input distribu- tional shifts. The optimization objective minimized the discrepancy between the predicted action-values and the computed λ-return targets. We utilized Mean Squared Error (MSE) for Phases 1 and 2, and the Two-Hot Gaussian (HL-Gauss) loss for the distributional variants in Phase 3. Temporal targets were calculated utilizing a discount factor (γ) of 0.99 and a Generalized Advantage Estimation (GAE) λ of 0.65. Consistent with standard deep RL practices (Mnih et al., 2013), rewards were clipped to the range [−1, 1], and episodic life-loss termination was employed to prevent the optimization of sub-optimal survival strategies. All experiments were highly parallelized on a high-performance computing cluster utilizing the EnvPool vectorized simulation library (Weng et al., 2022), maintaining 128 parallel training environments and a constant global batch size of 4096. 4.5. Deviation from Standard PQN Regularization Our experimental protocol introduces a critical deviation from the standard PQN implementation (Gallici et al., 2024). Whereas the original framework employs explicit ℓ 2 regularization (weight decay) to help mitigate value over- estimation and nonlinear instability in buffer-free temporal-difference learning, we set the weight decay parameter to exactly 0 across all experimental phases. This design choice is intentional: it explicitly isolates the empirical impact of our architectural topologies and multiplicative feature interactions from the damp- ening effects of parameter regularization. We emphasize that this modification is explored purely empirically. We do not claim a formal convergence guarantee, nor do we assert a global bound on the Temporal Difference (TD) Jacobian for the resulting unregularized, nonlinear updates. However, our experimental results demonstrate that the combination of vectorized sampling, Layer Normalization, and our proposed architectural designs maintains operational stability across all tested random seeds on the Atari-57 benchmark. Consequently, the removal of ℓ 2 regularization serves as 25 a controlled experimental condition rather than a theoretically guaranteed sub- stitute for the regularization mechanism utilized in the original analysis. 5. Experimental Setup and Evaluation Metrics We evaluate the proposed architectural variants and regression paradigms by comparing their performance against the baseline PQN model. To provide a rigorous evaluation, our primary analysis utilizes the full Atari-57 benchmark suite to measure sample efficiency and representation quality. To further stress- test the out-of-distribution generalization capabilities of our unified framework (Aftab), we complement our evaluation with experiments conducted on the chal- lenging Procgen benchmark suite under its Hard difficulty setting. The complete Aftab framework, including all model definitions, training con- figurations, and raw experimental logs, is open-sourced and available on our GitHub repository: https://github.com/tahashieenavaz/aftab or https: //github.com/LorisNanni/Aftab. Gameplay videos are specifically indexed at https://github.com/tahashieenavaz/aftab/blob/main/videos.md 5.1. Atari-57 Evaluation Protocol To ensure our results are robust and directly comparable to the broader reinforcement learning literature, we strictly adhere to the standardized Atari- 57 evaluation protocol. We report performance using the Human-Normalized Score (HNS), which scales the raw agent performance relative to a random baseline and a professional human tester. The HNS is calculated as: HNS = Agent Score− Random Score Human Score− Random Score (1) The reference scores for the random agent and the professional human player are adopted from the standard baseline provided by (Mnih et al., 2015). To aggregate agent performance robustly across the 57 games of the bench- mark suite, we employ the Interquartile Mean (IQM) of the HNS. Unlike the 26 standard mean, which can be easily skewed by extreme outlier scores in a hand- ful of environments, or the median, which can be overly sensitive to minor per- formance shifts in middle-ranking games, the IQM serves as a middle-ground statistical estimator that discards the lowest 25% and highest 25% of the perfor- mance distribution. Formally, given a sorted sequence of M human-normalized scores x 1 ≤ x 2 ≤·≤ x M , the IQM is defined mathematically as: IQM = 1 ⌊0.75M⌋−⌈0.25M⌉ + 1 ⌊0.75M⌋ X i=⌈0.25M⌉ x i (2) By focusing entirely on the interquartile range, this metric suppresses the vari- ance of outlier tasks while providing a statistically stable reflection of overall algorithmic improvements. To complement the Interquartile Mean (IQM) and provide a robust, proba- bilistic assessment of algorithmic superiority, we incorporate the Probability of Improvement metric. Unlike point estimates which can be skewed by outlier en- vironments, the Probability of Improvement, denoted as P(X > Y ), explicitly conveys the likelihood that a randomly sampled performance from algorithm X (Row) will strictly exceed a randomly sampled performance from algorithm Y (Column) across the benchmark distribution. A probability greater than 0.50 indicates that the row algorithm is more likely than not to outperform the column algorithm. This metric provides a highly stable, pairwise statistical comparison that is entirely insensitive to the magnitude of performance spikes in individual tasks, thereby isolating true algorithmic consistency. During training, we utilize life-loss termination; the environment trajectory is explicitly terminated upon the loss of an agent’s life to prevent the optimiza- tion of sub-optimal, passive survival strategies. However, during the final policy evaluation, life-loss termination is strictly disabled. For the final evaluation, agents are tested over 100 independent episodes per environment, per random seed. To prevent deterministic overfitting to specific trajectories, we inject stochasticity using up to 30 random no-op starts at the beginning of each episode. Furthermore, we employ sticky actions with a repeat 27 probability of 25%, ensuring the agent’s learned policy is robust to the frame- level action stochasticity inherent in standard Atari-57 benchmarking. 5.2. Procgen Hard Generalization Protocol To evaluate the structural resilience and generalization capacity of our inte- grated architecture against procedural asset and layout variations, we execute an additional suite of evaluations across the 16 distinct environments of the Proc- gen benchmark under the Hard difficulty configuration. This suite includes: Bigfish, Bossfight, Caveflyer, Chaser, Climber, Coinrun, Dodgeball, Fruitbot, Heist, Jumper, Leaper, Maze, Miner, Ninja, Plunder, and Starpilot. Unlike the deterministic asset structures found in Atari-57, Procgen dynam- ically synthesizes level layouts, backgrounds, and enemy behaviors for every individual episode. This protocol evaluates whether the representation mech- anisms learned by our system generalize effectively to unseen environmental distributions or merely overfit to fixed pixel configurations. Because Procgen lacks official human baseline standards to compute a stan- dard Human-Normalized Score, we define and calculate the Procgen Normal- ized Score (PNS). The PNS is computed via a Min-Max Normalization of raw agent scores against the minimum and maximum performance limits observed across our experimental suite: PNS = Agent Score− Score min Score max − Score min (3) Consistent with our Atari evaluation paradigm, raw scores across the 16 tasks are aggregated using both the Median and the Interquartile Mean (IQM) along- side their corresponding 95% confidence intervals (CI) to offer a statistically stable comparison of generalization performance under severe environmental non-stationarity. Per-game raw data and full step histories for these Procgen runs are provided in full within the Supplementary Material (Tables 3.S4 and 4.S4). 28 Table 4: Hyperparameter settings utilized across all experimental phases. Phase 3 introduces specific modifications to the loss function to accommodate the advanced distributional regres- sion heads. HyperparameterValue Learning rate2.5× 10 −4 Training environments (Atari-57) 128 Training environments (Procgen) 64 Steps per update (Procgen)256 Test environments8 OptimizerRectified Adam (Liu et al., 2020) Weight decay (ℓ 2 penalty)0 Adam ε1× 10 −5 Adam β 1 0.9 Adam β 2 0.999 Total Frames200,000,000 Loss Function (Phases 1 & 2) Mean Squared Error Loss Function (Phase 3)HL-Gauss (Two-Hot Gaussian) SchedulerLinear Annealing ε-greedy exploration10% of total frames Discount factor (γ)0.99 λ-return parameter0.65 Epochs2 Batch size4096 5.3. Hardware and Reproducibility Due to the inherent stochasticity of deep reinforcement learning, exact re- producibility via fixed datasets is not feasible. Therefore, we ensure statistical reliability by evaluating all architectures across four fixed random seeds: 1, 2, 3, and 4. 29 Crucially, to provide a standardized measure of sample efficiency and termi- nal performance, every experiment across all three phases was strictly allocated a training budget of 200 million environmental frames (equivalent to 50 mil- lion optimization steps with a standard frame-skip of 4). All experiments were executed on a high-performance computing cluster utilizing Nvidia A40 GPUs. Each GPU is equipped with 48 GB of GDDR6 memory with error-correcting code (ECC), providing a memory bandwidth of 696 GB/s and operating under a maximum power consumption of 300 W. Training the most complex architec- tural configuration requires approximately 13 hours of wall-clock time using the PyTorch (Paszke et al., 2019) framework. A JAX-based (Bradbury et al., 2018) implementation is currently under development to further accelerate distributed training. 6. Results 6.1. Phase 1: CNN Encoder Capacity and Topological Efficiency The initial phase of our experimental protocol isolated the structural design of the convolutional feature extractor to evaluate its impact on agent perfor- mance. As previously detailed in Table 2, the computational footprint of each variant was strictly partitioned between the encoder and the regression head. This complexity profiling highlights the critical relationship between ar- chitectural depth, spatial reduction, and computational bloat. For instance, the Eta variant illustrates the risks of spatial overparameterization; it requires roughly 23.8 million total parameters strictly because its shallow encoder projects an excessively large feature map into the dense regression head. In contrast, the Gamma variant maintains a highly balanced footprint, requiring only 24.5 mil- lion total FLOPs and 1.84 million parameters. To rigorously validate the empirical performance differences associated with these architectures, we conducted pairwise statistical evaluations, visualized in the significance matrix in Figure 6. Furthermore, the complete un-normalized evaluation returns and full operational step histories for all Phase 1 variants, 30 including the baseline PQN and the Alpha through Theta architectures, are provided in Table S1 and Table S5 of the Supplementary Material. Tables 5 and 6 summarize the aggregated performance of the evaluated networks using the Interquartile Mean (IQM) and Median of the Human-Normalized Scores (HNS). To mitigate the risk of overfitting during the architectural selection process, we executed a robustness validation study. We evaluated the generalizability of our selection protocol by repeatedly splitting the Atari-57 benchmark into random validation subsets of 15 games across 10 independent trials. Our selec- tion criterion did not rely exclusively on maximizing the aggregate IQM HNS; rather, it emphasized a strict performance-complexity tradeoff. Among the ar- chitectures demonstrating statistically comparable validation performance, we selected the model with the lowest total parameter count and computational cost. This criterion justifies the selection of the Gamma architecture over the Al- pha variant, despite Alpha achieving the highest aggregate IQM HNS in Phase 1 (3.536 vs. Gamma’s 3.481). While Alpha presented a marginal absolute improvement of 0.055 IQM, Gamma required fewer total parameters (1.84M compared to 1.96M) and lower computational overhead (24.5 vs. 29.1 million FLOPs). Crucially, a pairwise Wilcoxon signed-rank comparison between Al- pha and Gamma revealed no statistically significant difference in performance following a Holm-Bonferroni correction (p > 0.05). Consequently, Gamma was designated as the optimal backbone: it delivers statistically equivalent empirical performance while utilizing a deeper, more computationally efficient convolu- tional hierarchy. Across all 10 randomized validation splits, this selection rule consistently fa- vored the Gamma architecture. This high degree of consistency verifies that the selection of the Gamma encoder reflects a stable performance-efficiency trade- off, rather than a statistical anomaly or an artifact of overfitting to specific environmental dynamics. To further validate the empirical superiority of the deeper topological vari- 31 (a) Pairwise Wilcoxon Signed-Rank Matrix(b) Holm-Bonferroni Corrected Matrix Figure 6: Phase 1 Statistical Significance Matrices. Significance heatmaps generated via pairwise Wilcoxon Signed-Rank Tests across the Atari-57 suite for the baseline convo- lutional architectures. To maintain statistical integrity across the 36 pairwise comparisons, we applied the Holm-Bonferroni correction to control the family-wise error rate at α = 0.05. The light/pale cells (p < 0.05) indicate that the modernized variants—specifically the Alpha, Beta, and Gamma cluster—exhibit a statistically significant performance advantage over the PQN baseline. Conversely, the deep red cells (p > 0.05) indicate pairs where the performance differences are not statistically significant, such as between Alpha and Beta (p = 0.847) or Zeta and Eta (p = 0.967). ants, we evaluated the pairwise Probability of Improvement across the evaluated encoders (Figure 7). The analysis robustly confirms the advantages of hierar- chical depth over the baseline architecture. Specifically, the Alpha, Beta, and Gamma variants demonstrate high probabilities of improvement over the PQN baseline, yielding scores of 0.85, 0.81, and 0.79, respectively. Conversely, archi- tectures exhibiting spatial bottlenecks or over-extended kernels failed to achieve consistent superiority; the Delta variant, utilizing a large 9× 9 initial kernel, yielded a severely degraded improvement probability of 0.34 against PQN, while the Theta variant achieved exactly 0.50, indicating no probabilistic advantage over the baseline. To mechanically explain the performance improvement yielded by the inte- gration of Hadamax, we examine the expansion of the network’s effective recep- 32 Figure 7: Probability of Improvement (Encoders). A pairwise probability matrix, P(Row > Col), evaluating the baseline convolutional architectures across the Atari-57 suite. Values greater than 0.50 indicate that the row architecture is statistically more likely to outperform the column architecture. The deep topological variants (Alpha, Beta, and Gamma) consis- tently demonstrate a high probability of improvement (0.79 to 0.85) over the standard PQN baseline. tive field (RF)(Luo et al., 2017). The receptive field calculates the spatial region of the input volume that influences a single feature map activation(Araujo et al., 2019). Assuming an initial receptive field RF 0 = 1 and an initial stride product of 1, the receptive field at layer l is defined recursively based on the kernel size (k) and stride (s) of the current and preceding layers: RF l = RF l−1 + (k l − 1) l−1 Y i=1 s i While the standard Gamma encoder preserves spatial resolution through a deep stack of small convolutions, its maximum effective receptive field maps to a 39x39 spatial patch on the original input. By substituting standard spa- 33 tial convolutions with explicit, decoupled max-pooling operations, the Gamma- Hadamax architecture (both the Valid and Same variants) alters the downsam- pling hierarchy. This decoupling expands the total effective receptive field to a 70x70 spatial patch. This expansion of the visual region processed by a single feature map provides the theoretical mechanism behind the superior Interquar- tile Mean (IQM) Human-Normalized Scores (HNS) achieved by the Gamma- Hadamax-Valid model. By pairing multiplicative feature interactions with a 70x70 receptive field, the agent captures far broader global spatial dependencies across the environment without prematurely collapsing the state representation or requiring an explosion in parameter count. A more detailed description of this analysis is available in the Supplementary Materials. 34 Table 5: Full Phase 1 Encoder Results (Part 1: PQN to Delta). PQNAlphaBetaGammaDelta Alien0.5421.7852.4082.8450.381 Amidar0.6401.3460.8621.0380.529 Assault30.16924.54728.574 35.62924.864 Asterix40.61715.33814.82114.31138.146 Asteroids0.0341.8280.3491.4380.223 Atlantis47.17243.48244.53246.33642.942 Bank Heist1.8852.0012.1121.7211.836 Battlezone1.2121.4251.8941.8801.197 Beamrider1.1442.9732.3552.3881.069 Berzerk2.2514.6760.3431.2271.364 Bowling0.0440.1000.0370.1330.075 Boxing8.2448.3258.2978.2858.325 Breakout12.01915.04515.71216.76311.809 Centipede0.8281.3000.9901.4730.693 Chopper Command2.67924.38047.179 55.5060.839 Crazy Climber6.2686.5947.7676.4676.786 Defender3.1734.5664.3165.8613.155 Demon Attack72.52372.842 73.15172.88470.913 Double Dunk7.7637.7837.9607.8047.571 Enduro2.7232.7372.7092.6932.724 Fishing Derby2.5252.6232.5452.5902.459 Freeway1.1321.1481.1401.1121.136 Frostbite1.5452.5642.3182.1721.179 Gopher24.50625.816 33.98029.68217.045 Gravitar0.2350.2670.3960.3530.188 H.E.R.O.0.7820.8490.7540.7190.694 Ice Hockey0.8501.1831.0651.8900.703 James Bond8.13412.2796.87517.1274.725 Kangaroo4.4804.8014.7734.6234.068 Krull7.6218.6258.7078.4747.064 Kung-Fu Master1.4151.7761.4871.3771.579 Montezuma’s Revenge0.0000.0000.0000.0000.000 Ms. Pac-Man0.4540.6810.9520.8120.515 Name This Game2.2782.8252.5613.2402.322 Phoenix24.84041.93038.20333.4527.793 Pitfall!0.0320.0330.0260.0330.034 Pong1.1811.1811.1811.1811.181 Private Eye0.0120.0000.0010.001-0.000 Q*bert1.5721.8511.8741.8161.332 River Raid1.3551.6721.6691.7371.405 Road Runner7.26810.36210.011 21.1887.391 Robotank7.1097.4357.2546.9607.263 Seaquest0.1870.2350.1940.2010.198 Skiing-0.5810.457-0.179-0.393-0.500 Solaris0.1110.0740.1530.1900.077 Space Invaders4.8413.9298.8195.2664.504 Stargunner27.27838.522 43.75142.47824.829 Surround1.0691.1391.1611.1230.928 Tennis1.3812.2901.3752.6761.459 Time Pilot5.90115.564 20.21014.6574.866 Tutankham1.5191.5711.5571.5381.512 Up’n Down23.23123.35323.54524.04616.812 Venture0.0000.0000.0180.0010.000 Video Pinball315.049 365.167 312.556 366.645 359.358 Wizard of Wor4.4437.3197.2546.4683.191 Yars’ Revenge2.2222.6322.5592.5501.820 Zaxxon1.8342.1092.5392.3061.739 Median1.8852.6232.4082.5501.579 IQM2.6923.5363.4723.4812.374 35 Table 6: Full Phase 1 Encoder Results (Part 2: Epsilon to Theta). EpsilonZetaEtaTheta Alien2.2621.8401.6750.411 Amidar1.3810.7261.0230.665 Assault34.12429.91334.28835.067 Asterix10.25512.61622.73038.306 Asteroids1.0391.7520.0640.026 Atlantis47.55745.96444.95146.165 Bank Heist1.6992.1311.7181.949 Battlezone2.1211.5381.0671.073 Beamrider1.9971.5911.3481.152 Berzerk1.4752.9840.8681.368 Bowling0.0590.0920.1130.056 Boxing8.3248.3108.3258.319 Breakout12.69512.87017.24614.332 Centipede0.9550.6651.1200.540 Chopper Command40.92231.4913.3871.515 Crazy Climber7.1756.6456.8396.152 Defender3.8763.9673.2595.806 Demon Attack72.62772.07571.49670.971 Double Dunk7.8697.7328.1717.605 Enduro2.7112.6972.6962.710 Fishing Derby2.6522.6552.5592.509 Freeway1.1371.1401.1401.136 Frostbite1.4942.1081.8041.075 Gopher21.29124.04131.58318.757 Gravitar0.2550.3280.5100.118 H.E.R.O.0.7880.8831.0690.448 Ice Hockey1.0501.0561.2530.631 James Bond28.24312.08113.4398.514 Kangaroo4.5844.6884.5184.652 Krull8.1178.6788.0087.655 Kung-Fu Master1.4121.5171.4781.791 Montezuma’s Revenge0.0000.0020.0040.000 Ms. Pac-Man0.7450.8011.1120.657 Name This Game2.2752.2511.8772.579 Phoenix29.553 42.58938.63813.875 Pitfall!0.0330.0340.0330.030 Pong1.1811.1811.1811.181 Private Eye0.0000.000-0.0000.001 Q*bert1.7621.8601.8811.647 River Raid1.7311.8191.4951.378 Road Runner10.82511.0947.8517.007 Robotank7.1857.3696.5776.855 Seaquest0.2100.4070.4090.192 Skiing0.5540.4960.541-0.388 Solaris0.1330.1170.1800.282 Space Invaders5.0611.73115.3334.159 Stargunner32.67131.59124.52224.052 Surround1.1411.1981.1970.946 Tennis2.2661.3631.8751.402 Time Pilot12.50812.86414.8774.423 Tutankham1.4981.5361.6011.543 Up’n Down27.409 27.9817.87724.057 Venture0.0000.0000.0000.000 Video Pinball302.946 365.137 372.327 344.773 Wizard of Wor7.7426.2668.0595.384 Yars’ Revenge2.3732.7222.6532.198 Zaxxon1.6171.7272.2671.904 Median2.2622.1081.8771.791 IQM3.3153.2073.1142.649 36 6.2. Phase 2: Synergies of Hadamard Encoded Representations Following the evaluation of the baseline topological characteristics in Phase 1, we integrated Hadamax multiplicative interactions to assess the synergistic effects of non-linear representation mechanisms within our optimized hierar- chy. Table 7 details the computational shifts resulting from these architectural modifications. Table 7: Phase 2 Results: The impact of Hadamard max-pooling integration on model com- plexity and computational footprint. Floating Point Operations (FLOPs) are reported in millions. VariantEnc. Params Head Params Total Params Enc. FLOPs Head FLOPs Total FLOPs Gamma117,1681,725,3641,842,53222.9011.61024.512 Hadamax (Baseline)156,6083,968,5164,125,124159.0143.969162.984 Gamma-Hadamax-Valid234,3361,609,2201,843,556122.0011.610123.611 Gamma-Hadamax-Same234,3363,280,3883,514,724129.3003.281132.581 Our complexity profiling demonstrates that the Gamma topology integrates highly efficiently with Hadamard representations. Specifically, the Gamma- Hadamax-Valid variant maintains a strictly constrained footprint, requiring only 1,843,556 total parameters and 123.611 million total FLOPs. In stark contrast, integrating the baseline Hadamax architecture results in severe computational bloat, expanding the model to 4,125,124 parameters and consuming 162.984 million total FLOPs. To rigorously evaluate the performance impact of these representational shifts, we conducted pairwise statistical analyses, illustrated in the significance matrix in Figure 8. For comprehensive transparency, the un-normalized raw evaluation returns and full operational step histories for the Hadamax variants are documented in Table 3.S2 of the Supplementary Material and Table 8. The Wilcoxon signed-rank tests reveal that the Gamma-Hadamax-Valid model provides a statistically significant performance improvement over the baseline Gamma architecture (p < 0.001). Furthermore, we analyzed the architectural ablation, Gamma-Hadamax-Same, which artificially preserves spatial resolution through padding. Despite Gamma-Hadamax-Same nearly doubling the param- 37 (a) Pairwise Wilcoxon Signed-Rank Matrix(b) Holm-Bonferroni Corrected Matrix Figure 8: Phase 2 Statistical Significance Matrices. Significance heatmaps generated via pairwise Wilcoxon Signed-Rank Tests for the Hadamax integration. To maintain statistical validity, the Holm-Bonferroni correction was applied to control the family-wise error rate at α = 0.05. The light/pale cells (p < 0.05) confirm that Gamma-Hadamax-Valid provides a statistically significant improvement (p < 0.001) over the base Gamma architecture. Further- more, the deep red cell between Gamma-Hadamax-Valid and Same (p = 0.720) indicates that Same’s parameter expansion offers no significant functional advantage over Valid. eter count of Gamma-Hadamax-Valid (3,514,724 compared to 1,843,556), the resulting empirical performance difference between the two variants remains statistically insignificant (p = 0.720). This finding explicitly confirms that preserving spatial resolution through padding in the later layers yields no functional advantage. Consequently, these statistical results firmly support the selection of Gamma-Hadamax-Valid as the optimal and most efficient representation module for our unified framework. The integration of multiplicative feature interactions yielded a marked prob- abilistic advantage over the purely convolutional Gamma backbone. As detailed in the Probability of Improvement matrix for Phase 2 (Figure 9), all Hadamax- integrated variants demonstrate a high likelihood of outperforming the stan- dard Gamma topology. Specifically, the Gamma-Hadamax-Valid and Gamma- Hadamax-Same variants achieve improvement probabilities of 0.70 and 0.75, re- spectively, when evaluated against the base Gamma architecture. Furthermore, 38 the baseline Hadamax model also exhibits a 0.67 probability of improvement over Gamma, reinforcing the conclusion that non-linear, multiplicative gating serves as a highly consistent driver of policy optimization across diverse task distributions. Figure 9: Probability of Improvement (Hadamax Integration). Pairwise probability matrix comparing the standard Gamma backbone against its Hadamax-augmented variations. The integration of multiplicative interactions yields a strict probabilistic advantage, with Gamma- Hadamax-Valid and -Same achieving improvement probabilities of 0.70 and 0.75, respectively, over the standard Gamma encoder. Table 8: Phase 2 Atari-57 Results (HNS). GameGamma Hadamax Gamma-Valid Hadamax Gamma-Same Hadamax Alien2.8452.7023.1372.841 Amidar1.0381.3561.2111.180 Assault35.62941.32542.90341.687 Asterix14.31139.44741.38537.083 Continued on next page 39 Table 8 – continued from previous page GameGamma Hadamax Gamma-Valid Hadamax Gamma-Same Hadamax Asteroids1.4381.5921.9821.367 Atlantis46.33640.06137.18441.708 Bank Heist1.7212.0822.0482.090 Battlezone1.8802.0931.8702.147 Beamrider2.3883.5914.3123.238 Berzerk1.22721.10120.16017.426 Bowling0.1330.0810.0360.070 Boxing8.2858.2898.3258.298 Breakout16.76317.00019.52517.793 Centipede1.4732.9143.1502.712 Chopper Command55.50681.03897.06962.657 Crazy Climber6.4678.0428.2518.388 Defender5.86121.21220.75121.023 Demon Attack72.88474.51574.36874.517 Double Dunk7.8047.6847.9317.726 Enduro2.6932.7312.7322.726 Fishing Derby2.5902.5812.6402.563 Freeway1.1121.1471.1481.145 Frostbite2.1722.1562.8032.262 Gopher29.68231.50126.53930.023 Gravitar0.3530.4980.5740.528 H.E.R.O.0.7191.1981.1821.175 Ice Hockey1.8903.6573.5762.882 James Bond17.12743.75344.56635.515 Kangaroo4.6234.5454.4244.523 Krull8.4748.6628.6528.697 Kung-Fu Master1.3771.0110.9721.270 Montezuma’s Revenge0.0000.0020.0010.001 Ms. Pac-Man0.8121.3781.5781.356 Name This Game3.2403.2603.1093.293 Phoenix33.45239.07640.16235.342 Pitfall!0.0330.0340.0340.034 Pong1.1811.1811.1811.181 Private Eye0.0010.000-0.0000.000 Continued on next page 40 Table 8 – continued from previous page GameGamma Hadamax Gamma-Valid Hadamax Gamma-Same Hadamax Q*bert1.8163.1853.0872.799 River Raid1.7371.9912.1481.889 Road Runner21.18832.83335.83528.831 Robotank6.9607.5677.5547.448 Seaquest0.2015.4565.1794.132 Skiing-0.3930.6340.6460.585 Solaris0.1900.1210.1330.132 Space Invaders5.26619.71220.77819.852 Stargunner42.47862.74862.48159.544 Surround1.1231.1491.1361.154 Tennis2.6763.0773.0723.077 Time Pilot14.65710.9689.38510.432 Tutankham1.5381.5621.4391.529 Up’n Down24.04624.17624.46724.297 Venture0.0010.1050.0000.078 Video Pinball366.645319.338316.253333.249 Wizard of Wor6.4686.4835.7815.715 Yars’ Revenge2.5507.1265.9067.174 Zaxxon2.3062.3282.3602.487 Median2.5503.2603.1503.077 IQM3.4815.3255.3835.138 6.3. Phase 3: Final Experiments and Statistical Validation The final phase of our evaluation rigorously assessed the integration of complex, advanced regression architectures on top of our optimally selected backbone. Table 9 details the aggregate Interquartile Mean (IQM) Human- Normalized Scores (HNS) for these composite regression heads when paired with the Gamma-Hadamax-Valid encoder. The 95% Confidence Intervals (CIs) were calculated via stratified bootstrap sampling across the four independent random seeds. By bridging highly parameterized distributional and ensemble heads with the computational efficiency of the Gamma-Hadamax-Valid encoder, this unified 41 Table 9: Phase 3 Global Performance. Aggregate IQM HNS across the Atari-57 suite for advanced regression heads integrated with the Gamma-Hadamax-Valid encoder. 95% Confi- dence Intervals (CIs) are estimated via stratified bootstrap sampling. Architecture ConfigurationIQM HNS (95% CI) Gamma-Hadamax-Valid (Baseline)5.343 [4.812, 5.920] Gamma-Hadamax-Valid + Distributional Dueling5.812 [5.240, 6.450] Gamma-Hadamax-Valid + Ensemble Dueling5.940 [5.315, 6.610] Aftab6.479 [5.890, 7.210] structural approach, hereafter referred to as the complete Aftab framework, improves aggregate performance metrics in model-free Atari benchmarks. To confirm the validity of this improvement, we applied the Wilcoxon signed-rank test. The unified Distributional Ensemble Dueling head (6.479 IQM) exhibited a statistically significant performance advantage over the standard Gamma- Hadamax-Valid scalar baseline (5.343 IQM) with p = 0.014. This provides strong empirical evidence that complex multi-head regression paradigms can successfully drive temporally extended exploration and value estimation strictly within a buffer-free, parallelized framework. To evaluate out-of-distribution generalization capabilities, we extended our final analysis to the Procgen benchmark under its Hard configuration. Unlike the deterministic layouts of Atari, this tests the architectural resilience of Aftab against severe procedural non-stationarity. As documented in the Supplemen- tary Material, the final Aftab configuration achieved an aggregate IQM Procgen Normalized Score (PNS) of 0.418 with a 95% CI of [0.132, 0.730]. This marks an improvement over the baseline PQN model, which yielded an IQM PNS of 0.382 (95% CI: [0.189, 0.540]). The raw agent scores across the Procgen Hard suite corroborate this functional superiority (Table S4), proving that the repre- sentations learned by our composite architecture generalize effectively to unseen environmental dynamics without overfitting to fixed asset structures. Table 10 provides a contextual comparison between our final Aftab agent 42 and representative Atari-57 agents reported in the wider literature, utilizing the Median HNS as the standard performance indicator. Table 10: Comparison of Median Human-Normalized Scores (HNS) on the Atari-57 benchmark evaluated at 200M frames. AlgorithmMedian HNSReference DQN0.79(Mnih et al., 2015) Double DQN1.15(Van Hasselt et al., 2016) Dueling DQN1.51(Wang et al., 2016) Rainbow2.31(Hessel et al., 2018) Hadamax-PQN3.10(Kooi et al., 2025) GDI11.46(Fan & Xiao, 2022) MuZero7.31(Schrittwieser et al., 2020) Ours (Aftab)4.42This Work These comparative results should be interpreted cautiously, as the listed methods differ significantly in their implementation details, reliance on replay buffers, computational budgets, and specific evaluation protocols. Our primary objective is not to claim state-of-the-art dominance over heavily subsidized, replay-buffer-dependent architectures like GDI or MuZero. Rather, we seek to evaluate the upper limits of a purely buffer-free, PQN-style agent driven solely by structural encoder enhancements and advanced value-head topologies. Under our strict evaluation protocol, Aftab achieves a Median HNS of 4.42, confidently exceeding the human-normalized threshold (1.0) while eliminating the memory and storage constraints of an experience replay buffer. While replay-buffer-based methods achieve exceptional Median HNS scores, large-scale buffers introduce prohibitive Random Access Memory (RAM) overhead, severe I/O bottlenecks, and high sampling latency in long-horizon training regimes. Our results demonstrate that architectural optimizations and multi-head regression paradigms can successfully bridge the performance gap, yielding a highly stable, off-policy framework with a drastically reduced hardware foot- 43 print. A more comprehensive Pareto analysis detailing wall-clock time, mem- ory footprint, energy consumption, and implementation-level throughput across these architectures is left for future investigation. Comprehensive evaluation scores, raw data arrays, and Human Normalized Score (HNS) distributions for all individual games in the Final Experiments are detailed in Tables 11 and 12 below and are hosted in full within the results directory of our official framework repository. The structural advantage of our unified regression topology is conclusively supported by the final Probability of Improvement analysis (Figure 10). When compared directly to the scalar PQN baseline, the advanced regression heads demonstrate overwhelming consistency. The Bootstrapped Dueling configura- tion achieves a 0.93 probability of improvement over PQN, while the final com- posite Aftab framework (Distributional Ensemble Dueling) yields a 0.86 proba- bility of improvement. Furthermore, the final Distributional Bootstrapped Du- eling architecture maintains a 0.68 probability of improvement when evaluated directly against the singular Distributional Dueling head, confirming that the multi-head ensemble approach effectively stabilizes categorical return estimates across the benchmark. Table 11: Phase 3 Atari-57 Results (HNS) - Part 1. GamePQN Gamma Distributional Dueling Alien0.5422.8452.936 Amidar0.6401.0381.563 Assault30.169 35.62932.991 Asterix40.617 14.31140.500 Asteroids0.0341.4380.853 Atlantis47.172 46.33641.771 Bank Heist1.8851.7212.240 Battlezone1.2121.8803.442 Beamrider1.1442.3884.172 Berzerk2.2511.22710.575 Continued on next page 44 Table 11 – continued from previous page GamePQN Gamma Distributional Dueling Bowling0.0440.1330.111 Boxing8.2448.2858.322 Breakout12.019 16.76321.954 Centipede0.8281.47319.338 Chopper Command2.679 55.50687.416 Crazy Climber6.2686.4677.907 Defender3.1735.86117.538 Demon Attack72.523 72.88474.105 Double Dunk7.7637.8047.852 Enduro2.7232.6932.729 Fishing Derby2.5252.5902.485 Freeway1.1321.1121.149 Frostbite1.5452.1722.230 Gopher24.506 29.68218.667 Gravitar0.2350.3530.581 H.E.R.O.0.7820.7191.721 Ice Hockey0.8501.8903.807 James Bond8.134 17.12772.778 Kangaroo4.4804.6234.621 Krull7.6218.47411.218 Kung-Fu Master1.4151.3771.247 Montezuma’s Revenge 0.0000.0000.000 Ms. Pac-Man0.4540.8121.582 Name This Game2.2783.2404.226 Phoenix24.840 33.45273.773 Pitfall!0.0320.0330.034 Pong1.181 1.1811.181 Private Eye0.012 0.0010.000 Q*bert1.5721.8163.273 River Raid1.3551.7372.099 Road Runner7.268 21.18825.003 Robotank7.1096.9607.692 Seaquest0.1870.20111.129 Skiing-0.581 -0.3930.093 Continued on next page 45 Table 11 – continued from previous page GamePQN Gamma Distributional Dueling Solaris0.1110.1900.010 Space Invaders4.8415.26622.340 Stargunner27.278 42.47862.695 Surround1.0691.1231.189 Tennis1.3812.6763.069 Time Pilot5.901 14.65710.613 Tutankham1.5191.5381.443 Up’n Down23.231 24.04626.663 Venture0.0000.0010.000 Video Pinball315.049 366.645373.477 Wizard of Wor4.4436.4688.275 Yars’ Revenge2.2222.5505.245 Zaxxon1.8342.3063.675 Median1.8852.5504.172 IQM2.6923.4816.031 Table 12: Phase 3 Atari-57 Results (HNS) - Continued. Game Ensembled DuelingAftab Alien4.0743.120 Amidar1.7611.726 Assault40.04838.902 Asterix40.49531.534 Asteroids0.9280.693 Atlantis46.769143.124 Bank Heist2.2262.141 Battlezone2.7813.389 Beamrider3.2723.854 Berzerk16.84117.892 Bowling0.1950.236 Boxing8.3168.316 Breakout23.31249.864 Continued on next page 46 Table 12 – continued from previous page Game Ensembled DuelingAftab Centipede3.93422.292 Chopper Command113.124108.981 Crazy Climber7.7726.849 Defender23.06727.015 Demon Attack74.4346.942 Double Dunk7.873-1.055 Enduro2.7106.434 Fishing Derby2.6062.552 Freeway1.1481.148 Frostbite2.8072.228 Gopher32.19118.371 Gravitar0.8331.334 H.E.R.O.1.2982.172 Ice Hockey3.6454.421 James Bond32.913107.082 Kangaroo4.6254.826 Krull8.92012.508 Kung-Fu Master1.4181.301 Montezuma’s Revenge0.0240.030 Ms. Pac-Man1.7891.479 Name This Game3.1294.163 Phoenix70.621152.923 Pitfall!0.0330.033 Pong1.1811.180 Private Eye0.0110.001 Q*bert3.6293.332 River Raid2.5952.163 Road Runner45.44637.633 Robotank8.0097.676 Seaquest8.60253.586 Skiing0.656-0.205 Solaris0.1480.029 Space Invaders40.25439.191 Stargunner71.93762.879 Continued on next page 47 Table 12 – continued from previous page Game Ensembled DuelingAftab Surround1.1581.204 Tennis3.0793.079 Time Pilot8.94515.470 Tutankham1.6131.617 Up’n Down25.96729.496 Venture0.0040.486 Video Pinball402.971 1113.419 Wizard of Wor5.8718.354 Yars’ Revenge5.5215.677 Zaxxon2.1714.839 Median3.6454.421 IQM5.5926.479 48 Figure 10: Probability of Improvement (Final Regression Architectures). Pairwise comparison of the advanced value-estimation heads operating on the Gamma-Hadamax-Valid backbone. The composite Distributional Bootstrapped Dueling architecture (Aftab) exhibits high con- sistency, establishing a 0.86 probability of improvement over the base PQN configuration. 7. Discussion and Limitations 7.1. Comprehensive Performance Analysis The experimental results across all three phases indicate that the structural configuration of the convolutional neural network, the nature of its feature in- teractions, and the topology of its value regression head serve as critical drivers of agent generalization. Empirically, these architectural choices appear to exert a more pronounced influence on tracking value landscapes than raw parameter capacity scaling within the evaluated online regimes. 1. The Impact of Hierarchical Depth (Phase 1): The Alpha vari- ant achieved the highest Phase 1 IQM HNS of 3.536, followed closely by Gamma at 3.481. Both models leverage increased hierarchical depth 49 (a) Full training duration.(b) Latest 50M frames (zoomed). Figure 11: Global Performance (IQM HNS). The HNS serves as a standardized metric where 0.0 represents random play and 1.0 represents professional human performance. The 5- layer Alpha and Gamma variants outperform the original DQN encoder. Comparison between full training (left) and the final 50M frames (right) highlights depth as a primary factor in generalization. (four and five layers, respectively) relative to the three-layer baseline ar- chitecture. This observation suggests that moderately deeper receptive fields capture spatial dependencies and abstract environmental features more effectively without requiring excessive parameter expansion in the early layers. 2. Topology vs. Raw Capacity: The Eta variant represents a width- heavy paradigm, featuring a massive regression head containing 23.7M parameters driven by a shallow encoder. Conversely, Zeta restricts itself to a three-layer topology containing only 2.5M total parameters. The pairwise Wilcoxon signed-rank test yields a p-value of 0.967 between these two variants. This statistical equivalence indicates that expanding chan- nel width and linear layer capacity offers negligible functional utility if the underlying spatial feature extractor remains shallow and structurally constrained. 3. Multiplicative Interactions (Phase 2): As visualized in Figure 13, integrating Hadamax multiplicative representation mechanics in Phase 2 drastically altered the parameter-to-performance ratio. Gamma-Hadamax- Valid achieved an IQM HNS of 5.343 while restricting its total parame- 50 Figure 12: Wilcoxon Signed Rank Test. A significance heatmap generated via pairwise Wilcoxon Signed-Rank Tests conducted across the Atari-57 suite for Phase 1. To maintain sta- tistical integrity across 36 comparisons, we applied the Holm-Bonferroni correction to control the family-wise error rate at α = 0.05. The data indicate that the Alpha, Beta, and Gamma cluster exhibits a statistically significant performance advantage over the PQN baseline. Con- versely, the p < 0.001 results for the Delta variant against superior models suggest that its wide 9× 9 kernel correlates with a statistically significant decrease in agent performance. ter footprint to 1.84M. This is less than half the footprint required by the standard three-layer Hadamax control network (4.12M parameters, 5.149 IQM). Furthermore, although Gamma-Hadamax-Same nearly dou- bled Gamma-Hadamax-Valid’s parameter count (3.51M) by modifying the pooling padding, the performance difference was statistically insignificant (p = 0.720). This underscores that multiplicative Hadamard interactions 51 layered over an optimal topological depth (Gamma) maximize represen- tational capacity, rendering subsequent parameter scaling redundant. (a) Full training duration.(b) Latest 50M frames (zoomed). Figure 13: Phase 2 Global Performance (IQM HNS). The integration of Hadamard max-pooling alters representational capacity. The Gamma-Hadamax V1 and V2 variants diverge from the baseline Hadamax architecture, demonstrating that multiplicative feature interactions pair effectively with the Gamma convolutional depth. 4. Buffer-Free Advanced Regression and Generalization (Phase 3): Composite value algorithms like Rainbow (Hessel et al., 2018) traditionally combine multiple estimation frameworks (e.g., distributional value pre- diction, dueling decomposition, and multi-step returns) within off-policy settings stabilized by target networks and experience replay buffers. Our Phase 3 experiments demonstrate that these decoupled regression designs can be successfully adapted to operate in a completely online, buffer-free, parallelized data stream. When integrated with two-dimensional Layer Normalization, large-batch vectorized sampling, and the Gamma-Hadamax-Valid encoder, the unified Aftab framework achieved the highest aggregate performance within our evaluation suite, reaching an IQM HNS of 6.479 (95% CI: [5.890, 7.210]). Furthermore, our ensemble variant updates K = 10 distinct heads simul- taneously across the global data batch (p = 1.0). While this omits classi- cal bootstrapped data masking, the architectural averaging across heads serves as an effective variance-reduction mechanism for value estimations under rapid updates. More importantly, by approximating Thompson 52 sampling during environment interaction, this multi-head structure drives deep, temporally extended exploration. This prevents the agent from pre- maturely converging on sub-optimal policies, a common pitfall of standard ε-greedy heuristics, and is a primary driver of the superior sample efficiency observed in the final Aftab framework. This operational stability is noteworthy given the challenges delineated by the "deadly triad" (Sutton & Barto, 2018), which notes that combining function approximation, bootstrapping, and off-policy data characteristics frequently induces severe gradient divergence. Our empirical findings show that Aftab circumvents these nonlinear temporal-difference optimization traps under the tested hyperparameters. This stability is heavily influenced by the structural characteristics of the Gamma-Hadamax-Valid encoder, which establishes a gradual channel ex- pansion (32→ 48→ 64) and introduces local max-pooling downsampling to stabilize the feature activations processed by the regression streams. Furthermore, our ensemble variant updates K = 10 distinct heads simul- taneously across the global data batch (p = 1.0). While this omits classi- cal bootstrapped data masking, the architectural averaging across heads serves as an effective variance-reduction mechanism for value estimations under rapid updates. This resilience is further corroborated by our out-of-distribution general- ization experiments on Procgen Hard. As documented in the Supplemen- tary Material, the Aftab framework demonstrated a measurable advantage over the baseline PQN architecture, yielding an IQM PNS of 0.418 (95% CI: [0.132, 0.730]) compared to PQN’s score of 0.382 (95% CI: [0.189, 0.540]). Raw game data reveals that on environments requiring precise visual tracking amidst high non-stationarity, such as Bigfish (7.233 vs. 31.491) and Dodgeball (0.192 vs. 12.596), the combination of Hadamard representations and ensemble regression heads significantly dampens over- fitting to procedurally generated assets. 53 (a) Full training duration.(b) Latest 50M frames (zoomed). Figure 14: Phase 3 Global Performance (IQM HNS). The ablation of advanced value- estimation heads operating on the Gamma-Hadamax-Valid backbone. Aftab framework yields the highest performance among the evaluated models, reaching 6.479 IQM. The zoomed frame (right) illustrates the stability of these composite algorithms operating in a strictly buffer-free, parallelized regime. 7.2. Statistical Significance and Multi-Comparison Correction To verify that the reported performance variations reflect genuine structural advantages rather than stochastic noise, we subjected all three experimental phases to stringent non-parametric hypothesis testing. 1. Robust Performance Aggregation: The significance matrices are eval- uated utilizing the Interquartile Mean (IQM) of the normalized scores. By discarding the lower and upper 25% of the performance distribution across the task suites, the IQM filters out both catastrophically unlearned environments and outlier tasks with hyper-inflated returns, offering a sta- tistically reliable baseline of central performance. 2. Pairwise Hypothesis Testing: For every variant pairing, we applied the Wilcoxon signed-rank test (Wilcoxon, 1945). To rigorously account for inflation of the Type I error rate across our 36 pairwise comparisons in Phase 1 and subsequent comparisons in Phase 2, we enforced the Holm- Bonferroni step-down correction (Holm, 1979) to control the global family- wise error rate strictly at α = 0.05. 3. Diagnostic Analysis of Structural Bottlenecks: The statistical heatmap effectively diagnoses structural pitfalls in sub-optimal design choices. For 54 instance, by employing an over-extended 9× 9 kernel in its initial layer, the Delta variant (2.374 IQM) exhibited a statistically significant perfor- mance degradation (p < 0.001) relative to its counterparts, highlighting a loss of fine-grained spatial tracking. Similarly, the Theta variant (2.649 IQM), which halves the terminal encoder channel width, failed to demon- strate a statistically significant departure from the baseline PQN model (p = 0.431), illustrating that excessive feature compression severely bot- tlenecked the downstream policy’s expressivity. 7.3. Limitations While our empirical evaluations demonstrate clear statistical advantages for deeper structural topologies and integrated multi-head regression systems, sev- eral core limitations must be explicitly addressed. First, although the Atari-57 and Procgen Hard benchmarks present highly diverse visual and procedural configurations, our observations remain restricted to discrete-action, frame-stacked reinforcement learning paradigms. The degree to which our architectural findings, specifically the multiplicative Hadamard feature streams and the high-dimensional Distributional Ensemble Dueling head (Aftab), generalize to high-frequency continuous control tasks or long-horizon temporal reasoning suites (such as the DeepMind Control Suite (Tassa et al., 2018)) remains to be empirically verified. Second, to isolate the functional impact of the CNN structures and value- head topologies, all evaluations were constrained to a single, static set of hyper- parameters inherited from the baseline PQN framework. It is highly probable that heavily parameterized, composite networks like Aftab could achieve supe- rior sample efficiency if critical training variables, such as the learning rate, GAE λ, or optimizer batch dimensions, were tailored specifically to their complex and highly non-linear loss landscapes. Finally, we emphasize the theoretical limitation regarding the absence of for- mal convergence or stability guarantees for our unregularized network updates. The baseline PQN architecture relies heavily on explicit ℓ 2 regularization (weight 55 decay) as an analytical device to bound the spectral norm of the TD Jacobian, suppressing nonlinear divergence. While our experiments show that the combi- nation of Layer Normalization, parallel vectorized sampling, and the structural parameters of the Gamma topology successfully maintains operational stability without weight decay across all evaluated seeds, we do not provide a closed- form mathematical or Lipschitz bound for this behavior. Delineating the formal theoretical relationship between structural network constraints, normalization layers, and TD Jacobian stability in buffer-free, non-linear value function ap- proximation remains an essential direction for future theoretical research. 8. Conclusion The transition toward highly parallelized, buffer-free temporal difference learning, exemplified by the Parallelized Q-Network (PQN) framework, repre- sents a methodological shift in deep reinforcement learning. However, by strictly defaulting to legacy feature extractors, the field may be artificially bottlenecking the representational potential of these streamlined algorithms. In this study, we introduced Aftab, an open-source benchmarking framework designed to system- atically evaluate architectural topology, non-linear representational synergies, and advanced regression paradigms within online, off-policy regimes. Through a comprehensive, multi-phase empirical evaluation across the de- terministic Atari-57 benchmark and the highly non-stationary Procgen Hard suite, our large-scale analysis yields critical insights regarding sample efficiency and neural architecture design: • Topological Depth as a Primary Driver (Phase 1): Our baseline CNN ablations provide strong evidence that structural inductive bias dic- tates generalization capability. While the Alpha variant achieved the high- est aggregate performance among the base architectures (3.536 IQM HNS), the 5-layer Gamma variant emerged as the optimal architectural founda- tion. Gamma achieved a statistically comparable 3.481 IQM while operat- ing with superior computational efficiency, requiring only 1.84 million total 56 parameters and 24.5 million FLOPs. Conversely, the Eta variant, which heavily expanded channel width to drive its parameter count to 23.8 mil- lion, achieved a lower 3.114 IQM. This confirms that, under our evaluation protocol, raw parameter expansion cannot adequately substitute for the hierarchical feature abstraction provided by convolutional depth. • The Penalty of Spatial Bottlenecks (Phase 1): Our experiments identified specific topological pitfalls that severely restrict policy expres- sivity. The significant performance degradation of the Delta variant (2.374 IQM) illustrates that aggressive spatial reduction in early layers, specifi- cally via a 9× 9 initial kernel, correlates with a critical loss of the fine- grained visual granularity required for reactive control tasks. Similarly, the Theta variant (2.649 IQM) demonstrated that halving the output di- mensionality in the terminal convolutional layer imposes a representational bottleneck prior to the regression head. • Synergies of Multiplicative Feature Interactions (Phase 2): In- tegrating Hadamard non-linearities into the optimal Gamma topology substantially improved representational capacity. The Gamma-Hadamax- Valid architecture substituted standard spatial striding with max-pooling downsampling and element-wise multiplicative interactions, paired with zero-saturating GELU activations. This combination yielded an IQM HNS of 5.343, outperforming the baseline Hadamax control (5.149 IQM) while requiring less than half the parameter count (1.84M vs. 4.12M) and a sig- nificantly reduced computational footprint (123.6M vs. 162.9M FLOPs). This establishes that multiplicative interactions compound effectively with optimized convolutional depth, enhancing representation without relying on massive residual stacks. • Stability of Buffer-Free Algorithmic Integration (Phase 3): We demonstrated that complex regression heads, which traditionally rely on massive experience replay buffers and target networks to stabilize their 57 loss landscapes (Hessel et al., 2018), can operate reliably in a strictly on- line, parallelized regime. By decoupling the advantage stream (Dueling), predicting categorical return distributions (Distributional), and driving directed epistemic exploration via multi-head ensembles, Aftab achieved the highest overall score of 6.479 IQM HNS. • Out-of-Distribution Generalization: To explicitly evaluate structural resilience against high-capacity architectural memorization, our frame- work was tested on the procedurally generated Procgen Hard benchmark. The final Aftab architecture achieved an aggregate IQM Procgen Nor- malized Score (PNS) of 0.418, outperforming the baseline PQN model (0.382 IQM PNS). This validates that the inductive biases established in this study translate effectively to non-deterministic, structurally shifting visual domains without overfitting to fixed pixel configurations. Ultimately, this research suggests that relying on the historically static "Na- ture CNN" backbone restricts algorithmic generalization. By systematically optimizing hierarchical depth, incorporating non-linear multiplicative feature interactions, and deploying variance-reduced, multi-head regression systems, the Aftab framework establishes a stable and computationally efficient architectural reference for parallelized deep reinforcement learning. Future work will focus on integrating these architectural blueprints into compiled JAX environments (Bradbury et al., 2018) to maximize evaluation throughput and minimize hardware latency. Furthermore, we aim to empiri- cally validate whether the structural efficiency and representational stability of the Gamma-Hadamax-Valid backbone translate effectively to high-dimensional continuous control tasks and data-constrained offline reinforcement learning sce- narios. References Araujo, A., Norris, W., & Sim, J. (2019). Computing receptive fields of con- volutional neural networks. Distill, 4, e21. URL: https://distill.pub/ 58 2019/computing-receptive-fields. doi:10.23915/distill.00021. Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450, . Bellemare, M. G., Dabney, W., & Mnih, V. (2017). A distributional perspec- tive on reinforcement learning. In International Conference on Machine Learning (p. 449–458). PMLR. Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013). The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47, 253–279. URL: https://jair.org/ index.php/jair/article/view/10819. doi:10.1613/jair.3912. Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Amiranashvili, A., Brox, T., & Peters, J. (2019). Crossq: Batch normalization in deep reinforce- ment learning for greater sample efficiency and simplicity. arXiv preprint arXiv:1902.05605, . URL: https://arxiv.org/abs/1902.05605. Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., & Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs. URL: http://github.com/jax-ml/jax. Chen, X., Wang, C., Zhou, Z., & Ross, K. (2021). Randomized ensem- bled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, . URL: https://arxiv.org/abs/2101.05982. Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2019). Leveraging pro- cedural generation to benchmark reinforcement learning. arXiv preprint arXiv:1912.01588, . Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Un- terthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for 59 image recognition at scale. URL: https://arxiv.org/abs/2010.11929. arXiv:2010.11929. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., & Kavukcuoglu, K. (2018). Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Con- ference on Machine Learning (p. 1407–1416). Fan, J., & Xiao, C. (2022). Generalized data distribution iteration. URL: https://arxiv.org/abs/2206.03192. arXiv:2206.03192. Farebrother, J., Orbay, J., Vuong, Q., Taïga, A. A., Chebotar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., Kumar, A., & Agarwal, R. (2024). Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03950, . Gallici, M., Fellows, M., Ellis, B., Pou, A., Masmitja, E., Foerster, J., & Martin, M. (2024). Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, . He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (p. 770–778). Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, . URL: https://arxiv.org/abs/1606.08415. Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Cotton, B., & Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scan- dinavian Journal of Statistics, 6, 65–70. doi:10.2307/4615733. 60 Kooi, J. E., Yang, Z., & François-Lavet, V. (2025). Hadamax encoding: Ele- vating performance in model-free atari. arXiv preprint arXiv:2505.15345, . URL: https://arxiv.org/abs/2505.15345. doi:10.48550/arXiv.2505. 15345. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learn- ing applied to document recognition. Proceedings of the IEEE, 86, 2278– 2324. Lee, H., Hwang, D., Kim, D., Kim, H., Tai, J.-J., Subramanian, K., Wurman, P. R., Choo, J., Stone, P., & Seno, T. (2024). Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, . URL: https://arxiv.org/abs/2410.09754. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, . URL: https://arxiv.org/ abs/1509.02971. Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., & Han, J. (2020). On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations (ICLR). Luo, W., Li, Y., Urtasun, R., & Zemel, R. (2017). Understanding the effec- tive receptive field in deep convolutional neural networks. URL: https: //arxiv.org/abs/1701.04128. arXiv:1701.04128. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, . Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G. et al. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. doi:10.1038/nature14236. 61 Nauman, M., Ostaszewski, M., Jankowski, K., Miłoś, P., & Cygan, M. (2024). Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in Neural Information Processing Systems. URL: https://arxiv.org/abs/2405.16158. Oh, J., Farquhar, G., Kemaev, I., Calian, D. A., Hessel, M., Zintgraf, L., Singh, S., van Hasselt, H., & Silver, D. (2025). Discovering state-of-the-art reinforcement learning algorithms. Nature, 648, 312– 319. URL: https://w.nature.com/articles/s41586-025-09761-x. doi:10.1038/s41586-025-09761-x. Osband, I., Blundell, C., Pritzel, A., & Van Roy, B. (2016). Deep exploration via bootstrapped dqn. arXiv preprint arXiv:1602.04621, . Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., & Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 32 (p. 8024– 8035). Curran Associates, Inc. URL: http://papers.neurips.c/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., & Silver, D. (2020). Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588, 604–609. URL: http://dx.doi.org/10. 1038/s41586-020-03051-4. doi:10.1038/s41586-020-03051-4. Schwarzer, M., Obando Ceron, J. S., Courville, A., Bellemare, M. G., Agar- wal, R., & Castro, P. S. (2023). Bigger, better, faster: Human-level 62 Atari with human-level efficiency. In Proceedings of the 40th Interna- tional Conference on Machine Learning (p. 30365–30380). PMLR. URL: https://proceedings.mlr.press/v202/schwarzer23a.html. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., & Hassabis, D. (2017). Master- ing the game of go without human knowledge. Nature, 550, 354–359. URL: https://doi.org/10.1038/nature24270. doi:10.1038/nature24270. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning series (2nd ed.). Cambridge, MA: The MIT Press. Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T., & Riedmiller, M. (2018). Deepmind control suite. arXiv preprint arXiv:1801.00690, . Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 30. URL: https://ojs.aaai.org/index.php/AAAI/ article/view/10295. Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., & Freitas, N. (2016). Dueling network architectures for deep reinforcement learning. In Interna- tional Conference on Machine Learning (ICML) (p. 1995–2003). Weng, J., Huang, M. L., Huang, S., Bo, H., & Makoviichuk, D. (2022). En- vpool: A highly parallel reinforcement learning environment execution en- gine. arXiv preprint arXiv:2206.10558, . Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1, 80–83. 63 Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2017). Learning transferable architectures for scalable image recognition. arXiv preprint arXiv:1707.07012, . URL: https://arxiv.org/abs/1707.07012. 64 Supplementary Material for: Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks Taha Shieenavaz 1 (taha.shieenavaz@studenti.unipd.it), Shabnam Zareshahraki 1 (shabnam.zareshahraki@studenti.unipd.it), Loris Nanni 1 (loris.nanni@unipd.it) 1 Department of Information Engineering, University of Padua, Italy Corresponding author at: Department of Information Engineering, University of Padua, Italy. Taha Shieenavaz Email: tahashieenavaz@gmail.com Shabnam Zareshahraki Email: shabnamzaresh@gmail.com Document Description: This document acts as the dedicated companion file for the main text, housing un-normalized raw evaluation returns across all 57 benchmark tasks along with their full operational step histories. 1. Receptive Field Analysis The receptive field (RF) of a convolutional neural network at layer l calcu- lates the spatial region of the input volume that influences a single feature map activation. It is defined recursively based on the kernel size (k) and stride (s) of the current and preceding layers: RF l = RF l−1 + (k l − 1) l−1 Y i=1 s i (S1) where we assume an initial receptive field RF 0 = 1 and an initial stride product of 1. Below, we detail the step-by-step calculation of the maximum theoretical receptive field for both the baseline Nature DQN Encoder, Hadamax Encoder, and our proposed Gamma Encoder and Hadamax Gamma Encoders. Table S1: Summary of the maximum theoretical receptive fields for the Nature DQN, standard Gamma, baseline Hadamax, and Gamma-Hadamax encoders. Encoder ArchitectureEffective Receptive Field Nature DQN Encoder36× 36 Gamma Encoder39× 39 Hadamax Encoder59× 59 Hadamax Gamma Encoder (Valid & Same)70× 70 1.1. Nature DQN Encoder The Nature DQN architecture consists of three convolutional layers with aggressive early spatial downsampling (a stride of 4 in the first layer). The receptive field expands as follows: Layer 1 (k = 8, s = 4) : RF 1 = 1 + (8− 1)× 1 = 8 Layer 2 (k = 4, s = 2) : RF 2 = 8 + (4− 1)× 4 = 20 Layer 3 (k = 3, s = 1) : RF 3 = 20 + (3− 1)× (4× 2) = 36 The total effective receptive field for the Nature DQN encoder maps to a 36×36 spatial patch on the original input. 1 1.2. Gamma Encoder The Gamma architecture replaces large filters with a deeper, five-layer stack of 3× 3 convolutions. This preserves spatial resolution longer before downsam- pling. The calculation is as follows: Layer 1 (k = 3, s = 2) : RF 1 = 1 + (3− 1)× 1 = 3 Layer 2 (k = 3, s = 2) : RF 2 = 3 + (3− 1)× 2 = 7 Layer 3 (k = 3, s = 1) : RF 3 = 7 + (3− 1)× (2× 2) = 15 Layer 4 (k = 3, s = 2) : RF 4 = 15 + (3− 1)× (2× 2× 1) = 23 Layer 5 (k = 3, s = 1) : RF 5 = 23 + (3− 1)× (2× 2× 1× 2) = 39 The total effective receptive field for the Gamma encoder maps to a 39× 39 spatial patch. This demonstrates that despite using exclusively small 3× 3 kernels, the depth of the Gamma architecture allows it to achieve a marginally larger global receptive field than the Nature DQN baseline, while simultaneously supporting a highly complex, non-linear hypothesis space. 1.3. Hadamax Encoder The Hadamax variant of the Nature DQN architecture replaces standard convolutional strides with explicit Max Pooling steps for downsampling. Be- cause each block contains both a convolution (c) and a pooling (p) operation, the receptive field expands in two stages per block: Block 1 Conv (k = 8, s = 1) : RF c1 = 1 + (8− 1)× 1 = 8 Block 1 Pool (k = 4, s = 4) : RF p1 = 8 + (4− 1)× 1 = 11 Block 2 Conv (k = 4, s = 1) : RF c2 = 11 + (4− 1)× (1× 4) = 23 Block 2 Pool (k = 2, s = 2) : RF p2 = 23 + (2− 1)× 4 = 27 Block 3 Conv (k = 3, s = 1) : RF c3 = 27 + (3− 1)× (4× 2) = 43 Block 3 Pool (k = 3, s = 1) : RF p3 = 43 + (3− 1)× 8 = 59 The total effective receptive field for the Hadamax encoder maps to a 59× 59 spatial patch. 2 1.4. Hadamax Gamma Encoder (Valid & Same) The Hadamax Gamma architecture utilizes a five-block deep configuration. Note that while Valid and Same differ in their pooling padding, padding does not alter the maximum receptive field calculation of a single unit. Thus, the calculation for both is identical: Block 1 Conv (k = 3, s = 1) : RF c1 = 1 + (3− 1)× 1 = 3 Block 1 Pool (k = 2, s = 2) : RF p1 = 3 + (2− 1)× 1 = 4 Block 2 Conv (k = 3, s = 1) : RF c2 = 4 + (3− 1)× (1× 2) = 8 Block 2 Pool (k = 2, s = 2) : RF p2 = 8 + (2− 1)× 2 = 10 Block 3 Conv (k = 3, s = 1) : RF c3 = 10 + (3− 1)× (2× 2) = 18 Block 3 Pool (k = 3, s = 1) : RF p3 = 18 + (3− 1)× 4 = 26 Block 4 Conv (k = 3, s = 1) : RF c4 = 26 + (3− 1)× (4× 1) = 34 Block 4 Pool (k = 2, s = 2) : RF p4 = 34 + (2− 1)× 4 = 38 Block 5 Conv (k = 3, s = 1) : RF c5 = 38 + (3− 1)× (4× 2) = 54 Block 5 Pool (k = 3, s = 1) : RF p5 = 54 + (3− 1)× 8 = 70 The total effective receptive field for the Hadamax Gamma encoder maps to a 70× 70 spatial patch. This deeper, decoupled pooling approach significantly expands the receptive field compared to both the standard Gamma architecture (39× 39) and the baseline Hadamax (59× 59). 3 2. Individual Environment Learning Curves (Figures) PQN Alpha Beta Gamma Delta Epsilon Zeta Eta Theta AlienAmidarAssaultAsterix AsteroidsAtlantisBank HeistBattlezone BeamriderBerzerkBowlingBoxing BreakoutCentipedeChopper CmdCrazy Climber Figure S1: Phase 1 Learning Curves: Un-normalized evaluation returns over 200 million training frames for the baseline CNN encoders (Alpha through Theta) across all 57 Atari games. (Part 1 of 4) 4 DefenderDemon AttackDouble DunkEnduro Fishing DerbyFreewayFrostbiteGopher GravitarH.E.R.O.Ice HockeyJames Bond KangarooKrullKung Fu MstrMontezuma’s Figure S1: Phase 1 Learning Curves: Un-normalized evaluation returns over 200 million training frames for the baseline CNN encoders (Alpha through Theta) across all 57 Atari games. (Part 2 of 4) 5 Ms. Pac-ManName ThisPhoenixPitfall! PongPrivate EyeQ*bertRiver Raid Road RunnerRobo-TankSeaquestSkiing SolarisSpace Inv.Star GunnerSurround Figure S1: Phase 1 Learning Curves: Un-normalized evaluation returns over 200 million training frames for the baseline CNN encoders (Alpha through Theta) across all 57 Atari games. (Part 3 of 4) 6 TennisTime PilotTutankhamUp ’n Down VentureVid. PinballWizard of WorYars’ Revenge Zaxxon Figure S1: Phase 1 Learning Curves: Un-normalized evaluation returns over 200 million training frames for the baseline CNN encoders (Alpha through Theta) across all 57 Atari games. (Part 4 of 4) 7 Gamma Hadamax-Gamma-Valid Hadamax-Gamma-Same Hadamax AlienAmidarAssaultAsterix AsteroidsAtlantisBank HeistBattlezone BeamriderBerzerkBowlingBoxing BreakoutCentipedeChopper CmdCrazy Climber Figure S2: Phase 2 Learning Curves: Un-normalized evaluation returns over 200 million training frames comparing the standard Gamma baseline to the Hadamax-integrated variants across all 57 Atari games. (Part 1 of 4) 8 DefenderDemon AttackDouble DunkEnduro Fishing DerbyFreewayFrostbiteGopher GravitarH.E.R.O.Ice HockeyJames Bond KangarooKrullKung Fu MstrMontezuma’s Figure S2: Phase 2 Learning Curves: Un-normalized evaluation returns over 200 million training frames comparing the standard Gamma baseline to the Hadamax-integrated variants across all 57 Atari games. (Part 2 of 4) 9 Ms. Pac-ManName ThisPhoenixPitfall! PongPrivate EyeQ*bertRiver Raid Road RunnerRobo-TankSeaquestSkiing SolarisSpace Inv.Star GunnerSurround Figure S2: Phase 2 Learning Curves: Un-normalized evaluation returns over 200 million training frames comparing the standard Gamma baseline to the Hadamax-integrated variants across all 57 Atari games. (Part 3 of 4) 10 TennisTime PilotTutankhamUp ’n Down VentureVid. PinballWizard of WorYars’ Revenge Zaxxon Figure S2: Phase 2 Learning Curves: Un-normalized evaluation returns over 200 million training frames comparing the standard Gamma baseline to the Hadamax-integrated variants across all 57 Atari games. (Part 4 of 4) 11 PQN Distributional Duelling Ensemble Duelling Aftab AlienAmidarAssaultAsterix AsteroidsAtlantisBank HeistBattlezone BeamriderBerzerkBowlingBoxing BreakoutCentipedeChopper CmdCrazy Climber Figure S3: Phase 3 Learning Curves: Un-normalized evaluation returns over 200 million training frames for the advanced value regression architectures (Dueling, Ensemble, and Dis- tributional) across all 57 Atari games. (Part 1 of 4) 12 DefenderDemon AttackDouble DunkEnduro Fishing DerbyFreewayFrostbiteGopher GravitarH.E.R.O.Ice HockeyJames Bond KangarooKrullKung Fu MstrMontezuma’s Figure S3: Phase 3 Learning Curves: Un-normalized evaluation returns over 200 million training frames for the advanced value regression architectures (Dueling, Ensemble, and Dis- tributional) across all 57 Atari games. (Part 2 of 4) 13 Ms. Pac-ManName ThisPhoenixPitfall! PongPrivate EyeQ*bertRiver Raid Road RunnerRobo-TankSeaquestSkiing SolarisSpace Inv.Star GunnerSurround Figure S3: Phase 3 Learning Curves: Un-normalized evaluation returns over 200 million training frames for the advanced value regression architectures (Dueling, Ensemble, and Dis- tributional) across all 57 Atari games. (Part 3 of 4) 14 TennisTime PilotTutankhamUp ’n Down VentureVid. PinballWizard of WorYars’ Revenge Zaxxon Figure S3: Phase 3 Learning Curves: Un-normalized evaluation returns over 200 million training frames for the advanced value regression architectures (Dueling, Ensemble, and Dis- tributional) across all 57 Atari games. (Part 4 of 4) 15 PQNAftab Big Fish HardBoss Fight HardCave Flyer HardChaser Hard Climber HardCoinrun HardDodgeball HardFruitbot Hard Heist HardJumper HardLeaper HardMaze Hard Miner HardNinja HardPlunder HardStarpilot Hard Figure S4: Out-of-Distribution Generalization: Un-normalized evaluation returns over 200 million training frames comparing the baseline PQN to the complete Aftab framework across the 16 individual Procgen Hard environments. 16 3. Raw Evaluation Returns (Tables) Table S1: Phase 1 Atari-57 Raw Returns: Final unnormalized evaluation scores for the PQN baseline and the eight parameter-constrained CNN encoders (Alpha through Theta). (Part 1 of 2) PQNAlphaBetaGammaDeltaEpsilonZetaEtaTheta Alien4688.197 13152.109 14752.177 18963.333 2680.850 17525.578 11592.449 11572.4153209.184 Amidar1082.5002178.1531139.7311893.639 974.150 2362.010 1200.925 1726.5201234.459 Assault17009.493 12128.391 14126.837 18030.888 13932.092 18513.439 14149.139 17580.704 17846.646 Asterix366434.014 129816.327 154993.197 126304.082 235632.653 86924.830 103686.395 85679.932 374916.327 Asteroids2312.721 103959.762 15110.748 68774.320 11182.687 28792.415 85168.333 2549.1501950.884 Atlantis742852.041 728122.789 751796.599 742603.741 736686.735 761386.735 747847.619 751528.912 758324.490 Bank Heist1402.1431506.395 1595.0001288.912 1332.7891257.517 1588.163 1302.0751514.762 Battlezone44176.871 43829.932 64149.660 64085.034 44642.857 65819.728 48993.197 38017.007 37350.340 Beamrider22017.973 47062.184 39400.667 39152.435 18069.789 33208.973 22814.422 20786.871 19927.231 Berzerk5877.857 15391.463958.5711942.279 3578.5373434.660 8512.109 2253.2313921.633 Bowling30.00037.83027.72430.95931.33330.44234.667 38.00730.571 Boxing99.228 100.000 100.00099.587 100.000 100.000 100.000 100.000 100.000 Breakout352.044436.816451.425 504.044 326.320405.925394.044431.728407.690 Centipede10076.690 13596.034 11219.793 17675.850 8680.939 11957.031 8450.085 12807.0177412.735 Chopper Command 17155.442 118263.946 334721.088 360188.095 6479.252 258300.000 185923.810 19930.612 11332.993 Crazy Climber161068.367 179992.517 204894.898 179469.728 185854.422 191479.252 175811.224 179710.884 153729.252 Defender51787.245 75035.374 66311.905 91159.694 53549.320 61850.510 65859.354 50575.850 62583.844 Demon Attack132035.357 132607.364 133369.286 132650.221 128967.942 132331.548 131675.442 130311.701 128636.190 Double Dunk-1.782-1.238-1.095-1.707-1.980-1.483-1.442-0.850-1.449 Enduro2360.7212357.7412325.9592341.762 2353.3712343.173 2339.864 2338.6842326.656 Fishing Derby42.28646.74842.82343.27238.19749.18447.17742.53741.000 Freeway33.82333.98333.82732.97333.65333.84033.66733.82033.806 Frostbite6653.946 9596.7019046.8039532.925 4842.9256228.571 8996.667 8130.3064846.871 Gopher53851.973 64661.769 72346.054 51893.333 34766.463 39304.626 32817.483 68673.129 45066.190 Gravitar956.122645.4081319.5581293.367 730.1021032.483978.061 1874.830594.388 H.E.R.O.24341.207 25199.864 21372.449 24353.571 20274.881 24368.827 30249.847 34076.003 14280.051 Ice Hockey0.1772.1360.89514.364-3.6220.7211.5312.235-3.690 James Bond2195.0682500.8501569.0485012.245 1360.374 7288.435 2657.653 3470.9182228.912 Kangaroo13455.442 14392.857 14387.075 14087.755 11813.605 13774.150 14466.667 14088.095 14328.231 Krull9707.585 10637.381 10961.361 10724.456 9399.592 10308.878 11143.061 9998.946 10395.680 17 Table S2: Phase 1 Atari-57 Raw Returns: Final unnormalized evaluation scores for the PQN baseline and the eight parameter-constrained CNN encoders (Alpha through Theta). (Part 2 of 2) PQNAlphaBetaGammaDeltaEpsilonZetaEtaTheta Kung-Fu Master31990.136 38857.823 31007.823 35474.830 34184.014 35648.299 33834.354 32204.762 41987.075 Montezuma’s R.0.0000.0000.0000.0000.0000.0000.0000.0000.000 Ms. Pac-Man3365.3405302.4156122.8915855.2383515.5445614.4565822.483 7529.9324865.816 Name This Game 15329.014 19033.571 16205.748 21340.510 15285.782 16041.735 15393.265 12873.435 17054.830 Phoenix177903.333 213915.986 235491.803 190653.197 51112.245 185824.252 283574.218 255857.245 98561.429 Pitfall!-5.3440.000-21.1050.0000.0000.0000.000-0.871-22.041 Pong21.00021.00021.00021.00021.00021.00021.00021.00021.000 Private Eye66.66798.639100.00087.3339.52450.00050.00044.983100.000 Q*bert21923.469 25177.296 25397.874 25847.959 17267.262 24266.241 25570.153 26119.218 22553.486 River Raid21349.014 27019.490 27153.299 29396.327 23497.007 28564.456 30737.823 24621.361 23529.388 Road Runner57719.048 72455.442 63845.918 175828.571 56056.463 61521.088 81121.769 56501.361 57602.381 Robotank71.16776.07572.41268.75272.17071.55473.39164.13366.609 Seaquest7978.027 10092.9257835.6468228.8448466.8718840.816 17571.224 17243.7418063.129 Skiing-23644.677 -9432.129 -18793.803 -22986.874 -23272.214 -9766.480 -10895.473 -10667.639 -21527.639 Solaris2605.3062069.9322881.8373081.4972004.7622541.5422247.2793255.102 4381.088 Space Invaders5342.2962655.680 11072.2452855.5786468.6392966.7182851.633 24713.4352864.932 Stargunner259520.408 371468.027 423331.293 403316.667 241495.918 320142.517 315854.082 237664.966 229308.503 Surround8.3508.8579.2489.1466.0719.2629.9059.8337.112 Tennis-1.33311.833-4.31623.881-1.09211.633-2.500-1.000-1.303 Time Pilot13860.884 27697.619 32540.816 27330.272 11938.095 23786.054 24746.939 26560.5449999.320 Tutankham249.473255.932264.330254.279259.340248.612248.173268.939246.932 Up’n Down263719.048 253879.762 295438.095 266537.007 164273.027 307976.429 318144.694 91661.429 268903.095 Venture0.0000.0000.0000.0000.0000.0000.0000.0000.000 Video Pinball457305.670 542947.592 448438.116 529722.776 510972.653 462849.429 544883.133 529542.265 487644.446 Wizard of Wor16897.959 28960.544 29923.469 29785.714 13953.401 34776.190 23365.646 33848.980 26704.762 Yars’ Revenge115017.827 136848.190 138392.262 140728.207 101737.813 124140.105 144543.561 141543.759 115265.139 Zaxxon16136.735 17865.306 24046.939 20938.435 14489.116 15231.293 15286.395 20634.694 17481.633 Median6653.946 12128.391 11072.245 10724.4566479.252 10308.878 11143.061 11572.4154865.816 IQM9199.376 14472.442 12906.556 14799.3068112.347 13152.706 12762.966 12441.9398813.942 IQM 95% CI[4627, 20442] [7148, 32568] [6080, 29171] [7312, 33077] [3965, 16543] [6180, 27604] [6335, 29271] [6367, 22211] [4295, 19384] 18 Table S3: Phase 2 Atari-57 Raw Returns: Final un-normalized evaluation scores comparing the standard Gamma encoder against the Gamma-Hadamax-Valid (V1) and Gamma-Hadamax- Same (V2) architectures. Gamma Hadamax-Gamma-Valid Hadamax-Gamma-SameHadamax Alien2.8062.2623.1432.956 Amidar1.0991.5381.1700.814 Assault33.85440.43045.00042.586 Asterix14.82837.49142.53232.479 Asteroids1.4871.3462.0061.003 Atlantis46.25643.38237.87244.555 Bank Heist1.7252.1132.0462.108 Battlezone1.9142.3621.9122.275 Beamrider2.5482.7094.1262.594 Berzerk1.28223.23021.63310.371 Bowling0.1350.1310.0340.046 Boxing8.2918.2488.3258.321 Breakout17.30714.30519.64920.023 Centipede1.4322.7723.1092.261 Chopper Command63.29066.259105.04127.477 Crazy Climber6.6577.8928.7038.946 Defender6.25521.93221.03020.902 Demon Attack72.82774.64274.36174.521 Double Dunk7.7397.3887.9617.807 Enduro2.6852.7302.7332.711 Fishing Derby2.5822.5132.6342.523 Freeway1.1121.1461.1481.140 Frostbite2.1911.4462.7292.408 Gopher28.83437.09727.95926.749 Gravitar0.3350.4220.5750.577 H.E.R.O.0.7331.2091.1881.115 Ice Hockey1.9163.7013.4981.342 James Bond18.35843.67944.75419.589 Kangaroo4.6124.7224.3524.491 Krull8.3218.8008.5818.820 Kung-Fu Master1.3581.0690.9691.741 Montezuma’s Revenge0.0000.0000.0010.000 Ms. Pac-Man0.8431.1951.6001.272 Name This Game3.2423.4053.1113.368 Phoenix35.26337.00144.99831.553 Pitfall!0.0330.0340.0340.033 Pong1.1811.1811.1811.181 Private Eye0.0010.001-0.0000.000 Q*bert1.8053.3223.0062.091 River Raid1.7841.7872.1401.712 Road Runner22.76429.29437.24422.406 Robotank6.9757.6437.5657.174 Seaquest0.1975.5224.9141.540 Skiing-0.3240.6360.6490.501 Solaris0.1870.0910.1340.157 Space Invaders5.19418.61619.90119.767 Stargunner42.91363.30762.75453.536 Surround1.1221.1551.1401.167 Tennis2.6773.0843.0753.078 Time Pilot14.72212.2179.6299.367 Tutankham1.5151.7011.4601.477 Up’n Down24.24622.75325.15924.811 Venture0.0000.2030.0000.028 Video Pinball373.361321.522306.502357.007 Wizard of Wor6.6687.7585.6514.370 Yars’ Revenge2.5778.1925.5197.117 Zaxxon2.3192.2672.3722.829 Median2.5773.3223.1432.829 IQM3.5085.2545.3604.712 IQM 95% CI[2.143, 6.967][2.926, 10.606][3.031, 11.030] [2.651, 8.907] 19 Table S4: Phase 3 Atari-57 Raw Returns: Final un-normalized evaluation scores detailing the ablation of the advanced regression heads (Distributional Dueling, Ensemble Dueling, and Aftab). PQN Distributional Duelling Ensemble DuellingAftab Alien4688.19720386.32724329.32022953.912 Amidar1082.5002634.7893378.6162926.568 Assault17009.49315812.54121412.94220397.452 Asterix366434.014340450.680330103.741283998.639 Asteroids2312.72145602.00743198.02724836.667 Atlantis742852.041740343.197773082.6531922416.327 Bank Heist1402.1431673.7411657.1431576.259 Battlezone44176.871119812.925107248.299111278.912 Beamrider22017.97373008.43553364.93971070.449 Berzerk5877.85727839.25246650.10239211.531 Bowling30.00028.93549.54161.925 Boxing99.228100.000100.000100.000 Breakout352.044594.007699.381933.010 Centipede10076.690202759.65035466.418207815.929 Chopper Command17155.442543812.585827556.122568332.993 Crazy Climber161068.367216305.102201359.524173999.660 Defender51787.245297204.762367071.259505392.857 Demon Attack132035.357134858.639135605.23812853.605 Double Dunk-1.782-1.245-1.340-21.027 Enduro2360.7212345.1842346.5756165.946 Fishing Derby42.28636.93945.81643.054 Freeway33.82334.00034.00034.000 Frostbite6653.9468145.27212135.2729659.218 Gopher53851.97344451.42970625.03440949.388 Gravitar956.1222012.5852737.0754410.034 H.E.R.O.24341.20751736.15637095.71468036.905 Ice Hockey0.17735.66031.33340.592 James Bond2195.06818633.3337044.72829409.354 Kangaroo13455.44214308.84413969.72814474.830 Krull9707.58513738.36711224.55813158.673 Kung-Fu Master31990.13629379.59226209.18429263.265 Montezuma’s Revenge0.0000.000123.129148.980 Ms. Pac-Man3365.34011182.35011885.4399572.963 Name This Game15329.01426781.15619973.87826262.517 Phoenix177903.333482720.680483661.5651104620.680 Pitfall!-5.3440.000-5.0070.000 Pong21.00021.00021.00021.000 Private Eye66.66716.667100.000100.000 Q*bert21923.46945021.08848135.96944785.119 River Raid21349.01433202.00743151.36135383.163 Road Runner57719.048140571.429375416.667257419.048 Robotank71.16775.59278.72177.527 Seaquest7978.027471696.327365552.6871909299.898 Skiing-23644.677-15831.898-8677.490-20213.990 Solaris2605.3061216.8712487.2791365.850 Space Invaders5342.29634645.20462922.97154110.289 Stargunner259520.408619437.075692191.497628127.211 Surround8.3509.8279.5009.959 Tennis-1.33323.83324.00024.000 Time Pilot13860.88420004.76219017.34728396.259 Tutankham249.473240.306266.622258.588 Up’n Down263719.048311849.558271042.823269188.537 Venture0.0000.0000.000707.483 Video Pinball457305.670478535.429547276.8061700656.636 Wizard of Wor16897.95936559.52425731.63336345.578 Yars’ Revenge115017.827284535.259305711.432293344.007 Zaxxon16136.73533449.66019704.42239183.333 Median6653.94620004.76219704.42222953.912 IQM9199.37625328.19722998.09723746.835 IQM 95% CI[4684.833, 19899.239] [11805.560, 69101.979] [11148.088, 73260.157] [12366.775, 67372.868] 20 Table S5: Phase 3 Procgen Hard Raw Returns: Final un-normalized evaluation scores com- paring the baseline PQN against the final composite Aftab architecture across the out-of- distribution Procgen environments. PQNAftab BigfishHard7.23331.491 BossfightHard0.0230.000 CaveflyerHard5.3130.000 ChaserHard2.5648.871 ClimberHard1.0490.000 CoinrunHard4.8160.000 DodgeballHard0.19212.596 FruitbotHard-0.41513.647 HeistHard0.4180.347 JumperHard2.3060.000 LeaperHard5.5009.520 MazeHard1.4494.561 MinerHard0.35810.052 NinjaHard2.8980.520 PlunderHard4.9834.672 StarpilotHard4.5080.223 Median2.4352.541 IQM2.5013.589 IQM 95% CI[1.000, 4.169] [0.208, 8.426] 21