Paper deep dive
Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks
Taha Shieenavaz, Shabnam Zareshahraki, Loris Nanni
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/14/2026, 3:58:05 AM
Summary
The paper introduces Aftab, a composite architecture for the Parallelized Q-Network (PQN) framework that enhances performance through advanced CNN encoder topologies (specifically Gamma), Hadamax multiplicative representation learning, and advanced value-estimation heads (Distributional, Ensemble, and Dueling). Aftab significantly outperforms the standard PQN baseline on Atari-57 and demonstrates improved procedural generalization on Procgen-Hard, achieving higher sample efficiency and normalized scores while maintaining a memory-efficient, buffer-free training paradigm.
Entities (10)
Relation Signals (9)
Aftab → evaluatedon → Atari-57
confidence 99% · Extensive experiments on Atari-57 show that our final composite architecture, Aftab, achieves...
Aftab → outperforms → Parallelized Q-Network
confidence 98% · Aftab achieves an Interquartile Mean (IQM) Human-Normalized Score of 6.592, compared with 2.715 for the standard PQN baseline
Aftab → evaluatedon → Procgen-Hard
confidence 96% · We additionally evaluate Aftab on Procgen-Hard to assess performance under procedurally varying visual environments.
Aftab → uses → Hadamax
confidence 95% · integrating the Hadamax encoding paradigm with categorical, ensemble, and dueling value heads
Aftab → uses → Gamma
confidence 95% · The second phase... Gamma is therefore selected as the preferred performance–efficiency backbone... In the third phase, we hold the Gamma-Hadamax-Valid encoder fixed
Aftab → uses → Deep Ensemble
confidence 94% · we integrate... a Deep Ensemble formulation... The latter constitutes our final model, Aftab.
Aftab → uses → Dueling
confidence 94% · we integrate Distributional RL... Dueling value decomposition... and a Deep Ensemble formulation... The latter constitutes our final model, Aftab.
Aftab → uses →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms. Notably, the Parallelized Q-Network (PQN) algorithm enables off-policy value learning without relying on experience replay buffers or target networks. However, the representational capacity and computational efficiency of visual encoders operating in these buffer-free settings remain comparatively underexplored. In this work, we systematically investigate the architectural design space of Convolutional Neural Networks within PQN. We evaluate eight distinct CNN topologies while explicitly characterizing their parameter and computational requirements. We further study the effect of multiplicative representation learning and advanced value estimation by integrating the Hadamax encoding paradigm with categorical, ensemble, and dueling value heads. Extensive experiments on Atari-57 show that our final composite architecture, Aftab, achieves an Interquartile Mean (IQM) Human-Normalized Score of 6.592, compared with 2.715 for the standard PQN baseline, together with a 0.86 Probability of Improvement over PQN. We additionally evaluate Aftab on Procgen-Hard to assess performance under procedurally varying visual environments. Aftab achieves a normalized learning-curve Area Under the Curve (nAUC) of 0.541 compared with 0.216 for PQN. Overall, the results demonstrate that carefully designed encoder topology, multiplicative feature interactions, and advanced value-estimation heads can substantially improve performance within a parallelized, replay-free Q-learning framework while preserving its memory-efficient training paradigm. The complete Aftab framework, including model definitions, training configurations, reproducibility settings, and raw experimental logs, is open-sourced at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2608.07335v2
- Canonical: https://arxiv.org/abs/2608.07335v2
Trouble viewing inline? Open PDF directly →
Full Text
145,969 characters extracted from source content.
Expand or collapse full text
Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks Taha Shieenavaz 1 (tahashieenavaz@gmail.com), Shabnam Zareshahraki 1 (shabnamzaresh@gmail.com), Loris Nanni 1 (loris.nanni@unipd.it) 1 Department of Information Engineering, University of Padua, Italy Corresponding author at: Department of Information Engineering, University of Padua, Italy. Taha Shieenavaz Email: tahashieenavaz@gmail.com Shabnam Zareshahraki Email: shabnamzaresh@gmail.com Department of Information Engineering, University of Padua, Italy arXiv:2608.07335v2 [cs.LG] 13 Aug 2026 Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks Taha Shieenavaz a,∗ , Shabnam Zareshahraki a , Loris Nanni a a Department of Information Engineering, University of Padua, Italy Abstract Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms. Notably, the Parallelized Q-Network (PQN) algorithm enables off-policy value learning without relying on experi- ence replay buffers or target networks. However, the representational capacity and computational efficiency of visual encoders operating in these buffer-free settings remain comparatively underexplored. In this work, we systematically investigate the architectural design space of Convolutional Neural Networks within PQN. We evaluate eight distinct CNN topologies while explicitly char- acterizing their parameter and computational requirements. We further study the effect of multiplicative representation learning and advanced value estima- tion by integrating the Hadamax encoding paradigm with categorical, ensem- ble, and dueling value heads. Extensive experiments on Atari-57 show that our final composite architecture, Aftab, achieves an Interquartile Mean (IQM) Human-Normalized Score of 6.592, compared with 2.715 for the standard PQN baseline, together with a 0.86 Probability of Improvement over PQN. We addi- tionally evaluate Aftab on Procgen-Hard to assess performance under procedu- rally varying visual environments. Aftab achieves an IQM Procgen Normalized Score (PNS) of 0.418 compared with 0.382 for PQN. More substantially, its ∗ Corresponding author. Email addresses: tahashieenavaz@gmail.com (Taha Shieenavaz), shabnamzaresh@gmail.com (Shabnam Zareshahraki), loris.nanni@unipd.it (Loris Nanni) Preprint submitted to Expert Systems with ApplicationsAugust 14, 2026 normalized learning-curve Area Under the Curve (nAUC) is 0.541 compared with 0.216 for PQN, corresponding to approximately 2.50× the normalized performance accumulated over the complete 200-million-frame training trajec- tory. Overall, the results demonstrate that carefully designed encoder topol- ogy, multiplicative feature interactions, and advanced value-estimation heads can substantially improve performance within a parallelized, replay-free Q- learning framework while preserving its memory-efficient training paradigm. The complete Aftab framework, including model definitions, training config- urations, reproducibility settings, and raw experimental logs, is open-sourced at https://github.com/tahashieenavaz/aftab. Keywords: Deep Reinforcement Learning, Convolutional Neural Networks, Parallelized Q-Network (PQN), Buffer-Free Learning, Hadamax Representation, Distributional Value Estimation, Categorical Value Estimation, Deep Ensembles, Dueling Architecture, Procedural Generalization 1. Introduction Since the introduction of the Deep Q-Network (DQN) (Mnih et al., 2013, 2015), Deep Reinforcement Learning (DRL) has demonstrated that policies can be learned directly from high-dimensional visual observations. In these sys- tems, raw pixel inputs are transformed into latent representations that support downstream value estimation and action selection. The quality of this representation is therefore closely tied to the design of the visual feature extractor, which in value-based DRL is most commonly im- plemented as a Convolutional Neural Network (CNN) (LeCun et al., 1998). Computer vision architectures have evolved substantially since the original DQN formulation, with increasingly deep and expressive models such as resid- ual networks (He et al., 2016), neural architecture search systems (Zoph et al., 2017), and Vision Transformers (Dosovitskiy et al., 2021). Reinforcement learn- ing systems have likewise benefited from deeper visual encoders, including the residual architectures used in IMPALA (Espeholt et al., 2018) and AlphaGo 2 Zero (Silver et al., 2017). Nevertheless, many value-based model-free DRL al- gorithms continue to inherit variants of the compact three-layer convolutional encoder introduced with DQN (Mnih et al., 2013, 2015). Consequently, the ef- fect of encoder topology itself, independently of large increases in model capac- ity, remains comparatively underexplored in modern parallelized value-learning frameworks. This question is particularly relevant in simplified, highly parallelized algo- rithms such as the Parallelized Q-Network (PQN) (Gallici et al., 2024). PQN removes the experience replay buffer and target network used by conventional DQN and instead relies on vectorized sampling, Layer Normalization (Ba et al., 2016), and regularized temporal-difference updates to support stable off-policy learning. Subsequent approaches such as Hadamax (Kooi et al., 2025) have further enhanced representation learning in this setting through multiplicative feature interactions and explicit max-pooling. However, these advances have largely focused on modifying the learning or representation mechanism while retaining the underlying convolutional hierarchy. This leaves open an important question: how much of the performance of replay-free, parallelized Q-learning is determined by the topology of the visual encoder itself? We investigate this question through a controlled, three-phase empirical study within the PQN framework. Our primary evaluation uses the full Atari-57 benchmark (Bellemare et al., 2013), with high-throughput environment simula- tion provided by EnvPool (Weng et al., 2022). In the first phase, we evaluate eight CNN architectures, denoted Alpha through Theta, designed to examine differences in depth, spatial reduction, receptive-field structure, and parameter efficiency. Alpha achieves the highest Phase 1 IQM Human-Normalized Score (HNS) of 3.566, while Gamma achieves a closely matched IQM HNS of 3.508. The Alpha–Gamma difference is not statistically resolved after Holm–Bonferroni correction, while Gamma requires fewer parameters and fewer FLOPs. Gamma is therefore selected as the preferred performance–efficiency backbone for sub- sequent experiments. The second phase evaluates whether this deeper hierarchy can be combined 3 effectively with the Hadamax representation mechanism (Kooi et al., 2025). We compare a standard Hadamax baseline with two Gamma-based Hadamax vari- ants that differ in their pooling strategy: Gamma-Hadamax-Valid and Gamma- Hadamax-Same. The results show that Gamma-Hadamax-Valid provides a fa- vorable trade-off between representation quality and computational cost, and it is consequently selected as the shared encoder for the final phase of the study. In the third phase, we hold the Gamma-Hadamax-Valid encoder fixed and examine the effect of advanced value-estimation architectures. Specifically, we integrate Distributional RL (Bellemare et al., 2017), Dueling value decomposi- tion (Wang et al., 2016), and a Deep Ensemble formulation inspired by boot- strapped exploration (Osband et al., 2016; Nauman et al., 2024). We evaluate Distributional Dueling, Ensemble Dueling, and a unified Distributional Ensem- ble Dueling architecture. The latter constitutes our final model, Aftab. Across Atari-57, Aftab achieves an aggregate IQM HNS of 6.592 and a 0.86 Probability of Improvement over the original PQN baseline under our evaluation protocol. To examine whether the gains obtained on Atari transfer beyond fixed visual layouts, we additionally evaluate the final Aftab architecture on the 16 environ- ments of Procgen under the Hard configuration (Cobbe et al., 2019). This benchmark introduces substantial procedural variation in level layouts, visual assets, and environment dynamics. Aftab achieves an aggregate IQM Procgen Normalized Score (PNS) of 0.418 compared with 0.382 for PQN. Moreover, its normalized area under the learning curve is 0.541 compared with 0.216 for PQN, indicating that the advantage extends beyond terminal performance to the full training trajectory. Because the gains are heterogeneous across individual Proc- gen environments, we interpret these results as evidence of improved aggregate procedural generalization and sample efficiency rather than uniform dominance across tasks. Our objective is not to maximize unconstrained Atari performance relative to large replay-buffer-dependent agents such as GDI (Fan & Xiao, 2022) or MuZero (Schrittwieser et al., 2020). Instead, we investigate how far a replay- free, parallelized Q-learning framework can be improved through architectural 4 design alone. In addition, whereas the original PQN formulation employs ex- plicit ℓ 2 regularization, we set weight decay to zero throughout our controlled experiments and empirically assess whether normalization, vectorized sampling, and architectural structure are sufficient to maintain operational stability under the evaluated protocol. In summary, the main contributions of this work are: • Systematic encoder evaluation: We benchmark eight CNN topologies within PQN under controlled parameter and computational constraints, identifying Gamma as a favorable performance–efficiency trade-off. • Hadamax integration with a deeper backbone: We integrate the Gamma topology with Hadamard multiplicative interactions and max- pooling, showing that Gamma-Hadamax-Valid improves representational performance while avoiding the parameter expansion of less efficient alter- natives. • Advanced value-estimation ablation in a replay-free regime: We evaluate Distributional, Ensemble, and Dueling value-estimation mecha- nisms on a fixed Gamma-Hadamax-Valid encoder and combine them into the final Aftab architecture. • Evaluation of procedural generalization and sample efficiency: We assess the final Aftab model on Procgen Hard, reporting both terminal IQM PNS and learning-curve AUC to complement the Atari-57 analysis with a procedurally varying benchmark. We name the final composite model Aftab, Persian for “sunshine,” as a the- matic reference to Rainbow (Hessel et al., 2018) and to the combination of com- plementary value-learning mechanisms within a single replay-free, parallelized architecture. 5 2. Related Work Deep reinforcement learning has progressed through advances in representa- tion learning, value estimation, exploration, and optimization. In visual control, these components are tightly coupled: the encoder determines the latent rep- resentation available to the value function, while the learning algorithm deter- mines how effectively that representation can be optimized from non-stationary interaction data. This work lies at the intersection of four research direc- tions: convolutional encoder design, replay-free value learning, advanced value- estimation architectures, and procedural generalization. 2.1. Visual Encoders in Deep Reinforcement Learning The Deep Q-Network (DQN) demonstrated that convolutional feature ex- tractors could support end-to-end value learning directly from raw Atari obser- vations (Mnih et al., 2013, 2015). Its visual backbone consists of three convo- lutional layers followed by fully connected value-estimation layers and became the de facto encoder for a wide range of subsequent value-based agents. Double DQN (Van Hasselt et al., 2016), Dueling DQN (Wang et al., 2016), Categor- ical DQN (C51) (Bellemare et al., 2017), and Rainbow (Hessel et al., 2018) introduced substantial modifications to optimization or value estimation while largely preserving the same basic convolutional hierarchy. Other reinforcement-learning systems have demonstrated that substantially deeper visual representations can be effective. IMPALA (Espeholt et al., 2018), for example, employs a deeper residual encoder for large-scale multi-task learn- ing, while AlphaGo Zero (Silver et al., 2017) uses deep residual networks to construct rich state representations. More recent work has also investigated pa- rameter scaling and architectural simplicity in reinforcement learning, including Simba (Lee et al., 2024). These approaches demonstrate the potential benefits of greater representational capacity, but they often modify depth, width, resid- ual structure, and total parameter count simultaneously. Consequently, it can be difficult to isolate whether performance improvements arise from a particular topological choice or simply from increased model capacity. 6 Our work addresses this distinction in the context of PQN by comparing multiple convolutional topologies while explicitly tracking encoder parameters, head parameters, and computational cost. Rather than pursuing unconstrained network scaling, we focus on the performance–efficiency trade-offs produced by changes in depth, kernel structure, channel progression, and spatial downsam- pling. 2.2. Replay-Free and Parallelized Value Learning Classical DQN-style off-policy value learning relies on an experience replay buffer and a periodically updated target network to reduce temporal correla- tion and stabilize bootstrapped temporal-difference updates (Mnih et al., 2013, 2015). A substantial body of later work has improved sample efficiency and optimization behavior through alternative network and update formulations, including REDQ (Chen et al., 2021), CrossQ (Bhatt et al., 2019), and more re- cent scalable value-learning methods such as Bigger, Better, Faster (Schwarzer et al., 2023). The Parallelized Q-Network (PQN) (Gallici et al., 2024) takes a different ap- proach by removing both the large replay buffer and the target network. Instead, PQN relies on highly parallelized environment interaction, multi-step λ-returns, Layer Normalization (Ba et al., 2016), and explicit ℓ 2 regularization to improve the stability of online off-policy temporal-difference learning. This formula- tion is particularly well suited to modern vectorized simulation systems such as EnvPool (Weng et al., 2022), which provide high-throughput synchronous sam- pling, and to accelerator-oriented frameworks such as JAX (Bradbury et al., 2018). Hadamax (Kooi et al., 2025) extends this replay-free setting by modifying the representation mechanism itself. Instead of relying exclusively on strided convolutions, Hadamax introduces parallel normalized projections, element-wise Hadamard interactions, GELU activations, and explicit max-pooling. This im- proves representational expressivity without requiring a conventional replay- based learning pipeline. However, Hadamax retains a convolutional hierar- 7 chy closely related to the standard DQN/PQN backbone. The interaction between Hadamax-style multiplicative representations and alternative convo- lutional topologies therefore remains comparatively underexplored. The present study targets this gap by first identifying an efficient convolu- tional backbone within PQN and subsequently examining how that topology interacts with the Hadamax representation mechanism. 2.3. Advanced Value Estimation and Ensemble Exploration Architectural improvements in value-based reinforcement learning extend beyond the visual encoder. Dueling networks (Wang et al., 2016) decompose the action-value function into separate state-value and action-advantage streams, allowing the network to represent state quality independently of the relative utility of individual actions. Distributional reinforcement learning (Bellemare et al., 2017) instead models a distribution over returns rather than estimating only their expectation, providing a richer representation of value uncertainty and reward structure. More recently, value estimation has also been reformulated as a classification problem. The “Stop Regressing” framework (Farebrother et al., 2024) showed that categorical representations of scalar value targets can improve optimization and scalability. This perspective motivates the use of histogram-based classifi- cation objectives such as HL-Gauss in place of conventional mean-squared-error regression. A complementary line of work uses ensembles to improve exploration and value estimation. Bootstrapped DQN (Osband et al., 2016) approximates Thomp- son sampling using multiple independently parameterized Q-heads, enabling temporally extended exploratory behavior. Related ensemble-based approaches have demonstrated that independently initialized value functions can improve robustness and exploration in deep reinforcement learning (Nauman et al., 2024). Rainbow (Hessel et al., 2018) established that several value-learning enhance- ments, including categorical estimation and dueling decomposition, can be com- 8 plementary when combined within a replay-buffer-based DQN agent. However, the behavior of such composite value heads is less well characterized in replay- free, highly parallelized temporal-difference learning. In particular, their inter- action with multiplicative Hadamax representations has not been systematically evaluated. We therefore study Distributional Dueling, Ensemble Dueling, and their unified Distributional Ensemble Dueling formulation on a fixed Gamma- Hadamax-Valid encoder. 2.4. Procedural Generalization in Reinforcement Learning Generalization remains a central challenge in deep reinforcement learning, particularly when high-capacity visual policies can exploit regularities in a lim- ited set of training environments. Procgen (Cobbe et al., 2019) was introduced specifically to study this problem through procedurally generated environments in which layouts, visual assets, and task configurations vary across levels. The benchmark contains 16 environments and provides standardized difficulty set- tings designed to evaluate both learning efficiency and generalization across procedurally generated levels. Whereas Atari-57 provides a broad collection of visually and behaviorally distinct tasks, each individual Atari game retains comparatively fixed visual and structural regularities. Procgen therefore provides a complementary test of whether improvements obtained on Atari remain useful when the underlying level configuration varies throughout interaction. In this work, Procgen Hard is used as a secondary evaluation of the final Aftab architecture rather than as an additional architecture-selection benchmark. 3. Preliminaries We briefly formalize the value-based reinforcement learning setting and the architectural and optimization mechanisms underlying the experiments in this work. We first review DQN and PQN, and then introduce the Hadamax encoder, dueling value decomposition, categorical value estimation, and ensemble-based exploration. 9 3.1. Value-Based Reinforcement Learning and DQN We consider an agent interacting with an environmentE through a sequence of observations, actions, and rewards. The interaction is modeled as a Markov Decision Process (MDP), ⟨S,A,P,R,γ⟩, whereS denotes the state space,A the action space,P the transition dynamics, R the reward function, and γ ∈ [0, 1) the discount factor (Sutton & Barto, 2018). At time step t, the agent selects an action a t ∈ A and the environment transitions according to s t+1 ∼P(·| s t ,a t ), while emitting a scalar reward r t . In visual environments such as Atari (Belle- mare et al., 2013), the agent does not directly observe the underlying emulator state. Instead, the policy receives image observations, commonly represented as a stack of consecutive frames in order to retain short-term temporal information (Mnih et al., 2013). The objective is to maximize the expected discounted return R t = ∞ X k=0 γ k r t+k .(1) The optimal action-value function is defined as Q ∗ (s,a) = max π E [R t | s t = s, a t = a, π].(2) DQN (Mnih et al., 2013, 2015) approximates this function with a neural net- work Q(s,a;θ). For a transition (s,a,r,s ′ ), the network is trained by minimizing the squared temporal-difference error L(θ) =E h (y− Q(s,a;θ)) 2 i ,(3) where the one-step target is y = r + γ max a ′ ∈A Q(s ′ ,a ′ ;θ − ).(4) 10 Here, θ − denotes the parameters of a periodically updated target network. Classical DQN combines this delayed target network with an experience replay buffer, which reduces temporal correlation between consecutive training samples and stabilizes bootstrapped value updates. These mechanisms mitigate, rather than eliminate, the instabilities associated with the combination of off-policy learning, function approximation, and bootstrapping—commonly referred to as the “deadly triad” (Sutton & Barto, 2018). Throughout this paper, we use the term Nature DQN to refer to the 2015 DQN formulation (Mnih et al., 2015), including its standard three-layer convo- lutional visual encoder. 3.2. Parallelized Q-Network (PQN) The Parallelized Q-Network (PQN) (Gallici et al., 2024) modifies the con- ventional DQN learning pipeline by eliminating both the large replay buffer and the periodically updated target network. Instead, PQN collects transitions from many vectorized environments in parallel and performs learning directly from the resulting synchronous batches. The theoretical analysis of PQN considers the local stability properties of temporal-difference updates through the Jacobian of the expected update field. Let δ(φ) denote the expected TD parameter update for parameters φ. Its Jaco- bian is written as J (φ) =∇ φ δ(φ).(5) The PQN analysis decomposes sources of instability into terms associated with off-policy sampling and nonlinear function approximation. Within this analysis, Layer Normalization (Ba et al., 2016) is used to control the scale of intermediate activations, while network width and explicit ℓ 2 regularization contribute to controlling nonlinear update behavior (Gallici et al., 2024). These mechanisms motivate the use of normalization and regularization in replay-free temporal-difference learning. Importantly, the experiments in this work deliberately remove the explicit ℓ 2 penalty in order to isolate the empirical contribution of architectural design. 11 This deviation from standard PQN is described separately in Section 4. PQN additionally uses multi-step λ-returns. For a trajectory segment, the return is computed recursively as R λ t = r t + γ h λR λ t+1 + (1− λ) max a ′ Q φ (s t+1 ,a ′ ) i .(6) The parameter λ controls the interpolation between shorter-horizon TD tar- gets and longer-horizon returns. Combined with highly parallelized sampling, this formulation enables learning from recent trajectories without requiring a large replay buffer. Figure 1 contrasts the learning pipelines of Nature DQN and standard PQN. (a) Nature DQN (b) PQN Figure 1: Learning pipelines of Nature DQN and PQN. (a) Nature DQN stores tran- sitions in an experience replay buffer and evaluates bootstrapped targets using a periodically updated target network. (b) Standard PQN instead collects synchronous transitions from multiple vectorized environments and performs direct updates using the current learning net- work, thereby removing both the replay buffer and target network. 3.3. Hadamax Encoder The Hadamax (Hadamard Max-Pooling) encoder (Kooi et al., 2025) modifies the convolutional representation used by PQN by combining explicit spatial pooling with multiplicative feature interactions. Whereas conventional CNN blocks typically couple feature extraction and spatial reduction through strided 12 convolutions, Hadamax uses stride-one convolutional projections followed by max-pooling. A Hadamax block applies two parallel learned transformations to the same input. For hidden representation z j−1 , the output of block j can be written as z j = MP f LN z j−1 W j−1 1 ⊙ f LN z j−1 W j−1 2 ,(7) where W j−1 1 and W j−1 2 denote the learned kernels of the two parallel paths, LN denotes Layer Normalization, MP denotes max-pooling, and ⊙ is the element- wise Hadamard product. Hadamax uses the Gaussian Error Linear Unit (GELU) (Hendrycks & Gim- pel, 2016), GELU(x) = x 2 1 + erf x √ 2 . (8) Figure 2: GELU vs. ReLU. A visual comparison demonstrating the smooth, non-monotonic curve of the Gaussian Error Linear Unit (GELU) plotted alongside the Rectified Linear Unit (ReLU). Eq. 8 shows the mathematical formulation of the GELU activation function. Unlike ReLU, GELU does not hard-threshold all negative activations to zero. Retaining small negative activations before the Hadamard product can preserve information in both multiplicative branches and provides a smoother nonlinear- ity for optimization. Figure 3 illustrates the progression from a standard DQN block to the Layer- Normalized PQN block and finally to the multiplicative Hadamax formulation. 3.4. Dueling Network Architecture The dueling architecture (Wang et al., 2016) decomposes the action-value function into a state-value component V (s) and an action-dependent advantage 13 (a) Nature DQN (b) PQN (c) Hadamax Figure 3: Evolution of convolutional processing blocks. (a) Nature DQN uses convolu- tion followed by ReLU. (b) PQN introduces Layer Normalization before the nonlinearity. (c) Hadamax applies two parallel normalized convolutional projections, combines them through an element-wise Hadamard product, and performs spatial reduction through explicit max- pooling. component A(s,a). Given shared representation parameters θ, a value stream parameterized by β, and an advantage stream parameterized by α, the Q-value is computed as Q(s,a;θ,α,β) = V (s;θ,β) + " A(s,a;θ,α)− 1 |A| X a ′ ∈A A(s,a ′ ;θ,α) # . (9) Subtracting the mean advantage resolves the identifiability ambiguity be- tween the two streams and centers the advantage estimates around zero. This decomposition is particularly useful in states for which the choice among several actions has relatively little effect on the expected return. Figure 4: Dueling network architecture. A shared convolutional representation is sep- arated into a state-value stream and an action-advantage stream, which are subsequently combined to produce action-values. 14 3.5. Categorical Value Estimation: Distributional RL and HL-Gauss Standard value-based reinforcement learning predicts the expectation of the future return. Distributional reinforcement learning instead models the return as a random variable Z(s,a) (Bellemare et al., 2017). Under greedy control, the distributional Bellman relation can be written as Z(s,a) D = R(s,a) + γZ(s ′ ,a ∗ ),(10) where a ∗ = arg max a ′ E [Z(s ′ ,a ′ )],(11) and D = denotes equality in distribution. In C51 (Bellemare et al., 2017), the return distribution is approximated using N categorical support atoms z i ∈ [V min ,V max ], with corresponding probabilities p i (s,a). A Bellman-updated return distribu- tion is projected back onto the fixed categorical support and the network is optimized against this projected target. A related but conceptually distinct approach is to formulate scalar value prediction as a classification problem. Farebrother et al. (Farebrother et al., 2024) showed that categorical prediction of scalar value targets can improve optimization behavior in deep reinforcement learning. In this formulation, the network predicts logits over a fixed set of value bins, while a scalar target y is transformed into a categorical target distribution. In our Phase 3 distributional variants, this transformation is implemented using the Histogram Loss with Gaussian targets (HL-Gauss). Given a scalar target y, HL-Gauss places a Gaussian distribution centered at y and integrates its probability mass over neighboring support intervals. The target probability assigned to bin i can be written as p ∗ i (y) = Z z i +∆/2 z i −∆/2 1 σ √ 2π exp − (x− y) 2 2σ 2 dx,(12) 15 where ∆ denotes the bin width and σ controls the amount of target smoothing. The predicted logits are converted to categorical probabilities using softmax, and the network is trained using cross-entropy: L CE =− X i p ∗ i (y) logp i (s,a).(13) This formulation should be distinguished from C51 (Bellemare et al., 2017). C51 explicitly approximates the full distribution of future returns, whereas HL- Gauss represents a scalar bootstrapped target as a smooth categorical classifi- cation target. In this work, the categorical head is combined with the dueling and ensemble architectures described in Sections 3.4 and 3.6. Figure 5 summarizes the resulting optimization pipeline. Figure 5: Regression-as-classification optimization pipeline. The value network maps a state–action pair (s,a) to logits over fixed value bins, which are converted into predicted probabilities by softmax. Independently, the scalar bootstrapped target y is transformed into a smooth categorical target distribution using HL-Gauss. Training minimizes the cross-entropy between the predicted and target categorical distributions. 16 3.6. Bootstrapped and Ensemble-Based Exploration Bootstrapped DQN (Osband et al., 2016) promotes temporally extended exploration using an ensemble of K value-function heads that share a common feature extractor. Each head Q k (s,a;θ,θ k ) maintains independently initialized parameters θ k and is traditionally trained on a different subset of the observed data, often implemented through Bernoulli bootstrap masks m (k) t ∼ Bernoulli(p).(14) At the beginning of an episode, one head is sampled and used to determine actions for the duration of that episode. Because the heads represent different value estimates, this mechanism approximates posterior sampling and produces temporally coherent exploration rather than independent step-wise random ac- tions. Our Phase 3 ensemble formulation is inspired by this mechanism but differs from classical Bootstrapped DQN in its data assignment: all ensemble heads observe the full synchronous PQN batch (p = 1), while diversity is induced through independent initialization and the trajectories generated by parallel environments. The precise implementation is described in Section 4. 4. Methodology Our experimental methodology follows a progressive three-phase design in- tended to isolate the contributions of visual encoder topology, multiplicative representation mechanisms, and advanced value-estimation heads. Phase 1 eval- uates alternative convolutional backbones within a fixed PQN training frame- work. Phase 2 integrates the Hadamax representation mechanism into the back- bone selected in Phase 1. Phase 3 then holds the resulting encoder fixed while evaluating increasingly expressive value-estimation architectures. Within each phase, optimization settings and training budgets are held fixed except for the architectural component under investigation. Benchmark-specific environment and sampling settings, particularly the Procgen overrides, are de- scribed separately in Section 5. 17 4.1. Phase 1: Convolutional Encoder Architectures The first phase evaluates eight Convolutional Neural Network (CNN) ar- chitectures, denoted Alpha through Theta, against the PQN baseline. The architectures are designed to investigate how convolutional depth, kernel size, channel progression, spatial downsampling, and output dimensionality affect value learning while explicitly tracking differences in model capacity and com- putational cost. All Phase 1 variants are implemented within the same PQN framework (Gal- lici et al., 2024) and receive an identical Atari input consisting of four consecutive grayscale frames stacked into a tensor of shape 4× 84× 84 (Mnih et al., 2013). The learning algorithm, optimizer, training budget, and remaining hyperparam- eters are held fixed across Phase 1. Each convolutional layer in Table 1 is represented by the tuple (C in ,C out ,k,s,p), where C in and C out denote the input and output channel counts, k is the kernel size, s is the stride, and p denotes the amount of zero padding. All Phase 1 convolutional blocks apply two-dimensional Layer Normalization followed by a Rectified Linear Unit (ReLU), ReLU(x) = max(0,x). For each architecture, we separately report the number of parameters and Floating Point Operations (FLOPs) associated with the convolutional encoder and the downstream value-estimation head. This separation is necessary be- cause changes in convolutional stride, kernel size, and padding alter the spatial dimensions of the final feature map and consequently the dimensionality and parameter count of the fully connected head. The Phase 1 architectures are designed to separate topological effects from unconstrained model scaling. Six variants (Beta, Delta, Epsilon, Zeta, Eta, and Theta) use convolutional encoder parameter counts close to the PQN encoder 18 budget of approximately 78k parameters. Alpha and Gamma deliberately de- part from this strict encoder budget to examine increased hierarchical depth while remaining comparatively compact at the complete-model level. Table 1 details each variants’ structure. Encoder parameter count alone is therefore not treated as a sufficient mea- sure of model capacity. Each variant is characterized using encoder parameters, value-head parameters, total parameters, and total FLOPs, shown in Table 2. This distinction is important because a compact convolutional encoder can still produce a large flattened representation and therefore a highly parameterized dense head, as illustrated by the Eta architecture. 4.1.1. Phase 1 Architecture Variants The PQN baseline contains three convolutional layers and produces a flat- tened feature vector of size 3,136, resulting in approximately 1.76M total pa- rameters. Alpha extends the baseline to four convolutional layers and increases the encoder capacity to 174,752 parameters. Its additional depth produces a smaller flattened representation of size 2,304 and evaluates whether increased convolu- tional capacity improves performance without substantially enlarging the com- plete model. Beta also contains four convolutional layers but introduces a channel bot- tleneck in the later stages. The channel dimensionality decreases from 64 to 32 and then 16, testing whether additional depth remains beneficial when the terminal feature representation is compressed. Gamma extends the hierarchy to five convolutional layers while gradually increasing the number of channels from 32 to 48 and then 64. Its encoder con- tains approximately 117k parameters and the complete model approximately 1.84M parameters. Gamma therefore evaluates increased hierarchical depth without the large parameter expansion associated with substantially wider ar- chitectures. Delta retains a three-layer structure but employs a 9× 9 kernel in the first 19 convolution. This configuration evaluates the effect of a larger initial receptive field and more aggressive early spatial reduction while maintaining an encoder parameter count close to that of PQN. Epsilon uses a four-layer topology related to Gamma while remaining close to the baseline encoder parameter budget. Its reduced spatial compression pro- duces a flattened representation of size 4,096, thereby increasing the parameter count of the downstream value head. Zeta uses three convolutional layers with a constant channel width of 48. Although the encoder remains compact, its flattened feature vector has size 4,800, producing approximately 2.61M total parameters. Eta is the shallowest architecture, consisting of two convolutional layers with increased channel width. Limited spatial reduction produces a flattened representation of size 46,208 and consequently a dense head containing approx- imately 23.7M parameters. Eta therefore provides a deliberate test of shallow, high-capacity representation learning. Theta is the most compact complete architecture in Phase 1, with approxi- mately 1.20M total parameters. Its final convolution reduces the channel count to 32, producing a flattened representation of size 2,048 and testing strong ter- minal feature compression. The principal structural differences between Nature DQN, PQN, Gamma, and the subsequent Gamma-Hadamax architecture are illustrated in Figure 3. 4.1.2. Phase 1 Backbone Selection Protocol Since the objective of Phase 1 is to identify a performance–efficiency back- bone rather than simply selecting the architecture with the largest point esti- mate, model selection considers empirical performance together with computa- tional complexity. To assess the robustness of the selection procedure, we repeatedly sam- ple validation subsets of 15 Atari games across 10 randomized splits. Within each split, architectures exhibiting statistically comparable validation perfor- mance are compared according to total parameter count and computational 20 Table 1: Architectural specifications of the Phase 1 CNN feature extractors. Each layer is defined by (Input Channels, Output Channels, Kernel Size, Stride, Padding). All Phase 1 architectures apply two-dimensional Layer Normalization followed by ReLU after each convo- lution. VariantL1L2L3L4L5 PQN(4, 32, 8, 4, 0) (32, 64, 4, 2, 0) (64, 64, 3, 1, 0)– Alpha(4, 32, 4, 2, 1) (32, 64, 4, 2, 1) (64, 64, 3, 2, 1) (64, 64, 5, 1, 0)– Beta(4, 32, 6, 2, 2) (32, 64, 3, 1, 1) (64, 32, 4, 2, 1) (32, 16, 8, 1, 0)– Gamma (4, 32, 3, 2, 1) (32, 48, 3, 2, 1) (48, 64, 3, 1, 0) (64, 64, 3, 2, 0) (64, 64, 3, 1, 0) Delta(4, 24, 9, 4, 0) (24, 48, 5, 2, 0) (48, 96, 3, 1, 0)– Epsilon (4, 32, 3, 2, 1) (32, 48, 3, 2, 1) (48, 64, 3, 2, 0) (64, 64, 3, 1, 0)– Zeta(4, 48, 4, 2, 1) (48, 48, 4, 2, 1) (48, 48, 4, 2, 1)– Eta(4, 64, 4, 4, 0) (64, 128, 3, 1, 0)– Theta(4, 32, 7, 4, 2) (32, 64, 5, 2, 1) (64, 32, 3, 1, 0)– Table 2: Model complexity and computational footprint of the Phase 1 architectures. FLOPs are reported in millions and partitioned between the convolutional encoder and value- estimation head. Variant Enc. Params Head Params Total Params Enc. FLOPs Head FLOPs Total FLOPs PQN78,3041,686,5001,764,8047.7341.6109.347 Alpha174,7521,782,9481,957,70027.5411.61029.151 Beta89,0081,782,9481,871,95661.5151.61063.126 Gamma117,1681,725,3641,842,53222.9011.61024.512 Delta78,5521,850,5881,929,1406.1431.7747.917 Epsilon80,1122,179,8282,259,94013.2522.10115.354 Zeta77,2322,537,3962,614,62825.3622.46227.824 Eta78,40023,739,46023,817,86028.42223.66352.085 Theta76,2881,127,4281,203,7169.0651.05310.118 cost. Among statistically comparable candidates, preference is given to the architecture with lower complexity. The architecture selected through this procedure is subsequently fixed before the Phase 2 experiments. The corresponding selection frequencies and Phase 1 performance comparisons are reported in the Results section. 21 4.2. Phase 2: Hadamax Integration Phase 2 evaluates whether the convolutional topology selected in Phase 1 can be combined effectively with the Hadamax representation mechanism described in Section 3.3. This phase examines the interaction between deeper hierarchical structure and multiplicative feature transformations. In a Hadamax block, two stride-one convolutional projections of the same input are independently normalized and transformed using GELU activations. Their outputs are combined through an element-wise Hadamard product, af- ter which explicit max-pooling performs spatial reduction. Feature transfor- mation and downsampling are therefore separated rather than being jointly implemented by strided convolutions. We evaluate three Hadamax configurations. A detailed description of each is available on Table 3. 1. Hadamax (Baseline). This configuration serves as the Phase 2 control and follows the three-stage spatial hierarchy of the conventional PQN/DQN encoder while replacing each convolutional processing stage with a Hadamax block. The first block uses an 8× 8 convolution followed by 4× 4 max-pooling with stride 4, preserving the aggressive early spatial reduction of the original hierarchy. 2. Gamma-Hadamax-Valid. This architecture transfers the five-stage Gamma channel hierarchy to Hadamax blocks. Convolutional strides are set to one and spatial re- duction is performed explicitly through max-pooling. In Blocks 3 and 5, pooling uses kernel size 3, stride 1, and zero padding. These valid pooling operations reduce the spatial dimensions before the final flattening stage. 3. Gamma-Hadamax-Same. This configuration is identical to Gamma-Hadamax-Valid except for the pooling operations in Blocks 3 and 5, where padding is increased from 0 to 1. The additional padding preserves spatial resolution at these stages, producing a larger final feature map and a larger downstream value head. 22 The comparison therefore tests whether preserving this additional spa- tial information yields sufficient agent-performance gains to justify the increased parameter and computational cost. Table 3: Architectural specifications of the Phase 2 Hadamax variants. Convolutional op- erations are specified as (Input Channels, Output Channels, Kernel Size, Stride, Padding), and max-pooling operations are specified as [Kernel Size, Stride, Padding]. All Hadamax convolutional paths use Layer Normalization and GELU prior to the element-wise Hadamard product. VariantBlock 1Block 2Block 3Block 4Block 5 Hadamax (Baseline) Convolutional(4, 32, 8, 1, 4) (32, 64, 4, 1, 2) (64, 64, 3, 1, 1)– Max Pool[4, 4, 0][2, 2, 0][3, 1, 1]– Gamma-Hadamax-Valid Convolutional(4, 32, 3, 1, 1) (32, 48, 3, 1, 1) (48, 64, 3, 1, 1) (64, 64, 3, 1, 1) (64, 64, 3, 1, 1) Max Pool[2, 2, 0][2, 2, 0][3, 1, 0][2, 2, 0][3, 1, 0] Gamma-Hadamax-Same Convolutional(4, 32, 3, 1, 1) (32, 48, 3, 1, 1) (48, 64, 3, 1, 1) (64, 64, 3, 1, 1) (64, 64, 3, 1, 1) Max Pool[2, 2, 0][2, 2, 0][3, 1, 1][2, 2, 0][3, 1, 1] The Phase 2 selection follows the same performance–complexity principle used in Phase 1. When two configurations exhibit statistically comparable per- formance, the configuration requiring fewer parameters and less computation is preferred for the subsequent value-head experiments. 4.3. Phase 3: Advanced Value-Estimation Head Topologies In Phase 3, the encoder selected in Phase 2, Gamma-Hadamax-Valid, is held fixed. Only the downstream value-estimation architecture is modified. This design isolates the contribution of the value-estimation head from the encoder modifications introduced in the preceding phases. All Phase 3 heads receive the flattened representation produced by the Gamma-Hadamax-Valid encoder. The dense streams use an embedding dimen- sion of 512 and apply Layer Normalization prior to their final output projections. Three configurations are evaluated. 23 4.3.1. Distributional Dueling Head The first configuration combines the dueling decomposition (Wang et al., 2016) with categorical value estimation. Separate value and advantage streams are constructed from the shared encoder representation, and the action-advantage outputs are mean-centered before being combined with the value stream. Each state–action prediction is represented using N = 51 categorical bins spanning V min =−10,V max = 10. Rather than minimizing Mean Squared Error directly against a scalar target, the scalar λ-return is transformed into a smooth categorical target using the HL-Gauss formulation described in Section 3.5. The distributional standard deviation is derived from the configured sigma ratio sigma ratio = 0.75. Additional distributional value clipping is disabled, distributional_value_clip = 0. The dueling decomposition is applied at the logit level. Let ℓ V i (s) denote the value-stream logit for bin i and ℓ A i (s,a) the corresponding action-advantage logit. The combined state–action logit is ℓ i (s,a) = ℓ V i (s) + " ℓ A i (s,a)− 1 |A| X a ′ ℓ A i (s,a ′ ) # .(15) The resulting logits are optimized using cross-entropy against the HL-Gauss categorical target. For action selection, the categorical prediction associated with each action is converted to its expected scalar value, Q(s,a) = N X i=1 p i (s,a)z i ,(16) and greedy action selection is performed using these expected values. For consistency with the experimental terminology used throughout this paper, we refer to this configuration as Distributional Dueling. However, its training objective should be distinguished from the original C51 Bellman- distribution projection: in our implementation, HL-Gauss is applied to a scalar λ-return target. 24 4.3.2. Ensemble Dueling Head The second configuration combines the dueling architecture with an ensemble of (K=10) independently parameterized value heads. The Gamma-Hadamax- Valid encoder is shared across the ensemble, while each head contains its own state-value and action-advantage streams. The design is inspired by Bootstrapped DQN (Osband et al., 2016), in which diversity among independently initialized value heads is used to support tem- porally coherent exploration. In the present study, the ensemble is introduced primarily for this exploratory role rather than as a mechanism for improving value-estimation accuracy through prediction averaging or variance reduction. Our implementation differs from classical bootstrap training in its data- assignment mechanism. The bootstrap inclusion probability is set to (p=1.0), meaning that every head receives the complete synchronous training batch. Consequently, no Bernoulli data masking is used to create separate bootstrap subsets. Diversity among the heads therefore arises from their independent pa- rameterization and initialization rather than from distinct bootstrap samples. The ensemble consequently functions as a shared-encoder, multi-head ex- ploration architecture within the replay-free PQN regime. This configuration allows us to examine whether independently parameterized value heads can provide useful exploratory diversity under highly parallelized online training, without relying on replay-buffer-based bootstrap resampling. Any resulting performance difference is therefore interpreted as an empirical consequence of incorporating the ensemble mechanism rather than as evidence that additional heads intrinsically improve value-estimation accuracy. 4.3.3. Distributional Ensemble Dueling Head: Aftab The final configuration combines the categorical value-estimation mechanism with the K = 10-head ensemble. Each head is a complete Distributional Dueling module and produces |A|× 51 categorical logits. The Gamma-Hadamax-Valid encoder is shared across all heads, while each head maintains independently parameterized value and advantage streams. The 25 categorical configuration uses the same support range, [−10, 10], the same 51-bin discretization, and the same HL-Gauss sigma ratio of 0.75 as the Distributional Dueling configuration. The bootstrap probability remains fixed at p = 1.0. GELU activations are used within the dense value and advantage streams of the final composite model. This unified Gamma-Hadamax-Valid + Distributional Ensemble Dueling configuration constitutes the final architecture evaluated in this work, which we designate Aftab. 4.4. Training Configuration The experiments use the training defaults documented in the released Aftab implementation and summarized in Table 4. Unless a benchmark-specific over- ride is explicitly stated, these settings are held fixed across the evaluated archi- tectural variants. The optimizer is Rectified Adam (RAdam) (Liu et al., 2020) with learning rate 2.5 × 10 −4 , optimizer epsilon ε opt = 10 −5 , and momentum parameters β 1 = 0.9,β 2 = 0.999. The optimizer weight decay is fixed at 0.0. The global gradient norm is clipped at 10.0, and the dense value-estimation streams use an embedding di- mension of 512. Training uses ε-greedy exploration with a linear schedule for the exploration parameter ε. The schedule is annealed over the first 10% of the training hori- zon, after which the terminal exploration level is maintained for the remainder of training. This exploration parameter is distinct from the RAdam optimizer epsilon ε opt = 10 −5 , which serves as a numerical-stability constant in the opti- mizer. 4.4.1. Atari Training Configuration For Atari-57, training uses 128 parallel EnvPool environments (Weng et al., 2022). Each rollout collects 32 transitions from each training environment, yielding a batch size of 128× 32 = 4096. 26 Each rollout batch is divided into 32 mini-batches, 4096 32 = 128, resulting in a mini-batch size of 128. Each collected batch is optimized for two epochs. Each Atari experiment receives a total budget of 200 million observed frames. With the Atari frame skip fixed at four, this corresponds to 200 M 4 = 50 M envi- ronment action steps across the complete vectorized sampler. The 50-million- step count denotes the total interaction budget and not a per-environment bud- get. Atari observations use a stack of four consecutive frames. The environment configuration uses a maximum of 30 no-op actions at reset. Episodic-life termi- nation is enabled for training environments and disabled for test environments. Reward clipping is enabled for both the Atari training and Atari test en- vironment configurations. These options are benchmark-specific and are not automatically transferred to environments that do not support them. The discount factor and λ-return parameter are fixed to γ = 0.99,λ = 0.65. For the scalar Phase 1 and Phase 2 architectures, value estimates are op- timized using Mean Squared Error against the corresponding λ-return targets. The scalar Ensemble Dueling variant in Phase 3 retains the scalar regression ob- jective, whereas Distributional Dueling and Aftab use the HL-Gauss categorical objective described above. LayerNorm2d (Gallici et al., 2024) is applied throughout the convolutional encoders. Phase 1 uses ReLU activations, whereas the Hadamax-based encoders in Phases 2 and 3 use GELU within their multiplicative feature-processing paths. 4.4.2. Procgen-Specific Training Overrides Procgen retains the shared optimization defaults but modifies the vectorized sampling configuration. The number of training environments is reduced from 128 to 64, while the number of rollout steps per update is increased from 32 to 256. Consequently, each Procgen rollout contains 64× 256 = 16384 transitions. Using the same 32 mini-batches results in a Procgen mini-batch size of 16384 32 = 512. Procgen observations are used in their native RGB format with shape 3× 27 64× 64. The implementation queries the EnvPool configuration of each environment and passes only supported options. Atari-specific settings such as no-op ini- tialization, frame skip, frame stacking, episodic-life termination, and EnvPool reward clipping are therefore not passed to Procgen environments when unsup- ported. Because Procgen does not use the Atari frame-skip configuration, the 200- million-frame Procgen training budget corresponds directly to 200 million envi- ronment interaction frames. Further benchmark-specific evaluation details are provided in Section 5.4. 4.5. Deviation from Standard PQN Regularization Our experimental protocol deliberately modifies one component of the stan- dard PQN optimization configuration. Whereas PQN employs explicit regular- ization as part of its stabilization strategy (Gallici et al., 2024), the optimizer weight decay is set to 0.0 for all experiments in this study. The purpose of this modification is empirical. By removing explicit weight decay, all investigated architectures are evaluated under a common unregular- ized optimization setting, allowing architectural effects to be examined without introducing variant-specific regularization strengths. This modification should not be interpreted as a theoretical replacement for the stability mechanisms analyzed in PQN. We do not provide a formal conver- gence guarantee or a global bound on the TD update Jacobian for the result- ing nonlinear function approximators. Operational stability is instead assessed empirically through successful training across the fixed seeds and benchmark environments described in Section 5. 5. Experimental Setup and Evaluation Metrics We evaluate the proposed architectural variants under a controlled experi- mental protocol in which training budgets and optimization settings are held 28 fixed within each benchmark. Atari-57 serves as the primary benchmark for the three-stage architectural study, while Procgen Hard provides a complemen- tary evaluation of the final Aftab architecture under procedurally varying visual environments. The complete implementation, model definitions, training configurations, and experimental results are publicly available at https://github.com/tahashieenavaz/aftab and https://github.com/lorisnanni/aftab. Gameplay videos comparing rep- resentative PQN and Aftab policies are available at https://github.com/ tahashieenavaz/aftab/blob/main/videos.md. 5.1. Atari-57 Evaluation Protocol The primary evaluation uses the complete Atari-57 benchmark (Bellemare et al., 2013). Atari observations are represented as stacks of four consecutive grayscale frames with spatial resolution 84× 84, resulting in an input tensor of shape 4× 84× 84. Training is performed using 128 parallel EnvPool environments (Weng et al., 2022). Each optimization update collects 32 transitions from each environment, producing a rollout batch of 128× 32 = 4096 transitions. The batch is divided into 32 mini-batches of 128 samples and optimized for two epochs. Each Atari experiment receives a total interaction budget of 200 million observed frames. Because Atari uses a frame skip of four, this corresponds to 50 million environment action steps across the complete vectorized sampler. The 50-million-step count therefore represents the total interaction budget and not 50 million steps for each of the 128 parallel environments. The Atari environment configuration uses a maximum of 30 random no-op actions at reset, a frame skip of four, and a frame stack of four. Episodic-life ter- mination is enabled during training but disabled for the test environments. The implementation maintains eight test environments in parallel with the training environments. Test actions are selected greedily, without ε-greedy exploration. 29 The test environments use an independent seed stream relative to the train- ing environments. For a training seed s, the corresponding test environments are initialized using s + 1000, reducing overlap between the stochastic training and evaluation trajectories. 5.2. Atari Performance Metrics For each Atari game, performance is reported using the Human-Normalized Score (HNS), HNS = Score agent − Score random Score human − Score random .(17) The random-agent and human reference scores are taken from the standard Atari reference values reported by Mnih et al. (2015). Under this normaliza- tion, a score of 0 corresponds to random-agent performance and a score of 1 corresponds to the reference human performance level. Because Atari performance distributions can be strongly affected by extreme game-level scores, the primary aggregate statistic is the Interquartile Mean (IQM). The IQM corresponds to the mean over the central 50% of the em- pirical performance distribution and may be written in terms of the empirical quantile function F −1 as IQM = 1 0.75− 0.25 Z 0.75 0.25 F −1 (u)du.(18) We additionally report the median HNS to facilitate comparison with prior Atari-57 literature. The IQM is treated as the primary aggregate statistic be- cause it reduces the influence of exceptionally large scores in individual games while retaining information from a broader portion of the benchmark than the median alone. 5.3. Pairwise Statistical Analysis Aggregate scores are complemented by pairwise statistical analyses across the Atari-57 environments. For each pair of architectures, we apply the Wilcoxon 30 signed-rank test to the paired environment-level performance measurements. This non-parametric test evaluates whether the distribution of paired perfor- mance differences is centered around zero without assuming Gaussianity (Wilcoxon, 1945). Because multiple architecture pairs are evaluated within each experimental phase, the resulting p-values are adjusted using the Holm–Bonferroni step-down procedure (Holm, 1979). The family-wise significance level is fixed at α = 0.05. Importantly, IQM and the Wilcoxon signed-rank test serve different pur- poses in our evaluation. IQM is used as a robust benchmark-level performance aggregate, whereas the Wilcoxon test operates on paired environment-level per- formance measurements. The IQM itself is therefore not used as the input to the Wilcoxon test. We additionally report the Probability of Improvement, P (X > Y ), as a complementary pairwise measure. Values above 0.5 indi- cate that architecture X has a greater probability of outperforming architec- ture Y across the benchmark distribution, whereas a value near 0.5 indicates no clear directional advantage. This provides an interpretable complement to the hypothesis-testing analysis. Where reported, 95% confidence intervals for aggregate metrics are estimated using stratified bootstrap resampling over the benchmark evaluations. 5.4. Procgen Hard Procedural Evaluation Protocol Following the Atari-57 model-selection experiments, the final Aftab archi- tecture is compared with the original PQN baseline on the 16 environments of Procgen under the Hard configuration (Cobbe et al., 2019). The evalu- ated environments are Bigfish, Bossfight, Caveflyer, Chaser, Climber, Coinrun, Dodgeball, Fruitbot, Heist, Jumper, Leaper, Maze, Miner, Ninja, Plunder, and Starpilot. Procgen is treated as a secondary procedural evaluation rather than as an additional model-selection stage. Unlike Atari, whose individual games con- tain comparatively fixed visual and structural regularities, Procgen generates varying level layouts and visual configurations across episodes. The benchmark 31 therefore provides a complementary test of performance under procedural visual and structural variation. Procgen observations are used in their native RGB representation with shape 3× 64× 64. Atari-specific environment options are not applied to Procgen when unsupported by the environment. In particular, Atari frame stacking, no-op initialization, frame skipping, episodic-life termination, and EnvPool-specific Atari reward clipping are not passed to Procgen tasks when those options are unsupported. For Procgen, the implementation automatically changes the vectorized sam- pling configuration to 64 training environments and 256 steps per update. The resulting rollout batch therefore contains 64× 256 = 16384 transitions. Using the same 32 mini-batches as the Atari experiments gives a mini-batch size of 16384 32 = 512. Because Procgen does not use Atari-style frame skipping, its 200-million- frame budget corresponds directly to 200 million environment interaction frames. 5.5. Procgen Normalized Score The reward scales of the 16 Procgen environments differ substantially, and no single human-normalization reference is used in our experiments. We therefore report a within-study Procgen Normalized Score (PNS). Normalization is performed independently for each Procgen environment us- ing the seed-level scores included in the corresponding experimental comparison. For environment e, method m, and seed s, we define PNS m,e,s = G m,e,s − G min e G max e − G min e ,(19) where G m,e,s denotes the raw episodic score and G min e and G max e denote, re- spectively, the minimum and maximum seed-level scores used for normalization within environment e. The resulting per-environment PNS values are aggregated across the 16 Proc- gen tasks using both the median and IQM. Because the normalization limits are derived from the experimental scores themselves, PNS is intended specifically 32 for within-study comparison between PQN and Aftab and should not be inter- preted as an externally standardized Procgen score. 5.6. Learning-Curve Area Under the Curve Terminal Procgen performance characterizes only the final stage of train- ing. We therefore complement terminal PNS with an Area Under the Curve (AUC) analysis to characterize performance accumulated throughout the com- plete training trajectory. For each method m, environment e, and seed s, a raw-score AUC can first be computed as AUC raw m,e,s = Z T 0 G m,e,s (t)dt,(20) where G m,e,s (t) denotes the recorded raw Procgen score at training check- point t. The integral is approximated numerically using the trapezoidal rule over the recorded checkpoints. These raw-score AUC values are retained for per-environment and per-seed diagnostic analyses. Because Procgen reward scales differ substantially across tasks, raw AUC values are not used as the primary cross-environment aggre- gate: directly averaging them would give greater influence to environments with intrinsically larger reward magnitudes. For benchmark-level comparison, we instead integrate the normalized Proc- gen performance trajectory. Let PNS(t) denote the normalized Procgen perfor- mance at training checkpoint t. We define AUC PNS = Z T 0 PNS(t)dt,(21) where T is the complete training horizon. The integral is again approximated using the trapezoidal rule. When the horizontal axis is expressed in millions of environment frames, the complete training horizon is T = 200. We additionally report the normalized Area Under the Curve, 33 nAUC = AUC PNS T ,(22) which represents the average normalized Procgen performance accumulated over the complete training trajectory. Terminal IQM PNS and nAUC therefore capture complementary aspects of performance: terminal IQM PNS measures performance at the end of training, whereas nAUC summarizes the complete learning curve. 5.7. Optimization and Hyperparameter Configuration The principal hyperparameters are summarized in Table 4. Except for the Procgen-specific overrides to the number of training environments and rollout steps per update, the same optimization defaults are retained across the two benchmarks. Training uses ε-greedy exploration with a linear schedule for the exploration parameter ε. The schedule is annealed over the first 10% of the training hori- zon. This exploration parameter is distinct from the RAdam optimizer constant ε opt = 10 −5 . Table 4: Training and optimization settings used in the experiments. Procgen overrides the number of training environments and rollout steps per update; the corresponding batch and mini-batch sizes are therefore derived accordingly. HyperparameterAtari-57Procgen Hard Observation shape4× 84× 843× 64× 64 Training environments12864 Test environments88 Steps per update32256 Batch size4,09616,384 Mini-batches3232 Mini-batch size128512 Continued on next page 34 Table 4 – continued from previous page HyperparameterAtari-57Procgen Hard Training budget200M observed frames 200M environment frames Frame skip4– Frame stack4– Maximum no-op actions30– Training episodic lifeTrue– Test episodic lifeFalse– Training reward clippingTrue– Test reward clippingTrue– Learning rate2.5× 10 −4 OptimizerRectified Adam (RAdam) (Liu et al., 2020) Optimizer ε opt 1× 10 −5 Optimizer β 1 0.9 Optimizer β 2 0.999 Weight decay0.0 Discount factor γ0.99 λ-return parameter0.65 Epochs per update2 Gradient-norm clipping10.0 Embedding dimension512 ε-greedy scheduleLinear ε annealing ratio10% of training Phase 3 categorical and ensemble settings Categorical support bins51 Categorical support[−10, 10] HL-Gauss sigma ratio0.75 Distributional value clip0.0 Ensemble heads10 Continued on next page 35 Table 4 – continued from previous page HyperparameterAtari-57Procgen Hard Bootstrap inclusion probability1.0 Scalar configurations in Phases 1 and 2 are optimized using Mean Squared Error against the corresponding λ-return targets. The scalar Ensemble Duel- ing configuration in Phase 3 retains the regression objective, whereas Distribu- tional Dueling and Aftab use the HL-Gauss categorical cross-entropy objective described in Section 3.5. 5.8. Hardware and Reproducibility All reported architectures are evaluated using four fixed random seeds: 475284, 219842, 525975, 909314. These correspond to the reproducibility seeds dis- tributed with the public Aftab implementation through aftab_seeds. The same seed set is used throughout the reported experimental comparisons unless explicitly stated otherwise. Environment interaction and training are implemented in PyTorch (Paszke et al., 2019), with EnvPool used for vectorized environment simulation. Experiments were executed on NVIDIA A40 GPUs with 48 GB of GDDR6 ECC memory and a memory bandwidth of 696 GB/s. The most computationally demanding configuration requires approximately 13 hours of wall-clock training time on the reported hardware. The complete source code, model configurations, raw result logs, normalized score tables, learning curves, and statistical comparison figures are provided in the public Aftab repository. The experimental results reported in this paper were produced using the PyTorch implementation. A JAX implementation is under development and is not used for any result reported here. 36 6. Results This section reports the empirical findings of the three-stage architectural evaluation conducted on Atari-57, followed by the complementary evaluation of the final Aftab architecture on Procgen Hard. The first three subsections follow the progressive experimental design introduced in Section 4: Phase 1 evaluates convolutional encoder topology, Phase 2 examines the integration of Hadamax representations, and Phase 3 evaluates advanced value-estimation heads. Proc- gen is reported separately because it is used to evaluate the final selected archi- tecture under procedural variation rather than as an additional model-selection stage. 6.1. Phase 1: CNN Encoder Capacity and Topological Efficiency The first phase isolated the structural design of the convolutional feature extractor to evaluate its effect on Atari-57 performance. As detailed in Table 2, the computational footprint of each variant was partitioned between the encoder and the value-estimation head, allowing performance differences to be considered jointly with parameter count and computational cost. The evaluated architectures exhibited substantial differences in their parameter– computation trade-offs. Eta represents the most extreme case of spatial overpa- rameterization: although its convolutional encoder remains relatively compact, its large flattened representation expands the value-estimation head to approx- imately 23.7 million parameters. In contrast, Gamma requires approximately 1.84 million total parameters and 24.5 million FLOPs while introducing a deeper five-layer convolutional hierarchy. Tables 5 and 6 report the Atari-57 Human-Normalized Scores (HNS) for all evaluated architectures. Among the Phase 1 variants, Alpha achieved the highest aggregate IQM HNS of 3.566, followed closely by Gamma at 3.508. Both exceeded the PQN baseline IQM of 2.715. As the objective of Phase 1 was not solely to maximize the largest aggregate point estimate, architectural selection additionally considered model complexity. 37 Gamma requires approximately 1.84M total parameters compared with 1.96M for Alpha, and 24.5M FLOPs compared with 29.2M for Alpha. The absolute difference in IQM HNS between Alpha and Gamma is only 0.058. Pairwise Wilcoxon signed-rank tests were conducted using paired environment- level Atari-57 measurements. The corresponding uncorrected and Holm–Bonferroni- corrected p-value matrices are shown in Figure 6. These tests complement the aggregate IQM analysis by examining the consistency of paired game-level dif- ferences rather than differences between aggregate IQM point estimates them- selves. To assess whether the choice of Gamma depended strongly on a particular subset of Atari environments, we additionally repeated the architecture-selection procedure across 10 randomized validation splits, each containing 15 games. Within each split, architectures exhibiting statistically comparable validation performance were compared according to parameter count and computational cost. Gamma was selected in all 10 trials under this performance–complexity criterion, indicating that its selection was stable across the sampled validation subsets. The Probability of Improvement analysis provides a complementary pairwise view of the Phase 1 results and is shown in Figure 7. Together, the aggregate scores, pairwise statistical analysis, and randomized performance–complexity selection procedure motivated the use of Gamma as the backbone for Phase 2. Although Alpha attained the largest Phase 1 IQM point estimate, Gamma provided a closely matched aggregate score with fewer parameters and lower computational cost. Table 5: Full Phase 1 Encoder Results (Part 1: PQN to Delta). PQNAlphaBetaGammaDelta Alien0.5511.8212.4212.8060.370 Amidar0.6401.3400.8631.0990.545 Assault30.71823.87728.18633.85426.660 Asterix44.57915.71015.74314.82835.490 Asteroids0.0361.9660.3261.4870.230 Atlantis45.57943.77445.78046.25643.365 Bank Heist1.8941.9902.1231.7251.828 Continued on next page 38 Table 5 – continued from previous page PQNAlphaBetaGammaDelta Battlezone1.1971.3931.8261.9141.231 Beamrider1.2112.8602.2812.5481.094 Berzerk2.4565.3680.3431.2821.306 Bowling0.0440.0960.0370.1350.072 Boxing8.2088.3258.2908.2918.325 Breakout11.91814.73115.05017.30711.621 Centipede0.8181.2820.9661.4320.679 Chopper Command2.74426.64850.14863.2900.845 Crazy Climber6.2446.8267.8616.6576.789 Defender3.1604.5444.3926.2553.166 Demon Attack72.50872.86773.20672.82770.875 Double Dunk7.7407.8537.9927.7397.595 Enduro2.7302.7372.7052.6852.728 Fishing Derby2.5262.6182.5442.5822.463 Freeway1.1321.1481.1381.1121.137 Frostbite1.5692.6232.2942.1911.174 Gopher24.35027.62034.58328.83415.861 Gravitar0.2320.2530.3900.3350.189 H.E.R.O.0.7860.8360.7560.7330.694 Ice Hockey0.8401.1931.0771.9160.696 James Bond8.26612.3466.74918.3584.829 Kangaroo4.5294.8064.7644.6123.883 Krull7.5608.4898.6538.3217.208 Kung-Fu Master1.4051.7851.4701.3581.596 Montezuma’s Revenge0.0000.0000.0000.0000.000 Ms. Pac-Man0.4670.7240.9220.8430.507 Name This Game2.3022.8432.5383.2422.344 Phoenix26.28239.26136.56335.2637.772 Pitfall!0.0320.0330.0230.0330.034 Pong1.1811.1811.1811.1811.181 Private Eye0.0180.0000.0010.001-0.000 Q*bert1.5551.8711.8531.8051.332 River Raid1.3271.6411.6731.7841.421 Road Runner7.35310.69410.31722.7647.498 Robotank7.0947.5947.2736.9757.236 Seaquest0.1900.2400.1920.1970.199 Skiing-0.5220.451-0.182-0.324-0.518 Solaris0.1210.0780.1560.1870.083 Space Invaders4.9894.0048.8305.1944.768 Stargunner27.41038.65244.04542.91325.046 Surround1.0701.1371.1661.1220.923 Tennis1.3652.2941.3692.6771.460 Time Pilot5.92415.06721.04614.7224.790 Tutankham1.5301.5811.5831.5151.499 Up’n Down23.56323.16623.68824.24617.208 Venture0.0000.0000.0210.0000.000 Video Pinball318.311371.878312.513373.361361.368 Wizard of Wor4.5187.0597.1296.6683.270 Yars’ Revenge2.2392.6172.5702.5771.823 Zaxxon1.8312.1062.5702.3191.772 Median1.8942.6182.4212.5771.596 IQM2.7153.5663.4643.5082.388 IQM 95% CI[1.541, 4.477] [2.138, 6.292] [1.909, 6.242] [2.160, 7.182] [1.393, 3.870] 39 (a) Pairwise Wilcoxon signed-rank p-values.(b) Holm–Bonferroni-corrected p-values. Figure 6: Phase 1 statistical significance matrices. Pairwise Wilcoxon signed-rank tests computed from paired Atari-57 environment-level performance measurements. Holm– Bonferroni correction is applied across the 36 pairwise comparisons to control the family-wise error rate at α = 0.05. Table 6: Full Phase 1 Encoder Results (Part 2: Epsilon to Theta). EpsilonZetaEtaTheta Alien2.3841.8531.6400.401 Amidar1.3580.7031.0460.672 Assault36.76630.38333.37934.165 Asterix10.61113.26218.96945.196 Asteroids1.0501.8480.0570.026 Atlantis47.56446.54144.99646.273 Bank Heist1.6902.1301.7171.959 Battlezone2.0191.5061.0351.039 Beamrider2.0851.4921.2581.130 Berzerk1.5053.1450.8241.478 Bowling0.0600.0890.1110.057 Boxing8.3258.3028.3258.320 Breakout13.04412.71416.23114.331 Centipede0.9640.6391.1570.531 Chopper Command44.04133.1133.1951.494 Crazy Climber7.1916.7116.8516.090 Defender3.8663.9573.1926.064 Demon Attack72.66672.11071.40470.828 Double Dunk7.8197.7318.1137.719 Enduro2.7122.7022.6992.700 Fishing Derby2.6492.6252.5572.486 Freeway1.1381.1401.1401.135 Frostbite1.4912.0671.7681.095 Gopher20.64724.55832.24819.611 Gravitar0.2690.3160.5280.125 H.E.R.O.0.7730.8871.0760.448 Ice Hockey1.0341.0681.2420.645 Continued on next page 40 Table 6 – continued from previous page EpsilonZetaEtaTheta James Bond28.34612.11014.5498.973 Kangaroo4.5204.7614.5704.647 Krull8.1288.6537.9687.693 Kung-Fu Master1.4261.4471.4451.830 Montezuma’s Revenge0.0000.0020.0040.000 Ms. Pac-Man0.7390.8461.1190.667 Name This Game2.2672.2071.8792.598 Phoenix31.54944.39240.79514.803 Pitfall!0.0330.0340.0330.030 Pong1.1801.1811.1811.181 Private Eye0.0000.0000.0000.001 Q*bert1.7591.8641.8971.667 River Raid1.7261.8411.4661.397 Road Runner10.83311.0517.9757.113 Robotank7.1057.3016.5246.758 Seaquest0.2080.4110.4100.193 Skiing0.5650.4770.520-0.480 Solaris0.1180.1050.1910.295 Space Invaders5.9561.72815.4704.586 Stargunner32.75432.03324.76923.882 Surround1.1471.2031.1920.936 Tennis2.2661.3661.8741.388 Time Pilot12.47012.74914.5164.490 Tutankham1.4871.5101.6231.533 Up’n Down27.05028.1717.97424.838 Venture0.0000.0000.0000.000 Video Pinball298.742372.805374.307335.655 Wizard of Wor7.7486.0378.0705.707 Yars’ Revenge2.3282.7402.6832.194 Zaxxon1.6041.7312.2081.933 Median2.2662.0671.8791.830 IQM3.3413.1993.1062.688 IQM 95% CI[1.851, 6.077] [1.802, 5.791] [1.708, 5.692] [1.535, 4.504] 41 Figure 7: Probability of Improvement for the Phase 1 encoders. Pairwise probability matrix P(Row > Column) across the Atari-57 benchmark. Values above 0.50 indicate a directional advantage for the row architecture under this metric. 6.2. Phase 2: Hadamax Representation Integration Following the selection of Gamma in Phase 1, the second experimental phase evaluated whether the deeper convolutional hierarchy could be combined effec- tively with Hadamax multiplicative feature interactions. The computational characteristics of the resulting architectures are summarized in Table 7. Table 7: Computational footprint of the Phase 2 Hadamax integration architectures. Floating Point Operations (FLOPs) are reported in millions. VariantEnc. Params Head Params Total Params Enc. FLOPs Head FLOPs Total FLOPs Gamma117,1681,725,3641,842,53222.9011.61024.512 Hadamax (Baseline)156,6083,968,5164,125,124159.0143.969162.984 Gamma-Hadamax-Valid234,3361,609,2201,843,556122.0011.610123.611 Gamma-Hadamax-Same234,3363,280,3883,514,724129.3003.281132.581 The integration produced markedly different complexity profiles. Gamma- 42 Hadamax-Valid retained essentially the same total parameter count as the orig- inal Gamma architecture, with 1.844M and 1.843M parameters, respectively, despite replacing the standard convolutional transformations with the two-path Hadamax formulation. In contrast, the Hadamax baseline required approx- imately 4.13M parameters and 163.0M FLOPs. Gamma-Hadamax-Same in- creased the total parameter count to approximately 3.51M because retaining the later spatial dimensions enlarged the flattened representation supplied to the dense head. The corresponding Atari-57 results are reported in Table 8. Gamma achieved an IQM HNS of 3.508. Introducing the Hadamax formulation increased the IQM to 4.712 for the three-block Hadamax baseline, 5.254 for Gamma-Hadamax- Valid, and 5.360 for Gamma-Hadamax-Same. Gamma-Hadamax-Same therefore attained the largest Phase 2 IQM point estimate, but the difference relative to Gamma-Hadamax-Valid was only 0.106. This small aggregate difference was accompanied by a substantial complexity increase: Gamma-Hadamax-Same contains approximately 3.51M parameters compared with 1.84M for Gamma-Hadamax-Valid. The pairwise Wilcoxon signed-rank analyses are shown in Figure 8a. The corrected comparison supports a performance difference between the Gamma backbone and Gamma-Hadamax-Valid, whereas the Valid and Same variants do not exhibit a statistically detectable difference under the specified family- wise significance criterion. Since Gamma-Hadamax-Same did not provide a statistically established advantage sufficient to justify its much larger dense head, the performance– complexity selection criterion favored Gamma-Hadamax-Valid. This configura- tion was therefore fixed as the shared encoder for Phase 3. The corresponding Probability of Improvement matrix is shown in Figure 9, providing a complementary view of the environment-level directional differences among Gamma and the Hadamax based variants. 43 (a) Pairwise Wilcoxon signed-rank p-values.(b) Holm–Bonferroni-corrected p-values. Figure 8: Phase 2 statistical significance matrices. Pairwise Wilcoxon signed-rank tests for the Hadamax integration experiments. Holm–Bonferroni correction is applied to control the family-wise error rate at α = 0.05. Table 8: Phase 2 Atari-57 Results (HNS). GammaHadamax-Gamma-Valid Hadamax-Gamma-SameHadamax Alien2.8062.2623.1432.956 Amidar1.0991.5381.1700.814 Assault33.85440.43045.00042.586 Asterix14.82837.49142.53232.479 Asteroids1.4871.3462.0061.003 Atlantis46.25643.38237.87244.555 Bank Heist1.7252.1132.0462.108 Battlezone1.9142.3621.9122.275 Beamrider2.5482.7094.1262.594 Berzerk1.28223.23021.63310.371 Bowling0.1350.1310.0340.046 Boxing8.2918.2488.3258.321 Breakout17.30714.30519.64920.023 Centipede1.4322.7723.1092.261 Chopper Command63.29066.259105.04127.477 Crazy Climber6.6577.8928.7038.946 Defender6.25521.93221.03020.902 Demon Attack72.82774.64274.36174.521 Double Dunk7.7397.3887.9617.807 Enduro2.6852.7302.7332.711 Fishing Derby2.5822.5132.6342.523 Freeway1.1121.1461.1481.140 Frostbite2.1911.4462.7292.408 Gopher28.83437.09727.95926.749 Gravitar0.3350.4220.5750.577 H.E.R.O.0.7331.2091.1881.115 Ice Hockey1.9163.7013.4981.342 James Bond18.35843.67944.75419.589 Kangaroo4.6124.7224.3524.491 Continued on next page 44 Table 8 – continued from previous page GammaHadamax-Gamma-Valid Hadamax-Gamma-SameHadamax Krull8.3218.8008.5818.820 Kung-Fu Master1.3581.0690.9691.741 Montezuma’s Revenge0.0000.0000.0010.000 Ms. Pac-Man0.8431.1951.6001.272 Name This Game3.2423.4053.1113.368 Phoenix35.26337.00144.99831.553 Pitfall!0.0330.0340.0340.033 Pong1.1811.1811.1811.181 Private Eye0.0010.001-0.0000.000 Q*bert1.8053.3223.0062.091 River Raid1.7841.7872.1401.712 Road Runner22.76429.29437.24422.406 Robotank6.9757.6437.5657.174 Seaquest0.1975.5224.9141.540 Skiing-0.3240.6360.6490.501 Solaris0.1870.0910.1340.157 Space Invaders5.19418.61619.90119.767 Stargunner42.91363.30762.75453.536 Surround1.1221.1551.1401.167 Tennis2.6773.0843.0753.078 Time Pilot14.72212.2179.6299.367 Tutankham1.5151.7011.4601.477 Up’n Down24.24622.75325.15924.811 Venture0.0000.2030.0000.028 Video Pinball373.361321.522306.502357.007 Wizard of Wor6.6687.7585.6514.370 Yars’ Revenge2.5778.1925.5197.117 Zaxxon2.3192.2672.3722.829 Median2.5773.3223.1432.829 IQM3.5085.2545.3604.712 IQM 95% CI[2.182, 7.036][2.990, 10.524][3.034, 11.111][2.635, 9.232] 45 Figure 9: Probability of Improvement for the Hadamax integration experiments. Pairwise probability matrix comparing the Gamma backbone with the Hadamax-based vari- ants. 6.3. Phase 3: Advanced Value-Estimation Heads The third phase evaluated advanced value-estimation heads while keeping the Gamma-Hadamax-Valid encoder fixed. This design isolates the contribution of the Distributional Dueling, Ensemble Dueling, and combined Distributional Ensemble Dueling formulations from the encoder modifications established in the preceding phases. Table 9 reports the aggregate Atari-57 IQM HNS and 95% confidence in- tervals for the Phase 3 comparison. Confidence intervals are estimated us- ing stratified bootstrap resampling across the four fixed experimental seeds 475284, 219842, 525975, 909314 All three advanced value-estimation configurations substantially increased the aggregate IQM point estimate relative to PQN. Distributional Dueling reached an IQM HNS of 6.093, Ensemble Dueling reached 5.625, and Aftab 46 Table 9: Phase 3 aggregate Atari-57 performance. Distributional Dueling, Ensemble Du- eling, and Aftab use the Gamma-Hadamax-Valid encoder. 95% confidence intervals are estimated using stratified bootstrap resampling across the four fixed experimental seeds 475284, 219842, 525975, 909314. Architecture ConfigurationIQM HNS (95% CI) PQN2.715 [1.533, 4.486] Gamma-Hadamax-Valid + Distributional Dueling 6.093 [3.418, 10.868] Gamma-Hadamax-Valid + Ensemble Dueling5.625 [3.279, 11.988] Aftab6.592 [3.524, 13.536] attained the highest Phase 3 IQM of 6.592, with a 95% confidence interval of [3.524, 13.536]. For reference, the scalar Gamma-Hadamax-Valid encoder selected at the end of Phase 2 achieved an IQM HNS of 5.254. The Phase 3 results therefore indicate that the categorical and ensemble value-estimation mechanisms can be integrated successfully with the selected encoder, with Aftab attaining the largest aggregate point estimate among the evaluated final configurations. The bootstrap confidence intervals are relatively broad and overlap substan- tially. Accordingly, the aggregate point estimates should be interpreted together with the environment-level pairwise analysis rather than as evidence that every numerical difference among the Phase 3 heads is statistically resolved. Figure 10 reports the corresponding Probability of Improvement matrix. This analysis provides a complementary environment-level comparison of the advanced heads and the PQN baseline without relying exclusively on the mag- nitude of aggregate HNS values. Detailed per-environment Atari-57 results are reported in Table 10. The cor- responding raw evaluation returns and individual learning curves are provided in the Supplementary Material. 47 Figure 10: Probability of Improvement for the Phase 3 value-estimation architec- tures. Pairwise comparison of PQN and the advanced value-estimation heads evaluated in the final experimental phase. Table 10: Phase 3 Atari-57 Results (HNS). PQNDistributional Dueling Ensemble DuelingAftab Alien0.5512.9484.0793.201 Amidar0.6401.5651.8391.717 Assault30.71833.03540.91138.166 Asterix44.57940.60540.65233.011 Asteroids0.0360.9200.8900.627 Atlantis45.57941.33446.637150.786 Bank Heist1.8942.2432.2222.139 Battlezone1.1973.4242.9273.353 Beamrider1.2114.3723.1794.073 Berzerk2.45611.01117.44320.114 Bowling0.0440.1060.1950.237 Boxing8.2088.3258.3178.312 Breakout11.91821.47624.15150.740 Centipede0.81818.5643.59521.325 Chopper Command2.74486.611126.862119.595 Crazy Climber6.2448.0577.7616.812 Defender3.16018.04223.06431.197 Demon Attack72.50874.11874.4807.215 Double Dunk7.7407.8537.856-1.085 Enduro2.7302.7292.6976.846 Fishing Derby2.5262.4702.5922.554 Continued on next page 48 Table 10 – continued from previous page PQNDistributional Dueling Ensemble DuelingAftab Freeway1.1321.1491.1491.148 Frostbite1.5692.2342.8522.258 Gopher24.35020.59633.02519.747 Gravitar0.2320.5890.8401.368 H.E.R.O.0.7861.7181.3042.172 Ice Hockey0.8403.9203.6694.374 James Bond8.26671.45733.223109.931 Kangaroo4.5294.6504.6554.825 Krull7.56011.2398.87511.900 Kung-Fu Master1.4051.2681.4561.295 Montezuma’s Revenge0.0000.0000.0260.032 Ms. Pac-Man0.4671.6061.8061.496 Name This Game2.3024.2613.1374.172 Phoenix26.28276.02773.512165.021 Pitfall!0.0320.0340.0330.033 Pong1.1811.1811.1811.179 Private Eye0.0180.0000.0090.001 Q*bert1.5553.3763.6363.281 River Raid1.3272.1142.5832.146 Road Runner7.35326.38143.57936.413 Robotank7.0947.7238.0047.736 Seaquest0.19011.1628.63855.555 Skiing-0.5220.0950.655-0.223 Solaris0.1210.0010.1570.012 Space Invaders4.98922.33441.28043.490 Stargunner27.41063.40872.28965.620 Surround1.0701.1931.1571.207 Tennis1.3653.0733.0783.078 Time Pilot5.92410.2479.33015.709 Tutankham1.5301.4461.6361.599 Up’n Down23.56326.80325.46130.842 Venture0.0000.0000.0000.506 Video Pinball318.311355.299405.4441153.948 Wizard of Wor4.5188.5675.9368.513 Yars’ Revenge2.2395.5415.5525.659 Zaxxon1.8313.6612.1554.837 Median1.8944.2613.6364.374 IQM2.7156.0935.6256.592 IQM 95% CI[1.533, 4.486][3.418, 10.868][3.279, 11.988][3.524, 13.536] 49 (a) Pairwise Wilcoxon signed-rank p-values.(b) Holm–Bonferroni-corrected p-values. Figure 11: Phase 3 statistical significance matrices. Pairwise Wilcoxon signed-rank tests computed from paired Atari-57 environment-level performance measurements for PQN, Distributional Dueling, Ensemble Dueling, and Aftab. Panel (a) reports the uncorrected p- values, while panel (b) reports the corresponding Holm–Bonferroni-adjusted p-values. The correction controls the family-wise error rate at α = 0.05. 6.4. Performance on Procgen Hard After completing model selection on Atari-57, we evaluated the final Aftab architecture on the 16 Procgen environments under the Hard configuration. Unlike Phases 1–3, this experiment was not used to select an additional archi- tecture. Instead, the already selected Aftab configuration was compared with the original PQN baseline to examine its behavior under procedurally varying visual environments. Across the Procgen Hard suite, Aftab achieved an IQM Procgen Normalized Score (PNS) of 0.418, with a 95% confidence interval of [0.125, 0.741]. PQN achieved an IQM PNS of 0.382, with a corresponding 95% confidence interval of [0.194, 0.541]. Aftab therefore attained the higher IQM point estimate under the prespecified primary aggregate metric. The median provides a complementary view of the result. PQN attained a median PNS of 0.414, whereas Aftab attained 0.325. Thus, the aggregate ranking is not uniform across summary statistics: Aftab is favored by the IQM, while PQN is favored by the median. This difference reflects the heterogeneous 50 pattern of performance across the 16 Procgen tasks and motivates examination of both terminal and learning-curve-level results. To characterize performance throughout training, we additionally evaluated the Area Under the Curve of the normalized Procgen trajectories using the procedure defined in Section 5.6. Over the 200-million-frame training horizon, Aftab achieved a PNS-based AUC of 108.13, compared with 43.21 for PQN. The corresponding normalized AUC values are nAUC Aftab = 108.13 200 = 0.541, and nAUC PQN = 43.21 200 = 0.216. Aftab therefore accumulated approximately 2.50× the normalized perfor- mance of PQN over the complete evaluated learning trajectory. This learning- curve result is stronger than the difference in terminal IQM and indicates that Aftab maintained higher normalized performance for a larger portion of the common training budget. The environment-level raw-score AUC analysis provides an additional di- agnostic view. After computing AUC independently within each game and averaging across the four seeds, Aftab obtained the higher raw-score AUC in 12 of the 16 Procgen Hard environments. PQN retained the higher raw-score AUC in Caveflyer, Coinrun, Jumper, and Ninja. These raw AUC values are used only for within-environment diagnostic comparisons and are not directly averaged as the primary benchmark-level statistic, because reward magnitudes differ substantially between Procgen environments, Terminal performance was less uniformly favorable to Aftab. Based on fi- nal per-environment scores, Aftab exceeded PQN in 7 of the 16 environments: Bigfish, Chaser, Dodgeball, Fruitbot, Leaper, Maze, and Miner. PQN obtained the higher terminal score in the remaining nine environments. The Procgen result should therefore be interpreted as a higher primary IQM and substan- 51 tially stronger normalized learning-curve AUC rather than as uniform domi- nance across all tasks. Figure 12 summarizes the benchmark-level Procgen learning trajectories over the full 200-million-frame budget. Terminal IQM PNS characterizes the final aggregate performance, whereas nAUC summarizes performance accumulated throughout training. Complete per-environment learning curves, raw terminal returns, and environment-level AUC results are provided in the Supplementary Material. Overall, the Procgen experiment shows that the performance gains obtained by Aftab on Atari are not confined to a single fixed-layout benchmark. How- ever, because the magnitude and direction of the differences vary substantially across Procgen environments, we restrict the interpretation to performance un- der procedural variation rather than making a stronger claim of universal or out-of-distribution generalization. Figure 12: Procgen Hard procedural evaluation. Aggregate learning trajectories of PQN and Aftab across the 16 Procgen Hard environments over 200M training frames. Terminal IQM PNS measures final aggregate performance, while normalized Area Under the Curve (nAUC) summarizes normalized performance throughout training. 52 6.5. Contextual Comparison with Prior Atari-57 Agents Finally, Table 11 places the final Aftab result in the context of representative Atari-57 agents reported in the literature. Median HNS is used because it is widely reported in prior Atari studies. Table 11: Contextual comparison of Median Human-Normalized Scores (HNS) reported for representative Atari-57 agents evaluated at a nominal budget of 200M frames. Differences in evaluation protocol and algorithmic infrastructure should be considered when interpreting these values. AlgorithmMedian HNSReference DQN0.79(Mnih et al., 2015) Double DQN1.15(Van Hasselt et al., 2016) Dueling DQN1.51(Wang et al., 2016) Rainbow2.31(Hessel et al., 2018) Hadamax-PQN3.10(Kooi et al., 2025) GDI11.46(Fan & Xiao, 2022) MuZero7.31(Schrittwieser et al., 2020) Ours (Aftab)4.374This Work Aftab achieves a Median HNS of 4.374 under the evaluation protocol used in this study. This value exceeds those reported for DQN, Double DQN, Dueling DQN, Rainbow, and Hadamax-PQN in the cited studies, while remaining below the reported values of GDI and MuZero. These comparisons are intended only to provide broader performance con- text. The algorithms listed in Table 11 differ in training procedure, computa- tional requirements, use of replay buffers and target networks, implementation details, and potentially evaluation protocol. Consequently, the table should not be interpreted as a controlled head-to-head benchmark or as evidence of state-of-the-art performance. The principal comparison of this study remains the controlled evaluation against PQN and the architectural ablations conducted under the common ex- 53 perimental protocol described in Section 5. Within that setting, the progressive architectural study increases aggregate Atari-57 performance from the PQN IQM HNS of 2.715 to an Aftab IQM HNS of 6.592 while preserving the buffer- free PQN training paradigm. 7. Discussion and Limitations 7.1. Interpretation of the Three-Phase Architectural Study The three experimental phases collectively indicate that architectural struc- ture matters substantially within the buffer-free PQN training regime. Per- formance differences were observed not only when the number of parameters changed, but also when models with comparable overall capacity differed in convolutional hierarchy, spatial reduction, multiplicative feature interactions, or value-estimation topology. The results therefore support the view that pa- rameter count alone is insufficient to characterize the behavior of value networks in this setting. 7.1.1. Phase 1: Encoder Depth, Spatial Structure, and Capacity Phase 1 provides the clearest evidence that increasing raw parameter ca- pacity is not sufficient to guarantee improved Atari-57 performance. Alpha achieved the highest Phase 1 IQM HNS of 3.566, followed closely by Gamma at 3.508, compared with 2.715 for PQN. Alpha contains four convolutional layers, whereas Gamma contains five, both extending the three-layer hierarchy used by the PQN baseline. These results are consistent with the hypothesis that moderately deeper con- volutional hierarchies can provide useful representations for Atari control. How- ever, depth is not manipulated independently of kernel sizes, strides, channel progression, and output feature-map dimensions in these architectures. The ob- served advantage should therefore be attributed to the complete encoder topol- ogy rather than to depth alone. 54 (a) Full training duration.(b) Final 50M frames (zoomed). Figure 13: Phase 1 aggregate Atari-57 performance (IQM HNS). Human-Normalized Score (HNS) is normalized such that 0 corresponds to the random-agent reference score and 1 to the reference human score. The deeper Alpha and Gamma encoders attain the strongest aggregate Phase 1 performance among the evaluated variants. The right panel provides a magnified view of the final 50M frames. These results show an association between the eval- uated deeper encoder topologies and improved Atari performance, but do not by themselves establish convolutional depth as the sole causal factor. The comparison with Eta further illustrates the distinction between struc- tured feature extraction and raw parameter capacity. Eta contains approx- imately 23.8 million total parameters, primarily because its shallow encoder produces a very large flattened representation for the dense value head. Despite this large capacity, its IQM HNS is 3.106, below Alpha, Beta, and Gamma. By contrast, Gamma reaches an IQM of 3.508 with approximately 1.84 million parameters. Similarly, Zeta reaches an IQM HNS of 3.199 with approximately 2.61 million parameters. These observations indicate that enlarging a dense downstream representation does not necessarily reproduce the performance gains obtained through changes to convolutional topology. The Phase 1 selection procedure also illustrates why Gamma was preferred over Alpha despite Alpha obtaining the slightly higher IQM point estimate. Their aggregate IQMs differ by only 0.058, while Gamma requires approximately 1.84M total parameters and 24.5M FLOPs compared with approximately 1.96M parameters and 29.2M FLOPs for Alpha. Moreover, Gamma was selected in all 10 randomized 15-game validation splits under the predefined performance– 55 complexity criterion. The choice of Gamma should therefore be interpreted as a performance–efficiency decision rather than a claim that it achieved the highest absolute Phase 1 score. 7.1.2. Phase 2: Multiplicative Feature Interactions Phase 2 shows that the Gamma hierarchy remains effective when its stan- dard convolutional processing stages are replaced by the Hadamax formula- tion. Gamma-Hadamax-Valid increases the IQM HNS from 3.508 for Gamma to 5.254, while retaining nearly the same total parameter count: approximately 1.844M versus 1.843M. The three-stage Hadamax baseline reaches an IQM HNS of 4.712 but re- quires approximately 4.13M parameters and 163.0M FLOPs. This comparison suggests that the benefit observed in Phase 2 cannot be explained simply by adding the Hadamax operation or increasing parameter capacity. Instead, the results are consistent with an interaction between the multiplicative represen- tation mechanism and the spatial hierarchy in which it is embedded. Gamma-Hadamax-Same obtains the highest Phase 2 IQM point estimate of 5.360, slightly above the 5.254 obtained by Gamma-Hadamax-Valid. However, the Same variant contains approximately 3.51M parameters, compared with 1.84M for the Valid variant. The pairwise analysis does not establish a sta- tistically detectable advantage of Same over Valid under the corrected testing procedure. Consequently, Gamma-Hadamax-Valid provides the more favorable performance–complexity trade-off and was selected as the common encoder for Phase 3. Importantly, these results should not be interpreted as demonstrating that Hadamard interactions universally improve reinforcement-learning representa- tions. The experiment evaluates a specific family of architectures under a fixed training protocol. The results establish an empirical advantage for the tested Gamma-Hadamax configurations, while the mechanism responsible for that ad- vantage remains an open question. 56 (a) Full training duration.(b) Final 50M frames (zoomed). Figure 14: Phase 2 aggregate Atari-57 performance (IQM HNS). The Gamma- Hadamax variants improve aggregate performance relative to the original Gamma encoder under the evaluated training protocol. Gamma-Hadamax-Same achieves the largest IQM point estimate, whereas Gamma-Hadamax-Valid provides the more favorable performance– complexity trade-off. 7.1.3. Phase 3: Advanced Value Estimation in a Buffer-Free Regime Phase 3 demonstrates that categorical value estimation, dueling decompo- sition, and multi-head value estimation can be integrated with the buffer-free PQN framework without introducing an experience replay buffer. Using the selected Gamma-Hadamax-Valid encoder, Distributional Dueling reaches an IQM HNS of 6.093, Ensemble Dueling reaches 5.625, and Aftab reaches the highest Phase 3 IQM of 6.592, with a 95% confidence interval of [3.524, 13.536]. For comparison, the original PQN baseline reaches an IQM HNS of 2.715, while the scalar Gamma-Hadamax-Valid configuration selected after Phase 2 reaches 5.254. The Phase 3 confidence intervals are relatively broad and overlap substan- tially. Consequently, the ordering of the IQM point estimates should not be interpreted as establishing statistically resolved superiority between every pair of advanced heads. The stronger conclusion is that all three advanced value- estimation configurations can operate effectively within the evaluated buffer-free framework, with Aftab producing the largest aggregate IQM point estimate. The categorical components used in Distributional Dueling and Aftab should 57 also be interpreted carefully. As described in Section 4.3, the implementation converts scalar λ-return targets into categorical HL-Gauss targets. It there- fore does not implement the original C51 Bellman-distribution projection. The observed gains should consequently be attributed to the evaluated categorical value-estimation objective rather than to distributional reinforcement learning in the strict C51 sense. The ensemble component similarly differs from classical Bootstrapped DQN (Osband et al., 2016). Aftab uses K = 10 independently parameterized heads with bootstrap inclusion probability p = 1.0, so all heads receive the same synchronous training batch. The method therefore does not create classical bootstrap datasets through Bernoulli masking. Any benefit associated with the ensemble must arise despite this shared data assignment and cannot be attributed directly to conventional bootstrap resampling. These distinctions are important because they restrict several possible causal interpretations. In particular, the present experiments do not establish that the multi-head architecture approximates posterior sampling, implements Thomp- son sampling, prevents convergence to sub-optimal policies, or reduces value- estimation variance. Such mechanisms are plausible subjects for future analysis, but they are not directly measured by the experiments reported here. (a) Full training duration.(b) Final 50M frames (zoomed). Figure 15: Phase 3 aggregate Atari-57 performance (IQM HNS). Advanced value- estimation heads are evaluated using the selected Gamma-Hadamax-Valid encoder. Aftab obtains the largest aggregate IQM point estimate, reaching 6.592. The right panel magnifies the final 50M frames of training. 58 7.2. Optimization Stability in the Buffer-Free Setting The successful training of the Phase 3 architectures is notable because temporal- difference learning with nonlinear function approximation can exhibit unsta- ble optimization behavior, particularly in settings involving bootstrapping and changing data distributions (Sutton & Barto, 2018). Aftab nevertheless com- pleted the reported training runs under the common experimental configuration without requiring experience replay or optimizer weight decay. This observation should be interpreted as empirical operational stability rather than as evidence that the architecture eliminates the theoretical insta- bility associated with nonlinear temporal-difference learning. The experiments do not provide a formal convergence result, a global contraction argument, or a bound on the TD update Jacobian. Several components may contribute to the observed behavior, including Layer Normalization, large synchronous batches, gradient-norm clipping, the selected encoder topology, and the common optimizer configuration. Because these mechanisms were not independently ablated for stability, their individual causal contributions cannot be determined from the present study. The same caution applies to the absence of weight decay. The experiments demonstrate that all evaluated configurations could be trained with optimizer weight decay fixed at zero under the reported settings. They do not establish that weight decay is unnecessary in PQN generally or that the architectural changes provide an equivalent theoretical stabilization mechanism. 7.3. Performance Under Procedural Variation The Procgen Hard experiment provides a complementary assessment of Aftab after all architecture selection had been completed on Atari-57. Under the pri- mary terminal aggregate statistic, Aftab reaches an IQM PNS of 0.418 with a 95% confidence interval of [0.125, 0.741], whereas PQN reaches an IQM PNS of 0.382 with a confidence interval of [0.194, 0.541]. The difference in terminal IQM is therefore modest relative to the uncer- tainty of the estimates. Moreover, the median produces the opposite ordering: 59 PQN reaches a median PNS of 0.414, compared with 0.325 for Aftab. This indicates that the Procgen advantage is not uniform across aggregate summary statistics. The learning-curve analysis provides a stronger distinction. Aftab achieves a PNS-based AUC of 108.13, corresponding to an nAUC of 0.541, whereas PQN achieves an AUC of 43.21 and an nAUC of 0.216. Thus, Aftab accumulates approximately 2.50 times the normalized performance of PQN across the full training trajectory under the metric defined in Section 5.6. The environment-level results further emphasize the heterogeneous nature of this improvement. At the terminal checkpoint, Aftab exceeds PQN in 7 of the 16 Procgen environments: Bigfish, Chaser, Dodgeball, Fruitbot, Leaper, Maze, and Miner. For example, the raw terminal score on Bigfish increases from 7.233 for PQN to 31.491 for Aftab, while Dodgeball increases from 0.192 to 12.596. However, PQN obtains the larger terminal score in the remaining nine environments. The current raw Procgen result table therefore does not support a claim of uniform dominance. The individual terminal scores reported for these environments are descriptive rather than evidence of a statistically established environment-specific effect. The learning-curve picture is broader: the raw-score AUC diagnostic favors Aftab in 12 of the 16 environments and PQN in Caveflyer, Coinrun, Jumper, and Ninja. Because raw reward scales vary substantially across Procgen tasks, these per-environment raw AUC comparisons are not used as a direct cross-task aggregate; the normalized PNS-based AUC remains the primary learning-curve comparison. Accordingly, the Procgen experiment supports the narrower conclusion that the Aftab architecture retains useful performance characteristics under procedu- ral variation and accumulates substantially more normalized performance during training under the evaluated protocol. It does not establish out-of-distribution generalization in the stronger sense of evaluation on a formally specified held- out distribution of levels. No such claim is required for the observed Procgen result. 60 7.4. Statistical Interpretation and Multiple-Comparison Control The statistical analysis was designed to complement, rather than replace, the aggregate IQM comparisons. The IQM is used as the primary robust benchmark-level aggregate because Atari performance distributions contain extreme game-level values. By averag- ing the central 50% of the empirical distribution, IQM reduces the influence of both very small and exceptionally large normalized scores. The pairwise Wilcoxon signed-rank tests, however, are not performed on the IQM values. Instead, they operate on paired environment-level performance measurements. The two statistics therefore answer different questions: IQM summarizes aggregate benchmark performance, whereas the Wilcoxon test ex- amines the consistency of paired performance differences across environments. Because Phase 1 contains nine models, its complete pairwise analysis com- prises 36 comparisons. The corresponding p-values are adjusted using the Holm–Bonferroni step-down procedure to control the family-wise error rate at α = 0.05. The same correction principle is applied within the subsequent ex- perimental phases. This distinction is particularly important when interpreting individual ar- chitectures. For example, Delta obtains an IQM HNS of 2.388, below the PQN value of 2.715, while Theta obtains 2.688, close to the baseline. These observa- tions are consistent with the possibility that aggressive early spatial reduction or terminal feature compression can be detrimental under the tested config- urations. However, the experiment changes multiple architectural properties simultaneously, and therefore cannot uniquely attribute Delta’s performance to its 9× 9 first-layer kernel or Theta’s performance to its terminal channel width. The statistical matrices are consequently best interpreted as evidence about differences between complete architectures, not as causal tests of individual architectural components. 61 7.5. Limitations Several limitations define the scope of the conclusions that can be drawn from this study. First, the empirical study is restricted primarily to discrete-action visual reinforcement learning. Atari-57 and Procgen Hard provide considerable diver- sity in visual structure and task dynamics, but they do not establish that the same architectural choices will transfer to continuous-control, partially observed, language-conditioned, or substantially longer-horizon domains. Evaluating the Hadamax encoder and advanced value-estimation heads on benchmarks such as the DeepMind Control Suite (Tassa et al., 2018) would therefore provide an important test of their broader applicability. Second, the experimental design intentionally emphasizes controlled archi- tectural comparison rather than architecture-specific hyperparameter optimiza- tion. Within each benchmark, the principal optimization settings are held fixed across the relevant comparisons, with only the documented Procgen sampling overrides. Consequently, the reported scores should be interpreted as perfor- mance under a common protocol rather than as the maximum achievable per- formance of each architecture. Architecture-specific tuning of the learning rate, λ-return parameter, batch configuration, categorical support, or ensemble set- tings could alter the relative results. Third, all reported experiments use four fixed random seeds. Although this is sufficient for the controlled comparisons performed here and is complemented by bootstrap confidence intervals and paired environment-level analyses, a larger number of independent seeds would provide more precise estimates of between- run variability, particularly for the Phase 3 and Procgen comparisons, whose confidence intervals remain broad. Fourth, the Procgen Normalized Score is a within-study normalization. Its minimum and maximum values are derived from the compared experimental scores rather than from fixed external reference values. PNS is therefore useful for comparing PQN and Aftab within this experiment but should not be treated as a standardized Procgen metric for comparison with unrelated studies. 62 Fifth, the Procgen experiment evaluates performance under procedurally varying environments but does not define a separate held-out level distribution in the experimental protocol. The results therefore support claims regarding procedural variation, but not a stronger claim of formal out-of-distribution gen- eralization. Sixth, the three-phase experimental sequence is intentionally progressive: Gamma is selected in Phase 1, Gamma-Hadamax-Valid in Phase 2, and the Phase 3 heads are then evaluated on that fixed backbone. This design makes the experimental search tractable and interpretable, but it does not exhaus- tively evaluate every possible encoder–Hadamax–value-head combination. In- teractions between discarded Phase 1 encoders and the Phase 3 heads therefore remain unmeasured. Finally, we do not provide a formal convergence or stability guarantee for the unregularized nonlinear temporal-difference updates used in this study. Baseline PQN motivates explicit regularization as part of its stability analysis, whereas our experiments use optimizer weight decay equal to zero. The empirical results demonstrate successful training under the reported architecture, normalization, batching, and gradient-clipping settings, but they do not establish a closed-form spectral, Lipschitz, or TD-Jacobian bound for Aftab. Understanding how convolutional topology, multiplicative feature interac- tions, normalization, multi-head value estimation, and large-scale parallel sam- pling jointly influence the optimization dynamics of buffer-free temporal-difference learning remains an important direction for future theoretical and empirical work. 8. Conclusion This study introduced Aftab, an open-source benchmarking framework for systematically evaluating convolutional encoder topology, multiplicative fea- ture interactions, and advanced value-estimation heads within the parallelized, buffer-free PQN training regime. Through a progressive three-phase study on 63 Atari-57, followed by a complementary evaluation on Procgen Hard, we exam- ined how architectural design affects performance when the underlying training framework and interaction budget are held fixed. The principal findings of the study can be summarized as follows: • Encoder topology matters beyond parameter count. In Phase 1, Alpha achieved the highest IQM HNS among the base encoder variants at 3.566, while Gamma achieved a closely matched IQM of 3.508. Gamma, however, required approximately 1.84 million total parameters and 24.5 million FLOPs, compared with approximately 1.96 million parameters and 29.2 million FLOPs for Alpha. Moreover, Gamma was selected in all 10 randomized 15-game validation splits under the predefined performance– complexity criterion. In contrast, the substantially larger Eta architecture, with approximately 23.8 million total parameters, achieved an IQM HNS of 3.106. These results indicate that increasing model capacity alone does not reproduce the gains associated with the more effective convolutional topologies identified in this study. • Multiplicative feature interactions improve the selected back- bone under the evaluated protocol. In Phase 2, replacing the Gamma convolutional processing stages with Hadamax blocks substantially in- creased aggregate Atari-57 performance. Gamma-Hadamax-Valid achieved an IQM HNS of 5.254 while retaining approximately the same total param- eter count as Gamma (1.844M versus 1.843M). Gamma-Hadamax-Same achieved the largest Phase 2 IQM point estimate of 5.360, but required approximately 3.51 million parameters. Because the performance differ- ence between the Valid and Same variants was not statistically resolved under the corrected pairwise analysis, Gamma-Hadamax-Valid provided the more favorable performance–complexity trade-off and was selected for the final phase. • Advanced value-estimation heads can operate effectively in a buffer-free parallelized setting. With the Gamma-Hadamax-Valid 64 encoder fixed, Phase 3 evaluated categorical Distributional Dueling, En- semble Dueling, and their combined formulation. Distributional Dueling achieved an IQM HNS of 6.093, Ensemble Dueling achieved 5.625, and Aftab obtained the highest aggregate IQM HNS of 6.592. The categori- cal variants use HL-Gauss classification of scalar λ-return targets rather than the original C51 Bellman-distribution projection, while the ensemble uses K = 10 independently parameterized heads with bootstrap inclusion probability p = 1.0. The results therefore demonstrate that these value- estimation mechanisms can be integrated successfully into the replay-free PQN framework without requiring a target network or experience replay buffer. • The final architecture retains useful performance under procedu- ral variation. On Procgen Hard, Aftab achieved an IQM Procgen Nor- malized Score (PNS) of 0.418, compared with 0.382 for PQN. The median produced the opposite ordering, with 0.325 for Aftab and 0.414 for PQN, emphasizing that the terminal advantage was heterogeneous across tasks. The learning-curve analysis showed a substantially larger difference: Aftab achieved a PNS-based AUC of 108.13 and an nAUC of 0.541, compared with 43.21 and 0.216 for PQN, respectively. Thus, Aftab accumulated approximately 2.50× the normalized performance of PQN over the com- plete training trajectory. These results support improved learning-curve performance under procedural variation, while not constituting evidence of formal out-of-distribution generalization. Taken together, the results show that architectural choices remain highly consequential even within streamlined, highly parallelized value-learning sys- tems. The progression from the PQN baseline IQM HNS of 2.715 to the fi- nal Aftab IQM HNS of 6.592 was achieved through successive changes to en- coder topology, multiplicative feature processing, and value-estimation structure rather than through unconstrained parameter scaling. At the same time, the results emphasize that these components should be evaluated jointly with com- 65 putational cost and statistical uncertainty: the largest architecture or largest point estimate was not always the configuration ultimately selected. Aftab should therefore be viewed as both a concrete architecture and a reproducible experimental framework for studying neural design choices in par- allelized, replay-free Q-learning. The present results establish its effectiveness under the reported Atari-57 and Procgen Hard protocols, but they do not pro- vide a formal convergence guarantee or establish that the observed architectural relationships will transfer unchanged to other reinforcement learning domains. Future work will investigate whether the identified design principles transfer to continuous-control, partially observed, and longer-horizon environments, and whether architecture-specific optimization can further improve the performance of the selected components. Additional experiments with larger numbers of random seeds and explicit held-out procedural evaluation protocols would also provide stronger estimates of statistical variability and generalization. Finally, extending the framework to the planned JAX implementation may enable sub- stantially higher-throughput experimentation and make broader architectural searches computationally practical. References Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450, . Bellemare, M. G., Dabney, W., & Mnih, V. (2017). A distributional perspec- tive on reinforcement learning. In International Conference on Machine Learning (p. 449–458). PMLR. Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013). The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47, 253–279. URL: https://jair.org/ index.php/jair/article/view/10819. doi:10.1613/jair.3912. 66 Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Amiranashvili, A., Brox, T., & Peters, J. (2019). Crossq: Batch normalization in deep reinforce- ment learning for greater sample efficiency and simplicity. arXiv preprint arXiv:1902.05605, . URL: https://arxiv.org/abs/1902.05605. Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., & Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs. URL: http://github.com/jax-ml/jax. Chen, X., Wang, C., Zhou, Z., & Ross, K. (2021). Randomized ensem- bled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, . URL: https://arxiv.org/abs/2101.05982. Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2019). Leveraging pro- cedural generation to benchmark reinforcement learning. arXiv preprint arXiv:1912.01588, . Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Un- terthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. URL: https://arxiv.org/abs/2010.11929. arXiv:2010.11929. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., & Kavukcuoglu, K. (2018). Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Con- ference on Machine Learning (p. 1407–1416). Fan, J., & Xiao, C. (2022). Generalized data distribution iteration. URL: https://arxiv.org/abs/2206.03192. arXiv:2206.03192. Farebrother, J., Orbay, J., Vuong, Q., Taïga, A. A., Chebotar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., Kumar, A., & Agarwal, 67 R. (2024). Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03950, . Gallici, M., Fellows, M., Ellis, B., Pou, A., Masmitja, E., Foerster, J., & Martin, M. (2024). Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.04811, . He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (p. 770–778). Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, . URL: https://arxiv.org/abs/1606.08415. Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Cotton, B., & Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scan- dinavian Journal of Statistics, 6, 65–70. doi:10.2307/4615733. Kooi, J. E., Yang, Z., & François-Lavet, V. (2025). Hadamax encoding: Ele- vating performance in model-free atari. arXiv preprint arXiv:2505.15345, . URL: https://arxiv.org/abs/2505.15345. doi:10.48550/arXiv.2505. 15345. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learn- ing applied to document recognition. Proceedings of the IEEE, 86, 2278– 2324. Lee, H., Hwang, D., Kim, D., Kim, H., Tai, J.-J., Subramanian, K., Wurman, P. R., Choo, J., Stone, P., & Seno, T. (2024). Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, . URL: https://arxiv.org/abs/2410.09754. 68 Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., & Han, J. (2020). On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations (ICLR). Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, . Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G. et al. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. doi:10.1038/nature14236. Nauman, M., Ostaszewski, M., Jankowski, K., Miłoś, P., & Cygan, M. (2024). Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in Neural Information Processing Systems. URL: https://arxiv.org/abs/2405.16158. Osband, I., Blundell, C., Pritzel, A., & Van Roy, B. (2016). Deep exploration via bootstrapped dqn. arXiv preprint arXiv:1602.04621, . Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., & Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 32 (p. 8024– 8035). Curran Associates, Inc. URL: http://papers.neurips.c/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., & 69 Silver, D. (2020). Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588, 604–609. URL: http://dx.doi.org/10. 1038/s41586-020-03051-4. doi:10.1038/s41586-020-03051-4. Schwarzer, M., Obando Ceron, J. S., Courville, A., Bellemare, M. G., Agar- wal, R., & Castro, P. S. (2023). Bigger, better, faster: Human-level Atari with human-level efficiency. In Proceedings of the 40th Interna- tional Conference on Machine Learning (p. 30365–30380). PMLR. URL: https://proceedings.mlr.press/v202/schwarzer23a.html. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., & Hassabis, D. (2017). Master- ing the game of go without human knowledge. Nature, 550, 354–359. URL: https://doi.org/10.1038/nature24270. doi:10.1038/nature24270. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning series (2nd ed.). Cambridge, MA: The MIT Press. Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T., & Riedmiller, M. (2018). Deepmind control suite. arXiv preprint arXiv:1801.00690, . Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 30. URL: https://ojs.aaai.org/index.php/AAAI/ article/view/10295. Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., & Freitas, N. (2016). Dueling network architectures for deep reinforcement learning. In Interna- tional Conference on Machine Learning (ICML) (p. 1995–2003). Weng, J., Huang, M. L., Huang, S., Bo, H., & Makoviichuk, D. (2022). En- 70 vpool: A highly parallel reinforcement learning environment execution en- gine. arXiv preprint arXiv:2206.10558, . Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1, 80–83. Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2017). Learning transferable architectures for scalable image recognition. arXiv preprint arXiv:1707.07012, . URL: https://arxiv.org/abs/1707.07012. 71 Supplementary Material for: Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks Taha Shieenavaz 1 (tahashieenavaz@gmail.com), Shabnam Zareshahraki 1 (shabnamzaresh@gmail.com), Loris Nanni 1 (loris.nanni@unipd.it) 1 Department of Information Engineering, University of Padua, Italy Corresponding author at: Department of Information Engineering, University of Padua, Italy. Taha Shieenavaz Email: tahashieenavaz@gmail.com Shabnam Zareshahraki Email: shabnamzaresh@gmail.com Document Description: This document acts as the dedicated companion file for the main text, housing un-normalized raw evaluation returns across all 57 benchmark tasks along with their full operational step histories. 1. Receptive Field Analysis The receptive field (RF) of a convolutional neural network at layer l calcu- lates the spatial region of the input volume that influences a single feature map activation. It is defined recursively based on the kernel size (k) and stride (s) of the current and preceding layers: RF l = RF l−1 + (k l − 1) l−1 Y i=1 s i (S1) where we assume an initial receptive field RF 0 = 1 and an initial stride product of 1. Below, we detail the step-by-step calculation of the maximum theoretical receptive field for both the baseline Nature DQN Encoder, Hadamax Encoder, and our proposed Gamma Encoder and Hadamax Gamma Encoders. Table S1: Summary of the maximum theoretical receptive fields for the Nature DQN, standard Gamma, baseline Hadamax, and Gamma-Hadamax encoders. Encoder ArchitectureEffective Receptive Field Nature DQN Encoder36× 36 Gamma Encoder39× 39 Hadamax Encoder59× 59 Hadamax Gamma Encoder (Valid & Same)70× 70 1.1. Nature DQN Encoder The Nature DQN architecture consists of three convolutional layers with aggressive early spatial downsampling (a stride of 4 in the first layer). The receptive field expands as follows: Layer 1 (k = 8, s = 4) : RF 1 = 1 + (8− 1)× 1 = 8 Layer 2 (k = 4, s = 2) : RF 2 = 8 + (4− 1)× 4 = 20 Layer 3 (k = 3, s = 1) : RF 3 = 20 + (3− 1)× (4× 2) = 36 The total effective receptive field for the Nature DQN encoder maps to a 36×36 spatial patch on the original input. 1 1.2. Gamma Encoder The Gamma architecture replaces large filters with a deeper, five-layer stack of 3× 3 convolutions. This preserves spatial resolution longer before downsam- pling. The calculation is as follows: Layer 1 (k = 3, s = 2) : RF 1 = 1 + (3− 1)× 1 = 3 Layer 2 (k = 3, s = 2) : RF 2 = 3 + (3− 1)× 2 = 7 Layer 3 (k = 3, s = 1) : RF 3 = 7 + (3− 1)× (2× 2) = 15 Layer 4 (k = 3, s = 2) : RF 4 = 15 + (3− 1)× (2× 2× 1) = 23 Layer 5 (k = 3, s = 1) : RF 5 = 23 + (3− 1)× (2× 2× 1× 2) = 39 The total effective receptive field for the Gamma encoder maps to a 39× 39 spatial patch. This demonstrates that despite using exclusively small 3× 3 kernels, the depth of the Gamma architecture allows it to achieve a marginally larger global receptive field than the Nature DQN baseline, while simultaneously supporting a highly complex, non-linear hypothesis space. 1.3. Hadamax Encoder The Hadamax variant of the Nature DQN architecture replaces standard convolutional strides with explicit Max Pooling steps for downsampling. Be- cause each block contains both a convolution (c) and a pooling (p) operation, the receptive field expands in two stages per block: Block 1 Conv (k = 8, s = 1) : RF c1 = 1 + (8− 1)× 1 = 8 Block 1 Pool (k = 4, s = 4) : RF p1 = 8 + (4− 1)× 1 = 11 Block 2 Conv (k = 4, s = 1) : RF c2 = 11 + (4− 1)× (1× 4) = 23 Block 2 Pool (k = 2, s = 2) : RF p2 = 23 + (2− 1)× 4 = 27 Block 3 Conv (k = 3, s = 1) : RF c3 = 27 + (3− 1)× (4× 2) = 43 Block 3 Pool (k = 3, s = 1) : RF p3 = 43 + (3− 1)× 8 = 59 The total effective receptive field for the Hadamax encoder maps to a 59× 59 spatial patch. 2 1.4. Hadamax Gamma Encoder (Valid & Same) The Hadamax Gamma architecture utilizes a five-block deep configuration. Note that while Valid and Same differ in their pooling padding, padding does not alter the maximum receptive field calculation of a single unit. Thus, the calculation for both is identical: Block 1 Conv (k = 3, s = 1) : RF c1 = 1 + (3− 1)× 1 = 3 Block 1 Pool (k = 2, s = 2) : RF p1 = 3 + (2− 1)× 1 = 4 Block 2 Conv (k = 3, s = 1) : RF c2 = 4 + (3− 1)× (1× 2) = 8 Block 2 Pool (k = 2, s = 2) : RF p2 = 8 + (2− 1)× 2 = 10 Block 3 Conv (k = 3, s = 1) : RF c3 = 10 + (3− 1)× (2× 2) = 18 Block 3 Pool (k = 3, s = 1) : RF p3 = 18 + (3− 1)× 4 = 26 Block 4 Conv (k = 3, s = 1) : RF c4 = 26 + (3− 1)× (4× 1) = 34 Block 4 Pool (k = 2, s = 2) : RF p4 = 34 + (2− 1)× 4 = 38 Block 5 Conv (k = 3, s = 1) : RF c5 = 38 + (3− 1)× (4× 2) = 54 Block 5 Pool (k = 3, s = 1) : RF p5 = 54 + (3− 1)× 8 = 70 The total effective receptive field for the Hadamax Gamma encoder maps to a 70× 70 spatial patch. This deeper, decoupled pooling approach significantly expands the receptive field compared to both the standard Gamma architecture (39× 39) and the baseline Hadamax (59× 59). 3 2. Individual Environment Learning Curves (Figures) PQN Alpha Beta Gamma Delta Epsilon Zeta Eta Theta AlienAmidarAssaultAsterix AsteroidsAtlantisBank HeistBattlezone BeamriderBerzerkBowlingBoxing BreakoutCentipedeChopper CmdCrazy Climber Figure S1: Phase 1 Learning Curves: Human-Normalized Score learning curves over 200 million training frames for the baseline CNN encoders (Alpha through Theta) across all 57 Atari games. (Part 1 of 4) 4 DefenderDemon AttackDouble DunkEnduro Fishing DerbyFreewayFrostbiteGopher GravitarH.E.R.O.Ice HockeyJames Bond KangarooKrullKung Fu MstrMontezuma’s Figure S1: Phase 1 Learning Curves: Human-Normalized Score learning curves over 200 million training frames for the baseline CNN encoders (Alpha through Theta) across all 57 Atari games. (Part 2 of 4) 5 Ms. Pac-ManName ThisPhoenixPitfall! PongPrivate EyeQ*bertRiver Raid Road RunnerRobo-TankSeaquestSkiing SolarisSpace Inv.Star GunnerSurround Figure S1: Phase 1 Learning Curves: Human-Normalized Score learning curves over 200 million training frames for the baseline CNN encoders (Alpha through Theta) across all 57 Atari games. (Part 3 of 4) 6 TennisTime PilotTutankhamUp ’n Down VentureVid. PinballWizard of WorYars’ Revenge Zaxxon Figure S1: Phase 1 Learning Curves: Human-Normalized Score learning curves over 200 million training frames for the baseline CNN encoders (Alpha through Theta) across all 57 Atari games. (Part 4 of 4) 7 Gamma Hadamax-Gamma-Valid Hadamax-Gamma-Same Hadamax AlienAmidarAssaultAsterix AsteroidsAtlantisBank HeistBattlezone BeamriderBerzerkBowlingBoxing BreakoutCentipedeChopper CmdCrazy Climber Figure S2: Phase 2 Learning Curves:Human-Normalized Score learning curves over 200 million training frames comparing the standard Gamma baseline to the Hadamax-integrated variants across all 57 Atari games. (Part 1 of 4) 8 DefenderDemon AttackDouble DunkEnduro Fishing DerbyFreewayFrostbiteGopher GravitarH.E.R.O.Ice HockeyJames Bond KangarooKrullKung Fu MstrMontezuma’s Figure S2: Phase 2 Learning Curves: Human-Normalized Score learning curves over 200 million training frames comparing the standard Gamma baseline to the Hadamax-integrated variants across all 57 Atari games. (Part 2 of 4) 9 Ms. Pac-ManName ThisPhoenixPitfall! PongPrivate EyeQ*bertRiver Raid Road RunnerRobo-TankSeaquestSkiing SolarisSpace Inv.Star GunnerSurround Figure S2: Phase 2 Learning Curves: Human-Normalized Score learning curves over 200 million training frames comparing the standard Gamma baseline to the Hadamax-integrated variants across all 57 Atari games. (Part 3 of 4) 10 TennisTime PilotTutankhamUp ’n Down VentureVid. PinballWizard of WorYars’ Revenge Zaxxon Figure S2: Phase 2 Learning Curves: Human-Normalized Score learning curves over 200 million training frames comparing the standard Gamma baseline to the Hadamax-integrated variants across all 57 Atari games. (Part 4 of 4) 11 PQN Distributional Duelling Ensemble Duelling Aftab AlienAmidarAssaultAsterix AsteroidsAtlantisBank HeistBattlezone BeamriderBerzerkBowlingBoxing BreakoutCentipedeChopper CmdCrazy Climber Figure S3: Phase 3 Learning Curves: Human-Normalized Score learning curves over 200 million training frames for the advanced value regression architectures (Dueling, Ensemble, and Distributional) across all 57 Atari games. (Part 1 of 4) 12 DefenderDemon AttackDouble DunkEnduro Fishing DerbyFreewayFrostbiteGopher GravitarH.E.R.O.Ice HockeyJames Bond KangarooKrullKung Fu MstrMontezuma’s Figure S3: Phase 3 Learning Curves: Human-Normalized Score learning curves over 200 million training frames for the advanced value regression architectures (Dueling, Ensemble, and Distributional) across all 57 Atari games. (Part 2 of 4) 13 Ms. Pac-ManName ThisPhoenixPitfall! PongPrivate EyeQ*bertRiver Raid Road RunnerRobo-TankSeaquestSkiing SolarisSpace Inv.Star GunnerSurround Figure S3: Phase 3 Learning Curves: Human-Normalized Score learning curves over 200 million training frames for the advanced value regression architectures (Dueling, Ensemble, and Distributional) across all 57 Atari games. (Part 3 of 4) 14 TennisTime PilotTutankhamUp ’n Down VentureVid. PinballWizard of WorYars’ Revenge Zaxxon Figure S3: Phase 3 Learning Curves: Human-Normalized Score learning curves over 200 million training frames for the advanced value regression architectures (Dueling, Ensemble, and Distributional) across all 57 Atari games. (Part 4 of 4) 15 PQNAftab Big FishBoss FightCave FlyerChaser ClimberCoin RunDodgeballFruit Bot HeistJumperLeaperMaze MinerNinjaPlunderStarpilot Figure S4: Procgen Hard Procedural Evaluation: Un-normalized evaluation returns over 200 million training frames comparing the baseline PQN to the complete Aftab framework across the 16 individual Procgen Hard environments. 16 Table S1: Phase 1 Atari-57 Raw Returns: Final unnormalized evaluation scores for the PQN baseline and the eight parameter-constrained CNN encoders (Alpha through Theta). (Part 1 of 2) PQNAlphaBetaGammaDelta Alien4688.19713152.10914752.17718963.3332680.850 Amidar1082.5002178.1531139.7311893.639974.150 Assault17009.49312128.39114126.83718030.88813932.092 Asterix366434.014129816.327154993.197126304.082235632.653 Asteroids2312.721103959.76215110.74868774.32011182.687 Atlantis742852.041728122.789751796.599742603.741736686.735 Bank Heist1402.1431506.3951595.0001288.9121332.789 Battlezone44176.87143829.93264149.66064085.03444642.857 Beamrider22017.97347062.18439400.66739152.43518069.789 Berzerk5877.85715391.463958.5711942.2793578.537 Bowling30.00037.83027.72430.95931.333 Boxing99.228100.000100.00099.587100.000 Breakout352.044436.816451.425504.044326.320 Centipede10076.69013596.03411219.79317675.8508680.939 Chopper Command17155.442118263.946334721.088 360188.0956479.252 Crazy Climber161068.367179992.517 204894.898179469.728185854.422 Defender51787.24575035.37466311.90591159.69453549.320 Demon Attack132035.357132607.364 133369.286132650.221128967.942 Double Dunk-1.782-1.238-1.095-1.707-1.980 Enduro2360.7212357.7412325.9592341.7622353.371 Fishing Derby42.28646.74842.82343.27238.197 Freeway33.82333.98333.82732.97333.653 Frostbite6653.9469596.7019046.8039532.9254842.925 Gopher53851.97364661.76972346.05451893.33334766.463 Gravitar956.122645.4081319.5581293.367730.102 H.E.R.O.24341.20725199.86421372.44924353.57120274.881 Ice Hockey0.1772.1360.89514.364-3.622 James Bond2195.0682500.8501569.0485012.2451360.374 Kangaroo13455.44214392.85714387.07514087.75511813.605 Krull9707.58510637.38110961.36110724.4569399.592 Kung-Fu Master31990.13638857.82331007.82335474.83034184.014 Montezuma’s Revenge0.0000.0000.0000.0000.000 Ms. Pac-Man3365.3405302.4156122.8915855.2383515.544 Name This Game15329.01419033.57116205.74821340.51015285.782 Phoenix177903.333213915.986235491.803190653.19751112.245 Pitfall!-5.3440.000-21.1050.0000.000 Pong21.00021.00021.00021.00021.000 Private Eye66.66798.639100.00087.3339.524 Q*bert21923.46925177.29625397.87425847.95917267.262 River Raid21349.01427019.49027153.29929396.32723497.007 Road Runner57719.04872455.44263845.918175828.57156056.463 Robotank71.16776.07572.41268.75272.170 Seaquest7978.02710092.9257835.6468228.8448466.871 Skiing-23644.677 -9432.129-18793.803-22986.874-23272.214 Solaris2605.3062069.9322881.8373081.4972004.762 Space Invaders5342.2962655.68011072.2452855.5786468.639 Stargunner259520.408371468.027 423331.293403316.667241495.918 Surround8.3508.8579.2489.1466.071 Tennis-1.33311.833-4.31623.881-1.092 Time Pilot13860.88427697.61932540.81627330.27211938.095 Tutankham249.473255.932264.330254.279259.340 Up’n Down263719.048253879.762295438.095266537.007164273.027 Continued on next page 17 Table S1 – continued from previous page PQNAlphaBetaGammaDelta Venture0.0000.0000.0000.0000.000 Video Pinball457305.670542947.592448438.116529722.776510972.653 Wizard of Wor16897.95928960.54429923.46929785.71413953.401 Yars’ Revenge115017.827136848.190138392.262140728.207101737.813 Zaxxon16136.73517865.30624046.93920938.43514489.116 Table S2: Phase 1 Atari-57 Raw Returns: Final unnormalized evaluation scores for the PQN baseline and the eight parameter-constrained CNN encoders (Alpha through Theta). (Part 2 of 2) EpsilonZetaEtaTheta Alien17525.57811592.44911572.4153209.184 Amidar2362.0101200.9251726.5201234.459 Assault18513.43914149.13917580.70417846.646 Asterix86924.830103686.39585679.932 374916.327 Asteroids28792.41585168.3332549.1501950.884 Atlantis761386.735747847.619751528.912758324.490 Bank Heist1257.5171588.1631302.0751514.762 Battlezone65819.72848993.19738017.00737350.340 Beamrider33208.97322814.42220786.87119927.231 Berzerk3434.6608512.1092253.2313921.633 Bowling30.44234.66738.00730.571 Boxing100.000100.000100.000100.000 Breakout405.925394.044431.728407.690 Centipede11957.0318450.08512807.0177412.735 Chopper Command258300.000185923.81019930.61211332.993 Crazy Climber191479.252175811.224179710.884153729.252 Defender61850.51065859.35450575.85062583.844 Demon Attack132331.548131675.442130311.701128636.190 Double Dunk-1.483-1.442-0.850-1.449 Enduro2343.1732339.8642338.6842326.656 Fishing Derby49.18447.17742.53741.000 Freeway33.84033.66733.82033.806 Frostbite6228.5718996.6678130.3064846.871 Gopher39304.62632817.48368673.12945066.190 Gravitar1032.483978.0611874.830594.388 H.E.R.O.24368.82730249.84734076.00314280.051 Ice Hockey0.7211.5312.235-3.690 James Bond7288.4352657.6533470.9182228.912 Kangaroo13774.15014466.66714088.09514328.231 Krull10308.87811143.0619998.94610395.680 Kung-Fu Master35648.29933834.35432204.76241987.075 Montezuma’s Revenge0.0000.0000.0000.000 Ms. Pac-Man5614.4565822.4837529.9324865.816 Name This Game16041.73515393.26512873.43517054.830 Phoenix185824.252 283574.218 255857.24598561.429 Pitfall!0.0000.000-0.871-22.041 Pong21.00021.00021.00021.000 Private Eye50.00050.00044.983100.000 Q*bert24266.24125570.15326119.21822553.486 River Raid28564.45630737.82324621.36123529.388 Road Runner61521.08881121.76956501.36157602.381 Robotank71.55473.39164.13366.609 Continued on next page 18 Table S2 – continued from previous page EpsilonZetaEtaTheta Seaquest8840.81617571.22417243.7418063.129 Skiing-9766.480-10895.473-10667.639-21527.639 Solaris2541.5422247.2793255.1024381.088 Space Invaders2966.7182851.63324713.4352864.932 Stargunner320142.517315854.082237664.966229308.503 Surround9.2629.9059.8337.112 Tennis11.633-2.500-1.000-1.303 Time Pilot23786.05424746.93926560.5449999.320 Tutankham248.612248.173268.939246.932 Up’n Down307976.429 318144.69491661.429268903.095 Venture0.0000.0000.0000.000 Video Pinball462849.429 544883.133 529542.265487644.446 Wizard of Wor34776.19023365.64633848.98026704.762 Yars’ Revenge124140.105 144543.561 141543.759115265.139 Zaxxon15231.29315286.39520634.69417481.633 19 Table S3: Phase 2 Atari-57 Raw Returns: Final un-normalized evaluation scores comparing the standard Gamma encoder against the Gamma-Hadamax-Valid (V1) and Gamma-Hadamax- Same (V2) architectures. GammaHadamax-Gamma-Valid Hadamax-Gamma-SameHadamax Alien18963.33314507.89117811.59919762.347 Amidar1893.6392658.7521971.4861357.088 Assault18030.88821776.35423277.93922982.881 Asterix126304.082320721.088353770.408268579.932 Asteroids68774.32068211.90590121.83739592.245 Atlantis742603.741706664.286668376.531763619.728 Bank Heist1288.9121576.7351533.0951571.020 Battlezone64085.03483642.85763918.36789863.946 Beamrider39152.43547184.26564156.41544187.966 Berzerk1942.27958130.47655552.92531191.837 Bowling30.95937.03127.54830.667 Boxing99.58799.293100.000100.000 Breakout504.044410.425533.874547.340 Centipede17675.85023132.67734114.67726424.534 Chopper Command360188.095422371.088790388.095141528.912 Crazy Climber179469.728192591.837232073.469227088.095 Defender91159.694365917.177360621.599351619.898 Demon Attack132650.221135620.833135425.476135638.333 Double Dunk-1.707-2.306-1.136-1.415 Enduro2341.7622348.2072352.2992332.058 Fishing Derby43.27240.11647.39542.932 Freeway32.97334.00034.00033.806 Frostbite9532.9256893.74110607.89110759.048 Gopher51893.33383183.06156273.26559860.612 Gravitar1293.3671360.3742035.0342000.850 H.E.R.O.24353.57136814.57536880.81634529.915 Ice Hockey14.36435.32731.5513.262 James Bond5012.24511395.91812455.6124558.333 Kangaroo14087.75514363.26513030.27213900.340 Krull10724.45611037.17711042.14311045.170 Kung-Fu Master35474.83023623.81022875.17039016.327 Montezuma’s Revenge0.0000.0000.0000.000 Ms. Pac-Man5855.2388367.09911653.3848421.429 Name This Game21340.51021829.62620023.46921919.864 Phoenix190653.197237966.701286008.673201750.442 Pitfall!0.0000.000-0.6730.000 Pong21.00021.00021.00021.000 Private Eye87.33381.00014.62624.490 Q*bert25847.95944113.69039942.51727978.912 River Raid29396.32729878.06135587.65329210.612 Road Runner175828.571226921.769246640.476168071.088 Robotank68.75275.58276.26572.265 Seaquest8228.844263118.503208911.56563600.136 Skiing-22986.874-8848.003-8846.616-10685.082 Solaris3081.4972252.7892734.1502833.537 Space Invaders2855.57830371.24130697.70427396.701 Stargunner403316.667608855.102606955.442504784.354 Surround9.1469.8279.4019.486 Tennis23.88124.00023.83024.000 Time Pilot27330.27222304.76217815.64618702.721 Tutankham254.279275.857240.582257.537 Up’n Down266537.007271865.952288032.857269874.490 Continued on next page 20 Table S3 – continued from previous page GammaHadamax-Gamma-Valid Hadamax-Gamma-SameHadamax Venture0.00018.3670.0000.000 Video Pinball529722.776489526.680450481.476478247.367 Wizard of Wor29785.71433751.70126120.40819372.109 Yars’ Revenge140728.207440586.704260409.735370792.272 Zaxxon20938.43520152.72120314.96626114.966 21 Table S4: Phase 3 Atari-57 Raw Returns: Final un-normalized evaluation scores detailing the ablation of the advanced regression heads (Distributional Dueling, Ensemble Dueling, and Aftab). PQNDistributional Dueling Ensemble DuelingAftab Alien4688.19720386.32724329.32022953.912 Amidar1082.5002634.7893378.6162926.568 Assault17009.49315812.54121412.94220397.452 Asterix366434.014340450.680330103.741283998.639 Asteroids2312.72145602.00743198.02724836.667 Atlantis742852.041740343.197773082.6531922416.327 Bank Heist1402.1431673.7411657.1431576.259 Battlezone44176.871119812.925107248.299111278.912 Beamrider22017.97373008.43553364.93971070.449 Berzerk5877.85727839.25246650.10239211.531 Bowling30.00028.93549.54161.925 Boxing99.228100.000100.000100.000 Breakout352.044594.007699.381933.010 Centipede10076.690202759.65035466.418207815.929 Chopper Command17155.442543812.585827556.122568332.993 Crazy Climber161068.367216305.102201359.524173999.660 Defender51787.245297204.762367071.259505392.857 Demon Attack132035.357134858.639135605.23812853.605 Double Dunk-1.782-1.245-1.340-21.027 Enduro2360.7212345.1842346.5756165.946 Fishing Derby42.28636.93945.81643.054 Freeway33.82334.00034.00034.000 Frostbite6653.9468145.27212135.2729659.218 Gopher53851.97344451.42970625.03440949.388 Gravitar956.1222012.5852737.0754410.034 H.E.R.O.24341.20751736.15637095.71468036.905 Ice Hockey0.17735.66031.33340.592 James Bond2195.06818633.3337044.72829409.354 Kangaroo13455.44214308.84413969.72814474.830 Krull9707.58513738.36711224.55813158.673 Kung-Fu Master31990.13629379.59226209.18429263.265 Montezuma’s Revenge0.0000.000123.129148.980 Ms. Pac-Man3365.34011182.35011885.4399572.963 Name This Game15329.01426781.15619973.87826262.517 Phoenix177903.333482720.680483661.5651104620.680 Pitfall!-5.3440.000-5.0070.000 Pong21.00021.00021.00021.000 Private Eye66.66716.667100.000100.000 Q*bert21923.46945021.08848135.96944785.119 River Raid21349.01433202.00743151.36135383.163 Road Runner57719.048140571.429375416.667257419.048 Robotank71.16775.59278.72177.527 Seaquest7978.027471696.327365552.6871909299.898 Skiing-23644.677-15831.898-8677.490-20213.990 Solaris2605.3061216.8712487.2791365.850 Space Invaders5342.29634645.20462922.97154110.289 Stargunner259520.408619437.075692191.497628127.211 Surround8.3509.8279.5009.959 Tennis-1.33323.83324.00024.000 Time Pilot13860.88420004.76219017.34728396.259 Tutankham249.473240.306266.622258.588 Up’n Down263719.048311849.558271042.823269188.537 Continued on next page 22 Table S4 – continued from previous page PQNDistributional Dueling Ensemble DuelingAftab Venture0.0000.0000.000707.483 Video Pinball457305.670478535.429547276.8061700656.636 Wizard of Wor16897.95936559.52425731.63336345.578 Yars’ Revenge115017.827284535.259305711.432293344.007 Zaxxon16136.73533449.66019704.42239183.333 23 Table S5: Procgen Hard Raw Returns: Un-normalized evaluation returns over 200 million training frames comparing PQN and Aftab across the 16 Procgen Hard environments. PQN Aftab Big Fish7.233 31.491 Boss Fight 0.0230.000 Cave Flyer 5.3130.000 Chaser2.564 8.871 Climber1.0490.000 Coin Run 4.8160.000 Dodgeball 0.192 12.596 Fruit Bot -0.415 13.647 Heist0.4180.347 Jumper2.3060.000 Leaper5.500 9.520 Maze1.449 4.561 Miner0.358 10.052 Ninja2.8980.520 Plunder4.9834.672 Starpilot 4.5080.223 24 Table S6: Procgen Hard raw-score Area Under the Curve (AUC) by environment: For each Procgen Hard environment, the AUC is computed over the complete 200-million- frame training trajectory using trapezoidal integration across the recorded evaluation check- points and then averaged across the four experimental seeds. Bold values indicate the higher AUC within each environment. Because raw reward scales differ substantially across Procgen tasks, these values are intended only for within-environment comparison between Aftab and PQN and are not aggregated directly across environments. AftabPQN Big Fish148.35413.218 Boss Fight 2961.430 298.998 Cave Flyer70.669 113.820 Chaser119.001 -10.464 Climber128.45142.228 Coin Run111.235 159.278 Dodgeball 131.9114.117 Fruit Bot130.77910.437 Heist96.67368.645 Jumper19.699 93.572 Leaper148.85614.215 Maze97.86020.906 Miner175.3751.638 Ninja107.727 162.305 Plunder19.8603.061 Star Pilot180.68793.280 25