Paper deep dive

Why Architecture Choice Matters in Symbolic Regression

Chakshu Gupta

Year: 2026Venue: arXiv preprintArea: cs.NEType: PreprintEmbeddings: 21

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%

Last extracted: 6/21/2026, 6:29:40 AM

Summary

This paper investigates how architectural choices in gradient-based Symbolic Regression (SR) affect the recovery of mathematical targets. By using the Exp-Minus-Log (EML) operator, the study isolates the effect of tree structure and variable routing from expressiveness. The research demonstrates that even when architectures share the same target language, their performance varies significantly based on the interaction between variable routing, tree shape, and the operator's gradient profile. Key findings include that more expressive architectures do not guarantee better recovery, certain tree shapes (balanced trees) are universally difficult for these methods, and switching operators can invert shape preferences. The study concludes that the optimization landscape, rather than just expressiveness, is the primary determinant of success in gradient-based SR.

Entities (9)

Chakshu Gupta · person · 100%EML Operator · mathematical-operator · 100%Eq. 6 · architecture · 100%Georgia Institute of Technology · organization · 100%Hybrid · architecture · 100%RML Operator · mathematical-operator · 100%SML Operator · mathematical-operator · 100%Symbolic Regression · field · 100%V16 · architecture · 100%

Relation Signals (4)

Chakshu Gupta → affiliatedwith → Georgia Institute of Technology

confidence 100% · Chakshu Gupta cgupta65@gatech.edu Georgia Institute of Technology

Balanced Trees → arehardtorecoverwith → Symbolic Regression

confidence 100% · Balanced (non-chain) tree shapes are never recovered.

Eq. 6 → ismoreexpressivethan → V16

confidence 100% · Because its input set at internal nodes is a strict superset of V16’s, Eq. 6 is strictly more expressive.

EML Operator → hasasymmetricgradient → Eq. 6

confidence 90% · The left input is exponentially amplified... while the right is attenuated.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Symbolic regression discovers mathematical formulas from data. Some methods fix a tree of operators, assign learnable weights, and train by gradient descent. The tree's structure, which determines what operators and variables appear at each position, is chosen once and applied to every target. This paper tests whether that choice affects which targets are actually recovered. Three structures are compared, all sharing the same operator and target language but differing in how variables enter the tree; one is strictly more expressive. Across over 12,700 training runs, one structure recovers a target at 100% while another scores 0%, and the ranking reverses on a different target. Expressiveness guarantees that a solution exists in the search space, but not that gradient descent finds it: the most expressive structure fails on targets that a restricted alternative solves reliably. Switching the operator changes which targets succeed; reversing its gradient profile collapses recovery entirely. Balanced (non-chain) tree shapes are never recovered. These findings show that the optimization landscape, not expressiveness alone, determines what gradient-based symbolic regression recovers.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

20,527 characters extracted from source content.

Expand or collapse full text

Why Architecture Choice Matters in Symbolic Regression Chakshu Gupta cgupta65@gatech.edu Georgia Institute of TechnologyKhurjaIndia Abstract. Symbolic regression discovers mathematical formulas from data. Some methods fix a tree of operators, assign learnable weights, and train by gradient descent. The tree’s structure, which determines what operators and variables appear at each position, is chosen once and applied to every target. This paper tests whether that choice affects which targets are actually recovered. Three structures are compared, all sharing the same operator and target language but differing in how variables enter the tree; one is strictly more expressive. Across over 12,700 training runs, one structure recovers a target at 100% while another scores 0%, and the ranking reverses on a different target. Expressiveness guarantees that a solution exists in the search space, but not that gradient descent finds it: the most expressive structure fails on targets that a restricted alternative solves reliably. Switching the operator changes which targets succeed; reversing its gradient profile collapses recovery entirely. Balanced (non-chain) tree shapes are never recovered. These findings show that the optimization landscape, not expressiveness alone, determines what gradient-based symbolic regression recovers. †copyright: none†journalyear: 2026 1. Introduction Symbolic regression (SR) discovers mathematical formulas from data (Cava et al., 2021; Dong and Zhong, 2025; Makke and Chawla, 2024). One family of methods parameterizes a formula as a tree of operators whose inputs are controlled by learnable weights, and recovers the target by training those weights with gradient descent (Sahoo et al., 2018; Dong and Zhong, 2024). The tree’s structure, which determines what operators and variables are available at each position, is its architecture: a design choice made once and applied to every target. Prior work has compared SR methods that differ in architecture, grammar, and search simultaneously: SRBench (Cava et al., 2021) tests 14 methods on 252 problems but cannot isolate architecture effects. Gradient-based methods such as EQL (Sahoo et al., 2018) and eEQL (Dong and Zhong, 2024) each use a single fixed architecture without studying how architecture interacts with targets. MetaSymNet (Li et al., 2025) meta-learns the architecture but does not explain why fixed ones fail. Reinforcement-learning approaches (Petersen et al., 2021; Landajuela et al., 2022; Biggio et al., 2021) avoid fixed-architecture landscapes entirely. On the EML side, Odrzywolek (Odrzywolek, 2026) demonstrated SR with one architecture, and Ipek (Ipek, 2026b, a) applied EML to hardware and battery models without interaction analysis. No prior study has varied architecture while holding the target set constant. This paper isolates the architecture variable by using the EML (Exp-Minus-Log) operator (Odrzywolek, 2026), a single binary operation from which all elementary functions can be constructed. Because every node in an EML tree computes the same operation, targets differ only in tree shape and variable placement, making it possible to compare architectures that share the same target language. Three architectures are tested across more than 12,700 training runs; the results show that a target’s recoverability depends on architecture, not just on whether the architecture can express it. 2. Background The EML operator (Odrzywolek, 2026) computes eml(a,b)=ea−ln⁡beml(a,b)=e^a- b. Its two inputs receive asymmetric gradients: the left input is exponentially amplified (∂eml/∂a=ea /∂ a=e^a) while the right is attenuated (∂eml/∂b=−1/b /∂ b=-1/b). EML is a Sheffer operator: with only the constant 11, binary trees of EML nodes can express every elementary function (++, −-, ×, //, exp , ln , trigonometric, and more) (Odrzywolek, 2026). For instance, ex=eml(x,1)e^x=eml(x,1) and e=eml(1,1)e=eml(1,1). A depth-n EML tree is a perfect binary tree with 2n−12^n-1 operator nodes (depth 3 gives 7 nodes with 8 leaf positions). At depth 3, five distinct shapes exist. Four are chains, in which one branch at each level carries the computation while the other is a leaf. Chains are classified by which EML input (Left or Right) carries the active branch: Shape Template Active-branch path LR eml(eml(1,eml(⋅,⋅)),1)eml(eml(1,eml(·,·)),1) Left at root, Right at depth 1 RL eml(1,eml(eml(⋅,⋅),1))eml(1,eml(eml(·,·),1)) Right at root, Left at depth 1 R eml(1,eml(1,eml(⋅,⋅)))eml(1,eml(1,eml(·,·))) Right at both levels L eml(eml(eml(⋅,⋅),1),1)eml(eml(eml(·,·),1),1) Left at both levels The fifth shape is balanced: both root branches are full subtrees, giving eml(eml(⋅,⋅),eml(⋅,⋅))eml(eml(·,·),eml(·,·)). The L chain produces double exponentiation (eeae^e^a), which overflows on the training domain and is excluded from experiments. Three architectures are tested, summarized in Table 1. All three can represent any depth-3 EML tree with 1,x,y\1,x,y\ at the leaves; one is strictly more expressive. Eq. 6 is the architecture described in Equation 6 of Odrzywolek (Odrzywolek, 2026): at each internal node a softmax selects among 1,x,fchild\1,x,f_child\, giving variable x access to every level of the tree. Because its input set at internal nodes is a strict superset of V16’s, Eq. 6 is strictly more expressive. V16 is variant 16 in the released source code111Code: https://github.com/aodrzywolek/EML accompanying Odrzywolek (2026): a sigmoid at each internal node selects between 1,fchild\1,f_child\, and variables x and y enter only at the leaves through a 3-way softmax, making them symmetric. Hybrid uses V16’s sigmoid at all internal nodes except the root, where a 4-way softmax adds x and y as additional options; the name reflects its combination of V16’s body with partial variable access at the root. Table 1. Architectures at depth 3. Name Internal selection Params Variable routing Eq. 6 softmax 1,x,f\1,x,f\ 42 x at all nodes; y leaves only V16 sigmoid 1,f\1,f\ 38 x,yx,y leaves only (symmetric) Hybrid V16 + root 4-way 44 x,yx,y root and leaves 3. Methodology To test whether the architecture–target interaction is specific to EML or general to operators with gradient asymmetry, two additional operators are introduced: (1) sml(a,b) (a,b) =sinh⁡(a)−arctan⁡(b), = (a)- (b), (2) rml(a,b) (a,b) =arctan⁡(a)−sinh⁡(b). = (a)- (b). SML shares EML’s gradient direction: the left input receives unbounded gradients (∂sml/∂a=cosh⁡a /∂ a= a, which grows exponentially) while the right is bounded (∂sml/∂b=−1/(1+b2) /∂ b=-1/(1+b^2)). RML reverses this asymmetry. Neither SML nor RML is known to be universal; target trees are evaluated numerically. Each node’s inputs are weighted combinations of the available terms, with weights controlled by learnable parameters. Training minimizes mean squared error using Adam (Kingma and Ba, 2015) (learning rate 0.01) on a 21×2121× 21 grid over [−3,3]2[-3,3]^2 (441 points; targets with numerical overflow are filtered, leaving ntrain≥378n_train≥ 378). A two-phase schedule is used: a search phase (6,000 iterations at temperature τ=2.5τ=2.5, keeping softmax/sigmoid weights soft) followed by a hardening phase (2,000 iterations; τ anneals from 2.5 to 0.01 with entropy and binarity penalties, then snaps to one-hot selections). After hardening each node selects exactly one input, producing a concrete expression tree. Each combination of architecture, operator, tree shape, and leaf assignment is tested with 256 independent trials (64 seeds × 4 initializations) for V16 and Hybrid, or 64 trials (one initialization) for Eq. 6; Eq. 6 uses a single initialization because the original paper specifies one, and adding V16-style strategies to Eq. 6 would confound the comparison. Non-extreme rates (e.g., 17%, 39.5%) carry 95% Clopper–Pearson intervals of roughly ±5± 5 p at n=256n=256. A trial counts as exact recovery if the hardened tree matches the target formula, verified on 500 random points (RMSE <10−6<10^-6). Replacing V16’s initialization with Eq. 6’s (randn × 0.1, no bias) yielded 0/64 recoveries, confirming that the architecture differences reported below are not explained by initialization. 4. Results 4.1. Architecture–Target Interaction Table 2 shows that recovery depends on the combination of architecture and target, not on either factor alone. Eq. 6 acts as a specialist: 100% recovery (64/64) on exactly two targets and 0% (0/64) on three others. V16 is a generalist with nonzero recovery on all four chain shapes (16–99.6%) and x↔yx y symmetry from its leaf-only variable design. Hybrid is an RL specialist (95–96% on RL targets, near 0% elsewhere). No single architecture dominates: Eq. 6 scores 100% where V16 scores 17%, yet V16 scores 99.6% where Eq. 6 scores 0%. Expressiveness does not explain this pattern because Eq. 6 is strictly more expressive than V16 yet fails on targets V16 recovers. The difference lies in how each architecture’s variable routing interacts with the optimization landscape for a given target. Table 2. EML recovery rates (%, exact match). Target Shape Eq. 6 V16 Hybrid Paper (yx) LR 100 (64/64) 17 0.4 T1 (xy) RL 0 (0/64) 99.6 96 T4 (xy) R 0 (0/64) 39.5 0 T8 (xy) LR 1.6 16 0.4 T1_yx RL 0 — 95.3 T4_yx R 100 (64/64) — 0 4.2. Operator Swap Reverses Shape Preferences Switching from EML to SML (Table 3, Figure 1) inverts Eq. 6’s shape preferences: its best shape under EML is LR (100%), but under SML it shifts to RL and R (100%) while LR drops to 0% (Fisher exact p<10−15p<10^-15). Both operators amplify the left input, so the reversal traces to differences in right-side attenuation (−1/(1+b2)-1/(1+b^2) for SML versus −1/b-1/b for EML), which saturate differently on the [−3,3]2[-3,3]^2 domain. Despite the shape reversal, qualitative profiles persist: Eq. 6 remains a specialist, V16 a generalist (83–100% on all chains), and Hybrid an RL specialist. These profiles likely reflect structural properties of variable routing rather than artifacts of one operator. Heatmap showing recovery rates for three operators across architectures and tree shapes. Figure 1. Recovery by architecture, shape, and operator (— = inferred by symmetry). Table 3. SML recovery rates (%, exact match). Target Shape Eq. 6 V16 Hybrid S_LR (yx) LR 0 84 0.8 S_LR_xy LR 0 83 0.8 S_RL (xy) RL 100 100 100 S_RL_yx RL 0 100 100 S_R (xy) R 100 99.6 89.8 S_R_yx R 100 98.8 88.7 V16 dominates under SML, recovering all chain targets at 83% or above. Replacing EML with the right-amplified RML collapses all rates: Eq. 6 scores 0% on every target, V16’s best nominal rate is 2.0% (5/256, not significantly different from 0%; Fisher exact p=0.06p=0.06), and Hybrid’s RL preference vanishes. R and balanced shapes were not tested under RML because the collapse on chain targets, which are easier to recover than balanced shapes, made further experiments uninformative. 4.3. Balanced Trees Are Universally Hard The balanced depth-3 shape is tested with four leaf-placement variants per architecture–operator combination (Table 4). Every architecture scores 0% on every balanced variant (0/3,776 trials; Clopper–Pearson 95% upper bound 0.10%). The failure persists across 6 training configurations varying learning rate (0.01, 0.05), iteration budget (6,000, 12,000), and temperature (1.0, 2.5): 0/192. The contrast with chain shapes is stark: V16 recovers 83–100% of SML chains yet scores exactly 0% on balanced variants of the same operator. A plausible explanation is that balanced branching splits gradient signal roughly equally between the two root subtrees, preventing the concentration along a single path that drives chain recovery; gradient measurements for balanced shapes were not collected, and verifying this hypothesis is future work. Table 4. Balanced recovery: 0% across all combinations. Operator Eq. 6 V16 Hybrid Total SML (4 variants) 0/256 0/1024 0/1024 0/2,304 EML (4 variants) 0/256 0/1024 — 0/1,280 HP sensitivity — 0/192 — 0/192 Total 0/3,776 4.4. Gradient Measurements To investigate the mechanism behind the architecture–target interaction, gradient norms at leaf parameters are measured at iteration 1000 across 10 seeds (Table 5). For Eq. 6, the ratio tracks recovery: when x/y<1x/y<1 (variable y receives stronger gradients), recovery is 100%; when x/y>1x/y>1 (y is gradient-starved), recovery is 0%. Swapping EML for SML on the same shape (LR) flips the ratio from 0.67 to 1.78, consistent with the observed preference reversal. V16 shows narrower ratios across targets (0.77–2.00), consistent with its symmetric design preventing either variable from dominating the gradient landscape. Table 5. Leaf gradient ratio ‖∇x‖/‖∇y‖\| _x\|/\| _y\| at iteration 1000 (10-seed mean ± s.e.). Arch. Target Op. x/yx/y ratio Rec. Eq. 6 LR (yx) EML 0.67±0.030.67± 0.03 100% Eq. 6 RL (xy) EML 1.58±0.041.58± 0.04 0% Eq. 6 LR (yx) SML 1.78±0.121.78± 0.12 0% Eq. 6 RL (xy) SML 1.19±0.111.19± 0.11 100% V16 LR (yx) EML 0.77±0.050.77± 0.05 17% V16 RL (xy) EML 2.00±0.082.00± 0.08 99.6% The gradient trajectory during training (Figure 2) reveals the mechanism more clearly. On the 100% EML target the ratio undergoes a brief reversal, from 1.45 at iteration 100 to 0.14 at iteration 500, then settling at 0.67 by iteration 1000; this temporary window in which y dominates correlates with convergence to the correct formula. On the 0% target the ratio remains above 1.15 at every measured iteration, and y never receives sufficient gradient signal. These measurements establish a gradient-level correlate of the architecture–target interaction; the question of causation is addressed in Section 5.1. Line plot showing gradient ratio trajectories diverging for recovered vs unrecovered targets. Figure 2. Gradient ratio ‖∇x‖/‖∇y‖\| _x\|/\| _y\| during training (Eq. 6, 10-seed mean ± s.e.). 5. Discussion Selecting a single architecture for all targets, as current gradient-based SR methods do (Sahoo et al., 2018; Dong and Zhong, 2024; Odrzywolek, 2026), can yield 0% recovery on targets that a different architecture recovers at 100%. Neural architecture search (NAS) also studies architecture–task interaction, but NAS architectures typically differ in expressiveness (Elsken et al., 2019); here all architectures share the same target language, so recovery differences arise purely from optimization landscape geometry, a condition rarely met in benchmarks where methods differ simultaneously in grammar, search, and architecture (Cava et al., 2021). The cross-operator results point to four interacting factors that determine recovery: variable routing, tree shape, leaf assignment, and gradient direction. Four predictions were formulated before the RML and balanced experiments. One was confirmed (V16 preferences invert under RML); three were falsified (balanced trees were predicted to vary by architecture but proved universally hard; Eq. 6 and Hybrid were predicted to reverse preferences under RML but collapsed instead). Gradient direction interacts with variable routing more strongly than a simple reordering model predicts. 5.1. Limitations All three operators share the subtraction form f(a)−g(b)f(a)-g(b); whether the interaction extends to multiplicative or compositional operators is untested. The full three-architecture comparison exists only at depth 3; limited depth-4 experiments with V16 show the generalist profile collapses (RL drops from 99.6% to 0.8%), but a complete depth-4 matrix remains future work. The controlled-expressiveness design requires all targets to lie within the EML grammar, preventing direct comparison with standard SR benchmarks (SRBench, Feynman equations) that would require exponential-depth EML trees. SML targets have narrower output range than EML targets on [−3,3]2[-3,3]^2, which may partly explain V16’s higher SML rates (83–100% versus 16–99.6%). The shape-specific patterns (Eq. 6 under SML: LR 0%, RL 100%) cannot be explained by conditioning alone, since better conditioning would raise all shapes uniformly. The gradient measurements establish correlation, not causation. A mechanistic account via loss-landscape analysis would strengthen the framework but is beyond the scope of this study. 6. Conclusion The optimization landscape, not expressiveness alone, determines what gradient-based symbolic regression recovers. This motivates two practical directions: multi-architecture ensembles that run several structures in parallel and select the best result, and architecture-aware search that adapts variable routing to the target during optimization. The universal failure on balanced trees points to a deeper constraint that warrants direct investigation via loss-landscape analysis. Extending the interaction analysis to multiplicative operators, deeper trees, and standard SR benchmarks would test how far these findings generalize. References (1) Biggio et al. (2021) Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. 2021. Neural Symbolic Regression that Scales. In Proceedings of the 38th International Conference on Machine Learning (ICML). 936–945. Cava et al. (2021) William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabrício Olivetti de França, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason H. Moore. 2021. Contemporary Symbolic Regression Methods and their Relative Performance. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 34. Dong and Zhong (2024) Hao Dong and Jiangjun Zhong. 2024. Extended Equation Learner Solves Symbolic Regression Tasks with Evolutionary Components. IEEE Transactions on Evolutionary Computation (2024). Dong and Zhong (2025) Hao Dong and Jiangjun Zhong. 2025. A Survey on Symbolic Regression. arXiv preprint arXiv:2211.10873 (2025). Elsken et al. (2019) Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural Architecture Search: A Survey. Journal of Machine Learning Research 20, 55 (2019), 1–21. Ipek (2026a) Sinan Ipek. 2026a. Evaluating the Exp-Minus-Log Sheffer Operator for Battery Characterization. arXiv preprint arXiv:2604.13873 (2026). Ipek (2026b) Sinan Ipek. 2026b. Hardware-Efficient Neuro-Symbolic Networks with the Exp-Minus-Log Operator. arXiv preprint arXiv:2604.13871 (2026). Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations (ICLR) (2015). Landajuela et al. (2022) Mikel Landajuela, Chak Shing Lee, Jiachen Yang, Ruben Glatt, Claudio P. Santiago, Ignacio Aravena, Terrell Mundhenk, Garrett Mulcahy, and Brenden K. Petersen. 2022. A Unified Framework for Deep Symbolic Regression. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. Li et al. (2025) Wenqiang Li, Weijun Li, Lina Yu, Min Wu, Linjun Sun, Jingyi Liu, Yanjie Li, and Songsong Tian. 2025. MetaSymNet: A Dynamic Symbolic Regression Network Capable of Evolving into Arbitrary Formulas. In Proceedings of the AAAI Conference on Artificial Intelligence. Makke and Chawla (2024) Nour Makke and Sanjay Chawla. 2024. Interpretable Scientific Discovery with Symbolic Regression: A Review. Artificial Intelligence Review 57, 2 (2024). Odrzywolek (2026) Andrzej Odrzywolek. 2026. All Elementary Functions from a Single Operator. arXiv preprint arXiv:2603.21852v2 (2026). Institute of Theoretical Physics, Jagiellonian University. Petersen et al. (2021) Brenden K. Petersen, Mikel Landajuela, T. Nathan Mundhenk, Claudio P. Santiago, Soo K. Kim, and Joanne T. Kim. 2021. Deep Symbolic Regression: Recovering Mathematical Expressions from Data via Risk-Seeking Policy Gradients. In Proceedings of the International Conference on Learning Representations (ICLR). Sahoo et al. (2018) Subham Sahoo, Christoph Lampert, and Georg Martius. 2018. Learning Equations for Extrapolation and Control. In Proceedings of the 35th International Conference on Machine Learning (ICML). 4442–4450.

Instant research discovery