Paper deep dive

Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization

Benjamin Léger, Kazem Meidani, Christian Gagné

Year: 2026Venue: arXiv preprintArea: cs.NEType: PreprintEmbeddings: 51

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 4/14/2026, 1:43:07 AM

Summary

This paper investigates the effectiveness of SNIP, a multi-modal latent space optimization (LSO) framework for symbolic regression. The authors evaluate whether SNIP's cross-modal alignment between symbolic and numeric encoders actually guides symbolic search. Through empirical analysis, they demonstrate that the optimization process does not actively exploit the learned alignment and that the alignment itself is too coarse-grained to distinguish between structurally similar symbolic expressions, suggesting that fine-grained alignment is a critical requirement for future multi-modal LSO methods.

Entities (6)

CLIP · model · 100%Genetic Programming · methodology · 100%Latent Space Optimization · methodology · 100%SNIP · model · 100%Symbolic Regression · task · 100%InfoNCE · loss-function · 95%

Relation Signals (3)

SNIP → inspiredby → CLIP

confidence 100% · SNIP (Meidani et al., 2024), a contrastive pre-training model inspired by CLIP

SNIP → uses → Latent Space Optimization

confidence 95% · SNIP applies LSO to symbolic regression in a framework similar to the general approach

SNIP → optimizes → Symbolic Regression

confidence 90% · SNIP applies LSO to symbolic regression

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Symbolic regression (SR) aims to discover mathematical expressions from data, a task traditionally tackled using Genetic Programming (GP) through combinatorial search over symbolic structures. Latent Space Optimization (LSO) methods use neural encoders to map symbolic expressions into continuous spaces, transforming the combinatorial search into continuous optimization. SNIP (Meidani et al., 2024), a contrastive pre-training model inspired by CLIP, advances LSO by introducing a multi-modal approach: aligning symbolic and numeric encoders in a shared latent space to learn the phenotype-genotype mapping, enabling optimization in the numeric space to implicitly guide symbolic search. However, this relies on fine-grained cross-modal alignment, whereas literature on similar models like CLIP reveals that such an alignment is typically coarse-grained. In this paper, we investigate whether SNIP delivers on its promise of effective bi-modal optimization for SR. Our experiments show that: (1) cross-modal alignment does not improve during optimization, even as fitness increases, and (2) the alignment learned by SNIP is too coarse to efficiently conduct principled search in the symbolic space. These findings reveal that while multi-modal LSO holds significant potential for SR, effective alignment-guided optimization remains unrealized in practice, highlighting fine-grained alignment as a critical direction for future work.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

51,014 characters extracted from source content.

Expand or collapse full text

Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization Benjamin Léger IID / Mila, Université LavalQuebec City (Quebec)Canada benjamin.leger.1@ulaval.ca , Kazem Meidani Capital OneMcLeanUSA and Christian Gagné Canada-CIFAR AI Chair IID / Mila, Université LavalQuebec City (Quebec)Canada christian.gagne@gel.ulaval.ca Abstract. Symbolic regression (SR) aims to discover mathematical expressions from data, a task traditionally tackled using Genetic Programming (GP) through combinatorial search over symbolic structures. Latent Space Optimization (LSO) methods use neural encoders to map symbolic expressions into continuous spaces, transforming the combinatorial search into continuous optimization. SNIP (Meidani et al., 2024), a contrastive pre-training model inspired by CLIP, advances LSO by introducing a multi-modal approach: aligning symbolic and numeric encoders in a shared latent space to learn the phenotype-genotype mapping, enabling optimization in the numeric space to implicitly guide symbolic search. However, this relies on fine-grained cross-modal alignment, whereas literature on similar models like CLIP reveals that such an alignment is typically coarse-grained. In this paper, we investigate whether SNIP delivers on its promise of effective bi-modal optimization for SR. Our experiments show that: (1) cross-modal alignment does not improve during optimization, even as fitness increases, and (2) the alignment learned by SNIP is too coarse to efficiently conduct principled search in the symbolic space. These findings reveal that while multi-modal LSO holds significant potential for SR, effective alignment-guided optimization remains unrealized in practice, highlighting fine-grained alignment as a critical direction for future work. Symbolic Regression, Genetic Programming, Latent Space Optimization, Deep Learning, Multi-modal Learning †copyright: none†conference: Preprint – not peer-reviewed; 2026; †journalyear: 2026 1. Introduction While traditional Genetic Programming typically explores vast symbolic solution spaces through combinatorial search, recent methods (Mežnar et al., 2023; Liskowski et al., 2020; Kusner et al., 2017; Caetano et al., 2023) – referred to here as Latent Space Optimization Genetic Programming, or LSO-GP – transform this combinatorial problem into continuous optimization. Leveraging the success of deep representation learning methods, these approaches use neural encoders to map symbolic expressions into continuous and semantically dense latent spaces. Traditional black-box optimizers are subsequently used to conduct the search over this space. However, most existing LSO-GP methods focus exclusively on encoding the symbolic structure (i.e., genotype) of candidate solutions, without capturing information about their numeric behavior (i.e., phenotype or semantics).. Symbolic Regression (SR) is, however, a multi-modal task (Li et al., 2025; Meidani et al., 2024), as it aims to find interpretable symbolic equations from numeric data. Understanding the underlying genotype-phenotype mapping, that is, data-fitting accuracy, i.e., how well the solution fits the data, and has long been recognized as key for the design of effective SR heuristics (Winkler et al., 2018). The field of Semantic Genetic Programming (Vanneschi et al., 2014) has demonstrated how semantic information can guide symbolic search more efficiently. Moreover, SR inherently requires optimizing two distinct types of objectives: numerical accuracy, or how well the solution fits the data, and symbolic relevance, which includes interpretability and meaningful syntactic patterns (Bertschinger et al., 2024; Yu et al., 2025a). Rather than treating these modalities separately, exploiting their interactions and mutual information is crucial for designing effective search algorithms (Li et al., 2025). Recent advances in multi-modal models offer a potential solution to this challenge. Vision-Language Models (VLMs) like CLIP (Radford et al., 2021) can map complex data to information-rich continuous spaces while learning relationships between different modalities of the same concept. This capability suggests multi-modal models could provide the missing ingredient for data-driven Genetic Programming: algorithms that explore continuous spaces which are simultaneously informative about both the numeric and symbolic nature of candidate solutions. Building on this idea, SNIP (Meidani et al., 2024), inspired by CLIP, uses a pretrained bi-modal model to encode both numeric and symbolic modalities of mathematical equations. The model is trained to align these representations by mapping both modalities of the same equation close to each other in a shared latent space. For symbolic regression, SNIP conducts LSO search in the numeric latent space, with the premise that the learned inter-modal alignment will implicitly guide exploration of the symbolic space. However, the effectiveness of this approach depends on two critical assumptions. First, the LSO algorithm must actively exploit the learned alignment during search. Second, the alignment itself must be fine-grained enough to distinguish between symbolically different expressions. Existing literature on contrastive bi-modal models like CLIP reveals systematic failures to capture fine-grained semantic distinctions (Lewis et al., 2022; Wang et al., 2023; Tong et al., 2023; Chen et al., 2025), raising the question of whether SNIP’s alignment is sufficiently fine-grained for effective symbolic optimization. In this work, we investigate these two assumptions to better understand the strengths and limitations of multi-modal LSO for symbolic regression. Specifically, we make the following contributions: (1) We empirically demonstrate that SNIP’s current LSO formulation does not actively exploit the learned cross-modal alignment during optimization, even as fitness improves. (2) We evaluate the granularity of SNIP’s cross-modal alignment using retrieval tasks adapted from the contrastive learning literature, revealing that the alignment is too coarse to reliably distinguish between structurally similar symbolic expressions. (3) We discuss implications for future work, identifying fine-grained alignment as a critical direction for improving multi-modal LSO methods and outlining potential paths forward for symbolic regression. Figure 1. Overview of Latent Space Optimization and multi-modal alignment. a) Traditional LSO-GP framework : a continuous optimizer searches the latent space of a symbolic autoencoder. b) SNIP’s (Meidani et al., 2024) contrastive pre-training: symbolic and numerical encoders are trained to align embeddings of matched expression pairs with bat. c) Illustration of alignment granularity limitations : given a base expression f⋆=sin⁡(x+1)f = (x+1) and its numeric embedding ZV⋆Z_V (labeled N⋆N in the figure) encoded from observations (X,y⋆)(X,y ), the model should assign highest similarity between ZV⋆Z_V and the symbolic embedding ZS⋆Z_S (labeled S⋆S ) over embeddings of perturbed variants (S1,S2,S3S_1,S_2,S_3). In practice, the learned alignment often fails to make this distinction. TBD. 2. Preliminaries and background 2.1. Latent Space Optimization for SR Latent Space Optimization (LSO) transforms the combinatorial search in Genetic Programming (GP) algorithms into continuous optimization in learned representation spaces. The typical approach involves training an autoencoder to map symbolic expressions to continuous embeddings and reconstruct them via a decoder, minimizing syntactic reconstruction objectives such as cross-entropy loss. Once this space is constructed, gradient-free or gradient-based optimization algorithms can be utilized to search and sample candidates from this space, which are then decoded into symbolic expressions and evaluated for their fitness on the target data (as depicted in Fig 1-a). Prior LSO methods for symbolic regression (Caetano et al., 2023; Kusner et al., 2017; Mežnar et al., 2023) primarily encode symbolic structure without explicitly representing numeric semantics. 2.2. SNIP : Multi-modal Pretraining for Mathematical Equations SNIP (Meidani et al., 2024) is a multi-modal pre-training framework that jointly learns symbolic and numeric representations of mathematical expressions. Inspired by CLIP (Radford et al., 2021), SNIP uses contrastive learning to align both modalities in a shared latent space, aiming to capture the genotype-phenotype mapping between symbolic structure and numeric behavior. We provide an overview of SNIP’s architecture, training procedure, and its application to symbolic regression through latent space optimization. 2.2.1. Architecture and training SNIP’s architecture consists of two transformer encoders: one processing symbolic expressions (as prefix-order token sequences), another processing numeric observations (input-output pairs (x,y)(x,y)), resulting in latent vector representations for each modality, respectively denoted as ZSZ_S (for symbolic embeddings) and ZVZ_V (for numeric embeddings) in the rest of the paper. Both encoders are trained with the same contrastive objective used in CLIP, the InfoNCE loss (Oord et al., 2018), learning to align embeddings of matched symbolic-numeric pairs while separating unrelated pairs (as illustrated in Fig 1-b): (1) ℒInfoNCE=−∑(S,V)∈ℬ[log⁡exp⁡(ZS⋅ZV+/τ)∑Z∈ZV+,ZV−exp⁡(ZS⋅Z/τ)+logexp⁡(ZV⋅ZS+/τ)∑Z∈ZS+,ZS−exp⁡(ZV⋅Z/τ)], aligned L_InfoNCE=- _(S,V) [& (Z_S· Z_V^+/τ) _Z∈\Z_V^+,Z_V^-\ (Z_S· Z/τ)\\ &+ (Z_V· Z_S^+/τ) _Z∈\Z_S^+,Z_S^-\ (Z_V· Z/τ) ] aligned, where ℬB is a minibatch of randomly generated mathematical equations, τ a learnable temperature coefficient, and ZS+Z_S^+, ZV+Z_V^+, ZS−Z_S^- and ZV−Z_V^- denote positive (matched) and negative (unmatched) pairs for each modality. Concretely, for a given expression f with symbolic form S and numeric observations V, the positive pair is (ZS,ZV)(Z_S,Z_V) from the same expression, while negative pairs are formed by all other expressions in the batch. This formulation encourages a form of geometric coherence between the two modalities in a “joint” learned space, with the main objective of training models that capture the underlying relationship between them. 2.2.2. Latent Space Optimization for Symbolic Regression SNIP applies LSO to symbolic regression in a framework similar to the general approach described in Section 2.1, with a key difference: the search is conducted in the continuous latent space produced by the numeric encoder, rather than a discrete symbolic space. A decoder network, inherited and fine-tuned from prior neuro-generative SR work (Kamienny et al., 2022), maps latent vectors to symbolic expressions. The optimization procedure consists of the following steps: (1) The model encodes the target numerical dataset, resulting in the latent vector ZVtargetZ_V^target; (2) An initial population of latent vectors is initialized around ZVtargetZ_V^target by randomly perturbing the data and latent vectors; (3) A gradient-free (black-box) optimizer, such as Grey Wolf Optimizer (GWO) (Mirjalili et al., 2014), is employed to explore the continuous latent space, sampling new candidate vectors at each iteration; (4) Candidate vectors are decoded into symbolic expressions, whose constants are refined using BFGS algorithm (Fletcher, 2013); (5) Solutions are evaluated for fitness (R2R^2), which guides the optimizer’s search in subsequent iterations. 3. Alignment in Multi-Modal LSO 3.1. SNIP’s Performance: Promise and Gaps SNIP demonstrates promising capabilities for symbolic regression. The approach shows strong data-fitting accuracy and generates solutions with reasonable complexity on the SRBench benchmark (La Cava et al., 2021). Evidence suggests the model learns meaningful cross-modal relationships: LSO in the learned space yields better results than single-shot predictions decoded directly from the target’s numeric embedding, the model can quickly classify mathematical properties of expressions directly from their symbolic form, and visualizations reveal meaningful clusters of basic symbolic properties (e.g., number of variables, operator types) in the numeric space and vice versa (e.g., convexity, monotonicity in the symbolic space). However, the key hypothesis underlying SNIP’s multi-modal approach, that progressive moves in the continuous latent space translate to meaningful symbolic modifications via learned alignment, remains unvalidated. Recent work by Yu et al. (Yu et al., 2025a) demonstrates that despite SNIP’s data-fitting performance, it struggles to effectively retrieve relevant symbolic forms, showing significantly inferior symbolic retrieval rates compared to GP-based heuristics. Notably, SNIP fails to show substantial improvement over previous transformer-based generative models that include no explicit symbolic constraints in their search. This gap between data-fitting performance and symbolic retrieval raises fundamental questions about whether the learned alignment effectively guides symbolic search during optimization. 3.2. Limitations of Contrastive alignment and CLIP-like multi-modal models SNIP’s training objective mirrors CLIP’s (Radford et al., 2021), which learns joint vision-language representations through contrastive learning. The extensive literature analyzing CLIP reveals systematic limitations that may be relevant for understanding SNIP’s performance. While CLIP achieves impressive zero-shot performance on many tasks, its alignment captures high-level semantic similarity rather than fine-grained structure. Studies show CLIP behaves as a “bag of concepts”, failing to bind attributes to objects or distinguish compositional relationships (Lewis et al., 2022). Similarity scores do not vary faithfully with semantic changes (Wang et al., 2023), and the model struggles with quantifiers, negations, and spatial relations (Tong et al., 2023; Chen et al., 2025). For SNIP, analogous limitations could mean the model captures coarse numerical behavior but fails to distinguish symbolically different expressions with similar outputs. These findings are particularly concerning for symbolic regression. Effective LSO in the symbolic space requires the ability to model how small symbolic modifications translate into behavioral changes. If CLIP’s contrastive alignment fails on fine-grained visual-linguistic distinctions despite its scale and success, SNIP’s alignment may similarly struggle to capture the precise symbolic-numeric relationships needed for principled symbolic search. 3.3. Measuring Cross-Modal Alignment We refer to inter-modal alignment as how well the model associates the symbolic form of an expression with its numeric behavior (the two modalities) in the learned latent space. We quantify alignment using the cosine similarity between embeddings, which is the similarity function directly optimized by the InfoNCE training objective (Eq. 1): (2) alignment=a(ZS,ZV)=cos⁡(ZS,ZV)=ZS⋅ZV‖ZS‖‖ZV‖.alignment=a(Z_S,Z_V)= (Z_S,Z_V)= Z_S· Z_V\|Z_S\|\|Z_V\|. The InfoNCE loss (Eq. 1) explicitly maximizes this similarity for matched symbolic-numeric pairs of the same expression during training while minimizing it for unmatched pairs. In principle, this learned alignment could guide the optimization, that is if a candidate solution’s symbolic embedding ZSZ_S has high alignment with the target’s numeric embedding ZV∗Z_V^*, this could indicate symbolic relevance to the problem. However, whether the alignment is actually exploited during LSO and whether it is fine-grained enough to support symbolic search remains an open question. 3.4. Research Hypotheses Given the gap between SNIP’s data-fitting performance and its symbolic retrieval performance, combined with known limitations of contrastive alignment in similar models, we investigate two hypotheses for why the learned alignment may not effectively guide symbolic search: H1 – Algorithm exploitation:: The LSO procedure does not actively exploit the learned cross-modal alignment during optimization, even if the alignment quality is sufficient. H2 – Alignment granularity:: The learned alignment is too coarse-grained to distinguish between symbolically different expressions, preventing effective symbolic guidance even if the algorithm attempts to exploit it. These hypotheses are not mutually exclusive. Both could contribute to the observed gap in symbolic performance. In the following sections, we design experiments to test each hypothesis empirically. 4. Experimental Analysis To address the hypotheses outlined above, we pose two research questions. • Does SNIP’s LSO actively exploit the learned alignment, or does optimization occur independently of cross-modal relationships? (Verifying H1) • Is the learned alignment fine-grained enough to support symbolic search, or does it only capture coarse behavioral similarities? (Verifying H2) We answer these questions through two complementary studies. 4.1. Study 1: Cross-Modal Alignment During Optimization To verify H1, we investigate whether SNIP actively exploits cross-modal alignment during optimization. We run SNIP’s original LSO algorithm and track the following quantities for the best individual at each iteration t: • Rt2R^2_t fitness, indicating whether evolution improves data-fitting accuracy; • Inter-modal alignment between its symbolic embedding and the target’s numeric embedding (Eq. 2). While LSO gradient-free optimizer’s elitist nature ensures monotonically increasing Rt2R^2_t, the evolution of ata_t reveals whether the model’s search is guided by symbolic relevance, that is, how symbolically appropriate the current candidate solution is relative to the symbolic regression target. If the algorithm actively exploits the learned alignment to incorporate symbolic awareness, this quantity should progressively increase over the course of optimization. 4.1.1. Experimental details. We replicate SNIP’s experimental setup, hyperparameters, and model parameters (for full details, see (Meidani et al., 2024)), running optimization for 8080 iterations without early stopping to enable consistent comparison of metrics across the entire optimization trajectory. Dataset. We use the Feynman and Strogatz benchmark suites included in the popular symbolic regression benchmark SRBench (La Cava et al., 2021), keeping all equations with input dimensions D≤10D≤ 10. Encoders. Symbolic and numeric embeddings used to compute alignment were obtained using SNIP’s original pre-trained encoders (not fine-tuned for LSO), as provided by the authors. Initialization and first measurement. Following SNIP’s LSO procedure, the population is initialized via dataset augmentation: subsampling (p1=15p_1=15 individuals), target perturbation (p2=10p_2=10 individuals), and latent perturbation (p3=25p_3=25 individuals), totaling P=50P=50 individuals. The first measurement point (t=0t=0) corresponds to the model’s one-shot prediction from the target data encoding. Measurement t=1t=1 corresponds to the best individual after initialization and a BFGS constant refinement step. Measurement t=2t=2 onwards correspond to the best individual after each GWO iteration. 4.1.2. Results. Evolution of both metrics is reported in Fig. 2. R2R^2 shows the expected increasing trend, improving from 0.730.73 at t=0t=0 (one-shot prediction) to 0.960.96 at t=1t=1 (after initialization and BFGS) and reaching 0.990.99 at t=80t=80 (final iteration) on Feynman. The substantial R2R^2 improvement between t=0t=0 and t=1t=1 reveals the strong influence of initialization and constant refinement on data-fitting accuracy. In contrast to numeric accuracy, alignment remains essentially flat throughout optimization, starting at 0.0380.038 and decreasing slightly to 0.0010.001 by the final iteration on the Feynman set. These results demonstrate that the search algorithm does not actively exploit the learned alignment during optimization. Even as numerical fitness steadily improves, alignment quality neither increases nor guides the search toward symbolically relevant solutions. This indicates that the LSO procedure operates independently of the cross-modal relationships learned during pretraining. Similar conclusions can be drawn with Strogatz equations. This confirms H1: SNIP’s LSO procedure does not actively exploit the learned cross-modal alignment during optimization. Figure 2. Evolution of (a) R2R^2 fitness (top) and (b) cross-modal alignment (bottom) during LSO across all Feynman and Strogatz equations, averaged over all equations. While fitness improves consistently, alignment remains flat or decreases, indicating that optimization does not exploit cross-modal alignment to produce better symbolic solutions. Iteration 0 corresponds to the model’s one-shot prediction, iteration 1 to the best individual after initialization and BFGS constant refinement. Two subplots showing $R^2$ increasing from 0.73 to 0.990 and alignment decreasing from 0.038 to 0.001 over 80 iterations. 4.2. Study 2: Alignment Granularity We now address H2, by verifying whether the learned alignment is fine-grained enough to support symbolic search. While SNIP shows evidence of some cross-modal understanding (e.g., property prediction, operator clustering), effective symbolic optimization may require much finer granularity to distinguish between structurally similar expressions. We evaluate whether SNIP’s learned alignment can discriminate between structurally similar symbolic expressions through a retrieval task. Given a base expression f and its corresponding numerical observations (X,y)(X,y), we generate K−1K-1 perturbed variants f1′,f2′,…,fK−1′\f_1 ,f_2 ,…,f_K-1 \ that differ minimally from f in symbolic structure. The task is to identify the correct expression f among the K candidates using only the alignment (cosine similarity, Eq. 2) between the symbolic embedding ZSZ_S and the numerical embedding ZV∗Z_V^* encoded from (X,y)(X,y). 4.2.1. Experimental details. Dataset Construction. We evaluate on two datasets. First, the Feynman physics equations from SRBench with 2≤d≤102≤ d≤ 10 variables. The lower bound ensures applicability of variable substitution perturbations (which require at least two distinct variables), while the upper bound matches SNIP’s maximum capacity. After filtering equations without known ground-truth expressions and excluding 12 equations due to numerical issues or parsing failures, this yields 82 test cases. Second, 100100 synthetic expressions generated using SNIP’s training data distribution. As these match the pre-training data, they might be seen as an upper bound on expected performance. Perturbation Strategies. For each base equation, we apply up to four symbolic perturbation types designed to create structurally similar but semantically different expressions, summarized in Table 1. Table 1. Perturbation types used to generate candidate expressions. Each perturbation modifies a single element of the expression tree while preserving overall structure. Type Description Examples Unary operator swap Replace unary operator with sin↔cos↔tan semantically related alternative exp↔log ; ⋅↔(⋅)2↔(⋅)3 · (·)^2 (·)^3 Binary operator swap Replace binary operator with +⁣↔⁣−+ -; ×⁣↔⁣÷× any of the other three +⁣↔⁣×+ ×; −⁣↔⁣÷- Constant change Modify a numeric constant 2→32→ 3; π→eπ→ e; c→1.2c→ 1.2c Variable substitution Replace one variable with another x0→x1x_0→ x_1 For unary operators, we group operators into semantic families and allow swaps only within families: trigonometric (sin , cos , tan ), exponential/logarithmic (exp , log ), and power functions (⋅ ·, (⋅)2(·)^2, (⋅)3(·)^3). For binary operators, we allow swaps between any pair of the four arithmetic operators, covering both within-family swaps (e.g. +⁣↔⁣−+ -, ×⁣↔⁣÷× ) and cross-family swaps (e.g. +⁣↔⁣×+ ×, −⁣↔⁣÷- ). When multiple instances of a swappable operator exist in an expression, one is selected uniformly at random. As not all perturbations are applicable to every equation (e.g., unary operator swap requires the presence of a swappable unary operator), each test case includes in average K¯=3.67 K=3.67 candidates (1 base + 2.67 perturbations) for the Feynman set and K¯=4.42 K=4.42 for the synthetic set. Evaluation Protocol. For each test case, we: (1) Load data (X,y)(X,y) with N=200N=200 standardized input points; (2) Encode the numeric target: ZV∗=encodeV(X,y)Z_V^*=encode_V(X,y) using SNIP’s original pre-trained encoder; (3) Encode all K candidate expressions: ZS(k)=encodeS(fk)Z_S^(k)=encode_S(f_k) using the LSO-finetuned encoder; (4) Rank candidates by alignment a(fk)=cos⁡(ZS(k),ZV∗)a(f_k)= (Z_S^(k),Z_V^*); (5) Record the rank of the true expression f∗f^*. Metrics. We report retrieval accuracy: the fraction of test cases where the base expression ranks first. We compare against a random baseline corresponding to random selection over the K valid candidates for each given evaluation case. 4.2.2. Results. Table 2 presents retrieval performance on both datasets. On Feynman, the model achieves 18.3%18.3\% accuracy, below the random baseline of 27.327.3 (0.670.67× baseline). This indicates that the alignment signal not only fails to discriminate between similar expressions, but actively misleads the ranking: perturbed expressions often have higher similarity to the target than the correct expression. On Synthetic data, accuracy matches the random baseline exactly (23.0%23.0\%vs. 22.9%22.9\%), meaning that the model performs no better than chance even on in-distribution expressions. Figure 3 shows the rank distribution for Feynman: the correct expression ranks first in only 1515 cases (18.3%18.3\%), while it most frequently ranks second (47.6%47.6\%). Table 2. Overall retrieval performance on both datasets. The model performs worse than random chance on Feynman, not better than random chance on synthetic data. Metric Feynman Synthetic Test cases 82 100 Retrieval accuracy 18.3% 23.0% Random baseline 27.3% 22.9% Accuracy / Baseline 0.67×0.67× 1.00×1.00× Mean rank 2.29±0.922.29± 0.92 2.13±0.852.13± 0.85 Figure 3. Distribution of ranks for the correct expression across 82 test cases of Feynman set. The correct expression most frequently ranks second (39 cases), with only 15 cases achieving rank 1. Per-Perturbation Analysis. Table 3 shows which perturbation types most frequently “fool” the model on Feynman. Binary operator swaps are most problematic, fooling the model 65.8%65.8\% of the time and accounting for 49%49\% of all ranking failures. This suggests the alignment captures general functional form but cannot distinguish expressions that differ only in arithmetic operations. Variable substitutions are easiest to detect (only 24.4%24.4\% fooling rate), indicating the model does encode some variable-specific information. This last observation is intuitively unsurprising: during pretraining, the numeric encoder receives observations ordered by variable index (i.e. as a stacked tensor [X0,X1,…,XD][X_0,X_1,…,X_D]) while the symbolic encoder processes variable through similarily annoted tokens (x0,x1,…,xdx_0,x_1,…,x_d), making variable identity a straightforward correspondence to learn. Per-perturbation patterns are consistent on synthetic data: binary operator swaps remain most challenging (50.0%50.0\% fooling rate) and variable substitutions easiest to detect (12.0%12.0\% fooling rate). Table 3. Per-perturbation confusion analysis. “Fooling rate” indicates how often each perturbation type achieves higher similarity than the correct expression, “Wins” how often each perturbation is the ranks higher than all others. Perturbation Type Occurrences Fooling Rate Wins (rank 1) Unary operator swap 18 55.6% 5 Binary operator swap 79 65.8% 33 Constant change 49 49.0% 13 Variable substitution 82 24.4% 16 Impact of constant swaps. Since the handling and representations of constants have been shown to be challenging for transformer-based equation encoders (Kamienny et al., 2022; Li et al., 2022), and since SNIP performs a constant optimization step after each iteration, we repeat the experiment excluding constant perturbations entirely. Fig. 4 shows the impact of removing the “constant change” category on the overall results on the Feynman set. While excluding it improves accuracy from 18.3% to 23.2%, confirming that constant perturbations are indeed challenging, accuracy still falls short of the random baseline (23.2% vs 32.1%), demonstrating that the alignment’s limited discriminative power is not solely attributable to constant handling. The model also struggles with operator swaps, confirming that the coarse alignment granularity is a fundamental characteristic rather than a constant-specific artifact. Similar conclusions are noted with the Synthetic set. Figure 4. Sensitivity analysis excluding constant change perturbations on Feynman. (a) Accuracy comparison showing that even without constant changes, model accuracy (23.2%) remains below random baseline (32.1%). (b) Rank distribution comparison showing improved but still suboptimal performance when constant changes are excluded. Table 4. Retrieval performance comparison: Feynman benchmark vs. synthetic data from training distribution. Metric Feynman Synthetic Test cases 82 100 Retrieval accuracy 18.3% 23.0% Random baseline 27.3% 22.9% Accuracy / Baseline 0.67×0.67× 1.00×1.00× Mean rank 2.29±0.922.29± 0.92 2.13±0.852.13± 0.85 5. Discussion 5.1. Summary of Findings Our experimental investigation addressed two research questions about multi-modal latent space optimization for symbolic regression. Study 1 (Sec. 4.1) demonstrated that SNIP’s LSO procedure does not actively exploit the learned cross-modal alignment during optimization. Despite steady improvements in numerical fitness (R2R^2), alignment between candidate solutions and the target remains flat or decreases throughout the search process. This reveals that the algorithm operates independently of the cross-modal relationships learned during pretraining. Study 2 (Sec. 4.2) evaluated whether the learned alignment is sufficiently fine-grained to support symbolic search. Through retrieval tasks with structurally similar expressions, we found that SNIP’s alignment is too coarse-grained to reliably distinguish between expressions that differ in operators or constants. The model achieves only 18.3% retrieval accuracy, below the 27.3% random baseline, indicating that perturbed expressions often align more strongly with targets than correct expressions. The model’s inability to differentiate solutions with potentially significant behavioral differences when their symbolic forms are close rules out the possibility of using alignment to guide symbolic space optimization. The combination of Studies 1 and 2 reveals a fundamental gap: alignment neither increases during optimization (Study 1), nor would such increases guide symbolic search effectively if they occurred (Study 2). This explains the symbolic retrieval gap observed by Yu et al. (Yu et al., 2025a). Furthermore, these results show that using alignment as an explicit objective would be bottlenecked by the model’s limited discriminative capacity: even if alignment increased during evolution, this would likely not result in meaningful symbolic improvements. Despite these limitations, our analysis is constructive: by isolating two precise bottlenecks—algorithmic exploitation and alignment granularity—it provides a concrete roadmap for improving multi-modal LSO methods. The potential of such methods remains significant: learning the phenotype-genotype mapping in a continuous, optimizable space could shift SR from hand-crafted heuristics to data-driven search paradigms. Realizing this potential requires addressing the specific challenges identified here. 5.2. Implications for Multi-Modal LSO These results have broader implications beyond SNIP. They reveal a fundamental challenge for multi-modal latent space optimization methods in symbolic regression: learning cross-modal alignment that is simultaneously robust enough to emerge from contrastive pretraining and fine-grained enough to support symbolic search. The success of CLIP in vision-language tasks might suggest that similar architectures and objectives would naturally transfer to symbolic-numeric domains. Our findings demonstrate otherwise. The distinction between coarse-grained semantic alignment and fine-grained structural alignment is critical. In vision-language tasks, CLIP’s ability to capture high-level semantic similarity (e.g., “a dog playing in a park”) suffices for many applications. In symbolic regression, however, effective search requires understanding how minimal symbolic modifications (e.g., sin⁡(x)→cos⁡(x) (x)→ (x) or x+y→x×yx+y→ x× y) translate to behavioral changes. This demands a qualitatively different level of alignment granularity. For the broader LSO-GP community, this suggests that incorporating numeric information into latent spaces requires more than simply adding a numeric encoder and applying contrastive learning. The alignment must be explicitly designed and evaluated for the fine-grained discriminative capacity that symbolic search demands. This represents both a challenge and an opportunity: methods that successfully achieve fine-grained alignment could substantially advance the field. 5.3. Future Directions 5.3.1. Algorithm-Level Improvements A natural approach to leverage alignment would be to explicitly incorporate it into the optimization objective. For instance, the LSO algorithm could optimize both R2R^2 fitness and alignment quality (Equation (2)), guiding the search toward regions where candidate solutions are both numerically accurate and symbolically relevant to the target. However, our findings show this strategy is bottlenecked by alignment quality. Preliminary experiments confirm that while explicitly optimizing alignment increases ata_t during evolution, it does not improve symbolic retrieval performance when the underlying alignment is too coarse. This suggests that model-level improvements to alignment granularity are a prerequisite for effective algorithm-level exploitation. 5.3.2. Model-Level Improvements Addressing coarse alignment in contrastive models is an active research area. Several techniques have been proposed to improve fine-grained alignment in CLIP and similar vision-language models (Lewis et al., 2022; Wang et al., 2023). Adapting these techniques to mathematical expressions represents a promising direction. For symbolic regression, this might involve generating training data with carefully designed hard negatives (expressions that differ minimally but behave differently), incorporating structured contrastive objectives that explicitly reward fine-grained discrimination, or using auxiliary tasks that require detailed symbolic understanding. Additionally, refined fine-tuning strategies after pretraining could improve alignment for specific SR applications. Such model-level improvements could then be combined with algorithm-level guidance mechanisms, enabling LSO procedures that truly exploit multi-modal information for symbolic search. 5.3.3. Broader Applications Beyond latent space optimization, improved multi-modal alignment could benefit symbolic regression in other ways. If alignment quality reaches sufficient granularity, it could serve as a learned symbolic similarity metric, offering advantages over hand-crafted measures like edit distance or tree distance. Such metrics could improve benchmarking methodologies and enable more principled evaluation of symbolic retrieval. Additionally, multi-modal representations could guide traditional GP operators, informing initialization, selection, or variation to maintain both numerical accuracy and symbolic coherence, similar to recent neural-guided GP methods (Anthes et al., 2025; Han et al., 2025b). 5.4. Limitations Our analysis focuses on SNIP as a representative example of multi-modal LSO for symbolic regression. While SNIP uses widely adopted design choices (Transformer encoders, InfoNCE loss, contrastive learning), investigating alignment quality in methods with different architectures or training objectives would provide additional insights into the generality of our findings. Our retrieval evaluation in Study 2 employed specific perturbation strategies designed to test fine-grained discrimination; alternative evaluation protocols could reveal complementary aspects of alignment quality. Finally, while our findings identify alignment granularity as a critical bottleneck, we have not proposed or evaluated concrete solutions to improve it. However, the precise identification of this bottleneck—and the retrieval-based evaluation framework we introduce—provide both the target and the tools for future work on this problem. 6. Related works 6.1. Neural Genetic Programming Several approaches have explored integrating neural methods with Genetic Programming, either by learning continuous representations for optimization or by using neural models to guide traditional GP operators. 6.1.1. Latent Space Optimization for GP Several LSO-GP methods encode symbolic expressions into continuous latent spaces for optimization (Kusner et al., 2017; Dai et al., 2018; Mežnar et al., 2023; Caetano et al., 2023). These approaches focus on symbolic structure without explicitly representing numerical behavior. NEO (Liskowski et al., 2020) uses contrastive learning to enforce semantic locality in Boolean program synthesis, but exploits the finite output space and does not extend to symbolic regression. Our work investigates whether multi-modal approaches that jointly encode symbolic structure and numeric behavior can overcome these limitations. 6.1.2. Neural Guidance in Traditional GP Rather than conducting search in learned latent spaces, other methods use neural models to guide traditional GP operators such as initialization, selection, and variation (Liskowski et al., 2018; Anthes et al., 2025; Han et al., 2025b; Wyrwiński and Krawiec, 2025). These approaches maintain the discrete, tree-based representation of GP while leveraging neural networks to inform search decisions. If multi-modal alignment can be improved to achieve fine-grained discrimination, similar guidance mechanisms could be developed that account for both symbolic structure and numeric behavior. 6.2. Deep Learning for Symbolic Regression Building on the success of Transformer-based generative models, several works have approached symbolic regression through one-shot inductive learning (Biggio et al., 2021; Kamienny et al., 2022; Vastl et al., 2024; Li et al., 2022). These methods pretrain models on large-scale synthetic datasets to learn direct mappings from numerical observations to autoregressively decoded symbolic expressions, enabling instantaneous prediction without iterative search. While offering fast inference, these methods struggle with identifying underlying symbolic patterns, especially for out-of-distribution test cases (Yu et al., 2025a), motivating the inclusion of multi-modal information as in SNIP. Recent research has addressed fundamental limitations in neuro-generative SR methods. Yu et al. (Yu et al., 2025b) focus on improving training data quality through structural plausibility filtering based on ”Effective Information,” ensuring training sets better represent real physical formulas. Related issues of memorization bias (Sato and Sato, 2025) and generalization (Voigt et al., 2025) are actively being investigated. Additionally, several works explore using diffusion models instead of autoregressive generation (Bendinelli et al., 2023; Tymkow et al., 2025; Han et al., 2025a), offering alternative approaches to symbolic expression generation. Since SNIP inherits components from prior neuro-generative work (Kamienny et al., 2022), including its numeric encoder, symbolic decoder, and training data generation, these improvements represent orthogonal directions that could be combined with better alignment strategies. Our findings on alignment quality are complementary to these decoder-focused improvements: even with perfect decoders and training data, coarse alignment would still limit the effectiveness of multi-modal LSO. 6.3. Multi-Modal Approaches to Symbolic Regression Several recent works share the perspective that symbolic regression should be approached as a multi-modal task requiring joint optimization or understanding of symbolic and numeric modalities. Li et al. (Li et al., 2025) argue that SR cannot be properly modeled as sequence-to-sequence translation with word-to-word correspondence, but rather as a multi-modal task akin to image captioning. Their MMSR model optimizes symbolic, data-fitting, and alignment objectives simultaneously to predict expressions in a one-shot manner without iterative search. Bertschinger et al. (Bertschinger et al., 2024) similarly emphasize simultaneous optimization of symbolic and numerical objectives, framing SR as dual optimization of ”form and function.” They use neuroevolution to evolve neural network weights that bridge these often orthogonal metrics. MDLformer (Yu et al., 2025a) provides symbolic guidance by using Minimum Description Length (MDL) as a search objective, guiding Monte Carlo Tree Search toward symbolically principled solutions rather than purely minimizing prediction error. These works share our view that effective symbolic regression requires understanding and exploiting relationships between symbolic structure and numeric behavior. Our contribution is complementary: we investigate the fundamental requirement of fine-grained cross-modal alignment that underlies all such approaches. Our findings suggest that achieving sufficiently fine-grained alignment is more challenging than prior work has acknowledged, and that alignment quality should be explicitly evaluated rather than assumed. This has implications for any multi-modal SR method, whether based on LSO, one-shot prediction, or evolutionary search. 7. Conclusion We investigated the promise of multi-modal learning for Genetic Programming, focusing on SNIP’s application to symbolic regression. Through systematic experiments, we demonstrated two key limitations: (1) SNIP’s latent space optimization does not actively exploit the learned cross-modal alignment during search, and (2) the alignment learned through contrastive pre-training is too coarse-grained to distinguish between structurally similar expressions. These findings explain the gap between SNIP’s numerical success and its symbolic retrieval performance observed in prior work. While current bi-modal models do not yet achieve the fine-grained understanding required for principled symbolic search, our analysis clarifies the path forward: improving alignment granularity through refined training objectives, and designing optimization algorithms that explicitly leverage cross-modal relationships. We believe these directions can unlock the significant potential of multi-modal approaches for symbolic regression. References P. Anthes, D. Sobania, and F. Rothlauf (2025) Transformer semantic genetic programming for symbolic regression. In Proceedings of the Genetic and Evolutionary Computation Conference, p. 952–960. Cited by: §5.3.3, §6.1.2. T. Bendinelli, L. Biggio, and P. Kamienny (2023) Controllable neural symbolic regression. In International Conference on Machine Learning, p. 2063–2077. Cited by: §6.2. A. Bertschinger, J. Bagrow, and J. Bongard (2024) Evolving form and function: dual-objective optimization in neural symbolic regression networks. In Proceedings of the Genetic and Evolutionary Computation Conference, p. 277–285. Cited by: §1, §6.3. L. Biggio, T. Bendinelli, A. Neitz, A. Lucchi, and G. Parascandolo (2021) Neural symbolic regression that scales. In International Conference on Machine Learning, p. 936–945. Cited by: §6.2. V. Caetano, M. C. Teixeira, and G. L. Pappa (2023) Symbolic regression trees as embedded representations. In Proceedings of the Genetic and Evolutionary Computation Conference, Cited by: §1, §2.1, §6.1.1. Z. Chen, T. Xiao, J. Zhang, Y. Zheng, and X. Chen (2025) Understanding hardness of vision-language compositionality from a token-level causal lens. arXiv preprint. Cited by: §1, §3.2. H. Dai, Y. Tian, B. Dai, S. Skiena, and L. Song (2018) Syntax-directed variational autoencoder for structured data. In International Conference on Learning Representations, Cited by: §6.1.1. R. Fletcher (2013) Practical methods of optimization. John Wiley & Sons. Cited by: item 4. X. Han, C. Ning, J. Zhong, F. Yang, Y. Wang, and X. Mu (2025a) Discovering mathematical equations with diffusion language model. arXiv preprint arXiv:2509.13136. Cited by: §6.2. X. Han, J. Zhong, Z. Ma, X. Mu, and N. Gligorovski (2025b) Transformer-assisted genetic programming for symbolic regression. IEEE Computational Intelligence Magazine. Cited by: §5.3.3, §6.1.2. P. Kamienny, S. d’Ascoli, G. Lample, and F. Charton (2022) End-to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems, Vol. 35, p. 10269–10281. Cited by: §2.2.2, §4.2.2, §6.2, §6.2. M. J. Kusner, B. Paige, and J. M. Hernández-Lobato (2017) Grammar variational autoencoder. In International conference on machine learning, p. 1945–1954. Cited by: §1, §2.1, §6.1.1. W. La Cava, B. Burlacu, M. Virgolin, M. Kommenda, P. Orzechowski, F. O. de França, Y. Jin, and J. H. Moore (2021) Contemporary symbolic regression methods and their relative performance. Advances in neural information processing systems 2021 (DB1), p. 1. Cited by: §3.1, §4.1.1. M. Lewis, N. V. Nayak, P. Yu, Q. Yu, J. Merullo, S. H. Bach, and E. Pavlick (2022) Does clip bind concepts? probing compositionality in large image models. arXiv preprint. Cited by: §1, §3.2, §5.3.2. W. Li, W. Li, L. Sun, M. Wu, L. Yu, J. Liu, Y. Li, and S. Tian (2022) Transformer-based model for symbolic regression via joint supervised learning. In The Eleventh International Conference on Learning Representations, Cited by: §4.2.2, §6.2. Y. Li, J. Liu, M. Wu, L. Yu, W. Li, X. Ning, W. Li, M. Hao, Y. Deng, and S. Wei (2025) MMSR: symbolic regression is a multi-modal information fusion task. Information Fusion 114, p. 102681. Cited by: §1, §6.3. P. Liskowski, I. Błądek, and K. Krawiec (2018) Neuro-guided genetic programming: prioritizing evolutionary search with neural networks. In Proceedings of the Genetic and Evolutionary Computation Conference, p. 1143–1150. Cited by: §6.1.2. P. Liskowski, K. Krawiec, N. E. Toklu, and J. Swan (2020) Program synthesis as latent continuous optimization: evolutionary search in neural embeddings. In Proceedings of the Genetic and Evolutionary Computation Conference, p. 400–408. Cited by: §1, §6.1.1. K. Meidani, P. Shojaee, C. K. Reddy, and A. Barati Farimani (2024) SNIP: bridging mathematical symbolic and numeric realms with unified pre-training. In International Conference on Learning Representations, Cited by: Figure 1, §1, §1, §2.2, §4.1.1. S. Mežnar, S. Džeroski, and L. Todorovski (2023) Efficient generator of mathematical expressions for symbolic regression. Machine Learning 112 (11), p. 4563–4596. Cited by: §1, §2.1, §6.1.1. S. Mirjalili, S. M. Mirjalili, and A. Lewis (2014) Grey wolf optimizer. Advances in engineering software 69, p. 46–61. Cited by: item 3. A. v. d. Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §2.2.1. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1, §2.2, §3.2. S. Sato and I. Sato (2025) Can test-time computation mitigate memorization bias in neural symbolic regression?. arXiv preprint arXiv:2505.22081. Cited by: §6.2. S. Tong, E. Jones, and J. Steinhardt (2023) Mass-producing failures of multimodal systems with language models. Advances in Neural Information Processing Systems. Cited by: §1, §3.2. R. T. Tymkow, B. D. Schnapp, M. Valipour, and A. Ghodshi (2025) Symbolic-diffusion: deep learning based symbolic regression with d3pm discrete token diffusion. arXiv preprint arXiv:2510.07570. Cited by: §6.2. L. Vanneschi, M. Castelli, and S. Silva (2014) A survey of semantic methods in genetic programming. Genetic Programming and Evolvable Machines 15 (2), p. 195–214. Cited by: §1. M. Vastl, J. Kulhánek, J. Kubalík, E. Derner, and R. Babuška (2024) Symformer: end-to-end symbolic regression using transformer-based architecture. IEEE Access 12, p. 37840–37849. Cited by: §6.2. H. Voigt, P. Kahlmeyer, K. Lawonn, M. Habeck, and J. Giesen (2025) Analyzing generalization in pre-trained symbolic regression. arXiv preprint arXiv:2509.19849. Cited by: §6.2. T. Wang, K. Lin, L. Li, C. Lin, Z. Yang, H. Zhang, Z. Liu, and L. Wang (2023) Equivariant similarity for vision-language foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 11998–12008. Cited by: §1, §3.2, §5.3.2. S. M. Winkler, M. Affenzeller, B. Burlacu, G. Kronberger, M. Kommenda, and P. Fleck (2018) Similarity-based analysis of population dynamics in genetic programming performing symbolic regression. In Genetic Programming Theory and Practice XIV, p. 1–17. Cited by: §1. P. Wyrwiński and K. Krawiec (2025) Learning semantics-aware search operators for genetic programming. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, p. 659–662. Cited by: §6.1.2. Z. Yu, J. Ding, Y. Li, and D. Jin (2025a) Symbolic regression via mdlformer-guided search: from minimizing prediction error to minimizing description length. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §3.1, §5.1, §6.2, §6.3. Z. Yu, G. Wang, J. Ding, H. Wang, and Y. Li (2025b) Beyond formula complexity: effective information criterion improves performance and interpretability for symbolic regression. arXiv preprint arXiv:2509.21780. Cited by: §6.2.

Instant research discovery