Paper deep dive
Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences
Artem Sakhno, Ivan Sergeev, Alexey Shestov, Omar Zoloev, Elizaveta Kovtun, Gleb Gusev, Andrey Savchenko, Maksim Makarenko
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/22/2026, 5:20:34 AM
Summary
Embedding-Aware Feature Discovery (EAFD) is a framework that bridges the gap between pretrained event-sequence embeddings and interpretable feature-based pipelines. By using an LLM-driven agent, EAFD iteratively discovers features based on alignment (explaining existing embeddings) and complementarity (identifying missing predictive signals), consistently outperforming baseline models on financial transaction benchmarks.
Entities (5)
Relation Signals (3)
EAFD → improvesperformanceof → CoLES
confidence 95% · EAFD (CoLES) achieves consistent gains across all datasets.
EAFD → improvesperformanceof → NTP
confidence 95% · EAFD yields large relative improvements across all datasets, including +15.4% on Age Prediction and up to +19.3% on the DataFusion benchmark.
EAFD → evaluatedon → Rosbank
confidence 90% · We observe a consistent improvement in performance... on the Rosbank prediction task.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Industrial financial systems operate on temporal event sequences such as transactions, user actions, and system logs. While recent research emphasizes representation learning and large language models, production systems continue to rely heavily on handcrafted statistical features due to their interpretability, robustness under limited supervision, and strict latency constraints. This creates a persistent disconnect between learned embeddings and feature-based pipelines. We introduce Embedding-Aware Feature Discovery (EAFD), a unified framework that bridges this gap by coupling pretrained event-sequence embeddings with a self-reflective LLM-driven feature generation agent. EAFD iteratively discovers, evaluates, and refines features directly from raw event sequences using two complementary criteria: \emph{alignment}, which explains information already encoded in embeddings, and \emph{complementarity}, which identifies predictive signals missing from them. Across both open-source and industrial transaction benchmarks, EAFD consistently outperforms embedding-only and feature-based baselines, achieving relative gains of up to $+5.8\%$ over state-of-the-art pretrained embeddings, resulting in new state-of-the-art performance across event-sequence datasets.
Tags
Links
- Source: https://arxiv.org/abs/2603.15713v1
- Canonical: https://arxiv.org/abs/2603.15713v1
Trouble viewing inline? Open PDF directly →
Full Text
43,935 characters extracted from source content.
Expand or collapse full text
Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences Artem Sakhno 1 , Ivan Sergeev 1 , Alexey Shestov 1 , Omar Zoloev 1 , Elizaveta Kovtun 1 , Gleb Gusev 1 Andrey Savchenko 1,2 , Maksim Makarenko 1 1 Sber AI Lab 2 ISP RAS Research Center for Trusted AI Abstract Industrial financial systems operate on temporal event sequences such as transactions, user actions, and system logs. While recent research emphasizes representation learning and large language mod- els, production systems continue to rely heavily on handcrafted statistical features due to their in- terpretability, robustness under limited supervision, and strict latency constraints. This creates a per- sistent disconnect between learned embeddings and feature-based pipelines. We introduce Embedding- Aware Feature Discovery (EAFD), a unified frame- work that bridges this gap by coupling pretrained event-sequence embeddings with a self-reflective LLM-driven feature generation agent. EAFD it- eratively discovers, evaluates, and refines features directly from raw event sequences using two com- plementary criteria: alignment, which explains in- formation already encoded in embeddings, and complementarity, which identifies predictive sig- nals missing from them. Across both open-source and industrial transaction benchmarks, EAFD con- sistently outperforms embedding-only and feature- based baselines, achieving relative gains of up to +5.8% over state-of-the-art pretrained embed- dings, resulting in new state-of-the-art performance across event-sequence datasets. 1 Introduction Industrial machine learning systems commonly operate on temporal event sequences, including financial transac- tions [ Mollaev et al., 2025; Fadeev et al., 2025 ] , online purchase histories [ Liu et al., 2025b ] , electronic health records [ Draxler et al., 2025 ] , spatial trajectories [ Zhu et al., 2025 ] . In production environments, these sequences are pro- cessed under strict constraints on latency, throughput, and stability [ Xiao et al., 2024 ] , which has led to the widespread adoption of specialized encoder models that produce compact embeddings [ Brisaboa et al., 2018 ] , alongside handcrafted statistical interpretable features that encode domain-specific structure [ Verdonck et al., 2024; Shestov et al., 2025a ] . While recent research has increasingly focused on representa- tion learning [ Shestov et al., 2025b ] , real-world systems con- Figure 1: Blind spots of embeddings. Coefficient of determination (R 2 ) of a feature reconstruction based on embedding representations CoLES, NTP, and LLM4ES on the Rosbank dataset. Values high- light systematic representational blind spots across embeddings. tinue to rely heavily on feature-based pipelines [ Louhi, 2023; Faubel and Schmid, 2024 ] due to their ease of configuration for new tasks with limited labeled data, interpretability, and predictable runtime behavior. Recent advances in automated feature engineering [ de Winter et al., 2025 ] and AutoML [ Gijsbers et al., 2024 ] have greatly reduced the manual effort required to construct and select features for structured data. Many end-to-end Au- toML frameworks incorporate feature selection directly into the training pipeline [ Arik and Pfister, 2021 ] , while other approaches focus explicitly on automated feature construc- tion. For example, AutoFeat [ Horn et al., 2019 ] generates non-linear transformations for tabular data, and Deep Fea- ture Synthesis (DFS), implemented in Featuretools [ Kan- ter and Veeramachaneni, 2015 ] , produces fixed-size repre- sentations from sequential inputs using predefined aggrega- tion operators. More recently, research in automated fea- ture engineering has drifted toward the use of large lan- guage models (LLMs) as reasoning and orchestration compo- nents, moving toward fully automated and increasingly agen- tic frameworks that coordinate feature generation, selection, and evaluation [ Hollmann et al., 2023; Guo et al., 2024; Abhyankar et al., 2025 ] . Despite this progress, automated feature engineering for sequential data largely remains decou- pled from representation learning. As a result, deep embeddings and statistical features are arXiv:2603.15713v1 [cs.LG] 16 Mar 2026 optimized through largely separate workflows. The conse- quences of this separation become apparent when examining how well embeddings reconstruct different classes of hand- crafted features (Figure 1). While embeddings capture certain aggregate statistics with high fidelity, other feature groups exhibit persistently low reconstruction quality, revealing sys- tematic representational blind spots that vary across embed- ding models. This is problematic in practice: information missing from the embedding cannot be recovered by subse- quent models, leading to performance ceilings and redundant feature construction. In the absence of a clear interface be- tween representation learning and feature discovery, feature engineering remains unaware of representation gaps, and em- bedding refinement proceeds without interpretable guidance from structured signals. To address this gap: • We introduce Embedding-Aware Feature Discovery (EAFD) 1 , a unified framework that transforms the dis- connect between embeddings and structured features into an iterative, self-reflective reasoning process for feature discovery (Section 3). EAFD operates in two complementary regimes:an interpretability regime, which analyzes and explains the information already en- coded in pretrained embeddings by aligning with inter- pretable features, and a performance regime, which dis- covers complementary features that provide additional predictive signal not present in embeddings.EAFD anchors on pretrained embeddings and integrates them with an LLM-driven feature-generation agent that ac- tively explores raw event-sequence data, proposing and evaluating candidate features based on alignment and complementarity criteria. • We evaluate EAFD on four open-source event- sequence benchmarks, where it consistently outperforms embedding-only and feature-based baselines across datasets and backbone embeddings (Section 4.2). EAFD achieves relative gains of up to +5.8% over state-of- the-art embeddings and up to +19% over weaker rep- resentations, establishing a new state of the art on open-source transaction datasets. We further validate EAFD on a large-scale, proprietary, multitarget finan- cial dataset, where a single EAFD-enhanced represen- tation improves across classification and regression tar- gets, yielding gains of up to 12.55% and up to 3.87% error reduction, respectively. • We use EAFD’s explicit feedback to analyze multiple state-of-the-art embedding models, uncovering system- atic representational biases and information gaps (Sec- tion 4.3). Leveraging these insights, we propose sev- eral modifications for the CoLES framework, resulting in up to 1.20% relative improvement in churn predic- tion. We also demonstrate how EAFD’s embedding interpretability signals can be further coupled with a privacy-preserving feature erasure mechanism [ Seputis et al., 2025; Liu et al., 2025a ] that identifies and sup- presses sensitive attributes encoded in embeddings. Overall, EAFD provides a unified and practical framework 1 The source code is provided in the supplementary materials. for analyzing and enhancing event-sequence embeddings through feature discovery, bridging representation learning and feature-based pipelines across both benchmark datasets and large-scale industrial systems. 2 Related Works Automated feature engineering (AutoFE) [ de Winter et al., 2025 ] has been widely studied for tabular data. Classical systems range from LightAutoML’s data cleaning [ Vakhru- shev et al., 2021 ] to more complex approaches like Feature- tools [ Kanter and Veeramachaneni, 2015 ] and AutoCross [ Li et al., 2024 ] , which use predefined transformation gram- mars and cross-feature search. Neural methods, such as Tab- Net [ Arik and Pfister, 2021 ] , learn latent feature representa- tions directly, while OpenFE [ Zhang et al., 2023 ] improves generation diversity and efficiency via a fast two-stage prun- ing. Recent progress in large language models has ex- tended this line of work toward feature-generation agents. CAAFE [ Hollmann et al., 2023 ] shows that LLMs can synthesize meaningful features and executable code from natural-language descriptions, while follow-up approaches explore evolutionary search (LLM-FE [ Abhyankar et al., 2025 ] ), program synthesis (DS-Agent [ Guo et al., 2024 ] ), and table-structure reasoning [ Han et al., 2025 ] . Both clas- sical AutoFE systems and LLM-based feature agents exhibit a common limitation in modern industrial pipelines that rely on pretrained encoders: feature construction is performed in- dependently of the embedding space, without assessing re- dundancy or complementarity, and without explicit reasoning over temporal structure in event-sequence data. A complementary research direction investigates embed- ding probing [ Ko et al., 2025 ] and representation anal- ysis [ Tinaz et al., 2025 ] .Techniques such as linear probes [ Basile et al., 2025 ] , sparse decomposition [ Harle et al., 2025 ] , and centered kernel alignment [ Maniparambil et al., 2024 ] provide insight into what pretrained models en- code. These approaches have been applied extensively in language and vision models [ Li et al., 2025 ] and, more re- cently, in representation learning for tabular and sequential data [ Tennenholtz et al., 2024 ] . However, probing meth- ods are diagnostic only: they reveal alignment or separability properties of embeddings but do not generate new features or provide actionable guidance for downstream modeling. Event-sequence modeling further amplifies structural gap between feature construction and representation analysis, as temporal patterns such as recency, burstiness, and seasonality typically require explicit feature construction alongside spe- cialized sequence encoders [ Klenitskiy et al., 2025 ] . Exist- ing AutoFE and LLM-based feature agents do not incorporate feedback from pretrained event-sequence embeddings, either generating features without accounting for what the embed- ding already encodes [ Kanter and Veeramachaneni, 2015 ] or analyzing embeddings without producing complementary symbolic features [ Hollmann et al., 2023; Abhyankar et al., 2025 ] . As a result, the problem of joint, embedding-aware feature discovery remains largely unaddressed. Figure 2: Overview of Embedding-Aware Feature Discovery (EAFD). (a) Latent embedding pipeline mapping event sequences to con- tinuous user representations. (b) EAFD agent loop: an LLM-based generator proposes interpretable features from raw sequences, eval- uated by embedding–feature alignment and downstream utility, with reflective updates guiding iteration. (c) Feature outcomes: aligned features recover information encoded in the embedding, while complementary features capture predictive factors missing from it. (d) Em- bedding–feature space decomposition into latent, interpretable, and blind-spot regions. (e) Deployment and refinement: discovered features improve downstream tasks and support targeted encoder refinement (e.g., coverage, robustness, privacy). 3 Proposed Method Embedding-Aware Feature Discovery (EAFD) is an iterative framework for discovering interpretable and complementary features for event-sequence embeddings. Figure 2 illustrates the overall workflow. Given a collection of event sequences S =s i N i=1 with downstream labels y i , a pretrained encoder model f θ maps each sequence to a fixed-dimensional embed- dingz i = f θ (s i ) (Figure 2a). The encoder parameters θ are frozen throughout feature discovery and serve as a stable rep- resentation anchor. Feature candidates are defined as deter- ministic and interpretable mappings g k : S →R M , and are proposed by an LLM-based generator G conditioned on the raw event data and accumulated reflective feedback from pre- vious iterations (Figure 2b). To guide the discovery process, EAFD operates in two compatible regimes: interpretability and performance. Interpretability Regime. This regime aims to explain the information encoded in the latent space by aligning it with human-interpretable concepts. Here, we evaluate whether a generated feature g k recovers information already present in z. This is quantified by the Alignment Score, which measures the predictive consistency or correlation between the embed- dings and the feature-based predictor: A(g k ) = ψ z,g k (S) ,(1) where ψ is the alignment metric (R 2 of a small gradient boosting model mapping generated features to embedding vectors). High A(g k ) indicates that the feature effectively translates latent dimensions into interpretable logic. Performance Regime. This regime seeks to supplement the embeddings with additional predictive signals. The goal is to identify “blind spots”, information relevant to the task but missing fromz. We measure this via the Downstream Utility Score, which compares the loss of an embedding-only model with that of a joint model utilizing both representations: U (g k ) =L z,y −L [z,g k (S)],y .(2) A positive U (g k ) implies that the feature captures comple- mentary structure that improves downstream performance. While distinct in their objectives, these two regimes are employed simultaneously to drive the iterative feedback loop. To introduce additional features, the generator G is condi- tioned on a detailed reflection from the previous step. In the interpretability regime, this reflection includes the alignment scores A(g k ) alongside qualitative examples that distinguish features strongly coupled withz from those that are orthog- onal. In the performance regime, the feedback comprises the utility scores U (g k ) and explicit feature importance rankings derived from the downstream model on the validation set. The generator translates these reflective insights into exe- cutable code definitions for new candidate features. To ensure robustness, if the generated code fails to execute (e.g., due to syntax errors or runtime exceptions), the framework triggers a self-correction debug mode to fix the implementation itera- tively. Successfully executed features are then evaluated and, based on the aggregated signals shown in Figure 2c, catego- rized as aligned (A(g k ) high, U (g k )≈ 0) or complementary (U (g k )> 0). The categorization induces a decomposition of information across the joint feature-embedding space (Fig- ure 2d): some components remain purely latent in the em- bedding, others become interpretable through alignment with discovered features, and blind spots correspond to feature di- mensions that supplement the embedding with missing task- relevant information. The resulting features can be used directly in downstream tasks (Figure 2e), such as attribute classification and behav- ioral understanding (Section 4.2). The identified embedding blind spots can guide targeted encoder refinement, such as controlled adaptation or selective information erasure (Sec- tion 4.3), for privacy-sensitive applications. 4 Experiments This section evaluates the framework in two parts. The first part focuses on downstream performance when complemen- tary features are iteratively added to pretrained embeddings. The second part examines interpretability and analyzes how feature based projections reveal the structure of the embed- ding space. 4.1 Experimental Settings Datasets All datasets consist of transactional or behavioral event se- quences. We evaluate EAFD on four public financial bench- marks and one large-scale proprietary industrial dataset. Age Prediction [ ODS.AI, 2025 ] contains ∼44M transac- tions from 50K users, with relative timestamps, transaction categories, and anonymized amounts. Following a semi- supervised setup, 20K unlabeled sequences are used for em- bedding pretraining, while 30K labeled sequences are used for EAFD training and downstream evaluation. Gender Prediction [ Sber, 2025 ] is a demographic bench- mark with 8.4K labeled user sequences, each defined by rel- ative timestamps, merchant category codes, and transaction types, used to assess projection consistency across targets. Rosbank [ Boosters, 2025 ] includes transaction histories for 10K users over a three-month period, with churn labels avail- able for only 5K users. The remaining unlabeled sequences are leveraged for self-supervised embedding learning, en- abling evaluation in a label-constrained regime. DataFusion [ ODS.AI, 2024 ] targets churn prediction over a six-month horizon using nine months of transactional history. The dataset comprises 13M transactions from 96K users (64K labeled), with each record including timestamps, merchant category codes, currency, and amount. Proprietary Multi-target Dataset.We further validate EAFD on a private banking dataset with millions of users, requiring simultaneous prediction of age group, gender, and a continuous financial outcome forecasted one year ahead, re- flecting a realistic industrial multi-target setting. Implementation Details We use gpt-oss-120b via vLLM on 4×NVIDIA A100 GPUs. We set the context window to 100K tokens and the max- imum output length to 16K tokens. We run EAFD for 5 iterations and evaluate the generated features using a Cat- Boost [ Prokhorenkova et al., 2018 ] model. 4.2 Performance Evaluation We begin with experiments that use embeddings produced by a single encoder model. For each dataset, a pretrained en- coder f θ generates user embeddings z i = f θ (s i ) of fixed dimensionality. To evaluate method performance with em- beddings that capture distinct event-sequence information, we use three self-supervised encoders: CoLES [ Babaev et al., 2022 ] (RNN, contrastive loss), NTP [ Padhi et al., 2021 ] (RNN, NTP loss), and LLM4ES [ Shestov et al., 2025b ] (fine- tuned LLM, text-enriched sequences, NTP loss). The frame- work is run with a fixed feature budget, and the complemen- tary features with the highest uplift are selected. We compare the following approaches: • Embedding-only: z i used as the sole input to the down- stream model. • Aggregation features: statistical event-sequence aggre- gations from pytorch-lifestream [ Sakhno et al., 2025 ] and Featuretools [ Kanter and Veeramachaneni, 2015 ] , concatenated with z i . • LLM-based feature agents: CAAFE and LLMFE, which generate high-level features from tabular inputs composed of baseline aggregations and embeddings. • Non-LLM feature generation: OpenFE [ Zhang et al., 2023 ] , generating features from tabular inputs without joint model training. • AutoML pipelines: LightAutoML [ Vakhrushev et al., 2021 ] , performing integrated feature engineering and model selection on aggregated representations. • EAFD (ours): EAFD-generated features combined with z i under the same feature budget. Table 1 reports results on open-source financial datasets for different backbone embeddings. Across all datasets and tasks, EAFD consistently improves performance over the corresponding base representations, demonstrating that the discovered features capture complementary information not fully encoded in the embedding space. The most pronounced gains occur for NTP embeddings, where EAFD yields large relative improvements across all datasets, including +15.4% on Age Prediction and up to +19.3% on the DataFusion benchmark. EAFD effectively recovers task-relevant struc- ture that is weakly represented in NTP embeddings. EAFD also improves performance for stronger backbones: when applied to CoLES, it achieves consistent gains across all datasets, with particularly large improvements on Rosbank and DataFusion, while EAFD applied to LLM4ES yields smaller but stable improvements. Iterative Feature Generation Dynamics Figure 3 illustrates the iterative nature of the EAFD frame- work and its impact on the Rosbank prediction task. The line plot tracks the steady increase in the validation ROC-AUC score, while the stacked bars detail the volume and functional MethodAgeGenderRosbankDataFusion Acc∆%AUC∆%AUC∆%AUC∆% CoLES0.645–0.888–0.835–0.738– NTP0.540–0.849–0.798–0.648– LLM4ES0.651–0.849– Agg Features + CoLES0.649+0.620.889+0.110.832-0.360.740+0.27 Featuretools + CoLES0.644-0.160.889+0.110.863+3.350.760+2.98 LLMFE (Agg Features + CoLES)0.644-0.160.887-0.110.825-1.200.740+0.27 LLMFE (Featuretools + CoLES)0.644-0.160.8880.000.862+3.230.760+2.98 CAAFE (Agg Features + CoLES)0.638-1.090.877-1.240.840+0.600.747+1.22 CAAFE (Featuretools + CoLES)0.641-0.620.882-0.680.870+4.190.762+3.25 EAFD (CoLES)0.649+0.620.898+1.130.872+4.430.781+5.83 EAFD (NTP)0.623+15.370.870+2.470.865+8.400.773+19.29 EAFD (LLM4ES)0.652+0.15–0.866+2.00– Table 1: Performance comparison of EAFD against baseline embeddings and feature-based agents. Relative improvement (∆%) is reported w.r.t. the corresponding backbone embedding. Standard deviations are approximately: Age±0.004, Gender±0.007, Rosbank±0.005, DataFusion±0.008. Figure 3: EAFD iteration dynamics of Rosbank validation ROC- AUC and feature composition across EAFD iterations. categories of features added at each step. We observe a con- sistent improvement in performance, rising from a baseline of 0.8206 to a peak of 0.8396 by the fourth iteration. In total, EAFD utilizes 37 distinct aggregation functions over the five iterations. Qualitative analysis reveals that the most significant metric gains are often driven by interpretable temporal features, such as the time elapsed between the first and last transaction or transaction frequency within recent windows (e.g., count in the last 30 days). However, the frame- work also successfully identifies complex, high-utility sig- nals that capture behavioral nuances, including the Herfind- ahl–Hirschman Index (HHI) on MCC distributions, Exponen- tially Weighted Moving Averages (EWMA), and autocorrela- tion features applied to transaction amounts. Multi-target Feature Generation In this section, we validate EAFD on a large-scale industrial multi-target dataset. Table 2 summarizes the experimental re- sults, which demonstrate two critical capabilities of the pro- posed framework. First, EAFD consistently improves the performance of state-of-the-art pretrained embeddings.When applied to strong baselines such as CoLES and NTP, EAFD reduces re- gression Mean Absolute Error (MAE) by 3.87% and 1.55%, respectively. These results show that pretrained latent repre- sentations do not fully capture all task-relevant information, and that EAFD systematically recovers additional predictive signal through complementary feature discovery. MethodPrivate Dataset AgeGenderRegression CoLES0.7430.89811373 NTP0.7360.89510839 LLM4ES0.6920.78911462 EAFD (CoLES) 0.756 (+1.75%) 0.901 (+0.33%) 10933 (-3.87%) EAFD (NTP) 0.739 (+0.41%) 0.897 (+0.23%) 10671 (-1.55%) EAFD (LLM4ES) 0.714 (+3.18%) 0.888 (+12.55%) 11108 (-3.09%) Table 2: Performance comparison on the Private Dataset across clas- sification and regression tasks. Percentages denote relative change with respect to the corresponding base embedding; for regression, negative values indicate MAE reduction (improvement). Second, EAFD improves downstream performance when applied to weaker embeddings. Starting from the LLM4ES baseline, which underperforms contrastive encoders, EAFD increases Gender prediction performance by +12.55% and Age accuracy by +3.18%. These results show that EAFD recovers task-relevant semantic information not captured by the original embedding, raising LLM4ES performance to a level comparable with strong contrastive baselines. Beyond quantitative gains, EAFD demonstrates intrinsic adaptability by tailoring feature generation to the semantic requirements of each target, as visualized in Figure 4. For Age prediction, the generator prioritizes Activity patterns (≈ 65%), leveraging lifestyle signals encoded in merchant interactions. In contrast, Gender prediction shifts focus to a balanced combination of Amount (≈ 45%) and Categories (≈ 40%), capturing distinct spending magnitudes and pref- erences. The Regression task triggers a strategic increase in Time-based features, confirming that capturing temporal dy- namics is essential for forecasting. Figure 4: Task-adaptive distribution of discovered feature types. Feature category proportions generated by EAFD for age, gender, and regression tasks. 4.3 Interpretability Analysis To analyze the semantic content of embeddings, we recov- ered 43 EAFD features aggregated into four groups: Amount, Categories, Time, and Activity. Figure 1 reports the alignment score (R 2 ) for three architectures: • CoLES: Prioritizes static information. It dominates in Categories (0.816) and Activity (0.897) but fails to cap- ture temporal dynamics (0.327). • NTP: Shows a specialized profile with peak perfor- mance in Activity (0.942), but struggles with explicit fea- tures like Amount (0.330) and Categories (0.272). • LLM4ES: Excels in temporal modeling, surpassing baselines in Time (0.524). It maintains balanced seman- tic recovery but retrieves fewer Activity markers (0.599) than other encoders. Building on this analysis, we next showcase two concrete examples of how the guidance provided by EAFD can be ap- plied in practice. Encoder Refinement through Gap Analysis Standard pretrained encoders, such as CoLES or NTP, often overlook domain-specific patterns because they are optimized using general-purpose self-supervised objectives.EAFD identifies information gaps by detecting features that are in- sufficiently encoded in the latent space and uses this signal to guide representation refinement. The analysis of the CoLES encoder, illustrated in Figure 1, highlights a critical deficiency in capturing both numer- ical distribution patterns (amounts) and temporal dynamics. To mitigate these ”blind spots”, we enhanced the encoder’s input layer with several transformations, including log(x), exp(x), and Piecewise Linear Encoding [ Gorishniy et al., 2022 ] for transaction amounts, alongside Time2Vec embed- dings [ Kazemi et al., 2019 ] for temporal features. CategoryBase Encoder Enhanced Encoder Amount (R 2 )0.4510.484 (+7.39%) Category (R 2 )0.8160.687 (-15.85%) Time (R 2 )0.3270.608 (+85.68%) Other features (R 2 )0.8970.870 (-3.07%) Downstream (AUC)0.8350.845 (+1.20%) Table 3: Reconstruction Score R 2 of feature classes from embed- ding representations and downstream performance on the Rosbank dataset between Base and Enhanced Encoder. Table 3 compares the reconstruction quality across fea- ture categories and the downstream performances for the en- hanced and baseline CoLES encoders. The refined encoder substantially improves over the base model, increasing re- construction quality for temporal patterns by 85.68% and im- proving the modeling of transaction amounts, while reducing MCC precision (-15.85%). This reallocation of representa- tional capacity toward continuous features results in higher downstream performance, improving churn prediction from 0.835 to 0.845 (+1.20%). Privacy-Preserving Feature Erasure Figure 5: Selective feature erasure in embeddings. Top panels show R 2 reconstruction performance before and after erasing mcc (a) and trx amount (b) feature categories, while the bottom panels report the corresponding relative performance change (∆R 2 ). Privacy-preserving feature erasure studies how specific sensitive attributes can be selectively removed from learned embeddings while preserving their utility for downstream tasks. Here, the erasure process is guided by EAFD, which identifies and targets embedding-aligned feature groups for controlled information removal. We evaluate the effect of removing sensitive information from embeddings using an auxiliary de-correlation regular- izer based on the Hilbert–Schmidt Independence Criterion (HSIC). Given embeddings z ∈R d and a sensitive attribute s, the training objective becomes L =L CoLES + λHSIC(z,s),(3) where HSIC(z,s) penalizes statistical dependence between the learned representation and the sensitive attribute, and λ controls the strength of privacy enforcement. Figure 5 illustrates the obtained results on the DataFusion dataset using CoLES embeddings. We report the predictabil- ity (R 2 ) of different feature groups from the embedding be- fore and after applying HSIC-based de-correlation, together with the corresponding change ∆R 2 . Erasing mcc features substantially reduces their recover- ability (∆R 2 = −17.9 p.p.) with minimal impact on other attributes. Similarly, erasing trx amount primarily suppresses the targeted information (∆R 2 = −22.2 p.p.) while leav- ing unrelated features largely unaffected. Compared to the baseline embeddings trained without attribute erasing, the downstream ROC-AUC decreases by approximately 1-2%, depending on which attribute group is erased. These results demonstrate that the proposed HSIC regu- larization enables fine-grained, attribute-specific information removal. The effect remains mostly confined to the erased feature group, showing that sensitive information can be se- lectively reduced with minimal impact on the overall repre- sentational capacity of the embedding. 5 Ablation studies In this section, we study which components of the framework are necessary to obtain the reported gains. 5.1 Impact of LLM Generator Backbone Llama3.1-8BLlama3.3-70Bgpt-oss-20Bgpt-oss-120B 0.8350.8560.8590.872 Table 4: EAFD performance on the Rosbank (AUC) for different LLM generator backbones. In this ablation we analyze the effect of the LLM backbone used by the feature generator. Table 4 reports downstream performance on the Rosbank dataset across four generator models. While all models can perform reflective feature generation, reasoning-oriented backbones (gpt-oss) consistently produce more diverse and executable features. Smaller instruction- tuned models more frequently fail during code generation, triggering repeated self-correction and limiting scalability when generating more than 10 features. MethodNTPCoLES Embeddings0.6480.738 Featuretools0.728 (+12.3%)0.760 (+3.0%) OpenFE0.730 (+12.7%)0.753 (+2.0%) LightAutoML0.731 (+12.8%)0.760 (+3.0%) LLMFE0.728 (+12.3%)0.760 (+3.0%) CAAFE0.750 (+15.7%)0.762 (+3.3%) EAFD0.773 (+19.3%)0.781 (+5.8%) Table 5: Comparison of automated feature engineering methods across different embeddings. 5.2 Performance across Embedding Backbones Table 5 evaluates feature engineering methods across two event-sequence embedding backbones of different quality. EAFD consistently achieves the largest relative improve- ments on both the weaker NTP embedding (+19.3%) and the stronger CoLES embedding (+5.8%), outperforming all tab- ular feature engineering baselines in both cases. Overall, existing LLM-based feature engineering methods are primarily designed for static tabular inputs and do not ex- plicitly model event-sequence structure. When applied on top of embeddings, feature generation operates in an abstract rep- resentation space without grounding in the original temporal attributes, which limits the ability to recover complementary sequence-level information. This structural mismatch results in systematically weaker and less stable performance gains compared to EAFD. 6 Conclusion WeintroducedEmbedding-AwareFeatureDiscovery (EAFD), a principled framework for jointly reason- ing over learned embeddings and structured features in event-sequence data.Across four open-source financial benchmarks, EAFD consistently outperformed embedding- only and feature-based baselines, achieving relative gains of up to 5.8% for state-of-the-art embeddings and up to 19% for weaker representations.Beyond performance, EAFD diagnosed representational gaps and systematic biases in embedding models and leveraged these signals to design an alternative, more effective training scheme for CoLES, resulting in measurable downstream improvements.By enabling joint feature discovery in the combined embed- ding–feature space, EAFD produced representations that transferred effectively across multiple classification and regression targets on large-scale industrial dataset. Looking forward, this work can be extended toward multi- agent orchestration, where feature generation, selection, and refinement are handled by specialized agents coordinated via embedding-aware signals. Embedding-aware discovery can also be integrated with ensemble embeddings, enabling joint reasoning over multiple representations and adaptive feature exploration under representational uncertainty. More broadly, EAFD’s interpretation signals provide a foundation for auto- matic encoder and model tuning, where identified informa- tion gaps guide architecture, sampling, and training objec- tives, closing the loop between representation learning and system optimization. References [ Abhyankar et al., 2025 ] Nikhil Abhyankar, Parshin Sho- jaee, and Chandan K. Reddy. LLM-FE: Automated fea- ture engineering for tabular data with LLMs as evolution- ary optimizers. arXiv preprint arXiv:2503.14434, 2025. [ Arik and Pfister, 2021 ] Sercan ̈ O Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 35, pages 6679–6687, 2021. [ Babaev et al., 2022 ] Dmitrii Babaev, Nikita Ovsov, Ivan Kireev, Maria Ivanova, Gleb Gusev, Ivan Nazarov, and Alexander Tuzhilin. Coles: Contrastive learning for event sequences with self-supervision. In Proceedings of the 2022 International Conference on Management of Data, pages 1190–1199, 2022. [ Basile et al., 2025 ] Lorenzo Basile, Valentino Maiorca, Diego Doimo, Francesco Locatello, and Alberto Caz- zaniga. Head pursuit: Probing attention specialization in multimodal transformers. In The Thirty-ninth Annual Con- ference on Neural Information Processing Systems, 2025. [ Boosters, 2025 ] Boosters. Rosbank Challenge 1 Overview, 2025. [Online; accessed 11-February-2025]. [ Brisaboa et al., 2018 ] NievesRBrisaboa,Guillermo De Bernardo, Gonzalo Navarro, Tirso V Rodeiro, and Diego Seco. Compact representations of event sequences. In Data Compression Conference, pages 237–246. IEEE, 2018. [ de Winter et al., 2025 ] Casper de Winter, Flavius Frasincar, Bart de Peuter, Vladyslav Matsiiako, Enzo Ido, and Jas- mijn Klinkhamer. Automated feature engineering for auto- mated machine learning. Knowledge-Based Systems, page 113671, 2025. [ Draxler et al., 2025 ] Felix Draxler, Yang Meng, Kai Nel- son, Lukas Laskowski, Yibo Yang, Theofanis Karaletsos, and Stephan Mandt. Transformers for mixed-type event sequences.In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [ Fadeev et al., 2025 ] Egor Fadeev, Dzhambulat Mollaev, Aleksei Shestov, Dima Korolev, Omar Zoloev, Ivan A Kireev, Andrey Savchenko, and Maksim Makarenko. Latte: Learning aligned transactions and textual embed- dings for bank clients. In Proceedings of the 2025 Confer- ence on Empirical Methods in Natural Language Process- ing: Industry Track, pages 2635–2647, 2025. [ Faubel and Schmid, 2024 ] Leonhard Faubel and Klaus Schmid. Mlops: A multiple case study in industry 4.0. In 2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA), pages 01– 08. IEEE, 2024. [ Gijsbers et al., 2024 ] Pieter Gijsbers, Marcos LP Bueno, Stefan Coors, Erin LeDell, S ́ ebastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. Amlb: an automl benchmark. Journal of Machine Learning Re- search, 25(101):1–65, 2024. [ Gorishniy et al., 2022 ] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991–25004, 2022. [ Guo et al., 2024 ] Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang.Ds-agent: Automated data science by empowering large language models with case-based reasoning.arXiv preprint arXiv:2402.17453, 2024. [ Han et al., 2025 ] Sungwon Han, Sungkyu Park, and Seun- geon Lee. Tabular feature discovery with reasoning type exploration. arXiv preprint arXiv:2506.20357, 2025. [ Harle et al., 2025 ] Ruben Harle, Felix Friedrich, Manuel Brack, Stephan W ̈ aldchen, Bj ̈ orn Deiseroth, Patrick Schramowski, and Kristian Kersting. Measuring and guid- ing monosemanticity. In The Thirty-ninth Annual Confer- ence on Neural Information Processing Systems, 2025. [ Hollmann et al., 2023 ] Noah Hollmann, Samuel M ̈ uller, and Frank Hutter. Large language models for automated data science: Introducing caafe for context-aware auto- mated feature engineering. In Proceedings of the Thirty- seventh Conference on Neural Information Processing Systems (NeurIPS), 2023. [ Horn et al., 2019 ] Franziska Horn, Robert T. Pack, and Michael Rieger. The AutoFeat Python library for auto- mated feature engineering and selection. In ECMLPKDD Workshop on Automating Data Science, 2019. [ Kanter and Veeramachaneni, 2015 ] James Max Kanter and Kalyan Veeramachaneni.Deep feature synthesis: To- wards automating data science endeavors. In IEEE Inter- national Conference on Data Science and Advanced Ana- lytics (DSAA), pages 1–10, 2015. [ Kazemi et al., 2019 ] Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, San- jay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning a vector represen- tation of time. arXiv preprint arXiv:1907.05321, 2019. [ Klenitskiy et al., 2025 ] Anton Klenitskiy, Artem Fatkulin, Daria Denisova, Anton Pembek, and Alexey Vasilev. En- code me if you can: Learning universal user representa- tions via event sequence autoencoding. In Proceedings of the Recommender Systems Challenge 2025, pages 26–30. 2025. [ Ko et al., 2025 ] Myeongseob Ko, Hoang Anh Just, Charles Fleming, Ming Jin, and Ruoxi Jia. Probing hidden knowl- edge holes in unlearned llms. In The Thirty-ninth An- nual Conference on Neural Information Processing Sys- tems, 2025. [ Li et al., 2024 ] Yinhong Li, Kui Wang, and Hanwen Qu. Autocross: Automatic feature interaction learning with self-attention neural networks integrating feature graph in- formation. In 2024 2nd International Conference on Com- puter Network Technology and Electronic and Information Engineering (CNTEIE), pages 17–21. IEEE, 2024. [ Li et al., 2025 ] Yihao Li, Saeed Salehi, Lyle Ungar, and Konrad P Kording. Does object binding naturally emerge in large pretrained vision transformers?In The Thirty- ninth Annual Conference on Neural Information Process- ing Systems, 2025. [ Liu et al., 2025a ] Daogao Liu, Edith Cohen, Badih Ghazi, Peter Kairouz, Pritish Kamath, Alexander Knop, Ravi Ku- mar, Pasin Manurangsi, Adam Sealfon, Da Yu, et al. Ura- nia: Differentially private insights into ai use.arXiv preprint arXiv:2506.04681, 2025. [ Liu et al., 2025b ] Jiongnan Liu, Zhicheng Dou, Jian-Yun Nie, Zhenlin Chen, Guoyu Tang, Sulong Xu, and Ji-Rong Wen. Enhancing sequential personalized product search with external out-of-sequence knowledge. ACM Transac- tions on Information Systems, 43(4):1–25, 2025. [ Louhi, 2023 ] Jarkko Louhi. An empirical study on feature data management practices and challenges. Master’s the- sis, University of Helsinki, Faculty of Science, 2023. [ Maniparambil et al., 2024 ] Mayug Maniparambil, Raiym- bek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mo- hamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel E. O’Connor. Do vision and lan- guage encoders represent the world similarly? In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14334–14343, June 2024. [ Mollaev et al., 2025 ] Dzhambulat Mollaev, Ivan Kireev, Mikhail Orlov, Alexander Kostin, Ivan Karpukhin, Maria Postnova, Gleb Gusev, and Andrey Savchenko.Mul- timodal banking dataset:Understanding client needs through event sequences. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 6476–6480, 2025. [ ODS.AI, 2024 ] ODS.AI. Data fusion contest 2024, 2024. [Online; accessed 11-February-2025]. [ ODS.AI, 2025 ] ODS.AI. Sberbank Sirius Lesson Competi- tion, 2025. [Online; accessed 11-February-2025]. [ Padhi et al., 2021 ] Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman. Tabular transformers for modeling multivariate time series. In ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3565–3569. IEEE, 2021. [ Prokhorenkova et al., 2018 ] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categor- ical features. Advances in neural information processing systems, 31, 2018. [ Sakhno et al., 2025 ] Artem Sakhno, Ivan Kireev, Dmitrii Babaev, Maxim Savchenko, Gleb Gusev, and Andrey Savchenko. Pytorch-lifestream: Learning embeddings on discrete event sequences. In Proceedings of the Thirty- Fourth International Joint Conference on Artificial Intelli- gence, pages 11104–11108, 2025. [ Sber, 2025 ] Sber.Sberbank Gender Competition, 2025. [Online; accessed 11-February-2025]. [ Seputis et al., 2025 ] Dominykas Seputis, Yongkang Li, Karsten Langerak, and Serghei Mihailov. Rethinking the privacy of text embeddings: A reproducibility study of “text embeddings reveal (almost) as much as text”. In Proceedings of the Nineteenth ACM Conference on Rec- ommender Systems, pages 822–831, 2025. [ Shestov et al., 2025a ] Aleksei Shestov, Anton Klenitskiy, Daria Denisova, Amurkhan Dzagkoev, Daniil Petrovich, Andrey Savchenko, and Maksim Makarenko. Topologi- cal metric for unsupervised embedding quality evaluation, 2025. [ Shestov et al., 2025b ] Aleksei Shestov, Omar Zoloev, Mak- sim Makarenko, Mikhail Orlov, Egor Fadeev, Ivan Kireev, and Andrey Savchenko. Llm4es: Learning user embed- dings from event sequences via large language models. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 5238– 5242, 2025. [ Tennenholtz et al., 2024 ] Guy Tennenholtz, Yinlam Chow, ChihWei Hsu, Jihwan Jeong, Lior Shani, Azamat Tulep- bergenov, Deepak Ramachandran, Martin Mladenov, and Craig Boutilier. Demystifying embedding spaces using large language models. In The Twelfth International Con- ference on Learning Representations, 2024. [ Tinaz et al., 2025 ] Berk Tinaz, Zalan Fabian, and Mahdi Soltanolkotabi. Emergence and evolution of interpretable concepts in diffusion models. In The Thirty-ninth An- nual Conference on Neural Information Processing Sys- tems, 2025. arXiv:2510.21465. [ Vakhrushev et al., 2021 ] AntonVakhrushev,Alexander Ryzhkov, Maxim Savchenko, Dmitry Simakov, Rinchin Damdinov, and Alexander Tuzhilin. Lightautoml: Automl solution for a large financial services ecosystem. arXiv preprint arXiv:2109.01528, 2021. [ Verdonck et al., 2024 ] Tim Verdonck, Bart Baesens, Mar ́ ıa ́ Oskarsd ́ ottir, and Seppe vanden Broucke. Special issue on feature engineering editorial, 2024. [ Xiao et al., 2024 ] Zhibo Xiao, Luwei Yang, Tao Zhang, Wen Jiang, Wei Ning, and Yujiu Yang.Deep evolu- tional instant interest network for ctr prediction in trigger- induced recommendation. In Proceedings of the 17th ACM International Conference on Web Search and Data Min- ing, pages 846–854, 2024. [ Zhang et al., 2023 ] Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: Automated feature generation with expert-level performance. In International Confer- ence on Machine Learning, pages 41880–41901. PMLR, 2023. [ Zhu et al., 2025 ] Yuanshao Zhu, James Jianqiao Yu, Xi- angyu Zhao, Xun Zhou, Liang Han, Xuetao Wei, and Yux- uan Liang. Unitraj: Learning a universal trajectory founda- tion model from billion-scale worldwide traces. Advances in Neural Information Processing Systems, 38, 2025.