Paper deep dive
Learning to Predict, Discover, and Reason in High-Dimensional Event Sequences
Hugo Math
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/22/2026, 5:40:08 AM
Summary
This dissertation presents a framework for automated fault diagnostics in high-dimensional event sequences (specifically Diagnostic Trouble Codes in vehicles) by integrating Transformer-based sequence modeling, causal discovery (OSCAR, CARGO, TRACE), and multi-agent systems (CAREP) for Boolean rule synthesis.
Entities (7)
Relation Signals (3)
Hugo Math â developed â CarFormer
confidence 95% · Part I introduces Transformer-based architectures for predictive maintenance... CarFormer
CAREP â synthesizes â Boolean EP rules
confidence 95% · multi-agent system (CAREP) that automates the synthesis of Boolean EP rules.
OSCAR â recovers â Direct Causes
confidence 90% · OSCAR and CARGO recover the direct causes of each outcome label
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Electronic control units (ECUs) embedded within modern vehicles generate a large number of asynchronous events known as diagnostic trouble codes (DTCs). These discrete events form complex temporal sequences that reflect the evolving health of the vehicle's subsystems. In the automotive industry, domain experts manually group these codes into higher-level error patterns (EPs) using Boolean rules to characterize system faults and ensure safety. However, as vehicle complexity grows, this manual process becomes increasingly costly, error-prone, and difficult to scale. Notably, the number of unique DTCs in a modern vehicle is on the same order of magnitude as the vocabulary of a natural language, often numbering in the tens of thousands. This observation motivates a paradigm shift: treating diagnostic sequences as a language that can be modeled, predicted, and ultimately explained. Traditional statistical approaches fail to capture the rich dependencies and do not scale to high-dimensional datasets characterized by thousands of nodes, large sample sizes, and long sequence lengths. Specifically, the high cardinality of categorical event spaces in industrial logs poses a significant challenge, necessitating new machine learning architectures tailored to such event-driven systems. This thesis addresses automated fault diagnostics by unifying event sequence modeling, causal discovery, and large language models (LLMs) into a coherent framework for high-dimensional event streams. It is structured in three parts, reflecting a progressive transition from prediction to causal understanding and finally to reasoning for vehicle diagnostics. Consequently, we introduce several Transformer-based architectures for predictive maintenance, scalable sample- and population-level causal discovery frameworks and a multi-agent system that automates the synthesis of Boolean EP rules.
Tags
Links
- Source: https://arxiv.org/abs/2603.16313v2
- Canonical: https://arxiv.org/abs/2603.16313v2
Trouble viewing inline? Open PDF directly â
Full Text
430,491 characters extracted from source content.
Expand or collapse full text
Dissertation Learning to Predict, Discover, and Reason in High-Dimensional Event Sequences Exploiting the Predictive-Causal Duality of Autoregressive Models for Vehicle Diagnostics Automation Hugo Math Submitted in March 2026 at the Faculty of Applied Computer Science University of Augsburg to obtain the doctoral degree Dr. rer. nat. Faculty of Applied Computer Science University of Augsburg arXiv:2603.16313v2 [cs.AI] 19 Mar 2026 Advisor:Prof. Dr. rer. nat. Rainer Lienhart Reviewers:Prof. Dr. rer. nat. Rainer Lienhart Prof. Dr. rer. nat. Elisabeth Andre Thesis defense: June 2026 Abstract Electronic control units (ECUs) embedded within modern vehicles generate a large num- ber of asynchronous events known as diagnostic trouble codes (DTCs). These discrete events form complex temporal sequences that reflect the evolving health of the vehi- cleâs subsystems. In the automotive industry, domain experts manually group these codes into higher-level error patterns (EPs) using Boolean rules to characterize system faults and ensure safety. However, as vehicle complexity grows, this manual process becomes increasingly costly, error-prone, and diïŹicult to scale. Consequently, man- ufacturers urgently require automated solutions capable of anticipating malfunctions, reducing downtime, and lowering warranty and repair costs. Notably, the number of unique DTCs in a modern vehicle is on the same order of magnitude as the vocabulary of a natural language, often numbering in the tens of thousands. This observation moti- vates a paradigm shift: treating diagnostic sequences as a language that can be modeled, predicted, and ultimately explained. Traditional statistical approaches fail to capture the rich dependencies and do not scale to high-dimensional datasets characterized by thousands of nodes, large sample sizes, and long sequence lengths. Specifically, the high cardinality of categorical event spaces in industrial logs poses a significant challenge, necessitating new machine learning architectures tailored to such event-driven systems. This thesis addresses automated fault diagnostics by unifying event sequence model- ing, causal discovery, and large language models (LLMs) into a coherent framework for high-dimensional event streams. It is structured in three parts, reflecting a progressive transition from prediction to causal understanding and finally to reasoning for vehicle diagnostics. Part I introduces Transformer-based architectures for predictive mainte- nance from DTC sequences. Part I develops a suite of three scalable causal discovery frameworks that leverage pretrained autoregressive models as neural density estimators. OSCAR and CARGO recover the direct causes of each outcome label at the sample- and population-level respectively, while TRACE addresses a complementary problem: recovering the event-to-event causal graph from a single sequence. Finally, Part I integrates these causal structures with large language models in a multi-agent system ( CAREP) that automates the synthesis of Boolean EP rules. Together, these contribu- tions establish a coherent, scalable pipeline from raw event streams to human-readable diagnostic knowledge. i Acknowledgments First and foremost, I would like to express my deepest gratitude to my supervisor, Prof. Rainer Lienhart, for his openness to ideas, constant investment throughout my PhD, and scientific rigor. His support and vision were essential in shaping both this thesis and my growth as a researcher. I am grateful for the opportunity to have conducted this research under his guidance. I also extend my sincere thanks to Prof. Elisabeth Andre for her time and effort in reviewing this thesis. I am also sincerely thankful to my PhD colleagues, Julian Lorenz, Daniel Kienzle, Robin Schön and Mrunmai Phatak, for their help, support, and camaraderie. They made this journey not only productive but also enjoyable. The countless discussions, collaborations, and shared moments have been a source of motivation and inspiration. I warmly thank my BMW supervisors, Stefan Oelsner for his deep expertise, creativity, and knowledge, and Stefan Mueller for his rigorous guidance, constructive feedback, and support in every phase of the Promotion program. I also thank Stefan Kirsch for his interest, support and the opportunity to connect academic research with real-world industrial challenges at BMW. I am thankful to Michael Decker and Ingo Stock for their support and time reviewing this thesis. Their insights and the collaboration between BMW and the University of Augsburg greatly enriched this work. Additionally, I would like to acknowledge all my BMW colleagues, particularly Bernd Postmaier for his orientation and advice, Stefan Benesch for operational guidance, and Lisa Jobst for her positive spirit in the oïŹice. Finally, I am deeply grateful to my family, especially my parents, Bertrand and Audrey Math, for their constant encouragement and unwavering support. They greatly helped me stay motivated and keep moving forward. I hope I can make them as proud as I am to have them as parents. I also want to thank my sister, Laura, for her presence, and my two grandfathers, Henri and Jean, whose memory continues to inspire me. And of course, my close friends, particularly Nathan Haudot, for his encouragement and companionship throughout this journey. v Contents Abstractiii Acknowledgmentsv Contentsvii 1 Introduction1 1.1 From Industrial Challenge to Scientific Question. . . . . . . . . . . . . .2 1.2 Research Areas, Challenges and Open Problems. . . . . . . . . . . . . . .2 1.3 Contributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 1.3.1 Predictive Sequence Modeling of Vehicle Diagnostics. . . . . . . .4 1.3.2 Multi-Label Causal Discovery in High-Dimensional Event Sequences6 1.3.3 Scalable Sample-Level Event-to-Event Causal Discovery in Event Sequences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 1.3.4 Automated Causal Reasoning with Multi-Agent Systems for Error Pattern Rule Automation . . . . . . . . . . . . . . . . . . . . . . .11 1.3.5 Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 1.4 List of Publications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 1.5 Thesis Outline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 I Predictive Sequence Modeling15 2 Foundations of Event Sequence Modeling17 2.1 Hawkes Processes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 2.1.1 Background. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 2.1.2 Self-Exciting Property. . . . . . . . . . . . . . . . . . . . . . . . .18 2.1.3 Mutually Exciting Hawkes Processes. . . . . . . . . . . . . . . . .19 2.1.4 Limitations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 2.1.5 The High-Dimensional Discrete Event Stream Regime. . . . . . .20 2.2 Language Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 2.2.1 Word Level. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 2.2.2 Word Embeddings. . . . . . . . . . . . . . . . . . . . . . . . . . .21 2.2.3 Transformers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 2.2.3.1 Attention. . . . . . . . . . . . . . . . . . . . . . . . . . .22 2.2.3.2 Feed-forward Layers. . . . . . . . . . . . . . . . . . . . .22 2.2.3.3 Normalization. . . . . . . . . . . . . . . . . . . . . . . .23 2.2.3.4 Positional Encoding. . . . . . . . . . . . . . . . . . . . .23 vii Contents 2.2.3.5 Autoregressive Transformers. . . . . . . . . . . . . . . .23 2.3 Metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24 2.3.1 Next-Event Prediction. . . . . . . . . . . . . . . . . . . . . . . . .24 2.3.2 Next-Time Prediction. . . . . . . . . . . . . . . . . . . . . . . . .24 2.3.2.1 Confident Predictive Maintenance Window. . . . . . . .25 2.4 Dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 2.4.1 Vehicular Fault Sequences. . . . . . . . . . . . . . . . . . . . . . .26 2.4.1.1 Diagnostic Trouble Code: DTCs. . . . . . . . . . . . . .27 2.4.1.2 Error Patterns. . . . . . . . . . . . . . . . . . . . . . . .27 2.4.1.3 Experimental Datasets used Across Chapters. . . . . . .27 3 Autoregressive Architectures for Error Pattern Prediction29 3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 3.2 Background. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 3.3 Dataset Description. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30 3.3.1 Time and Mileage. . . . . . . . . . . . . . . . . . . . . . . . . . .31 3.4 Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31 3.4.1 CarFormer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32 3.4.1.1 Embeddings. . . . . . . . . . . . . . . . . . . . . . . . .32 3.4.1.2 Multi-Task Learning. . . . . . . . . . . . . . . . . . . . .33 3.4.2 EPredictor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 3.4.2.1 Limitations of Next-Event Prediction. . . . . . . . . . .35 3.4.2.2 Multi-Label Event Prediction. . . . . . . . . . . . . . . .35 3.4.2.3 Imbalanced Labels. . . . . . . . . . . . . . . . . . . . . .37 3.4.2.4 Losses. . . . . . . . . . . . . . . . . . . . . . . . . . . . .37 3.5 Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37 3.5.1 CarFormer Pre-training. . . . . . . . . . . . . . . . . . . . . . . .38 3.5.1.1 Ablation I: CarFormer Embeddings. . . . . . . . . . . .38 3.5.2 EPredictor Experiments. . . . . . . . . . . . . . . . . . . . . . . .39 3.5.3 Ablation I: EPredictor Architecture & CPMW. . . . . . . . . . .39 3.6 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40 3.7 Outlook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41 4 Multimodal Sequence Modeling for Error Pattern Classification43 4.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43 4.2 Related Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44 4.2.1 Failure Detection in Vehicles. . . . . . . . . . . . . . . . . . . . .44 4.2.2 Multimodal Fusion & Learning. . . . . . . . . . . . . . . . . . . .44 4.3 Dataset Extension with Environmental Conditions. . . . . . . . . . . . .46 4.3.1 Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46 4.4 BiCarFormer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48 4.4.1 DTC Embeddings. . . . . . . . . . . . . . . . . . . . . . . . . . .48 4.4.2 Positional Embeddings. . . . . . . . . . . . . . . . . . . . . . . . .48 4.4.3 Environmental Embeddings. . . . . . . . . . . . . . . . . . . . . .48 viii Contents 4.4.4 Co-attention for Vehicle Event Sequences. . . . . . . . . . . . . .49 4.4.5 Multimodal Learning. . . . . . . . . . . . . . . . . . . . . . . . . .51 4.5 Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51 4.5.1 Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51 4.5.2 Metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 4.5.3 Multi-label Classification Performance Analysis. . . . . . . . . . .52 4.5.4 Cross-Attention Scores Interpretations. . . . . . . . . . . . . . . .53 4.5.5 Error Pattern in Battery Electric Vehicles. . . . . . . . . . . . . .55 4.6 Applications to Downstream Tasks. . . . . . . . . . . . . . . . . . . . . .56 4.7 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 4.8 Outlook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 I Causal Discovery in High-Dimensional Event Sequences via Neural Density Estimation59 5 Foundations of Information Theory and Causal Discovery61 5.1 Information Theory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61 5.1.1 Marginal, Conditional, and Relative Entropy. . . . . . . . . . . .61 5.1.1.1 Definitions. . . . . . . . . . . . . . . . . . . . . . . . . .61 5.1.1.2 Properties. . . . . . . . . . . . . . . . . . . . . . . . . .62 5.1.2 Mutual Information and its variants. . . . . . . . . . . . . . . . .63 5.1.2.1 Definitions. . . . . . . . . . . . . . . . . . . . . . . . . .63 5.1.2.2 Properties. . . . . . . . . . . . . . . . . . . . . . . . . .64 5.2 Graphical Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64 5.2.1 Bayesian Networks. . . . . . . . . . . . . . . . . . . . . . . . . . .64 5.2.1.1 Motivation. . . . . . . . . . . . . . . . . . . . . . . . . .64 5.2.1.2 Definitions. . . . . . . . . . . . . . . . . . . . . . . . . .64 5.3 Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65 5.3.1 Causal Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . .65 5.3.1.1 Assumptions. . . . . . . . . . . . . . . . . . . . . . . . .66 5.3.1.2 Classes of Causal Discovery Algorithms. . . . . . . . . .66 5.3.2 Multi-Label Causal Discovery. . . . . . . . . . . . . . . . . . . . .67 5.3.3 Evaluation Metrics for Multi-Label Causal Discovery. . . . . . . .67 5.3.3.1 Set-Based Metrics. . . . . . . . . . . . . . . . . . . . . .68 5.3.4 Evaluation Metrics for Event-to-Event Causal Discovery. . . . . .68 5.3.5 Structural Hamming Distance (SHD). . . . . . . . . . . . . . . . .68 6 Sample-Level Multi-Label Causal Discovery69 6.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69 6.2 Related Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70 6.2.1 Neural Autoregressive Density Estimation. . . . . . . . . . . . . .70 6.2.2 Transformers as Causal Learners. . . . . . . . . . . . . . . . . . .71 6.2.3 Multi-label Causal Discovery. . . . . . . . . . . . . . . . . . . . .71 ix Contents 6.3 Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71 6.3.1 Autoregressive Event Sequence Models.. . . . . . . . . . . . . . .71 6.3.2 Assumptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72 6.3.3 Lemmas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73 6.3.4 Conditional Mutual Information Estimation via Autoregressive Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .74 6.3.5 Sequential Markov Boundary Recovery. . . . . . . . . . . . . . . .75 6.3.6 Computation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77 6.4 Causal Indicator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78 6.5 Empirical Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79 6.5.1 Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79 6.5.2 Vehicle Event Sequences Dataset. . . . . . . . . . . . . . . . . . .79 6.5.3 Comparisons. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79 6.5.4 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80 6.6 Limitations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81 6.7 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82 6.8 Outlook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83 7 Population-Level Multi-Label Causal Discovery85 7.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85 7.2 Related Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85 7.3 Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87 7.3.1 Structural Fusion of Markov Boundaries. . . . . . . . . . . . . . .87 7.3.2 Aggregation under Imperfect CI-tests. . . . . . . . . . . . . . . .88 7.3.3 Adaptive Fusion for Structural Discovery in Long-Tail Distributions89 7.4 Empirical Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90 7.4.1 Benchmark Against Multi-Label Causal Discovery Methods. . . .90 7.4.2 Ablation on Aggregation Criteria for Phase 2. . . . . . . . . . . .90 7.5 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92 7.6 Outlook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92 8 Sample-Level Event-to-Event Causal Discovery93 8.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93 8.2 Related Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94 8.2.1 Causal Discovery in Event Sequences. . . . . . . . . . . . . . . . .94 8.3 Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95 8.3.1 Data-Generating Process. . . . . . . . . . . . . . . . . . . . . . .95 8.3.2 Sample-Level Causality. . . . . . . . . . . . . . . . . . . . . . . .96 8.3.3 Summary-Level Causality. . . . . . . . . . . . . . . . . . . . . . .96 8.3.4 Autoregressive Density Estimation. . . . . . . . . . . . . . . . . .97 8.3.5 Assumptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .97 8.4 Single Stream Causal Discovery. . . . . . . . . . . . . . . . . . . . . . . .98 8.4.1 Conditional Mutual Information. . . . . . . . . . . . . . . . . . .98 8.4.2 Estimation and Approximation Error. . . . . . . . . . . . . . . .98 x Contents 8.4.3 Identifiability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101 8.4.4 Lagged Effects via Simulated Interventions. . . . . . . . . . . . .102 8.5 Algorithm: Parallel Causal Discovery. . . . . . . . . . . . . . . . . . . . .103 8.5.1 Scalability of TRACE. . . . . . . . . . . . . . . . . . . . . . . . .104 8.6 Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105 8.6.1 Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105 8.6.2 Comparative Analysis. . . . . . . . . . . . . . . . . . . . . . . . .106 8.6.3 Scalability and Robustness Analysis. . . . . . . . . . . . . . . . .106 8.6.4 Application to Vehicles Diagnostics. . . . . . . . . . . . . . . . . .108 8.7 Limitations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .109 8.7.1 Causal SuïŹiciency. . . . . . . . . . . . . . . . . . . . . . . . . . .109 8.7.2 Temporal Precedence & Instantaneous Effects. . . . . . . . . . . .110 8.7.3 Consistency Through Time. . . . . . . . . . . . . . . . . . . . . .111 8.7.4 Amortization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .111 8.8 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .111 8.9 Outlook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112 I Automated Causal Reasoning and Logic Synthesis115 9 Multi-Agent Causal Reasoning for Automated Error Pattern Rule Synthesis117 9.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .117 9.2 Related Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119 9.2.1 Multi-Agent Fault Diagnosis. . . . . . . . . . . . . . . . . . . . .119 9.2.2 Error Pattern Rule Automation. . . . . . . . . . . . . . . . . . . .119 9.3 Methodology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .120 9.3.1 Causal Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . .120 9.3.2 Contextual Information. . . . . . . . . . . . . . . . . . . . . . . .122 9.4 Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122 9.4.1 Evaluation Metrics. . . . . . . . . . . . . . . . . . . . . . . . . . .122 9.4.1.1 Structural Evaluation. . . . . . . . . . . . . . . . . . . .123 9.4.1.2 Semantic Evaluation. . . . . . . . . . . . . . . . . . . . .123 9.4.2 Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .123 9.4.3 Comparison. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .123 9.4.4 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .124 9.5 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125 9.6 Outlook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126 10 Conclusion and Outlook131 10.1 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .131 10.2 Outlook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133 10.2.1 Toward a Unified Causal Graph over Events and Outcomes. . . .133 10.2.2 Toward Multimodal and Multi-Resolution Causal Discovery. . . .134 xi Contents 10.2.3 Unseen Domain and Continuous Learning. . . . . . . . . . . . . .134 10.2.4 Industrial Foundation Models. . . . . . . . . . . . . . . . . . . . .135 List of Figures137 List of Tables143 Bibliography145 A Appendix of Part I167 A.1 Error Pattern Prediction. . . . . . . . . . . . . . . . . . . . . . . . . . . .167 A.1.1 Multivariate Hawkes Process Log-likelihood Derivation. . . . . . .167 A.1.2 Models Definition. . . . . . . . . . . . . . . . . . . . . . . . . . . .167 A.1.3 Random Event Injection. . . . . . . . . . . . . . . . . . . . . . . .168 A.2 Multimodal Error Pattern Offline Prediction. . . . . . . . . . . . . . . . .168 B Appendix of Part I171 B.1 Sample-Level Multi-Label Causal Discovery. . . . . . . . . . . . . . . . .171 B.1.1 PyCausalFS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .171 B.1.2 MI-MCF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .171 B.1.3 NADEs Quality.. . . . . . . . . . . . . . . . . . . . . . . . . . . .171 B.1.4 Sampling Procedure for CMI. . . . . . . . . . . . . . . . . . . . .171 B.1.5 Sampling Number. . . . . . . . . . . . . . . . . . . . . . . . . . .173 B.1.6 Dynamic Thresholding. . . . . . . . . . . . . . . . . . . . . . . . .173 B.1.7 Implementation of OSCAR. . . . . . . . . . . . . . . . . . . . . .173 B.2 Population-Level Multi-Label Causal Discovery. . . . . . . . . . . . . . .176 B.2.1 Definition of Scoring Criteria. . . . . . . . . . . . . . . . . . . . .176 B.2.2 Criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .177 B.2.3 Frequency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .177 B.2.4 Expected FPR Adjustment. . . . . . . . . . . . . . . . . . . . . .177 B.2.5 Mutual Information. . . . . . . . . . . . . . . . . . . . . . . . . .177 B.2.6 Implementation of Phase 2. . . . . . . . . . . . . . . . . . . . . .178 B.3 Sample-Level Event-to-Event Causal Discovery. . . . . . . . . . . . . . .180 B.3.1 Flexible Assumptions. . . . . . . . . . . . . . . . . . . . . . . . .180 B.3.2 Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180 B.3.3 Additional Ablations. . . . . . . . . . . . . . . . . . . . . . . . . .181 B.3.3.1 Number of ParticlesN. . . . . . . . . . . . . . . . . . .181 B.3.3.2 Empirical Validation ofΔ-Strong Faithfulness.. . . . . . .181 B.3.3.3 Scaling Law for Causal Identifiability. . . . . . . . . . .183 B.3.4 Additional Summary Causal Graph. . . . . . . . . . . . . . . . . .183 B.4 CAREP: Multi-Agent System for Automatic Rule Generation. . . . . . .183 B.4.1 Input Format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .183 xii Glossary CAREPCausalAutomatedReasoning forErrorPattern, a multi-agent system for automated error rule genera- tion.i,11,117,132 CARGOCausalAggregation viaRobustGraphOperations, a scalable multi-label causal discovery framework de- signed to construct global causal structures from a batch of high-dimensional event sequences.7,85 CML Causal Machine Learning is a field of machine learn- ing that learns cause-and-effect relationships in data to improve robustness of machine learning models (intervention, counterfactual, causal graphs).4,118 DNN Deep Neural Networks, neural network architectures comprising multiple layers of parameterized transfor- mations, enabling the hierarchical learning of repre- sentations from raw data. 1 DTC Diagnostic Trouble Code, standardized codes gener- ated by a vehicleâs electronic control unit to indicate a precise event within the vehicle (malfunctions, soft- ware update,·). i,4,27 ECU Electronic Control Unit, embedded system control- ling one or more electrical systems or subsystems in a vehicle.i,1 EPError Pattern, a higher-level abstraction of DTC se- quences defined by domain experts to characterize specific vehicle faults using Boolean rules, example shown in Fig. 9.1.1,27 LLM Large Language Model refers to a Transformer model with billions of parameters, typically trained via self- supervised learning objectives on a large-scale text corpora.i,1,4,113,117 NADE A Neural Autoregressive Density Estimator is an au- toregressive model that factorizes the joint distribu- tion over a sequence into a product of conditionals, each parameterized by a neural network. 7,69,70 xiii Contents NLPNatural Language Processing, the study of compu- tational techniques for processing human language. 20 OSCAROne-ShotCausalAutoRegressive Discovery is a causal discovery method described in Chapter6that infers a directed acyclic graph representing events- to-outcome relationships from a single sequence of events leading to outcomes. 7,85 RAG Retrieval-Augmented Generation is a technique that improves the accuracy of the LLM answers by allow- ing the model to access and cite external data sources before generating a response.4,118 TPPTemporal Point Process, a stochastic process used to model sequences of discrete events over continuous time.31,112 TRACETemporalReconstruction viaAutoregressiveCausal Estimation is a causal discovery method described in Chapter8that infers a summary and instance-time causal graph [6] from a single sequence of discrete events.9 xiv Notations We summarize the notation used throughout this thesis here. We followed the standard notation fromDeep Learning[42] available athttps://github.com/goodfeli/dlbook_ notation/ Numbers and Arrays aA scalar (integer or real) aA vector A matrix A tensor Sets and Graphs A set XAn alphabet RThe set of real numbers 0,1The set containing 0 and 1 0,1, . . . , nThe set of all integers between0andn [a, b]The real interval includingaandb (a, b]The real interval excludingabut includingb A subtraction, i.e., the set containing the elements ofAthat are not inB GA graph DA dataset Pa G (X i )The parents ofX i inG xv Contents Probability and Information Theory XA discrete random variable with alphabetX X t A random variable referred to as an event occurring at a time steptin a sequence x t A realization of a random variable at a time stept YA discrete random variable characterizing an outcome or a label (e.g., an error pattern) drawn fromY X=X 1 ,·, X n A set of discrete random variables P(X)A probability distribution over a discrete random variableX P Ξ (X)A probability distribution parametrized byΞ, e.g., a pre- trained model Ë P ( X ) A probability distribution over a discrete random variable X estimated empirically based on a data sample p(x)The probability mass function for the random variable,Xi.e., p(x) =P(X=x), xâ X. We denotep(x)instead ofp X (x) for convenience XâŒPRandom variableXhas distributionP E p(x) [f(X)]orEf(X)Expectation off(X)with respect toP(X) Var(f(X))Variance off(X)underP(X) H(X)Entropy of the random variableX D KL (Pâ„Q)Kullback-Leibler divergence ofPandQ N(x;ÎŒ,ÎŁ)Gaussian distribution overxwith meanÎŒand covarianceÎŁ Functions f:AâBThe functionfwith domainAand rangeB logx,lnxNatural logarithm ofx, we follow the convention thatlnx= logx Ï(x)Logistic sigmoid, 1 1+exp(âx) xvi 1 Introduction The increasing digitization of modern vehicles has transformed automotive systems into complex, interconnected networks of electronic control units (ECUs) continuously gen- erating streams of Diagnostic Trouble Codes (DTCs). Each vehicle now asynchronously produces thousands of discrete events that document the behavior and health of its subsystems. These data consist of vast collections of sequences, each containing hun- dreds of events drawn from a large vocabulary of unique codes, effectively forming high- dimensional datasets. For decades, automotive manufacturers have relied on domain experts to manually analyze these codes and define higher-level Boolean rules called Error Patterns (EPs) to characterize system faults and ensure safety. However, with the increasing complexity of vehicles and the proliferation of embedded software, this manual process has become prohibitively expensive, time-consuming, and error-prone. This challenge illustrates a broader industrial problem: how can we automatically learn, understand, and reason about event-driven systems? The answer is not merely one of prediction, but of understanding why events occur and how they interact with each other, thereby transforming data-driven systems into explainable and autonomous diagnostic agents. Over the past decade, the rise of Deep Neural Networks ( DNNs) has profoundly re- shaped the landscape of artificial intelligence. From image recognition to speech under- standing, their ability to model complex, nonlinear relationships has led to significant progress across multiple fields. Among these advances, Transformer architectures have become the cornerstone of modern deep learning, spanning vision, audio, and sequence modeling. They underpin the development of Large Language Models ( LLMs) that excel not only at compressing vast textual corpora but also at exhibiting emergent induction mechanisms. Beyond a certain scale of parameters, training data, and with appropri- ate fine-tuning, LLMs appear to exhibit remarkable reasoning capabilities. Additionally, language models embody a general principle: sequences, whether words, medical signals, or machine events, encode meaning through their temporal structure. This observation opens a new research frontier, extending the power of language modeling to machine- generated data in industrial settings. In such domains, event logs, sensor readings, med- ical test results, or diagnostic codes constitute a machine-generated âlanguageâ through which complex systems express their internal state. By interpreting this language, deep learning models can uncover hidden patterns of operation, anticipate failures before they occur, and even infer causal relationships within large-scale systems. The same archi- tectures that change fundamentally natural language processing now hold the potential to transform how industries understand, monitor, and reason about the behavior of complex machines. 1 1 Introduction 1.1 From Industrial Challenge to Scientific Question The automotive industry presents one of the most demanding real-world contexts for developing such intelligent systems. Vehicles must be safe, reliable, and explainable, yet their internal states are observed indirectly through asynchronous, heterogeneous, and noisy event logs. Traditional statistical and rule-based approaches struggle to capture the rich dependencies underlying these sequences; they remain applicable only when the event vocabulary is small and sequences are short. Additionally, diagnostic data exhibits all the hallmarks of complex event-driven domains: extremely high cardinality of event types, sparse temporal observations, and a high degree of redundancy. It is essential to develop techniques that not only anticipate such events but also explain their underlying causal mechanisms. These characteristics make automotive diagnostics an ideal testbed for advancing machine learning at scale, particularly in event-sequence modeling, multimodal learning, causal discovery, and multi-agent systems. What begins as a challenge in fault detection thus becomes a general scientific endeavor: building models that not only predict, but also explain and reason over sequences of events. 1.2 Research Areas, Challenges and Open Problems This thesis builds upon the convergence of three complementary research areas: Sequence ModelingModern language models have demonstrated remarkable ability to capture long-range dependencies and contextual relations in structured sequences. Be- yond their success in natural language, they establish a general and practical paradigm: training a model on a large-scale, self-supervised task such as next-token prediction, and then adapting it to downstream applications where labeled data are scarce. In contrast, machine-generated data in complex systems such as vehicles, aircraft, hospitals, or computer networks is inherently noisy, repetitive, and scattered across time. Vehicle diagnostic logs, for instance, comprise tens of thousands of distinct DTCs, each irregularly triggered by sensor anomalies, environmental fluctuations, or transient software behavior. Similar challenges arise in other domains: sequences of electrocar- diogram spikes or clinical events in healthcare, network intrusions and security alerts in cybersecurity, flight logs in aeronautics, and transactional traces in financeâall of which consist of heterogeneous event streams. These data are often non-stationary, dominated by repetitive or redundant events, and corrupted by noise or incomplete information, which makes it diïŹicult for classical methods to extract stable temporal dependencies or causal relations. Modeling such event sequences with modern sequence models, such as Transform- ers [ 151], offers a powerful alternative. Their self-attention mechanism enables the cap- ture of long-range dependencies, modeling of multiscale temporal dynamics, and inte- gration of heterogeneous modalities, such as sensor readings, timestamps, and textual metadata, through fusion mechanisms. Despite these advantages, the use of autoregres- sive and bidirectional Transformers for machine-generated event data, particularly in 2 1.2 Research Areas, Challenges and Open Problems automotive diagnostics, remains underexplored. In addition, the applicability of Trans- formers to prognosis and long-horizon forecasting in event-driven systems remains poorly studied. Causal Discovery in Event SequencesPredictive accuracy alone is insuïŹicient in safety-critical domains. To truly anticipate failures, prevent accidents, or mitigate risks, one must understandwhyevents occur and how they influence one another. This is the central goal of causal discovery: recovering, from observational data, the underlying mechanisms that generate the observed phenomena. In practice, this involves learning a compact Directed Acyclic Graph (DAG) that captures direct causes, mediating effects, and potential confounders influencing an outcome such as a vehicle defect or a disease. Once established, this DAG can be used to perform causal inference, i.e., to answer counterfactual questions such as âwhat ifâ or to measure the average causal effect of a variable on the outcome. In event-driven domains, causal discovery becomes particularly intricate. Unlike static cross-sectional data or regularly sampled time series, event sequences are asynchronous, irregularly spaced, and highly context-sensitive. In these settings, the causal effect of an event is not fixed; it may vary significantly depending on its precise timing, its position within a sequence, and its co-occurrence with other markers. In general, two main perspectives emerge. The first, events-to-outcome causal discovery (or multi-label causal discovery), seeks to identify which events or combinations thereof cause a higher-level outcome, such as a disease from symptoms, a security breach from system alerts, or an EP from DTCs. The second, event-to-event causal discovery, focuses on uncovering causal influences between events themselves, forming the basis for understanding asynchronous system behavior and performing root-cause analysis. Traditional causal discovery methods, such as constraint-based algorithms (PC [ 140]), score-based optimization (GES [21]), and functional causal models (LiNGAM [134]), are ill-suited to this setting. They typically assume low-dimensional variables. When applied to high-dimensional sequences with thousands of discrete event types, long tem- poral dependencies, and intense noise, these methods quickly become intractable due to their computational complexity and statistical assumptions. Even recent approaches for time series and event sequences (THP [ 180], SHTP [119], CAUSE [174], CASCADE [24]) are applied to dozens or, at most, a hundred variables on short sequences which is not applicable in many scenarios. In practice, they fail to scale, restricting their applica- bility in industrial contexts where datasets are both massive and heterogeneous. As a result, the field of causal discovery in large-scale event sequences remains largely under- explored. Practical adoption is hindered by three central challenges: (1) the combina- torial explosion of possible event interactions, (2) the lack of scalable, GPU-compatible algorithms, and (3) the absence of production-ready methods that reuse pre-existing architectures for inference, like Transformers or other autoregressive models. As a con- sequence, practitioners are seeking automated solutions to identify causal relationships in high-dimensional sequence datasets in a reasonable amount of time, leveraging existing pretrained sequence models. In the automotive domain, EPs are defined as deterministic 3 1 Introduction Boolean rules that encode causal relationships betweenDTCs. By reinterpreting these rules as a causal discovery problem, it becomes possible to identify potential causes as DTCs for unknown EPs and hence create new rules based on the DTCs as causes of an EP. Automated and Explainable Reasoning SystemsBeyond discovering causal relations, the ultimate goal in industrial AI is to enable autonomous reasoning over complex au- tomation tasks. Such a system can not only infer causal structures but also explain, justify, and act upon them to automate manual processes. While causal discovery pro- vides the skeleton of knowledge, it often remains abstract, numerical, or incomplete. Such causal graphs must be translated into interpretable reasoning chains that domain experts can trust and validate. This requires bridging two complementary worlds: data- driven inference from causal discovery algorithms and domain knowledge. Recent ad- vances in LLMs have opened new possibilities for reasoning, automation, and the in- tegration of external knowledge to form intelligent systems. These models can process both structured and unstructured information, including textual descriptions, metadata, documentation, and relational rules, either directly in the prompt (in-context learning) or via retrieval-augmented generation (RAG), making them ideal candidates for reason- ing over causal evidence. Yet, using LLMs alone remains insuïŹicient: without grounding in causal structure or domain constraints, they may hallucinate or reason inconsistently. Conversely, causal discovery methods alone provide semantic knowledge but lack the adaptability of LLMs. This gap motivates a hybrid paradigm that combines the structured and principled outputs of causal discovery algorithms with the natural-language reasoning capabilities of LLMs, often referred to as Causal Machine Learning ( CML). By organizing these components into multi-agent systems, each agent can specialize in one task. One in- ferring causal dependencies, another contextualizes them using metadata and textual descriptions. The result is a collaborative reasoning process where causal evidence is grounded, verified, and articulated in human-readable form. In the automotive indus- try, they could automatically generate interpretable Boolean rules that describe how diagnostic events combine to form EPs. These rules, which currently require extensive manual expert labor, could then be fully automated. More broadly, multi-agent systems powered by causal discovery could be extended to any domain where reducing LLMsâ hallucinations and increasing interpretability are essential, from medical diagnostics and industrial health monitoring to cybersecurity incident analysis. 1.3 Contributions The contributions of this thesis can be summarized as follows. 1.3.1 Predictive Sequence Modeling of Vehicle Diagnostics In the first contribution of this thesis, we explore how modern sequence models can anticipate system failures in vehicles before they occur [92]. 4 1.3 Contributions Time Event type DTC1DTC2DTC3 EP when what S = <s> DTC1 DTC2 DTC3 </s>P(EP|S) Figure 1.1:Error Pattern Prediction (when and what). Based on the past sequenceS of diagnostic trouble codes (DTCs), manufacturers want to prevent error patterns (EPs) from happening by predicting their likelihood and time of occurrence. Autoregressive Transformers for Vehicle Event SequencesWe draw an analogy be- tween natural language and vehicular diagnostic data: just as words that form structured sentences convey meaning, DTCs form spatio-temporal sequences encoding the vehicleâs state at each mileage and time. This analogy motivates the use of Transformer-based language models for automotive fault prediction. To this end, we introduce two autore- gressive Transformer architectures [ 151]: âąCarFormer, designed to model the generative process of DTC sequences by pre- dicting the following DTC and its time of occurrence. âąEPredictor, which extends this formulation to EPs by jointly predictingwhatEP will occur andwhen, given a history of observed DTCs. Together, these models learn to anticipate vehicle faults in an autoregressive manner, providing both event type and time of occurrence. Despite the complex data distribu- tion of a stochastic process such as the DTCs and EPs appearance, their high event- type cardinality (>10 4 ), irregular sampling, and unbalanced label distributions, they demonstrate robust performance across multiple metrics, outperforming the classical autoregressive Transformer model. In particular, they could anticipate a given EP ap- pearance with 80% F1-score within an interval of58.4±13.2h when predicting the time of occurrence. Predictive Maintenance Metric for Autoregressive ModelsWe introduce the Confi- dent Predictive Maintenance Window (CPMW), a novel evaluation protocol that mea- sures how early in an observed sequence of events (DTCs) a model can issue a confi- dent outcomes (EPs) prediction. Unlike accuracy-at-end-of-sequence metrics, CPMW 5 1 Introduction Time, Mileage Event type DTC1DTC2DTC3 50C1bar 2mV 4A 1mV32% 32s 60C 3bar Error Pattern(s): EPs DTC4 4V 80C Figure 1.2:Error Pattern Prediction using Multimodal Sequences. Past diagnostic trouble codes only provide limited information about complex and overlapping er- ror patterns. In addition, domain experts rely on environmental conditions (e.g., temperature, voltage, humidity, ...) to identify more accurately error patterns. rewards early, high-confidence predictions, directly reflecting the operational value of predictive maintenance. Offline Error Pattern Prediction using Multimodal LearningWhile modern diag- nostic systems primarily rely on sequences of vehicular DTCs, they overlook valuable contextual information such as raw sensory data (e.g., temperature, humidity, and pres- sure) known as environmental conditions. Experts often use such sensory information to separate overlapping EPs that might share the same DTCs in their Boolean rule definition. To this end, we introduceBiCarFormer, a bidirectional multimodal Trans- former that integrates environmental sensor data with DTC sequences. Through special fusion mechanisms, BiCarFormer learns interdependencies between discrete diagnostic events and environmental conditions, improving predictive performance compared to standalone bidirectional Transformers (+9%precision on average) and enabling more interpretable predictions through attention scores interpretation. 1.3.2 Multi-Label Causal Discovery in High-Dimensional Event Sequences In the second part of this thesis, we move from predicting events to understanding why they occur. While the introduced autoregressive models capture the temporal regular- ities of diagnostic sequences, they remain fundamentally correlational. We introduce two causal discovery framework for discrete sequences, each one targetingsample- and population-level, that is, discovering causal relationships on respectively one sequence or a multitude. 6 1.3 Contributions Sample-LevelTo address this challenge, we introduceOSCAR(One-ShotCausal AutoRegressive discovery). OSCAR identifies, for each outcome label, the minimal set of causal events (the so-called Markov Boundary) directly from a single observed sequence, as formalized in Table1.1. We refer to this regime as sample-level causal discovery throughout this thesis to emphasize the key property: the causal graph is inferred from and specific to one individual sequence, without requiring a population of observations 1 . OSCAR reuses the pretrained autoregressive Transformers from Part I as neural autore- gressive density estimators ( NADEs) to estimate the conditional mutual information (CMI) between events and labels given the past events. They quantify the extent to which each event contributes to a future outcome. This formulation enables eïŹicient, GPU-parallelized computation of causal dependencies. Theoretically grounded under standard assumptions, we show that OSCAR recovers the Markov Boundaries of each label within an associated Bayesian Network [ 70]. In practice, it also introduces causal indicators that distinguish excitatory and inhibitory effects between events, offering a more nuanced view than deterministic causal graphs [68]. We validate OSCAR on a large-scale vehicular dataset comprising over tens of thou- sands of DTCs and hundreds of Error Patterns (EPs). Classical constraint-based causal discovery methods fail to scale beyond a few hundred variables whereas OSCAR suc- cessfully identifies meaningful causal relations between DTCs and EPs, revealing inter- pretable causal structures for phenomena such as steering wheel degradation or battery power limitation (Fig. 1.3). In summary, the key contributions of OSCAR are as follows: âąSample-level Multi-Label Causal Discovery, recovering the Markov Bound- ary of each outcome label from a single observed sequence without requiring a population of observations. âąNADEsreuse, repurposing pretrained autoregressive Transformers as neural au- toregressive density estimators for conditional mutual information estimation, es- tablishing a formal bridge between Parts I and I of this thesis. âąCausal indicators, annotating each discovered dependency with a signed measure of excitatory or inhibitory influence, going beyond binary edge presence. âąExtreme-scale, handling 29,100 event types and 474 outcome labels simultane- ously a regime where classical constraint-based methods fail entirely. Population-levelWhile OSCAR performs causal discovery at the level of individual se- quences (sample-level), real-world systems demand a broader perspective: understanding global causal relationships across the entire observational dataset (population-level). To address this, we introduceCARGO(CausalAggregation viaRobustGraphOperations), a scalable framework that merges sample-level causal graphs into a unified global rep- resentation. CARGO treats causal discovery as a two-phase process. In the first phase, 1 This terminology supersedes the label âone-shotâ used in the associated conference publications [94, 91], where âone-shotâ was used informally to mean âfrom a single sequenceâ. 7 1 Introduction 0.37 ± 0.19 -0.50 ± 0.21 0.58 ± 0.25 -0.50 ± 0.17 0.68 ± 0.18 -0.30 ± 0.22 0.45 ± 0.24 -0.63 ± 0.22 0.37 ± 0.18 -0.46 ± 0.21 0.51 ± 0.24 0.56 ± 0.22 -0.26 ± 0.20 0.41 ± 0.23 -0.50 ± 0.23 0.33 ± 0.18 0.44 ± 0.18 -0.53 ± 0.21 0.59 ± 0.24 0.66 ± 0.20 -0.28 ± 0.21 0.42 ± 0.24 -0.52 ± 0.24 Power limitation Restricted path planning Software error Radar: Init failed Camera error Voltage error Odometer degradation Invalid msg signal Security sync failure Steering degraded Type 1 Steering degraded Type 2 Low washer fluid Figure 1.3:Anonymized Causal Graph.Example of a sequence of events (DTCs) that lead to a steering wheel degradation and a power limitation as outcomelabels. The causal indicators are shown invioletif inhibitory andorangeorredif excitatory, depending on the magnitude. The voltage error DTC is the biggest causal driver for the power limitation EP while the security sync failure seems to have the biggest inhibitory effect on all EPs. OSCAR extracts sample-level causal graphs. In the second phase, CARGO aggregates these causal graphs based on the frequencies of edge appearances across labels using a novel adaptive frequency-based fusion strategy. Motivated by long-tail distributions in labeled data, the edges are retained based on their frequency of appearance and the number of available samples per label. This aggregation process enables the recovery of global Markov Boundaries for each label while filtering out spurious relations caused by noise, sparsity, or rare events. Empirical evaluation demonstrates that CARGO not only scales where classical algorithms fail, but also increases the accuracy and recall compared to using OSCAR alone. The ablation studies confirm the robustness of its aggregation phase across multiple thresholding and scoring strategies. In summary: âąAdaptive frequency-based fusion, aggregating sample-level graphs into a global structure via a label-frequency-aware thresholding strategy motivated by the long- tail distribution of EP samples. âąLinear scalability, processing 300,000 sequences with 29,100 event types on mod- ern GPUs where quadratic constraint-based methods fail. âąGlobal Markov Boundary recovery, with theoretical guarantees under mild assumptions. 1.3.3 Scalable Sample-Level Event-to-Event Causal Discovery in Event Sequences Multi-label causal discovery methods are limited to local structure learning, ignoring the dense information of inter-event dependencies. To fully diagnose complex phenomena like cascading vehicle failures, we must recover the complete causal graph. This presents 8 1.3 Contributions 7.32 7.06 0.30 3.72 0.98 0.19 1.77 0.34 0.16 1.83 0.330.25 0.34 0.34 0.92 0.37 2.79 0.90 0.45 0.20 0.45 0.21 1.19 t=1t=2t=3t=4t=5t=6t=7t=8t=9t=10t=11t=12t=13t=14t=15t=16t=17t=18t=19t=20t=21t=22t=23t=24t=25t=26t=27t=28t=29t=30t=31t=32t=33 Tailgate Blocked Tailgate Lift Limited Augmented View Limited Camera Calib Error Camera Out- of-Range BT Audio Play Phonebook Access Phonebook Access Engine Intervention Variance Cruise Range Stat Ignition Off Prevented Window Emergency Mode Interior Light Map Implausible Speed Limit Msg Missing Foresight Motor Fault FAS Signal Error Baseboard Supply Limited FAS Signal Error Baseboard Supply Limited Battery Temp Fault Secondary Network Fault Remote Driving Limited Wiper Limited Radar List Overflow FAS Signal Error Radar Interference 48V Temp Derating LDW Signal Inconsist. TAP Events Stat LCW Events Stat Radar List Overflow Substitute Speed Used PWM Actuation Available Instance Time Causal Graph 7.32 7.06 0.30 3.72 0.98 0.19 1.77 0.34 0.16 0.45 1.19 1.83 0.33 0.34 0.34 0.92 0.37 2.79 0.90 0.20 0.45 0.21 Tailgate Blocked Tailgate Lift Limited Augmented View Limited Camera Calib Error Camera Out- of-Range BT Audio Play Phonebook Access Cruise Range Stat Ignition Off Prevented Window Emergency Mode Interior Light Map Implausible Speed Limit Msg Missing Foresight Motor Fault FAS Signal Error Radar List Overflow Substitute Speed Used Baseboard Supply Limited Battery Temp Fault Secondary Network Fault Remote Driving Limited Wiper Limited Radar Interference 48V Temp Derating LDW Signal Inconsist. TAP Events Stat LCW Events Stat PWM Actuation Available Engine Intervention Variance Instance Summary Causal Graph Figure 1.4:Anonymized Instance-Time Causal Graph for Diagnostic Defect Cas- cade.Temporal evolution of a diagnostic defect cascade in a vehicle (|X|â29,100). TRACE effectively captures causal relationships, revealing distincterror clusters at different time steps (e.g., initial sensor failures att= 3triggering mechanical faults att= 12, battery at issuet= 17). This enables actionable root-cause analy- sis by isolating the specific onset of a failure mechanism and its strength using the conditional mutual information. a significantly more challenging scenario due to the combinatorial explosion of possible edge connections, as opposed to simple events-to-outcome relationships. We therefore introduceTRACE(TemporalReconstruction viaAutoregressiveCausalEstimation), a framework that uses the density estimation capabilities of autoregressive models to recover the causal graph from a single sequence. By treating the event stream as a high-order Markov chain, TRACE employs an autoregressive model to perform scal- able conditional independence testing via its learned distributions. TRACE inherits OSCARâs linear scalability with the vocabulary size, extending it to event-to-event de- pendencies. Our results show that causal structure becomes identifiable well before the autoregressive model fully converges during the pretraining. In summary: âąSample-Level Event-to-Event Causal Discovery, we propose TRACE, to the best of our knowledge, the first framework designed to recover the summary causal graph from a single high-dimensional sequence by amortizing the learning of dy- namics via a pre-trained AR model. This addresses a substantially underexplored area in causal discovery. âąBackbone Agnosticism, the framework strictly decouplesdensity estimation (Phase 1) fromcausal discovery(Phase 2). Consequently, TRACE can leverage any state-of-the-art autoregressive backbone (e.g., Transformers, Mamba, RNNs). 9 1 Introduction Causal Scope Events-to-Outcome (DTCsâError Pattern) Event-to-Event (DTCsâDTCs) Sample-Level (Single Sequence) OSCAR One-Shot Multi-Label Causal Discovery Markov Boundary per label from a single sequence Chapter6 TRACE Scalable Amortized Causal Discovery Instance-time causal graph from a single sequence Chapter8 Population-Level (Full Dataset) CARGO Graph Aggregation for Causal Discovery Global Markov Boundaries via adaptive graph fusion Chapter7 Open Problem Population-level summary causal graph over event types/outcomes Requires aggregation of instance-time and summary graphs with acyclicity constraints Future work Table 1.1:The four regimes of causal discovery in discrete event sequences, defined by the cross-product ofcausal scope(single-sequence inference vs. population-level aggregation) anddependency type(events-to-outcome vs. event-to-event). The three shaded contributions of this thesis âOSCAR,TRACE, andCARGOâ share a common backbone: pretrained autoregressive Transformers repurposed as neural autoregressive density estimators (NADEs) for conditional mutual information es- timation. The remaining cell, population-level event-to-event discovery, constitutes an open problem and a direct avenue for future work. âąLinear Complexity in High Dimensions, TRACE scales linearly with the vo- cabulary size, bypassing the combinatorial explosion of constraint-based structural learning. âąIdentifiability under Generative Approximation, we theoretically establish that causal identifiability is achievable for any autoregressive model that suïŹi- ciently approximates the data-generating process. We derive error bounds for the identifiability, proving that the causal graph is recoverable up to a noise floor determined by the modelâs convergence. âąReal-World Applicability, we empirically validate TRACE on synthetic struc- tural causal models with challenging vocabulary, memory requirements, and apply it to vehicle diagnostic logs, demonstrating its practical utility in industrial set- tings. 10 1.3 Contributions 1.3.4 Automated Causal Reasoning with Multi-Agent Systems for Error Pattern Rule Automation To fully automate the construction of EPs, we introduceCAREPforCausalAutomated Reasoning forErrorPatterns, a multi-agent system for the automatic synthesis of EP rules in high-dimensional automotive event sequences. It comprises three agents: âąACausal Discovery Agentthat estimates candidate causes and causal indica- tors for each unknown EP using OSCAR and CARGO. âąAContextual Information Agentthat incorporates DTC descriptions, known EP rules, and metadata to simulate expert knowledge. âąAnOrchestrator Agentthat coordinates the two agents to synthesize five plau- sible rules for these EPs, alongside a natural language explanation. Through its agent-based design, CAREP translates causal evidence and contextual in- formation into symbolic rules, along with their explanations, so that engineers can use them directly for vehicle fault analysis. To evaluate CAREP, we masked a random subset of the known rules and tested the frameworkâs ability to identify them. Results show that CAREP consistently outperforms LLM-only baselines by a large margin when reconstructing the rule, going up to83%precision in recovering the correct elements in the rule, while offering interpretable explanations at scale. Finally, the reuse of the same pretrained model, first as a predictor in Part I, then as a density estimator in PartII, and finally as the causal backbone of CAREP in PartIII, forms the conceptual spine of the thesis and the unifying insight of this work. 1.3.5 Software In addition to the theoretical, methodological and architectural contributions presented in this dissertation, a significant effort has been dedicated to practical and scalable software. The core algorithmic frameworks have been formalized and open-sourced as a Python library namedseq2cause[ 88] 2 . Released under the permissive MIT License and publicly available via the Python Package Index (PyPI),seq2causeprovides a universal, plug-and-play pipeline designed to transform standard autoregressive sequence models such as Transformers into robust causal discovery frameworks. This software contribution not only ensures the repro- ducibility of the experiments detailed in the subsequent chapters but also equips the broader machine learning community with a practical, open-source implementation to uncover causal mechanisms in complex, event-driven systems. 2 https://pypi.org/project/seq2cause/ 11 1 Introduction 1.4 List of Publications Most parts of this thesis have been published in the academic literature and presented at international conferences. The following list gives an overview of the publications: Harnessing Event Sensory Data for Error Pattern Prediction in Vehicles: A Language Model Approach. [92] Hugo Math, Rainer Lienhart, Robin Schön, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 18, p. 19423â19431, 2025, Philadelphia, USA, February 2025. One-Shot Multi-Label Causal Discovery in High-Dimensional Event Sequences. [94] Hugo Math, Robin Schön, Rainer Lienhart, Conference on Neural Information Pro- cessing Systems (NeurIPS) Workshop 2025 on CauScien: Uncovering Causality in Sci- ence, San Diego, USA, December 2025. Towards Practical Multi-label Causal Discovery in High-Dimensional Event Sequences via One-Shot Graph Aggregation. [91] Hugo Math, Rainer Lienhart, Conference on Neural Information Processing Systems (NeurIPS) Workshop 2025 on Structured Probabilistic Inference & Generative Modeling, San Diego, USA, December 2025. Context-Informed Sequence Classification: A Multimodal Approach to Vehicle Di- agnostics. [89] Hugo Math, Rainer Lienhart, International Conference on Learning Representations (ICLR) Workshop on Time Series in the Age of Large Models, April 2026. Neuro-Symbolic Rule Discovery: Empowering LLMs with Causality for Vehicle Di- agnostics. [93] Hugo Math, Julian Lorenz and Rainer Lienhart, International Conference on Learn- ing Representations (ICLR) Workshop on Logical Reasoning of Large Language Models, April 2026. Scalable Sample-Level Causal Discovery in Event Sequences via Autoregressive Den- sity Estimation (under review). [90] Hugo Math, Rainer Lienhart, under review at the International Conference on Ma- chine Learning (ICML), Seoul, South Korea,Available on arXiv:2602.01135, July 2026. 12 1.5 Thesis Outline 1.5 Thesis Outline This thesis is organized into three main parts. PartI(Chapters2â4) establishes the modeling foundations. Chapter 2 provides background on event sequence modeling and Transformers. Chapters 3 and 4 present our predictive architectures for DTC sequences, evaluated on large-scale BMW data. PartII(Chapters 5â8) develops three causal dis- covery frameworks. Chapter 5 provides mathematical foundations. Chapters 6, 7, and 8 present OSCAR, CARGO, and TRACE respectively, with theoretical guarantees and scalability experiments. Part I (Chapters 9â10) moves from causal discovery to auto- mated reasoning and introduces CAREP, a multi-agent system. The thesis concludes in Chapter 10with a synthesis of the main findings and perspectives for future work. We summarize how predictive modeling, causal discovery, and automated reasoning can be unified into a coherent pipeline for scalable, interpretable diagnostics of vehicle error codes and patterns. Finally, we discuss open challenges and future directions. 13 Part I Predictive Sequence Modeling 15 2 Foundations of Event Sequence Modeling This chapter introduces the formal machinery used throughout the thesis: event sequence modeling, Transformer architectures, and evaluation metrics. We give a formal defini- tion of a sequence of events and common ways of representing them. Then, we highlight challenges of traditional statistical methods and provide a brief introduction to Trans- formers [151,26,122]. We introduce their primary components: positional embedding techniques, feed-forward layers, layer normalization, and the attention mechanism. At the end of this chapter, we investigate various metrics for evaluating language model predictions in classification and regression. 2.1 Hawkes Processes The Hawkes Process [53] has arguably been the most studied modeling technique for the occurrence of discrete events in a sequence, spanning financial and earthquake data [107, 74]. Its key property, the so-calledself-excitation, is that the occurrence of one event may trigger a series of similar events. For instance, earthquakes trigger aftershocks, stock-market transactions may trigger a chain reaction, or a neuron fires and triggers brain activity elsewhere. To do so, we model the number of event occurrences up to timetthrough a stochastic processN(t). In some extensions of the basic Hawkes process, a random variable called amark, which provides auxiliary information such as spatial coordinates or sensory mea- surements is attached to each event. In the automotive context, a vehicle generates a sequence of discrete events (DTCs), where each arrival is characterized by a timestamp tand a specific mileagemacting as the mark. Furthermore, we consider the multi- variate setting where multiple Hawkes processes aremutually exciting: an occurrence in one event stream increases the probability of future occurrences in other streams. For example, in a social network, a message may trigger further messages from different users. This cross-excitation is the fundamental mechanism explored in this work. However, scaling these models remains a challenge: when faced with thousands of distinct event types across numerous long sequences, how can we eïŹiciently learn a robust model and reason about the underlying interactions? In Part I, we address the statistical and computational challenges of learning robust models from event sequences. 2.1.1 Background Definition 1(Counting Process).A stochastic processN(t), also notedN t , defined for tâ„0, is a counting process starting atN(0) = 0ifN(t)only takes values in0,1,2,· 17 2 Foundations of Event Sequence Modeling and increases in jumps of size +1. The random jump timest 1 , t 2 ,·form a point process, with0< t 1 < t 2 <·andt 0 â0. The counting process can be defined in terms of the point process as: N(t) = â X i=1 It i â€t= X t i â€t 1 Thepoint processrepresentation enables us to formally define amarked event sequence (Def.3). Although event sequences can refer to multivariate time series as well [52], we abbreviate the term and refer solely toevent sequencesas a discrete sequence of events, such as: Definition 2(Event Sequence).Let each event be composed of a time of occurrence tâR + and an event typexâXforming a pair(x, t).Xis a finite set of discrete event types. A sequence of lengthL+ 1is constructed with multiple pairs of events, such as S=(x 0 , t 0 ), ...,(x L , t L )where0< t 1 < ... < t L witht 0 â0. Definition 3(Marked Event Sequence).A Marked Event Sequence is an event sequence (Def.2) where each eventx i is augmented with a markm i â M. The event is then defined as a triple(x i , t i , m i ). The inclusion of marks is mostly used in PartIand PartIIIwhen dealing with the mileage and other sensory or metadata information per eventx i . Definition 4(Conditional Intensity Function).The conditional intensity function of N(t)is defined, fortâ„0, by λ â âlim ââ0 E(N(t+ â)âN(t)|H t ) â if the limit exists. HereH t is the history ofN(t)such thatH t â(x i , t i )âXĂR + |t i < t. Remark 1.In the context of vehicle logs or manufacturing trajectories,Sis a realiza- tion of a discrete-event stochastic process. While we acknowledge the Hawkes process framework common in the literature for modeling instantaneous rates asλ â (t), our pri- mary interest lies in the conditional dependencies between the event typesxwithin the sequence. 2.1.2 Self-Exciting Property Hawkes [53] defined the counting process (named after him) by specifying its conditional intensity process: Definition 5(Hawkes Process).A Hawkes process is a counting processN(t)whose conditional intensity process fortâ„0is: λ â (t) =λ+ X t i <t ÎŒ(tât i ) whereλ >0is a background arrival rate andÎŒ:R + âR + is the excitation function. 18 2.1 Hawkes Processes The excitation functionÎŒcontrols how past event occurrence will affect the rate of future occurrences. A common and simple excitation function isÎŒ(t) =αexp(âÎČt) (exponentially decaying intensity). In this case,(N(t), λ â (t))is aMarkov process[70] such that the current intensityλ(t)does not depend on the entire history of events. 2.1.3 Mutually Exciting Hawkes Processes Definition 6(Mutually-Exciting Hawkes processes).ConsiderN(t) = (N 1 (t),·, N |X| (t)) as a collection of|X|counting processes withN k (t)âs occurrence times denoted ast k 1 , t k 2 etc. They are mutually-exciting Hawkes processes ifN k (t)âs conditional intensity follows: λ â k (t) =λ+ |X| X j=1 X t j i <t ÎŒ j,k (tât j i )fork= 1,·,|X| whereÎŒ j,k (s)â„0 Remark 2.Denote the integrals of theÎŒ j,k excitation functions asÏ j,k = R â 0 ÎŒ j,k (s)ds and theexcitation matrixasΊ = (Ï j,k )âR |X|Ă|X| . TheΊmatrix is conventionally treated as an approximation of the causal adjacency matrix of a graph with|X|nodes and is typically used to extract causal relationships [162]. The|X| 2 parameter count renders this approach computationally intractable in our high-dimensional regime. Part IIadopts an alternative that avoids this bottleneck. 2.1.4 Limitations Number of parameters.TheN j t process has either an excitatory effect onN k t (Ï j,k >0) or no effect onN k t (Ï j,k = 0)forj=korjÌž=k. Importantly, these processes haveO(|X| 2 ) parameters to fit, which is often intractable in practice for many applications [ 74]. For |X|= 10 3 the number of parameters is already1million. Moreover, the actual number of scalars being optimized in practice is more than|X| 2 , since there is the base intensity λ, the parameters of the excitation functionsÎŒ(decay rates) and interaction magnitude. The literature often assumes|X| 2 +|X|[ 74]. Optimization.To understand why the Hawkes process is insuïŹicient for our setting, we first show that the Maximum Likelihood Estimation (MLE) of its parameters is computationally intractable at scale. We therefore seek the parameters (base rateα, decayÎČ) that make the observed sequence of events most probable. The likelihood for any point process parametrized byΞwith observationst 1 ,·, t L within a time horizon 0â€tâ€Tcan be computed as the sum of the log-likelihood for each process: lnL(Ξ|t 1 ,·, t L , T) = |X| X k=1 L X i=1 lnλ â k (t i )â |X| X k=1 Z T 0 λ â k (t)dt(2.1) The full derivation can be found in Appendix A.1.1. Importantly, Eq.2.1cannot be computed easily since the sum over all event types|X|and the double summation over 19 2 Foundations of Event Sequence Modeling all pairs of eventt i andt j , leading toO(|X| 2 ·N(T) 2 )at worst andO(|X| 2 ·N(T))if using exponential decay [74]. This is a regime where classical MLE is computationally prohibitive and statistically prone to overfitting [9,78]. In contrast, the standard self- exciting Hawkes process with exponential decay leads toO(N(T))complexity [74] but implies a greater parametric assumption. 2.1.5 The High-Dimensional Discrete Event Stream Regime While the Mutually-Exciting Hawkes process provides a rigorous theoretical foundation for interacting events, it assumes that maintaining a continuous intensity functionλ â k (t) for every dimensionkâ1, . . . ,|X|is computationally feasible. However, in regimes of highly structured data with a large number of discrete events (patient trajectories [106], event logs [30], vehicle diagnostics [92], NLP [130]), representing each continuous in- tensity function orstreamseparately is ineïŹicient. For instance, in modern automotive diagnostics, the event space cardinality approaches |X| â 10 4 , while the actual event occurrences are sparse due to structural relationship (eventAalways excites the occur- rence ofBdespite the history). We assume such a high-dimensional 1 regime throughout the thesis. This shift in perspective is crucial. It aligns the problem with Neural Sequence Mod- eling[ 144] andLanguage Modeling[122], where the focus is on predicting the next token x i+1 from a high-dimensional discrete sequence of words, rather than regressing|X| parallel sequences. This justifies our use of autoregressive architectures (in Part I) to approximate the conditional densityP(x i+1 |H)and enables the scalable causal discov- ery techniques developed further in PartII. 2.2 Language Models Although we employ the term event sequence modeling, we define it in the Natural Lan- guage Processing ( NLP) realm and adopt its terminology (such as tokens, vocabulary). Therefore, we give an overview of the important concepts in NLP as used in this work. 2.2.1 Word Level Sentence and WordsBefore introducing the modeling framework, we formally define the notion of a sentence as a special case of a unit-time event sequence: Definition 7(Sentence as a Unit-Time Event Sequence).A sentenceSis a special case of an event sequence (Def.2) characterized byunit-time arrivals. Given a vocabulary 1 Throughout this thesis, high-dimensional refers mainly to the cardinality of the discrete event space |X|rather than the ambient dimension of a feature vector in the classical statistical sense. This usage is consistent with the event sequence and causal discovery literature [6,41], where dimensionality indexes vocabulary size and refers to the number of nodes in a graph. The regime studied here is characterized simultaneously by large|X|, long sequence lengths, and large sample counts, all of which contribute to the described regime. 20 2.2 Language Models (the event type spaceX), a sentenceSis a sequence of events defined as: S= (x i , t i ) L i=0 wheret i =i,âiâ0, . . . , L(2.2) where events are so-called âwordsâ. TokenizationTo translate words into machine-interpretable data, a step calledtok- enizationconverts a sequence of words intotokens. These tokens can be seen as word fragments. Consequently, a word is often split into smaller units (sub-word tokenization) to optimize the compression of a natural language [26] into a fixed-sizevocabulary. In event sequence modeling, it is common to treat events as whole tokens, since we do not know their underlying semantic structure. Language PreprocessingTo obtain all distinct tokens, there is often a preprocessing stage where we iterate over the entire observational datasetD=S 0 ,·, S m of se- quences and sort the different tokens by their frequency, i.e., we obtain a vocabulary with the most common words. The vocabulary is referred to asX. The words that do not appear in the vocabulary are replaced by the unknown token, often defined as: [U N K]. A start-of-sequence token< S >or[CLS]in BERT [26] is used asx 0 , and an end-of-sequence token< /S >or[EOS]forx L the last word. 2.2.2 Word Embeddings To get a dense representation of our sequence, we embed each token into ad-dimensional space using a learnable embedding matrixW e âR VĂd whereV=|X|is the number of distinct tokens (vocabulary). To perform this projection in parallel, we create a sequence of one-hot encoded vectors from the tokensx i L i=0 asYâR LĂV . The word embedding E is obtained via a linear projection: E=YW e âR LĂd (2.3) wheredâȘV. Typically,W e is learned from scratch, especially adapted for event sequence modeling, where there is no pretrained word embedding. 2.2.3 Transformers Throughout the thesis, we will primarily employ Transformer-based architectures. This architecture underpins the vast majority of modern deep learning, whether in vision [82], time series [4], or language [147]. It relies on several key concepts such as the attention mechanism, feed-forward layers, layer normalization [ 7], position embeddings, and self-supervised learning. We provide a brief overview of these elements. 21 2 Foundations of Event Sequence Modeling 2.2.3.1 Attention Scaled Dot-Product AttentionThe majority of research utilizing Transformer models for sequence data employs the architecture introduced by [151]. We define three linear projection matricesQ=UW Q âR LĂd ,K=UW K âR LĂd , andV=UW V âR LĂd which are called respectivelyquery,key, andvalue.W Q ,W K ,W V are trainable weights. Scaled dot-product attention computes a similarity between queries and keys to weight the importance of each tokenâs value, allowing the model to focus on relevant parts of the sequence when making predictions. It is computed as: A=Attention(Q,K,V) =softmax QK T â d k V(2.4) whereAâR LĂd . The softmax is applied row-wise. In the original paper, [151] argue that scaling theQKdot product enables a better stability of the softmax function since the gradients become extremely small. Therefore thescalingfactord k is introduced in Eq.2.4. ForcausalorautoregressiveTransformers, a lower triangular matrix Ì Aâ 0,ââ LĂL is applied additively to the pre-softmax logits of Eq.2.4, setting future positions toââto prevent tokens from attending to future tokens [122]. Multi-Head AttentionIn the same paper, [151] extended the single scaled dot-product into amulti-head attention(MHA). They divided the attention scores computation into multiple heads to perform multiple attentions in parallel. In this case, the feature di- mensiondis divided intohparallel attention heads of size d h âN. It is usually done in practice by reshaping the query, key matrices into respectivelyQâR LĂ d h Ăh and KâR LĂ d h Ăh before feeding them to the attention scores calculation: MHA(Q,K,V) = [head 1 ,·,head h ]·W O (2.5) with: head i =Attention(Q i ,K i ,V i )(2.6) whereQ i âR LĂ d h ,K i âR LĂ d h ,V i âR LĂ d h andW O âR hĂd V Ăd is a learnable projection matrix. The authors show that each head learns different token-to-token relationships from a single sequence. 2.2.3.2 Feed-forward Layers The multi-head-attention scores are processed using a combination of layer normal- ization, position-wise feed-forward neural network (FFN), and residual connections to ensure a proper gradient flow and to learn the attention scores patterns between tokens: U âČ =LayerNorm(A+U) H=LayerNorm(U âČ +FFN(U âČ )) (2.7) 22 2.2 Language Models whereHâR LĂd is the resulting hidden state. The so-called position-wise FFN layers are typically two layers that project thed-dimensional input to an intermediate size, often4Ăd. These FFNs are applied independently for each position: FFN(U) =f(UW 1 +b 1 )W 2 +b 2 (2.8) wherefan activation function (ReLU [2] in the original paper),W 2 a learnable weight matrix andb 2 a bias vector. Then, residual connections [54] are applied in Eq.2.7to improve gradient flow and training stability. This layer is stackedktimes, leading to a Transformer withklayers. 2.2.3.3 Normalization The normalization layer helps stabilize the dynamics of the deep neural networkâs hidden states [7] by rescaling each output neuron before the activation function. This mitigates the problem ofinternal covariance shift[65], which arises from the fact that the weights of one layer are highly dependent on the outputs of the neurons in the previous layer. Modern Transformers such as LLaMA [147] use root-mean-square (RMS) normaliza- tion [169] where each activation is divided by the RMS of the layerâs inputs, omitting the mean-centering step of classical layer normalization. 2.2.3.4 Positional Encoding DeterministicDue to the nature of the attention scores, the architecture does not have a sense of ordering, as the input is just an array of vectors. Therefore, [151] added a so-calledpositional encodingto the word embedding vector for each sequential step. This vector comprises a fixed periodic function, such as: PE i,j := ( sin(iĂÏ j/d 0 )ifjmod 2 = 0 cos(iĂÏ (jâ1)/d 0 )ifjmod 2 = 1 (2.9) whereiis the index of theiâth token andÏ 0 the initial frequency (usually10 â4 ). Then, it is added to the word embedding to form the input embedding: U=PE+E(2.10) LearnableIn most PyTorch [111] and Hugging Face [158] implementations of Trans- formers (e.g., BERT [26], GPT [122]), a learnable positional embedding is obtained via a learnable lookup tableW PE âR LĂd for each sequential steps0,1,2,·, L. It is added to the word embeddings in Eq. 2.10 2.2.3.5 Autoregressive Transformers Autoregressive or causal Transformers have a causal attention mask applied in Eq.2.4. They perform the next token prediction via the chain-rule factorization of the joint 23 2 Foundations of Event Sequence Modeling distributionP(X 0 ,·, X L )[15], such as the probability of observing a sequence of event typess= (x 0 ,·, x L )is: P(s) = L Y i=0 p(x i |x 0 , x 1 , . . . , x iâ1 )(2.11) wherep(x)is the probability mass function, i.e.,P(X=x). These models are trained using next-token prediction [122], by minimizing the negative log-likelihood of observing sequences. 2.3 Metrics We measure the prediction of language models using common classification metrics that are interpretable and anchored in the machine learning domain. We also define our own metric later in Chapter3for predictive maintenance. 2.3.1 Next-Event Prediction Next-event prediction is often evaluated using the Accuracy of the inferred next event [173,180,78]. However, in the presence of class imbalance, Accuracy alone can be misleading [172]. Therefore, we include Precision and Recall to provide a general view of model performance. Accuracyis defined as the proportion of correct predictions among the total number of cases examined: ACC= T P+T N T P+T N+F P+F N (2.12) Precision(or positive predictive value) quantifies the reliability of the positive predic- tions, measuring the proportion of inferred events that are correctly identified: P= T P T P+F P (2.13) Recall(or sensitivity) measures the completeness of the discovery, capturing the pro- portion of actual positive events that were recovered by the model: R= T P T P+F N (2.14) 2.3.2 Next-Time Prediction Mean Absolute ErrorMAE is the average of the absolute difference between the actual t i and predicted values Ë t i : MAE= 1 m m X i=0 |t i â Ë t i |(2.15) 24 2.4 Dataset For instance, if one finds an MAE of5h, it indicates that the model has, on average, an error of 5 hours when predicting the time of occurrence. Eq.2.15weighs all errors equally, making it less sensitive to outliers. Root Mean Square Erroror RMSE, adds a square function inside the sum, averages across all points, and takes the square root: RMSE= v u u t 1 m m X i=0 (t i â Ë t i ) 2 (2.16) Unlike MAE, RMSE is more sensitive to outliers as it disproportionately penalizes larger errors. Nevertheless, RMSE remains less interpretable as the scaling of values changes. 2.3.2.1 Confident Predictive Maintenance Window Letζ:0, . . . , L âRbe a performance metric evaluated as a function of the number of observed events (e.g., precision or mean absolute error). Given a confidence threshold ΞâR, we define theonset of confident predictionx Ξ as the earliest observation index at whichζexceeds (or falls below, for error metrics)Ξand remains so on average: x Ξ = min iâ0, . . . , L|ζ(i)â·â Ξ ,(2.17) whereâ·ââ â„,â€depending on whetherζis a quality score (e.g., F1,â·â=â„) or an error metric (e.g., MAE,â·â=â€). LetÎŒ seq denote the average sequence length across the evaluation set. Definition 8(Confident Predictive Maintenance Window).TheConfident Predictive Maintenance Window(CPMW) is the interval[x Ξ , ÎŒ seq ]within which the model produces confident predictions, as determined by thresholdΞ. To quantify the overall quality of a model within the CPMW, we define theCPMW Area Under the Curve(CPMW ζ ) as the normalised integral ofζover this interval: CPMW ζ = 1 ÎŒ seq âx Ξ Z ÎŒ seq x Ξ ζ(x) dx,(2.18) whereζis assumed to be continuous and integrable on[x Ξ , ÎŒ seq ]. In practice,ζis estimated at discrete observation steps and Eq. (2.18) is approximated by the trapezoidal rule. Importantly,A higherCPMW F1 indicates reliable early classification; a lower CPMW MAE indicates accurate early time-of-occurrence estimation. 2.4 Dataset This thesis uses proprietary BMW vehicle datasets. Due to confidentiality constraints, we will briefly describe the dataset in terms of its nature, dimensions, and scale, but no specific numerical values, vehicle identifiers, or customer information can be disclosed. AllmentionedGitHubrepositoriescontainsyntheticoranonymizeddataonly;noBMW dataormappingsareincluded. 25 2 Foundations of Event Sequence Modeling NotationDescription xDiscrete event type, in our case a DTC. dAbsolute mileage of the vehicle in km. mMileage of the vehicle in km since the first DTC (x 0 ) occurred in a se- quenceSsuch asm i =d i âd 0 tsUnix timestamp attached to each DTC. tNumber of hours passed since the first DTC occurred in a sequence. More specifically,t i is defined as t i =ts i âts 0 . SSequence of triplets (event type, time, mileage) defined asS= (x i , t i , m i ) L i=0 of lengthLwith in- dex starting from 0. S l Multi-labeled sequence of triplets (event type, time, mileage) lead- ing to an outcome defined asS l = (x i , t i , m i ),y L i=0 of lengthLwith index starting from 0 andyâ 0,1 |Y| . S (i) Thei-th sequence in a datasetD. S (i) l Thei-th multi-labeled sequence in a datasetD. sSequence of lengthLcontain- ing only the event types, e.g., (x 0 , x 1 ,·, x L ). iâ0, ..., Lindex of element in (event) se- quence. Table 2.1:List of dataset symbols and their respective meanings for the datasets used in this thesis. 2.4.1 Vehicular Fault Sequences A vehicular fault sequence is commonly referred to as an event sequence, where each event includes an error code and a timestamp. If specified, the vehicleâs mileage at the time the error code was triggered can be included asm i , forming a marked event sequence (Def. 3). To obtain a full marked event sequenceS=(x i , t i , m i ) L i=0 , we obtain the last known timestampts L and mileaged L and select all DTCs that are no further than: 26 2.4 Dataset (1) a given period in the past (ts L âts i â€30days) and (2) a given distance in the past (d L âd i â€300km). In addition to that, the dataset is joined with error patterns (EPs) per sequence. In most of our experiments we have labeled sequences, where the labels are the EPs. LetD=S (1) l ,·, S (m) l be a dataset of multi-labeled sequences S (k) l = ((x 0 , t 0 ),·,(x L , t L ),y)whereyâ0,1 |Y| is a multi-one hot vector (multiple EPs can happen per sequence). For each chapter, we will recall the used dataset and its characteristics, such as the number of distinct DTCs|X|and EPs|Y|, the number of sequences, and their average lengths. An overview of the notations can be found in Table2.1. 2.4.1.1 Diagnostic Trouble Code: DTCs We construct aDiagnostic Trouble Code(DTC) indicating the precise error from 3 pieces of information arriving at the same timestamp and mileage: (1) the ID number of the ECU, (2) an error code (Base-DTC), and (3) aFault-Byte. A single DTC token comprises these three elements: DT C=ECU||Base-DTC||Fault-Byte(2.19) where||denotes a concatenation. 2.4.1.2 Error Patterns Error patterns (EPs) are a higher-level abstraction of DTC sequences to characterize specific vehicle faults. They are defined as Boolean combinations over DTCs. When denotingerror pattern, we refer to it as their intrinsic name (e.g., a precise battery defect, a PCB fault), while when denotingerror pattern rules, we refer directly to the Boolean combinations that define this error pattern. An example of an EP rule for(y 1 ) based on some diagnosis trouble codes(x i )is shown in Eq. 2.20such as: y 1 =x 1 &x 5 &x 8 & (x 12 |x 3 ) & !x 10 & !x 20 (2.20) A tangible EP example for a battery failure could be: (voltage drop&temp high)|(charging error& !battery replaced)(2.21) 2.4.1.3 Experimental Datasets used Across Chapters We used the following dataset settings across chapters 2 : âą(Chapter 3): 1.7 million sequences with|X|= 8,710DTCs and|Y|= 254EPs from one BMW model range. âą(Chapter 4): 5 million sequences with|X|= 22,137DTCs and|Y|= 360EPs from different BMW model ranges. 2 We will recall them in their respective experiment sections. 27 2 Foundations of Event Sequence Modeling âą(Chapters 6â9):300,000sequences with|X|= 29,100DTCs and|Y|= 474EPs from different BMW model ranges. 28 3 Autoregressive Architectures for Error Pattern Prediction Anticipating bothwhaterror pattern will occur andwhenit will appear is an important step toward predictive maintenance and improved vehicle safety. In this chapter, we present an approach that draws an analogy between natural language processing and modeling multivariate vehicle event streams. Similar to words in a sentence, DTCs unfold as sequences over time, enriched with contextual information such as mileage and time of occurrence. To capture these dependencies, we introduce two autoregres- sive Transformer-based architectures:CarFormer, an autoregressive Transformer trained with a novel self-supervised strategy, andEPredictor, a decoder that jointly predicts the type and timing of future EPs. This chapter is based on the following publication: Harnessing Event Sensory Data for Error Pattern Prediction in Vehicles: A Language Model Approach.[ 92] Hugo Math, Rainer Lienhart, Robin Schön. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 18, p. 19423â19431, 2025, Philadelphia, USA, February 2025. 3.1 Introduction A central challenge in vehicle diagnostics lies in predicting bothwhatkind of error will manifest andwhenit is likely to occur. Unlike individual DTCs, which may be noisy or repetitive (e.g., due to recurring electrical glitches or software updates), EPs characterize entire fault sequences, such as engine or battery failures. Fig. 1.1illustrates this setting, where the task is to infer, given a past sequence of DTCs, the probability distribution over future EPs and their expected time of occurrence. Recent research has begun to address predictive maintenance using DTC sequences, for instance, with recurrent neural networks (RNNs) [ 50] or Transformer-based architec- tures [49] that aim to predict the next DTC. However, focusing solely on the next DTC is insuïŹicient for two reasons. First, minor or noisy DTCs can obscure critical patterns, making it diïŹicult to distinguish between trivial and critical faults. Second, as the event vocabulary grows often into the tens of thousands, the accuracy of next-token prediction deteriorates rapidly, much like in natural language modeling with large vocabularies and long sequences [ 8]. Historically, as seen in the foundation, Hawkes Processes and their neural variants have advanced the state of the art in event modeling for next event and time predic- tion tasks [ 53,31,132]. Transformer-based models like BERT [26] and GPT-3 [18] have 29 3 Autoregressive Architectures for Error Pattern Prediction gained overwhelming popularity due to their attention-based architecture, flexibility, parallelization, and state-of-the-art performance in sequence modeling. Consequently, models adapted to discrete-time sequences using Transformers have emerged naturally [173,180,137], achieving state-of-the-art performance in next-event prediction bench- marks. Inspired by the analogy between natural language processing and vehicle event streams, we adopt this perspective for DTCs and EPs: DTCs play the role of words, while EPs correspond to a sentence label (e.g., sequence classification for sentiment analysis). Formally, we represent a DTC sequence as a sentence of the form â< s > DT C 1 DT C 2 . . . DT C n < /s >â We make several modifications to the vanilla Transformer from [151], incorporating continuous-time and mileage positional embeddings as additional context. Using two distinct training phases, we introduce CarFormer, a pretrained model acting as an en- coder and EPredictor, a decoder Transformer-based model that generates a probability distribution over a set of error patterns for each event stepito determinewhatEPs will most likely happen and estimates a time forwhenit will occur. 3.2 Background To model asynchronous discrete events in a sequence, we typically reuse the vanilla Transformer [ 151] and create two embeddings. Time EmbeddingTime embedding replaces the traditional positional encoding, which grants the Transformer model positional information of each token within the sequence. Instead of using the token positioniin Eq. 2.9, the continuous event time occurrencet i is used. This provides an asynchronous representation of the event positions in a sequence. Event-Type EmbeddingTo get a dense representation of our sequence, we embed each event into ad-dimensional space using an embedding matrixL VĂd whereV=|X|is the number of distinct events. As it is done for word embeddings in Eq. 2.3, we create a sequence of one-hot encoded vectors from the event typesx i L i=0 asYâR LĂV . The event-type embeddingEis defined asE=YLâR LĂd and the input embeddingUas U=E+PâR LĂd . 3.3 Dataset Description OverviewIn this chapter, we use an anonymized vehicular DTC sequence dataset of 1.7Ă10 6 sequences with, on average, 150 DTCs per sequence. Each sequence belongs to a unique vehicle and the dataset is focused on a specific BMW model range. An overview of the DTC elements 1 can be found in Table3.1. 1 DTC composition defined in2.4.1.1and notation in2.1 30 3.4 Methodology Data # of valuesDescription DTC8710 Diagnostic Trouble Code ECU61Electronic Control Unit Base-DTC 7726Error Code Fault-Byte2Binary Value Table 3.1:Number of Distinct values of the DTC elements 3.3.1 Time and Mileage We plot the distribution oft i andm i in Fig.3.1. We observe peaks at zero on both distributions due to truncation and missing values. To feed our model with the timetfeature, we need a scaling method to mitigate the left-tail skew. Usinglog(t+ 1)is a natural choice. At the same time, we want to approximately maptinto the range of[â1,1]. Therefore, we apply the following non-linear functionf t :R + âRtot: t âČ =f t (t) = log(t+ 1)â1âtâR + (3.1) InTPPs the time is usually represented as inter-event time or its logarithm[132,31], making our approach anchored in the literature. 3.4 Methodology We introduce the following two models:CarFormerandEPredictorfor autoregressive DTC and EP prediction. Figure 3.1:Time and Mileage Distribution. Distribution oft i andm i (relative time and mileage) in our dataset. 31 3 Autoregressive Architectures for Error Pattern Prediction 3.4.1 CarFormer The CarFormer model can be seen in Fig.3.2. The following defines the main architec- tural ideas. 3.4.1.1 Embeddings CarFormer uses four different embeddings to capture the spatial and temporal depen- dencies of irregular event appearance. These embeddings differ from the positional embeddingPused in TPPs [180], [175], and next-DTC prediction studies [49], [50]. We embed both timetand mileagemand use a rotation matrix to induce absolute and relative event positions such as: âąEvent-type embeddingEâR LĂd is obtained like described in section3.2. âąAbsolute time embeddingTâR LĂd is constructed on-the-fly at each forward pass by a linear transformationt i,j =t âČ i w j +b j wherew j , b j are learnable parameters andt âČ i is the scaled time at event stepi. âąMileage embeddingMâR LĂd is obtained via a learnable lookup tableW m max Ăd wherem max = 300km. Each roww m âR d corresponds to the learnable embed- ding vector for the discrete mileage m . The continuous mileage m i â R + is cast to an integer valuem=âm i â. We attempted to leverage the continuous mileage by appropriately linearly interpolating the embeddingsw m andw m+1 , but the time complexity increased without observing any benefits. âąRotary Position Embedding (RoPE)R d Î Due to the permutation invariance of the Transformer model and the scattered timet, we still need to integrate positional event information. To do so,Q,Kare rotated using the orthogonal matrixR d Î from [ 143] as a function of the absolute event positioniin the sequence S. This method has two advantages: (1) it is not learnable (less likely to overfit), and (2) it integrates natively the relative position instead of alteringAwith a learnable bias like in [131]. Unlike some papers suggesting using only a time embedding plus an event embedding [ 180], [173], we argue that, like in [137], encoding solely the position with a time em- bedding leads to ineïŹicient sequential learning for our Transformer model and leads to an early plateau in our pretraining. We make the distinction between our event-type embeddingEand the other information per event (time and mileage), which we call context embeddingCE=T+M. Continuous Time Mileage Aware AttentionWe modify the vanilla Transformer from [151] by (1) adding the context embedding to the projected event-type embedding at every layer [147], and then (2) a Rotary Position Embedding (RoPE) [143] is applied to both queryQand keyK: Q=R d Î (W Q E+CE), 32 3.4 Methodology K=R d Î (W K E+CE) where 2 Î =Ξ i =Ξ â2(iâ1)/d 0 , iâ[1,2, . . . , d/2], Ξ 0 = 10 4 . More specifically, the inner product between queryq m and keyk n takes the event-type embeddinge m ,e n where m â n is their relative position with context CE m and CE n : q T m k n = (R d Î,m (W q e m +CE m )) T R d Î,n (W k e n +CE n ) =e T m W q R d Î,nâm W k e n +e T m W q R d Î,nâm CE n + CE T m R d Î,nâm W k e n +CE T m R d Î,nâm CE n =(1): query-to-key+(2): query-to-ce (3): ce-to-key+(4): ce-to-ce (3.2) whereR d Î,nâm = (R d Î,m ) T R d Î,n is a sparse orthogonal matrix. The terms provide (1) standard attention capturing similarity between token embeddings, (2) query attending to the spatiotemporal information of the other token (vice versa for (3)) and (4) the interaction of the spatiotemporal information only of both token only. Consequently, they provide a richer query and key representations when computing the attention scores. The final the attention scores is computed as: A=softmax (R d Î (W Q E+CE))(R d Î (W K E+CE)) T â 3d (3.3) AddingCEafter the projection to query and key can be seen as a refinement ofQ,K byT,M, providing additional context to the attention scores. We also add a scaling factor to compensate for the additional terms 3 in Eq.3.2. 3.4.1.2 Multi-Task Learning Next-Event Prediction.We use a standard language modeling objective which aims to minimize the cross-entropy loss between our output distributionËx i generated by our modelâsNext Event Headand the next eventx i+1 . [137] used a BERT [26] model trained on a masked event modeling task, which is commonly used for bidirectional models. However, they simultaneously applied a causal mask, resulting in a loss of the bidirectional property. We argue that, by doing so, we lose substantial sample eïŹiciency; thus, we will retain a standard next-token prediction.Ëx i is the probability distribution produced by theNext Event Head, which integrates an RMS normalization [ 169] and one linear layer. The cross-entropy loss betweenËx i andx i+1 (= a one-hot vector in0,1 V ) is obtained by: L c :=â L X i=0 V X j=0 x i+1,j log(Ëx i,j )(1âÎŽ i,r )(3.4) 2 Note the reciprocal relationship between the standard PE frequencyÏ 0 and the RoPE baseΞ 0 3 Verified empirically in the ablation experiments (Table3.3) via therotcross-key-value-scaled-ce-2 model. 33 3 Autoregressive Architectures for Error Pattern Prediction , Time Event-type Embedding ... Masked Multi Head Attention RMS Norm Value Query Key RoPE RMS Norm Context Embedding Feed Forward RMS Norm Next Event HeadNext Time HeadRandom Event Head H(S) N x Figure 3.2:CarFormer Architecture. A Transformer architecture is used alongside spatio- temporal embeddings (context embedding) injected to the query and key. A RoPE is then applied to induce the relative position of the DTCs. Three predictive heads are used to learn the complex dynamics of the DTC sequences. whereÎŽ i,r is the Kronecker delta, iff event typex i belongs to the set of random event typesRand0otherwise. Next Event Time Prediction.In addition, we compute the Huber loss [66] between the estimated inter-event timeâ Ë t âČ i for the eventx i and the ground truthât âČ i =f t (t i+1 )â f t (t i )to deal with outliers and prevent exploding gradients withÎČ= 1, Δ i = â Ë t âČ i âât âČ i . L t := L X i=0 (1âÎŽ i,r ) ( 0.5Δ 2 i if|Δ i |< ÎČ, (|Δ i |â0.5)otherwise, (3.5) t âČ is obtained using a log, hence we are essentially computing a kind of Mean Squared Logarithmic Error (MSLE) but with aÎČ, useful to stabilize training and help conver- gence. Random Event Prediction.Finally, a binary classifier that predicts whether an event was true or randomly generated is added. This choice is motivated by several papers 34 3.4 Methodology [137,38] stating that a model should learn when an event does not happen to reinforce the negative evidence of no observable events within each inter-event. At each stepi, a random event is injected with probabilityp. If a random event is successfully injected, the process continues until a failure occurs (i.e., the event is not injected). This allows for multiple random events to be injected in a row, allowing for more complexity. The L r loss is defined as the binary cross-entropy loss between the probability distribution Ëy r i generated by ourRandom Event Headand the ground truthy r i at event stepi: L r :=â |R| X i=0 y r i log(Ëy r i ) + (1ây r i ) log(1âËy r i )(3.6) Total Loss.The total loss is defined as follows: L= 1 Lâ|R| (L c +k 1 L t ) +k 2 1 |R| L r (3.7) wherek 1 , k 2 are trade-off between the different loss terms,Lthe sequence length,Rthe set of random events injected inS. 3.4.2 EPredictor We now describe the architecture of EPredictor in detail (Fig.3.3). We motivate this problem formulation and leverage the autoregressive property of CarFormer to introduce EPredictor. 3.4.2.1 Limitations of Next-Event Prediction Predicting only the next DTC in a sequence of DTC faults has inherent limitations and remains a diïŹicult task. For instance, in [ 49], they used DTCs, whose ECU, Base-DTC, and Fault-Byte data have a cardinality of 83, 419, and 64, respectively, and only report a81%top-5 accuracy for next DTC prediction. This is because DTCs are not always correlated or causally linked. Instead, we are also using repair and warranty data to predict more important events, such as EPs (error patterns). Repair and warranty data differ from DTCs since they are defined after observing all DTCs and characterizing a whole sequenceSand not an individual eventx i . 3.4.2.2 Multi-Label Event Prediction We define that occurrence of error patternyat indexi=Lin a sequenceS. Note that multiple EPs can occur at the same time. We can now define a supervised multi- label classification learning problem of predicting EPs. Multi-label classification has attracted growing attention in the context of event prediction. [ 175] uses an LSTM for fault detection. More recently, [136] treats concurrent event predictions as a multi-label classification problem and models such data using a Transformer architecture. To define the multi-label event prediction task withNlabels (âĄEPs), we reuse each S=(x i , t i , m i ) L i=0 and attach a binary vectoryâ[0,1] N to indicate the EPs occurring 35 3 Autoregressive Architectures for Error Pattern Prediction Time P(EP|S') Masked Multi Head Attention RMS Norm Key Masked Multi Head Attention Add & RMS Norm Value Query Output Event- type Embeding Value Query Output Context Embedding RoPE Add & RMS Norm Output H(S) CarFormer Key Feed Forward Next EP HeadNext EP Time Head S' P(EP|SâČ) SâČ Minimum context c ... ... Feed Forward RoPE Figure 3.3:EPredictor Architecture. The hidden stateH(S)is extracted from CarFormer and fed to the second attention layer as value and key. The event-type embedding Eis fed into the first layer after being normalized, and the context embedding CEcarrying the spatio-temporal information is added directly after the RoPE application to the keys and queries of the second layer. at timet L . Critically,yis invariant per sequenceS, meaning for all events withinSthe ground truthywill be the same. By applying a causal mask to both CarFormer and EPredictor, we enable predictive maintenance, since tokens can only attend to previous tokens, as shown in Figure 3.3. With EPredictor, we leverage theseq2seqnature of Transformers, where CarFormer outputs a sequenceH(S)of tokens encoded in a high-dimensional spaced, positioning tokens with similar characteristics nearby. Then, this hidden representationH(S)is fed into EPredictor, which acts as an autoregressive multi-label classifier for EPs. We approach EP prediction as a machine translation task. By utilizing the contextualized hidden statesH(S)from CarFormer as key and value (i.e., throughcross-attention), this effectively transitions our model from a âseq2seqâ to a âdtc2epâ framework. 36 3.5 Experiments 3.4.2.3 Imbalanced Labels In our case, and in most real-world problems, EPs are highly imbalanced across the dataset, which makes the event prediction task considerably more challenging, especially when the least frequent classes are the most important to detect [175]. In natural language processing, traditional upsampling methods involve perturbingSby shuffling and replacing tokens. However, in event data, we cannot afford to lose spatial and temporal information. Therefore, we inject random events(x i , m i , t i )with the same probability ofp= 0.05as in Section 3.4.1.2. We upsample the different EP classes to a minimum ofΞ 1 = 6000and downsample the most popular ones to a maximum of Ξ 2 = 12000. We also drop classes below 100 appearances across the dataset. 3.4.2.4 Losses We define a minimum contextc= 30which acts as a âminimum historyâ of DTCs to retain. TheNext EP Headoutputs a vector of probabilitiesËy i =sigmoid(MLP c (H i ))for each historyH 1 , ..., H i ,iâc, ..., LwhereH i âR d is the generated hidden represen- tation from EPredictor at stepi. For the regression task we forecast the time until the EP(s) occurrenceâ Ë t âČ i = MLP t (H i ) where the ground truth isât âČ i =f t (t L ,30)âf t (t i ,30). Formally, we define our binary cross-entropy loss over theNpossible EPs for one step ias follows: L ep i :=â 1 N N X j=1 y j log(Ëy i,j ) + (1ây j ) log(1âËy i,j )(3.8) The total loss acrossSwithLevents and contextcis: L ep := 1 Lâc+ 1 L X i=c L ep i (3.9) We use the Huber loss [ 66] and defineΔ i = â Ë t âČ i âât âČ i , withÎČ= 1thus: L t := 1 Lâc L X i=c ( 0.5Δ 2 i if|Δ i |< ÎČ, |Δ i |â0.5otherwise, (3.10) Our final loss to minimize is then:L=L ep +ÎłL t 3.5 Experiments We implemented CarFormer and EPredictor models using PyTorch, the code is publicly available 4 . 4 https://github.com/Mathugo/AAAI2025-CarFormer-EPredictor 37 3 Autoregressive Architectures for Error Pattern Prediction 3.5.1 CarFormer Pre-training We trained the CarFormer model for 70,000 steps with a learning rate of5Ă10 â4 , scheduled using a cosine warm restart with 10,000 warm-up steps, and a weight decay of 0.1. The loss coeïŹicients were set toα= 1andÎČ= 1. The model architecture included 12 attention heads, 6 layers, and a feature size of 600, utilizing the GELU activation function in all feed-forward layers. We employed the AdamW optimizer with a batch size of 192 and a sequence length of 258, resulting in approximately 34 million parameters. It is important to note that on all experiments we fixed the same number of parameters when introducing news embeddings for all evaluated model to provide a fair comparison, same for EPredictor. The training data was split into85%training and 15%testing without up-sampling, and random events were injected with a probability ofp= 0.05per sample. On average, each training session lasted about 20 hours on an Nvidia A10G GPU. 3.5.1.1 Ablation I: CarFormer Embeddings Multiple CarFormer models with different embedding choices were evaluated on next- token prediction accuracy (ACC) and on the regression task using mean absolute percent- age error (MAPE) and root mean square error (RMSE) to determine the best-performing CarFormer model. The different choices wererot(RoPE),time(absolute time embed- ding added to the inputU),mileage(also added toU),m2c, andc2mare additional dot-products ([ 55]). In our case, we are more interested in the next-event prediction task, thus we are willing to sacrifice some MAPE % over the ACC (%). ModelACC(%) MAPE(%) RMSE rot-ce22.643.20.04770 time21.482.90.04762 time-mileage21.383.00.04785 time-c2m-m2c 21.583.50.04794 time-m2c21.523.60.04823 GPT19.89-- Table 3.2:Overall prediction performance of CarFormer with different embeddings. Best results are in bold. Using onlytimegave the best MAPE (2.9) but not the best ACC (21.48), suggesting that other features might improve the model predictions. For the mileage integration, our intuition was that doing an early summation of the two embeddings (T,M) seemed to denature the inputU(ACC oftime-mileage<ACC oftime), but the mileage of the vehicle could help differentiate between different DTCs. We tried to modify the attention dot products, which seemed to help the ACC a bit (time-c2m-m2c,time-m2c), but increased the MAPE drastically. So we fused it withCEdirectly inQ,K. Then, by applying a RoPE to the transformed input, the ACC increased while preserving the RMSE, leading to the best performing model in terms of ACC, namely:rot-ce. 38 3.5 Experiments ModelF1 MAPE MAE CPMW F1 âCPMW MAE â rotcross-query-key-ce-1-282.69 32.45 0.026852.800.888 rotcross-query-key-ce-282.6931.180.025456.610.882 rotcross-query-ce-282.6931.180.025453.000.884 rotcross-key-value-ce-284.3833.47 0.026365.060.904 rotcross-key-value-scaled-ce-2 84.3831.440.0252 67.630.874 rotnocross-ce-1-280.74 37.61 0.027542.950.927 cross-speed83.53 34.73 0.026049.280.877 cross-mixffn83.41 33.37 0.027054.390.891 time-cross-query83.34 35.89 0.027545.710.896 Table 3.3:EPredictor evaluation results with different model architectures on the test set (no up- nor down-sampling). The MAPE and F1 are shown in (%) 3.5.2 EPredictor Experiments We evaluate multi-labeled classification using the micro-F1 score [172]. To better un- derstand and enhance our modelâs predictive maintenance capabilities, we introduce the concept ofConfident Predictive Maintenance Window (CPMW, Def.8), which repre- sents the interval within which our model can make reliable predictive maintenance predictions. We quantify this with theCPMW Area Under Curve(CPMW ζ ). The F1 score, MAE, and MAPE have been calculated on average for all observations in Table 3.3, and additionally for each historyH i t in Fig.3.4to understand how each model per- forms with different numbers of observations. To monitor the predictive maintenance capability of each model, the CPMW F1 and CPMW MAE were computed. 3.5.3 Ablation I: EPredictor Architecture & CPMW Embedding Architecture and Cross-attentionWe explored several architectural changes and their impact on the CPMW: we applied a RoPE (rot), a cross-attention withquery orkeyorvalueto the second multi-head attention block (cross), added the context em- beddingCE(ce) to layer1and/or2, applied a scaling factor of â 3dtoQ,Kas shown in Eq.2.4(scale), injected a relative matrixS rel into the attention scores (speed), and applied a mixed feed-forward network (mixffn) [160] to the mileage embedding. The timemodel refers toT, which is also added toE. Finally, we trained a model with and without theRandom Event Head. ResultsOur experiments revealed several key insights: By applying acrossatten- tion, we can see improvement in all metrics (rotnocross-ce-1-2), which is consistent with the machine translation analogyâdtc2errorpatternâ. The best cross-attention results were obtained whenH(S)was used as thekey-value. Adding the mileage via an MLP layer (cross-mixffn) seemed to help the MAPE (-2.5%), the CPMW F1 (+9)and also the CPMW MAE (â0.005)compared totime-cross-querymodel, suggesting that mileage is beneficial for both tasks. This is consistent since EPs are also dependent on the traveled distances between DTCs and the different stationary behavior of the vehicle. 39 3 Autoregressive Architectures for Error Pattern Prediction Furthermore, models incorporating a RoPE (rot) performed significantly better in both regression and classification tasks, as in the pre-training, highlighting the performance of RoPE in machine translation tasks [143]. AddingCEto the last layer (ce-2) yielded the best results, as opposed to adding it to both layers (ce-1-2). Surprisingly, doing feature engineering onT,Mwith aspeedmatrixS rel did not help the metrics, which could indicate some missing modalities (e.g., mileage) during training, thusCEis more adapted for real-world scenarios. We monitored the need for ourRandom Event Head and noticed+1.2%in the F1 Score and+10.1in the CPMW f1 . 20406080100120140160180200 Number of observations 55 60 65 70 75 80 85 90 F1 micro (%) Confident Predictive Maintenance Window rotcross-key-value-ce-2 rotcross-key-value-scaled-ce-2 rotcross-query-key-ce-2 time-cross-query cross-mixffn cross-speed rotnocross-ce-1-2 x = 81, Confident predictions x = 158, Average Sequence length Figure 3.4:Classification Performance Comparison Between Architectural Changes. F1 Score comparison with multiple EPredictor architectures as a function of the number of observations. This answerswhatEP is most likely to occur. When taking the best performing model (rotcross-key-value-scaled-ce-2), the model en- tered the CPMW after 81 observations, i.e.,half of the sequence. Within this window, the model obtained an error ofâ65±14hwhen estimating the time of EP occurrence (Figure 3.5), highlighting the modelâs predictive capability within the CPMW. Other- wise, the average absolute error across all observations was approximately58.4±13.2h. By experimenting with these modifications, we aimed to identify the optimal architecture for predictive maintenance. The findings reveal that cross-attention, context embedding in the second layer, and scaled attention significantly improve performance within the CPMW. 3.6 Summary This chapter presented a novel approach to predictive maintenance in vehicles by mod- eling diagnostic event sequences as a language. By introducing two Transformer-based architectures,CarFormerandEPredictor, we demonstrated that it is possible to jointly 40 3.7 Outlook 406080100120140160180200 Number Of Observations 0 50 100 150 200 Mean Absolute Error (h) Confident Predictive Maintenance Window Average Std Deviation x = 81, Confident predictions x = 158, Average Sequence length Figure 3.5:Regression Error As a Function of the Number of Observations. Evolution of the MAE as a function of the number of observations for the best performing model, this effectively tests forwhenan EP is most likely to occur. forecast bothwhenandwhaterror patterns are likely to occur, even in the presence of highly unbalanced, irregular, and high-cardinality diagnostic data. Our results show that autoregressive modeling of DTC sequences provides a powerful means to capture the tem- poral and contextual dependencies underlying vehicle faults. The proposed framework effectively reconciles traditional event sequence modeling and large-scale fault prediction, moving toward a unified, data-driven foundation for predictive maintenance. Beyond its experimental performance, the proposed system has clear practical implications. In principle, once deployed,EPredictorcan operate in real-world scenarios by continuously analyzing DTC streams and estimating the probability and timing of forthcoming error patterns. When a suïŹiciently confident prediction is made, the vehicle can proactively notify the user of a potential critical fault and recommend a visit to a service center. Such a system has the potential to enhance vehicle safety, reduce maintenance costs, and ultimately support the broader vision of intelligent and self-diagnostic automotive systems. The pretrained CarFormer and EPredictor backbones are directly applicable for a broad range of downstream tasks, including sequence clustering, anomaly detection, and causal discovery, as demonstrated in the subsequent parts of this thesis. 3.7 Outlook Several limitations remain. First, predictive performance is sensitive to the quality and convergence of CarFormerâs pretraining; a poorly trained backbone degrades EPredictor accordingly. Second, the CPMW metric shows that reliable predictions require observing approximately half the sequence, which may be too late for certain use cases. Third, the highly imbalanced distribution of error patterns means that rare but critical EPs remain harder to detect, despite the upsampling strategy employed. These constraints motivate the following chapter, which incorporates additional modalities to improve the classification of overlapping and rare error patterns. 41 4 Multimodal Sequence Modeling for Error Pattern Classification Chapter3demonstrated that DTC sequences alone provide a strong predictive signal, but highlighted a key failure mode: error patterns sharing identical Boolean DTC rules cannot be disambiguated from the event stream alone. Chapter4addresses this limita- tion directly, by introducing environmental sensor data as a complementary modality. Naturally, contextual information such as raw sensory data (e.g., temperature, humidity, and pressure) is beneficial for engineers to classify sequences of error codes into vehicle failures. Yet, it introduces unique challenges due to its complexity and the noisy nature of real-world data. This chapter is based on the following publication: Context-Informed Sequence Classification: A Multimodal Approach to Ve- hicle Diagnostics.[ 89] Hugo Math, Rainer Lienhart, International Conference on Learning Representations (ICLR) Workshop on Time Series in the Age of Large Models, April 2026. This chapter introduces BiCarFormer: a multimodal approach to multi-label sequence classification of error codes into EPs that integrates DTC sequences and environmental conditions. BiCarFormer is a bidirectional Transformer model tailored for vehicle event sequences, employing embedding fusions and a co-attention mechanism to capture the relationships between diagnostic codes and environmental data. 4.1 Introduction Diagnostic experts routinely find it necessary to incorporate contextual information as complex EPs seem to overlap in their rule definition (Eq.2.20). Prior work relies only on DTCs to infer the next DTCs (using Transformers [ 151,49] and RNNs [50]). While en- vironmental data can enhance the classification or clustering of EPs, it poses integration challenges due to its volume, variability, and ultimately high dimensionality. Modern time series foundation models (TSFMs) such as Chronos [4] operate on one modality and on a continuous event sequence. Vehicle data is intrinsically multimodal, consist- ing of continuous signals (voltage, RPM) interleaved with discrete event codes (DTCs), a modality mix that purely continuous TSFMs still struggle to integrate. Our goal is to uncover the appearance of DTCs and environmental conditions that correlate with certain EPs. We demonstrate that our approach significantly improves the multi-label sequence classification of EPs, outperforming traditional models that rely solely on DTCs [ 49,50] and classical Transformers such as BERT [26]. We further explain classification 43 4 Multimodal Sequence Modeling for Error Pattern Classification predictions by interpreting cross-attention scores. We show that BiCarFormer learns to detect fluctuations of quantized continuous values, despite encoding different units and thousands of different environmental conditions. 4.2 Related Work 4.2.1 Failure Detection in Vehicles Historically, machine learning models have been employed to detect machine failures. It includes techniques based on correlation [176], Bayesian networks [73], decision tree models [139], and more recently neural state-space models [56] to predict machine fail- ures or remaining useful life (RUL). These methods rely on historical data to identify patterns and predict potential failures. Bayesian networks can model the probabilistic dependencies among various components of the vehicle. Some papers explore the use of Deep Neural Networks (DNNs). For example, [179] uses a combination of convolutional, fully connected, and Transformer modules within one architecture for the classification of machine event logs. In Chapter 3, we suggested using the warranty data of vehicles to create a supervised learning problem where we attach EPs to each DTC sequence. We further model these multivariate vehicle event sequences and focus on classifying error patterns to benchmark the proposed methods. 4.2.2 Multimodal Fusion & Learning Introducing other modalities in a Transformer model has been extensively studied [163, 100]. One common and simple way to fuse two token embeddingsX A âR L a Ăd and X B âR L b Ăd from modality A and B (assumed to be sequences) is viaearly summation. Specifically,token-wise weighted summingof multiple embeddings at the input level is defined as: U=αX a âÎČX b (4.1) In MedBERT [123], the authors summed three types of embeddings (diagnostic codes, the order, and the position of each visit) to form inputU. This method has several advantages: it is simple to integrate into a Transformer and does not significantly alter the computation. However, the different tokens need to be aligned or projected to a latent space ifL a Ìž=L b , and it is not clear how one should weigh the different embeddings when summing usingα, ÎČ. For example, BERT uses an early summation of its token embeddingEand its positionPEsuch thatU=E+PE. In the diagnostic domain, DTC-TranGRU [ 49] uses three separate DTC components embeddings that they concatenate alongdto form a global DTC embeddingDâR LĂd (we refer to this method asearly concatenation). U=concat(X a ,X b )âR LĂ(d b +d a ) (4.2) 44 4.2 Related Work This enables finer-grained integration of additional features by selecting their embedding sizes,d a andd b . It also preserves the distinct characteristics of each feature by main- taining separate dimensions, thus potentially learning more nuanced representations and interactions in the deeper attention layers. However, it usually requires more computa- tion since we concatenate alongd, resulting in more parameters [163] for the point-wise feed-forward layers and the projection matrices. It also does not work well with low cardinality features such as token type ids from BERT [26] because it will only span across a very small size in the hidden dimensiond, thus early summation is preferable for this case. By combining embeddings at an early stage, models can learn a unified representation that captures the interactions between different features. This can be particularly useful if the features are highly correlated. These methods are used in a wide variety of domains, in particular in medicine [123,133] with electronic health records (EHR). When dealing with complex modalities such as audio or images that are not necessarily aligned, the attention mechanism of Transformers [151] is usually used for fusion. Late fusion in Transformers usually involves encoding the different modalities through independent Transformer encoders to extract high-level representationsZ a ,Z b : Z a =Tf a (X a )âR L a Ăd Z b =Tf b (X b )âR L b Ăd Then, these representations can be concatenated and encoded via another third Trans- former (Hierarchical Attention) to output a fused representation using multi-head at- tention: Z=Tf(concat(Z a ,Z b ))âR L a Ă(d a +d b ) One can also fuseZ a ,Z b with similarity products like in CLIP [121]. The CLIP model introduces a new self-supervised multimodal learning task where the model learns which caption goes with which image and demonstrates SOTA performance for 30 different existing computer vision datasets. The problem with this method is its limited cross- modal interaction, where we first encode separately the modalities to fuse them later. Perhaps the model would benefit from learning multimodal dependencies earlier in the architecture. At the other end, middle fusion (ormid fusion) methods involvecross-attentionand more generallyco-attentionmechanisms. In ViLBERT [ 82] and LXMBERT [145], the attention models enable computation of the attention scoresAas a function of the image and text input. For example, usingX a as queryQ a andX b asK b ,V b , we can compute the resultingC aââb âR L a Ăd and vice versa (one input attends to another). In co-attentionboth inputs attend to each other by computing two attentions simultane- ously (C aââb ,C bââa ), which enables multimodal learning in both ways. One caveat to these methods is that by not projectingX a ,X b into a fixed attention scores matrix but computing a newAâR L a ĂL b , ifL b â«L a , we drastically increase the computational overhead due to the quadratic time complexityO(L 2 )of thevanillaattention. We also get two hidden representations for each modality:H a ,H b . 45 4 Multimodal Sequence Modeling for Error Pattern Classification The productQK T carries most of the computation, thus one may consider selecting specific query and key multiplications, resulting in a sparse attention variant like Long- Former [14], or BigBird [168]. These models use a sliding, global, and random attention combination, reducingO(L 2 )to a linear complexity ofO(wĂL)for LongFormer. An- other method is to project the keys and values into a lower-dimensional space, like in Linformer [153], or use a multimodal bottleneck Transformer [100] to compute the cross- attention partially in a restricted latent space. Feature fusion in event sequence models has received comparatively little attention in the literature. The majority of papers aim at fusing the timet i information per event typex i by performing anearly summation [179,173,180] to integrate the time component, sometimes omitting the positional em- bedding for the time embedding [180], sometimes summing both [137,78], or not taking it into account [49]. 4.3 Dataset Extension with Environmental Conditions We extend the Chapter3dataset with 5 million sequences, containing an average of Lâ150±90DTCs per sequence from different BMW model ranges. Each sequence S raw belongs to a unique vehicle. InS raw , each DTC occurrence is attached additionally with a small sequenceS (i) e =(d j , v j , c j ) L ( i ) e j=0 of environmental conditions, respectively with elements description, value, and unit. These elements construct a single event(x i , t i , m i , S (i) e ). To obtain a complete marked event sequenceS raw =(x i , t i , m i , S (i) e ) L i=0 (Def.3), we apply the same 30-day, 300-km observation window as in Chapter 2. We then splitS raw into two distinct sequences: S=(x i , t i , m i ) L i=0 for the DTCs, andS e =(d i , v i , c i ) L e i=0 for only the environmental conditions with lengthL e â2275±2310, making it highly unaligned withSand orders of magnitude longer. An overview of the elements is provided in Table4.1as well as the number of distinct values for each feature in Table 4.2. 4.3.1 Integration Incorporating sensory information eïŹiciently, such as temperature or pressure value alongside discrete codes, remains an open problem [ 179,49]. Intuitively, it might seem Notations Description S e Sequence of environmental conditions triplets defined as S e =(d i , v i , c i ) L e i=0 of lengthL e â«L dEnvironmental condition description, e.g., temperature increase, vehicle speed, pressure increase vEnvironmental condition value can be int, float, string, null cEnvironmental condition unit, e.g., A, V, bar, °C, sec Table 4.1:New symbols introduced in this chapter. For base DTC notation, see Table2.1 46 4.3 Dataset Extension with Environmental Conditions trivial to rely on temperature for diagnosing a specific defect in a vehicle engine or a voltage measurement in a battery to identify a cell failure. Experts often analyze this data to make decisions about EPs in vehicles. However, the integration of environmental conditions comes with three significant challenges: 1.Dimensionality:descriptions have a high cardinality (>10 3 )and values can be multi-types: strings, Booleans, integers, floats, or NaN. 2.Variability:differs across each DTC: e.g., we cannot be certain that we will observe âtemperatureâ for DTC1 and âpressureâ for DTC2. 3.Volume:there are multiple environmental conditions per DTC, which can be re- dundant, noisy, and duplicated across the sequenceS e . TheDimensionalityproblem increases model complexity and often necessitates exten- sive data engineering. TheVariabilitymakes integration into a machine learning model challenging, and lastly, theVolumerequires substantial data infrastructure to manage environmental conditions, since it is often several times the size of DTC data and sig- nificantly increases the sequence length, a known challenge in Transformers [78]. We partially address these challenges by: âąremovingnullvalues, units, or descriptions. âąremoving duplicates per DTC: the same(d, v, c)triplet occurring multiple times within the sameS (i) e is removed, retaining only one instance. This significantly re- duced redundancy and noise in the data, though we still observedL e â2275±2310 triplets perS e . One caveat of this approach is thenonalignmentof environmental conditions and DTC when multiple DTCs occur at the samet i . In such cases, we drop subsequent environmental conditions after the first occurrence. âąselecting the top-18most popular units. To ingest the resultingS e into a Transformer, we map our continuous valuesvâR + into discretetokens[ 163] with a limited vocabulary. Therefore, we search for the optimal bins for each unitc. A variation of the Greenwald-Khanna algorithm [45] up to a maximum of4000tokens per unituto limit a potential18ĂΞ= 72000values. In practice, they overlap and do not reach the maximum (Table 4.2). Data# of valuesDescription DTC22,137Diagnostic Trouble Code ECU132Electronic Control Unit Base-DTC17,044Error Code Fault-Byte2Binary Value E. Condition Description2,559(See Table 1:d) E. Condition Value3,288(See Table 1:v) E. Condition Unit18Most popular units:u Table 4.2:Number of distinct values after filtering for each feature. 47 4 Multimodal Sequence Modeling for Error Pattern Classification 4.4 BiCarFormer 4.4.1 DTC Embeddings We use a Bidirectional Transformer model [26] that we train with mask language mod- eling. Each DTC element is embedded in a specific feature space, namely:ECU (D ecu âR LĂd ecu ),Base-DTC(D base âR LĂd base ), and theFault-Byte(D f âR LĂd ) using separate lookup tables. The first two are concatenated along the feature dimen- sion, like in [ 49], and D f is added to the result like a token type id in BERT [ 26]. This reduces the vocabulary size and trainable parameters compared to [92] and ensures that the embeddings with high cardinality are preserved independently, thus capturing more relationships between theECU, theBase-DTC, and theFault-byte. We obtain the input DTC embeddingDâR LĂd : D=concat(D base ,D ecu ) +D f During the pretraining phase, we mask only the Base-DTC token, since it provides the primary information. 4.4.2 Positional Embeddings Understanding the positioning of failure events in both the temporal and spatial (mileage) dimensions is beneficial for predictive maintenance. Failure events in our dataset exhibit spatial and temporal patterns (Figure4.1), which could indicate stationary behavior of the vehicle or recurring temporal failures [ 92]. Prior work, such as [179], employs adap- tive binning to discretize time in long event sequences. However, given our moderate sequence length (258 DTCs with 30-day and 300-km intervals), binning is unnecessary. Additionally, lookup table embeddings, often used in discrete event modeling, would be computationally expensive, introduce unnecessary parameters, and fail to preserve meaningful distance relationships inR + . We adopt a continuous time embedding [ 173] which is the deterministic positional en- coding introduced in Chapter 2(Eq.2.9) where we replace the token position with the continuous timet i . We do it similarly to create a mileage embeddingMâR LĂd/2 . Both embeddings,T,M, are concatenated along the feature dimension to form a unified positional representation. This preserves their independent contributions and prevents interference when fused withD. Our final fused inputUâR LĂd is given by: U=concat(D base ,D ecu ) +D f +concat(T,M)(4.3) 4.4.3 Environmental Embeddings To better capture relationships between the different environmental conditions elements (d, v, u), we create three distinct learnable embeddings: (1) the description of the envi- ronmental conditionsD e âR L e Ăd d (2) the discretized valueV e âR L e Ăd v (3) its unit U e âR L e Ăd . We use a mix of early summation and concatenation to fuse these em- beddings at the input level and obtain the total environmental conditions embedding 48 4.4 BiCarFormer 0100200300400500600700 Time Since First Event (h) 050100150200250300 Mileage of the Vehicle Since First Event (km) Event Occurrence Figure 4.1:Spatio-temporal Point Process Representation of a Vehicle.Spatial (mileage) and Temporal (time) point process representation of events from a vehi- cle. Bold vertical lines indicate multiple events occurring at the same timet i or mileagem i . EâR L e Ăd : E=concat(V e ,D e ) +U e To fuse the environmental conditions, we cannot simply align them with their respective DTC to perform an early token-wise summation or concatenation withD. Given that we utilize real-world data (noisy, redundant) and perform rigorous data filtering, there are too many missing environmental conditions for each DTC. Consequently, we chose to create a separate sequenceS e that is much longer thanS, where we will concatenate all environmental conditions and fuse them into one embeddingE. This straightforward method is flexible, andEcan be employed in a middle fusion manner inside the attention mechanism [ 163]. 4.4.4 Co-attention for Vehicle Event Sequences The overall architecture of BiCarFormer is shown in Figure4.2. Our architecture is directly inspired by the co-attention mechanism of ViLBERT [82], which enables mul- timodal learning by computing attention scores conditioned on each modality. Two multi-head attention layers are processingSandS e , resulting in two attention scores computation conditioned on:D(DTCs) andE(environmental conditions) fused embed- dings. We apply a RoPE (Rotary Position Encoding) [ 143] like in the previous chapter, on two sets of queries and keys (one for each cross-attention) to induce the absolute and relative position of tokens (Figure4.2). More specifically, the fused input embedding vectorsu m âR d ande n âR d from token positionsm, nare projected through weightsW q ,W k ,W v . The query and key are given by applying two different RoPE [143]: q m =e imΞ u W q u m (4.4) k n =e inΞ e W k e n (4.5) 49 4 Multimodal Sequence Modeling for Error Pattern Classification whereΞ u =diag(Ξ 1 , . . . , Ξ d/2 )withΞ i =Ξ â2i/d 0,u ,Ξ 0,u = 5000. Same forΞ e but with Ξ 0,e = 80000due toL e â«L. We use an alignment function [17]f:Râ[0,1]to produce the attention weights between two tokens: a(u m ,e n , m, n) =f(q T m k n ) =a dtcâenv =f u T m W T q e i(nΞ e âmΞ u ) W k e n (4.6) We also compute Eq.4.6fora envâdtc =a(e m ,u n , n, m)with separate weights forq m ,k n . Finally, two cross-attended context vectors are produced: c dtcâenv (m) = L e X n=1 a dtcâenv (m, n)v e,n âR d (4.7) c envâdtc (n) = L X m=1 a envâdtc (n, m)v u,m âR d (4.8) where valuesv u ,v e are obtained fromV e =W e v E,V u =W u v D. We tried two differ- ent alignment functionsf[17] for Eq.4.6. The widely usedsoftmaxand1.5-entmax Time, Mileage ECU ... Multi Head Attention RMS Norm VdQd RoPE Add & RMS Norm Feed Forward Add & RMS Norm Base Code Head Diagnosis Trouble Codes Event Type N x Time Base Code Fault-Byte Mileage °C Temp Increase ... Environmental Conditions 25 Temp Increase °C 65 °C Pressure Decrease 75 Temp Increase Bar 3 Operating Voltage V 12 °C 90 OperatingVoltage V 12 Temp Increase Multi Head Attention RMS Norm Qe Kd Add & RMS Norm Add & RMS Norm Feed Forward RoPE ValueDescription Unit Value HeadDescription Head Ke Ve He(Se)Hd(S) Mask Figure 4.2:BiCarFormer Architecture. Both parallel Transformers are computing cross- attention scores conditioned on each modalityQ,K,V. Two final representations are generated for each modality: DTC (H d )and environmental conditions (H e ). Multiple hierarchical embeddings are defined at the input level to account for DTC- specific features (ECUs, Fault-Bytes, Base-Code). 50 4.5 Experiments [116]. The standardsoftmaxoutputs dense attention scores, which might take too many environmental conditions into account. Since our input data is noisy and redundant, we would like to extract salient information; a sparse alignment function seems like an intuitive choice. However, in practice, we did not find a benefit using 1.5-entmax in the pretraining and classification where we observed a 30% slowdown in iterations per second. Consequently, we retained thesoftmax. As a consequence, two hidden states H d âR LĂd andH e âR L e Ăd (Figure4.2) for each modality are computed after fully connected layers, residual connections and root-mean-squared normalizations [169,147]. 4.4.5 Multimodal Learning To reinforce the relationship between DTC events and environmental conditions, we not only mask the DTC but also the environmental conditions(d, v)and let the unit unmask to reduce training complexity. This enables multimodal learning by reconstructingS e usingSand vice versa. As a result, BiCarFormer learns to benefit from this extra modality, which can be confirmed by inspecting the cross entropy loss of theBase-DTC classification:L dtc where using both modalities helps to reduce the pretraining loss (Appendix.A.2). However, due to multitasking, the losses are less stable and may require gradient clipping and a smaller learning rate to stabilize training. BiCarFormer is trained with three cross-entropy losses balanced by static coeïŹicients:α= 0.5, ÎČ= 0.3, Îł= 0.2: L=αL dtc +ÎČL value +ÎłL description (4.9) 4.5 Experiments 4.5.1 Settings We evaluated our model against established Transformer architectures, including BERT [26] (without the Next Sentence Prediction task) and DTC-TranGRU [49]. Since DTC- TranGRU is autoregressive [49], we trained it using next-token prediction and three sep- arate heads, each classifying one of the three DTC components. To assess each modelâs performance, we focused on multi-label classification of error patterns in the downstream task. For BERT and BiCarFormer we masked 15% of the tokens during the pretraining. The different models comprised approximately 25 million parameters, with the same hidden sized= 600, ensuring a fair comparison. For our downstream task of multi- label sequence classification, we used either the [CLS] token to perform classification or in the case of the BiCarFormer, the[CLS] dtc and[CLS] env after simply concatenating them along the feature dimensiondand fusing with a smallM LP. We used the [EOS] token for DTC-TranGRU rather than the [CLS]. Each classifier has 3,461,160 trainable parameters and consists of a small MLP with layer normalization, residual connections, and a sigmoid activation. We froze all backbones during classification. The dataset was partitioned into training, validation, and testing sets, adhering to a ratio of 75%, 15%, and 15%, respectively, with about 360 labeled error patterns. For the pretraining, we used a learning rate of10 â4 with a cosine scheduler and warm-up of 2000 steps for all 51 4 Multimodal Sequence Modeling for Error Pattern Classification models. We used the AdamW optimizer [80] withÎČ 1 set to 0.9 andÎČ 2 to 0.999. We applied a weight decay of 0.1 and a percentile gradient clipping of 5. Finally, we trained on an NVIDIA A10G GPU using FP16 with a batch size of 32 for the pretraining and 192 for the classifiers. 4.5.2 Metrics The evaluations were conducted across 360 different error patterns. To measure clas- sification performance, we employed multiple metrics, namely AUROC (Area Under the Receiver Operating Characteristic), Precision, Recall, and F1 Score [109]. The last three were computed using a confidence threshold of 0.8. The metrics were aggregated using different averaging: micro, macro, and sample. Micro computes metrics broadly and favors frequent classes. It provides an overall picture of model performance, but it does not reflect the performanceper-classorper-instance. In contrast, macro averaging treats all classes equally by computing metrics independently for each class and averages the results. In multi-label classification, the focus is often onper-instanceperformance rather thanper-classperformance. This means we prioritize the overall quality of multi- label predictions for each instance, rather than optimizing for the accuracy of individual labels in isolation. For this, the sample averaging computes metricsper-instanceand then averages across all samples. 4.5.3 Multi-label Classification Performance Analysis Table4.3presents a comparative analysis of BiCarFormer against established sequence- to-sequence models for multi-label error pattern classification. Multi-label classification is particularly challenging due to the varying complexity of error patterns, where a single instance may be associated with multiple diagnostic trouble codes (DTCs) and environ- mental conditions. DTC-TranGRU [ 49] underperforms, particularly inper-classand per-instanceevaluations, likely due to the limitations of masked attention mechanisms Micro and Macro Averaging ModelAUROC (Micro)F1 Score (Micro)F1 Score (Macro) BiCarFormer0.8090.770.71 DTC-TranGRU [49]0.6020.360.28 BERT [26]0.7680.710.63 Sample Averaging ModelPrecision (Sample)Recall (Sample)F1 Score (Sample) BiCarFormer0.680.620.64 DTC-TranGRU [49]0.230.190.20 BERT [26]0.590.530.55 Table 4.3:Downstream evaluation of multi-label error pattern classification. Each model has the same number of parameters (25M). Results are grouped by averaging type (mi- cro/macro vs. sample). 52 4.5 Experiments and model architecture. These constraints make it diïŹicult for the model to capture long-range dependencies in the input sequence, leading to a lower AUROC (0.602) and poor F1 scores (Micro: 0.36, Macro: 0.28, Sample: 0.2). While BERT [26] is already achieving relatively high AUROC and F1 Micro, BiCarFormer significantly enhances the AUROC (Micro) by +4%, and the F1 Score (Micro) by +6%. The improvement of BiCarFormer is especially visible forper-classaveraging. When dealing with many error patterns, some are more diïŹicult to distinguish due to their nat- ural complexity and overlaps. BiCarFormer, however, better differentiates rare classes, suggesting an improved capacity to generalize across a diverse range of error patterns. This is confirmed in this experiment, where we see the gaps in classification performance when averaging across classes, with a +8% F1 Macro compared to BERT. This means that classes with small instances, usually hard to classify using only the DTCs, can be better differentiated using environmental conditions. Finally,per-instanceprecision and recall are the biggest improvement with +9% in F1 Score, Precision, and Recall compared to BERT. These results confirm that BiCarFormer effectively leverages this additional information and highlights the predictive improvement compared to standard Transformers. 4.5.4 Cross-Attention Scores Interpretations We would like to understand how the co-attention mechanism enables multimodal learn- ing and enhances downstream tasks. We took a random test sample with a specific battery issue and ran it on BiCarFormer to analyze the different cross-attention score patterns. Figure 4.4presents a heat-map visualization ofA dtcâenv . This helps us to 0250500750100012501500 Environmental Condition Token Index 0 1 2 3 4 5 Summed Attention Score Head 0 Head 1 Head 2 Head 3 Head 4 Head 5 Figure 4.3:Attention Distribution between Environmental Condition Tokens. Amount of attention received by each environmental condition token from the DTCs. The y-axis was truncated to improve clarity as well as the number of heads printed. We takeA dtcâenv of the last layer. 53 4 Multimodal Sequence Modeling for Error Pattern Classification understand general attention patterns from DTCs to environmental conditions. Cross- attention uncovers natural relationships from certain DTCs linked to specific environ- mental conditions. Consider DTC 190, we can see that this token focuses more on a local series of environmental conditions with unitkm/hrather than other units. Whereas DTC 33 and 485 attend more to°C. This result is attributable to specific environmen- tal conditions that characterize certain DTCs, i.e., the temperature might define DTC1 while DTC2 might indicate a rise in the voltage. Due to the duplicated environmental conditions unit acrossS e , one DTC might attend to the next environmental conditions âvoltageâ also later in the sequence, creating these line patterns on Figure4.4with DTC 190 & 33 & 485. As a consequence, we notice the local temporal patterns between DTCs and their surrounding environmental conditions (Figure4.4). Therefore, the ini- tial DTCs should prioritize the initial environmental conditions inS e , with an offset resulting from the difference in sequence length betweenLandL e . For instance, DTC 190 at positions 1 and 6 should prioritize the initial environmental conditions. We would like to assess whether the DTCs attend to a few key environmental condi- tions. From the same test sample, Figure4.3plots the amount of attention that DTCs give to each of the environmental conditions. We observed different clusters per head. Each head learns to focus on specific environmental conditions, sometimes particular 1-[CLS][CLS] 9-243km/h 21-358v 16-73°c 20-334°c 9-286km/h 21-445v 20-120°c 16-85°c 21-440v 20-112°c 16-96°c 9-145km/h 5-[SEP][SEP] 9-243km/h 21-358v 16-73°c 20-334°c 9-286km/h 21-445v 20-120°c 16-85°c 21-440v 20-112°c 16-96°c 9-145km/h 5-[SEP][SEP] 9-243km/h 21-358v 16-73°c 20-334°c 9-286km/h 21-445v 20-120°c 16-85°c 21-440v 20-112°c 16-96°c 9-145km/h 5-[SEP][SEP] 9-243km/h 21-358v 16-73°c 20-334°c 9-286km/h 21-445v 20-120°c 16-85°c 21-440v 20-112°c 16-96°c 9-145km/h 5-[SEP][SEP] 9-243km/h 20-334°c 21-358v 16-73°c 20-120°c 21-445v 9-286km/h 16-85°c 21-440v 16-96°c 9-145km/h 20-112°c 5-[SEP][SEP] 9-243km/h 20-334°c 21-358v 16-73°c 20-120°c 21-445v 9-286km/h 16-85°c 21-440v 16-96°c 9-145km/h 20-112°c 5-[SEP][SEP] 9-243km/h 20-334°c 21-358v 16-73°c 20-120°c 21-445v 9-286km/h 16-85°c 21-440v 16-96°c 9-145km/h 20-112°c 5-[SEP][SEP] 9-243km/h 20-334°c 21-358v 16-73°c 20-120°c 21-445v 9-286km/h 16-85°c 21-440v 16-96°c 9-145km/h 20-112°c 5-[SEP][SEP] 21-358v 16-73°c 20-334°c 9-243km/h 16-85°c First 110 Environmental Conditions (description is anonymized) 1 190 194 193 192 174 190 194 193 192 174 719 899 1305 422 296 1378 409 410 1783 381 408 64 33 485 1812 81 211 74 400 1011 477 First 32 DTCs (anonymized) 0.000.020.040.060.080.100.12 Attention Score Intensity Figure 4.4:Cross-Attention Scores forA dtcâenv .The DTCs are shown on the y-axis (anonymized), and the environmental conditions with their 3 elements(d, v, u)con- catenated are shown on the x-axis (the descriptiondis anonymized). The intensity of each cell reflects the attention weight, where darker shades indicate higher at- tention values. 54 4.5 Experiments indices likehead 1(iâ749and800), andhead 2(iâ960). The [CLS] token is tradi- tionally heavily attended. Interestingly, the last part of the sequence seems to be more attended by specific heads:iâ[1300,1500], signaling that we can capture long-range dependencies between DTCs and tokens of environmental conditions even with a big difference in sequence length (i.e., withL e â«L). Moreover, late environmental condi- tions might carry a lot of defect signals, which makes sense as the sequence approaches the time of the critical failure event occurring at timet i =t L . Thus, attention scores increase correspondingly. 4.5.5 Error Pattern in Battery Electric Vehicles With the increasing popularity ofBEV(battery-electric vehicle), there is a pressing need to diagnose failures in these vehicle types. We take the example of a concrete error pattern, such as abattery aging, which is a common phenomenon in electric vehicles. And draw the cumulative attention of e. condition units for the same test sample as before. Since attention heads focus on a certain part ofS e (Figure4.3), some should focus on special units (Figure4.6) such as km/h (head 0, 4, 9) while other on °C (head 6, 7) and voltage (head 1, 2). Now, more interestingly, we take a head that tends to specialize in one unit likehead 2for voltage and draw the evolution of the discretized voltage value as a function of their index inS e in Figure4.5. We essentially try to answer if we have a fluctuation of voltage values and a correlated change in the attention pattern. We also draw the associated attention scores received from DTCs and obtain multipletrigger pointswhere the voltage value strongly correlates with attention scores. km/h °c v [SEP] % 1/min a counts mohm mv Environmental Condition Unit 0 20 40 60 80 100 120 Summed Attention Score Head 0 Head 1 Head 2 Head 3 Head 4 Head 5 Head 6 Head 7 Head 8 Head 9 Head 10 Head 11 Figure 4.5:Attention Distribution between Environmental Conditional Units. Amount of attention received by each environmental unit from the DTCs in a battery agingerror pattern from a battery electric vehicle. We takeA dtcâenv of the last layer and print all heads. 55 4 Multimodal Sequence Modeling for Error Pattern Classification We note that this is not the same depending on the heads; some exhibit exponentially decreasing attention scores shapes, while others exhibit noisy signals. As a consequence, BiCarFormer is able to extract specific fluctuations of continuous voltage values from a tokenized input. Engineers may extract trigger points and analyze how these specific variations affect the vehicle, providing a more explainable result. 0 100200300400 Environmental Condition Voltage Index 0 200 400 600 800 1000 Discretized Voltage Value Trigger Point 0.2 0.4 0.6 0.8 1.0 Attention Score Received From DTCs Figure 4.6:Discretized Voltage and Attention Scores Evolution between Environ- mental Token Position. Discretized voltage variation as a function of envi- ronmental condition tokens and the associated amount of attention received from DTCs. We only take triplets(d, v, u)fromS e withu=âvâ and extractA dtcâenv from the last layer andhead 2. The sample is taken from abattery agingerror pattern. BiCarFormer implicitly learns the fluctuation of discretized continuous values to make a prediction. 4.6 Applications to Downstream Tasks Having a separate hidden state for the environmental conditions also enables unsuper- vised learning in settings where EPs are unavailable. The hidden stateH e could provide a more accurate dimensionality reduction and observe better decision boundaries between unlabeled EPs using clustering techniques. Also, if provided with domain knowledge, we could directly filterS e beforehand to take specific environmental conditions. Our model and overall paradigm enable retention of this architectural choice and reduce the computation by injecting domain knowledge directly into the preprocessing steps. An- other possible useful downstream application is explainability. We would like to see the contributions of environmental conditions and DTCs to the EPs classification. Since error patterns are hard-coded defined rules, we could try to derive new rules for un- known error patterns based on feature attribution methods [ 83], cross-attention scores, perturbation-based methods or multi-label causal discovery with Transformers. BiCar- 56 4.7 Summary Former model classifies EPs a posteriori (offline setting); thus, the inference optimization might not be needed since we do not rely on edge computing capabilities within the vehi- cles. Nevertheless, due to the quadratic time complexity of cross-attention scores, there is an extensive need to optimize Eq.4.7, and Eq.4.8. Alternatives like sparse attention [78,168,14], token merging [16] or pooling should be adapted for co-attention in event sequences. 4.7 Summary We introduced BiCarFormer, a bidirectional multimodal Transformer designed to clas- sify EPs by jointly encoding discrete DTC sequences and continuous environmental conditions. Through a co-attention mechanism and hierarchical embedding fusion, Bi- CarFormer learns cross-modal dependencies between failure events and their surround- ing sensory context. On a real-world automotive dataset comprising 22,137 DTCs and 360 error patterns, BiCarFormer achieves an AUROC of 0.809, a micro F1 score of 0.77, and a sample-level F1 score of 0.64, representing improvements of +4%, +6%, and +9% respectively over a standard BERT baseline trained on DTC sequences alone. The gains are most pronounced in per-class and per-instance evaluations, confirming that environmental conditions provide discriminative signal for rare and overlapping er- ror patterns that share identical DTC rule definitions. Interpretability analysis of the cross-attention scores reveals that individual attention heads specialize in specific en- vironmental units, such as voltage or temperature, and implicitly detect fluctuations in discretized continuous values that correlate with fault onset. This provides an addi- tional layer of transparency beyond what DTC-only models offer. However, two practical limitations merit acknowledgment. First, the quadratic complexity of the co-attention mechanism with respect to the environmental condition sequence length (L e â«L) con- stitutes a computational bottleneck for long sequences, motivating future work on sparse or pooled cross-attention variants. Second, BiCarFormer classifies EPs offline (a pos- teriori), in contrast to EPredictorâs online, causal-mask-based operation; it is therefore better suited to retrospective diagnostic analysis than real-time on-board deployment. 4.8 Outlook PartIhas demonstrated that large-scale Transformers can serve as effective predictive and diagnostic models for high-dimensional vehicle event streams. The introduced mod- els demonstrated that NLP techniques such as word embeddings, positional encoding, and transformer models can be adapted to event sequences to capture the hierarchical dependencies among diagnostic trouble codes (DTCs). The connection is not merely se- quential: the autoregressive models of Part I serve as neural density estimators in Part I, a repurposing that distinguishes this thesis from prior work. By interpreting attention scores, automotive engineers can better understand BiCarFormer predictions. However, we must establish cause-and-effect relationships between DTCs and EPs rigorously, as relying on attention scores alone lacks reliability [ 156,11]. While predictive models en- 57 4 Multimodal Sequence Modeling for Error Pattern Classification able early fault detection, they remain fundamentallycorrelational: a high F1 score tells us what will fail and when, but not why. Without understanding the underlying causal mechanisms, any prediction remains fragile, susceptible to distributional shift, and inca- pable of supporting counterfactual reasoning or root-cause analysis. Establishing such causal relationships requires moving beyond correlation-based models toward methods that explicitly encode the data-generating process. This demands a formal framework for causal discovery, grounded in probability and information theory, that is capable of operating on the scale of tens of thousands of event types over multiple samples. Part I introduces precisely this foundation. 58 Part I Causal Discovery in High-Dimensional Event Sequences via Neural Density Estimation 59 5 Foundations of Information Theory and Causal Discovery Having established predictive models in Part I, this chapter turns to understanding. Causal discovery requires measuring statistical dependencies between random variables in graphical models, a task for which information theory provides the right tools. We therefore introduce the related background, including the common definitions and mod- eling techniques required for PartII. LetUdenote the set of all (discrete) random variables. We define the event set X=X 1 , . . . , X n âU, and the label setY=Y 1 , . . . , Y l âU. When explicitly said, eventX (t i ) i represent the occurrence ofX i at stepiand timet i and similarly forY (t i ) i . We use this notation to improve clarity, since we mainly deal with stationary stochastic process (Assumption3). 5.1 Information Theory In this section, we present the primary mathematical tools from probability and infor- mation theory that we will use in Part I. 5.1.1 Marginal, Conditional, and Relative Entropy 5.1.1.1 Definitions Definition 9(Entropy).The entropy [130] of a discrete random variableXwith a probability mass functionp(x) =P(X=x)is defined by H(X) =â X xâX p(x) logp(x)(5.1) It can be thought of as the amount of surprise of the random variableXand thus quantifies its uncertainty. We measure it innatsas we use the cross-entropy loss of PyTorch [111] in our experiments. Definition 10(Conditional Entropy).We can define the conditional entropy as the entropy of a random variable given another random variable H(Y|X) =â X xâX,yâY p(x, y) log p(x, y) p(x) (5.2) 61 5 Foundations of Information Theory and Causal Discovery This can also be written as a function of the posteriorp(y|x)using the chain rule of probability: H(Y|X) =â X xâX,yâY p(x)p(y|x) logp(y|x) =âE p(x,y) logp(Y|X)(5.3) It can be thought of as theremaininguncertainty inYwhen we already knowX. Definition 11(Relative Entropy).The relative entropy is defined as the distributional difference or Kullback-Leibler-Divergence(D KL )[23] between two probability distribution PandQ: D KL (P||Q) = X xâX p(x) log p(x) q(x) (5.4) It is an asymmetric âdistanceâ measure between two probability distributions. Definition 12(Joint Entropy).The joint entropyH(X, Y)of a pair of discrete random variables(X, Y)with joint distributionP(X, Y)is defined as: H(X, Y) =â X xâX X yâY p(x, y) logp(x, y)(5.5) which is also the expected value oflogp(x, y): H(X, Y) =âE p(x,y) logp(X, Y) Definition 13(Information Gain).The information gainI G of a random variableY obtained from an observation of the realization ofX=xis defined as I G (Y, x) =H(Y)âH(Y|x)(5.6) It can also be written as a distributional difference I G (Y, x) =D KL (P(Y|X=x)||P(Y))(5.7) Information gain is used to construct decision trees [120]. It will be our primary tool for performing causal discovery from posterior distributions. 5.1.1.2 Properties The following describes the key properties regarding the different types of entropy. Lemma 1(Non-Negativity of the Entropy).[23] The entropy of a random variable is always non-negative H(X)â„0 Lemma 2(Chain Rule).[23] We can decompose the joint entropy into a sum of marginal and conditional entropy H(X, Y) =H(X) +H(Y|X) 62 5.1 Information Theory Lemma 3(Conditioning Reduces Entropy).[23] Conditioning the entropy of a random variableXon multiple random variables reduces its entropy H(X)â„H(X|Y)â„H(X|Y, Z)â„·â„0 Lemma 4(Bounded).[23] The entropy of a random variableXis bounded by the log of its support|X| log|X|â„H(X) This follows from the fact thatH(X)is concave inpand is non-negative. 5.1.2 Mutual Information and its variants 5.1.2.1 Definitions Definition 14(Mutual Information).The reduction in uncertainty due to another ran- dom variable is called the mutual information (MI) [23]. For two random variablesX, Y with joint probability mass functionp(x, y), it is given as the relative entropy between the joint distributionP(X, Y)and the product distributionP(X)P(Y) I(Y, X) =D KL (P(X, Y)||P(X)P(Y)) = X xâX,yâY p(x, y) log p(x, y) p(x)p(y) (5.8) It can also be defined as the difference betweenH(Y)andH(Y|X) I(Y, X) =H(Y)âH(Y|X)(5.9) Consequently, the mutual informationI(Y, X)is the reduction in uncertainty ofXdue to the knowledge ofY. We writeI(Y, X)for mutual information, the semicolon notation I(X;Y)is equivalent. We now define the conditional mutual information (CMI), which is the reduction in uncertainty ofXdue to the knowledge ofYgivenZ. Definition 15(Conditional Mutual Information).The conditional mutual information of random variables X and Y given Z is defined by I(Y, X|Z) =H(Y|Z)âH(Y|X, Z) (5.10) =E p(x,y,z) log p(X, Y|Z) p(X|Z)p(Y|Z) (5.11) Zcan be also multiple random variables such asZ=X 0 , X 1 ,·, X i . This notation will be particularly used in this PartII. Now that we have defined the conditional mutual information, we can define conditional independence between random variables. Definition 16(Conditional Independence).VariablesXandYare said to be condi- tionally independent given a variable setZ, ifP(X, Y|Z) =P(X|Z)P(Y|Z), denoted as Xâ„Y|Z. Inversely,XÌžâ„Y|Zdenotes the conditional dependence. Using the conditional mutual information [ 23] to measure the independence relationship, this implies that I(Y, X|Z) = 0âXâ„Y|Z 63 5 Foundations of Information Theory and Causal Discovery 5.1.2.2 Properties Lemma 5(Symmetry of Mutual Information).In its standard definition, the MI is symmetric: I(Y, X) =I(X, Y) =H(Y)âH(Y|X) =H(X)âH(X|Y) Lemma 6(Non-negativity of mutual information).For any two random variableX, Y, I(Y, X)â„0(5.12) It is also the case for the conditional mutual information: I(X, Y|Z)â„0(5.13) 5.2 Graphical Models Graphical models [70] provide a simple way to represent probabilistic dependencies across random variables. For instance, a physician must infer the most likely diagnoses, given the patientâs symptoms, test results and genetics. A diagnostic expert might want to evaluate the causes of defects in a vehicle based on the series of DTCs that preceded it. 5.2.1 Bayesian Networks 5.2.1.1 Motivation Our goal is to represent the joint distribution P over the set of random variablesU. To reduce the specification of every possible random variable assignment, the independence properties in the distribution can be used to represent distributions via directed acyclic graphs [ 70], i.e., graphs that have oriented edges and no cycles. Bayesian Networks [115] have served as a modeling technique for a variety of decision problems and benefit from the extensive literature on causal discovery associated with such directed acyclic graphs (DAGs). In particular, we build upon Pearlâs model of causality, which assumes that the underlying causal structure is in the form of a directed acyclic graph (DAG) [ 114]. Such a graph describes the causal relationships between the random variables. 5.2.1.2 Definitions Definition 17(Bayesian Network).[115] LetPdenote the joint distribution over a variable setUof a directed acyclic graph (DAG)G. The triplet<U,G, P >constitutes a Bayesian Network (BN) if it satisfies the Markov condition: every random variable is independent of its non-descendant variables given its parents inG. Each nodeX i âU represents a random variable. The directed edge(X i âX j )encodes a probabilistic dependence. The joint probability distribution can be factorized as P(X 0 ,·, X n ) = n Y i=0 P(X i |Pa G (X i ))(5.14) 64 5.3 Tasks with Pa G (X i )the parents of nodeX i inG. We now defineI-maps, which will help us identify graphs that represent the same set of conditional independence. Definition 18(I-maps).[70] LetPbe a distribution overU, we defineI(P)to be the set of independence assertions of the formXâ„Y|Zthat hold inP We can rewrite the statement thatPsatisfies the local independencies associated with the graphGasI(G)â I(P). The graphGencodes a set of conditional independencies I(G), where each element corresponds to a conditional independence relationXâ„Y|Z, meaning thatXandYare conditionally independent given the variableZ(Def.16). Definition 19(Faithfulness).[141] Given a BN<U,G, P >,Gis faithful toPif and only if every conditional independence present inPis entailed byGi.e.,I(P)â I(G) and the Markov condition holds.Pis faithful if and only if there exist a DAGGsuch thatGis faithful toP. That is, the conditional independencies inPcorrespond exactly to those implied by the DAG. Definition 20(Markov Equivalence Class).Two distinct graphsG,G âČ are said to belong to the same Markov Equivalence Class (MEC) if they have the same set of conditional independencies, i.e.,I(G) =I(G âČ ). Causal Discovery algorithm often recovers not the exact graphGbut rather an MEC [21, 140,41], since orienting the edges is perilous and involves, in general, domain knowledge or additional assumptions. Definition 21(Markov Boundary).[149] In a faithful BN<U,G, P >, for a set of variablesZâUand labelYâU, if all other variablesXâXâZare independent ofYconditioned onZ, and any proper subset ofZdoes not satisfy the condition, then Zis the Markov Boundary ofY:MB(Y). Many local structure learning (LSL) algorithms seek a reduced Bayesian Network in the form of theMarkov Boundaryof certain random variables. This serves primarily for feature selection [ 77]. 5.3 Tasks 5.3.1 Causal Discovery Causal discovery aims to recover the underlying causal structure among a set of random variablesXfrom observational data and or interventions [114]. Formally, the goal is to identify a directed acyclic graph (DAG)G= (X, E)whose edges encode the direct causal relations such that the joint distributionP(X)satisfies the Markov condition with respect toG(Definition 17). Depending on the assumptions and available information, the tasks consist of recovering either: 65 5 Foundations of Information Theory and Causal Discovery âąThefull DAGG, âąAMarkov Equivalence Class (MEC)(Def.20), âąOr theMarkov Boundaryof selected target variables (Def.21), Each representation offers a different level of identifiability. In high-dimensional settings such as error codes in vehicles, recovering the complete DAG is often infeasible; Many methods, therefore, focus on local structures (Markov Boundaries) or on equivalence classes of DAGs. 5.3.1.1 Assumptions To enable identifiability, we must make several assumptions that we causal discovery algorithms typically rely on classical assumptions such as: Assumption 1(Causal SuïŹiciency).All relevant variables are observed, and there are no hidden confounders afecting the events and labels. Furthermore, to be able to detect any conditional independencies inGviaP(i.e., through a statistical test) the faithfulness assumption is adopted (Def.19). For se- quences, we assume temporal precedence (or priority [ 62]) which simply edge orientation by making causality asymmetric in time. That is: Assumption 2(Temporal Precedence).Given a perfectly recorded sequence of events ((x 0 , t 0 ),·,(x L , t L ))with labels(y L , t L )and monotonically increasing time of occur- rence0â€t 1 †· â€t L , an eventx i is allowed to influence any subsequent eventx j such thatt i â€t j andi < j. Formally, the graphG= (U,E),(x i , x j )âE=ât i †t j and stepi < j Violations of these assumptions (hidden confounders, cycles, non-stationarity) are com- mon in practice due to the inaccurate recording of events, hidden confounders, and so forth. 5.3.1.2 Classes of Causal Discovery Algorithms Constraint-Based Methods.These algorithms determine graph structure by perform- ing a series of conditional independence tests (Def.16). They are mostly employed in tabular data where a lot of data samples are presents with a few number of variables. Examples include PC [ 140], and local structure learning methods such as IAMB [148], MBB-BY-MBB [161], or HITON [3]. Granger Methods.In temporal data, Granger [44] causality is commonly employed to assess pairwise dependencies [162,119]. It is based on the assumption thatcauses precede effects and should improve the predictability of the effect. Recently, [ 51] pro- posed a Granger-inspired causal discovery framework in multivariate time series using 66 5.3 Tasks an encoder-decoder architecture. Their method achieves state-of-the-art root-cause iden- tification via autoregressive modeling of structured time series. [119] explores Granger causality under low-resolution temporal data using Hawkes processes [53] and shows gains in F1 across time granularities. However, their setup also assumes relatively small event vocabularies. CASCADE [24] recovers DAGs from temporal event data under a Poisson process assumption but is limited to smaller event spaces (âŒ200types). All these approaches remain limited and are applied in poor experimental settings in which the number of variables (or events) does not reflect real-world and industrial applica- tions. Additionally, Granger causality refers more to probabilistic causation [33] than Pearl causality [115]. It is often considered a relatively weak notion of causality that ex- cludes instantaneous and inhibitory effects and is usually unable to discover true causal dependencies. In Granger causality, baking a cake is causal to a birthday. Score-Based Methods.These methods search over graph structures to maximize a scoring function (e.g., Bayesian Information Criterion: BIC or BDeu). The Greedy Equivalence Search (GES) [21] is the most popular method and relies on decomposable, score-equivalent metrics (Definitions 27â29). Optimization-Based and Neural Structural Learning Methods.A recent line of work formulates causal discovery as a continuous optimization problem, enabling gradient- based learning of DAG structures. The NOTEARS framework [178] introduced a smooth acyclicity constraint that relaxes the discrete DAG search into a differentiable objec- tive, making it possible to learn graph structures via standard gradient descent. Then, GOLEM [ 102] proposed a more scalable likelihood-based formulation, while DAG-GNN [167] leveraged variational autoencoders and graph neural networks. Temporal and Point-Process-Based Models.In sequential domains, Hawkes Processes and Temporal Point Processes model temporal causation via excitation kernels [ 74]. In PartIwe mentioned that the excitation matrixΊâR |X|Ă|X| was used as a proxy for causality. This excitation matrix is usually binarized to produce the adjacency matrix of the corresponding graph. These approaches impose parametric assumptions and are interesting to discover causal relationships between different event streams [ 174], rather than a single high-dimensional one like in vehicle fault sequences. 5.3.2 Multi-Label Causal Discovery In multi-label settings, the objective is to recover the direct causes of each labelY j among a large set of eventsX. Instead of estimating the complete graph, one seeks the Markov Boundary:MB(Y j )âX. 5.3.3 Evaluation Metrics for Multi-Label Causal Discovery In the multi-label setting (events-to-outcome), we aim to identify the set of direct tempo- ral causesP j âXfor each outcome labely j . Following the Markov assumption, this set 67 5 Foundations of Information Theory and Causal Discovery corresponds to the Markov BoundaryMB(Y j )of the label. We evaluate the alignment between the ground-truth boundaryMB(Y j )and the inferred boundary d MB(Y j ). 5.3.3.1 Set-Based Metrics To measure the local recovery of the causal neighborhood for each label, we utilize the following metrics: âąPrecision(P) quantifies the reliability of the inferred causes, measuring the pro- portion of discovered event types that are true causal parents: P= | d MB(Y j )â©MB(Y j )| | d MB(Y j )| âąRecall(R) (or Sensitivity) measures the completeness of the discovery, capturing the proportion of the true causal mechanism recovered by the model: R= | d MB(Y j )â©MB(Y j )| |MB(Y j )| âąF1-Score(F 1 ) provides the harmonic mean of precision and recall. It is our pri- mary metric for causal discovery, as it penalizes both the hallucination of spurious causes and the omission of weak causal signals: F 1 = 2· P·R P+R 5.3.4 Evaluation Metrics for Event-to-Event Causal Discovery For global structure recovery (Chapter8), we reconstruct the full event-type adjacency matrix Ë Aâ 0,1 |X|Ă|X| . Unlike set-based evaluation, graph-based evaluation must account for the structural topology of the system. 5.3.5 Structural Hamming Distance (SHD) The Structural Hamming Distance is the standard metric for comparing two causal graphsGand Ë G. It counts the minimum number of edge operations (additions, deletions, or orientation reversals) required to transform the estimated graph into the ground truth. In our directed temporal setting, where orientation is fixed by time, SHD reduces to the L 1 distance between the flattened adjacency matrices: SHD(A, Ë A) = |X| X i=1 |X| X j=1 A ij â Ë A ij 68 6 Sample-Level Multi-Label Causal Discovery Building upon the predictive foundations established in the previous chapter, where autoregressive Transformer models were used to anticipatewhenandwhaterror patterns occur, we now turn to a complementary and equally essential question:whythese failures arise. We observe that autoregressive models trained on next-event prediction already encode the conditional distributions required by information theory metrics such as information gain (Def.13), potentially usable for conditional independence testing in a graphical model. We therefore reuse the pretrained CarFormer and EPredictor weights, without fine-tuning, asNADEs. Consequently, in this chapter we address the challenge of causal discovery in large-scale event sequences, in which outcomes, such as error patterns, emerge from preceding diagnostic events in a single sample during inference (sample-level). It is based on the following publication: One-Shot Multi-Label Causal Discovery in High-Dimensional Event Sequences. [94] Hugo Math, Robin Schön, Rainer Lienhart, Conference on Neural Information Pro- cessing Systems (NeurIPS) Workshop on CauScien: Uncovering Causality in Science, San Diego, USA, December 2025. 6.1 Introduction Understanding causal relations among diagnostic events is a crucial next step toward interpretable and automated reasoning in large-scale event-driven systems. Causal dis- covery in sequential data has broad relevance across domains such as healthcare [123, 57], cybersecurity [86], flight operations [84], and vehicle diagnostics [117]. Although recent progress in sequence modeling, particularly through Transformers [151,122,147] has improved our ability to capture temporal dependencies, most methods remain fo- cused on prediction rather than explanation. Traditional causal discovery methods, such as constraint-based or Granger-causal approaches [ 44], face severe computational bot- tlenecks when applied to industrial-scale event data [77,166,52,41]. Their reliance on repeated conditional independence tests grows combinatorially with the number of event types, rendering them impractical for modern diagnostic systems. Interpreting attention weights as a measure of causality is a promising direction [101, 125] but lacks theoretical guarantees and often admits strong parametric assumptions as in [125]. Some authors strongly criticize the reliance on attention scores, arguing that they provide no explanatory value [ 156]. On the other hand, repurposing Transformers as neural density estimators [64,97] offers a promising direction, enabling the modeling 69 6 Sample-Level Multi-Label Causal Discovery of sequences using graphical models [70] and the use of the learned joint probability distribution. Nevertheless, all of these methods often aim to recover a single global causal graph. A representation that is not only diïŹicult to interpret for a lot of different events but also misaligned with how practitioners reason about causality in practice. In operational settings, reasoning about causality typically occurs at thesample-level: given a single temporal sequence of observed events, one seeks to identify the specific causes leading to an outcome such as a system failure, defect, or disease with one sample during inference. For instance, in vehicle diagnostics, engineers may ask:âWhich diagnostic events most likely caused this particular error pattern?âThis local perspective motivates a more eïŹicient, explainable form of sample-level causal discovery. To address this underexplored problem this chapter introduces OSCAR, to the best of our knowledge, the first sample-level causal discovery method that extends the pre- dictive modeling framework of the previous chapter toward causal reasoning. OSCAR performs a single-pass, fully parallelized causal extraction on GPUs and recovers an interpretable causal graph with quantified causal indicators and uncertainty measures. This enables practical multi-label causal discovery at a scale previously unattainable, supporting thousands of event types and hundreds of outcome labels simultaneously. This chapter, therefore, marks a conceptual shift: from purely predictive modeling to causal understanding, laying the foundation for the automated reasoning systems devel- oped in the following chapters. 6.2 Related Work 6.2.1 Neural Autoregressive Density Estimation NADEs were initially introduced for density estimation via chain-rule factorization of the joint distribution using neural networks in the original paper of Bengio [15], later extended through recurrent architectures [ 22,58] and Transformers [151]. These mod- els are trained using next-token prediction by minimizing the negative log-likelihood of observing the sequences= (x 0 , . . . , x L ). Recall that the joint probabilityP(s)can be expressed using the chain rule as in Eq.2.11. Recent work has explored autoregressive models as tools for causal inference. For example, [ 39] leverages density estimators to simulate interventions and compute average treatment effects. [64] shows that autore- gressive language models can approximate Bayesian networks (Fig.6.2), treating the model itself as a statistical engine for causal inference. These findings motivate our use of pretrained Transformers to estimate conditional mutual information (Def. 15) between events and labels. Specifically, we repurposed these models asNADEs for both the events and labels, allowing us to quickly estimate the conditional probabilities of the next eventx i and labelsygiven past events(x 0 ,·, x iâ1 ). 70 6.3 Methodology 6.2.2 Transformers as Causal Learners Transformer-based models have gained growing attention in causal discovery literature. [104] showed that when trained on sequences generated from in-context Markov chains, they can implicitly learn latent causal graphs, where attention weights align with the adjacency matrix of the actual causal structure. For sequential data, [125] analyzes self-attention under the assumption that data is generated by a linear-Gaussian struc- tural causal model (SCM) [141]. They relate the covariance of endogenous variables to attention scores and apply conditional independence (CI) tests to the final layerâs outputs to recover a partial ancestral graph. Our work builds on this idea by leveraging Transformers to grant full access to the historyH t using the self-attention mechanism. 6.2.3 Multi-label Causal Discovery Multi-label causal discovery seeks to identify the Markov Boundary orMB(Def.21) of each label, i.e., its minimal set of parents, children, and spousesâsuch that the label is conditionally independent of all other variables given itsMB[149]. This boundary constitutes an optimal feature set for tasks such as explainable modeling and feature selection under the faithfulness assumption. However, rather than learning the full joint causal graph, which is known to be NP- hard [20]âwe focus on recovering local causal structure(LCS) [166]: discovering minimal sub-graphs from events to labels within a single sequence. This formulation makes the problem tractable in high dimensions and better suited for real-world production scenarios. Unlike event-to-event causal learning, multi-label causal discovery remains unexplored in event sequences [ 41,52], yet its potential applications are broad across various domains. 6.3 Methodology Fig.6.1depicts OSCAR, our causal discovery method for event sequences. The lemmas and theorems are directly followed by their proofs. 6.3.1 Autoregressive Event Sequence Models. We begin by repurposing the two Transformer architectures introduced in PartIto perform next event (CarFormeras Tf x ) and next labels (EPredictoras Tf y ) prediction. These two autoregressive Transformers model the conditional probability distribution of the next events and labels conditioned on the past sequence of observed eventsZ= (X 1 ,·, X iâ1 ) =S <i , the predictive distributions are defined as: P Ξ x (X i |Z)âSoftmax(h x iâ1 ) =Tf x (S <i ) (6.1) P Ξ y (Y|X i ,Z)âSigmoid(h y i ) =Tf y (S â€i )(6.2) 71 6 Sample-Level Multi-Label Causal Discovery Time P(Yi|S') S' P(Yi+1|S) S Minimum context c ... ... Z_i^e Next Multi-label Neural Autoregressive Density Estimator: Tf_y Next Event Neural Autoregressive Density Estimator: Tf_e Time P(Xi+1|S) ... Event Type Labels Zi^y P(Xi+1|S) y hat _i x hat _i (1) Extract Conditional probabilities for sequence with label Y3 (2) Compute the CMI I(X_i, Y_i+1|Z) by sampling from Tf_e If I(X_i, Y_i+1, 3|Z) > theta, X is a cause of Y1 (3) Draw an edge from X_i to Y1 Xi Xi+1 Xc Xi-1 ... Y1 Y2 ... (4) Compare MBs and use back-door criterion to get inter- label edges Yn MB1 MB2 (backbone) Time ... Event Type Extract Conditional Probabilities Hidden States Next-Event Predictor: Next-Label Predictor: Time ... Event Type Sampling Parallel independance testing using CMI Draw an edge from if CMI Extract Probabilities from Event Sequence Next-Event Predictor: Next-Label Predictor: Time time ... Event Type 1) Infer next events: Infer Compute CI-test Draw an edge from if CMI ) context 3) Infer hidden states 2) top-k + nucleus sampling 4) Infer next labels 5) CMI Thresholding on each label Time Event Type sample times Hidden states Compute and Figure 6.1:The Overview of OSCAR:One-Shot multi-labelCausalAutoRegressive dis- covery.ddenotes the hidden dimension,Lthe sequence length,MB 1 ,MB 2 the Markov Boundary ofY 1 , Y 2 respectively. All green and blue areas represent paral- lelized operations. Here,h x iâ1 ,h y i âR d are the logits produced by the two Transformer heads of Tf x and Tf y parametrized byΞ x , Ξ y . The majority of Tf x (except the heads) serves as a backbone for Tf y , see Fig.6.1. 6.3.2 Assumptions Working with causal structure learning from observed data requires several assumptions. As stated in the foundation, we assume: (1) the causal Markov assumptions [ 115], which state that a variable is conditionally independent of its non-descendants given its parents, (2) no hidden confounders (A1), (3) temporal precedence (A2) and (4) a faithfulness via the defined BN (Def.19). Formally, we assume the following throughout this chapter: Assumption 3(Stationary Markovian Dynamic System).[70] We assume that the probability distributionP(Y (t j+1 ) j |Z (t j ) )is the same for allt. In this case we represent our transition probabilityP(Y j |Z), so that for any,tâ„0,as: P(Y (t i+1 ) j =y|Z (t i ) =z) =P(Y j =y|Z=z)(6.3) Stationarity allows us to compute the correct posteriorsP(Y (t i ) j |Z, X i )even at different time stept i . Assumption 4(Oracle Models).We assume that two autoregressive Transformer mod- els, Tf x and Tf y , are trained via maximum likelihood on a dataset of multi-labeled event sequencesD=S (1) l ,·, S (n) l , and can perfectly approximate the true conditional dis- tributions of events and labels: P(X i |Pa G (X i )) =P Ξ x (X i |Pa G (X i )) =Tf x (S <i )(6.4) P(Y j |Pa G (Y j )) =P Ξ y (Y j |Pa G (Y j )) =Tf y (S â€i )(6.5) 72 6.3 Methodology This is a mild to strong assumption. Although we do not take any structural as- sumption about the data generating process, we assume that the two autoregressive models perfectly captures the true distribution. We challenge this assumption in the experiments as well as in Chapter8. Assumption 5(Bounded Lagged Effects).Once we observed events up to timestampt i and stepiasZ=X 0 ,·, X iâ1 , any future lagged copy of eventX (t i +Ï) i is independent ofY j conditioned onZ: Y j â„X (t i +Ï) i |Z WhereÏ=t i+1 ât i is a finite bound on the allowed time delay for causal influence. In other words, we allow the causal influence of eventX i onY j until the next eventX i+1 is observed. We acknowledge that this may not hold for data with strong lagged effects (e.g., financial transactions), but it is relevant for log-based and error code-based data. An extended discussion on the impact of assumptions is provided later in Section6.6 6.3.3 Lemmas We can derive the Markov Boundary of each label in the Bayesian Network constructed from a multi-labeled sequence (Fig.6.2). Lemma 7(Markov Boundary Equivalence).In a multi-label event sequenceS l and under the temporal precedence assumption A2, the Markov Boundary of each labelY j is only its parents such thatâXâUâPa G (Y j ), Xâ„Y j |Pa G (Y j )âMB(Y j ) =Pa G (Y j ). Proof.Let<U,G, P >be the BN composed of the events from the multi-labeled se- quence. Following the temporal precedence assumption A 2, the labelsy L can only be caused by past events(x 0 ,·, x L ); we assume that labels do not cause any other labels. Thus,Y j has no descendants, so no children and spouses. Therefore, together with the Markov Assumption we know thatâXâ UâPa G (Y j ):Y j â„X|Pa G (Y j ). Which is the definition of the MB (Def. 21). ConsequentlyMB(Y j ) =Pa G (Y j ). To recover the causal graphG, we must make sure it is identifiable from the observational data. Time Event Type X 0 · X Lâ2 X Lâ1 Y L Identification X 0 · X Lâ2 X Lâ1 Y 1 Y 2 MB 1 MB 2 Figure 6.2:Example of a Causal Graphextracted from a multi-labeleventsequence where MB 1 represents the Markov Boundary ofY 1 andMB 2 the Markov Boundary of Y 2 . 73 6 Sample-Level Multi-Label Causal Discovery Lemma 8(Identifiability ofG).Assuming the faithfulness condition holds for the true causal graphG. Let Tf x and Tf y be Oracle models perfectly approximating the actual conditional distributions of events and labels, respectively. The joint distributionP Ξ x ,Ξ y can then be constructed, and any conditional independence detected from the distributions estimated by Tf x and Tf y corresponds to a conditional independence inG: X i ℠Ξ x ,Ξ y Y j |Z=âX i â„ G Y j |Z. Where℠Ξ x ,Ξ y denotes the independence entailed by the joint probabilityP Ξ x ,Ξ y . Proof.We assume that the data is generated by the associated causal graphGfollowing the BN from a multi-labeled sequenceS l . Given that the Oracle models Tf x and Tf y are trained to perfectly approximate the true conditional distributions, for any variableU i in the graph, we have: P(U i |Pa G (U i )) = ( P(Y j |Pa G (Y j )) =P Ξ y (Y j |Pa G (Y j )),ifU i âY P(X i |Pa G (X i )) =P Ξ x (X i |Pa G (X i )),otherwise. The joint distributionP Ξ x ,Ξ y can then be constructed using the chain rule P Ξ x ,Ξ y (X 0 ,·, X i , Y 1 ,·, Y c ) = i Y k=0 P(X k |P a G (X k )) c Y l P(Y l |Pa G (Y l )) By the faithfulness assumption [115], if the conditional independencies hold in the data, they must also hold in the causal graphG: X i â„Y j |Z=âX i â„ G Y j |Z Since we can approximate the true conditional distributions, it follows that: X i ℠Ξ x ,Ξ y Y j |Z=âX i â„Y j |Z=âX i â„ G Y j |Z Where℠Ξ x ,Ξ y denotes the independence entailed by the joint probabilityP Ξ x ,Ξ y . As a result, the graphGcan be identified from the observational data. 6.3.4 Conditional Mutual Information Estimation via Autoregressive Models OSCAR works like a constraint-based causal discovery algorithm, where the condition- ing set of nodesZincreases over time as we unfold the Bayesian Network in Fig.6.2. Specifically, we would like to assess how much additional information an eventX i oc- curring at stepiprovides about labelY j when we already know the past sequence of eventsZ=X 0 ,·, X iâ1 . We essentially try to answer if: P(Y j |X i ,Z) =P(Y j |Z)âD KL (P(Y j |X i ,Z)â„P(Y j |Z)) = 0 whereD KL denotes theKullback-Leibler divergence(Def.11). The distributional dif- ference between the conditionalsP(Y j |X i ,Z), P(Y j |Z)is akin to Information GainI G conditioned on past eventsz=x 0 ,·, x iâ1 (Def. 13): 74 6.3 Methodology I G (x i , Y j |z)âD KL (P(Y j |X i =x i ,Z=z)||P(Y j |Z=z))(6.6) Which is equal to the difference between the conditional entropies denoted asH: I G (Y j , x i |z) =H(Y j |z)âH(Y j |x i , z)(6.7) More generally, we can use the CMI to assess conditional independence (Def.16), which is simply the expected value of the information gainI G (Y j , x i |z)such as: I(Y j , X i |Z)âH(Y j |Z)âH(Y j |X i ,Z) =E p(z) [I G (Y j , X i |Z=z)](6.8) It can be interpreted as the expected value over all possible contextZof the deviation from independence ofX i , Y j in this context. To approximate Eq.6.8, a naive Monte Carlo estimation [29] is performed where we drawNrandom variations of the conditioning set z (l) =x (l) 0 , . . . , x (l) iâ1 , denoting thel-th sampledparticlefromP Ξ 1 : Ë I N (Y j , X i |Z) = 1 N N X l=1 I G (Y j , X i |Z=z (l) )(6.9) Under Assumption4(Oracle model), the estimator Ë I N is unbiased with respect to the true distributionP. Additionaly, sinceI G (Y j , X i |Z=z)is a difference between conditional entropies (Eq. 6.7), it is thus bounded uniformly [23] by the log of supports |Y|(Lemma4) such as: 0< I G (Y j , X i |Z=z (l) ) =H(Y j |z (l) )âH(Y j |X i , z (l) ))â€H(Y j )â€log|Y| Thus the posterior variance off i =I G (Y j , X i |Z=z (l) )satisfiesÏ 2 f i âE p(z) [f 2 i (p(z)]â I 2 (f i )<+â[29] then the variance of Ë I N (f i )) is equal tovar( Ë I N (f i )) = Ï 2 f t N and from the strong law of large numbers the estimation Ë I N converges almost surely to the conditional mutual information: Ë I N a.s. ââ Nâ+â E p(z) [I G (Y j , X i |Z=z)]âI(f i ). (6.10) 6.3.5 Sequential Markov Boundary Recovery In practice, a label-specific thresholdÏ j â0is applied to Eq.6.9to identify conditional independence: Y j Ìžâ„X i |ZâI(Y j , X i |Z)> Ï j â0(6.11) Ï j is dynamically computed for each label based on the mean and standard deviation of the CMI values across the sequence, such that:Ï j =ÎŒ Y j +kÂ·Ï Y j , wherekcontrols the confidence interval. We analyze the effect ofkin Appendix.B.2. 1 Averaging the information gain over multiple sampled past event histories enables tractable computa- tion as we do not need to enumerate over all possible contextsx 0 ,x 1 ,x 2 ,x 2 ,x 1 ,x 0 ,x 1 ,x 2 ,x 0 ,· 75 6 Sample-Level Multi-Label Causal Discovery Theorem 1(Markov Boundary Identification in Event Sequences).IfS (k) l a multi- labeled sequence drawn from a datasetD=S (1) l ,·, S (n) l where two Oracle Models Tf x and Tf y were trained on, then under causal suïŹiciency (A1), bounded lagged effects (A5) and temporal precedence (A2), the Markov Boundary of each labelY j in the causal graphGcan be identified using conditional mutual information for CI-testing. Proof.By induction over the sequence lengthLof the multi-label sequenceS (k) l , we want to show that under temporal precedence A2, bounded lagged effects A5, causal suïŹiciency A 1, Oracle Models A4the Markov Boundary of label Y j can be identified in the causal graphG. Letâs defineM L j as the estimated Markov Boundary ofY j after observingLevents. Base CaseL=1: Consider the BN for stepL= 1following the Markov assumption [115] with two nodesX 0 , Y j . Using Tf x ,Tf y as Oracle Models A4, we can express the conditional probabilities for any nodeU: P(U|Pa G (U)) = ( P(X 0 ) =P Ξ x (X 0 |[CLS])ifUâX P(Y j |X 0 ) =P Ξ y (Y j |X 0 )otherwise (6.12) whereP(.|[CLS])is considered the marginal distributionP(.). Assuming that P is faithful (Def.19) toG, no hidden confounders bias the estimate (A1) and temporal precedence (A2), we can estimate the CMI6.8such that iffI(X 0 , Y j |â )>0âY j ̞℠Ξ x ,Ξ y X 1 =âY j Ìžâ„ G X 1 (Lemma 8). Since we assume temporal precedence A2, we can orient the edge such thatX 0 must be a parent ofY j inG. Using Lemma7, we know that Pa G (Y j ) =MB(Y j ) =âX 1 â MB ( Y j ) , thus we must include X 0 in M 1 j , otherwise not. HeredityForL=i, we obtainedM i j with the BN up to stepL=i. Now forL=i+ 1, the BN hasi+2nodes denoted asU âČ = (X 0 ,·, X i , X i+1 , Y j ). Using the Oracle Models A 4and following the Markov assumption [115], we can estimate the following conditional probabilities for any nodesUâU âČ : P(U|Pa G (U)) = ( P(Y j |Pa G (Y j )) =P Ξ y (Y j |Pa G (Y j )),ifUâY P(X|Pa G (X)) =P Ξ x (X|Pa G (X)),otherwise. (6.13) By bounded lagged effects (A 5) we know that the causal influence of pastX â€i onY j has expired. In addition, no hidden confounders (A1) bias the independence testing. Finally, using Eq.6.8, we can estimate the CMI such that iffI(Y j , X i+1 |Z)>0â Y j ̞℠Ξ x ,Ξ y X i+1 |Z=âY j Ìžâ„ G X i+1 |Z(Lemma 8). Since we assume temporal precedence A2, we can orient the edge so thatX i+1 must be a parent ofY j inG. Using Lemma7, we know that Pa G (Y j ) =MB(Y j ) =âX i+1 â MB(Y j ). Thus,X i+1 âM i+1 j which represent theMB(Y j )for stepi+ 1. Finally,M i+1 j still recovers the Markov Boundary ofY j such that âUâU âČ âM i+1 j , Y j â„U|M i+1 j 76 6.3 Methodology Intuitively, Theorem1states that if our Transformers perfectly approximate the true joint distribution, then testing conditional mutual information at each step is suïŹicient to recover the Markov Boundary of each label sequentially. By induction, we prove that with bounded lagged effects of the previous events, we can restrict their causal influence and recover the correctMBof each label in the associated Bayesian Network (Fig.6.2). 6.3.6 Computation ContextFollowing the EPredictor architecture seen in Chapter3, to ensure stable con- ditional entropy estimates and reliable predictions from Tf y , the CMI is computed after observingcevents (context). This design choice also enables out-of-the-box paralleliza- tion. By samplingNvariations of the prefix sequenceS â€c , the CMI is independently computed across positionsiâ[c, L]. One caveat is the phenomenon of entropy satura- tion [130], whereby the conditional entropyH(Y j |Z i )diminishes asZ i =S â€i grows longer. Recall the Lemma3: H(Y j |X i+1 ,Z i )â€H(Y j |X i ,Z iâ1 ). In other words, once a suïŹiciently informative context is observed, future uncertainty becomes minimal. Therefore, context c and sequence length L must be carefully selected to balance informativeness and computational eïŹiciency. In our experiments, we set c= 15andL= 128. An ablation oncand the quality of the NADEs can be found in the Appendix B.1.3. ParallelizationA key advantage of our approach is its scalability. Fig.6.1shows all parallelized steps on GPUs. Unlike traditional methods whose complexity depends on the event and label cardinality|X|and|Y|[ 77],ourmethodisagnostictoboth. CMI estimations are independently performed for all positionsiâ[c, L], with the sampling pushed into the batch dimension and results averaged across particles using Eq.6.9. This reduces the per-sample computational overhead via batch parallelism fromO(BSĂNĂL) sequential forward passes to a single batched forward pass asO(1). A simple PyTorch [ 111] implementation of OSCAR and a top-k/p sampling is given in AppendixB.1.7. The main computation can be summarized through these simple steps: 1logits_x = tfx(**batch)['prediction_logits'] 2x_hat = F.softmax(logits_x, dim=-1) 3sampled = topk_p_sampling(batch['input_ids'], x_hat, c=c, n=N) Listing 6.1:Step 1: Next-event prediction and sampling. The next-event Transformertfxproduces logits over next event types. We apply top- k/nucleus sampling to expand the batch intoNcandidates in parallel. Only the firstc events are sampled. 77 6 Sample-Level Multi-Label Causal Discovery 1out_y = tfy(input_ids=sampled.reshape(-1,L), attention_mask=attention_mask.repeat( N, 1)) 2prob_y = torch.sigmoid(out_y['logits']).reshape(bs, N, L-c, -1) 3prob_y = torch.clamp(prob_y, eps, 1-eps) Listing 6.2:Step 2: Next-label prediction. The label Transformertfyevaluates all samples in one forward pass starting fromc, yielding conditional probabilitiesP(Y j |Z)andP(Y j |X i ,Z). We then calculate the bi- naryD KL : 1y_z, y_zx = prob_y[...,:-1,:], prob_y[...,1:,:] 2cmi = torch.mean( 3y_zx*torch.log(y_zx/y_z) + 4(1-y_zx)*torch.log((1-y_zx)/(1-y_z)), 5dim=1 6) # (bs, L, |Y|) Listing 6.3:Step 3: Conditional mutual information estimation. Conditional mutual information is averaged across the sampling dimension, producing a compact(bs, Lâc,|Y|)tensor: 1mu, std = cmi.mean(dim=1), cmi.std(dim=1) 2mask = cmi >= (mu + k*std).unsqueeze(1) Listing 6.4:Step 4: Dynamic thresholding. Finally, dynamic per-label thresholds identify causal events by evaluating their values across the sequence length dimension. 6.4 Causal Indicator While deterministic DAGs reveal structural dependencies, they often obscure themag- nitudeanddirectionof influence between variables. In many settings, a small subset of causal events may exert disproportionate influence on the probability of a label. Also, causal relationships can be eitherexcitatoryorinhibitory. That is, the presence of a cause may either increase or decrease the likelihood of its effect. For instance, ifP(Y j |X i ,Z)< P(Y j |Z)thenX i negatively influencesY j , yet still constitutes a valid causal relationship [ 114]. Without quantifying the effect direction and strength, such cases may mislead an operator. Given that we can estimate both conditionalsP(Y j |X i ,Z)andP(Y j |Z), we define thecausal indicatorC â[â1,1] between an eventX i and a labelY j under causal suïŹiciency and the contextZthat we assume fixed for every measurement [ 35]: C(Y j , X i ;Z) :=E p(z) [P(Y j = 1|X i = 1,Z=z)âP(Y j = 1|X i = 0,Z=z)](6.14) 78 6.5 Empirical Evaluation where the expectation is taken over contextsz (l) sampled from the learned model dis- tributionP Ξ using the same Monte Carlo estimation as Eq.6.9. We compute the mean and standard deviations over contextsz (l) to provide uncertainty estimates. Eq.6.14 align with the average causal effect (ACE) [114] iff the conditioningZfully blocks all unwanted backdoor paths A1. Here,C>0indicates a positive influence andC<0 reflects an inhibitory effect. While several metrics for causal strength existâincluding Causal Power [19] and Goodâs measure [35], we adopt this measure for its simplicity of interpretation. An operator can easily read it and get a sense of the rise in likelihood of the labelY j . We employ the term causalindicatorto distinguish it from causal strength measures, which, if using this formulation, can be problematic as pointed out by [68]. Ours serves more as an indication than a strength, which is here the conditional mutual information. 6.5 Empirical Evaluation 6.5.1 Settings We used ag4dn.12xlargeinstance from AWS Sagemaker to run comparisons. It contains 48 vCPUs and 4 NVIDIA T4 GPUs. During inference, we used fp16 for Tf y and fp32 for Tf x . We used a combination of F1-Score, Precision, and Recall with different averaging [172] to perform the comparisons. The code for OSCAR, Tf x ,Tf y and the evaluation are provided 2 as well as the anonymized version of the dataset for reproducibility purposes. 6.5.2 Vehicle Event Sequences Dataset We evaluated our method on an anonymized real-world vehicular test set ofn= 300,000 sequences. It contains|Y|= 474different error patterns and about|X|= 29,100different DTCs forming sequences ofâ150±90events. We used 105m parameters backbones as Tf x ,Tf y . The two NADEs did not see the test set during training. We set the elements of the EP rules (as defined in 2.4.1.2, Eq.2.20) as the correct Markov Boundary for each labely j in the tested sequences. Importantly, the rules are subject to change over time by automotive engineers, contain some noise, and are biased, which makes it more diïŹicult to extract the trueMB. Further, there is about12%missing ground truthMB rules for certainY j . 6.5.3 Comparisons Although no existing method directly targets one-shot multi-label causal discovery [41], we benchmark OSCAR against local structure learning (LSL) algorithms that estimate global Markov Boundaries. This includes established approaches such as CMB [ 37], MB- by-MB [152], PCD-by-PCD [165], IAMB [148] from thePyCausalFSpackage [166], as well as the more recent, state-of-the-art MI-MCF [85]. 9 random folds of the test data were created and converted into a multi-one-hot data-frame where one row represents 2 https://github.com/Mathugo/OSCAR-One-Shot-Causal-AutoRegressive-discovery 79 6 Sample-Level Multi-Label Causal Discovery one sequence, and each column represents an event type or label(X,Y). We set the target nodes as the labels withPyCausalFS. 6.5.4 Results We first drewn= 50,000random sequences from our dataset and performed comparisons (Table6.1). We observed that even under this reduced setup, LSL algorithms failed to compute the Markov Boundaries within a 3-day timeout, far exceeding practical limits for deployment. OSCAR, on the other hand, shows robust classification over a large amount of events(29,100), especially55%precision, in a matter of minutes. This behavior highlights the current infeasibility of multi-label causal discovery in massive discrete event sequences. This positions OSCAR as a more feasible approach for large- scale sample-level causal discovery in production environments. To enable at least partial comparison, we further sub-sampled ton= 500sequences (Table6.2) to enable a faster computation. However, the number of labels in the test set is approximately the same forn= 500samples. This resulted in a poor number of CI tests for the baselines. As a result, LSL algorithms output emptyMBsets after multiple hours. Especially MI-MCF with even500samples suffers from its expensive CMI testing. Thus, traditional algorithms suffer from having either too many samples and taking days to compute or too little data to even function. Algorithm PrecisionâRecallâF1âRunning Time (min)â IAMB--->4320 CMB--->4320 MB-by-MB--->4320 PCDbyPCD--->4320 MI-MCF--->4320 OSCAR55.26±1.42 31.37±0.82 40.02±1.0311.7 Table 6.1:Comparisons ofMBretrieval withn= 50,000samples,|X|= 29,100,|Y|= 474 averaged over6âfolds. Classification metrics averaging is âweightedâ and shown as one-shot for OSCAR. The symbol â-â indicates that the algorithm did not output the MBwithin 3 days. Metrics are given in%. Algorithm PrecisionâRecallâF1âRunning Time (min)â IAMB0.0±0.00.0±0.00.0±0.0129.4 CMB0.0±0.00.0±0.00.0±0.0128.7 PCDbyPCD0.0±0.00.0±0.00.0±0.0129.1 MB-by-MB0.0±0.00.0±0.00.0±0.0140.3 MI-MCF0.0±0.00.0±0.00.0±0.0>1440 OSCAR54.78±2.91 30.39±2.39 39.92±2.250.14 Table 6.2:Comparisons ofMBretrieval withn= 500samples over9folds. 80 6.6 Limitations We illustrate the explainability provided by our method for the task of explaining error patterns observed in a vehicle (Fig.1.3). A concrete use case for OSCAR in this context is to refine or develop new error-pattern rules based on OSCAR output predictions, such as non-common causal variables [159] between labels, thereby improving the automation of quality processes. 6.6 Limitations Our approach relies on several assumptions that enable sample-level causal discovery under practical and computational constraints. Temporal PrecedenceTemporal precedence (A2) simplifies causal directionality. How- ever, this relies heavily on precise event time-stamping. Even though we only test X i âY j , this could falsify the conditioning testZand introduce suprious relation- ships. Bounded Lagged EffectsThe bounded lagged effects (A5) assumption enables us to restrict causal influence and recover theMBof each label using Theorem1. In most real- world sequences where relevant history is limited, this holds empirically. Nonetheless, in highly delayed causal chains, like financial transactions, some influences may be missed. Causal SuïŹiciencyAs with many causal discovery approaches, we assume all relevant variables are observed (A 1). It constitutes a mild assumption, especially in sample-level causal discovery where fewer data points are observed. Inter-label EffectsBy definition, the labels are explained solely by events. While simplifying causal discovery, this assumption could be relaxed in future work by using thedooperator [ 114] to perform interventions on common causal variables of multiple labels. For example, our current framework estimates the Markov Boundaries for each label independently. However, inter-label dependencies can exist, particularly when labels share overlapping Markov Boundaries (e.g M B 1 = [ X 1 , X 3 ] , M B 2 = [ X 1 , X 2 ] . We aim to investigate a âPhase 2â for OSCAR, focusing on inter-label dependencies through simulated interventions. For instance, if we consider a sequence of two labels Y 1 , Y 2 with the MB above, we could perform counterfactual interventions by applying do(X 1 = 0), do(X 3 = 0). We then compute the average change in the likelihood ofY 1 , which, if non-zero, indicates a dependence betweenY 1 andY 2 . [ 159] points out that the assumptions of these inter-label dependencies are already anchored in the Markov Boundaries; we do the same in this chapter. NADEsUsing NADEs allows for relaxation of structural assumption about the under- lying data-generating processes (e.g., Poisson Processes or SCMs). As demonstrated by the Ablations B.1.3, the effectiveness of OSCAR hinges on the capacity of Tf x and Tf y to 81 6 Sample-Level Multi-Label Causal Discovery approximate true conditional probabilities (A4) and provide Oracle CI-test. While as- suming Oracle tests is common in the literature [161,77] and necessary to recover faithful causal structures, this remains a strong assumption. It is valid only to the extent that the models are perfectly trained. In multi-label classification settings, performance may degrade in underrepresented regions of the data distribution. For example, we analyze on a reduced dataset, the performance of OSCAR as a function of theMBlength: 10 0 10 1 10 2 Markov Boundary Length 0 10 20 30 40 50 Recall % 10 0 10 1 10 2 Markov Boundary Length 0 20 40 60 80 Precision % Macro Micro Weighted 10 0 10 1 10 2 Markov Boundary Length 0 10 20 30 40 50 F1 Score % Number of Samples Imbalance threshold 10 1 10 2 10 3 10 4 Number of Samples Figure 6.3:Evolution of the Classification Performance as a Function of the Markov Boundary Length|MB(Y j )|(n= 45969samples). We can identify that the Markov Boundary complexity is not the bottleneck but rather the number of sam- ples per class (imbalance threshold), which reduces classification performance. Fig.6.3reveals the classification performance depending on the number of events in the ground truthMB. On the same plot is drawn ingreythe number of samples that each MBlength contains (to account for imbalance). We observe that generally, a bigger |MB(Y j )|does not imply a reduction in performance, highlighting the capability of OS- CAR to retrieve complex Markov Boundaries in high-dimensional data. However, we observe that past acertain number of samples (imbalance thresholdin redâ7Ă10 2 samples), the classification metrics are directly correlated with the number of samples per|MB(Y j )|. This indicates that Tf x ,Tf y struggle to output proper conditional prob- abilities for rare classes, this results in a non faithful CI-test. Therefore, when using OSCAR and more generally assumption A 4, one should carefully assess class imbalance in the pretraining phase. 6.7 Summary OSCAR extends autoregressive models toward causal discovery by repurposing them as NADEs. The proposed methodology combines theoretical guarantees with paral- lelized computation on GPUs. Under assumptions, such as temporal precedence, causal suïŹiciency, and bounded lagged effects, OSCAR can reliably recover the local causal structure underlying each observed sequence, as we have shown. Thanks to its paral- lelized implementation on GPUs, the approach scales seamlessly to tens of thousands of 82 6.8 Outlook event types and hundreds of labels, achieving causal discovery for50,000of sequences in minutes where traditional constraint-based methods fail to compute within days. By introducing acausal indicator, OSCAR further quantifies the direction and magnitude of influence between events and outcomes, making causal discovery not only feasible but also interpretable for practitioners. Applied to a large-scale vehicular dataset, OSCAR demonstrated the ability to uncover the correct EP rules. However, we highlighted the limitations of the assumptions, including the impact of under-training of EPredictor on rare classes, which can, in practice, break the Oracle assumption. Nevertheless, this con- stitutes an important conceptual change in our overall frameworkâfrom predictingwhen andwhatfaults will occur, to understandingwhythey arise. Such causal interpretability is essential in many industrial problems, in fault diagnostics but also in production lines where defects arrive sequentially. 6.8 Outlook Chapter 7 addresses a distinct challenge: while OSCAR recovers causal structure within a single observed sequence, practitioners require a consensus causal graph that is stable across the full vehicle fleet. To this end, Chapter7introduces CARGO, a graph aggre- gation method that uses the extracted per-sample causal graph of OSCAR to provide a global population-level structure. 83 7 Population-Level Multi-Label Causal Discovery It is essential in many real-world systems to recover the global set of causes for each outcome (e.g., diseases, manufacturing defects) across the entire available data. We aim to recover all the causes of each error pattern to potentially suggest new rules or automate the rules of unknown error patterns. This chapter is based on the following publication: Towards Practical Multi-label Causal Discovery in High-Dimensional Event Sequences via One-Shot Graph Aggregation. [91] Hugo Math, Rainer Lienhart.NeurIPS 2025 Workshop on Structured Probabilistic Inference & Generative Modeling, San Diego, USA, December 2025. 7.1 Introduction Although the local view of causal relationships enables interpretable causal reasoning within individual sequences, real-world systems such as fleets of vehicles, patient pop- ulations, or distributed networks often share underlying causal mechanisms that recur across many realizations. Understanding these global causal dependencies is essential to building coherent diagnostic or decision-making frameworks that generalize beyond a single observation. It remains currently unknown how to perform such population-level causal discovery across industrial-scale event sequences [ 86,123,57,84,117] as standard methods are computationally intractable. To address this, we reinterpret multi-label causal discovery for event sequences as a form of Bayesian model averaging [ 59,115], where each sequence is treated as a sample from a local causal model. Specifically, each sequence induces a sample-level causal graph (i.e., a directed acyclic graph (DAG)) [94] inferred fromOSCAR. Together, they are fused to form a unified global structure [28]. This process, known as structural fusion [113], aggregates local graphs into a consensus causal graph over all observed samples. For this purpose, we introduce CARGO: (CausalAggregation viaRobustGraphOperations), a scalable two-phase causal discovery framework. 7.2 Related Work Bayesian Model AveragingFusing BNs has two primary applications: averaging mod- els from different experts to learn a global representation [ 59], or performing causal discovery in distributed settings with federated learning algorithms [154,47]. 85 7 Population-Level Multi-Label Causal Discovery Formally, given a set of Bayesian NetworksB k m k=1 with associated DAGsG (k) = (V k , E k ) m k=1 ,V k âUsharing the same finite set of nodeU, structural fusion aims to construct the DAGG â = (V, E), VâU. Multiple fusion methods exist and leverage either the probability distributionPby doing Bayesian Model Averaging [59] or focus on the structural learning (Fig.7.1) ofG â [127,113,40,118,47], i.e., only merging the edgesE k m k=1 to the correct nodesV k m k=1 . Bayesian Model Averaging is considered to be an NP-hard [113] problem. We focus on the structural learning and seek the merged edgesE= S m i=1 E Ï i . The consistent node orderingÏensures acyclicity. The fused DAG G â is the minimal I-map (Def.18) of the intersection of the conditional independencies across all DAGsG (k) = (V k , E k ) m k=1 . X 1 X 2 Y 1 X 2 X 3 Y 2 X 1 X 3 Y 2 Structural Fusion X 1 X 2 X 3 Y 1 Y 2 Figure 7.1:Structural Fusion Example: individual set of causal graphsG (k) (left) ag- gregated into a fused DAG for multi-label event sequences (right) using a simple union. Greedy Equivalence SearchGreedy equivalence search (GES) [21] is one of the most theoretically sound methods to recover a Markov equivalence class (MEC: Def. 20) of a DAG. Particularly in the context of infinite samples, GES provides theoretical guarantees of reaching the true graph. Formally, GES searches for the MEC of the graphG â from the observational datasetDwith distributionPusing a scoring functionf s . This defines the optimization problem as: G â = arg G maxf s (G,D)(7.1) [21] proved that, under parametric assumption, a large number of samples and using the Bayesian Information Criterion (BIC) as criterionf s , GES is guaranteed to recover an MEC ofG â since it is score equivalent (Def. 28) and locally consistent (Def.29). Some contemporary works point toward decomposing classical causal discovery for high- dimensional datasets into a subproblems (divide-and-conquerapproaches). [72] introduce a ring-based distributed algorithm for learning high-dimensional BN, [28] explores a dis- tributed approach for large-scale causal structure learning and [ 98] for Markov Bound- aries. 86 7.3 Methodology Frequency-BasedMultiple heuristics have been developed to merge multiple BNs. One uses an edge frequency cutoff [142], another relies on estimating the proportion of false positive edges and integer linear programming (ILP) [36] or a mix of both: [146]. As pointed out by [36], choosing a frequency cutoffÏis particularly challenging. Moreover, under class imbalance and long-tail problem in labeled datasets [170], the theoretical property of frequency approaches, such as the law of large numbers, might not hold. 7.3 Methodology We now explain the two phases of CARGO, which are (1) the sample-level causal dis- covery (2) graph aggregation using adaptive thresholding. 7.3.1 Structural Fusion of Markov Boundaries LetG (k) = (V k , E k ) m k=1 be the set of generated DAGs by the Phase 1 from the dataset Dcontainingmi.i.d multi-labeled sequencesS (k) l m k=1 drawn from a joint distribution P(X, Y). Each graphG (k) represents local Markov Boundaries identified within a multi- labeled sequenceS (k) l . Our objective is to fuse these local graphs into a single, global consensus graphG â = (U, E)(see Fig.7.1) with the events always as parents of labels Y j such as: Pa G (Y j )âX 0 ,·, X n Time P(Yi|S') S' P(Yi+1|S) S Minimum context c ... ... Z_i^e Next Multi-label Neural Autoregressive Density Estimator: Tf_y Next Event Neural Autoregressive Density Estimator: Tf_e Time P(Xi+1|S) ... Event Type Labels Zi^y P(Xi+1|S) y hat _i x hat _i (1) Extract Conditional probabilities for sequence with label Y3 (2) Compute the CMI I(X_i, Y_i+1|Z) by sampling from Tf_e If I(X_i, Y_i+1, 3|Z) > theta, X is a cause of Y1 (3) Draw an edge from X_i to Y1 Xi Xi+1 Xc Xi-1 ... Y1 Y2 ... (4) Compare MBs and use back-door criterion to get inter- label edges Yn MB1 MB2 (backbone) Time ... Event Type Extract Conditional Probabilities Hidden States Next-Event Predictor: Next-Label Predictor: Time ... Event Type Sampling Parallel independance testing using CMI Draw an edge from if CMI Extract Probabilities from Event Sequence Next-Event Predictor: Next-Label Predictor: Time time ... Event Type 1) Infer next events: Infer Compute CI-test Draw an edge from if CMI ) context 3) Infer hidden states 2) top-k + nucleus sampling 4) Infer next labels 5) CMI Thresholding on each label Time Event Type sample times Hidden states Compute and One Shot Causal Discovery Backward Equivalence Search Maximise a criterion by removing edges Get the optimal Graph Generate EP Rules with Causal Strength and uncertainty measures Unknowns EP (backbone) Sample times Hidden states Compute and access Apply the adaptive threshold to the frequencies of Get one- shot graphs Fit long-tail support distribution of labels to obtain threshold function One-Shot (Phase 1) Graph aggregation (Phase 2) Batch of labeled sequences Figure 7.2:Overview of CARGO. Phase 1 (one-shot [94] or sample-level) is on top, and Phase 2 (Adaptive Thresholding) is on the bottom.ddenotes the hidden dimension, Lthe sequence length,mthe number of samples andMB 1 ,MB 2 the Markov Boundary ofY 1 , Y 2 . All green and blue areas are parallelized. 87 7 Population-Level Multi-Label Causal Discovery A naive fusion approach, such as taking the simple union of all edgesE= S m k=1 E Ï k works if and only if the Oracle models (A4) yield faithful CI-tests in Phase 1 such that the local graphsG (k) are faithful toP(X, Y)([113], Theorem 4). Here, the orderingÏdoes not matter since we are dealing with Markov Boundaries. Thus,G â is naturally a DAG because we considered previously that outcome labels are solely explained by events, which simplifies acyclicity. We define the Bernoulli variableB k i,j for each potential edge (X i âY j )within each sequenceS (k) l : B k i,j = ( 1if the edgeX i âY j is present inG (k) 0otherwise Under the Oracle Models assumption (A 4), Phase 1 acts as a perfect conditional inde- pendence tester. Consequently, the detection of an edge in a local graphG (k) corresponds precisely to a true causal dependency in the global graphG â . The probability of this event P(B k i,j = 1)is therefore the true marginal probability of the edgeâs existence, which we denote asÏ i,j . The empirical frequency,ËÏ i,j (m), of the edge(X i âY j )after observing m sequences is the sample mean of these i.i.d Bernoulli variables Ë Ï i,j ( m ) = 1 m P m k=1 B k ij . By the Law of Large Numbers (LLN), as the number of i.i.d sequencesmtends to in- finity, the empirical frequency converges in probability to the true expected value of the random variable: ËÏ i,j (m) p ââE[B (k) i,j ] =Ï i,j Given a suïŹiciently large number of sequences, the empirical frequencyËÏ i,j (m)serves as a consistent estimator for the true probability of the edgeâs existence in the global DAG G â . 7.3.2 Aggregation under Imperfect CI-tests The assumption of an Oracle CI-tester, while necessary for initial theoretical guaran- tees and common in the literature [98], is invariably violated in practice due to fac- tors like model capacity, limited data or class imbalance as seen in the experiments of Chapter 6. The extracted sample-level graphs will most likely violate the indepen- dencies inG (k) and thus inG â . Let us model the performance of our one-shot CI-test for any potential edgeX i âY j with the following error rates: (1) False Positive Rate (Type I Error):α=P(detect|edge is spurious) (2) True Positive Rate (Sensitivity): 1âÎČ=P(detect|edge is causal). We operate under the reasonable assumption that our classifier is significantly better than random, implying that1âÎČâ«Î±. The expected value of our Bernoulli variableB k i,j is now: E[B k ij ] =P(B k ij = 1) =P(detect|causal)P(causal) +P(detect|spurious)P(spurious) = (1âÎČ)Ï ij +α(1âÏ ij ) The empirical frequency now converges to this new expectation. For a true edge(Ï i,j = 1), the empirical frequency converges to a high value:ËÏ ij (m) p ââ1âÎČand for spurious 88 7.3 Methodology edge(Ï i,j = 0)it converges to a low value:ËÏ ij (m) p ââα. This reveals the critical role of frequency aggregation as a mechanism for separating signal from noise. 7.3.3 Adaptive Fusion for Structural Discovery in Long-Tail Distributions A primary challenge in real-world data is the long-tail distribution of outcome labels, where a few âheadâ labels possess abundant data while the vast majority of âtailâ labels are data-sparse [170]. 0500010000150002000025000 Sample Count (m j ) 0.1 0.2 0.3 0.4 0.5 Threshold ( m j ) Evolution of Adaptive Threshold with Long-Tail Sample Distribution(n= 300000) Adaptive Threshold Sample Count Distribution 0 20 40 60 80 100 Frequency of Labels Figure 7.3:Adaptive Thresholding FunctionÏ j (m j )across varying supportm j , illustrating the logistic decay fromÏ max toÏ min . To resolve this, we introduce anadaptive thresholdingstrategy (Fig.7.3) that tailors the edge inclusion criterion to the statistical power available for each label. We define a label-specific thresholdÏ j ,as a logistic decay function of its sample supportm j : Ï j (m j ) = (Ï max âÏ min )· 1 1 +e k(logm j âlogm 0 ) +Ï min (7.2) This function smoothly interpolates between a user-defined maximum threshold,Ï max (prioritizing precision for the tail), and a minimum,Ï min (prioritizing recall for the head). The functionâs behavior is calibrated by the data distribution. Such that decay midpointm 0 is set to the median of all label supports, providing a robust anchor point against skew. The decay ratekis made inversely proportional to the log-inter-quartile range of supports such that: k= 2 log 3 logq 75 âlogq 25 (7.3) Consequently, for rare labels with smallm j , the high variance of the frequency estimate necessitates a high thresholdÏ j (m j )that acts as a strong regularizer. For common labels with largem j , the LLN guarantees the convergence ofËÏ i,j to the true edge probability, justifying a lower threshold to capture a more complete causal structure. 89 7 Population-Level Multi-Label Causal Discovery 7.4 Empirical Evaluation The empirical evaluation of CARGO follows the same experimental setting (baselines and dataset configuration) as introduced in Chapter6. The implementation is included in AppendixB.2.6. This task is challenging, as it involves handling cumulative noise across sequences from the imperfect CI-tests, managing severe label imbalance, and ensuring that aggregated structures remain both interpretable and statistically valid. 7.4.1 Benchmark Against Multi-Label Causal Discovery Methods ResultsTable7.1shows the performance of the multi-label causal discovery algorithms, the sample-level performance of OSCAR and the global Markov boundary retrieval of CARGO. Since the evaluation setup is the same, all baseline LSL algorithms failed to compute complete Markov Boundaries within a three-day timeout, as demonstrated in Chapter6. We observe that, even though the imperfect conditional independence tests introduce cumulative errors and the classification task becomes considerably more chal- lenging since, in the global setting, the model aims to recover the underlying global causal rules rather than the more straightforward local rules identifiable within a single sequence, CARGO still achieves a significant performance improvement. It surpasses OS- CAR by a clear margin across all metrics, e.g., the Precision is (60.6±1.5vs.55.26±1.42) for the sample-level performance, confirming the benefit of tailored graph aggregation for large-scale causal discovery. 7.4.2 Ablation on Aggregation Criteria for Phase 2 CriteriaWe provide an Ablation of the different criteria used in the structural fusion of Markov Boundaries.Unionstands for a simple union over all edges without any removal. Frequencyor edge voting, counts how often isX i â MB(Y j ). Then apply a static frequency thresholdÏ= [0.05,0.25,0.5,0.8].MIuses the mutual information between events and labels as a criterion in a Background Equivalence Search [ 21].Expected AlgorithmPrecisionâRecallâF1âRunning Time (min)â IAMB--->4320 CMB--->4320 MB-by-MB--->4320 PCDbyPCD--->4320 MI-MCF--->4320 OSCAR (sample-level)55.3±1.4 31.3±0.8 40.0±1.011.7 CARGO (population-level) 60 . 6 ± 1 . 5 45 . 8 ± 1 . 7 45 . 8 ± 1 . 2 11 . 8 Table 7.1:Comparisons ofMBretrieval withm= 50,000samples averaged over6âfolds with |Y|= 474,|X|= 29,100nodes. Averaging is âweightedâ. The symbol â-â indicates that the algorithm did not output theMBswithin 3 days. Metrics are given in%. OSCAR reports the sample-level performance (Phase 1). 90 7.4 Empirical Evaluation FPR(false positive ratio) [36] describes two beta distributions which are fitted using the distribution of the mutual informationI(X i , Y j )extracted from Phase 1. The lower tail is used for outlier detection. Different FPR are chosenÎČ= [0.01,0.05,0.15,0.2]. Detailed definitions can be found in AppendixB.2.2. ResultsFigure7.4illustrates the impact of aggregation choices during Phase 2. A naive Unionmaximizes recall (84%for weighted) but suffers from poor precision. Optimizing a local scoring criterion (BES mi) did not enhance the performance over a basicUnion. In contrast, fitting Beta distributions to detect outliers using their mutual information appears to perform better; hence,frequency betaoutperforms alternatives, particularly in terms of lower FPR. Experimental results for frequency approaches with static thresholds 0 20 40 60 80 Macro Average Precision (%)Recall (%)F1 Score (%) 0100000200000300000 Number of Samples 0 20 40 60 80 Weighted Average 0100000200000300000 Number of Samples 0100000200000300000 Number of Samples BES_mi_imbalance_par adaptive_thresholding (ours) frequency_0.05 frequency_0.25 frequency_0.5 frequency_0.8 frequency_beta_0.01 frequency_beta_0.05 frequency_beta_0.15 frequency_beta_0.2 union Figure 7.4:Criterion Ablation for Structural Fusion. Comparison of different criteria for the structural fusion (Phase 2) as a function of the number of samplesm. With |Y|= 474,|X|= 29,100nodes. 91 7 Population-Level Multi-Label Causal Discovery corroborate the analysis in Section7.3.2. Specifically, we observe a trade-off dictated by class support: strict cut-offs (Ïâ[0.5,0.8]) yield the lowest weighted F1-scores, as they penalize classes with large support (m j ). Conversely, lowering the threshold (Ïâ [0.05,0.25]) leads to a substantial gain in weighted precision (+40%), but at the expense of macro-averaged metrics (â20%precision), as rare classes become overwhelmed by noise. Our adaptive thresholding criterion outperformed all baselines by applying conserva- tive thresholds to small supports and lenient thresholds to large supports. Consequently, it achieves state-of-the-art performance across all metrics, ranking first with F1-scores of44.88%(weighted) and40.9%(macro), and precision scores of62.8%and66.1%, respectively. 7.5 Summary CARGO completes the population-level events-to-outcome causal discovery cell (Ta- ble1.1). Moving from sample-level to population-level discovery significantly increases the taskâs complexity, as it requires the model to recover generalizable causal mecha- nisms rather than sequence-specific associations. Yet CARGO achieves strong perfor- mance (60.6%precision), especially across a remarkable order of magnitude of10 4 event types and10 2 labels, demonstrating both robustness and scalability in real-world auto- motive diagnostics. It demonstrates that complex causal dependencies can be eïŹiciently inferred from noisy and imbalanced data. Under the temporal assumptions, large sam- ples, faithfulness, and a perfect CI-test, CARGO recovers the set of Markov Boundaries. However, we highlighted the practical limitations, in particular long-tail distributions, which will most likely introduce spurious edges due to the imperfect CI tests. 7.6 Outlook OSCAR and CARGO together form a complete pipeline for multi-label causal discovery, identifying, for each outcome label, the set of events that causally drive it. Nevertheless, both methods share an important scope restriction: they model events-to-outcome de- pendencies, treating the causal structure between events themselves as opaque. In many diagnostic scenarios, however, understanding how one DTC causally triggers another is equally essential for root-cause analysis and cascading failure diagnosis, e.g., why does DTC A tend to precede DTC B? This limitation motivates a change in problem formula- tion: rather than asking which events cause a given label, we now ask how events cause each other. Chapter 8addresses this more general challenge with TRACE, a framework for scalable sample-level event-to-event causal discovery in discrete sequences. 92 8 Sample-Level Event-to-Event Causal Discovery A complete diagnostic picture requires understanding how events causally influence each other, the event-to-event setting tackled in this chapter. In this setting, it becomes possible to identify root-causes, mediating effects and recover the global causal graphs. We therefore study causal discovery from a single observed sequence of discrete events generated by a stochastic process, as encountered in vehicle logs, manufacturing systems, or patient trajectories. This regime is particularly challenging due to the absence of repeated samples, high dimensionality, and long-range temporal dependencies of the single observation during inference. This chapter is based on the following contribution: Scalable Sample-Level Causal Discovery in Event Sequences via Autoregressive Den- sity Estimation (under review). [90] Hugo Math, Rainer Lienhart, Under Review to the International Conference on Machine Learning (ICML), Seoul, South Korea, July 2026. 8.1 Introduction Suppose we observe a single realization of a discrete stochastic process, for instance, the symptoms, tests, and disease evolution of a patient [76,57], a manufacturing lineâs tests, or some diagnostic codes generated by a vehicle. Which of these discrete events influences the occurrence of the others? Contemporary causal discovery algorithms for sequences infer relations across multiple parallel streams (e.g., distinct users or sensors). These methods are ill-suited for re- covering causal structure within a single event stream, nor do they scale to the long, noisy, and heterogeneous sequences encountered in modern industrial systems. More- over, sample-level causal discovery (see Chapter 6) poses a challenging problem, as it requires identifying conditional independencies among event types from a single realiza- tion of the process. In the era of large-scale pretraining, Autoregressive Language Models (AR LMs) [ 122, 147] have emerged as powerful density estimators, encoding rich conditional distribu- tions over complex contexts to predict the next token [ 30]. Consequently, amortized causal discovery gained a surge of interest in the recent literature [81,10], where the heavy computational cost is shifted to pretraining a single model that can infer causal structures. We propose to take this a step further by repurposing existing predictive priors rather than training specialized models. This effectively transforms a production forecaster into a causal discovery engine without the need for task-specific retraining. 93 8 Sample-Level Event-to-Event Causal Discovery A. Traditional Setting (e.g., PCMCI, Hawkes) Input: Multivariate Seq. Variables (|X|<100: small) Time ( T ââ ) Iterative Solver (CI Tests / Regression) Output: Global Summary GraphG sum B. Our Setting(TRACE) Session-based Input: Single Event Sequence Variables (|X|>1000:massive) ABAC... s t 0 t 1 t 2 t 3 Time Sequence LengthL Pretrained LM(Ξ) (AR Density Estimation) TRACE (Parallel CMI) AB C Output: Instance Summary GraphG s Figure 8.1:Methodological Shift. (A) Traditional Causal Discovery in Sequences (e.g., PCMCI, Hawkes, Granger) relies on iterative solvers (CI-tests) over long multivariate time series (Tâ â).(B) Our TRACE Approachprocesses a single sequence (e.g., event logs, user interactions, patient trajectories) through a pretrained autoregressive (AR) model as density estimator to compute the Con- ditional Mutual Information (CMI) in parallel, enabling scalable causal discovery over massive vocabularies (|X|>1000). 8.2 Related Work 8.2.1 Causal Discovery in Event Sequences Distinguishing between the multi-stream and single-stream paradigm is key, as they fundamentally address different causal problems (Fig.8.1). We point out the current methodsâ limitations in Table8.1. Multiple Streams (Standard).The most common paradigm considers a long multi- variate time series, each corresponding to an individual entity (e.g., sensor, user, or machine). The goal is to uncover how the occurrence of events in one sequence influ- ences the occurrences in others. To recall the main methods, it is traditionally done via Granger-based [ 44,135,101,174] (TCDF, CAUSE), constraint-based [126] (PCMCI), functional [63] (VARLiNGAM) or optimization-based methods [110] (DYNOTEARS). Modern information-theoretic variants (NPHC [1], CASCADE [24]) and neural point processes (THP [ 180], SHTP [119]). As established in Chapters5and6, these methods are limited to small number of nodes in the resulting graph. 94 8.3 Methodology Method Class Discrete Events High Dim. Non- Param. Lags Instance- Level Linear Complexity Constraint-based (PCMCI, FCI)Ă33Ă Score-based (DYNOTEARS)Ă3Ă Granger (TCDF, CAUSE)Ă3Ă Noise-based (VarLiNGAM)Ă3Ă Hawkes / TPP Models (THP, SHP)3Ă3Ă Info-Theoretic (NPHC, CASCADE)3Ă33Ă TRACE (Ours)333333 Table 8.1:Comparison of causal discovery methods.Discrete Events: Operates on discrete event sequences (e.g., text, logs) rather than multivariate time series.High Dim: Computationally tractable for large vocabularies (|X|>10 3 ).Non-Param.: Ag- nostic to the functional form (e.g., linearity) of causal relationships.Lags: Models delayed causal effects.Sample-Level: Infers a local causal graph specific to a sin- gle sequence, rather than a global graph.Linear Complexity: Complexity scales linearly with the vocabulary size|X|. Single Stream (The Sample-Level Regime).Our setting differs fundamentally (Fig.8.1) as we operate in the âsession-basedâ regime common to NLP and system logs: we observe many short, independent sequences over a massive vocabulary (often with|X|>1000). During inference, only one realization of the process is often available. Here, the goal is to understand if event typeAcauses typeB. This regime is significantly harder due to the sparsity of specific event pairs in high-dimensional vocabularies and the lack of independent trials during inference. While recent works attempt to interpret attention weights in Transformers as causal graphs [ 125], this is heavily criticized for being a poor proxy of causality [11]. 8.3 Methodology An overview of TRACE can be found in Fig.8.2. A key advantage of TRACE is its architectural agnosticism. The framework decouples the density estimation (Phase 1) from causal discovery (Phase 2). Consequently, TRACE can leverage any state-of-the-art autoregressive backbone (e.g., Transformers, Mamba, RNNs). 8.3.1 Data-Generating Process We model thedata-generating process(DGP) as a non-stationary stochastic process 1 X t , tâNtaking values in a finite, discrete alphabetX. We assume that the process forms a high-order Markov chain [70] with transition distributionP(X t |X <t ), where X <t =X 0:tâ1 âX 0 ,·, X tâ1 . This implies that the pastX 0:tâ1 is not independent of the futureX t+1 conditioned on the presentX t . 1 We slightly abuse the notation compared to Chapters6,7by usingX t instead ofX i to denote an event occurrence at time stept. This is more aligned with dynamical systems and stochastic processes literature [ 70]. 95 8 Sample-Level Event-to-Event Causal Discovery Phase 1: Self-Supervised Training s N . . . s 2 s 1 Training Sequences Train AR Model ( Ξ ) (language model, rnn) Phase 2: Sample-Level Causal Discovery (Inference) Input Sequences A B A C AR Model ( Ξ ) A single sequence is observed AR Model ( Ξ ) Use the learned probabilistic dynamics AR Model ( Ξ ) P Ξ Parallelized CMI (CI-Tests) A t 0 B t 1 A t 2 C t 3 Instance Graph G t,s Projection AB C Summary Graph G s Figure 8.2:TRACE Methodology. Phase 1 (Training):An autoregressive (AR) model (e.g., LM, RNN) is pretrained on a corpus of event sequences via next-token predic- tion to learn the process dynamics (P Ξ ).Phase 2 (Inference):A single sequence sis passed through the frozen model. We then estimate conditional mutual in- formation (Parallelized CMImodule) to prune non-causal edges and form the Instance-Time Causal GraphG t,s . Finally, this graph is projected onto the event types to recover theSummary Causal GraphG s . 8.3.2 Sample-Level Causality Our analysis begins at the level of the specific observed sequence. To bridge the gap between the multinomial realizations= (x 0 , . . . , x L )and causal structure, we define the Binary Event ProcessE t L t=0 as E t â1 X t =x t Here,E t is a binary random variable representing the realization of the specific event type observed at timet. This transformation allows us to represent the causal dependencies specific to this sequence as a DAG, which we term theInstance-Time Causal Graph. Definition 22(Instance-Time Causal Graph).Letsbe a realization of the processX t . The Instance-Time Causal GraphG t,s = (T,E t )is a DAG where the nodesT=0, . . . , L correspond to the time steps of the sequence. A directed edge(tâk)âtexists inE t if and only if the realization of the event attâkis a cause of the event att. This graphG t,s (illustrated in Fig.8.2, right) represents the unrolled causal history of the sequences. 8.3.3 Summary-Level Causality Operators are often confronted with observing a single sequence during inference to un- derstand generalizable rules (e.g., âFire causes Smokeâ) rather than specific timestamps (G t,s ). We adapt the terminology from [ 6] regarding summary causal graph (SCG), which treats the unique event types inXas the variables of interest. Therefore, for a single sequences, we aim to recover itsInstance Summary Causal Graph. Definition 23(Instance Summary Causal Graph).LetX s â Xbe the set of unique event types ins. The Instance SCGG s = (X s ,E s )is the surjective projection of the Instance GraphG t,s ontoX s . Specifically, a type-level edgeuâvexists inG s if and only if it appears at least once in the instance graph: uâvâE s ââ ât, ks.t.((tâk)ât)â 96 8.3 Methodology E t â§x tâk =uâ§x t =v Hence, our causal discovery objective is to identify the parents Pa G t,s (t)for each node in the sequence (e.g., detecting Fire@t 1 âSmoke@t 2 ) and aggregate them to reconstruct the instance summary causal graphG s (e.g., FireâSmoke), illustrated in Fig.8.2 (right). Remark 3(Cyclicity in Summary Graphs).Consistent with the standard literature [6], the summary graphG s is an abstraction of the time-unrolled graph and is therefore permitted to contain cycles. 8.3.4 Autoregressive Density Estimation TRACE leverages the same density estimation capability as OSCAR (Chapter6): a pretrained AR model denoted Tf Ξ (CarFormer) estimates the next token probabilities P Ξ . We map it to the binary event process as: P Ξ (E t = 1|X <t =x <t ) = [Softmax(Tf Ξ (x <t ))] x t (8.1) The model is trained on a single event stream rather than multi-labeled sequences (Chap- ter 3). This formulation allows us to utilize the joint distributions learned by the AR model for the structure learning ofG t,s . Therefore, in contrast to recent works probing causal reasoning abilities of AR models (e.g., language models) via prompting [79,69], we focus on extracting causal structure from the learned probabilistic dynamics. 8.3.5 Assumptions In particular,G t,s implies (1) theMarkov assumption[115], such that a variable is conditionally independent of its non-descendants given its parents, and (2)consistency through time[ 6]. Following Chapter6, we assume temporal precedence (A2) and causal suïŹiciency (A 1). Specifically to this chapter, we assume: Assumption 6(ΔâOracle Model).We assume that the AR model Tf Ξ , trained via maximum likelihood estimation on a dataset generated by the true distributionP, has converged such that the Kullback-Leibler divergence is bounded byΔ: D KL (P(X t |X <t )||P Ξ (X t |X <t ))â€Î”(8.2) And that the total variation distanceÎŽ(P, P Ξ )†1 2 . We thus define Tf Ξ as anΔ-Oracle model. By Pinskerâs inequality [23], Eq.8.2implies the total variation distanceÎŽ(P, P Ξ )to be bounded by p Δ/2: D KL (P(X t |X <t )||P Ξ (X t |X <t ))â€Î” =â r 1 2 D KL (P(X t |X <t )||P Ξ (X t |X <t ))†r Δ 2 97 8 Sample-Level Event-to-Event Causal Discovery =âÎŽ(P, P Ξ )†r Δ 2 By Pinskerâs inequality(8.3) WhereÎŽ(P, P Ξ ) = sup A |P(X)âP Ξ (X)|. Essentially, we aim at being more flexible than Chapter6via a possible non-stationary stochastic process and an imperfect density estimator (A6). 8.4 Single Stream Causal Discovery In this section, we describe how we can derive a CI-test to construct the Instance-Time Causal GraphG t,s within single streams. 8.4.1 Conditional Mutual Information In a sequences= (x 0 ,·, x L ), we would like to assess how much additional information the realizationx t (E t = 1)provides about the next event occurrenceX t+1 =x t+1 when we already know the past observationX <t . We essentially try to answer whether: P(E t+1 |E t , X <t ) =P(E t+1 |X <t ) As seen in Chapter5, Information GainI G (Def.13) characterizes the remaining uncer- tainty in E t+1 once we know the realization e t conditioned on x <t : I G (E t+1 , e t |x <t ) =D KL (P(E t+1 |e t , x <t )â„P(E t+1 |x <t )) (8.4) =H(E t+1 |x <t )âH(E t+1 |e t , x <t ) The conditional independence (Def.16) between eventE t and eventE t+1 is assessed with the CMI such as: I(E t+1 , E t |X <t )âH(E t+1 |X <t )âH(E t+1 |E t , X <t ) =E e t ,x <t [I G (E t+1 , e t |x <t )] (8.5) We can deduce the following CI-test: E t+1 Ìžâ„E t |X <t âI(E t+1 , E t |X <t )>0(8.6) Remark 4.We condition on the full trajectoryX <t rather than the coarsened binary historyE <t to test for all observed events in the history, thereby blocking potential back- door paths (confounders) that would otherwise remain hidden in the binary projection. 8.4.2 Estimation and Approximation Error The previous Eq.8.5involves an expectation over the distribution of historiesx <t . Since the true distributionPis unknown, we utilize theΔ-Oracle model Tf Ξ as a proxy to simulateNi.i.d history particlesx (l) <t N l=1 âŒP Ξ (X <t ). We define the empirical estimator Ë I N of the CMI as the Monte Carlo estimation: 98 8.4 Single Stream Causal Discovery Ë I N (E t+1 ;E t |X <t ) = 1 N N X l=1 E e t âŒP Ξ [I G (E t+1 , e t |x (l) <t )] |z f Ξ (x ( l ) <t ) (8.7) Proposition 1(Convergence to theΔ-Proxy).The estimator Ë I N is a consistent estimator of theΔ-Oracle induced CMI denoted asI Ξ . By the Strong Law of Large Numbers, as Nââ: Ë I N a.s. ââ Nâ+â I Ξ (E t+1 ;E t |X <t ) (Proof Sketch)It follows directly from the Strong Law of Large Numbers (SLLN), as the samples are drawn i.i.d. fromP Ξ and the information gain termf Ξ (x (l) <t )is bounded bylog 2, ensuring finite variance. Proof.The particlesx (l) <t are sampled directly from the model Tf Ξ . Letf Ξ (x (l) <t )represents the estimation CMI for a fixed historyx <t andI Ξ the CMI with the approximated distributionP Ξ . Expressing this as a difference of conditional entropies: 0â€E e t âŒP Ξ I G E t+1 , e t |x (l) <t =H Ξ E t+1 |x (l) <t âH Ξ E t+1 |E t , x (l) <t â€H Ξ (E t+1 )â€log 2. (8.8) Thus the posterior variance off Ξ (x (l) <t )satisfiesÏ 2 f âE x <t [f 2 Ξ (x <t )]âI 2 Ξ (f Ξ )<+â[ 29] then the variance of Ë I N (f)is equal tovar( Ë I N (f)) = Ï 2 f N and from the strong law of large numbers: Ë I N a.s. ââ Nâ+â E e t âŒP Ξ ,x <t âŒP Ξ [I G (E t+1 , e t |x <t )]âI Ξ (f Ξ )(8.9) Since we know that the estimator Ë I N converges to the approximated CMI asI Ξ , we derive an approximation error bound to characterize the remaining noise induced by the imperfectΔ-Oracle model. Theorem 2(Total Error Bound in theΔ-Regime).Let Ë I N be the Monte Carlo estimator of the approximated CMII Ξ . Assuming the AR modelP Ξ approximates the true DGPP as anΔ-Oracle model (A 6), the asymptotic error of the true CMIIis bounded by: lim sup Nââ |Iâ Ë I N |â€2 p Δ/2 ln(2) + 2(1 + p Δ/2)h b p Δ/2 1 + p Δ/2 ! whereh b (·)is the binary entropy functionh b (p) =âplnpâ(1âp) ln(1âp). (Proof Sketch)We use the Alicki-Fannes-Winter [157] inequality for the difference between conditional entropies of two distributions with a small total variation distance ÎŽ(P, P Ξ )corresponding to anΔ-Oracle model. 99 8 Sample-Level Event-to-Event Causal Discovery Proof.By definition, the Conditional Mutual Information is the difference of two con- ditional entropies (Eq.8.5): I(E t ;E t âČ |X <t ) =H(E t |X <t )âH(E t |E t âČ , X <t ) Our framework operates in the Teacher Forcing regime (sequence-to-sequence model). We do not sample the historyX <t from the modelâs joint distribution. Instead, we estimate the CMI conditioned on the observed historyx (l) <t . Consequently, the relevant error metric is the per-step conditional divergence at timet, given the fixed history. Letâ =|IâI Ξ |be the CMI estimation error. By the triangle inequality: ââ€|H P (E t |X <t )âH Ξ (E t |X <t )| | z Term A +|H P (E t |E t âČ , X <t )âH Ξ (E t |E t âČ , X <t )| |z Term B We apply the sharp continuity bound for conditional entropy in classical systems ([157], Lemma 2). LetÏandÏbe the true and model distributions respectively. Let ÎŽ=|(P(E t |·)âP Ξ (E t |·)| 1 be the total variation distance. Since the target variableE t is binary (d A = 2), the bound is: |H Ï âH Ï |â€ÎŽln(d A ) + (1 +ÎŽ)h b ÎŽ 1 +ÎŽ From Assumption 6,Ύ†1 2 . Substitutingd A = 2(soln 2in nats since we are using cross-entropy loss in PyTorch [ 111]): |H P (E t |X <t )âH Ξ (E t |X <t )|â€ÎŽln(2) + (1 +ÎŽ)h b ÎŽ 1 +ÎŽ Term B represents the same entropy difference conditioned on an augmented set E t âČ , X <t . Since the target dimensiond A remains 2, the same bound applies. Summing the terms: |IâI Ξ |â€2ÎŽln(2) + 2(1 +ÎŽ)h b ÎŽ 1 +ÎŽ Substituting the Pinsker upper boundÎŽ= p Δ/2and noting that the right-hand side (h b (x)) is monotonically increasing inÎŽ, we obtain the final bound in terms of the oracle score Δ : |IâI Ξ |â€2 p Δ/2 ln(2) + 2(1 + p Δ/2)h b p Δ/2 1 + p Δ/2 ! (8.10) Finally, we decompose the total error into estimation variance and approximation bias using the triangle inequality: | Ë I N âI|=|( Ë I N âI Ξ ) + (I Ξ âI)|†| Ë I N âI Ξ | | z Estimation Error +|I Ξ âI| | z Approximation Bias (8.11) 100 8.4 Single Stream Causal Discovery Given that Ë I N is a consistent estimator of the modelâs internal CMI,I Ξ (Prop.1). By the Strong Law of Large Numbers, Ë I N a.s. ââI Ξ asNââthe stochastic estimation error vanishes, leaving only the irreducible approximation bias: lim sup Nââ | Ë I N âI|â€0 +|IâI Ξ | We thus obtain the final bound in terms of the oracle score as: lim sup Nââ | Ë I N âI|â€2 p Δ/2 ln(2) + 2(1 + p Δ/2)h b p Δ/2 1 + p Δ/2 ! This confirms that minimizing the cross-entropy loss (Δ) directly minimizes the upper bound on structural causal error using the CMI as causal strength. Crucially, this bound implies that as the AR model approaches the true distribution (Δâ0), the causal identification error vanishes. 8.4.3 Identifiability Standard Faithfulness [140] (Def.19) relies on Oracle CI-tests returning exact zeros (for the conditional mutual information), an unrealistic premise under finite samples and imperfect density estimation. We instead adopt a variant of Strong Faithfulness [150], which requires valid causal associations to exceed a detection thresholdÏ(i.e.,I > Ï). Crucially, in our setting, this lower boundÏis by the estimatorâs bias (Theorem2) (if we assumeNâ+â). Definition 24(Δ-Strong Faithfulness).LetÏ Î” be the asymptotic approximation error bound of the model (Theorem2). A distributionPisΔ-Strongly Faithfulto a causal graphGwith respect to the estimatorP Ξ if, for every active edgeE t âE t âČ witht < t âČ , thetrueCMI satisfies: I(E t ;E t âČ |X <t )>2Ï Î” (8.12) Identifiability is guaranteed provided the true causal signal dominates the modelâs ap- proximation error (verified experimentally in Appendix B.3.3.2). Lemma 9(Identifiability of the Instance-Time Causal Graph).Let Ë I N be the consistent Monte Carlo estimator of the CMI derived from theΔ-Oracle modelP Ξ . Under the assumption ofΔ-Strong Faithfulness (Def. 24), the instance-time causal graphG t,s is identifiable with probability 1 asNââ. Proof.Letâ =| Ë I N âI|be the total estimation error. From Theorem2, we have a finite bound corresponding to a noise floorÏ Î” such aslim sup Nââ ââ€Ï Δ . We analyze the two cases for binary classification of the edgeE ij : Case 1: No Edge (H 0 ).If the edge is absent,I= 0. The estimator is bounded by the noise floor: 0â€| Ë I N âI|â€Ï Δ 101 8 Sample-Level Event-to-Event Causal Discovery ââ0â€ Ë I N â€Ï Δ The CI-test is rejected (Correct Rejection). Case 2: Active Edge (H 1 ).If the edge exists, by Definition24,I= 2Ï Î” +Îłfor someÎł >0. We have: | Ë I N âI|= â =â âââ€ Ë I N âIâ€â =âIâââ€ Ë I N â€â +I =â2Ï Î” +ÎłâÏ Î” â€ Ë I N â€â +I =âÏ Î” +Îłâ€ Ë I N â€â +I(8.13) =âÏ Î” < Ë I N (8.14) SinceÎł >0we have Ë I N > Ï Î” . The estimator detects an edge (Correct Detection). Thus the graph is identifiable. 8.4.4 Lagged Effects via Simulated Interventions To evaluate the lagged effects of an eventE t onE t âČ witht < t âČ , we control for the in- termediate events, so-calledmediatorsM=X t+1:t âČ â1 by simulating a Controlled Direct Effect (CDE) [ 114] ofE t onE t âČ . Definition 25(Randomized Interventional Do-Operator).LetM=X t+1:t âČ â1 be the set of intermediate events between causeE t and effectE t âČ . We define the intervention do(MâŒQ)as the expectation over counterfactual realizations sampled from a proposal Q(e.g., Uniform over|X|) and average this effect forNparticlesm (l) as with the Monte Carlo estimation (Eq. 8.7) such as: P(E t âČ |do(MâŒQ), X <t )âE MâŒQ [P(E t âČ |M, X <t )](8.15) â 1 N N X l=1 P(E t âČ |m (l) , X <t ) Remark 5.This effectively marginalizes out the intermediate causal mechanisms only if we assume that there are no hidden confounders (Assumption 1). Using the previous definition, we modify Eq. 8.4to detect lagged information gain fromE t toE t âČ , namelyI L G : Definition 26(Lagged Information Gain).LetE t be the cause,E t âČ be the effect (t < t âČ ) andM=X t+1:t âČ â1 the set of intermediate events. The Lagged Information GainI L G is defined: I L G (E t âČ ;e t |x <t )âD KL P(E t âČ |do(MâŒQ), x <t ) P E t âČ |do(MâŒQ), e t , x <t (8.16) Under the do-operator, conditioning on the realizede t is still admissible since the intervention affectsMonly (the intermediates). 102 8.5 Algorithm: Parallel Causal Discovery 8.5 Algorithm: Parallel Causal Discovery TRACE uses a series of parallelizable tensor operations on GPUs. Instead of iterating sequentially,I L G is evaluated for all candidate edges simultaneously. We start from an unknownG un representing the single stream and iteratively remove edges based on the CMI estimation Ë I N to obtainG t,s . An overview of the process can be found in Fig.8.3. We introduce Theorem3, which guarantees the soundness of our algorithm when returning the strong instance-time causal graph. Theorem 3(Soundness of TRACE for the Instance-Time Causal Graph).LetG t,s be the Instance-Time Causal Graph of a sequencesgenerated by a stochastic process. Assume the underlying distributionPisΔ-Strongly Faithful toG t,s (Def.24) and that TRACE uses a consistent CMI estimator Ë I N with thresholdÏ Î” (Lemma9) and the corresponding Δ-Oracle ModelP Ξ . Then, under Causal SuïŹiciency (A1) and Temporal Precedence (A2), it recovers the correct Instance-Time Causal GraphG t,s asymptotically asNââ. (Proof Sketch)By induction, we show that for each sequential stept, we can recover the potential causesE <t of the effect eventE t using the consistent CMI estimator (Prop.1) which generates aΔ-Strong Faithful CI-test (Lemma9) and control for lagged effects using (Def.26). By temporal precedence (A2) and causal suïŹiciency (A1), we can conclude that no other events will affect the effect eventE t and thus verify the heredity. Proof.We proceed by induction on the time indextâ 1, . . . , Lknowing temporal precedence (Assumption 2). Goal:We show that for everyt, the estimated parent set c P a(x t )is exactly the true parent setP a(x t )in the instance-time causal graphG t,s . Base Case (t= 1):Consider the first eventx 1 . By the temporal precedence and causal suïŹiciency (Assumption1),x 1 has no ancestors in the observed sequence. Thus, ĂNParticles 1. Observed Seqs x (l) 0:c x 1 x 2 x 3 2. Noise SamplesM (l) m (l) 2 m (l) 3 3. Input Construction x (l) 0:c x 1 m (l) 2 m (l) 3 x (l) 0:c x 1 x 2 m (l) 3 x (l) 0:c x 1 x 2 x 3 4. Inference Tf Ξ AR Model X do 5. Output Tensor p 0 . . .. . . p 1 . . .. . . p 2 . . .. . . p 3 . . .. . . 6. Compare 0 I 1â2 I 1â3 00 I 2â3 000 P raw Shift [:â1] Shift [1 :] P base P do D KL (P base ||P do ) Averaged acrossN Fixed Contextx (l) 0:c Observed Historyx â€j Intervened Mediatorsm (l) j+1:iâ1 Figure 8.3:Overview of TRACE Parallel CI-tests.We construct a single broadcasted tensorX do where each rowjincrementally fixes the historyx â€j while randomizing the future (staircase pattern). The model processes this tensor in parallel to produce raw probabilitiesP raw (grey). We then compute the Causal Mutual Information by comparing adjacent rows: the distribution at rowjâ1serves as the baseline (P base , blue) for the intervention at rowj(P do , red). 103 8 Sample-Level Event-to-Event Causal Discovery the true parent set isP a(x 1 ) =â . The TRACE algorithm evaluates candidatese 1âk for kâ„1. Since no such events exist in the sequence, the candidate set is empty. TRACE returns c P a(E 1 ) =â . Thus, c P a(E 1 ) =P a(E 1 ). Heredity:Assume that for all time stepsj < t, the algorithm has correctly identified the local structure (though note that the decision fore t depends only on the historyx <t , not on previous graph decisions). We consider the eventE t . With the full variant, the algorithm iterates through all valid past eventsE tâk âE <t as candidate parents.For each candidate, we apply the decision rule based on the estimator Ë I N (Def.24) assuming that no hidden confounders alters the CI-tests (Assumption1): âąCase 1:E tâk is a True Parent (E tâk âP a(E t )).By theΔ-Strong Faithfulness assumption (Def. 24), the true conditional mutual information satisfiesI >2Ï Î” . By Lemma9(Identifiability), this ensures that the estimator satisfies Ë I N > Ï Î” asymptotically. Consequently, TRACEacceptsthe edge. âąCase 2:E tâk is Not a Parent (E tâk /âP a(E t )).By the Causal Markov Condition, conditioned on the historyx <t (which contains the true parents),E t is independent of non-descendants. Thus,I(E t ;E tâk |X <t ) = 0. By Lemma9, the estimator is bounded by the noise floor: Ë I N â€Ï Δ . Consequently, TRACErejects the edge. Since the algorithm makes the correct decision for every candidateE tâk individually, the resulting set c P a(E t )is identical toP a(E t ). Conclusion:By induction, c P a(E t ) =P a(E t )for allt= 1, . . . , L. Since the graphG t,s is defined by the union of these parent sets, TRACE recoversG t,s exactly. Consequently, by Def. 23, TRACE recovers the instance summary causal graphG s . 8.5.1 Scalability of TRACE To parallelize the Monte-Carlo estimation (Eq.8.5) and counterfactual sampling (Eq.8.16) we avoid computing the CMI for the full trajectoryx 0:tâ1 but a truncated version, called contextsimilarly to Chapter6, such as: x <t âx 0:c forc < tand0< câȘL We argue that truncating the history enables parallelized CI-tests on GPUs. In all experiments, we setc=max(0.1L,20). Although it might break Markovianity for long sequence, empirical results show robustness to this truncation. We provide ablation to unseen sequence lengths during training and show our method to be robust to high delayed effects in Fig 8.5. Sparse ApproximationIn contrary to the sample-level multi-label causal discovery case, for each time steptwe must performtCI-tests (one for every potential lag per step). For a sequence of lengthL, the total number of CI-tests is given by P L t=1 t= L(L+1) 2 104 8.6 Experiments which grows quadratically with the sequence length. As a result, even on multiple GPUs, inference becomes computationally intractable. To solve this, we propose asparse variantfor which we bound the lagged effects of previous events on future events up to a memorym(Assumption7). Thus the DGP X t becomes anm-order Markov chain. WithmâȘL, TRACE scales linearly with the sequence lengthL. The memory complexity transitions from: O(N·(Lâc)·L·|X|) Bounded Memory ââO(N·m·L·|X|) 8.6 Experiments We evaluate TRACE on synthetic linear Structural Causal Models (SCMs) and real- world vehicle logs. All baselines utilize the same frozen backbone to evaluate the contri- bution of the inference mechanism. TRACE is implemented in Python. A more complete protocol description can be found in Appendix B.3.2as well as additional ablations. 8.6.1 Settings Synthetic Linear-SCMWe validate TRACE on sequences generated by linear SCMs with controllable memorym, sequence lengthL, and vocabulary size|X|. Our evalu- ation proceeds in two phases: (1) We train a standard AR LM (LLaMA architecture [147]) on the SCM and validate the training by monitoring the oracle scoresËΔ(Eq.B.6, normalizedΔ) (2) We then apply TRACE to recover the summary causal graph of each single observation. To evaluate performance, we perform atomic interventions by uni- formly randomizingE t and measuring the average KL divergence over 10 counterfactual between post-intervention and observational distributions ofE t âČ . If the divergence is aboveÏ >0.05, an edgeE t âE t âČ exists inG t,s . We then report the Precision, Recall, and Structural Hamming Distance (SHD) against this ground truth. We benchmark TRACE against four distinct baseline types: âąNeural Granger: A Granger-causal discovery method that uses the same AR Model as TRACE but computes the difference in probability rather than the CMI to detect causality. âąAttention: We train a BERT [26] model on the same SCM with the same model capacity and training steps as Tf Ξ and extract the attention scores at the deepest layer. A thresholdÏ= 0.02is applied to get the adjacency matrix. âąSaliency(InputĂGradient): A local sensitivity baseline that estimates feature im- portance by computing the gradient of the target tokenâs log-probability with respect to the input embeddings [ 138]. âąShapley Value Sampling: An axiomatic attribution method rooted in cooperative game theory [83]. Shapley values estimate theaverage marginal contributionof a tokene t to the prediction ofe t âČ by sampling permutations of the input history. âąNaive baselines: A random guesser that predicts edges(E t âE t âČ )with a fixed probabilityÏ= 0.01and a frequency baseline that assumes the top-kmost frequent 105 8 Sample-Level Event-to-Event Causal Discovery event types are universal causes for all other events. These tests reveal whether the task is non-trivial. 8.6.2 Comparative Analysis As detailed in Table8.2, TRACE establishes state-of-the-art performance for causal dis- covery on discrete sequences generated from single streams and outperforms the strongest baseline by over 0.20 F1 Score. While attention scores alone fail to distinguish correla- tion from causation (F10.50), the Neural Granger baseline achieves a respectable F1 of 0.69but suffers from high variance (SHD100.2±14.6). Saliency methods, traditional in NLP, also fail to distinguish causal links between events, especially with poor precision 0.51. This disparity highlights a critical insight:measuring the CMI is a far more robust signal of causalitythan monitoring the probability fluctuation of a single target token (Granger) or relying on metrics that are not anchored in causal discovery (Saliency, attention scores). MethodSHD (â) F1 (â) Precision (â) Random218.8±3.20.01±0.000.04±0.01 Frequency723.0±6.70.09±0.000.06±0.00 Attention (BERT)321.0±150.50±0.010.35±0.01 Saliency (Input x Gradient LLaMA) 160.2±6.550.67±0.010.51±0.01 Shapley Value Sampling (LLaMA) 148.0±5.090.60±0.010.55±0.01 Neural Granger (LLaMA)100.2±14.60.69±0.040.71±0.04 TRACE(LLaMA)28.6±2.80.91±0.010.89±0.01 Table 8.2:Identifiability Comparison.Comparison of causal discovery performance on syn- thetic SCMs with|X|= 1000, L= 64, Δ= 0.05, Ï= 3.10 â5 , N= 128.TRACE significantly outperforms local (Saliency) and global (Granger/Shapley) baselines, achieving over 20 points higher F1 while maintaining high precision. Results across 10 runs are reported. 8.6.3 Scalability and Robustness Analysis Breaking the Curse of DimensionalityStandard causal discovery algorithms suffer from combinatorial explosions when the variable count or state space increases. In Fig.8.4, we challenge TRACE with massive vocabularies to validate its scalability to massive event types. Remarkably, we observe thatdiscovery performance is invari- ant to the vocabulary size|X|. The F1 score remains stable atâ0.81even as the search space grows exponentially and the underlying SCM entropy increases. This con- firms a key advantage of our approach: by leveraging a pre-trained AR model, TRACE enables causal discovery over massive event vocabularies at scales that were infeasible before. 106 8.6 Experiments 10 2 10 3 Vocabulary Size |î| (Log Scale) 0.0 0.2 0.4 0.6 0.8 1.0 Score (Higher is Better) Pred: 0.66 Pred: 0.59 Pred: 0.57 Pred: 0.58 Pred: 0.56 Pred: 0.54 Pred: 0.53 Params: 0.42 (m) Params: 0.76 (m) Params: 1.64 (m) Params: 4.45 (m) Params: 9.97 (m) Params: 16.67 (m) Params: 31.34 (m) F1 Score Precision Recall SHD (Error) 17.5 20.0 22.5 25.0 27.5 30.0 32.5 SHD (Lower is Better) Ground Truth (|î|=50) TRACE Pred (|î|=50) Ground Truth (|î|=4000) TRACE Pred (|î|=4000) Figure 8.4:Scalability to High-Dimensional Event Spaces.Evaluation of structural identifiability across exponentially growing vocabulary sizes.Top:Evolution of discovery metrics. TRACE exhibitsperformance invariance, maintaining F1 â0.81even as the combinatorial search space explodes.Bottom:Visual ex- amples of recovered summary graphsG s at scale. Shannon redundancy as (Pred =1âH(P)/H max ) confirm that TRACE succeeds even in high-entropy regimes. (ËΔ= 0.01, L= 64, N= 64, Ï= 10 â4 ). Unseen Sequence LengthsWe evaluate TRACEâs ability to scale to unseen sequence length during training of the AR Model, as well as the maximum GPU memory con- sumption for theFulland theSparsevariant, which computes up tomlagged effects per event. As shown in Fig. 8.5(b), the sparse variant of TRACE achieves linear memory scaling, whereas the full variant exhibits quadratic memory growth. For the classifica- tion metrics, we observe a quick degradation in Recall as the sequence length exceeds the training window. Therefore, the modelâs ability to identify all causal links diminishes as the temporal context becomes increasingly out-of-distribution. Precision remains re- markably high and stable(â0.95) across all tested lengths. This suggests that the model becomes more conservative with longer sequences. Thisconservative failure modeis highly desirable and further supported by Fig. 8.5(a). Identifiability Precedes Convergence.A central question is whether the AR model must be perfect (Δâ0) to recover causal structure. Fig.8.5(a) reveals a critical find- ing:exact convergence is not a necessary condition for identifiability. We observe a distinct phase transition where the coarse-grained causal graph is recovered 107 8 Sample-Level Event-to-Event Causal Discovery 10 â2 10 â1 10 0 Oracle Score Δ (Log Scale) 0.0 0.2 0.4 0.6 0.8 1.0 Discovery Metrics Out of Regime ( Δ > 0.1) (a) Robustness to Generative Error 20406080100120 Sequence Length L 0.0 0.2 0.4 0.6 0.8 1.0 Train Length (b) Scalability to Length 01020304050 Memory Horizon m 0.0 0.2 0.4 0.6 0.8 1.0 (c) Long-Range Dependencies 60 70 80 90 100 110 120 130 SHD 0 2000 4000 6000 8000 10000 12000 14000 Peak VRAM (MB) 10 20 30 40 50 60 70 80 SHD F1 ScorePrecisionRecallSHDVRAM (Full)VRAM (Sparse) Figure 8.5:Robustness and Scalability Analysis(|X|= 1000, N= 128, Ï= 10 â4 , L= 64). Evolution of causal discovery performance (F1, Precision, Recall, SHD).(a) Ro- bustness to Generative Error:Performance as a function of the modelâs oracle scoreΔ. TRACE exhibits a phase transition, recovering structure even for imper- fect models (Δ <0.1) and maintaining high Precision even as fidelity degrades.(b) Scalability to Length:Performance and GPU memory usage vs. sequence length L. TheSparsevariant demonstrates linear memory scaling (O(mL)), enabling in- ference on sequences far exceeding the training length (L= 64), whereas the Full variant scales quadratically.(c) Long-Range Dependencies:Robustness to in- creasing delayed-effectsm. TRACE maintains F1>0.8even as dependencies span one third of the sequence (m= 20), confirming the methodâs ability to capture distant causal mechanisms. early in the training dynamics (Δâ0.1), before the model masters fine-grained transi- tion probabilities. TRACE exhibits aconservative failure mode. As approximation errorΔincreases, the model defaults to âblindnessâ (lower Recall) rather than âhallucina- tionâ (lower Precision), maintaining near-perfect precision (>0.95) even in high-entropy regimes. This suggests that model uncertainty manifests as a failure to detect weak sig- nals rather than the generation of false positivesâa desirable property for safety-critical applications. Deep Temporal Dependencies.In Fig.8.5(c), we stress-test the method by extend- ing the memory horizon of the underlying SCM up tom= 48. TRACE maintains high stability (F1 > 0 . 80 ) up to m = 20 , confirming that our parallelized intervention mechanism effectivelycaptures long-range dependencies. While precision naturally softens as the number of events behind detected increases, hence necessitating a bigger thresholdÏ. 8.6.4 Application to Vehicles Diagnostics To evaluate TRACE, we use the same dataset as in Chapters6and7. The training loss reported of Tf x isL AR = 1.91nats. We apply TRACE to analyze complex electrical 108 8.7 Limitations 7.32 7.06 0.30 3.72 0.98 0.19 1.77 0.34 0.16 1.83 0.330.25 0.34 0.34 0.92 0.37 2.79 0.90 0.45 0.20 0.45 0.21 1.19 t=1t=2t=3t=4t=5t=6t=7t=8t=9t=10t=11t=12t=13t=14t=15t=16t=17t=18t=19t=20t=21t=22t=23t=24t=25t=26t=27t=28t=29t=30t=31t=32t=33 Tailgate Blocked Tailgate Lift Limited Augmented View Limited Camera Calib Error Camera Out- of-Range BT Audio Play Phonebook Access Phonebook Access Engine Intervention Variance Cruise Range Stat Ignition Off Prevented Window Emergency Mode Interior Light Map Implausible Speed Limit Msg Missing Foresight Motor Fault FAS Signal Error Baseboard Supply Limited FAS Signal Error Baseboard Supply Limited Battery Temp Fault Secondary Network Fault Remote Driving Limited Wiper Limited Radar List Overflow FAS Signal Error Radar Interference 48V Temp Derating LDW Signal Inconsist. TAP Events Stat LCW Events Stat Radar List Overflow Substitute Speed Used PWM Actuation Available Instance Time Causal Graph 7.32 7.06 0.30 3.72 0.98 0.19 1.77 0.34 0.16 0.45 1.19 1.83 0.33 0.34 0.34 0.92 0.37 2.79 0.90 0.20 0.45 0.21 Tailgate Blocked Tailgate Lift Limited Augmented View Limited Camera Calib Error Camera Out- of-Range BT Audio Play Phonebook Access Cruise Range Stat Ignition Off Prevented Window Emergency Mode Interior Light Map Implausible Speed Limit Msg Missing Foresight Motor Fault FAS Signal Error Radar List Overflow Substitute Speed Used Baseboard Supply Limited Battery Temp Fault Secondary Network Fault Remote Driving Limited Wiper Limited Radar Interference 48V Temp Derating LDW Signal Inconsist. TAP Events Stat LCW Events Stat PWM Actuation Available Engine Intervention Variance Instance Summary Causal Graph Figure 8.6:Instance-Time Causal Graph.Temporal evolution of a diagnostic defect cas- cade in a vehicle (|X|â29,100). TRACE effectively captures causal relationships, revealing distincterror clustersat different time steps (e.g., initial sensor failures att= 3triggering mechanical faults att= 12, battery at issuet= 17). This enables actionable root-cause analysis by isolating the specific onset of a failure mechanism and their strength using the CMI Ë I N cascades, specifically focusing on battery and sensor degradation scenarios as shown in Fig.8.6withG t,s and in Fig.B.6withG s . In standard approaches, these events are often collapsed into a static correlation graph, obscuring the order of operations and their causal relationship. 8.7 Limitations We now include a discussion regarding the main assumptions taken in this chapter. 8.7.1 Causal SuïŹiciency A fundamental assumption in causal discovery isCausal SuïŹiciency(Assumption1)â the premise that no unobserved confounders influence the system. Since TRACE relies on pre-trained backbones which may have learned from noisy or incomplete data, we empirically evaluate the robustness of TRACE under controlled violations of causal suïŹiciency, focusing on two realistic forms of hidden confounding. Measurement Error (Noise Injection).In Fig.8.7(a), we simulate measurement error by randomly replacing valid tokens in the history with noise (P noise ). While Recall naturally degrades as the true causal parents are obscured,Precision remains high 109 8 Sample-Level Event-to-Event Causal Discovery (>0.8) even when 40% of the context is corrupted. This confirms that TRACE does not hallucinate false edges from noisy inputs; if the causal signalXâYis destroyed by measurement error, the model assignsCM Iâ0rather than hallucinating a spurious link. Missing Intermediaries (Temporal Drops).In the same Fig.8.7(b), we simulate miss- ing data by randomly dropping time steps, effectively hiding intermediate nodes in the causal chain (XâZ hidden âY). This is a more critical scenario where the Markov conditioning sets are incomplete as intermediate causal nodes are unobserved. 0.00.10.20.30.40.5 Noise Probability P noise 0.0 0.2 0.4 0.6 0.8 1.0 Discovery Score High Precision (No Hallucination) (a) Robustness to Measurement Noise 0.00.10.20.30.40.5 Drop Probability P drop (b) Robustness to Missing Intermediaries F1 ScorePrecisionRecall Figure 8.7:Robustness to Hidden Confounding.Evaluation of TRACE under violations of causal suïŹiciency.(a)Measurement Error: Random noise is injected into the context. Precision stays high, indicating resistance to hallucination.(b)Tempo- ral Drops: Time steps are randomly deleted, thus conditioning sets are broken. TRACE still recovers structure despite missing intermediaries but the discovery scores quickly decrease. 8.7.2 Temporal Precedence & Instantaneous Effects TRACE assumes that causal influence respects temporal precedence, which means ob- servations are perfectly recorded over time and therefore does not model instantaneous causal effects between events occurring at the same time index. This assumption is standard in sequential causal discovery and ensures that the recovered causal graph is acyclic and identifiable in the single observed sequence setting. From a theoretical standpoint, instantaneous effects are not identifiable from a single observed trajectory without additional parametric assumptions, repeated samples, or access to interventions. In practice, apparent simultaneity oftenarises from time dis- cretization, logging resolution, or batching effects. TRACE interprets such casesthrough the earliest observable temporal ordering, yielding a conservative but identifiable causal structure. As a result, the recovered summary graph captures directed causal influence with positive temporal delay, rather than true simultaneity. 110 8.8 Summary 8.7.3 Consistency Through Time TRACE operates in a one-shot regime, where causal structure must be inferred from a single observed trajectory rather than from repeated i.i.d. samplesduring inference. To make this statistically meaningful, we assumeconsistency through time: the causal mechanisms governing the generation of events are invariant across time indices. In other words, causal directionality does not reverse over time. For instance, ifAâB, it is subsequently assumed thatBÌžâA. This assumption is strictly weaker than stationarity [6]. While the marginal distribu- tion ofX t may vary over time, the underlying causal dependenciesâencoded by the directed edges of the instance-time causal graph remain stable. This form of causal in- variance is standard in sequential settings and underlies the validity of summary causal graphs that collapse time-indexed relations into event-to-event dependencies (Def.23). 8.7.4 Amortization TRACE leverages amortized causal discovery, shifting the computational burden to the self-supervised pretraining of an autoregressive model. However, because the base en- tropy of the true data-generating process is rarely known in practice, the generative errorΔcannot be directly deduced from the cross-entropy loss. This makes identifying the optimal model capacity and training duration challenging; as a practical heuristic, we recommend slight over-parameterization and extended training. Furthermore, while TRACE achieves linear scaling during inference with respect to sequence length and vocabulary size (|X|), this eïŹiciency does not account for the prerequisite pretraining phase, which scales non-linearly with|X|. 8.8 Summary We presentedTRACE, a framework for single-sequence causal discovery in high-dimensional discrete event streams. By repurposing a pretrained autoregressive model as a condi- tional density estimator, TRACE recovers the instance-time and summary causal graphs without any task-specific retraining, scalinglinearlywith the event vocabulary size. Our comparative analysis demonstrates that measuring causal influence via condi- tional mutual information is fundamentally more discriminative than attribution-based alternatives:TRACEachieves an F1 score of0.91±0.01, outperforming the strongest attribution baseline (Shapley value sampling,0.60) by more than 30 points and Neural Granger causality (0.69) by over 20 points. Critically, this gap is not merely a matter of model capacity, all methods share the same pretrained backbone, but reflects a funda- mental distinction betweencorrelation(captured by saliency and Granger-type scores) andcausal dependence(captured by the CMI under controlled interventions). This re- sult provides direct empirical evidence that causal structure learning is necessary for reliable discovery in event sequences, and cannot be replaced by post-hoc attribution. On the theoretical side, Theorem 2establishes a finite bound on the gap between the estimated CMI Ë I N and the true causal strengthI, showing that this gap is governed 111 8 Sample-Level Event-to-Event Causal Discovery exclusively by the modelâs cross-entropy approximation errorΔ. Lemma9then guaran- tees that the instance-time causal graph is identifiable with probability 1 asNâ â, provided the true causal signal exceeds the noise floorÏ Î” induced byΔ, a condition we termΔ-Strong Faithfulness. Taken together, these results confirm that structural identifiability is achievable under imperfect density estimation, and that minimizing the pretraining cross-entropy loss directly tightens the causal identification bound.TRACE thus represents a principled, GPU-scalable step toward causal discovery in real-world, high-dimensional event streams, where both the scale and the noise level of data preclude oracle-level approximations. 8.9 Outlook The discovery frameworks presented in this partOSCAR,CARGO, andTRACEcol- lectively establish a scalable pipeline for uncovering causal structures in high-dimensional single-event streams. Several frontiers nonetheless remain open. Toward MDL-grounded Causal Scoring.A conceptually significant direction concerns the theoretical foundations of the causal scoring function underlyingTRACE. At its core,TRACEdecides whether eventE t causesE t âČ by measuring how much knowing E t reduces uncertainty aboutE t âČ , given the observed historyâa quantity formalized as the conditional mutual information. This choice is empirically effective, yet it can be given a deeper justification through the lens of theMinimum Description Length(MDL) principle [ 124,46]. MDL is a general principle of model selection grounded in information theory: given a dataset, the best explanation is the one that leads to the shortest description of the data [ 46]. Concretely, one compares two descriptions of the same observations, a model Mand the data encodedgiventhat model and selecting the model minimizing their combined lengthL(M) +L(D |M). It is a mathematical formulation of the Occamâs razor principle, stating that when faced with competing explanations for the same phe- nomenon, the simplest oneârequiring the fewest assumptions is usually the best. Ap- plied to causal discovery, this principle instantiatescausal minimality: the true causal graph is the simplest one that fully accounts for the observed dependencies, and any additional edge would only increase the total description length without improving the fit. MDL-based methods have been successfully applied to causal inference in discrete data [ 24] and, most closely related to the present setting, to learning Granger-causal networks inTPP[67]. TRACEâs approximation error bound (Theorem2) reveals a direct structural par- allel with these ideas. Cross-entropyisa compression objective: a model achieving a lower cross-entropy provides a shorter expected code for the data under Shannonâs source-coding theorem [ 23]. MinimizingΔduring pretraining therefore simultaneously tightens the noise floorÏ Î” , sharpens the identifiability threshold, and, in the language of MDL, corresponds to finding a more compact two-part description of the event sequence. Formalizing this connection constitutes a promising theoretical direction. 112 8.9 Outlook Unified Event-to-Event and Events-to-Outcome Discovery.A unified framework sup- porting simultaneous recovery of both event-to-event and events-to-outcome causal de- pendencies within a single sequence would provide a more complete picture of system dynamics. As indicated by the open cell in Table1.1, the population-level counterpart ofTRACEremains an open problem, requiring new aggregation strategies and, likely, explicit acyclicity constraints to reconcile instance-time graphs across sequences. From Causal Graphs to Actionable Rules.Despite providing a robust causal back- bone, the numerical and graphical representations produced byTRACEremain a silent language for the domain expert. Causal graphs identify statistical dependencies but do not inherently capture the symbolic logicâsuch as Boolean combinations of eventsâ or the contextual nuance required for formal diagnostic standards. There remains a substantial gap between numerical causal evidence and the actionable rules that practi- tioners can validate and trust. PartIIIcloses this methodological arc. By transitioning from causal discovery to automated reasoning, it embeds the causal structures recovered byOSCAR,CARGOwithin a multi-agent framework powered byLLMs. As a result, the system synthesizes interpretable Boolean rules alongside natural language explana- tions, effectively transforming raw causal evidence into human-interpretable diagnostic expertise. 113 Part I Automated Causal Reasoning and Logic Synthesis 115 9 Multi-Agent Causal Reasoning for Automated Error Pattern Rule Synthesis While PartIfocused on learning event representations and PartIIon uncovering causal dependencies, PartIIIexplores how these causal insights can be operationalized. It is based on the following contribution: Neuro-Symbolic Rule Discovery: Empowering LLMs with Causality for Vehicle Di- agnostics. [93] Hugo Math, Julian Lorenz and Rainer Lienhart. International Conference on Learn- ing Representations (ICLR) Workshop on Logical Reasoning of Large Language Models, April 2026. LLMs and agentic systems have demonstrated the capacity to integrate heterogeneous information sources for structured reasoning tasks. From this perspective, we introduce CAREP: (CausalAutomatedReasoning forErrorPatterns), a multi-agent reasoning sys- tem that leverages the causal structures discovered by OSCAR and CARGO to generate interpretable EP rules with their natural language explanation automatically. CAREP integrates three coordinated agents: (1) acausal discovery agentthat refines DTCâEP relationships using the underlying causal graphs, (2) acontextual information agentthat enriches reasoning with semantic and metadata knowledge, and (3) anorchestrator agent that synthesizes candidate Boolean rules and produces transparent reasoning traces. We evaluated both the capacity of LLMs and CAREP to generate accurate Boolean rules (i.e., correct DTCs) and the accuracy of those rules based on their truth tables relative to the ground truth. CAREP demonstrates the ability to automatically in- fer accurate and human-readable EP rules, outperforming purely language-model-based baselines while providing interpretability. This framework thus completes the method- ological trajectory of the thesis, transforming predictive modeling into causal discovery and finally into automated causal reasoning. 9.1 Introduction Beyond causality, automation requires interpretability. Presenting unexplained rules is risky in safety-critical domains such as the automotive industry or in medical data. Large Language Models (LLMs) [108,25,147] have recently demonstrated emergent capabilities in planning, explanation, abduction, reasoning, and in-context learning [27, 171]. Especially when using Chain-of-Thought [155] (CoT) to guide them through step- by-step reasoning processes. Since CoT is not grounded in the external world, it is 117 9 Multi-Agent Causal Reasoning for Automated Error Pattern Rule Synthesis limited in its ability to reason reactively or update its knowledge, thereby propagating factual hallucinations and errors throughout the reasoning process. ReAct [164] goes further by creating areason to actandact to reasonframework where the LLM has access to external environments (e.g., Wikipedia) to incorporate additional information into the reasoning. Outputs One-Shot Causal Discovery Batch of unknown sequences of error patterns (backbone) Sample times Hidden states Compute and access Apply the adaptive threshold to the frequencies of Get one- shot graphs Fit long-tail support distribution of labels to obtain threshold function One-Shot (Phase 1) Graph aggregation (Phase 2) Batch of labeled sequences ACE PMI MI Frequencies Titan V2 Embedding Traceable Reasoning EP1 has potentially the rule: DTC is likely causing EP1 due to a high causal indicators notably the average causal effect. also appears when DTC is here due to the high point-wise mutual information, and appear to have a causal effect on also. It is associated with the CCM 128 [2] which is triggered when there is a water pump [2] issue. It is directly related with the description of [1]. References Sources: dtc descriptions.csv [1] ccms_description.csv [2] ep_known descriptions.csv [3] (backbone) pretrained on Sample context times Hidden states Compute the CMI and causal indicators Get graphs and aggregate using their frequencies Causal Discovery Batch of unknown sequences of error patterns Aggregate the DAGs for each unknown error patterns and give causal indicators Causal Reasoner Agent Orchestrator Agent (c) Contextual Information Agent (b) Causal Discovery (a) Generate m DAGs: Input Main: A batch of sequences of unknown EPs to extract next-DTC and next-EP probability Contextual Information: DTC descriptions Known EPs rule and descriptions Check control message (ccms) Metadata: model ranges, production step Traceable Reasoning EP1 has potentially the rule: DTC is likely causing EP1 due to a high causal indicators notably the average causal effect. also appears when DTC is here due to the high point-wise mutual information, and appear to have a causal effect on also. It is associated with the CCM 128 [2] which is triggered when there is a water pompe [2] issue. It is directly related with the description of [1]. Traceable Reasoning Looking at the data, I notice several key DTCs causing this error pattern with varying frequencies and causal relationships: 1. (frequency: 0.539) - This is the most frequent DTC, appearing in over half of the samples 2. (frequency: 0.137) - Second most frequent DTC 3. (frequency: 0.105) - Third most frequent DTC From the sample sequences, I observe that CCM 3 is consistently triggered across all samples. Looking at the DTC descriptions in the retrieved documents, I can see that several of these DTCs are related to the fuel system. DTCs like are likely related to fuel tank issues [1]. I also found a known error pattern that seems relevant: Fuel level sensor in the left tank does not reach the empty threshold with logic ")" which is related to fuel level sensors [3] Based on the PMI values, there are negative correlations between many of the DTCs, suggesting they might be alternative paths to the same error condition rather than co-occurring conditions. I propose the following rules: - reasoning: The most frequent DTCs in the samples. They co-occur (PMI > 0) in most sample trigerring CCM 3. Their high causal indicators (average causal effect: 0.45 and 0.3) support this rule. -reasoning: Looking at the frequencies and causal indicators, either or or sufices to trigger the rules. This aligns with the known error pattern above. - reasoning: The function of combined with create a pattern similar to known error patterns involving fuel problems. The PMI of and is strong (1.86), but this rule has medium confidence due to the lower frequency of these DTCs ... Traceable Reasoning Looking at the data, I notice several key DTCs causing this error pattern. I propose the following rules: - reasoning: The most frequent DTCs in the samples. They co-occur (PMI > 0) in most samples triggering CCM 3. Their high causal indicators (average causal effect: 0.45 and 0.3) support this rule. - reasoning: Looking at the frequencies and causal indicators, either or or sufices to trigger the rules. This aligns with the known error pattern "Fuel level sensors do not reach the threshold" - reasoning : The function of combined with create a pattern similar to known error patterns involving fuel problems. The PMI of and is strong (1.86), but this rule has medium confidence due to the lower frequency of these DTCs From the sample sequences, I observe that CCM 3 [2] is consistently triggered across all samples. Looking at the DTC descriptions in the retrieved documents, I can see that several of these DTCs are related to the fuel system. DTCs like are likely related to fuel tank issues [1]. I also found a known error pattern that seems relevant: Fuel level sensor in the left tank does not reach the empty threshold with logic ")" which is related to fuel level sensors [3]. Based on the PMI values, there are negative correlations between many of the DTCs, suggesting they might be alternative paths to the same error condition rather than co-occurring conditions. DTC Sequences Metadata Orchestrator Agent Contextual Agent Causal Estimator Agent 0x1283F Time Diagnostic Trouble Codes 0x09931 0x84324 (a) Raw Events(b) Reasoning(c) Error Pattern Rule EP = 0x1283F | 0x84324 ... Figure 9.1:Illustration of Error Pattern Automation in Modern Vehicles. (a) A ve- hicle with an unknown defect generates a sequence of Diagnostic Trouble Codes (DTCs) over time. (b) A domain expert analyzes the DTCs and the vehicleâs meta- data (e.g., model, descriptions). (c) Provide a Boolean rule to identify this error pattern. Through retrieval-augmented generation (RAG), external memory access, and con- text injection [75], LLMs can use raw text as references to improve question answering. Importantly, although causal graphs provide a theoretically grounded, valuable source of information, they alone rarely yield a complete, interpretable rule. A DTC may be discovered as a strong causal parent of an EP, but paired with the textual metadata can reveal whether the DTC reflects a root cause, a downstream effect, or an irrelevant side-symptom. LLMs offer a practical computational mechanism for integrating these heterogeneous evidence sources, resolving contradictions, and translating causal signals into symbolic form through the Boolean expression of an EP. Nevertheless, a single model is often insuïŹicient to handle the diverse subtasks involved in automation. The growing paradigm ofagentic systems[ 129,177,61] highlights how specialized agents, each with distinct capabilities, can collaborate through structured dialogues or planning loops. This opens the door to human-interpretable, automated causal reasoning in complex industrial pipelines [ 32]. For example, often referred to as causal machine learning (CML), it offers a more nuanced and interpretable approach to black box models and has found interesting applications [48]. That is, for LLMs, as noted by [96], they are sensitive and fragile without semantic or neuro-symbolic knowledge. For tasks requiring the selection of correct tokens, accuracy decreases exponentially with the number of tokens, making them unsuitable for high-dimensional event sequences alone. 118 9.2 Related Work 9.2 Related Work 9.2.1 Multi-Agent Fault Diagnosis The application of multi-agent systems for industrial diagnostics and failure analysis has become a prominent research area. [99] established an early framework using an agent-based inference engine to provide eïŹicient and reliable assistance for vehicle fail- ure diagnosis. In the domain of autonomous safety, [87] proposed an explainable system designed to interpret complex vehicle accidents, emphasizing the need for transparent decision-making. More recently, the scope of agentic systems has expanded to the manu- facturing sector. [112] introduced an agentic AI framework for âself-healingâ production lines capable of autonomous root cause analysis and fault correction. While these works share the goal of automating diagnostics through agents, CAREP distinguishes itself by addressing four critical challenges: âąExtreme Cardinality: CAREP is specifically designed for large-scale event streams containing over 29,100 unique DTC types, a scale where traditional inference en- gines, causal inference and discovery typically struggle. âąCausal Integration: It integrates a dedicated causal discovery agent (utilizing OS- CAR and CARGO) to filter irrelevant events before reasoning begins. âąMetadata Integration: Through embedding models, the metadata is integrated as references and used by a contextual information agent. âąSymbolic Synthesis: Unlike systems that provide static alerts, CAREP automates the synthesis of symbolic Boolean rules (Error Patterns) alongside natural language reasoning traces, effectively translating raw statistical evidence into actionable human-readable expertise. 9.2.2 Error Pattern Rule Automation The creation of EPs (Fig.9.1) has emerged as production and vehicle manufacturing have become increasingly complex. As demonstrated in the foundations, to identify new defects, experts analyze the diagnostic trouble code (DTC) sequence for each vehicle. Intuitively, this reasoning task is not trivial. Experts must draw on their prior obser- vations and knowledge of specific DTCs that may cause each EP. That is where the Boolean operators are useful. They allow us to construct different rules to characterize precise EPs. It sometimes requires the absence of certain DTCs (!NOT) to separate overlapping EPs. Or the presence of multiple DTCs at the same time (&AND) or a particular culprit (|OR). Therefore, the different operators(&,|,!)involve different sta- tistical perspectives. Intuitively, the!operator measures inhibitory strength [ 35], where the likelihood of observingy 1 will be lowered if we observex 10 (Fig9.1). On the other hand,&and|are more associated with excitatory events that will raise the likelihood of an EP. Knowing only the causes of an EP is not suïŹicient to elaborate a new rule. It is necessary to have indicators that quantify how DTCs are related to one another and 119 9 Multi-Agent Causal Reasoning for Automated Error Pattern Rule Synthesis to EPs. Finally, rules are subject to change and are dynamically updated by a domain expert based on the new incoming data. As a result, automating a rule is a complex, dynamic problem. 9.3 Methodology LetD=S 1 l ,·, S m l be a dataset of multi-labeled sequences. Each labely j is defined as a Boolean rule between some eventsx i i , xâ X. Such that if in a sequenceS (k) l , the Boolean rule is true for some labely j , then it is present inS (k) l .Dcontains multiple labels with unknown Boolean rules denoted asy un â Y. We aim to find the unknown rule of each labely un using the observed sequences inD. The general architecture of CAREP is shown in Fig.9.2. The agentic system outputs a reasoning explanation for each estimated error pattern rule. To increase predictionâs plu- rality, CAREP outputs5rules per EP with different levels of confidence (high, medium, low) that are reflected in the provided explanation. Each agent receives a system prompt that specifies their task and uses CoT [ 155]. The orchestrator is responsible for structur- ing comprehensive, traceable reasoning for each rule based on the other agentsâ outputs. We now examine each component separately. 9.3.1 Causal Discovery Population-Level Causal GraphsWe explicitly model the unknown error patternsy un as the target in the autoregressive classifier Tf y âs (EPredictor) output logits 1 . This forms a supervised unknown class learning strategy and enables us to extract the posteriors P Ξ y (Y=y un |.). To provide new causes for each unknown EPs, we employ OSCAR (k= 2.75, N= 64) and CARGO to extract causal relationships from the datasetD. As a result, we get as output a collection of DAGs asG â j c j , representing unknown error pattern aggregated causal relationships, alongside causal indicators. Average Causal EffectWe reuse the causal indicator defined in the one-shot phase in Eq. ( 6.14) asACE(Average Causal Effect). This indicates, on average, how much the DTC increased or decreased the likelihood of an error pattern. We report its expected value as the mean and its standard deviation over the sampled particlesz (l) used in the Monte Carlo estimation of Eq. (6.9): Ë C ÎŒ (Y j , X i )â 1 N N X l=1 P(Y j = 1|X i = 1, Z=z (l) )âP(Y j = 1|Z=z (l) ) (9.1) Ë C Ï (Y j , X i )â v u u t 1 N N X l=1 P(Y j = 1|X i = 1, Z=z (l) )â Ë C ÎŒ (Y j , X i ) 2 (9.2) 1 The training labels are used but the Boolean rule is generated at inference time. 120 9.3 Methodology Outputs One-Shot Causal Discovery Batch of unknown sequences of error patterns (backbone) Sample times Hidden states Compute and access Apply the adaptive threshold to the frequencies of Get one- shot graphs Fit long-tail support distribution of labels to obtain threshold function One-Shot (Phase 1) Graph aggregation (Phase 2) Batch of labeled sequences ACE PMI MI Frequencies Titan V2 Embedding Traceable Reasoning EP1 has potentially the rule: DTC is likely causing EP1 due to a high causal indicators notably the average causal effect. also appears when DTC is here due to the high point-wise mutual information, and appear to have a causal effect on also. It is associated with the CCM 128 [2] which is triggered when there is a water pump [2] issue. It is directly related with the description of [1]. References Sources: dtc descriptions.csv [1] ccms_description.csv [2] ep_known descriptions.csv [3] (backbone) pretrained on Sample context times Hidden states Compute the CMI and causal indicators Get graphs and aggregate using their frequencies Causal Discovery Batch of unknown sequences of error patterns Aggregate the DAGs for each unknown error patterns and give causal indicators Causal Reasoner Agent Orchestrator Agent (c) Contextual Information Agent (b) Causal Discovery (a) Generate m DAGs: Input Main: A batch of sequences of unknown EPs to extract next-DTC and next-EP probability Contextual Information: DTC descriptions Known EPs rule and descriptions Check control message (ccms) Metadata: model ranges, production step Traceable Reasoning EP1 has potentially the rule: DTC is likely causing EP1 due to a high causal indicators notably the average causal effect. also appears when DTC is here due to the high point-wise mutual information, and appear to have a causal effect on also. It is associated with the CCM 128 [2] which is triggered when there is a water pompe [2] issue. It is directly related with the description of [1]. Traceable Reasoning Looking at the data, I notice several key DTCs causing this error pattern with varying frequencies and causal relationships: 1. (frequency: 0.539) - This is the most frequent DTC, appearing in over half of the samples 2. (frequency: 0.137) - Second most frequent DTC 3. (frequency: 0.105) - Third most frequent DTC From the sample sequences, I observe that CCM 3 is consistently triggered across all samples. Looking at the DTC descriptions in the retrieved documents, I can see that several of these DTCs are related to the fuel system. DTCs like are likely related to fuel tank issues [1]. I also found a known error pattern that seems relevant: Fuel level sensor in the left tank does not reach the empty threshold with logic ")" which is related to fuel level sensors [3] Based on the PMI values, there are negative correlations between many of the DTCs, suggesting they might be alternative paths to the same error condition rather than co-occurring conditions. I propose the following rules: - reasoning: The most frequent DTCs in the samples. They co-occur (PMI > 0) in most sample trigerring CCM 3. Their high causal indicators (average causal effect: 0.45 and 0.3) support this rule. -reasoning: Looking at the frequencies and causal indicators, either or or sufices to trigger the rules. This aligns with the known error pattern above. - reasoning: The function of combined with create a pattern similar to known error patterns involving fuel problems. The PMI of and is strong (1.86), but this rule has medium confidence due to the lower frequency of these DTCs ... Traceable Reasoning Looking at the data, I notice several key DTCs causing this error pattern. I propose the following rules: - reasoning: The most frequent DTCs in the samples. They co-occur (PMI > 0) in most samples triggering CCM 3. Their high causal indicators (average causal effect: 0.45 and 0.3) support this rule. - reasoning: Looking at the frequencies and causal indicators, either or or sufices to trigger the rules. This aligns with the known error pattern "Fuel level sensors do not reach the threshold" - reasoning : The function of combined with create a pattern similar to known error patterns involving fuel problems. The PMI of and is strong (1.86), but this rule has medium confidence due to the lower frequency of these DTCs From the sample sequences, I observe that CCM 3 [2] is consistently triggered across all samples. Looking at the DTC descriptions in the retrieved documents, I can see that several of these DTCs are related to the fuel system. DTCs like are likely related to fuel tank issues [1]. I also found a known error pattern that seems relevant: Fuel level sensor in the left tank does not reach the empty threshold with logic ")" which is related to fuel level sensors [3]. Based on the PMI values, there are negative correlations between many of the DTCs, suggesting they might be alternative paths to the same error condition rather than co-occurring conditions. Figure 9.2:CAREP: A Causal Reasoning Agentic System for Error Pattern Au- tomation. (a) Represents the causal discovery phase, where we extract candidate DTC causes for the unknown error pattern alongside causal indicators. It then feeds the causal reasoner agent. (b) The descriptions and metadata are extracted through a Titan V2 Embeddings and fed into the contextual information agent. (c) The orchestrator agent manages the two agents and provides traceable reasoning to explain why the unknown error pattern may match these rules. Co-occurrenceThe different Boolean operators in the defined EP rules in Eq. (2.20) imply different statistical perspectives. In the previous section, we captured the elements in the rule, i.e., the set of DTC causes for a given label. However, we should also provide co-occurrence estimates of event pairs to properly address the OR, AND, and NOT operators. Traditionally, Point-wise Mutual Information (PMI) [ 23] is used to measure co-occurrence strength. It is defined as: pmi(x i , x j ) = log p(x i , x j ) p(x i )p(x j ) (9.3) wherep(x i ), p(x j )are probability mass functions. Along with the previous empirical frequency of eventx i in the set of causes of labely j , denoted asËÏ i,j , we also compute empirical joint frequencies of pairs of events within the same label-specific graphG â j . Formally, given a collection of DAGs(V k , E k ) m k=1 , the 121 9 Multi-Agent Causal Reasoning for Automated Error Pattern Rule Synthesis marginal and joint empirical probabilities are estimated as Ëp(x i |y j ) = 1 m j m X k=1 1X i âV k , Y j âV k , Ëp(x i , x â |y j ) = 1 m j m X k=1 1X i âV k , X â âV k , Y j âY k , wherem j denotes the number of samples inDwhere labelY j is present, and1·is the indicator function. The PMI between two eventsx i andx â conditioned on a labely j is then given by: d pmi(x i , x â |y j ) = log Ëp(x i , x â |y j ) Ëp(x i |y j ) Ëp(x â |y j ) .(9.4) Intuitively, d pmi(x i , x â |y j )>0indicates that eventsx i andx â co-occur more often than expected under independence, while a negative value suggests mutual exclusivity. 9.3.2 Contextual Information We add to each explanation ofy un contextual information about the description of DTCs, EPs, and the known error pattern rules. These descriptions are processed into embedding vectors using aTitan V2 Embedding 2 from AWS Bedrock. They are then used as a Retrieval-Augmented Generation (RAG) [ 75] system, in which the contextual information agent queries are matched against the description embeddings generated by the embedding model. It is incorporated into the context of the contextual information and orchestrator agents. Now, directly in the prompt (in-context learning[ 27]), we randomly inject 10 DTC sequences that exhibit the same unknown EP we aim to identify. As well as metadata, such as the vehicle model range and the triggered message printed on the board (check control message: CCM). The overall data inputs are shown in Fig. 9.2, and the JSON input to the orchestrator is shown in Appendix.B.7. 9.4 Experiments 9.4.1 Evaluation Metrics This task requires multiple levels of evaluation. Suppose that we have an automated method to extract a Boolean ruleR â for each unknown EPy un in a batch of sequences. We then want to evaluate how good this generated rule is compared to the ground truth. Hence, we distinguish between two evaluation types: (1)structural, where we compare the Boolean rules directly as a classification set, and (2)semantic, where we compare the corresponding truth-tables of the estimated Boolean expressions to the truth-table of the ground truth using the Sympy package [ 95]. 2 https://aws.amazon.com/de/blogs/aws/amazon-titan-text-v2-now-available-in-amazon-bedrock- optimized-for-improving-rag/ 122 9.4 Experiments 9.4.1.1 Structural Evaluation Specifically, for (1) we evaluate:Are the DTCs in the estimated rules correct?For this, we divide the estimated sets and ground truth using the Boolean operators as separators and perform a standard multi-label classification: dtc1 & dtc2 & !dtc5|dtc3=â[dtc1, dtc2, dtc3, dtc5] This evaluation is the same as the one described in Part IIfor the causal discovery algorithms recovering Markov Boundaries, i.e., sets of causes of an EP. 9.4.1.2 Semantic Evaluation For (2) we enumerate all possible assignments of the present Boolean variables and compute the truth table of the estimated rules and the ground truth to express:Is the rule logically correct ?We calculate the accuracy, precision, recall, F1 of the predicted value of the truth tables (e.g., Tab 9.1). x 1 x 2 Ground Truth (x 1 &x 2 )Predicted Rule (x 1 |x 2 ) 0000 0101 1001 1111 Table 9.1:Illustrative truth table for semantic evaluation. The accuracy using the predicted rule is 50% . 9.4.2 Settings We used the same experimental setup and dataset as in Chapters6and7. We created 5 folds of50,000sequences and randomly masked20different EPs with at leastnâ„100 sequences. We set their masked rule as the ground truth for each labely un j and average the results across the five folds. We used a combination of F1-Score, Precision, and Recall with different averaging methods [ 172] to compare with LLM-only baselines. 9.4.3 Comparison We compared CAREP against multiple LLMs, such as Claude Sonnet 3.5 and 3.7 3 , GPT4.1 and GPT4.1 mini 4 . We further investigated whether smaller LLMs [13] with fewer parameters impacted the evaluation. The LLMs are compared using only the observed DTC sequences and their descriptions, hence in Fig.9.2the orange region is removed. We also added CAREP without the causal indicators. We then performed the structural and semantic evaluation of the generated Boolean rules presented in the 3 https://w.anthropic.com/news/claude-3-7-sonnet 4 https://openai.com/index/gpt-4-1/ 123 9 Multi-Agent Causal Reasoning for Automated Error Pattern Rule Synthesis 01234 shard 0 20 40 60 80 100 120 140 complexity Rule Complexity Distribution by Shard 020406080100 num_codes 0 200 400 600 800 1000 Count Distribution of Rule Length (# DTCs) num_and num_or num_not 0 1000 2000 3000 4000 5000 6000 Total Operator Count in Dataset Figure 9.3:Analysis of Ground Truth Error Patterns.(Left)Complexity Distribu- tion by Fold:The consistency of rule complexity scores across fold confirms a balanced data split strategy. (Middle)Rule Length Histogram:The distribu- tion of Diagnostic Trouble Codes (DTCs) per rule reveals a long-tail nature, with some complex rules involving over 20 unique codes. (Right)Operator Frequency: The significant presence of logical NOT (ÂŹ) and OR (âš) operators highlights the non-trivial, inhibitory causal structures. foundation chapter. The top-3and top-5metrics are computed from the five estimates, ordered by decreasing confidence (high, medium, low), to provide a clearer picture. We average the metrics across labels (macro average). 9.4.4 Results Figure9.4reports the performance of our method CAREP against multiple LLM base- lines (Claude Sonnet 3.5/3.7, GPT4.1, and GPT4.1 mini). CAREP consistently outper- forms all standalone LLMs across both evaluation protocols. The ground-truth EP rules used for evaluation were manually defined by diagnostic experts and may contain noise or inconsistencies. Results should be interpreted accordingly. Semantic evaluationIn terms of truth-table agreement, CAREP substantially im- proves over LLMs in capturing the logical structure of error patterns. CAREP achieves aRecall@1of0.70, compared to0.25for the best-performing baseline (Claude Sonnet 3.7). Precision and F1 scores follow the same trend, showing that CAREPâs causal dis- covery step yields more faithful Boolean rules rather than over-generalized expressions. Interestingly, LLMs still reachâ0.50semanticAccuracy@5, indicating that DTCsâ ob- servations and descriptions alone allow them to approximate the ground truth partially. However, they lack the consistency and reliability required for deployment that CAREP provides. Structural evaluationWhen evaluating whether the predicted rules include the correct DTCs, CAREPâs advantage becomes even more pronounced. ForPrecision@1, CAREP achieves0.78versus only0.25for Claude Sonnet 3.7 and0.05for GPT4.1 mini. Similar margins are observed in F1 and accuracy. GPT4.1 and mini, in particular, perform 124 9.5 Summary Recall@1Recall@3Recall@5 0.0 0.5 1.0 0.25 0.38 0.41 0.14 0.25 0.30 0.12 0.12 0.13 0.11 0.120.12 0.53 0.54 0.61 0.70 0.780.78 (a) Semantic Evaluation Recall@1Recall@3Recall@5 0.26 0.48 0.52 0.12 0.19 0.22 0.07 0.110.11 0.12 0.220.22 0.50 0.510.51 0.47 0.57 0.61 (b) Structural Evaluation F1@1F1@3F1@5 0.0 0.5 1.0 0.29 0.42 0.46 0.19 0.30 0.33 0.15 0.15 0.16 0.16 0.180.18 0.53 0.55 0.59 0.58 0.660.66 F1@1F1@3F1@5 0.22 0.40 0.45 0.11 0.20 0.24 0.06 0.100.10 0.08 0.130.13 0.48 0.50 0.52 0.48 0.58 0.60 Precision@1Precision@3Precision@5 0.0 0.5 1.0 0.40 0.55 0.59 0.34 0.43 0.44 0.29 0.290.29 0.31 0.360.36 0.64 0.670.67 0.60 0.700.70 Precision@1Precision@3Precision@5 0.25 0.47 0.53 0.12 0.30 0.33 0.07 0.120.12 0.10 0.130.13 0.750.75 0.79 0.78 0.830.83 Accuracy@1Accuracy@3Accuracy@5 0.0 0.5 1.0 0.48 0.56 0.58 0.46 0.480.48 0.45 0.450.45 0.45 0.460.46 0.64 0.68 0.69 0.65 0.770.77 Accuracy@1Accuracy@3Accuracy@5 0.16 0.29 0.33 0.08 0.15 0.17 0.04 0.060.06 0.05 0.080.08 0.37 0.38 0.40 0.38 0.46 0.48 Claude Sonnet 3.7 Claude Sonnet 3.5 GPT4.1 GPT4.1 mini CAREP (w/o causal indicators) CAREP Figure 9.4:Performance Comparison of CAREP Against Standalone LLMs. (a) Se- mantic evaluation represents how well the estimated error pattern rules fit the ground truth in terms of Boolean expression. (b) Structural evaluation or multi- label classification reveals if the estimated rules contain the correct DTCs. poorly (F1@1<0.1), demonstrating that without causal discovery, LLMs fail to correctly identify the true DTCs present in the error pattern rules. In addition, unlike OSCAR and CARGO which were evaluated previously using only the structural evaluation, CAREP benefits from the contextual information and the multiple output Boolean rules since it obtains75%Precision@1and83%Precision@5contrary to60%Precision(Table 7.1) for CARGO and55%for OSCAR. 9.5 Summary We introduced CAREP, a multi-agent causal-reasoning framework that automates the discovery of error pattern rules from large-scale event sequences of error codes. By combining causal discovery, RAG systems, and agents, CAREP consistently outper- 125 9 Multi-Agent Causal Reasoning for Automated Error Pattern Rule Synthesis MethodPrecisionâRecallâF1âTime (min)â Phase 1: Sample-Level Causal Discovery (OSCAR) Instance GraphG (i) 55±1.431±0.8 40±1.011.7 Phase 2: Population-Level Causal Discovery (CARGO) Consensus GraphG â y 61±1.546±1.7 46±1.211.8 Phase 3: CAREP (End-to-End) CAREP83±1.1 60±1.3 61±1.112 Table 9.2:Ablation of causal discovery performance across the pipeline stages: (1) Local Discovery, (2) Global Aggregation, and (3) The complete framework (CAREP). The baselines are evaluated using the structural (does the rule contain the correct variables, ignoring the logic). âTimeâ represents cumulative runtime. By adding semantic information (metadata) and LLMsâ reasoning, CAREP adds a sig- nificant prediction margin in selecting the correctcausesof an outcome (EP). forms state-of-the-art LLM baselines while producing interpretable reasoning traces es- sential for safety-related deployment. The ablation across the three pipeline stages (Table9.2) confirms that neither causal structure alone nor language reasoning alone is suïŹicient: the performance gap between CARGO and CAREP demonstrates that con- textual grounding is indispensable for correct Boolean rule synthesis. We emphasize, however, that CAREP is designed as an assistive tool rather than a fully autonomous decision-maker. Given the high-stakes nature of automotive diagnostics, unsupervised rule deployment carries inherent risks; therefore, human experts must remain in the loop to validate and supervise the systemâs outputs. The methods and findings of this chap- ter therefore conclude the full pipeline of this thesis, from raw event-sequence modeling through causal structure recovery to symbolic rule generation, leaving the question of how to scale, adapt, and generalize this pipeline as the central open problem motivating the directions below. 9.6 Outlook Several directions offer high potential for extending CAREP toward a more general and robust automated reasoning system. A first and most immediate direction is to replace the events-to-outcome causal backbone (OSCAR and CARGO) with aunified causal graphthat jointly encodes event-to-event and events-to-outcome dependencies, the open cell of Table1.1. Such a graph would allow CAREP to reason not only about which DTCs cause a given error pattern, but also about the cascading sequence of fault propagation leading to it. Incorporating event-to-event causal edges from TRACE into CAREPâs causal discovery agent is a natural first step; the richer structural evidence would enable the orchestrator agent to synthesize temporally ordered Boolean rules, capturing not just the presence but the sequential ordering of causes, a property that current EP rules do not yet cover. Scaling this to the full population-level setting, where a unified DAG 126 9.6 Outlook over both event types and outcome labels is aggregated across a fleet, constitutes the primary research goal of theseq2causeproject [88]. A second direction concerns the language backbone of the reasoning agents. The current CAREP implementation relies on general-purpose LLMs that lack the specialized knowledge of automotive subsystems, DTC taxonomy, descriptions and diverse technical jargon. Domain-adapted language models, pretrained or continuously fine-tuned on automotive documentation, service manuals, and fault-knowledge bases would reduce hallucination rates and improve the semantic fidelity of generated rules, particularly for rare error patterns where contextual retrieval alone is insuïŹicient. 127 129 10 Conclusion and Outlook This thesis demonstrated that high-dimensional discrete event sequences, long treated as intractable by statistical models and causal discovery methods, can be systematically modeled, causally analyzed, and transformed into interpretable diagnostic knowledge us- ing a unified pipeline of autoregressive Transformers, information-theoretic causal tests, and multi-agent reasoning. 10.1 Conclusion This thesis aimed to address one of the core challenges in modern industrial systems: transforming massive, high-dimensional event streams into interpretable and actionable knowledge for practitioners and researchers. By focusing on the automotive domain, a setting that epitomizes complexity, scale, and safety-critical constraints, we progressively built a pipeline that evolves from sequence modeling to causal discovery and, finally, to automated reasoning for error pattern automation. In PartI, we demonstrated that event sequences, such as diagnostic trouble codes (DTCs), can be modeled as a language. By introducing CarFormer and EPredictor, we showed that Transformer-based architectures can accurately learn spatio-temporal dependencies in irregular, unbalanced event streams and forecast both when and what failures are likely to occur. We also showed that incorporating additional modalities, such as sensor readings, improves predictive performance with BiCarFormer, a bidirectional Transformer model. This laid the foundation for predictive maintenance systems capable of real-time, self-diagnostic behavior. In Part I, we moved beyond prediction to understanding. We introduced three com- plementary frameworks for causal discovery in sequences: OSCAR and TRACE, which perform sample-level causal discovery using autoregressive Transformers as density esti- mators, and CARGO, which aggregates these sample-level graphs into coherent global causal graphs. A key technical insight underpinning this is the functional duality be- tween prediction and causal discovery: an autoregressive model trained to predict the next event implicitly learns the joint distribution of the sequence, and can therefore be directly reused, without any fine-tuning, as a neural density estimator for conditional mutual information estimation. Critically, the teacher-forcing mechanism of sequence- to-sequence training (i.e., conditioning on observed rather than model-generated history) allows allNconditional log-probabilities required for a CMI estimate to be computed in a single forward pass, collapsing what would otherwise beNsequential inference calls into a single vectorized operation and making the approach tractable on the scale of tens of thousands of distinct events. Together, they enable, for the first time, causal dis- covery in sequences with a considerable number of distinct events and labels, exceeding 131 10 Conclusion and Outlook tens of thousands of DTCs and hundreds of error patterns in our experiments. These data settings have not been explored in the literature and remain a major obstacle to the adoption of causal discovery in large-scale datasets. These three methods are an- chored in modern parallelization of algorithms on GPUs, where Transformers and more generally autoregressive models are the fundamental starting point. The methods are implemented within common frameworks such as Hugging Face and PyTorch. This re- sults in three practical causal discovery methods that practitioners can deploy alongside their event sequence model, and scale its infrastructure using only GPUs. In addition, they are theoretically grounded in causal discovery and probability theory. We proved both experimentally and theoretically that a perfect oracle model is not required to recover the causal graph and provided the soundness of our approaches under several assumptions. We analyzed the limitations of such assumptions and proposed robust methods, such as adaptive thresholding for aggregating graphs with imbalanced label distributions. Finally, in Part I, we transitioned toward automating error patterns in vehicles, a complex reasoning task. We highlighted how we could build an entire framework powered by eïŹicient causal discovery, contextual information retrieval, and LLMs to reason over dense, multimodal inputs. We show that a multi-agent-based system (CAREP) can generate plausible rules that are highly similar to those created by domain experts. Beyond their direct automotive applications, the methods proposed in this dissertation have broader implications. The combination of autoregressive modeling, scalable causal inference, and multi-agent reasoning opens new pathways for domains such as healthcare, cybersecurity, and industrial process monitoring, wherever complex event-driven systems demand reliable and explainable decision support. It is important to acknowledge the assumptions and limitations that bound these contributions. The causal discovery frameworks developed in Part I rely on causal suïŹiciency, the absence of hidden confounders, which may not hold in all industrial deployments where unmeasured variables influence observed events. The evaluation of CAREP in Part I is conducted against EP rules that were themselves constructed by domain experts; while this provides a practical benchmark, it introduces a degree of circularity that future work should address through independent validation. Finally, all experiments are conducted on proprietary BMW data, which, while ensuring industrial relevance, limits direct reproducibility; the open-source seq2cause library and accompa- nying synthetic datasets are provided to partially mitigate this constraint. In summary, this dissertation establishes a clear progression from statistical learning to causal understanding and automated reasoning. By integrating large-scale modeling with causal discovery and agent-based frameworks, we provided a unified foundation for the next generation of autonomous systemsâones capable not just of prediction, but of interpretable, causally-aware decision-making across science and industry. 132 10.2 Outlook 10.2 Outlook The work presented in this thesis brings together sequence modeling, causal discovery, and multi-agent reasoning into a unified framework for automated fault understanding in industrial-scale event sequences. While the proposed methods demonstrate strong performance in automotive diagnostics and point toward a new paradigm for large- scale event understanding, many open problems remain, both theoretical and practical. These challenges define a rich landscape for future research that extends far beyond the automotive domain. 10.2.1 Toward a Unified Causal Graph over Events and Outcomes Table1.1organizes the causal discovery landscape of this thesis along two axes: causal scope (sample-level vs. population-level) and dependency type (events-to-outcome vs. event-to-event). Three of the four cells are addressed by the contributions of this thesis (OSCAR, TRACE, and CARGO), yet the fourth cell, population-level event-to-event discovery, remains perhaps the most scientifically important open problem. Filling the open cell requires aggregating instance-time causal graphs produced by TRACE across thousands of sequences into a single, general consistent summary graph over event types. This is the natural event-to-event counterpart of CARGO, and its realization faces three non-trivial obstacles. First, instance-time graphs encode when a causal edge fires within a specific sequence, making naive frequency-based aggregation sensitive to temporal in- consistencies across sequences of varying length and context. Second, enforcing acyclicity constraints on a graph with tens of thousands of nodes after aggregation is computa- tionally demanding and requires careful post-processing beyond what CARGOâs Phase 2 currently provides. Third, robust uncertainty quantification over aggregated edges, such as distinguishing persistent structural causes from spurious co-occurrences driven by rare events demands principled statistical thresholding at a scale that has not yet been stud- ied. However, filling this cell individually still leaves the deeper problem unresolved: the two dependency types are not independent. In practice, a DTC may causally trigger an- other DTC (event-to-event), which in turn causes an error pattern (events-to-outcome), forming a causal chain that spans both columns of Table 1.1simultaneously. At the sample-level, OSCAR and TRACE can in principle be run jointly on the same sequence, but their outputs, a Markov Boundary per label and an instance-time causal graph over events are not currently unified into a single coherent object. At the population-level, the problem compounds: one would require a mixed graph that simultaneously encodes directed edges between event types and directed edges from event types to outcome la- bels, with consistent semantics across both. We refer to this as the joint population-level causal graph, and it constitutes the most ambitious open problem in this line of work. Such a graph would allow a practitioner to trace a complete causal path from a root- cause DTC, through intermediate triggering events, to the final error pattern entirely from observational data, without any manual rule definition. Addressing this problem will require extending the aggregation framework of CARGO to handle heterogeneous node types (events and labels), developing acyclicity constraints that respect the asym- 133 10 Conclusion and Outlook metry between the two node classes, and designing evaluation protocols that go beyond current set-based and Hamming-distance metrics. The seq2cause library is architected with this unification in mind, and we consider the joint population-level causal graph the single most impactful direction for future research emerging from this thesis. 10.2.2 Toward Multimodal and Multi-Resolution Causal Discovery Throughout this thesis, causal discovery primarily operates on discrete event streams possibly with outcomes. Yet modern vehicles, like many stochastic systems, generate multimodal data, including environmental conditions, textual descriptions, or camera- based observations. Extending causal discovery to this multimodal setting, by combin- ing discrete events with continuous signals, images, or text, would enable richer causal graphs that can explain failures through finer-grained interactions. Foundations from multimodal Transformers, contrastive pretraining, and cross-modal attention could sup- port such expansion. A promising direction is causal fusion learning, where causal edges are allowed to form across modalities and at multiple temporal resolutions (millisecond sensor bursts vs. day-level DTCs), similar to summary graphs in time series [5]. This would help address phenomena such as cascading failures arising from the joint inter- action of software and hardware subsystems, and enrich the EP rules by incorporating environmental conditions. 10.2.3 Unseen Domain and Continuous Learning A central challenge for deploying intelligent diagnostic systems in real-world environ- ments is their ability to generalize beyond the distribution from which they were trained. Vehicles evolve across software versions, model years, component suppliers, and even regional driving conditions. Models trained once and deployed indefinitely, whether sequence predictors, causal discovery frameworks, or multi-agent reasoning systems, in- evitably face degradation as the data distribution drifts. Such a system needs to con- tinuously learn from new data while retaining previously acquired knowledge. This opens a rich avenue for future research at the intersection of continual learning and multi-agent adaptation. Classical continual learning methods based on regulariza- tion, replay, or architectural expansion [ 43] provide the first building blocks, but remain insuïŹicient for highly structured, relational tasks such as diagnostic rule synthesis. More recent theoretical developments, such as the âNested Learningâ perspective [12], chal- lenge the assumption that deep architectures inherently build hierarchical abstractions, suggesting instead that models must be explicitly encouraged to discover stable mecha- nisms that persist across contexts. They use different updating frequencies of neurons across layers, mimicking brain oscillations. Future systems may therefore incorporate continual updates during Transformersâ training and enable smooth, continuous synthesis of EP rules, with agents detecting when new vehicles introduce unseen failure modes, unseen DTC combinations, or entirely novel clusters of behaviors. Multi-agent architectures are particularly promising in this setting: one agent may monitor distribution shift, another may propose updated causal 134 10.2 Outlook graphs for new domains. Beyond adaptation, such systems could also identifywhen previously learned EP rules should be revised, pruned, or merged, enabling long-term autonomous maintenance of diagnostic knowledge bases. 10.2.4 Industrial Foundation Models Recent progress in large language models raises the question of whether analogous foun- dation models for event sequences can be developed. Such models would unify sequence modeling, forecasting, causal inference, anomaly detection, and reasoning capabilities across domains such as healthcare, cybersecurity, aviation, and industrial operations. The introduced Transformer-based architectures, coupled with causal discovery tools, could serve as stepping stones toward such foundation models. Pretraining on large- scale, heterogeneous event-log corpora, comparable to natural-language corpora, would enable these models to generalize across systems and support in-context learning, zero- shot, or few-shot diagnostics on unseen devices, procedures, or scenarios. It is essential to strike a balance between the world of words and events through adapted pretraining and fine-tuning tasks. Recent approaches adapt diffusion models for discrete data [ 128]. This could help build more robust bidirectional models rather than traditional left-to-right models [105], particularly for sequence classification and machine translation tasks. 135 List of Figures 1.1Error Pattern Prediction (when and what). Based on the past sequenceSof diagnostic trouble codes (DTCs), manufacturers want to prevent error patterns (EPs) from happening by predicting their likelihood and time of occurrence.. . . . . . . . . . . . . . . . . . . . . . . . . . . .5 1.2Error Pattern Prediction using Multimodal Sequences. Past di- agnostic trouble codes only provide limited information about complex and overlapping error patterns. In addition, domain experts rely on envi- ronmental conditions (e.g., temperature, voltage, humidity, ...) to identify more accurately error patterns.. . . . . . . . . . . . . . . . . . . . . . . .6 1.3Anonymized Causal Graph.Example of a sequence of events (DTCs) that lead to a steering wheel degradation and a power limitation as out- come labels. The causal indicators are shown invioletif inhibitory and orangeorredif excitatory, depending on the magnitude. The voltage error DTC is the biggest causal driver for the power limitation EP while the security sync failure seems to have the biggest inhibitory effect on all EPs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 1.4Anonymized Instance-Time Causal Graph for Diagnostic Defect Cascade.Temporal evolution of a diagnostic defect cascade in a vehicle (|X|â29,100). TRACE effectively captures causal relationships, reveal- ing distincterror clustersat different time steps (e.g., initial sensor failures att= 3triggering mechanical faults att= 12, battery at issue t= 17). This enables actionable root-cause analysis by isolating the spe- cific onset of a failure mechanism and its strength using the conditional mutual information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 3.1Time and Mileage Distribution. Distribution oft i andm i (relative time and mileage) in our dataset.. . . . . . . . . . . . . . . . . . . . . . .31 3.2CarFormer Architecture. A Transformer architecture is used alongside spatio-temporal embeddings (context embedding) injected to the query and key. A RoPE is then applied to induce the relative position of the DTCs. Three predictive heads are used to learn the complex dynamics of the DTC sequences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 137 List of Figures 3.3EPredictor Architecture. The hidden stateH(S)is extracted from CarFormer and fed to the second attention layer as value and key. The event-type embeddingEis fed into the first layer after being normalized, and the context embeddingCEcarrying the spatio-temporal information is added directly after the RoPE application to the keys and queries of the second layer.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36 3.4Classification Performance Comparison Between Architectural Changes. F1 Score comparison with multiple EPredictor architectures as a function of the number of observations. This answerswhatEP is most likely to occur. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40 3.5Regression Error As a Function of the Number of Observations. Evolution of the MAE as a function of the number of observations for the best performing model, this effectively tests forwhenan EP is most likely to occur.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41 4.1Spatio-temporal Point Process Representation of a Vehicle.Spa- tial (mileage) and Temporal (time) point process representation of events from a vehicle. Bold vertical lines indicate multiple events occurring at the same timet i or mileagem i .. . . . . . . . . . . . . . . . . . . . . . . .49 4.2BiCarFormer Architecture. Both parallel Transformers are comput- ing cross-attention scores conditioned on each modalityQ,K,V. Two final representations are generated for each modality: DTC (H d )and environmental conditions (H e ). Multiple hierarchical embeddings are defined at the input level to account for DTC-specific features (ECUs, Fault-Bytes, Base-Code). . . . . . . . . . . . . . . . . . . . . . . . . . . .50 4.3Attention Distribution between Environmental Condition To- kens.Amount of attention received by each environmental condition token from the DTCs. The y-axis was truncated to improve clarity as well as the number of heads printed. We takeA dtcâenv of the last layer..53 4.4Cross-Attention Scores forA dtcâenv .The DTCs are shown on the y-axis (anonymized), and the environmental conditions with their 3 ele- ments(d, v, u)concatenated are shown on the x-axis (the descriptiond is anonymized). The intensity of each cell reflects the attention weight, where darker shades indicate higher attention values.. . . . . . . . . . . .54 4.5Attention Distribution between Environmental Conditional Units. Amount of attention received by each environmental unit from the DTCs in abattery agingerror pattern from a battery electric vehicle. We take A dtcâenv of the last layer and print all heads. . . . . . . . . . . . . . . . .55 138 List of Figures 4.6Discretized Voltage and Attention Scores Evolution between En- vironmental Token Position. Discretized voltage variation as a func- tion of environmental condition tokens and the associated amount of at- tention received from DTCs. We only take triplets(d, v, u)fromS e with u=âvâ and extractA dtcâenv from the last layer andhead 2. The sample is taken from abattery agingerror pattern. BiCarFormer implicitly learns the fluctuation of discretized continuous values to make a prediction.. . .56 6.1The Overview of OSCAR:One-Shot multi-labelCausalAutoRegressive discovery.ddenotes the hidden dimension,Lthe sequence length,MB 1 ,MB 2 the Markov Boundary ofY 1 , Y 2 respectively. All green and blue areas rep- resent parallelized operations. . . . . . . . . . . . . . . . . . . . . . . . . .72 6.2Example of a Causal Graphextracted from a multi-labeleventse- quence whereMB 1 represents the Markov Boundary ofY 1 andMB 2 the Markov Boundary ofY 2 .. . . . . . . . . . . . . . . . . . . . . . . . . . . .73 6.3Evolution of the Classification Performance as a Function of the Markov Boundary Length|MB(Y j )|(n= 45969samples). We can identify that the Markov Boundary complexity is not the bottleneck but rather the number of samples per class ( imbalance threshold), which reduces classification performance.. . . . . . . . . . . . . . . . . . . . . .82 7.1Structural Fusion Example: individual set of causal graphsG (k) (left) aggregated into a fused DAG for multi-label event sequences (right) using a simple union.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86 7.2Overview of CARGO. Phase 1 (one-shot [94] or sample-level) is on top, and Phase 2 (Adaptive Thresholding) is on the bottom.ddenotes the hidden dimension,Lthe sequence length,mthe number of samples andMB 1 ,MB 2 the Markov Boundary ofY 1 , Y 2 . All green and blue areas are parallelized. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87 7.3Adaptive Thresholding FunctionÏ j (m j )across varying supportm j , illustrating the logistic decay fromÏ max toÏ min . . . . . . . . . . . . . . . .89 7.4Criterion Ablation for Structural Fusion. Comparison of different criteria for the structural fusion (Phase 2) as a function of the number of samplesm. With|Y|= 474,|X|= 29,100nodes.. . . . . . . . . . . . . .91 8.1Methodological Shift. (A) Traditional Causal Discovery in Se- quences(e.g., PCMCI, Hawkes, Granger) relies on iterative solvers (CI- tests) over long multivariate time series (Tâ â).(B) Our TRACE Approachprocesses a single sequence (e.g., event logs, user interactions, patient trajectories) through a pretrained autoregressive (AR) model as density estimator to compute the Conditional Mutual Information (CMI) in parallel, enabling scalable causal discovery over massive vocabularies (|X|>1000). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94 139 List of Figures 8.2TRACE Methodology. Phase 1 (Training):An autoregressive (AR) model (e.g., LM, RNN) is pretrained on a corpus of event sequences via next-token prediction to learn the process dynamics (P Ξ ).Phase 2 (In- ference):A single sequencesis passed through the frozen model. We then estimate conditional mutual information (Parallelized CMImod- ule) to prune non-causal edges and form theInstance-Time Causal Graph G t,s . Finally, this graph is projected onto the event types to recover the Summary Causal GraphG s . . . . . . . . . . . . . . . . . . . . . . . . . . .96 8.3Overview of TRACE Parallel CI-tests.We construct a single broad- casted tensorX do where each rowjincrementally fixes the historyx â€j while randomizing the future (staircase pattern). The model processes this tensor in parallel to produce raw probabilitiesP raw (grey). We then compute the Causal Mutual Information by comparing adjacent rows: the distribution at rowjâ1serves as the baseline (P base , blue) for the intervention at rowj(P do , red).. . . . . . . . . . . . . . . . . . . . . . .103 8.4Scalability to High-Dimensional Event Spaces.Evaluation of struc- tural identifiability across exponentially growing vocabulary sizes.Top: Evolution of discovery metrics. TRACE exhibitsperformance invari- ance, maintaining F1â0.81even as the combinatorial search space ex- plodes.Bottom:Visual examples of recovered summary graphsG s at scale. Shannon redundancy as (Pred=1âH(P)/H max ) confirm that TRACE succeeds even in high-entropy regimes. (ËΔ= 0.01, L= 64, N= 64, Ï= 10 â4 ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .107 8.5Robustness and Scalability Analysis(|X|= 1000, N= 128, Ï= 10 â4 , L= 64). Evolution of causal discovery performance (F1, Precision, Recall, SHD).(a) Robustness to Generative Error:Performance as a function of the modelâs oracle scoreΔ. TRACE exhibits a phase transition, recovering structure even for imperfect models (Δ <0.1) and maintain- ing high Precision even as fidelity degrades.(b) Scalability to Length: Performance and GPU memory usage vs. sequence length L . The Sparse variant demonstrates linear memory scaling (O(mL)), enabling inference on sequences far exceeding the training length (L= 64), whereas the Full variant scales quadratically.(c) Long-Range Dependencies:Robust- ness to increasing delayed-effectsm. TRACE maintains F1>0.8even as dependencies span one third of the sequence (m= 20), confirming the methodâs ability to capture distant causal mechanisms. . . . . . . . . . . .108 8.6Instance-Time Causal Graph.Temporal evolution of a diagnostic defect cascade in a vehicle (|X| â29,100). TRACE effectively captures causal relationships, revealing distincterror clustersat different time steps (e.g., initial sensor failures att= 3triggering mechanical faults att= 12, battery at issuet= 17). This enables actionable root-cause analysis by isolating the specific onset of a failure mechanism and their strength using the CMI Ë I N . . . . . . . . . . . . . . . . . . . . . . . . . .109 140 List of Figures 8.7Robustness to Hidden Confounding.Evaluation of TRACE under violations of causal suïŹiciency.(a)Measurement Error: Random noise is injected into the context. Precision stays high, indicating resistance to hallucination.(b)Temporal Drops: Time steps are randomly deleted, thus conditioning sets are broken. TRACE still recovers structure despite missing intermediaries but the discovery scores quickly decrease.. . . . .110 9.1Illustration of Error Pattern Automation in Modern Vehicles. (a) A vehicle with an unknown defect generates a sequence of Diagnostic Trouble Codes (DTCs) over time. (b) A domain expert analyzes the DTCs and the vehicleâs metadata (e.g., model, descriptions). (c) Provide a Boolean rule to identify this error pattern. . . . . . . . . . . . . . . . . .118 9.2CAREP: A Causal Reasoning Agentic System for Error Pat- tern Automation. (a) Represents the causal discovery phase, where we extract candidate DTC causes for the unknown error pattern alongside causal indicators. It then feeds the causal reasoner agent. (b) The de- scriptions and metadata are extracted through a Titan V2 Embeddings and fed into the contextual information agent. (c) The orchestrator agent manages the two agents and provides traceable reasoning to explain why the unknown error pattern may match these rules.. . . . . . . . . . . . .121 9.3Analysis of Ground Truth Error Patterns.(Left)Complexity Distribution by Fold:The consistency of rule complexity scores across fold confirms a balanced data split strategy. (Middle)Rule Length Histogram:The distribution of Diagnostic Trouble Codes (DTCs) per rule reveals a long-tail nature, with some complex rules involving over 20 unique codes. (Right)Operator Frequency:The significant pres- ence of logical NOT (ÂŹ) and OR (âš) operators highlights the non-trivial, inhibitory causal structures. . . . . . . . . . . . . . . . . . . . . . . . . . .124 9.4Performance Comparison of CAREP Against Standalone LLMs. (a) Semantic evaluation represents how well the estimated error pattern rules fit the ground truth in terms of Boolean expression. (b) Structural evaluation or multi-label classification reveals if the estimated rules con- tain the correct DTCs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125 A.1Confident Predictive Maintenance Window Evolutionwith and w/o the random head.. . . . . . . . . . . . . . . . . . . . . . . . . . . . .169 A.2Pretraining Loss Comparison. Base-DTC classification loss compari- son with and without multimodal learning.. . . . . . . . . . . . . . . . .169 B.1Ablation of the Number of ParticlesN. Results are reported using 1-sigma error bar. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .173 B.2Sensitivity Analysis. Evolution of the sample-level F1 Score, Precision, and Recall as a function of coeïŹicientk. Results are reported using 1- sigma error bar. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .174 141 List of Figures B.3Ablation of the Number of ParticlesN. Evolution of causal dis- covery performance as a function of the number of particlesNat|X|= 1000, m= 6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182 B.4Sensitivity Analysis.Classification metrics relative to the filtering thresholdÏwith|X|= 1000, Δ= 0.04, c= 6, m= 6, L= 64, N= 128).. .182 B.5The Scaling Law of Causal Identifiability.We perform a sensitivity analysis across vocabulary sizes|X|â100, . . . ,2000to find the optimal filtering thresholdÏ opt (maximizing F1-score) withΔ= 0.04. The result reveals a power-law relationshipÏ opt â|X| â0.96 .. . . . . . . . . . . . . . .183 B.6Instance Summary Causal Graph.The global causal structureG s aggregated from TRACE inferences over the validation set. While the instance graph (Fig. 8.6) detailswhenevents occur, this summary graph captures the invariant mechanism types. The density of the graph high- lights the complexity of modern vehicle architectures, where high-degree nodes often represent central control units (e.g., ECU, Battery Manage- ment) that propagate cascading faults. The conditional mutual informa- tion is reported as causal strength.. . . . . . . . . . . . . . . . . . . . . .184 B.7Input Examples Given to the Agentic System. Thepotential causes are output by the causal discovery algorithm. For instance,DTC3is a cause ofunknown EPand has an Average Causal Effect (ACE) of 0.7, i.e., it increases the likelihood of observing the EP on average by 70% (Eq. (9.1)). Examples of defective vehicles exhibiting this error pattern are provided in theDTC samples. . . . . . . . . . . . . . . . . . . . . . . .185 142 List of Tables 1.1The four regimes of causal discovery in discrete event sequences, defined by the cross-product ofcausal scope(single-sequence inference vs. population-level aggregation) anddependency type(events-to-outcome vs. event-to-event). The three shaded contributions of this thesis âOS- CAR,TRACE, andCARGOâ share a common backbone: pretrained autoregressive Transformers repurposed as neural autoregressive density estimators (NADEs) for conditional mutual information estimation. The remaining cell, population-level event-to-event discovery, constitutes an open problem and a direct avenue for future work.. . . . . . . . . . . . .10 2.1 List of dataset symbols and their respective meanings for the datasets used in this thesis.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26 3.1 Number of Distinct values of the DTC elements. . . . . . . . . . . . . . .31 3.2 Overall prediction performance of CarFormer with different embeddings. Best results are in bold.. . . . . . . . . . . . . . . . . . . . . . . . . . . .38 3.3 EPredictor evaluation results with different model architectures on the test set (no up- nor down-sampling). The MAPE and F1 are shown in (%) 39 4.1 New symbols introduced in this chapter. For base DTC notation, see Table 2.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46 4.2 Number of distinct values after filtering for each feature.. . . . . . . . . .47 4.3 Downstream evaluation of multi-label error pattern classification. Each model has the same number of parameters (25M). Results are grouped by averaging type (micro/macro vs. sample). . . . . . . . . . . . . . . . . . .52 6.1 Comparisons ofMBretrieval withn= 50,000samples,|X|= 29,100,|Y|= 474averaged over6âfolds. Classification metrics averaging is âweightedâ and shown as one-shot for OSCAR. The symbol â-â indicates that the algorithm did not output theMBwithin 3 days. Metrics are given in%. 80 6.2 Comparisons ofMBretrieval withn= 500samples over9folds.. . . . .80 7.1 Comparisons ofMBretrieval withm= 50,000samples averaged over 6âfolds with|Y|= 474,|X|= 29,100nodes. Averaging is âweightedâ. The symbol â-â indicates that the algorithm did not output theMBs within 3 days. Metrics are given in%. OSCAR reports the sample-level performance (Phase 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90 143 List of Tables 8.1 Comparison of causal discovery methods.Discrete Events: Operates on discrete event sequences (e.g., text, logs) rather than multivariate time series.High Dim: Computationally tractable for large vocabu- laries (|X|>10 3 ).Non-Param.: Agnostic to the functional form (e.g., linearity) of causal relationships.Lags: Models delayed causal effects. Sample-Level: Infers a local causal graph specific to a single sequence, rather than a global graph.Linear Complexity: Complexity scales linearly with the vocabulary size|X|. . . . . . . . . . . . . . . . . . . . . .95 8.2Identifiability Comparison.Comparison of causal discovery perfor- mance on synthetic SCMs with|X|= 1000, L= 64, Δ= 0.05, Ï= 3.10 â5 , N= 128.TRACEsignificantly outperforms local (Saliency) and global (Granger/Shap- ley) baselines, achieving over 20 points higher F1 while maintaining high precision. Results across 10 runs are reported. . . . . . . . . . . . . . . . .106 9.1 Illustrative truth table for semantic evaluation. The accuracy using the predicted rule is50%.. . . . . . . . . . . . . . . . . . . . . . . . . . . . .123 9.2Ablation of causal discovery performance across the pipeline stages: (1) Local Discovery, (2) Global Aggregation, and (3) The com- plete framework (CAREP). The baselines are evaluated using the struc- tural (does the rule contain the correct variables, ignoring the logic). âTimeâ represents cumulative runtime. By adding semantic information (metadata) and LLMsâ reasoning, CAREP adds a significant prediction margin in selecting the correctcausesof an outcome (EP). . . . . . . . . .126 B.1 Ablations of the performance of Phase 1 (Sample-Level MB retrieval) as a function of different NADEs withn= 50,000andn= 500samples, aver- aged over 5-folds. Classification metrics use weighted averaging. Metrics are given in%. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172 B.2 Sample-level classification performance and elapsed time (sec) across dif- ferent sampling methods. Best results are inbold, and equal performance are in underline.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172 144 Bibliography [1]Massil Achab, Emmanuel Bacry, StĂ©phane GaĂŻffas, Iacopo Mastromatteo, and Jean-François Muzy. âUncovering causality from multivariate hawkes integrated cumulantsâ. In:J. Mach. Learn. Res.18.1 (Jan. 2017), p. 6998â7025.issn: 1532- 4435. [2]Abien Fred Agarap. âDeep Learning using Rectified Linear Units (ReLU)â. In: arXiv e-prints, arXiv:1803.08375 (Mar. 2018), arXiv:1803.08375.doi:10.48550/ arXiv.1803.08375. arXiv:1803.08375 [cs.NE]. [3]C.F. Aliferis, Ioannis Tsamardinos, and A Statnikov. âHITON: A Novel Markov Blanket Algorithm for Optimal Variable Selectionâ. In:AMIA ... Annual Sym- posium proceedings / AMIA Symposium. AMIA Symposium2003 (Feb. 2003), p. 21â5. [4]Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mer- cado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, and Yuyang Wang.Chronos: Learning the Language of Time Series. 2024. arXiv: 2403.07815 [cs.LG].url:https://arxiv.org/abs/2403. 07815 . [5]Charles K. Assaad, Emilie Devijver, and Eric Gaussier. âDiscovery of extended summary graphs in time seriesâ. In:Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence. Ed. by James Cussens and Kun Zhang. Vol. 180. Proceedings of Machine Learning Research. PMLR, 2022, p. 96â106. url: https://proceedings.mlr.press/v180/assaad22a.html. [6]Charles K. Assaad, Emilie Devijver, and Eric Gaussier. âSurvey and Evaluation of Causal Discovery Methods for Time Seriesâ. In:J. Artif. Int. Res.73 (May 2022).issn: 1076-9757.doi: 10.1613/jair.1.13428.url:https://doi.org/ 10.1613/jair.1.13428. [7]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.Layer Normalization. 2016. arXiv:1607.06450 [stat.ML].url:https://arxiv.org/abs/1607. 06450. [8]Gregor Bachmann and Vaishnavh Nagarajan. âThe Pitfalls of Next-Token Pre- dictionâ. In:ICLR 2024 Workshop: How Far Are We From AGI. 2024.url: https://openreview.net/forum?id=v9L38gCohh. 145 Bibliography [9]Emmanuel Bacry, Martin Bompaire, StĂ©phane Gaffas, and Jean-Francois Muzy. âSparse and low-rank multivariate Hawkes processesâ. In:Journal of Machine Learning Research21.50 (2020), p. 1â32.url:http://jmlr.org/papers/v21/ 15-114.html. [10]Vahid Balazadeh, Hamidreza Kamkari, Valentin Thomas, Junwei Ma, Bingru Li, Jesse C. Cresswell, and Rahul Krishnan. âCausalPFN: Amortized Causal Effect Estimation via In-Context Learningâ. In:The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2025.url:https://openreview.net/ forum?id=RblaNJGx8C. [11]Jasmijn Bastings and Katja Filippova. âThe elephant in the interpretability room: Why use attention as explanation when we have saliency methods?â In:Proceed- ings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. Ed. by Afra Alishahi, Yonatan Belinkov, Grzegorz Chru- paĆa, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad. Online: Association for Computational Linguistics, Nov. 2020, p. 149â155.doi:10.18653/v1/2020. blackboxnlp-1.14.url:https://aclanthology.org/2020.blackboxnlp- 1.14/. [12]Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. âNested Learning: The Illusion of Deep Learning Architecturesâ. In:The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2025.url:https: //openreview.net/forum?id=nbMeRvNb7A . [13]Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Mu- ralidharan, Yingyan Celine Lin, and Pavlo Molchanov.Small Language Models are the Future of Agentic AI. 2025. arXiv: 2506.02153 [cs.AI].url:https: //arxiv.org/abs/2506.02153 . [14]Iz Beltagy, Matthew E. Peters, and Arman Cohan.Longformer: The Long-Document Transformer. 2020. arXiv: 2004.05150 [cs.CL].url:https://arxiv.org/abs/ 2004.05150. [15]Yoshua Bengio and Samy Bengio. âModeling High-Dimensional Discrete Data with Multi-Layer Neural Networksâ. In:Advances in Neural Information Process- ing Systems. Ed. by S. Solla, T. Leen, and K. MĂŒller. Vol. 12. MIT Press, 1999. url: https://proceedings.neurips.c/paper_files/paper/1999/file/ e6384711491713d29bc63fc5eeb5ba4f-Paper.pdf. [16]Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Fe- ichtenhofer, and Judy Hoffman. âToken Merging: Your ViT But Fasterâ. In: The Eleventh International Conference on Learning Representations. 2023.url: https://openreview.net/forum?id=JroZRaRw7Eu. [17]Gianni Brauwers and Flavius Frasincar. âA General Survey on Attention Mech- anisms in Deep Learningâ. In:IEEE Trans. on Knowl. and Data Eng.35.4 (Apr. 2023), p. 3279â3298.issn: 1041-4347.doi: 10.1109/TKDE.2021.3126456.url: https://doi.org/10.1109/TKDE.2021.3126456. 146 Bibliography [18]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. âLanguage Models are Few-Shot Learnersâ. In:Advances in Neural Information Processing Systems. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin. Vol. 33. Curran Associates, Inc., 2020, p. 1877â1901. url:https://proceedings.neurips.c/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. [19]Patricia Cheng. âFrom Covariation to Causation: A Causal Power Theoryâ. In: Psychological Review104.2 (1997), p. 367â405.doi:10.1037/0033-295x.104. 2.367 . [20]David Maxwell Chickering. âLearning Bayesian Networks is NP-Completeâ. In: Learning from Data: Artificial Intelligence and Statistics V. Ed. by Doug Fisher and Hans-J. Lenz. New York, NY: Springer New York, 1996, p. 121â130.isbn: 978-1-4612-2404-4.doi:10.1007/978-1-4612-2404-4_12.url:https://doi. org/10.1007/978-1-4612-2404-4_12. [21]David Maxwell Chickering. âOptimal structure identification with greedy searchâ. In:J. Mach. Learn. Res.3.null (Mar. 2003), p. 507â554.issn: 1532-4435.doi: 10.1162/153244303321897717.url:https://doi.org/10.1162/153244303321897717. [22]Kyunghyun Cho, Bart van MerriĂ«nboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. âLearning Phrase Rep- resentations using RNN EncoderâDecoder for Statistical Machine Translationâ. In:Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP). Ed. by Alessandro Moschitti, Bo Pang, and Walter Daelemans. Doha, Qatar: Association for Computational Linguistics, Oct. 2014, p. 1724â1734.doi: 10.3115/v1/D14-1179.url:https://aclanthology.org/ D14-1179/. [23]T.M. Cover.Elements of Information Theory. Wiley series in telecommunications and signal processing. Wiley-India, 1999.isbn: 9788126508143.url:https:// books.google.de/books?id=3yGJrqyanyYC. [24]Joscha CĂŒppers, Sascha Xu, Musa Ahmed, and Jilles Vreeken. âCausal Discovery from Event Sequences by Local Cause-Effect Attributionâ. In:Advances in Neural Information Processing Systems37 (2024). [25]DeepSeek-AI.DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Re- inforcement Learning. 2025. arXiv:2501.12948 [cs.CL].url:https://arxiv. org/abs/2501.12948. 147 Bibliography [26]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. âBERT: Pre-training of Deep Bidirectional Transformers for Language Understandingâ. In:North American Chapter of the Association for Computational Linguistics. 2019.url:https://api.semanticscholar.org/CorpusID:52967399. [27]Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. âA Survey on In-context Learningâ. In:Proceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing. Ed. by Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen. Miami, Florida, USA: Association for Computa- tional Linguistics, Nov. 2024, p. 1107â1128.doi: 10.18653/v1/2024.emnlp- main.64 .url:https://aclanthology.org/2024.emnlp-main.64/. [28]Shuyu Dong, MichĂšle Sebag, Kento Uemura, Akito Fujii, Shuang Chang, Yusuke Koyanagi, and Koji Maruhashi. âDCILP: A Distributed Approach for Large-Scale Causal Structure Learningâ. In:AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA. Ed. by Toby Walsh, Julie Shah, and Zico Kolter. AAAI Press, 2025, p. 16345â16353.doi: 10.1609/AAAI.V39I15.33795.url:https://doi.org/ 10.1609/aaai.v39i15.33795. [29]Arnaud Doucet, Nando de Freitas, and Neil Gordon. âAn Introduction to Sequen- tial Monte Carlo Methodsâ. In:Sequential Monte Carlo Methods in Practice. Ed. by Arnaud Doucet, Nando de Freitas, and Neil Gordon. New York, NY: Springer New York, 2001, p. 3â14.isbn: 978-1-4757-3437-9.doi:10.1007/978-1-4757- 3437-9_1 .url:https://doi.org/10.1007/978-1-4757-3437-9_1. [30]Felix Draxler, Yang Meng, Kai Nelson, Lukas Laskowski, Yibo Yang, Theofanis Karaletsos, and Stephan Mandt. âTransformers for Mixed-type Event Sequencesâ. In:The Thirty-ninth Annual Conference on Neural Information Processing Sys- tems. 2025.url: https://openreview.net/forum?id=MtwsRjPZhf. [31]Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. âRecurrent Marked Temporal Point Processes: Embedding Event History to Vectorâ. In:Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD â16. New York, NY, USA: Association for Computing Machinery, 2016, p. 1555â1564.isbn: 9781450342322.doi: 10.1145/2939672.2939875.url:https://doi.org/10. 1145/2939672.2939875 . [32]Hao Duong Le, Xin Xia, and Zhang Chen. âMulti-Agent Causal Discovery Us- ing Large Language Modelsâ. In:arXiv e-prints, arXiv:2407.15073 (July 2024), arXiv:2407.15073.doi:10 . 48550 / arXiv . 2407 . 15073. arXiv:2407 . 15073 [cs.AI]. [33]Ellery Eells.Probabilistic Causality. Cambridge Studies in Probability, Induction and Decision Theory. Cambridge University Press, 1991. 148 Bibliography [34]Angela Fan, Mike Lewis, and Yann Dauphin. âHierarchical Neural Story Genera- tionâ. In:Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers). Ed. by Iryna Gurevych and Yusuke Miyao. Melbourne, Australia: Association for Computational Linguistics, July 2018, p. 889â898.doi:10.18653/v1/P18-1082.url:https://aclanthology. org/P18-1082/. [35]Branden Fitelson and Christopher Hitchcock. âProbabilistic Measures of Causal Strengthâ. In:Causality in the Sciences(Jan. 2010).doi:10.1093/acprof: oso/9780199574131.003.0029. [36]Holger Fröhlich and Gunnar W. Klau. âReconstructing Consensus Bayesian Net- work Structures with Application to Learning Molecular Interaction Networksâ. In:German Conference on Bioinformatics 2013. Ed. by Tim BeiĂbarth, Martin Kollmar, Andreas Leha, Burkhard Morgenstern, Anne-Kathrin Schultz, Stephan Waack, and Edgar Wingender. Vol. 34. Open Access Series in Informatics (OA- SIcs). Dagstuhl, Germany: Schloss Dagstuhl â Leibniz-Zentrum fĂŒr Informatik, 2013, p. 46â55.isbn: 978-3-939897-59-0.doi:10.4230/OASIcs.GCB.2013.46. url:https://drops.dagstuhl.de/entities/document/10.4230/OASIcs. GCB.2013.46. [37]Tian Gao and Qiang Ji. âLocal Causal Discovery of Direct Causes and Effectsâ. In:Advances in Neural Information Processing Systems. Ed. by C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett. Vol. 28. Curran Associates, Inc., 2015.url:https://proceedings.neurips.c/paper_files/paper/ 2015/file/fcdf25d6e191893e705819b177cddea0-Paper.pdf . [38]Tian Gao, Dharmashankar Subramanian, Karthikeyan Shanmugam, Debarun Bhattacharjya, and Nicholas Mattei. âA Multi-Channel Neural Graphical Event Model with Negative Evidenceâ. In:Proceedings of the AAAI Conference on Ar- tificial Intelligence34 (Apr. 2020), p. 3946â3953.doi: 10.1609/aaai.v34i04. 5810. [39]Sergio Garrido, Stanislav Borysov, Jeppe Rich, and Francisco Pereira. âEstimat- ing causal effects with the neural autoregressive density estimatorâ. In:Journal of Causal Inference9.1 (2021), p. 211â228.doi: doi:10.1515/jci-2020-0007. url:https://doi.org/10.1515/jci-2020-0007. [40]Jinwoo Go and Tobin Isaac. âRobust Expected Information Gain for Optimal Bayesian Experimental Design Using Ambiguity Setsâ. In: The 38th Conference on Uncertainty in Artificial Intelligence. 2022.url:https://openreview.net/ forum?id=HU9IxO8oqlc. [41]Chang Gong, Chuzhe Zhang, Di Yao, Jingping Bi, Wenbin Li, and YongJun Xu. âCausal Discovery from Temporal Data: An Overview and New Perspectivesâ. In: ACM Comput. Surv.57.4 (Dec. 2024).issn: 0360-0300.doi:10.1145/3705297. url:https://doi.org/10.1145/3705297. 149 Bibliography [42]Ian Goodfellow, Yoshua Bengio, and Aaron Courville.Deep learning. Vol. 1. MIT Press, 2016. [43]Ian J. Goodfellow, Mehdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio. âAn Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networksâ. In:CoRRabs/1312.6211 (2013).url:https://api.semanticscholar. org/CorpusID:12730344 . [44]C W J Granger. âInvestigating Causal Relations by Econometric Models and Cross-Spectral Methodsâ. In:Econometrica37.3 (July 1969), p. 424â438.url: https://ideas.repec.org/a/ecm/emetrp/v37y1969i3p424-38.html. [45]Michael Greenwald and Sanjeev Khanna. âSpace-eïŹicient online computation of quantile summariesâ. In:SIGMOD Rec.30.2 (May 2001), p. 58â66.issn: 0163- 5808.doi:10.1145/376284.375670.url:https://doi.org/10.1145/376284. 375670. [46]Peter Grunwald.A tutorial introduction to the minimum description length prin- ciple. 2004. arXiv:math/0406077 [math.ST].url:https://arxiv.org/abs/ math/0406077 . [47]Xianjie Guo, Kui Yu, Lin Liu, and Jiuyong Li. âFedCSL: A Scalable and Accurate Approach to Federated Causal Structure Learningâ. In:Proceedings of the AAAI Conference on Artificial Intelligence38.11 (Mar. 2024), p. 12235â12243.doi: 10.1609/aaai.v38i11.29113.url:https://ojs.aaai.org/index.php/AAAI/ article/view/29113. [48]Ishansh Gupta, Adriana Martinez, Sergio Correa, and Hendro Wicaksono. âA comparative assessment of causal machine learning and traditional methods for enhancing supply chain resiliency and eïŹiciency in the automotive industryâ. In: Supply Chain Analytics10 (2025), p. 100116.issn: 2949-8635.doi: https:// doi.org/10.1016/j.sca.2025.100116.url:https://w.sciencedirect. com/science/article/pii/S2949863525000160 . [49]Abdul Basit Hafeez, Eduardo Alonso, and Atif Riaz. âDTC-TranGru: Improving the performance of the next-DTC Prediction Model with Transformer and GRUâ. In:Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing (2024).url: https://api.semanticscholar.org/CorpusID:269951398. [50]Abdul Basit Hafeez, Eduardo Alonso, and Aram Ter-Sarkisov. âTowards Se- quential Multivariate Fault Prediction for Vehicular Predictive Maintenanceâ. In: 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA). 2021, p. 1016â1021.doi:10.1109/ICMLA52953.2021.00167. [51]Xiao Han, Saima Absar, Lu Zhang, and Shuhan Yuan. âRoot Cause Analysis of Anomalies in Multivariate Time Series through Granger Causal Discoveryâ. In:The Thirteenth International Conference on Learning Representations. 2025. url: https://openreview.net/forum?id=k38Th3x4d9. 150 Bibliography [52]Uzma Hasan, Emam Hossain, and Md Osman Gani. âA Survey on Causal Dis- covery Methods for I.I.D. and Time Series Dataâ. In:Transactions on Machine Learning Research(2023). Survey Certification.issn: 2835-8856.url:https: //openreview.net/forum?id=YdMrdhGx9y. [53]Alan G. Hawkes. âSpectra of some self-exciting and mutually exciting point pro- cessesâ. In:Biometrika58.1 (Apr. 1971), p. 83â90.issn: 0006-3444.doi: 10. 1093/biomet/58.1.83. eprint:https://academic.oup.com/biomet/article- pdf/58/1/83/602628/58-1-83.pdf.url:https://doi.org/10.1093/biomet/ 58.1.83. [54]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. âDeep Residual Learn- ing for Image Recognitionâ. In:Proceedings of 2016 IEEE Conference on Com- puter Vision and Pattern Recognition. CVPR â16. Las Vegas, NV, USA: IEEE, June 2016, p. 770â778.doi:10.1109/CVPR.2016.90.url:http://ieeexplore. ieee.org/document/7780459. [55]Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. âDeBERTa: Decoding- enhanced BERT with Disentangled Attentionâ. In:International Conference on Learning Representations. 2021.url:https://openreview.net/forum?id= XPZIaotutsD . [56]Rui He, Zhigang Tian, and Mingjian Zuo. âMachine prognostics under varying operating conditions based on state-space and neural network modelingâ. In:Me- chanical Systems and Signal Processing182 (2023), p. 109598.issn: 0888-3270. doi:https://doi.org/10.1016/j.ymssp.2022.109598.url:https://w. sciencedirect.com/science/article/pii/S0888327022006896 . [57]Weijie He, Xiaohao Mao, Chao Ma, Yu Huang, JosĂ© Miguel HernĂ ndez-Lobato, and Ting Chen. âBSODA: A Bipartite Scalable Framework for Online Disease Diagnosisâ. In:Proceedings of the ACM Web Conference 2022. W â22. New York, NY, USA: Association for Computing Machinery, 2022, p. 2511â2521. isbn: 9781450390965.doi: 10.1145/3485447.3512123.url:https://doi.org/ 10.1145/3485447.3512123. [58]Sepp Hochreiter and JĂŒrgen Schmidhuber. âLong Short-Term Memoryâ. In:Neu- ral Comput.9.8 (Nov. 1997), p. 1735â1780.issn: 0899-7667.doi: 10.1162/ neco.1997.9.8.1735.url:https://doi.org/10.1162/neco.1997.9.8.1735. [59]Jennifer A. Hoeting, David Madigan, Adrian E. Raftery, and Chris T. Volin- sky. âBayesian model averaging: a tutorialâ. In:Statistical Science14.4 (1999), p. 382â417. [60]Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. âThe Curious Case of Neural Text Degenerationâ. In:International Conference on Learning Representations. 2020.url:https://openreview.net/forum?id=rygGQyrFvH. 151 Bibliography [61]Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, Yusuf Roohani, Ryan Li, Lin Qiu, Junze Zhang, Yin Di, Shruti Marwaha, Jennefer Carter, Xin Zhou, Matthew Wheeler, Jon Bernstein, Mengdi Wang, Peng He, Jingtian Zhou, Michael Snyder, and Jure Leskovec.Biomni: A General-Purpose Biomedical AI Agent. June 2025.doi:10.1101/2025.05.30.656746. [62]David Hume.A Treatise of Human Nature. Ed. by P. H. Nidditch and L. A. Selby-Bigge. Reprinted from the original edition with analytical index. Oxford: Oxford University Press, 1978.isbn: 9780198245889. [63]Aapo HyvĂ€rinen, Kun Zhang, Shohei Shimizu, and Patrik O. Hoyer. âEstimation of a Structural Vector Autoregression Model Using Non-Gaussianityâ. In:Journal of Machine Learning Research11.56 (2010), p. 1709â1731.url:http://jmlr. org/papers/v11/hyvarinen10a.html. [64]Daniel Jiwoong Im, Kevin Zhang, Nakul Verma, and Kyunghyun Cho.Using Deep Autoregressive Models as Causal Inference Engines. 2024. arXiv:2409.18581 [cs.LG].url:https://arxiv.org/abs/2409.18581. [65]Sergey Ioffe and Christian Szegedy. âBatch normalization: accelerating deep net- work training by reducing internal covariate shiftâ. In:Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37. ICMLâ15. Lille, France: JMLR.org, 2015, p. 448â456. [66]Aryan Jadon, Avinash Patil, and Shruti Jadon. âA Comprehensive Survey of Re- gression Based Loss Functions for Time Series Forecastingâ. In:ArXivabs/2211.02989 (2022).url:https://api.semanticscholar.org/CorpusID:253383974. [67]Amirkasra Jalaldoust, Katerina HlavĂĄckovĂĄ-Schindler, and Claudia Plant. âCausal Discovery in Hawkes Processes by Minimum Description Lengthâ. In:Thirty- Sixth AAAI Conference on Artificial Intelligence; Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence; The Twelveth Symposium on Educational Advances in Artificial Intelligence. Proceedings of the ... National Conference on Artificial Intelligence 6. AAAI Press, June 2022, p. 6978â6987. isbn: 978-1-57735-876-3.doi: 10.48550/arXiv.2206.06124. [68]Dominik Janzing, David Balduzzi, Moritz Grosse-Wentrup, and Bernhard Schölkopf. âQuantifying causal influencesâ. In:The Annals of Statistics(Mar. 2012).doi: 10.1214/13-AOS1145. [69]Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. âCausal Rea- soning and Large Language Models: Opening a New Frontier for Causalityâ. In: Transactions on Machine Learning Research(2024). Featured Certification.issn: 2835-8856.url:https://openreview.net/forum?id=mqoxLkX210. [70]D. Koller and N. Friedman.Probabilistic Graphical Models: Principles and Tech- niques. Adaptive computation and machine learning. MIT Press, 2009.isbn: 9780262013192.url:https://books.google.co.in/books?id=7dzpHCHzNQ4C. 152 Bibliography [71]Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. âEstimating mu- tual informationâ. In:Phys. Rev. E69 (6 June 2004), p. 066138.doi:10.1103/ PhysRevE.69.066138.url:https://link.aps.org/doi/10.1103/PhysRevE. 69.066138. [72]Jorge D. Laborda, Pablo Torrijos, JosĂ© M. Puerta, and JosĂ© A. GĂĄmez. âA Ring- Based Distributed Algorithm for Learning High-Dimensional Bayesian Networksâ. In:Symbolic and Quantitative Approaches to Reasoning with Uncertainty: 17th European Conference, ECSQARU 2023, Arras, France, September 19â22, 2023, Proceedings. Berlin, Heidelberg: Springer-Verlag, 2023, p. 123â135.isbn: 978-3- 031-45607-7.doi: 10.1007/978-3-031-45608-4_10.url:https://doi.org/ 10.1007/978-3-031-45608-4_10 . [73]Helge Langseth and Luigi Portinale. âBayesian networks in reliabilityâ. In:Re- liability Engineering & System Safety92.1 (2007), p. 92â108.issn: 0951-8320. doi:https://doi.org/10.1016/j.ress.2005.11.037.url:https://w. sciencedirect.com/science/article/pii/S0951832005001985. [74]Patrick J. Laub, Young Lee, Philip K. Pollett, and Thomas Taimre. âHawkes Models and Their Applicationsâ. In:Annual Review of Statistics and Its Appli- cation12.Volume 12, 2025 (2025), p. 233â258.issn: 2326-831X.doi:https: //doi.org/10.1146/annurev-statistics-112723-034304.url:https:// w.annualreviews.org/content/journals/10.1146/annurev-statistics- 112723-034304 . [75]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich KĂŒttler, Mike Lewis, Wen-tau Yih, Tim RocktĂ€schel, Se- bastian Riedel, and Douwe Kiela. âRetrieval-augmented generation for knowledge- intensive NLP tasksâ. In:Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS â20. Red Hook, NY, USA: Curran Associates Inc., 2020.isbn: 9781713829546. [76]Jing Li, Botong Wu, Xinwei Sun, and Yizhou Wang. âCausal Hidden Markov Model for Time Series Disease Forecastingâ. In:2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)(2021), p. 12100â12109.url: https://api.semanticscholar.org/CorpusID:232417085. [77]Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino, Jiliang Tang, and Huan Liu. âFeature Selection: A Data Perspectiveâ. In:CoRR abs/1601.07996 (2016). arXiv:1601.07996.url:http://arxiv.org/abs/1601. 07996 . [78]Zhuoqun Li and Mingxuan Sun. âSparse Transformer Hawkes Process for Long Event Sequencesâ. In:Machine Learning and Knowledge Discovery in Databases: Research Track. Ed. by Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, and Francesco Bonchi. Cham: Springer Nature Switzerland, 2023, p. 172â188.isbn: 978-3-031-43424-2. 153 Bibliography [79]Stephanie Long, Tibor Schuster, and Alexandre PichĂ©. âCan Large Language Models Build Causal Graphs?â In:NeurIPS 2022 Workshop on Causality for Real- world Impact. 2022.url:https://openreview.net/forum?id=LQQoJGw8JD1. [80]Ilya Loshchilov and Frank Hutter. âDecoupled Weight Decay Regularizationâ. In:International Conference on Learning Representations. 2019.url:https: //openreview.net/forum?id=Bkg6RiCqY7 . [81]Sindy Löwe, David Madras, Richard Zemel, and Max Welling. âAmortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Dataâ. In:Proceed- ings of the First Conference on Causal Learning and Reasoning. Ed. by Bernhard Schölkopf, Caroline Uhler, and Kun Zhang. Vol. 177. Proceedings of Machine Learning Research. PMLR, 2022, p. 509â525.url:https://proceedings. mlr.press/v177/lowe22a.html. [82]Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. âViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasksâ. In: Proceedings of the 33rd International Conference on Neural Information Pro- cessing Systems. Red Hook, NY, USA: Curran Associates Inc., 2019. [83]Scott M. Lundberg and Su-In Lee. âA unified approach to interpreting model predictionsâ. In:Proceedings of the 31st International Conference on Neural In- formation Processing Systems. NIPSâ17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 4768â4777.isbn: 9781510860964. [84]Qian Luo, Lin Zhang, Zhiwei Xing, Huan Xia, and Zhao-Xin Chen. âCausal Dis- covery of Flight Service Process Based on Event Sequenceâ. In:Journal of Ad- vanced Transportation2021.1 (2021), p. 2869521.doi: https://doi.org/10. 1155/2021/2869521. eprint:https://onlinelibrary.wiley.com/doi/pdf/ 10.1155/2021/2869521.url:https://onlinelibrary.wiley.com/doi/abs/ 10.1155/2021/2869521. [85]Lin Ma, Liang Hu, Yonghao Li, Weiping Ding, and Wanfu Gao. âMI-MCF: A Mu- tual Information-Based Multilabel Causal Feature Selectionâ. In:IEEE Transac- tions on Neural Networks and Learning Systems(2025), p. 1â15.doi:10.1109/ TNNLS.2025.3556128. [86]Liam Daly Manocchio, Siamak Layeghy, Wai Weng Lo, Gayan K. Kulatilleke, Mo- hanad Sarhan, and Marius Portmann. âFlowTransformer: A transformer frame- work for flow-based network intrusion detection systemsâ. In: Expert Systems with Applications241 (2024), p. 122564.issn: 0957-4174.doi:https://doi.org/10. 1016/j.eswa.2023.122564.url:https://w.sciencedirect.com/science/ article/pii/S095741742303066X. [87]Yasmin Mansy, Nourhan Ehab, and Amr ElMougy. âThe Problem of Many Ve- hicles: An Explainable System for Autonomous Multi-agent Accidentsâ. In:In- telligent Systems and Applications. Ed. by Kohei Arai. Cham: Springer Nature Switzerland, 2024, p. 38â58.isbn: 978-3-031-66431-1. 154 Bibliography [88]Hugo Math.seq2cause: Sample- and Population-Level Causal Discovery from Event Sequences using Autoregressive Models. Version 0.1.4. 2026.doi:10.5281/ zenodo.19068730.url:https://doi.org/10.5281/zenodo.19068730. [89]Hugo Math and Rainer Lienhart. âContext-Informed Sequence Classification: A Multimodal Approach to Vehicle Diagnosticsâ. In:1st ICLR Workshop on Time Series in the Age of Large Models. 2026.url: https://openreview.net/forum? id=G4iAE9xOpb. [90]Hugo Math and Rainer Lienhart.Scalable Sample-Level Causal Discovery in Event Sequences via Autoregressive Density Estimation. 2026. arXiv:2602.01135 [cs.LG].url:https://arxiv.org/abs/2602.01135. [91]Hugo Math and Rainer Lienhart. âTowards Practical Multi-label Causal Dis- covery in High-Dimensional Event Sequences via One-Shot Graph Aggregationâ. In:NeurIPS 2025 Workshop on Structured Probabilistic Inference & Generative Modeling. 2025.url:https://openreview.net/forum?id=1HZfpuDVeW. [92]Hugo Math, Rainer Lienhart, and Robin Schön. âHarnessing Event Sensory Data for Error Pattern Prediction in Vehicles: A Language Model Approachâ. In:Pro- ceedings of the AAAI Conference on Artificial Intelligence39.18 (Apr. 2025), p. 19423â19431.doi:10.1609/aaai.v39i18.34138.url:https://ojs.aaai. org/index.php/AAAI/article/view/34138 . [93]Hugo Math, Julian Lorenz, and Rainer Lienhart. âNeuro-Symbolic Rule Discov- ery: Empowering LLMs with Causality for Vehicle Diagnosticsâ. In:ICLR 2026 Workshop on Logical Reasoning of Large Language Models. 2026.url:https: //openreview.net/forum?id=M5ZszfsJxm . [94]Hugo Math, Robin Schön, and Rainer Lienhart. âOne-Shot Multi-Label Causal Discovery in High-Dimensional Event Sequencesâ. In:NeurIPS 2025 Workshop on CauScien: Uncovering Causality in Science. 2025.url: https://openreview. net/forum?id=z7NT8vGWC2 . [95]Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, OndĆej ÄertĂk, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sar- taj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pe- dregosa, Matthew J. Curry, Andy R. Terrel, Ć tÄpĂĄn RouÄka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. âSymPy: symbolic computing in Pythonâ. In:PeerJ Computer Science3 (Jan. 2017), e103. issn: 2376-5992.doi: 10.7717/peerj-cs.103.url:https://doi.org/10. 7717/peerj-cs.103. [96]Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. âGSM-Symbolic: Understanding the Limita- tions of Mathematical Reasoning in Large Language Modelsâ. In:The Thirteenth International Conference on Learning Representations. 2025.url: https : / / openreview.net/forum?id=AjXkRZIvjB. 155 Bibliography [97]Farhad Moghimifar, Afshin Rahimi, Mahsa Baktashmotlagh, and Xue Li. âLearn- ing Causal Bayesian Networks from Textâ. In:Proceedings of the 18th Annual Workshop of the Australasian Language Technology Association. Ed. by Maria Kim, Daniel Beck, and Meladel Mistica. Virtual Workshop: Australasian Lan- guage Technology Association, Dec. 2020, p. 81â85.url:https://aclanthology. org/2020.alta-1.9/. [98]Ehsan Mokhtarian, Sina Akbari, AmirEmad Ghassami, and Negar Kiyavash. âA Recursive Markov Boundary-Based Approach to Causal Structure Learningâ. In:Proceedings of The KDDâ21 Workshop on Causal Discovery. Ed. by Thuc Duy Le, Jiuyong Li, Greg Cooper, Sofia Triantafyllou, Elias Bareinboim, Huan Liu, and Negar Kiyavash. Vol. 150. Proceedings of Machine Learning Research. PMLR, Aug. 2021, p. 26â54.url:https://proceedings.mlr.press/v150/ mokhtarian21a.html. [99]Salama A. Mostafa, Aida Mustapha, Ahmed Abdulbasit Hazeem, Shihab Hamad Khaleefah, and Mazin Abed Mohammed. âAn Agent-Based Inference Engine for EïŹicient and Reliable Automated Car Failure Diagnosis Assistanceâ. In:IEEE Access6 (2018), p. 8322â8331.doi:10.1109/ACCESS.2018.2803051. [100]Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. âAttention bottlenecks for multimodal fusionâ. In:Proceedings of the 35th International Conference on Neural Information Processing Systems. NIPS â21. Red Hook, NY, USA: Curran Associates Inc., 2024.isbn: 9781713845393. [101]Meike Nauta, Doina Bucur, and Christin Seifert. âCausal Discovery with Attention- Based Convolutional Neural Networksâ. English. In:Machine Learning and Knowl- edge Extraction1.1 (Jan. 2019), p. 312â340.issn: 2504-4990.doi: 10.3390/ make1010019 . [102]Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. âOn the role of sparsity and DAG constraints for learning linear DAGsâ. In:Proceedings of the 34th Inter- national Conference on Neural Information Processing Systems. NIPS â20. Red Hook, NY, USA: Curran Associates Inc., 2020.isbn: 9781713829546. [103]Toan Q. Nguyen and Julian Salazar. âTransformers without Tears: Improving the Normalization of Self-Attentionâ. In:Proceedings of the 16th International Conference on Spoken Language Translation. Ed. by Jan Niehues, Rolando Cat- toni, Sebastian StĂŒker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico. Hong Kong: Association for Computational Linguistics, Nov. 2019.url: https: //aclanthology.org/2019.iwslt-1.17. [104]Eshaan Nichani, Alex Damian, and Jason D. Lee. âHow transformers learn causal structure with gradient descentâ. In:Proceedings of the 41st International Con- ference on Machine Learning. ICMLâ24. Vienna, Austria: JMLR.org, 2024. 156 Bibliography [105]Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. âLarge Language Diffusion Modelsâ. In:The Thirty-ninth Annual Conference on Neural Information Process- ing Systems. 2025.url:https://openreview.net/forum?id=KnqiC0znVF. [106]Siddhartha Nuthakki, Sunil Neela, Judy W. Gichoya, and Saptarshi Purkayastha. âNatural language processing of MIMIC-I clinical notes for identifying diagnosis and procedures with neural networksâ. In:CoRRabs/1912.12397 (2019). arXiv: 1912.12397.url:http://arxiv.org/abs/1912.12397. [107]Yosihiko Ogata et al. âStatistical studies of the occurrence of small earthquakes: I. A point process model for earthquake sequencesâ. In:Annals of the Institute of Statistical Mathematics30.1 (1978), p. 379â395. [108]OpenAI. âGPT-4 Technical Reportâ. In:ArXivabs/2303.08774 (2023).url:https: //arxiv.org/abs/2303.08774. [109]Juri Opitz. âA Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practiceâ. In:Transactions of the Association for Computational Linguistics12 (June 2024), p. 820â836.issn: 2307-387X.doi: 10.1162/tacl_a_00675. eprint:https://direct.mit.edu/tacl/article- pdf/doi/10.1162/tacl\_a\_00675/2465598/tacl\_a\_00675.pdf.url: https://doi.org/10.1162/tacl\_a\_00675. [110]Roxana Pamfil, Nisara Sriwattanaworachai, Shaan Desai, Philip Pilgerstorfer, Konstantinos Georgatzis, Paul Beaumont, and Bryon Aragam. âDYNOTEARS: Structure Learning from Time-Series Dataâ. In:Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics. Ed. by Silvia Chiappa and Roberto Calandra. Vol. 108. Proceedings of Machine Learning Re- search. PMLR, 2020, p. 1595â1605.url: https://proceedings.mlr.press/ v108/pamfil20a.html. [111]Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gre- gory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. âPyTorch: an imperative style, high-performance deep learning libraryâ. In:Proceedings of the 33rd International Conference on Neural Infor- mation Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2019. [112]Kevin Patel. âAgentic AI for Self-Healing Production Lines: Autonomous Root Cause Analysis and Correctionâ. In:Journal of Information Systems Engineering and Management10 (June 2025), p. 661â682.doi:10.52783/jisem.v10i52s. 10772 . [113]Jose M. Peña. âFinding consensus Bayesian network structuresâ. In:J. Artif. Int. Res.42.1 (Sept. 2011), p. 661â687.issn: 1076-9757. [114]Judea Pearl.Causality: Models, Reasoning and Inference. 2nd. USA: Cambridge University Press, 2009.isbn: 052189560X. 157 Bibliography [115]Judea Pearl.Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1988. isbn: 1558604790. [116]Ben Peters, Vlad Niculae, and AndrĂ© F. T. Martins. âSparse Sequence-to-Sequence Modelsâ. In:Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics. Ed. by Anna Korhonen, David Traum, and LluĂs MĂ rquez. Florence, Italy: Association for Computational Linguistics, July 2019, p. 1504â 1519.doi:10.18653/v1/P19-1146.url:https://aclanthology.org/P19- 1146/. [117]Parivash Pirasteh, Slawomir Nowaczyk, Sepideh Pashami, Magnus Löwenadler, Klas Thunberg, Henrik Ydreskog, and Peter Berck. âInteractive feature extraction for diagnostic trouble codes in predictive maintenance: A case study from auto- motive domainâ. In:Proceedings of the Workshop on Interactive Data Mining. WIDMâ19. New York, NY, USA: Association for Computing Machinery, 2019. isbn: 9781450362962.doi:10.1145/3304079.3310288.url:https://doi.org/ 10.1145/3304079.3310288. [118]JosĂ© M. Puerta, Juan A. Aledo, JosĂ© A. GĂĄmez, and Jorge D. Laborda. âEïŹicient and accurate structural fusion of Bayesian networksâ. In:Information Fusion66 (2021), p. 155â169.issn: 1566-2535.doi: https://doi.org/10.1016/j. inffus.2020.09.003.url:https://w.sciencedirect.com/science/ article/pii/S156625352030364X . [119]Jie Qiao, Ruichu Cai, Siyu Wu, Yu Xiang, Keli Zhang, and Zhifeng Hao. âStruc- tural Hawkes processes for learning causal structure from discrete-time event se- quencesâ. In:Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. IJCAI â23. Macao, P.R.China, 2023.isbn: 978-1-956792- 03-4.doi: 10.24963/ijcai.2023/633.url:https://doi.org/10.24963/ ijcai.2023/633 . [120]J. R. Quinlan. âInduction of Decision Treesâ. In:Machine Learning1 (1986), p. 81â106. [121]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. âLearning Transferable Visual Models From Natural Language Supervisionâ. In:Proceedings of the 38th International Conference on Machine Learning. Ed. by Marina Meila and Tong Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, July 2021, p. 8748â8763. url: https://proceedings.mlr.press/v139/radford21a.html. [122]Alec Radford and Karthik Narasimhan.Improving Language Understanding by Generative Pre-Training. 2018.url:https : / / api . semanticscholar . org / CorpusID:49313245. 158 Bibliography [123]Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. âMed-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease predictionâ. In:NPJ Digital Medicine4 (2020).url:https: //api.semanticscholar.org/CorpusID:218889776. [124]J. Rissanen. âPaper: Modeling by shortest data descriptionâ. In:Automatica14.5 (Sept. 1978), p. 465â471.issn: 0005-1098.doi: 10.1016/0005-1098(78)90005- 5.url:https://doi.org/10.1016/0005-1098(78)90005-5. [125]Raanan Yehezkel Rohekar, Yaniv Gurwicz, and Shami Nisimov. âCausal In- terpretation of Self-Attention in Pre-Trained Transformersâ. In:Thirty-seventh Conference on Neural Information Processing Systems. 2023.url:https:// openreview.net/forum?id=DS4rKySlYC. [126]Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdi- novic. âDetecting and quantifying causal associations in large nonlinear time series datasetsâ. In:Science Advances5.11 (2019), eaau4996.doi:10.1126/sciadv. aau4996. eprint:https://w.science.org/doi/pdf/10.1126/sciadv. aau4996.url:https://w.science.org/doi/abs/10.1126/sciadv.aau4996. [127]JosĂ© del Sagrado and SerafĂn Moral. âQualitative Aggregation of Bayesian Net- worksâ. In:Data Fusion and Perception. Ed. by Giacomo Della Riccia, Hans- Joachim Lenz, and Rudolf Kruse. Vienna: Springer Vienna, 2001, p. 91â108. isbn: 978-3-7091-2580-9.doi: 10.1007/978-3-7091-2580-9_5.url:https: //doi.org/10.1007/978-3-7091-2580-9_5. [128]Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Mar- roquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. âSimple and Effective Masked Diffusion Language Modelsâ. In:The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.url: https: //openreview.net/forum?id=L4uaAR4ArM. [129]Ranjan Sapkota, Konstantinos Roumeliotis, and Manoj Karkee.AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges. May 2025. doi:10.48550/arXiv.2505.10468. [130]Claude Elwood Shannon. âPrediction and Entropy of Printed Englishâ. In:Bell System Technical Journal30 (Jan. 1951), p. 50â64.url:http://languagelog. ldc.upenn.edu/myl/Shannon1950.pdf . [131]Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. âSelf-Attention with Rela- tive Position Representationsâ. In:Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 2 (Short Papers). Ed. by Marilyn Walker, Heng Ji, and Amanda Stent. New Orleans, Louisiana: Association for Computa- tional Linguistics, June 2018, p. 464â468.doi:10.18653/v1/N18-2074.url: https://aclanthology.org/N18-2074. 159 Bibliography [132]Oleksandr Shchur, Ali Caner TĂŒrkmen, Tim Januschowski, and Stephan GĂŒnne- mann. âNeural Temporal Point Processes: A Reviewâ. In:International Joint Con- ference on Artificial Intelligence. 2021.url:https://api.semanticscholar. org/CorpusID:233181707. [133]Benjamin Shickel, Brandon Silva, Tezcan Ozrazgat-Baslanti, Yuanfang Ren, Kia Khezeli, Ziyuan Guan, Patrick J. Tighe, Azra Bihorac, and Parisa Rashidi. âMulti- dimensional patient acuity estimation with longitudinal EHR tokenization and flexible transformer networksâ. In:Frontiers in Digital Health4 (2022).doi: 10.3389/fdgth.2022.1029191.url:https://par.nsf.gov/biblio/10401752. [134]Shohei Shimizu, Patrik O. Hoyer, Aapo HyvĂ€rinen, and Antti Kerminen. âA Lin- ear Non-Gaussian Acyclic Model for Causal Discoveryâ. In:J. Mach. Learn. Res. 7 (Dec. 2006), p. 2003â2030.issn: 1532-4435. [135]Ali Shojaie and George Michailidis. âDiscovering graphical Granger causality us- ing the truncating lasso penaltyâ. In:Bioinformatics26 (2010), p. i517 âi523. url:https://api.semanticscholar.org/CorpusID:388284. [136]Xiao Shou, Tian Gao, Dharmashankar Subramanian, Debarun Bhattacharjya, and Kristin P. Bennett. âConcurrent Multi-Label Prediction in Event Streamsâ. In:Proceedings of the AAAI Conference on Artificial Intelligence37.8 (June 2023), p. 9820â9828.doi: 10.1609/aaai.v37i8.26172.url:https://ojs.aaai. org/index.php/AAAI/article/view/26172. [137]Xiao Shou, D. Subramanian, Debarun Bhattacharjya, Tian Gao, and Kristin P. Bennet. âSelf-Supervised Contrastive Pre-Training for Multivariate Point Pro- cessesâ. In:ArXivabs/2402.00987 (2024).url: https://api.semanticscholar. org/CorpusID:267406624. [138]Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. âLearning Important Features Through Propagating Activation Differencesâ. In:Proceedings of the 34th International Conference on Machine Learning. Ed. by Doina Precup and Yee Whye Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR, 2017, p. 3145â3153.url: https://proceedings.mlr.press/v70/shrikumar17a. html. [139]Ruben Sipos, Dmitriy Fradkin, Fabian Moerchen, and Zhuang Wang. âLog-based predictive maintenanceâ. In:Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . KDD â14. New York, NY, USA: Association for Computing Machinery, 2014, p. 1867â1876.isbn: 9781450329569.doi:10.1145/2623330.2623340.url:https://doi.org/10. 1145/2623330.2623340. [140]Peter Spirtes and Clark Glymour. âAn Algorithm for Fast Recovery of Sparse Causal Graphsâ. In:Social Science Computer Review9.1 (1991), p. 62â72. doi: 10 . 1177 / 089443939100900106. eprint:https : / / doi . org / 10 . 1177 / 089443939100900106.url:https://doi.org/10.1177/089443939100900106. 160 Bibliography [141]Peter Spirtes, Clark Glymour, and Richard Scheines. âCausation, Prediction, and Search, 2nd Editionâ. In:Causation, Prediction, and Search (Second Edition). 2001.url:https://api.semanticscholar.org/CorpusID:124969922. [142]Emma Steele and Allan Tucker. âConsensus and Meta-analysis regulatory net- works for combining multiple microarray gene expression datasetsâ. In:Journal of Biomedical Informatics41.6 (Dec. 2008), p. 914â926.issn: 1532-0464.doi: 10.1016/j.jbi.2008.01.011.url:https://doi.org/10.1016/j.jbi.2008. 01.011. [143]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. âRoFormer: Enhanced transformer with Rotary Position Embeddingâ. In: Neurocomput.568.C (Mar. 2024).issn: 0925-2312.doi:10.1016/j.neucom. 2023.127063.url:https://doi.org/10.1016/j.neucom.2023.127063. [144]Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. âSequence to sequence learning with neural networksâ. In:Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2. NIPSâ14. Montreal, Canada: MIT Press, 2014, p. 3104â3112. [145]Hao Hao Tan and Mohit Bansal. âLXMERT: Learning Cross-Modality Encoder Representations from Transformersâ. In:Conference on Empirical Methods in Natural Language Processing. 2019.url: https://api.semanticscholar.org/ CorpusID:201103729. [146]Pablo Torrijos, JosĂ© M. Puerta, JosĂ© A. GĂĄmez, and Juan A. Aledo.Informed Greedy Algorithm for Scalable Bayesian Network Fusion via Minimum Cut Anal- ysis. 2025. arXiv: 2504.00467 [cs.LG].url:https://arxiv.org/abs/2504. 00467. [147]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, TimothĂ©e Lacroix, Baptiste RoziĂšre, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lam- ple.LLaMA: Open and EïŹicient Foundation Language Models. 2023. arXiv: 2302. 13971 [cs.CL].url:https://arxiv.org/abs/2302.13971. [148]Ioannis Tsamardinos, Constantin Aliferis, and Alexander Statnikov. âAlgorithms for Large Scale Markov Blanket Discoveryâ. In: Jan. 2003, p. 376â381. [149]Ioannis Tsamardinos and Constantin F. Aliferis. âTowards Principled Feature Selection: Relevancy, Filters and Wrappersâ. In:Proceedings of the Ninth Inter- national Workshop on Artificial Intelligence and Statistics. Ed. by Christopher M. Bishop and Brendan J. Frey. Vol. R4. Proceedings of Machine Learning Re- search. Reissued by PMLR on 01 April 2021. PMLR, Jan. 2003, p. 300â307. url:https://proceedings.mlr.press/r4/tsamardinos03a.html. [150]Caroline Uhler, Garvesh Raskutti, Peter BĂŒhlmann, Bin Yu, et al. âGeometry of the faithfulness assumption in causal inferenceâ. In:The Annals of Statistics41.2 (2013), p. 436â463. 161 Bibliography [151]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. âAttention is All you Needâ. In: Advances in Neural Information Processing Systems. Ed. by I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol. 30. Curran Associates, Inc., 2017.url:https://proceedings.neurips. c/paper_files/paper/2017/file/3f5e243547dee91fbd053c1c4a845a- Paper.pdf. [152]Changzhang Wang, You Zhou, Qiang Zhao, and Zhi Geng. âDiscovering and orienting the edges connected to a target variable in a DAG via a sequential local learning approachâ. In:Computational Statistics and Data Analysis77 (2014), p. 252â266.issn: 0167-9473.doi: https://doi.org/10.1016/j.csda.2014. 03.003.url:https://w.sciencedirect.com/science/article/pii/ S0167947314000802. [153]Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma.Linformer: Self-Attention with Linear Complexity. cite arxiv:2006.04768. 2020.url:http: //arxiv.org/abs/2006.04768. [154]Zhaoyu Wang, Pingchuan Ma, and Shuai Wang. âTowards Practical Federated Causal Structure Learningâ. In:Machine Learning and Knowledge Discovery in Databases: Research Track: European Conference, ECML PKDD 2023, Turin, Italy, September 18â22, 2023, Proceedings, Part I. Berlin, Heidelberg: Springer- Verlag, 2023, p. 351â367.isbn: 978-3-031-43414-3.doi: 10.1007/978-3-031- 43415-0_21 .url:https://doi.org/10.1007/978-3-031-43415-0_21. [155]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. âChain-of-thought prompting elic- its reasoning in large language modelsâ. In:Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS â22. Red Hook, NY, USA: Curran Associates Inc., 2022.isbn: 9781713871088. [156]Sarah Wiegreffe and Yuval Pinter. âAttention is not not Explanationâ. In:Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP). Ed. by Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, p. 11â20.doi: 10.18653/v1/D19-1002.url:https://aclanthology.org/ D19-1002/ . [157]Andreas Winter. âTight Uniform Continuity Bounds for Quantum Entropies: Conditional Entropy, Relative Entropy Distance and Energy Constraintsâ. In: Communications in Mathematical Physics347.1 (Mar. 2016), p. 291â313.issn: 1432-0916.doi:10.1007/s00220-016-2609-8.url:http://dx.doi.org/10. 1007/s00220-016-2609-8 . 162 Bibliography [158]Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement De- langue, Anthony Moi, Pierric Cistac, Tim Rault, RĂ©mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.HuggingFaceâs Transformers: State-of-the-art Natural Language Processing. 2020. arXiv:1910.03771 [cs.CL].url:https: //arxiv.org/abs/1910.03771. [159]Xingyu Wu, Bingbing Jiang, Yan Zhong, and Huanhuan Chen. âMulti-label Causal Variable Discovery: Learning Common Causal Variables and Label-specific Causal Variablesâ. In:CoRRabs/2011.04176 (2020). arXiv:2011.04176.url:https: //arxiv.org/abs/2011.04176 . [160]Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. âSegFormer: simple and eïŹicient design for semantic segmentation with transformersâ. In:Proceedings of the 35th International Conference on Neural In- formation Processing Systems. NIPS â21. Red Hook, NY, USA: Curran Associates Inc., 2021.isbn: 9781713845393. [161]Feng Xie, Zheng Li, Peng Wu, Yan Zeng, Chunchen Liu, and Zhi Geng. âLocal causal structure learning in the presence of latent variablesâ. In:Proceedings of the 41st International Conference on Machine Learning. ICMLâ24. Vienna, Austria: JMLR.org, 2024. [162]Hongteng Xu, Mehrdad Farajtabar, and Hongyuan Zha. âLearning Granger Causal- ity for Hawkes Processesâ. In:Proceedings of The 33rd International Conference on Machine Learning. Ed. by Maria Florina Balcan and Kilian Q. Weinberger. Vol. 48. Proceedings of Machine Learning Research. New York, New York, USA: PMLR, June 2016, p. 1717â1726.url: https://proceedings.mlr.press/ v48/xuc16.html . [163]Peng Xu, Xiatian Zhu, and David A. Clifton. âMultimodal Learning With Trans- formers: A Surveyâ. In:IEEE Trans. Pattern Anal. Mach. Intell.45.10 (Oct. 2023), p. 12113â12132.issn: 0162-8828.doi:10.1109/TPAMI.2023.3275156. url:https://doi.org/10.1109/TPAMI.2023.3275156. [164]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. âReAct: Synergizing Reasoning and Acting in Language Modelsâ. In:ArXivabs/2210.03629 (2022).url:https://api.semanticscholar.org/ CorpusID:252762395. [165]Jianxin Yin, You Zhou, Changzhang Wang, Ping He, Cheng Zheng, and Zhi Geng. âPartial orientation and local structural learning of causal networks for predic- tionâ. In:Proceedings of the Workshop on the Causation and Prediction Chal- lenge at WCCI 2008. Ed. by Isabelle Guyon, Constantin Aliferis, Greg Cooper, AndrĂ© Elisseeff, Jean-Philippe Pellet, Peter Spirtes, and Alexander Statnikov. Vol. 3. Proceedings of Machine Learning Research. Hong Kong: PMLR, June 2008, p. 93â105.url: http://proceedings.mlr.press/v3/yin08a.html. 163 Bibliography [166]Kui Yu, Xianjie Guo, Lin Liu, Jiuyong Li, Hao Wang, Zhaolong Ling, and Xindong Wu. âCausality-based Feature Selection: Methods and Evaluationsâ. In:ACM Comput. Surv.53.5 (Sept. 2020).issn: 0360-0300.doi:10.1145/3409382.url: https://doi.org/10.1145/3409382. [167]Yue Yu, Jie Chen, Tian Gao, and Mo Yu. âDAG-GNN: DAG Structure Learning with Graph Neural Networksâ. In:Proceedings of the 36th International Confer- ence on Machine Learning. Ed. by Kamalika Chaudhuri and Ruslan Salakhutdi- nov. Vol. 97. Proceedings of Machine Learning Research. PMLR, 2019, p. 7154â 7163.url:https://proceedings.mlr.press/v97/yu19a.html. [168]Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. âBig Bird: Transformers for Longer Sequencesâ. In:Advances in Neural Information Processing Systems. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin. Vol. 33. Curran Associates, Inc., 2020, p. 17283â17297. [169]Biao Zhang and Rico Sennrich. âRoot Mean Square Layer Normalizationâ. In: Advances in Neural Information Processing Systems. Ed. by H. Wallach, H. Larochelle, A. Beygelzimer, F. d' AlchĂ©-Buc, E. Fox, and R. Garnett. Vol. 32. Curran Associates, Inc., 2019.url:https://proceedings.neurips.c/paper_ files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf. [170]Chongsheng Zhang, George Almpanidis, Gaojuan Fan, Binquan Deng, Yanbo Zhang, Ji Liu, Aouaidjia Kamel, Paolo Soda, and JoĂŁo Gama. âA Systematic Review on Long-Tailed Learningâ. In:IEEE Transactions on Neural Networks and Learning SystemsPP (Feb. 2025), p. 1â21.doi: 10.1109/TNNLS.2025.3539314. [171]Libo Zhang and Yue Ning.Large Language Models as Interpolated and Extrapo- lated Event Predictors. 2024. arXiv:2406.10492 [cs.CL].url:https://arxiv. org/abs/2406.10492 . [172]Min-Ling Zhang and Zhi-Hua Zhou. âA Review On Multi-Label Learning Algo- rithmsâ. In:Knowledge and Data Engineering, IEEE Transactions on26 (Aug. 2014), p. 1819â1837.doi:10.1109/TKDE.2013.39. [173]Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. âSelf-Attentive Hawkes Processâ. In:Proceedings of the 37th International Conference on Ma- chine Learning . Ed. by Hal DaumĂ© I and Aarti Singh. Vol. 119. Proceedings of Machine Learning Research. PMLR, July 2020, p. 11183â11193.url:https: //proceedings.mlr.press/v119/zhang20q.html. [174]Wei Zhang, Thomas Kobber Panum, Somesh Jha, Prasad Chalasani, and David Page. âCAUSE: learning granger causality from event sequences using attribu- tion methodsâ. In:Proceedings of the 37th International Conference on Machine Learning. ICMLâ20. JMLR.org, 2020. 164 Bibliography [175]Wenyu Zhang, Devesh K. Jha, Emil Laftchiev, and Daniel Nikovski. âMulti- label Prediction in Time Series Data using Deep Neural Networksâ. In:CoRR abs/2001.10098 (2020). arXiv:2001.10098.url:https://arxiv.org/abs/ 2001.10098. [176]Pushe Zhao, Masaru Kurihara, Junichi Tanaka, Tojiro Noda, Shigeyoshi Chikuma, and Tadashi Suzuki. âAdvanced correlation-based anomaly detection method for predictive maintenanceâ. In:2017 IEEE International Conference on Prognostics and Health Management (ICPHM). 2017, p. 78â83.doi:10.1109/ICPHM.2017. 7998309. [177]Weike Zhao, Chaoyi Wu, Yanjie Fan, Xiaoman Zhang, Pengcheng Qiu, Yuze Sun, Xiao Zhou, Yanfeng Wang, Ya Zhang, Yongguo Yu, Kun Sun, and Weidi Xie. An Agentic System for Rare Disease Diagnosis with Traceable Reasoning. 2025. arXiv:2506.20430 [cs.CL].url:https://arxiv.org/abs/2506.20430. [178]Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. âDAGs with NO TEARS: continuous optimization for structure learningâ. In:Proceedings of the 32nd International Conference on Neural Information Processing Systems. NIPSâ18. Red Hook, NY, USA: Curran Associates Inc., 2018, p. 9492â9503. [179]Yun Zhou, Yawei Wang, Huan Song, Tesfagabir Meharizghi, Mohamad Al Jaza- ery, Panpan Xu, Denisse Colin-Magana, and Aruna Abeyakoon. âDeep sequence modeling for event log-based predictive maintenanceâ. In:KDD 2023 Interna- tional Workshop on Mining and Learning from Time Series (MileTS). 2023.url: https://w.amazon.science/publications/deep- sequence- modeling- for-event-log-based-predictive-maintenance . [180]Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. âTrans- former Hawkes Processâ. In:Proceedings of the 37th International Conference on Machine Learning. Ed. by Hal DaumĂ© I and Aarti Singh. Vol. 119. Proceed- ings of Machine Learning Research. PMLR, July 2020, p. 11692â11702.url: https://proceedings.mlr.press/v119/zuo20a.html. 165 A Appendix of PartI A.1 Error Pattern Prediction This section contains the appendix material for Chapter3. A.1.1 Multivariate Hawkes Process Log-likelihood Derivation The likelihood for any point process parametrized byΞwith observationst 1 ,·, t L within a time horizon0â€tâ€Tcan be computed as the sum of the log-likelihood for each process [74]: lnL(Ξ|t 1 ,·, t L , T) = |X| X k=1 lnL k (Ξ|t 1 ,·t L ) where each term is defined by: lnL k (t 1 ,·, t L ) = ln " L Y i=1 λ â k (t i ) # exp (â Z T 0 λ â k (t)dt) ! (A.1) This expression reduces to: = L X i=1 lnλ â k (t i ) + ln exp (â Z T 0 λ â k (t)dt) = L X i=1 lnλ â k (t i )â Z T 0 λ â k (t)dt(A.2) Finally, we have for the total log-likelihood: lnL(Ξ|t 1 ,·, t L , T) = |X| X k=1 L X i=1 lnλ â k (t i )â |X| X k=1 Z T 0 λ â k (t)dt(A.3) A.1.2 Models Definition A GPT [122] model is added to improve comparison over baseline models. We kept the original implementation of the feed-forward layers [151] but initialize the intermediate layers with:W l ⌠N(0, 2 L â d l ). We used root-mean-square normalization (RMS Norm) 167 A Appendix of PartI from [169] instead of traditional layer normalization. We initialized all other linear layers using the SMALLINIT schema [103]W l ⌠N 0, q 2 d l +4d l . The names printed in the Tab3.2are model variations based on the embeddings given to CarFormer: time: Along with the event typeEan absolute time embedding is added such as input U=E+T mileage: Same as time except we add a mileage embeddingMsuch asU=E+M rot: A RoPE [143] is applied toQ,Ksuch asQ=R d Î W Q E,K=R d Î W K E ce: We add a context embeddingCE=T+MtoQ,Ksuch asQ=UW Q +CEto every attention layers. m2c, c2m: Inspired by the Disentangled Attention mechanism from DeBerta [55]. At- tention scoreA i,j between tokensiandjis computed from hidden states vec- torH i at event stepiand a mileage vectorM i at event stepisuch as: A i,j =H i ,M i ĂH j ,M j T =H i H T j +H i M T j +M i H T j +M i M T j =âcontent-to- contentâ+âcontent-to-mileageâ+âmileage-to-contentâ+âmileage-to-mileageâ= âc2câ+âc2mâ+âm2câ+âm2mâ. A.1.3 Random Event Injection LetA i be the random variable representing the number of random events injected at stepi. We can say that: P(A i =r) = (1âp)p r forr= 0,1,2, . . . WhereA i is the number of random events injected at stepi,ris the number of trials (injected events) until the first failure,pis the probability of injecting a random event. Therefore, it is trivial to derive the expected number of random events injected per sequenceSof lengthL: Therefore, the expected total number of injected fake events over the entire sequence of lengthLis: E " L X i=1 A i # =Lp Which in our case would translateâŒ150Ă0.05 = 7.5random events per sequence. A.2 Multimodal Error Pattern Offline Prediction This section contains the appendix material for Chapter4. 168 A.2 Multimodal Error Pattern Offline Prediction 0255075100125150175200 Number of observations 65 70 75 80 85 90 95 100 F1 micro (%) Confident Predictive Maintenance Window c = w/o random head c = w random head x = 92, Confident predictions x = 158, Average Sequence length Figure A.1:Confident Predictive Maintenance Window Evolutionwith and w/o the random head. 01000020000300004000050000600007000080000 Training Steps 2 4 6 8 10 Base Code Classification Loss BiCarFormer with multimodal learning BiCarFormer with base-code masking only Figure A.2:Pretraining Loss Comparison. Base-DTC classification loss comparison with and without multimodal learning. 169 B Appendix of PartII B.1 Sample-Level Multi-Label Causal Discovery This section contains the appendix material for the sample-level multi-label causal dis- covery Chapter6. B.1.1 PyCausalFS Local structure learning algorithms were all used withα= 0.1in the associated code: https://github.com/wt-hu/pyCausalFS/tree/master/pyCausalFS/LSL. B.1.2 MI-MCF MI-MCF [85] was used for comparison following the oïŹicial implementation athttps: //github.com/malinjlu/MI-MCFwe usedα= 0.05, L= 268, k 1 = 0.7, k 2 = 0.1. B.1.3 NADEs Quality. We did several ablations on the quality of the NADEs and their impact on the sample- level multi-label causal discovery phase. In particular, Table B.1presents multiple Tf x ,Tf y with respectively 90 and 15 million parameters or 34 and 4 million param- eters. We also varied the context window (conditioning setZ), trained on different amounts of data (Tokens), and reported the classification results on the test set of Tf y alone. We did not output the running time since it was always the same for all NADEs: 1.27minutes for 50,000 samples and0.14for 5000. B.1.4 Sampling Procedure for CMI We performed an ablation (TabB.2) on the effect of sampling methods to estimate the expected value over all possible contextZ. We used one A10 GPU on a sample of the test dataset (4000 random samples) composed of 205 labels with a batch size of 4 during inference. We tested top-k sampling withk=20,35[34] with and w/o a temperature scaler ofTto log-probabilitiesËxsuch that Ëx âČ =softmax(logËx/T) and a combination of top-k and a top-nucleus sampling [ 60] with different probability massp=0.8,1.2and finally a permutation of token position within the context c. We fixed a dynamic threshold with z scorek= 3and performed 10 runs. We then reported the mean and standard deviation for each classification metric and elapsed time (sec). 171 B Appendix of PartII Tokens Param. Context Prec. (â) Rec. (â)F1 (â) Tfy F1 (â) Forn= 50,000samples 1.5B105mc= 447.95±1.05 30.65±0.51 37.39±0.6788.6 1.5B105mc= 12 54.62±1.03 29.88±0.73 38.63±0.8590.43 1.5B105mc= 1555.26±1.42 31.37±0.82 40.02±1.0390.57 1.5B105mc= 20 49.52±1.5931.76±0.8536.54±1.1091.19 1.5B105mc= 30 36.65±1.18 22.75±0.78 26.57±0.9192.64 300m47mc= 20 39.49±1.77 26.30±0.89 29.01±1.1083.6 Forn= 500samples 1.5B105mc= 12 54.84±4.5531.45±2.23 39.95±2.8390.43 1.5B105mc= 15 55.04±3.36 29.90±1.78 38.74±2.2490.57 1.5B105mc= 20 48.84±4.0131.65±2.3736.19±2.6591.19 300m47mc= 20 38.23±2.91 25.31±2.39 27.92±2.2583.6 Table B.1:Ablations of the performance of Phase 1 (Sample-LevelMBretrieval) as a function of different NADEs withn= 50,000andn= 500samples, averaged over 5-folds. Classification metrics use weighted averaging. Metrics are given in%. Sampling increases the predictive performance of OSCAR by a large margin. More in- terestingly, different sampling types have different effects on specific averaging. This has a âsmoothingâ effect on the CMI curve when multiple labels are present in the sequence. Without upsampling, the CMIâs sensitivity across labels increases, making it more chal- lenging to determine a threshold and a potential cause. We observe that, broadly, top-k sampling yields better results, particularly when followed by top-p=0.8. Sampling with the same tokens (Permutation) is not a good choice; sampling from the next-event pre- diction Tf x yielded better results. We will chooseTop-k+p=0.8for the increased F1 Micro and high F1 Macro, and Weighted. ProposalF1 Micro (%) F1 Macro (%) F1 Wtd. (%) Time (s) w/o Sampling14.0712.2916.6749.30±0.30 Permutation18.22±0.3613.75±0.0919.21±0.03 557.82±0.13 Top-k=2026.77±0.7123.83±0.1929.25±0.07 557.40±0.13 Top-k=3526.57±0.9624.08±0.2329.30±0.07 557.35±0.10 Top-k=35+T=0.827.36±0.6523.77±0.2128.98±0.07 557.45±0.11 Top-k=35+T=1.226.59±1.4924.62±0.29 29.52±0.06557.45±0.12 Top-k=25+p=0.827.98±0.6723.82±0.2829.18±0.07 558.07±0.07 Top-k=35+p=0.828.82±0.7524.06±0.2529.17±0.07 558.16±0.14 Top-k=35+p=0.926.39±0.9924.12±0.3129.26±0.11 558.11±0.12 Top-k=35+p=0.9+T=0.927.63±0.7523.90±0.2429.04±0.09 558.07±0.12 Top-k=35+p=0.9+T=1.126.75±1.3024.47±0.2429.45±0.09 558.06±0.11 Table B.2:Sample-level classification performance and elapsed time (sec) across different sam- pling methods. Best results are inbold, and equal performance are in underline. 172 B.1 Sample-Level Multi-Label Causal Discovery B.1.5 Sampling Number We experimented with different numbers ofNfor the sampling method across different averaging methods (micro, macro, weighted), Fig .B.1. We performed eight different runs and reported the average, standard deviation, and elapsed time. In general, sampling with a largerNtends to decrease the standard deviation and yield more reliable Markov Boundary estimation. As we process more samples, the model gradually improves with logarithmic growth until it converges to a final score. We also verify that our time complexity is linear with the number of samplesN. Based on these results, we generally chooseN= 68. 050100150200 Number of Samples 22 24 26 28 30 32 F1 Score (%) Micro Macro Weighted 050100150200 Number of Samples 20 25 30 35 40 Precision (%) 050100150200 Number of Samples 22 24 26 28 30 32 Recall (%) 050100150200 Number of Samples 0 100 200 300 Elapsed Time (sec) Figure B.1:Ablation of the Number of ParticlesN. Results are reported using 1-sigma error bar. B.1.6 Dynamic Thresholding We performed ablations on the effect ofkduring the dynamic thresholding of the CMI in Eq.6.11to assess conditional independence in Fig.B.2. To balance the classification metrics across the different averaging, we setk= 2.75. B.1.7 Implementation of OSCAR The following is the implementation of OSCAR in PyTorch [111]. 1deftopk_p_sampling(z, prob_x, c:int, n:int= 64, p:float= 0.8, k: int= 35, 173 B Appendix of PartII 24 Trust Interval Multiplier (k) 20 25 30 F1 Score (%) Micro Macro Weighted 24 Trust Interval Multiplier (k) 20 30 40 50 60 Precision (%) 24 Trust Interval Multiplier (k) 15 20 25 30 35 Recall (%) 0.00.20.40.60.81.0 0.0 0.2 0.4 0.6 0.8 1.0 Figure B.2:Sensitivity Analysis. Evolution of the sample-level F1 Score, Precision, and Recall as a function of coeïŹicientk. Results are reported using 1-sigma error bar. 2cls_token_id:int= 1, temp:float= None): 3#Samplejustthecontext 4input_ = prob_x[:, :c] 5 6#Top-kfirst 7topk_values , topk_indices = torch.topk(input_, k=k, dim=-1) 8 9#Top-povertop-kvalues 10sorted_probs , sorted_idx = torch.sort(topk_values , descending=True, dim=-1) 11cum_probs = torch.cumsum(sorted_probs , dim=-1) 12mask = cum_probs > p 13 14#Ensureatleastonetokeniskept 15mask[..., 0] = 0 16 17#Maskandnormalize 18filtered_probs = sorted_probs.masked_fill(mask, 0.0) 19filtered_probs += 1e-8#fornumericalstability 20filtered_probs /= filtered_probs.sum(dim=-1, keepdim=True) 21 22#Unscrambletomatchtheoriginaltop-kindices 23#Needtoreorderthesortedindicesbacktotheoriginaltop-k 24reorder_idx = torch.argsort(sorted_idx , dim=-1) 25filtered_probs = torch.gather(filtered_probs , -1, reorder_idx) 26 174 B.1 Sample-Level Multi-Label Causal Discovery 27batched_probs = filtered_probs.unsqueeze(1).repeat(1, n, 1, 1) #(bs,n,seq_len,k) 28batched_indices = topk_indices.unsqueeze(1).repeat(1, n, 1, 1) #(bs,n,seq_len,k) 29 30sampled_idx = torch.multinomial(batched_probs.view(-1, k), 1) #(bs*n*seq_len,1) 31sampled_idx = sampled_idx.view(-1, n, c).unsqueeze(-1) 32 33sampled_tokens = torch.gather(batched_indices , -1, sampled_idx). squeeze(-1) 34sampled_tokens[..., 0] = cls_token_id 35 36#Reconstructfullsequence 37z_expanded = z.unsqueeze(1).repeat(1, n, 1)[..., c:] 38returntorch.cat((sampled_tokens , z_expanded), dim=-1) 39 40fromtorchimportnn 41defOSCAR(tfe: n.Module, tfy: n.Module, batch:dict[str, torch.Tensor ], c:int, n:int, eps:float=1e-6, topk:int=20, k:int=2.75, p =0.8) -> torch.Tensor: 42#tfe,tfy:arethetwoautoregressiveTransformers(eventtypeand label) 43#batch:dictionarycontainingabatchofinput_idsand attention_maskofshape(bs,L)toexplain. 44#c:scalarnumberdefiningtheminimumcontexttostart inferring,alsothesamplinginterval. 45#n:scalarnumberrepresentingthenumberofsamplesforthe samplingmethod. 46#eps:floatfornumericalstability 47#topk:Thenumberoftop-kmostprobabletokenstokeepfor sampling 48#k:Numberofstandarddeviationstoaddtothemeanfordynamic thresholdcalculation 49#p:Probabilitymassfortop-pnucleus 50 51o = tfe(attention_mask=batch['attention_mask'], input_ids=batch[' input_ids'])['prediction_logits']#Inferthenexteventtype 52x_hat = torch.n.functional.softmax(o, dim=-1) 53 54b_sampled = topk_p_sampling(batch['input_ids'], x_hat, c, k=topk, n= n, p=p)#Samplingupto(bs,n,L) 55n_att_mask = batch['attention_mask'].unsqueeze(1).repeat(1, n, 1) 56 57withtorch.inference_mode(): 58o = tfy(attention_mask=n_att_mask.reshape(-1, b_sampled.size(-1) ), input_ids=b_sampled.reshape(-1, b_sampled.size(-1)))#flatten andinfer 59prob_y_sampled = o['ep_prediction'].reshape(b_sampled.size(0), n , batch[ 'input_ids'].size(-1)-c, -1)#reshapeto(bs,n,L-c) 60 61#Ensureprobsarewithin(eps,1-eps) 62prob_y_sampled = torch.clamp(prob_y_sampled , eps, 1 - eps) 63 175 B Appendix of PartII 64y_hat_i = prob_y_sampled[..., :-1, :]#P(Yj|z) 65y_hat_iplus1 = prob_y_sampled[..., 1:, :]#P(Yj|z,x_i) 66 67#ComputetheCMI&CSandaverageacrosssamplingdim 68cmi = torch.mean(y_hat_iplus1*torch.log(y_hat_iplus1/y_hat_i)+ (1-y_hat_iplus1)*torch.log((1-y_hat_iplus1)/(1-y_hat_i)), dim=1) 69#(BS,L,Y) 70cs = y_hat_iplus1 - y_hat_i 71cs_mean = torch.mean(cs, dim=1) 72cs_std = torch.std(cs, dim=1) 73 74#Confidenceintervalforthreshold 75mu = cmi.mean(dim=1) 76std = cmi.std(dim=1) 77dynamic_thresholds = mu + std * k 78 79#Broadcasttoselectanindividualdynamicthreshold 80cmi_mask = cmi >= dynamic_thresholds.unsqueeze(1) 81 82cause_token_indices = cmi_mask.nonzero(as_tuple=False) 83#(num_causes,3)-->eachrowis[batch_idx,position_idx, label_idx] 84returncause_token_indices , cs_mean, cs_std, cmi_mask Remark 6.Since tfy contains tfe as backbone, in practice we need only one forward pass from tfy and extract also Ë x, so tfe is not needed. We let it to improve understanding and clarity. B.2 Population-Level Multi-Label Causal Discovery This section contains the appendix material for Chapter7. B.2.1 Definition of Scoring Criteria Definition 27.(Decomposable Criterion). We say that a scoring criterionS(G,D)is decomposable if it can be written as a sum of measures, each of which is a function only of one node and its parents. In other words, a decomposable scoring criterionf s applied to a DAGGcan always be expressed as: S(G,D) = n X i f s (X i ,Pa G (X i ))(B.1) Definition 28(Score equivalent).[21]. A scoref s is score equivalent if it assigns the same score to all the graphs in the same MEC. Definition 29(Local Consistency).[21] LetDcontainmi.i.d samples from some distributionP(.). LetGbe any possible DAG andG âČ a different DAG obtained by adding the edgeiâjtoG. A scoref s is locally consistent if both hold: 176 B.2 Population-Level Multi-Label Causal Discovery âąIfX i Ìžâ„ p X j |P a G (X j ),thenf s (G âČ ,D)> f s (G,D) âąIfX i â„ p X j |P a G (X j ),thenf s (G âČ ,D)< f s (G,D) B.2.2 Criteria This section presents the different criteria used for comparison in the experimental eval- uation of Chapter7. B.2.3 Frequency Frequency-based heuristics that apply fixed thresholdsÏto the empirical frequency of the occurrence ofX i in each of the Markov BoundaryMB(Y j ). Formally, for each label Y j , after merging all local edge sets into a global setE= S m i=1 E i , we evaluate each candidate variableX i âMB j based on its frequency of appearance across the local models. If this frequency exceeds the thresholdÏ, the variable is retained in the final mergedMB j ; otherwise, it is discarded. B.2.4 Expected FPR Adjustment Same principle as in [36] except that we fit the two Beta distributions on the mutual information ofI(Y j , X i )instead of the raw frequencies. B.2.5 Mutual Information BESThe Backward Equivalence Search (BES) is the second phase of GES [21], where edges are removed one after the other to maximize a criterionS. Heuristics approaches [ 118,127] aim to solve this problem by optimizing: E= arg E i âΔ max X eâE i S(e)(B.2) WhereΔdenotes the search space (all possible edges overU) andS(e)a criterion function for edge relevance (e.g. edge frequency, thresholds,·). This formulation accounts for the characteristics of the underlying edges but not for the overall network structure, complexity, or missing data [113], leading to aconsensus fusion approach[146]. Estimating Mutual InformationWe want to reuse the estimated conditional mutual information, Eq.6.8, and profit from the parallelized inference of Phase 1 (Fig.7.2). As argued out by [68], a causal strength measure (or criterion)C X i âY j should possess multiple properties. Notably, ifC X i âY j = 0, then the joint distribution satisfies the Markov condition with respect to the DAG obtained by removing the arrowX i âY j . The true DAG readsX i âYiffC X i âY j =I(X i , Y j ). It is a natural criterion for merging edges across multiple causal graphs. However, it remains tricky to estimate [ 71]. Using the chain rule of conditional mutual information 177 B Appendix of PartII [23], we can rewrite it as: I(Y j , X i |Z) =I(Y j , X i )âI(Y j , X i ,Z)(B.3) WhereI(Y j , X i ,Z)is the interaction information [23], which tells us whether knowing Zexplains away the dependency betweenX i andY j (negative interaction), or enhances it (positive interaction): I(Y j , X i ,Z)âI(Y j ,Z)âI(Y j ,Z|X i ) I(Y j ,Z)can be estimated using the same Monte-Carlo sampling as forI(Y j , X i |Z) 6.8. SinceI(Y j ,Z) =H(Y j )âH(Y|Z j ), the marginalp(y)is needed. Fortunately, the datasetDis large enough, hence the frequencies ofy j are recovered empirically and an estimateËp(y), which we assume to be equal to the true marginalp(y). We acknowledge that under a restricted dataset,Ëp(y)might differ fromp(y). This yields to: I(Y j ,Z) =E p(z) D KL (P(Y j |Z)|| Ë P(Y j )) =E p(z) I G (Y j ,Z)(B.4) Formally, we assume that for long sequences i.e.,iâ+â, our event sequences form a stationary ergodic stochastic process andI(Y j ,Z|X i )is negligible compare toI(Y j ,Z) sinceZis containing most of the information to predictY j . This reduces the mutual information to I(Y j , X i )âI(Y j , X i |Z) +I(Y j |Z) Criterion.We propose aClass-Aware Information Gain (CAIG)score for evalu- ating candidate edges during Phase 2 of CARGO. Givenmi.i.d. samples from a dataset D, CAIG balances three key factors: mutual information derived from information gain, class imbalance, and network complexity. For each label nodeY j , with candidate parent setPa G j , the CAIG score is: f s (G,D) = n X j=1 f I (Y j ,Pa G j )âα·|Pa G j |·log m m j + 1 (B.5) Withf I (Y j ,Pa G j ) = P X i âPa G j I(Y j , X i ),αis a regularization hyperparameter,m j is the number of positive instances for classY j . This formulation encourages informative yet parsimonious graph structures and cor- rects for underrepresented labels via the regularization term. It is also eïŹicient since CAIG isdecomposable[21] like BIC with the locals I . This criterion is denoted asBES mi imbalance parin our experiments in Fig.7.4. B.2.6 Implementation of Phase 2 1importrandom 178 B.2 Population-Level Multi-Label Causal Discovery 2defcreate_auto_adaptive_threshold_fn(all_m_j, tau_max=0.5, tau_min =0.05, k=None, m0='median'): 3m_0 = np.median(all_m_j) 4 5ifk == None: 6q25, q75 = np.percentile(all_m_j, [25, 75]) 7ifq75 == q25: 8k = 1.0 9else: 10log_iqr = np.log(q75) - np.log(q25) 11k = (2 * np.log(3)) / log_iqr 12 13defthreshold_function(m_j): 14log_m_j = np.log(m_j + 1e-9) 15log_m_0 = np.log(m_0) 16logistic_decay = 1 / (1 + np.exp(k * (log_m_j - log_m_0))) 17return(tau_max - tau_min) * logistic_decay + tau_min 18 19returnthreshold_function 20 21defadaptive_thresholding_frequency(graphs:list, 22present_labels:dict, 23frequency_threshold:float= 0.5, 24k:float=None, 25tau_min:float=0.05, 26tau_max:float=0.5, 27m0:str='median', 28verbose=False, 29**kwargs): 30 31#Frequencyvoting:keepedgesappearingwithfrequency>threshold acrosssamples. 32 33#:paramgraphs:listoflocalgraphs(e.g.,fromPhase1).Each graphisadict[label][token]=listof#stats. 34#:parampresent_labels:labelspresentinevaluation 35#:paramfrequency_threshold:e.g.0.5formajority,0.8for conservative 36#:return:filtered_labels,sample_per_label,elapsed_time 37 38start_time = datetime.now() 39#Step1:Aggregategraphs 40labels, sample_per_label = union(graphs)#user-definedunion function 41old_labels = labels.copy() 42nodes = count_nodes(labels) 43samples =len(graphs) 44 45#Createthresholdfunction 46auto_threshold_fn = create_auto_adaptive_threshold_fn(list( sample_per_label.values()), k=k, tau_max=tau_max, tau_min=tau_min, m0=m0) 47 48#Step2:Frequencyvotingwithdynamicthresholds 179 B Appendix of PartII 49edge_counts = defaultdict(lambda: defaultdict(int))#edge_counts[ label][token]=count 50 51forgingraphs: 52forlabel, token_dicting.items(): 53iflabelnotinlabels: 54continue 55fortokenintoken_dict: 56edge_counts[label][token] += 1 57 58#Step3:Keepedgesabovefrequencythreshold 59filtered_labels = defaultdict(dict) 60forlabelinlabels: 61total = sample_per_label.get(label, samples)#fallbackto totalgraphsifmissing 62fortoken, countinedge_counts[label].items(): 63freq = count / total 64iffreq >= auto_threshold_fn(sample_per_label.get(label, 1)) : 65filtered_labels[label][token] = 'frequency': freq 66ifverbose: 67print(f'[label]tokentokenkept(freq=freq:.2f )') 68 69nb_of_edges =sum(len(v)forvinfiltered_labels.values()) 70print(f'Time:(datetime.now()-start_time).total_seconds():.2fs') 71returnfiltered_labels , sample_per_label , (datetime.now() - start_time).total_seconds() B.3 Sample-Level Event-to-Event Causal Discovery B.3.1 Flexible Assumptions Assumption 7(Bounded Lagged Effects).There exists an integermâ„1, referred to as the lag bound, such that for any sequencesand any target eventX j with indext > m, all past events prior totâmare conditionally independent ofX t given itsmmost recent predecessor events. Formally, X t â„X 0:tâm X tâm:tâ1 B.3.2 Evaluation We used anml.g5.4xlargeinstance from AWS Sagemaker, which contains 8 vCPUs and 1 NVIDIA A10G as GPU with 24GiB for training and inference. Justification for Evaluating on SCMs.As we saw, standard causal discovery algo- rithms for multivariate time series (e.g., PCMCI, Neural Hawkes Processes, CASCADE) are not applicable in our setting. Moreover, evaluating causal discovery in large scale event sequences is notoriously diïŹicult due to the lack of ground-truth annotations in 180 B.3 Sample-Level Event-to-Event Causal Discovery real-world traces (e.g., server logs, medical records). Furthermore, generic high-order MC are computationally intractable to materialize due to the state space|X| m ofX t and unlearnable in high dimensions without structural assumptions. Synthetic DataWe introduce a synthetic benchmark based on a linear Structural Causal Model (SCM). This DGP models the transition distributionP(X t |X tâm:tâ1 )via an additive mechanism where past events exert independentexcitatoryorinhibitory influences on future outcomes such as: P(X t |X tâm:tâ1 ) =softmax b+ m X k=1 W (k) x tâk ·λ(k) ! whereW (k) âR |X|Ă|X| is a sparse weight matrix representing the causal influence of an event at lagk, andλ(k)is a temporal decay function. Importantly, weightsW i,j can bepositive (excitation) or negative (inhibition), allowing for real-world causal mechanisms present in sequences. We tune the sparsity of the weight matrixWto obtain a predictability score Pred= H(P) H max = H(P) log(|X|) superior or equal to58%across the benchmarked SCMs. Practical Monitoring: The Oracle Score.Since the true entropyH(P)and thus the true KL divergence are unknown, we must approximateΔempirically. We observe that the cross-entropy loss decomposes asL AR (Ξ) =H(P) +D KL (P||P Ξ ). We propose the Oracle ScoreËΔas a normalized estimator of the excess entropy: ËΔ(P Ξ ) = L AR (Ξ)âH(P) H max âH(P) (B.6) whereH max = log|X|is the maximum entropy (uniform noise) andHis the irreducible entropy of the DGP (approximated by the minimum validation loss observed). This metric provides a vocabulary-agnostic measure of fit:ËΔâ0implies convergence to the theoretical limit (P Ξ âP). B.3.3 Additional Ablations B.3.3.1 Number of ParticlesN We observe in Fig.B.3that the number of particlesNreduce the amount of missing causal relationships (Recall) and SHD. AfterN >256, however, we did not observe significant changes. B.3.3.2 Empirical Validation ofΔ-Strong Faithfulness. Fig.B.4demonstrates that the standard faithfulness assumption (Ï= 0) is unten- able in practice, as it fails to distinguish causal signals from finite-sample approxima- tion noise (resulting in low precision). Conversely, the distinct performance peak at Ï opt â1.4Ă10 â5 empirically validates theΔ-Strong Faithfulnesshypothesis (Def. 24), 181 B Appendix of PartII 248163264128256300350400 Number of Particles (N) 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Causal Discovery Quality Precision Recall F1 Score SHD (Error) 0 10 20 30 40 50 Structural Hamming Distance Figure B.3:Ablation of the Number of ParticlesN. Evolution of causal discovery per- formance as a function of the number of particlesNat|X|= 1000, m= 6 confirming that causal discovery requires a minimum signal-to-noise ratio. Especially, Ï opt is orders of magnitude lower than the worst-case theoretical bound derived in Theo- rem2, suggesting that theaverage-caseestimation error is significantly tighter than the Alicki-Fannes-Winter inequality implies. Figure B.4:Sensitivity Analysis.Classification metrics relative to the filtering thresholdÏ with|X|= 1000, Δ= 0.04, c= 6, m= 6, L= 64, N= 128). 182 B.4 CAREP: Multi-Agent System for Automatic Rule Generation B.3.3.3 Scaling Law for Causal Identifiability In the rightΔ-regimeΔâ[0,0.1], we observe an inverse scaling lawÏ opt â |X| â1 for the optimal thresholdÏin Fig.B.5: Ï(X) = C |X| , C= 1.72·10 â2 This indicates that while the density estimation task becomes harder in high dimensions (higher entropy), the structural identifiability actually improves. The sparsity of the event space dilutes spurious correlations, effectively lowering the noise floor and allowing for the discovery of weaker causal signals. 10 2 10 3 Vocabulary Size |î| 10 â5 10 â4 Optimal Threshold Ï opt Empirical Optimal Fit: Ïâ 1.72eâ 02 â |î| â0.96 Figure B.5:The Scaling Law of Causal Identifiability.We perform a sensitivity analysis across vocabulary sizes|X|â100, . . . ,2000to find the optimal filtering threshold Ï opt (maximizing F1-score) withΔ= 0.04. The result reveals a power-law relation- shipÏ opt â|X| â0.96 . B.3.4 Additional Summary Causal Graph B.4 CAREP: Multi-Agent System for Automatic Rule Generation This section contains the appendix material for the chapter9. B.4.1 Input Format 183 B Appendix of PartII 7.32 7.06 0.30 3.72 0.98 0.19 1.77 0.34 0.16 1.83 0.330.25 0.34 0.34 0.92 0.37 2.79 0.90 0.45 0.20 0.45 0.21 1.19 t=1t=2t=3t=4t=5t=6t=7t=8t=9t=10t=11t=12t=13t=14t=15t=16t=17t=18t=19t=20t=21t=22t=23t=24t=25t=26t=27t=28t=29t=30t=31t=32t=33 Tailgate Blocked Tailgate Lift Limited Augmented View Limited Camera Calib Error Camera Out- of-Range BT Audio Play Phonebook Access Phonebook Access Engine Intervention Variance Cruise Range Stat Ignition Off Prevented Window Emergency Mode Interior Light Map Implausible Speed Limit Msg Missing Foresight Motor Fault FAS Signal Error Baseboard Supply Limited FAS Signal Error Baseboard Supply Limited Battery Temp Fault Secondary Network Fault Remote Driving Limited Wiper Limited Radar List Overflow FAS Signal Error Radar Interference 48V Temp Derating LDW Signal Inconsist. TAP Events Stat LCW Events Stat Radar List Overflow Substitute Speed Used PWM Actuation Available Instance Time Causal Graph 7.32 7.06 0.30 3.72 0.98 0.19 1.77 0.34 0.16 0.45 1.19 1.83 0.33 0.34 0.34 0.92 0.37 2.79 0.90 0.20 0.45 0.21 Tailgate Blocked Tailgate Lift Limited Augmented View Limited Camera Calib Error Camera Out- of-Range BT Audio Play Phonebook Access Cruise Range Stat Ignition Off Prevented Window Emergency Mode Interior Light Map Implausible Speed Limit Msg Missing Foresight Motor Fault FAS Signal Error Radar List Overflow Substitute Speed Used Baseboard Supply Limited Battery Temp Fault Secondary Network Fault Remote Driving Limited Wiper Limited Radar Interference 48V Temp Derating LDW Signal Inconsist. TAP Events Stat LCW Events Stat PWM Actuation Available Engine Intervention Variance Instance Summary Causal Graph Figure B.6:Instance Summary Causal Graph.The global causal structureG s aggregated from TRACE inferences over the validation set. While the instance graph (Fig. 8.6) details when events occur, this summary graph captures the invariant mecha- nism types. The density of the graph highlights the complexity of modern vehicle architectures, where high-degree nodes often represent central control units (e.g., ECU, Battery Management) that propagate cascading faults. The conditional mu- tual information is reported as causal strength. 184 B.4 CAREP: Multi-Agent System for Automatic Rule Generation 1"unknownEP": 2"potential_causes": 3"DTC3": 4"frequency": 0.78, 5"ACE_mean": 0.70, 6"ACE_std": 0.13, 7"pmi": 8"DTC345": -1.89 9 10, 11"DTC345": 12"frequency": 0.49, 13"ACE_mean": 0.59, 14"ACE_std": 0.12, 15"pmi": 16"DTC3": -1.89 17 18, 19"dtcs_samples": 20"0X0010X00080X01200X8900..META..CCM111", 21"0x60520X02040X01290X3410..META..CCM111", 22... 23 24 25 Figure B.7:Input Examples Given to the Agentic System. Thepotential causesare out- put by the causal discovery algorithm. For instance,DTC3is a cause ofunknown EPand has an Average Causal Effect (ACE) of 0.7, i.e., it increases the likelihood of observing the EP on average by 70% (Eq. (9.1)). Examples of defective vehicles exhibiting this error pattern are provided in theDTC samples. 185