Paper deep dive

Deep learning from a control vantage point

Álvaro Rodríguez Abella, João Pedro Silvestre, Bahman Gharesifard, Paulo Tabuada

Year: 2026Venue: arXiv preprintArea: eess.SYType: PreprintEmbeddings: 136

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 10/14/2026, 9:52:29 AM

Summary

This paper provides an overview of deep learning from a control theory perspective, highlighting the mathematical connections between neural network architectures and control systems. It explains how residual neural networks (ResNets) can be interpreted as the discretization of continuous-time control systems, enabling the use of geometric nonlinear control to analyze their function approximation capabilities. Additionally, it discusses how transformers can be viewed as the discretization of consensus algorithms and how backpropagation relates to optimal control. The article reviews feedforward networks, ResNets, transformers, large language models, and selective state-space models.

Entities (9)

Residual Neural Networks · model-architecture · 98%Transformers · model-architecture · 98%Backpropagation · algorithm · 95%Feedforward Neural Networks · model-architecture · 95%Large Language Models · model-architecture · 95%Optimal Control · theory · 95%Selective State Space Models · model-architecture · 92%Consensus Algorithm · algorithm · 90%Geometric Nonlinear Control · theory · 90%

Relation Signals (6)

Transformers → iscomponentof → Large Language Models

confidence 95% · transformers, the key architectural component of most large language-models

Residual Neural Networks → isdiscretizationof → Continuous-time Control System

confidence 95% · residual neural networks can be interpreted as the discretization of a continuous-time control system

Backpropagation → isrelatedto → Optimal Control

confidence 95% · connection between backpropagation... and optimal control

Transformers → isdiscretizationof → Consensus Algorithm

confidence 92% · transformers... can be seen the discretization of a consensus algorithm

Selective State Space Models → isinspiredby → Control Models

confidence 90% · selective state-space models, a recent class of models inspired by the models used in control

Residual Neural Networks → uses → Geometric Nonlinear Control

confidence 90% · ability of a residual neural network to approximate arbitrary continuous functions... can be solved using geometric nonlinear control

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:This article offers an overview of recent developments in deep learning from a control vantage point. It reviews several classes of models including feedforward and residual neural networks, transformers, large-language models, and selective state space models while explaining how control theory plays an important role in understanding their training, properties, or architecture design. The article explains how residual neural networks can be interpreted as the discretization of a continuous-time control system thereby enabling the use of control theoretic techniques for its analysis. To illustrate the power of this bridge between control and deep learning, two technical results are discussed: 1) the ability of a residual neural network to approximate arbitrary continuous functions is a non-standard controllability problem that can be solved using geometric nonlinear control and 2) transformers, the key architectural component of most large language-models, can be seen the discretization of a consensus algorithm and thus large language-models exhibit consensus. Another point of contact between control theory and deep learning is the connection between backpropagation, the algorithm used to train neural networks, and optimal control. The article addresses how the older connection between these problems, established in the 80's using discrete-time models can be reconciled with the more recent approach using continuous-time models. Several open problems and directions for future research are discussed, including selective state-space models, a recent class of models inspired by the models used in control.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

132,363 characters extracted from source content.

Expand or collapse full text

Copyright Notice © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: https://doi.org/10.1109/MCS.2026.3715420 arXiv:2610.14385v1 [eess.SY] 9 Oct 2026 Deep learning from a control vantage point FEEDFORWARD, RESIDUAL, TRANSFORMERS, LLMS, AND SELECTIVE STATE-SPACE MODELS Á. Rodríguez Abella , J. P. Silvestre , B. Gharesifard , and P. Tabuada Comillas Pontifical University, Queens University, University of California at Los Angeles D eep neural networks, in their different shapes and flavors, are having a remarkable impact in almost every area of science and engineering. While they are also influencing the field of control, the purpose of this article is to discuss recent efforts in using ideas from control to better under- stand and design neural networks. The connections between neural networks and control are both old and new. It was already observed in the ’80s that backpropagation, the main technique to train neural networks, is closely related to optimal control. A different connection emerged more recently, motivated by a class of neural networks termed residual neural networks. It was noticed that their update equations are similar to those resulting from the forward Euler discretization of differential equations. This insight motivated the use of differential equation models, and control systems in par- ticular, to describe deep neural networks. Consequently, several research efforts were initiated to tackle a range of problems in deep learning by leveraging the interplay between control and deep neural networks. It is from this control vantage point that this article offers a gentle introduction to deep learning with the objective of making its key ideas more accessible to a control audience while emphasizing how control theory has played, and continues to play, a fundamental role in deep learning. The choice of contents is based on the last author’s plenary talk at the 2024 IEEE CDC and it reflects the authors’ idiosyncratic view of the area. Any omission of relevant research efforts is simply a consequence of the authors’ incomplete view of the rapidly moving interface between control theory and deep learning. Other articles adopting a control theory perspective include [1], [2] and complement this article. ) Notwithstanding the topic of this article, the authors emphatically state that no AI tools were used to generate its contents. Summary T his article offers an overview of recent developments in deep learning from a control vantage point. It reviews several classes of models including feedforward and residual neural networks, transformers, large-language models, and selective state space models while explaining how control theory plays an important role in understanding their train- ing, properties, or architecture design. The article explains how residual neural networks can be interpreted as the discretization of a continuous-time control system thereby enabling the use of control theoretic techniques for its analy- sis. To illustrate the power of this bridge between control and deep learning, two technical results are discussed: 1) the ability of a residual neural network to approximate arbitrary continuous functions is a non-standard controllability prob- lem that can be solved using geometric nonlinear control and 2) transformers, the key architectural component of most large language-models, can be seen the discretization of a consensus algorithm and thus large language-models exhibit consensus. Another point of contact between con- trol theory and deep learning is the connection between backpropagation, the algorithm used to train neural net- 1066-033X/26©2026IEEEOCTOBER 2026 « IEEE CONTROL SYSTEMS 1 Notwithstanding the topic of this article, the authors emphatically state that no AI tools were used to generate its contents. works, and optimal control. The article addresses how the older connection between these problems, established in the 80’s using discrete-time models can be reconciled with the more recent approach using continuous-time models. Several open problems and directions for future research are discussed, including selective state-space models, a recent class of models inspired by the models used in control. A CONCISE HISTORICAL ACCOUNT OF ARTIFICIAL NEURAL NETWORKS Despite the recent rise into prominence of neural networks (NNs), their origins date back to the ’40s. This section provides a brief overview of the historical context of NNs. Readers not familiar with NNs are encouraged to first read the sidebar “What are feedforward neural networks?”. NNs were originally introduced as simple models of biological neurons and synapses. They were composed of two main components: neurons and the connections between them. An influential early model was introduced by McCulloch and Pitts in 1943 [3], who described neurons as Boolean threshold elements. Building on this line of work, Rosenblatt introduced the perceptron in 1958 [4], usually regarded as one of the first trainable artificial N architectures. Mathematically, each neuron can be described by a function transforming the input z in P R n into the output z out “ σpw z in ` bq P R,(0.1) where w P R 1ˆn denotes the weights, b P R denotes the bias, and σ : R Ñ R is the activation function σpzq “ # 1, z ě 0, 0, z ă 0. (0.2) The perceptron was a trainable network of such artificial neurons with supervised learning used to adjust its pa- rameters. This network could, in theory, solve any classification task whose classes are linearly separable, see “Classifica- tion vs regression tasks” sidebar. However, as noted in the book Perceptrons (1969) [5], it could not solve simple nonlinear problems such as classifying the inputs of the XOR function into the class tp0, 0q,p1, 1qu, mapped by the XOR function to 0, and the class tp0, 1q,p1, 0qu, mapped to 1. This limitation arises from the architectural limitations of single layer perceptron networks, and the lack, at the time, of an effective method for training networks with multiple layers. The book [5] is often cited as a contributing factor to the so-called “AI Winter”, a period where artificial intelligence (AI) research stagnated dramatically. For an overview of how AI relates to NNs and deep learning see the sidebar “AI, machine learning, or deep learning?”. The main breakthrough that pushed the field forward was the introduction of backpropagation. This approach introduced differentiable activation functions, such as sig- moids, along with a practical method for training mul- tilayer NNs [6], [7]. Other derivations of backpropaga- tion appeared earlier in different contexts such as [8]. Although these developments still shared some similarities with biological models, such as the sigmoid function used in the Wilson–Cowan model [9], the connection between artificial NNs and neuroscience gradually weakened. By the late ’80s, the two fields had largely diverged, with NNs increasingly driven by empirical results rather than biological realism. Nevertheless, same later models such as spiking NNs [10] retained closer ties to neuroscience, drawing inspiration from classical models including the Hodgkin–Huxley equations [11]. As the field progressed, many other N architectures were introduced. Among the most prominent were recur- rent neural networks (RNNs) [12], which were constructed as sequence-to-sequence models, that is, as systems trans- forming input signals into output signals. RNNs possess, in theory, the ability to retain information over arbitrarily long time horizons, allowing current outputs to be related to past inputs. In practice, this capability is limited by sev- eral issues, most notably the vanishing gradient problem (the use of gradients in the training of NNs is discussed in Optimal control and backpropagation) [13]. Despite these limitations, RNNs were widely used. Among the many different architectures that appeared in the literature, the following three have deep connections to control: residual neural networks, transformers, and selective state-space models. They are described in more detail in the sidebars “What are feedforward neural net- works?”, “What are residual neural networks?”, “What are transformers?”, and “What are recurrent neural networks and selective state-space models?” DIFFERENTIAL EQUATIONS, CONTROL SYSTEMS, AND NEURAL NETWORKS As discussed in A concise historical account of artificial neural networks, differential equations were already used in the 2 IEEE CONTROL SYSTEMS » OCTOBER 2026 CLASSIFICATION VS REGRESSION TASKS C lassification and regression are two common machine learning tasks. Both tasks can be seen as the learning of a mapΦ : X Ñ Y . The goal of classification is to separate data into finitely many different classes, hence, Y is a finite set. The classes are said to be linearly separable when X “ R m , for some m P N, and they can be separated by an affine subspace of X . In a regression task, data is to be mapped to real-valued outputs and thus Y is a subset of R n , for some n P N. AI, MACHINE LEARNING, OR DEEP LEARNING? A I is the acronym for Artificial Intelligence, a term that en- compasses a multitude of techniques aiming at creating machine intelligence. One such technique is machine learning but AI also includes other techniques such as logical and symbolic reasoning. Machine learning focuses on algorithms that learn from data while deep learning is a narrower class of such algorithms utilizing deep neural networks, that is, neural networks with a large number of layers. Consequently, (0.19) can be regarded as a control system and the problem of training a N becomes an open-loop control design problem. ’50s to model biological NNs. Surprisingly, it was recently observed that differential equations can also be used to de- scribe artificial NNs. This recent connection was motivated by an architecture called residual networks (ResNets), see sidebar “What are residual neural networks?” for a more detailed description of ResNets. What distinguishes ResNets from other architectures is how the state at layer k P t1, . . . ,ℓ ́ 1u, denoted by zpkq P R n , gives rise to the state at layer k` 1 zpk` 1q “ zpkq` wpkqΣpWpkqzpkqq.(0.18) The functionΣ : R n Ñ R n describes the action of an activation function σ : R Ñ R on each coordinate Σpvq “ pσpv 1 q, σpv 2 q, . . . , σpv n q. The argument ofΣ in (0.18) is the linear function of zpkq defined by the weights Wpkq P R nˆn . Note also the presence of the scalar weight wpkq P R. Although the original formulation in [18] allows for w to be a matrix, it is pedagogical to focus on the scalar case. It was observed in [30], [31], [32], [33] that (0.18) has a striking similarity with the numerical solution of the differential equation 9 z c ptq “ΣpW c ptqz c ptqq(0.19) given by the forward Euler method zpt k ` τq “ zpt k q` τΣpW c pt k qzpt k q, where τ P R ` is the step size and t k “ kτ. Hence, if the weights wpkq are small, it is conceivable that ResNets can be regarded as the discretization of (0.19). Before elaborating further on the possible equivalence between ResNets and differential equations, it matters to delve into the control perspective afforded by this connection. In (0.19), W c is a map W c : r0, 1s Ñ R nˆn transforming time t P r0, 1s into a matrix of weights. For convenience, the time domain was normalized to r0, 1s. The map W c is to be learned from data, that is, it is to be designed. This suggests that W c can be interpreted as a control input. Consequently, (0.19) can be regarded as a control system and the problem of training a N becomes an open-loop control design problem. This perspective brings into clear view how control theory can play a fundamental role in deep learning. Examples of how this perspective brought solutions to new and existing problems include an alternative connection between backpropagation and control (an earlier connection was identified in the ’80s), establishing deep neural networks as universal function approximators, and explaining the attention mechanism in transformers through consensus. Since the interplay between control systems and ResNets is the unifying theme of this article, a detailed discussion of when ResNets can be seen as the discretiza- tion of differential equations, or a differential equation can be seen as the continuous limit of a ResNet, is in order. Although outside the scope of this article, there is also a parallel line of work relating NNs and partial differential equations [34], [35], [36]. Fixed inputs Considerfirstthecasewherethecontrolinput W c : r0, 1s Ñ R nˆn isknownandfixedwherewe normalized time to the interval r0, 1s for simplicity. This represents situations where analysis or design was OCTOBER 2026 « IEEE CONTROL SYSTEMS 3 WHAT ARE FEEDFORWARD NEURAL NETWORKS? A neural network (N) is a family of functionsΦ W : R n 1 Ñ R o parameterized by a set of weights W . The functionΦ W is built by composing simpler functions Φ W “ h ̋ g ℓ ́1 ̋ g ℓ ́2 ̋ . . . ̋ g 1 ,(S3) where each g k : R n k Ñ R n k`1 takes a different form depending on the architecture and h : R n ℓ Ñ R o is chosen according to the task to be performed. The length or depth of the N is denoted byℓ P N. A N is said to be a deep neural network when ℓ is large. The domains and co-domains of the functions g i are often interpreted as layers, especially when a N is represented as a graph, see Figure S1. Under such interpretation, the state zpkq P R n k describes the contents of each of the n k neurons in layer k and the function g k transforms the state zpkq of layer k to the state zpk ` 1q P R n k`1 of layer k ` 1. In general, different layers have different dimensions. For notational simplicity, fix n def “ max k n k , and lift each function g k : R n k Ñ R n k`1 to the function r g k : R n Ñ R n r g k pzq “ pg k ,1 pz 1 , . . . , z n k q, . . . , g k ,n k`1 pz 1 , . . . , z n k q, 0, . . . , 0 looomooon n ́n k`1 zeros q that has the same domain and codomain. In this article g k is used to directly denote its lifted version. Layer 1Layer 2Layer 3 Layer 4 Input Ouput <latexit sha1_base64="T5FgXpxDMIAwf1PLA6RhwGI/G+g=">AAACZnicdVHLTgIxFO2MLxxfoDEu3DQSEtyQGWPUJYkb4woTeSRASKfTgUpnOmnvGJDwD271z/wDP8MOsBDQmzQ5Oef09vZcPxFcg+t+WfbG5tb2Tm7X2ds/ODzKF44bWqaKsjqVQqqWTzQTPGZ14CBYK1GMRL5gTX94n+nNV6Y0l/EzjBPWjUg/5iGnBAzVeOt5Ze+yly+6FXdWeB14C1BEi6r1CtZjJ5A0jVgMVBCt256bQHdCFHAq2NTppJolhA5Jn01mQ05xyVABDqUyJwY8Y5d8JNJ6HPnGGREY6FUtI//Tso56+VUNEVFjFUyd0m86Gf3hzXqClGKFFgTYyIy0xLZTCO+6Ex4nKbCYzv8VpgKDxFnAOOCKURBjAwhV3ASC6YAoQsGswenwmIo0YBOj9pUZfeqY9L3VrNdB46ri3VTcp+tiFS/2kEPn6AKVkYduURU9oBqqI4pe0Dv6QJ/Wt31on9pnc6ttLe6coKWy8Q+Vw7wu</latexit> z 1 (1) <latexit sha1_base64="KAUl/NurT5i4vbD1YZgL4qKxH0Y=">AAACZnicdVHLTgIxFC3jC0dF0BgXbhoJCW7IDDHqksSNcYWJPBIgpNMpUOlMJ+0dA5L5B7f6Z/6Bn2EHWAjoTZqcnHN6e3uuFwmuwXG+MtbW9s7uXnbfPjg8yh3nCydNLWNFWYNKIVXbI5oJHrIGcBCsHSlGAk+wlje+T/XWK1Oay/AZphHrBWQY8gGnBAzVfOtXy+5VP190Ks688CZwl6CIllXvFzKPXV/SOGAhUEG07rhOBL0ZUcCpYIndjTWLCB2TIZvNh0xwyVA+HkhlTgh4zq74SKD1NPCMMyAw0utaSv6npR316qsaAqKmyk/s0m86mvzhTXuClGKNFgTYxIy0wnZiGNz1ZjyMYmAhXfxrEAsMEqcBY58rRkFMDSBUcRMIpiOiCAWzBrvLQypin82MOlRm9MQ26bvrWW+CZrXi3lScp+tiDS/3kEUX6BKVkYtuUQ09oDpqIIpe0Dv6QJ+ZbytnnVnnC6uVWd45RStl4R+Xw7wv</latexit> z 2 (1) <latexit sha1_base64="DJaw8Aq7r3hTb7AOO4pnYuJch2U=">AAACZnicdVHLTgIxFC3jC0dF0BgXbhoJCW7IDDHqksSNcYWJPBIgpNMpUOlMJ+0dA5L5B7f6Z/6Bn2EHWAjoTZqcnHN6e3uuFwmuwXG+MtbW9s7uXnbfPjg8yh3nCydNLWNFWYNKIVXbI5oJHrIGcBCsHSlGAk+wlje+T/XWK1Oay/AZphHrBWQY8gGnBAzVfOu75epVP190Ks688CZwl6CIllXvFzKPXV/SOGAhUEG07rhOBL0ZUcCpYIndjTWLCB2TIZvNh0xwyVA+HkhlTgh4zq74SKD1NPCMMyAw0utaSv6npR316qsaAqKmyk/s0m86mvzhTXuClGKNFgTYxIy0wnZiGNz1ZjyMYmAhXfxrEAsMEqcBY58rRkFMDSBUcRMIpiOiCAWzBrvLQypin82MOlRm9MQ26bvrWW+CZrXi3lScp+tiDS/3kEUX6BKVkYtuUQ09oDpqIIpe0Dv6QJ+ZbytnnVnnC6uVWd45RStl4R+Xwbwv</latexit> z 1 (2) <latexit sha1_base64="HUkDLp2SNZROCaJnC7iCOplgwsk=">AAACZnicdVHLTgIxFC3jC0dF0BgXbhoJCW7IjBp1SeLGuMJEHgkQ0ukUqHSmk/aOAcn8g1v9M//Az7ADLAT0Jk1Ozjm9vT3XiwTX4DhfGWtjc2t7J7tr7+0f5A7zhaOGlrGirE6lkKrlEc0ED1kdOAjWihQjgSdY0xvdp3rzlSnNZfgMk4h1AzIIeZ9TAoZqvPXc8tVFL190Ks6s8DpwF6CIFlXrFTKPHV/SOGAhUEG0brtOBN0pUcCpYIndiTWLCB2RAZvOhkxwyVA+7ktlTgh4xi75SKD1JPCMMyAw1KtaSv6npR318qsaAqImyk/s0m86Gv/hTXuClGKFFgTY2Iy0xLZj6N91pzyMYmAhnf+rHwsMEqcBY58rRkFMDCBUcRMIpkOiCAWzBrvDQypin02NOlBm9MQ26burWa+DxmXFvak4T9fFKl7sIYvO0DkqIxfdoip6QDVURxS9oHf0gT4z31bOOrFO51Yrs7hzjJbKwj+Zv7w</latexit> z 1 (3) <latexit sha1_base64="GxBpU824vrdJVHw3E0rnWSUX6UQ=">AAACZnicdVHLTgIxFC3jC0dF0BgXbhoJCW7IjCHqksSNcYWJPBIgpNMpUOlMJ+0dAxL+wa3+mX/gZ9iBWQjoTZqcnHN6e3uuFwmuwXG+MtbW9s7uXnbfPjg8yh3nCydNLWNFWYNKIVXbI5oJHrIGcBCsHSlGAk+wlje+T/TWK1Oay/AZphHrBWQY8gGnBAzVfOu75epVP190Ks6i8CZwU1BEadX7hcxj15c0DlgIVBCtO64TQW9GFHAq2NzuxppFhI7JkM0WQ85xyVA+HkhlTgh4wa74SKD1NPCMMyAw0utaQv6nJR316qsaAqKmyp/bpd90NPnDm/QEKcUaLQiwiRlphe3EMLjrzXgYxcBCuvzXIBYYJE4Cxj5XjIKYGkCo4iYQTEdEEQpmDXaXh1TEPpsZdajM6HPbpO+uZ70JmtcV96biPFWLNZzuIYsu0CUqIxfdohp6QHXUQBS9oHf0gT4z31bOOrPOl1Yrk945RStl4R+bvbwx</latexit> z 1 (4) <latexit sha1_base64="F0VvJPop9beXpjrbnu950jtlK6Y=">AAACZnicdVHLTgIxFC3jC0dF0BgXbhoJCW7IDDHqksSNcYWJPBIgpNMpUOlMJ+0dA5L5B7f6Z/6Bn2EHWAjoTZqcnHN6e3uuFwmuwXG+MtbW9s7uXnbfPjg8yh3nCydNLWNFWYNKIVXbI5oJHrIGcBCsHSlGAk+wlje+T/XWK1Oay/AZphHrBWQY8gGnBAzVfOtXy9Wrfr7oVJx54U3gLkERLaveL2Qeu76kccBCoIJo3XGdCHozooBTwRK7G2sWETomQzabD5ngkqF8PJDKnBDwnF3xkUDraeAZZ0BgpNe1lPxPSzvq1Vc1BERNlZ/Ypd90NPnDm/YEKcUaLQiwiRlphe3EMLjrzXgYxcBCuvjXIBYYJE4Dxj5XjIKYGkCo4iYQTEdEEQpmDXaXh1TEPpsZdajM6Ilt0nfXs94EzWrFvak4T9fFGl7uIYsu0CUqIxfdohp6QHXUQBS9oHf0gT4z31bOOrPOF1Yrs7xzilbKwj+Zwbww</latexit> z 2 (2) <latexit sha1_base64="HKCabKYADODuQHswwObWB4ls7lM=">AAACZnicdVHLTgIxFC3jC8cXaIwLN42EBDdkBo26JHFjXGEijwQI6XQKVDrTSXvHgGT+wa3+mX/gZ9gBFgJ6kyYn55ze3p7rRYJrcJyvjLWxubW9k9219/YPDo9y+eOGlrGirE6lkKrlEc0ED1kdOAjWihQjgSdY0xvdp3rzlSnNZfgMk4h1AzIIeZ9TAoZqvPUqpavLXq7glJ1Z4XXgLkABLarWy2ceO76kccBCoIJo3XadCLpTooBTwRK7E2sWEToiAzadDZngoqF83JfKnBDwjF3ykUDrSeAZZ0BgqFe1lPxPSzvq5Vc1BERNlJ/Yxd90NP7Dm/YEKcUKLQiwsRlpiW3H0L/rTnkYxcBCOv9XPxYYJE4Dxj5XjIKYGECo4iYQTIdEEQpmDXaHh1TEPpsadaDM6Ilt0ndXs14HjUrZvSk7T9eFKl7sIYvO0QUqIRfdoip6QDVURxS9oHf0gT4z39ahdWqdza1WZnHnBC2VhX8Am7+8MQ==</latexit> z 2 (3) <latexit sha1_base64="+OOQSR/2viSU5lBSAUvyYyeF7wc=">AAACZnicdVHLTgIxFC3jC8cXaIwLN42EBDdkhhB1SeLGuMJEHgkQ0ukUqHSmk/aOAQn/4Fb/zD/wM+zALAT0Jk1Ozjm9vT3XiwTX4DhfGWtre2d3L7tvHxweHZ/k8qdNLWNFWYNKIVXbI5oJHrIGcBCsHSlGAk+wlje+T/TWK1Oay/AZphHrBWQY8gGnBAzVfOtXStXrfq7glJ1F4U3gpqCA0qr385nHri9pHLAQqCBad1wngt6MKOBUsLndjTWLCB2TIZsthpzjoqF8PJDKnBDwgl3xkUDraeAZZ0BgpNe1hPxPSzrq1Vc1BERNlT+3i7/paPKHN+kJUoo1WhBgEzPSCtuJYXDXm/EwioGFdPmvQSwwSJwEjH2uGAUxNYBQxU0gmI6IIhTMGuwuD6mIfTYz6lCZ0e2Sd9dz3oTNCtl96bsPFULNZzuIYsu0RUqIRfdohp6QHXUQBS9oHf0gT4z39axdW5dLK1WJr1zhlbKwj+dvbwy</latexit> z 2 (4) <latexit sha1_base64="Y/v4/7udKxMsEtGvgzZywvXEAao=">AAACZnicdVHLTgIxFC3jC8cXaIwLN42EBDdkBo26JHFjXGEijwQI6XQKVDrTSXvHgGT+wa3+mX/gZ9gBFgJ6kyYn55ze3p7rRYJrcJyvjLWxubW9k9219/YPDo9y+eOGlrGirE6lkKrlEc0ED1kdOAjWihQjgSdY0xvdp3rzlSnNZfgMk4h1AzIIeZ9TAoZqvPWuSpXLXq7glJ1Z4XXgLkABLarWy2ceO76kccBCoIJo3XadCLpTooBTwRK7E2sWEToiAzadDZngoqF83JfKnBDwjF3ykUDrSeAZZ0BgqFe1lPxPSzvq5Vc1BERNlJ/Yxd90NP7Dm/YEKcUKLQiwsRlpiW3H0L/rTnkYxcBCOv9XPxYYJE4Dxj5XjIKYGECo4iYQTIdEEQpmDXaHh1TEPpsadaDM6Ilt0ndXs14HjUrZvSk7T9eFKl7sIYvO0QUqIRfdoip6QDVURxS9oHf0gT4z39ahdWqdza1WZnHnBC2VhX8Am8G8MQ==</latexit> z 3 (2) <latexit sha1_base64="4lLloYE9ZLV+88jloWUWDSl7dBI=">AAACZnicdVHLTgIxFC3jC8cXaIwLN42EBDdkRoy6JHFjXGEijwQI6XQKVDrTSXvHgGT+wa3+mX/gZ9gBFgJ6kyYn55ze3p7rRYJrcJyvjLWxubW9k9219/YPDo9y+eOGlrGirE6lkKrlEc0ED1kdOAjWihQjgSdY0xvdp3rzlSnNZfgMk4h1AzIIeZ9TAoZqvPUqpcplL1dwys6s8DpwF6CAFlXr5TOPHV/SOGAhUEG0brtOBN0pUcCpYIndiTWLCB2RAZvOhkxw0VA+7ktlTgh4xi75SKD1JPCMMyAw1KtaSv6npR318qsaAqImyk/s4m86Gv/hTXuClGKFFgTY2Iy0xLZj6N91pzyMYmAhnf+rHwsMEqcBY58rRkFMDCBUcRMIpkOiCAWzBrvDQypin02NOlBm9MQ26burWa+DxlXZvSk7T9eFKl7sIYvO0QUqIRfdoip6QDVURxS9oHf0gT4z39ahdWqdza1WZnHnBC2VhX8Anb+8Mg==</latexit> z 3 (3) <latexit sha1_base64="lq/YwZn6Mh2ilmlAF7bryyyDCH8=">AAACZnicdVHLTgIxFC3jC8cXaIwLN42EBDdkRom6JHFjXGEijwQI6XQKVDrTSXvHgGT+wa3+mX/gZ9gBFgJ6kyYn55ze3p7rRYJrcJyvjLWxubW9k9219/YPDo9y+eOGlrGirE6lkKrlEc0ED1kdOAjWihQjgSdY0xvdp3rzlSnNZfgMk4h1AzIIeZ9TAoZqvPWuS5XLXq7glJ1Z4XXgLkABLarWy2ceO76kccBCoIJo3XadCLpTooBTwRK7E2sWEToiAzadDZngoqF83JfKnBDwjF3ykUDrSeAZZ0BgqFe1lPxPSzvq5Vc1BERNlJ/Yxd90NP7Dm/YEKcUKLQiwsRlpiW3H0L/rTnkYxcBCOv9XPxYYJE4Dxj5XjIKYGECo4iYQTIdEEQpmDXaHh1TEPpsadaDM6Ilt0ndXs14Hjauye1N2niqFKl7sIYvO0QUqIRfdoip6QDVURxS9oHf0gT4z39ahdWqdza1WZnHnBC2VhX8An728Mw==</latexit> z 3 (4) <latexit sha1_base64="5ClEY6oR3Xz2i5qkCGZCqeVIoyg=">AAACZnicdVHLTgIxFC3jC8cXaIwLN42EBDdkhhB1SeLGuMJEHgkQ0ukUqHSmk/aOAQn/4Fb/zD/wM+zALAT0Jk1Ozjm9vT3XiwTX4DhfGWtre2d3L7tvHxweHZ/k8qdNLWNFWYNKIVXbI5oJHrIGcBCsHSlGAk+wlje+T/TWK1Oay/AZphHrBWQY8gGnBAzVfOtXS5Xrfq7glJ1F4U3gpqCA0qr385nHri9pHLAQqCBad1wngt6MKOBUsLndjTWLCB2TIZsthpzjoqF8PJDKnBDwgl3xkUDraeAZZ0BgpNe1hPxPSzrq1Vc1BERNlT+3i7/paPKHN+kJUoo1WhBgEzPSCtuJYXDXm/EwioGFdPmvQSwwSJwEjH2uGAUxNYBQxU0gmI6IIhTMGuwuD6mIfTYz6lCZ0e2Sd9dz3oTNCtl96bsPFULNZzuIYsu0RUqIRfdohp6QHXUQBS9oHf0gT4z39axdW5dLK1WJr1zhlbKwj+dwbwy</latexit> z 4 (2) <latexit sha1_base64="mmXuO8VKGKsJ1hSx+Vep7bxOVAw=">AAACZnicdVHLTgIxFC3jC0dF0BgXbhoJCW7IjCHqksSNcYWJPBIgpNMpUOlMJ+0dAxL+wa3+mX/gZ9iBWQjoTZqcnHN6e3uuFwmuwXG+MtbW9s7uXnbfPjg8yh3nCydNLWNFWYNKIVXbI5oJHrIGcBCsHSlGAk+wlje+T/TWK1Oay/AZphHrBWQY8gGnBAzVfOtXy9Wrfr7oVJxF4U3gpqCI0qr3C5nHri9pHLAQqCBad1wngt6MKOBUsLndjTWLCB2TIZsthpzjkqF8PJDKnBDwgl3xkUDraeAZZ0BgpNe1hPxPSzrq1Vc1BERNlT+3S7/paPKHN+kJUoo1WhBgEzPSCtuJYXDXm/EwioGFdPmvQSwwSJwEjH2uGAUxNYBQxU0gmI6IIhTMGuwuD6mIfTYz6lCZ0e2Sd9dz3oTNK8r7k3FeaoWazjdQxZdoEtURi66RTX0gOqogSh6Qe/oA31mvq2cdWadL61WJr1zilbKwj+hvbw0</latexit> z 4 (4) FIGURE S1 Neural network depicted as a graph. The vertically stacked circles represent neurons and form a layer. The arrows from layer k to layer k ` 1 represent the different entries of the weight matrix Wpkq. Feedforward Networks Feedforward networks are among the earliest N architectures and still widely used. In a feedforward N, data moves only in one direction: it enters at the first or input layer as zp1q, is successively transformed by the sequence of functions g k producing zpk ` 1q, and results in the output hpzpℓq. In feed- forward NNs the maps g k take the form g k pzq “ΣpWpkqzq,(S4) where the functionΣ : R n Ñ R n operates element-wise, that is Σpvq “ pσpv 1 q, σpv 2 q, . . . , σpv n q with σ : R Ñ R an activation function. The argument ofΣ is the linear function of z defined by the weights Wpkq P R nˆn . In the literature, the argument ofΣ is often an affine function Wpkqz ` bpkq. To simplify the presentation, a linear represen- tation is used; it can be obtained by writing Wpkqz ` bpkq in the linear form « Wpkqbpkq 01 f « z 1 f . In this article W and z are used to directly denote their lifted variants. Classical feedforward NNs used the logistic sigmoid σpv i q “ p1` e ́v i q ́1 or the hyperbolic tangent σpv i q “ tanhpv i q as activation functions. While smooth, these functions can saturate and attenuate gradients thus making training more challenging. To facilitate deeper architectures, modern networks predominantly employ the Rectified Linear Unit (ReLU), σpv i q “ maxt0, v i u [14], and its variants. More recently, smooth gating- style activations like the Gaussian Error Linear Unit (GELU) [15], defined as σpv i q “ v i 2 ́ 1` e ́ v i ? 2 ̄ ̄ , epv i q “ 2 ? π ş v i 0 e ́t 2 dt , have become standard in transformer-based models. Task Dependence The choice of the function h in (S3) must be made according to the task being solved. Two common cases are: Regression: In a regression problem the output of a N is an element of R o and it is common to choose h to be affine hpzpℓq “ Azpℓq` c ,(S5) where A P R oˆn and c P R o are weights learned from data. Classification: In a classification problem, the output is the class to which the input belongs. Rather than directly producing the class, NNs produce a vector that is interpreted as the probability of the input belonging to each class. Hence, if there are o P N classes, the output is a vector in ∆ o def “ # p P R o : p i ě 0, o ÿ i“1 p i “ 1 + . The output map h in (S3) is the composition of two maps. The first is an affine map acting on zpℓq and producing logits q P R o q “ Azpℓq` c , and the second converts the logits into an element p P∆ o using the softmax function p i “ softmax i pqq def “ e q i ř o j“1 e q j ,i P t1, . . . , ou.(S6) Each p i is then interpreted as the probability of the given input zp1q belonging to class i . A more detailed introduction to this class of networks can be found in [16, Chapter 6]. 4 IEEE CONTROL SYSTEMS » OCTOBER 2026 WHAT ARE RESIDUAL NEURAL NETWORKS? I n recent years, the surge in computational power and large-scale datasets has enabled the training of substan- tially deeper networks, beginning with milestones such as AlexNet [17], which established convolutional neural networks as the standard for visual recognition. However, as depth in- creases, the training of such networks becomes notoriously dif- ficult. Deep residual networks (ResNets) [18] addressed this by introducing residual (skip) connections, which allow the gradient to bypass intermediate layers, see Figure S1. Formally, the functions g k (see (S3)) in a ResNet are of the form g k pzq “ z ` W 2 pkqΣpW 1 pkqzq,(S7) where W 1 pkq, W 2 pkq P R nˆn are weight matrices. It has been empirically shown that this architectural change, with respect to feedforward NNs, effectively smoothens the training loss landscape and mitigates the vanishing gradient problem [19] (the use of gradients in the training of NNs is discussed in Opti- mal control and backpropagation) thereby enabling the reliable training of ResNets with hundreds of layers. <latexit sha1_base64="6r4FrMs/VLvzKs+fzcidCaMnE08=">AAACYXicdVHLagIxFI3Tl05fapduhopQKMhMKW2XQjelK4X6ABXJZK4ampkMyZ2iiF/QbftxXfdHmtFZVG0vBA7nnNzcnOvHgmt03a+ctbd/cHiUL9jHJ6dn58VSuaNlohi0mRRS9XyqQfAI2shRQC9WQENfQNd/fUz17hsozWX0gvMYhiGdRHzMGUVDta5Hxapbd1fl7AIvA1WSVXNUyj0PAsmSECJkgmrd99wYhwuqkDMBS3uQaIgpe6UTWKzmWzo1QwXOWCpzInRW7IaPhlrPQ984Q4pTva2l5H9a2lFvvqoxpGqugqVd+03Hsz+8aU+UUmzRgiLMzEgbbD/B8cNwwaM4QYjY+l/jRDgonTRbJ+AKGIq5AZQpbgJx2JQqytBswB7wiIkkgIVRJ8qMvrRN+t521rugc1P37upu67bacLM95EmFXJIr4pF70iBPpEnahBEg7+SDfOa+rYJVtMprq5XL7lyQjbIqPxyyuqs=</latexit> + <latexit sha1_base64="06mHlaZGyeXaa99JEYFCp5Wheo8=">AAACenicdVFNTwIxEC3rF65foBcTLxsICcSE7BKiHkm8GE8Y5SMBsul2CzZ0t5t21oAEf41X/T/+Fw92YQ8COskkr++9TqczXsSZAtv+yhhb2zu7e9l98+Dw6Pgklz9tKxFLQltEcCG7HlaUs5C2gAGn3UhSHHicdrzxbaJ3XqhUTIRPMI3oIMCjkA0ZwaApN3fecWvlcaX/yEYBLndcRx9edVbcXNGu2ouwNoGTgiJKo+nmM/d9X5A4oCEQjpXqOXYEgxmWwAinc7MfKxphMsYjOls0PrdKmvKtoZA6Q7AW7IoPB0pNA087AwzPal1LyP+0pKJafVVBgOVU+nOz9JuOJn94k5ogBF+jOQY60S2tsL0YhjeDGQujGGhIlv8axtwCYSVDt3wmKQE+1QATyfRALPKMJSagV2P2WUh47NOZVkdStz439fSd9Vlvgnat6lxV7Yd6sWGne8iiC1RAZeSga9RAd6iJWoigN/SOPtBn5tsoGBXjcmk1MumdM7QSRv0HIq3Csg==</latexit> W 2 (k)⌃(W 1 (k)z(k)) <latexit sha1_base64="167RAvS3RLREjAic849QQgutm18=">AAACZHicdVFdS8MwFM3q16xTN4dPghTHQF9GK6I+DnwRnya4D9jGSNNsC0ubktzK6uhf8FX/mn/A32G69cFueiFwOOfk5uZcN+RMgW1/FYyt7Z3dveK+eVA6PDouV046SkSS0DYRXMieixXlLKBtYMBpL5QU+y6nXXf2kOrdVyoVE8ELxCEd+ngSsDEjGFLq7XJ2NSrX7Ia9LGsTOBmooaxao0rhaeAJEvk0AMKxUn3HDmG4wBIY4TQxB5GiISYzPKGL5YiJVdeUZ42F1CcAa8nmfNhXKvZd7fQxTNW6lpL/aWlHlX9VgY9lLL3ErP+mw/kf3rQnCMHXaI6BzvVIObYfwfh+uGBBGAENyOpf44hbIKw0XstjkhLgsQaYSKYDscgUS0xAL8EcsIDwyKMLrU6kHj0xdfrOetaboHPdcG4b9vNNrWlneyiiM3SBLpGD7lATPaIWaiOCpugdfaDPwrdRMqrG6cpqFLI7VZQr4/wHofW71A==</latexit> z(k) <latexit sha1_base64="YNAv12h4flec+yrpclKzjwvAbuY=">AAACZnicdVFdSwJBFB23L9vKtIgeehkSwQhkN6J6FHqJngzyA1RkdnbUydmdZeZuaIv/odf6Z/2Dfkaz6kNqXRg4nHPmzp1zvUhwDY7zlbE2Nre2d7K79t7+Qe4wXzhqaBkryupUCqlaHtFM8JDVgYNgrUgxEniCNb3Rfao3X5nSXIbPMIlYNyCDkPc5JWCoxlt5dOle9PJFp+LMCq8DdwGKaFG1XiHz2PEljQMWAhVE67brRNBNiAJOBZvanViziNARGbBkNuQUlwzl475U5oSAZ+ySjwRaTwLPOAMCQ72qpeR/WtpRL7+qISBqovypXfpNR+M/vGlPkFKs0IIAG5uRlth2DP27bsLDKAYW0vm/+rHAIHEaMPa5YhTExABCFTeBYDokilAwa7A7PKQi9lli1IEyo09tk767mvU6aFxV3JuK83RdrDqLPWTRGTpHZeSiW1RFD6iG6oiiF/SOPtBn5tvKWSfW6dxqZRZ3jtFSWfgHplm8RA==</latexit> z(k+ 1) FIGURE S1 Graphical description of a skip connection in a ResNet. conducted using the model (0.19) and the resulting conclusions are to be carried over, or implemented, on a ResNet. One example is the use of (0.19) to establish existence of inputs guaranteeing universal approximation, as discussed in Universal approximation, and then seeking to determine if ResNets have the same ability. Using the Euclidean norm, denoted by ̈, questions of this type can be distilled to: Given a desired error tolerance ε P R ` , is there a number of layersℓ ě 2 and a choice of weights w so that the mismatch between the solution z c : r0, 1s Ñ R n of 9 z c ptq “ΣpW c ptqz c ptqq and the solution z : t1, 2, . . . ,ℓu Ñ R n of the recursion zp1q “ z c p0q, zpk` 1q “ zpkq` wpkqΣ ˆ W c ˆ k ́ 1 ℓ ́ 1 ̇ zpkq ̇ ,(0.20) satisfies › › › › z c ˆ k ́ 1 ℓ ́ 1 ̇ ́ zpkq › › › › ď ε(0.21) for every k P t1, . . . ,ℓu? Note how the control input W c : r0, 1s Ñ R nˆn directly defines the weights W : t1, 2, . . . ,ℓu Ñ R nˆn of the ResNet Wpkq “ W c ˆ k ́ 1 ℓ ́ 1 ̇ and the only degrees of freedom are the number of layers ℓ and the weights w that play the role of step size. This is a classical problem in the numerical analysis of differential equations and the answer is affirmative under suitable technical conditions, see, for example [37]. A particular choice of such conditions is continuous differentiability ofΣ, boundedness of W c , and the assumption that (0.21) only needs to hold for solutions with z c p0q belonging to a compact set. Note that implicit in the problem formulation is that solutions of (0.19) exist in the time domain r0, 1s. In general, lower values of ε will require largerℓ, that is, deeper ResNets. Since ε in (0.21) can be rendered as small as desired, the differential equation model (0.19) can be seen as the limit of a ResNet whenℓ Ñ 8 for a suitable choice of weights w such as wpkq “ 1ℓ. Learned inputs Whereas the case of known and fixed control inputs can be resolved using numerical analysis, there are situations where the control inputs are not known beforehand. One such example occurs when the model (0.19) is used to design an algorithm that learns W c : r0, 1s Ñ R nˆn from data. Denote such algorithm by a map φ c : pR n ˆ R n q s Ñ pR nˆn q r0,1s sending a data set D P pR n ˆ R n q s into W c “ φ c pDq. The notation pR nˆn q r0,1s denotes the set of all maps from r0, 1s to R nˆn and a data set is a collection of pairs D “ ppx 1 , y 1 q,px 2 , y 2 q, . . . ,px s , y s q where y i is the evalu- ation of some unknown function on the data point x i . When using supervised learning, the objective of training a N is to learn the values of the weights so that the N approximates the unknown function. Training of NNs is described in more detail in Optimal control and backpropa- gation. Typically, φ c models the construction of W c through the minimization of a loss functional based on the training data W c “ argmin x W c ż 1 0 L c ́ D, x W c ptq ̄ dt.(0.22) In order to implement such algorithm, (0.19) is discretized to obtain a ResNet and φ c is discretized to obtain a family of maps φ ℓ : pR n ˆ R n q s Ñ pR nˆn q t1,2,...,ℓu parameterized by the lengthℓ of the network. When φ c is defined through (0.22), φ ℓ would also be defined by the minimization of a suitably discretized loss W “ argmin x W ℓ ́1 ÿ k“1 L ℓ ́ D, x Wpkq ̄ .(0.23) OCTOBER 2026 « IEEE CONTROL SYSTEMS 5 WHAT ARE TRANSFORMERS? A common requirement in modern applications of NNs is to approximate functions mapping signals to signals, also known as sequence-to-sequence mappings. One example is the translation of text from one language to another. In such settings, the input is a sequence ps 1 , . . . , s m q,s i P R n (S8) and the output can be a sequence or a classification label. It has been empirically shown that the transformer architec- ture [20] outperforms feedforward NNs in learning sequence- to-sequence maps. Transformers handle sequences by build- ing intermediate token representations and repeatedly mixing information across different elements of the sequence using an attention mechanism. Attention mechanisms were popularized in neural machine translation [21] and later instantiated as self- attention in transformers [20]. This sidebar provides a brief introduction to these concepts. The transformer architecture can still be described by a map similar to (S3) Φ W “ h ̋ g ℓ ́1 ̋ g ℓ ́2 ̋ . . . ̋ g 1 ̋ f .(S9) The function f : R nˆm Ñ R dˆm is called the embedding map and sends the matrix representation of the sequence (S8) S “ ” s 1 |s 2 | . . .|s m ı P R nˆm (S10) into a token matrix in R dˆm . The maps g k : R dˆm Ñ R dˆm describe the transformer blocks, and the map h is the output map. The domain of h is R dˆm and its co-domain depends on the task. Embedding and positional information: The embedding map f is typically of the form fpSq “ ” Eps 1 , 1q|Eps 2 , 2q| . . .|E m ps m , mq ı for a function E : R n ˆ N Ñ R d whose form depends on the specific application and whose parameters are learned from data. For the sake of brevity, we also call each element z i p1q “ Eps i , iq(S11) a token although the term token is often reserved for the elements s i . Collecting tokens column-wise leads to the token matrix representation Zp1q def “ ” z 1 p1q|. . . |z m p1q ı P R dˆm (S12) that will be used in the next sections. Self-attention and the transformer block At each layer k P t1, . . . ,ℓ ́ 1u, the transformer updates Zpkq P R dˆm using multi-head self-attention followed by a token-wise feedforward N layer, each combined with a residual connection and a normalization function [20]. Denote by a P N the total number of heads in a transformer. A head is simply a parallel path in the architecture along which an attention mechanism is implemented. The output of all the heads is then combined into a single output. Define the key K j pkq P R eˆm , query Q j pkq P R eˆm , and value V j pkq P R e 1 ˆm matrices for the j -th head as Q j pkq “ W Q j pkqZpkq, K j pkq “ W K j pkqZpkq, V j pkq “ W V j pkqZpkq, where W Q j pkq, W K j pkq P R eˆd and W V j pkq P R e 1 ˆd are weight matrices. The output of the attention mechanism is given by summing over all the attention heads ApZpkqq “ a ÿ j“1 W o j V j pkqsoftmax ̃ K J j pkqQ j pkq ? e ̧ ,(S13) where softmax is applied column-wise and W o j P R dˆe 1 is another weight matrix. The matrix K J j pkqQ j pkq contains the inner products between the columns of Z when using the inner product defined by the weight matrix W J K j pkqW Q j pKq. In particular, the entry pi 1 , i 2 q of K J j pkqQ j pkq measures how much token i 1 is aligned with token i 2 . The division by ? e prevents the inner products in K J j pkqQ j pkq from growing with the dimension. The standard transformer block update, that is, the map g k in (S9), can then be written as the composition of two maps: r Zpkq “ N ` Zpkq` ApZpkqq ̆ ,(S14) Zpk ` 1q “ N ` r Zpkq` R k p r Zpkqq ̆ ,(S15) where N : R dˆm Ñ R dˆm is a normalization map applied token-wise (column-wise) and R k pZpkqq “ pr k pz 1 q, . . . , r k pz m q, with r k defined by r k pz i q “ W 2 pkqΣpW 1 pkqz i q, as depicted in Figure S1. In the original transformer architecture, there are two types of transformer blocks: encoders and decoders. In the encoder blocks, attention is typically computed over all token positions 1, . . . , m. In the decoder blocks, attention is made causal, that is, (S13) is modified so that attention for token i only depends on the tokens that precede it, that is, tokens 1 through i ́ 1 [20]. Output and text generation: The transformer output is task-dependent. For classification, the output map h is typically the composition of two maps, similarly to what is done in feed- forward NNs, see sidebar “What are residual neural networks?”. The first is an affine map acting on a designated token (often the final token or a special classification token) to obtain logits. The second map is the entry-wise softmax to produce a distribution over classes. For text generation, the output map is similarly defined but the class set is the vocabulary of the language being used. The output map sends a token matrix to logits over all the vocabulary words, and the softmax converts these logits into a probability distribution for the next token. The next token is then selected by a decoding rule, passed through the embedding map f and appended to the end of the token matrix Zp1q. The process is then restarted by processing the enlarged token matrix starting with the map g 1 . This text generation process ends when a special end of sentence or end of text token is generated. In practice, decoding often modifies the softmax distribution before sampling. Two common rules are: (i) top-k sampling, which restricts sampling to the k most probable tokens; and (i) top-p sampling, which restricts sampling to the smallest set of tokens whose cumulative probability is at least p, see [22]. Readers seeking additional background on large language models and transformers may consult [23, Chapter 7-8]. <latexit sha1_base64="6r4FrMs/VLvzKs+fzcidCaMnE08=">AAACYXicdVHLagIxFI3Tl05fapduhopQKMhMKW2XQjelK4X6ABXJZK4ampkMyZ2iiF/QbftxXfdHmtFZVG0vBA7nnNzcnOvHgmt03a+ctbd/cHiUL9jHJ6dn58VSuaNlohi0mRRS9XyqQfAI2shRQC9WQENfQNd/fUz17hsozWX0gvMYhiGdRHzMGUVDta5Hxapbd1fl7AIvA1WSVXNUyj0PAsmSECJkgmrd99wYhwuqkDMBS3uQaIgpe6UTWKzmWzo1QwXOWCpzInRW7IaPhlrPQ984Q4pTva2l5H9a2lFvvqoxpGqugqVd+03Hsz+8aU+UUmzRgiLMzEgbbD/B8cNwwaM4QYjY+l/jRDgonTRbJ+AKGIq5AZQpbgJx2JQqytBswB7wiIkkgIVRJ8qMvrRN+t521rugc1P37upu67bacLM95EmFXJIr4pF70iBPpEnahBEg7+SDfOa+rYJVtMprq5XL7lyQjbIqPxyyuqs=</latexit> + <latexit sha1_base64="167RAvS3RLREjAic849QQgutm18=">AAACZHicdVFdS8MwFM3q16xTN4dPghTHQF9GK6I+DnwRnya4D9jGSNNsC0ubktzK6uhf8FX/mn/A32G69cFueiFwOOfk5uZcN+RMgW1/FYyt7Z3dveK+eVA6PDouV046SkSS0DYRXMieixXlLKBtYMBpL5QU+y6nXXf2kOrdVyoVE8ELxCEd+ngSsDEjGFLq7XJ2NSrX7Ia9LGsTOBmooaxao0rhaeAJEvk0AMKxUn3HDmG4wBIY4TQxB5GiISYzPKGL5YiJVdeUZ42F1CcAa8nmfNhXKvZd7fQxTNW6lpL/aWlHlX9VgY9lLL3ErP+mw/kf3rQnCMHXaI6BzvVIObYfwfh+uGBBGAENyOpf44hbIKw0XstjkhLgsQaYSKYDscgUS0xAL8EcsIDwyKMLrU6kHj0xdfrOetaboHPdcG4b9vNNrWlneyiiM3SBLpGD7lATPaIWaiOCpugdfaDPwrdRMqrG6cpqFLI7VZQr4/wHofW71A==</latexit> z(k) <latexit sha1_base64="YNAv12h4flec+yrpclKzjwvAbuY=">AAACZnicdVFdSwJBFB23L9vKtIgeehkSwQhkN6J6FHqJngzyA1RkdnbUydmdZeZuaIv/odf6Z/2Dfkaz6kNqXRg4nHPmzp1zvUhwDY7zlbE2Nre2d7K79t7+Qe4wXzhqaBkryupUCqlaHtFM8JDVgYNgrUgxEniCNb3Rfao3X5nSXIbPMIlYNyCDkPc5JWCoxlt5dOle9PJFp+LMCq8DdwGKaFG1XiHz2PEljQMWAhVE67brRNBNiAJOBZvanViziNARGbBkNuQUlwzl475U5oSAZ+ySjwRaTwLPOAMCQ72qpeR/WtpRL7+qISBqovypXfpNR+M/vGlPkFKs0IIAG5uRlth2DP27bsLDKAYW0vm/+rHAIHEaMPa5YhTExABCFTeBYDokilAwa7A7PKQi9lli1IEyo09tk767mvU6aFxV3JuK83RdrDqLPWTRGTpHZeSiW1RFD6iG6oiiF/SOPtBn5tvKWSfW6dxqZRZ3jtFSWfgHplm8RA==</latexit> z(k+ 1) <latexit sha1_base64="6r4FrMs/VLvzKs+fzcidCaMnE08=">AAACYXicdVHLagIxFI3Tl05fapduhopQKMhMKW2XQjelK4X6ABXJZK4ampkMyZ2iiF/QbftxXfdHmtFZVG0vBA7nnNzcnOvHgmt03a+ctbd/cHiUL9jHJ6dn58VSuaNlohi0mRRS9XyqQfAI2shRQC9WQENfQNd/fUz17hsozWX0gvMYhiGdRHzMGUVDta5Hxapbd1fl7AIvA1WSVXNUyj0PAsmSECJkgmrd99wYhwuqkDMBS3uQaIgpe6UTWKzmWzo1QwXOWCpzInRW7IaPhlrPQ984Q4pTva2l5H9a2lFvvqoxpGqugqVd+03Hsz+8aU+UUmzRgiLMzEgbbD/B8cNwwaM4QYjY+l/jRDgonTRbJ+AKGIq5AZQpbgJx2JQqytBswB7wiIkkgIVRJ8qMvrRN+t521rugc1P37upu67bacLM95EmFXJIr4pF70iBPpEnahBEg7+SDfOa+rYJVtMprq5XL7lyQjbIqPxyyuqs=</latexit> + <latexit sha1_base64="hB0AW9Ea6w1dPotedBwo0RLiBJA=">AAACYXicdVHLTgIxFC3jC8YX4JLNRELiiswYoy4xbowrSOSRACGdzgUaO9NJe8dACF/gVj/OtT9iB2YhoDdpcnLO6e3tuX4suEbX/cpZe/sHh0f5gn18cnp2XiyVO1omikGbSSFVz6caBI+gjRwF9GIFNPQFdP3Xx1TvvoHSXEYvOI9hGNJJxMecUTRU62FUrLp1d1XOLvAyUCVZNUel3PMgkCwJIUImqNZ9z41xuKAKOROwtAeJhpiyVzqBxWq+pVMzVOCMpTInQmfFbvhoqPU89I0zpDjV21pK/qelHfXmqxpDquYqWNq133Q8+8Ob9kQpxRYtKMLMjLTB9hMc3w8XPIoThIit/zVOhIPSSbN1Aq6AoZgbQJniJhCHTamiDM0G7AGPmEgCWBh1oszoS9uk721nvQs613Xvtu62bqoNN9tDnlTIJbkiHrkjDfJEmqRNGAHyTj7IZ+7bKlhFq7y2WrnszgXZKKvyA0hwusE=</latexit> A <latexit sha1_base64="fCD/22rBc12QpBD01hctQHS+wgM=">AAACYXicdVHLagIxFI3Tl05fapduhorQlcyU0nYpdFO6KAr1ASqSyVw1NDMZkjtFEb+g2/bjuu6PNKOzqNpeCBzOObm5OdePBdfoul85a2//4PAoX7CPT07PzoulckfLRDFoMymk6vlUg+ARtJGjgF6sgIa+gK7/+pDq3TdQmsvoBecxDEM6ifiYM4qGaj2PilW37q7K2QVeBqokq+aolHsaBJIlIUTIBNW677kxDhdUIWcClvYg0RBT9konsFjNt3RqhgqcsVTmROis2A0fDbWeh75xhhSneltLyf+0tKPefFVjSNVcBUu79puOZ394054opdiiBUWYmZE22H6C4/vhgkdxghCx9b/GiXBQOmm2TsAVMBRzAyhT3ATisClVlKHZgD3gERNJAAujTpQZfWmb9L3trHdB57ru3dbd1k214WZ7yJMKuSRXxCN3pEEeSZO0CSNA3skH+cx9WwWraJXXViuX3bkgG2VVfgBiSbrO</latexit> N <latexit sha1_base64="fCD/22rBc12QpBD01hctQHS+wgM=">AAACYXicdVHLagIxFI3Tl05fapduhorQlcyU0nYpdFO6KAr1ASqSyVw1NDMZkjtFEb+g2/bjuu6PNKOzqNpeCBzOObm5OdePBdfoul85a2//4PAoX7CPT07PzoulckfLRDFoMymk6vlUg+ARtJGjgF6sgIa+gK7/+pDq3TdQmsvoBecxDEM6ifiYM4qGaj2PilW37q7K2QVeBqokq+aolHsaBJIlIUTIBNW677kxDhdUIWcClvYg0RBT9konsFjNt3RqhgqcsVTmROis2A0fDbWeh75xhhSneltLyf+0tKPefFVjSNVcBUu79puOZ394054opdiiBUWYmZE22H6C4/vhgkdxghCx9b/GiXBQOmm2TsAVMBRzAyhT3ATisClVlKHZgD3gERNJAAujTpQZfWmb9L3trHdB57ru3dbd1k214WZ7yJMKuSRXxCN3pEEeSZO0CSNA3skH+cx9WwWraJXXViuX3bkgG2VVfgBiSbrO</latexit> N <latexit sha1_base64="RBEg9fm4uSkkVwYvrc0iMC3QspA=">AAACY3icdVHLSsNAFJ3GV019tNWdCMFScFUSEXVZcCOu6qMPaEuZTCbp0EkmzNxIS8knuNVv8wP8DydtF6bVCwOHc87cuXOuG3OmwLa/CsbW9s7uXnHfLB0cHh2XK9WOEokktE0EF7LnYkU5i2gbGHDaiyXFoctp153cZ3r3jUrFRPQKs5gOQxxEzGcEg6ZenkeTUblmN+xFWZvAWYEaWlVrVCk8DjxBkpBGQDhWqu/YMQznWAIjnKbmIFE0xmSCAzpfTJhadU15li+kPhFYCzbnw6FSs9DVzhDDWK1rGfmflnVU+VcVhFjOpJea9d90P3Dm/UEIfgazTHQqR4px/YT8O+GcxbFCdCILP/lJ9wCYWXpWh6TlACfaYCJZDoQi4yxxAT0DswBiwhPPDrXaiD16Kmp03fWs94EnauGc9Own65rTXu1hyI6QxfoEjnoFjXRA2qhNiIoQO/oA30Wvo2SUTVOl1ajsLpzgnJlnP8AR7y7sA==</latexit> R k FIGURE S1 Diagrammatic representation of a transformer block architecture described by (S14) and (S15). 6 IEEE CONTROL SYSTEMS » OCTOBER 2026 WHAT ARE RECURRENT NEURAL NETWORKS AND SELECTIVE STATE-SPACE MODELS? Recurrent Neural Networks An early architecture for sequence-to-sequence modeling is the recurrent neural network (RNN) [12]. All the N architectures so far described were formulated as composition of maps that update the state from one layer to the next. These maps were memoryless whereas RNNs employ memoryfull maps defined by a nonlinear control system. The memory of such maps is denoted by cpk , iq P R d with k denoting the dependence on the layer and i denoting the dependence on the position (in the input sequence) of the token being processed. As illustrated in Figure S1, a RNN can be can interpreted as a two-dimensional architecture in which information moves horizontally, from layer 1 to layerℓ, and also vertically, from position 1 to position m. The following update equations explain how the memory and state are updated cpk , iq “ΣpApkqcpk , i ́ 1q` Bpkqzpk , iqq , zpk ` 1, iq “ΣpCpkqcpk , iqq , cpk , 0q “ 0, (S16) for matrices Apkq P R dˆd , Bpkq P R dˆn and Cpkq P R nˆd . In practice, RNNs can be difficult to train due to the well known vanishing/exploding gradient problem. Some variants such as Long Short-Term Memory (LSTM) networks [24] can mitigate this problem, but the training process remains inherently se- quential in i thus limiting parallelization. A good introduction to this class of models can be found in [16, Chapter 10]. Selective State-Space Models More recently, selective state-space models (SSMs) have emerged as an alternative approach to transformers [25], [26]. Among all the different models, which are explained in depth in [27], the most prominent one is the Mamba model [28] whose defining feature is the use of input dependent weight matrices. <latexit sha1_base64="stNnzLsojexjKcKwFF51ziK7788=">AAACl3icdVHbSsMwGM7qadbTplfiTXEMNhijFU834kCQ4ZWC2wQ7RpqmWzBtSvJXdmBP4tN4q0/g25jOIm7qDyEf3/flz3/wYs4U2PZHzlhaXlldy6+bG5tb2zuF4m5biUQS2iKCC/ngYUU5i2gLGHD6EEuKQ4/Tjvd0leqdZyoVE9E9jGLaDXE/YgEjGDTVK5yQilNzqheBvlJoV2vjGVN1XXNcOUq1fqY531qvULLr9iys38DJQAllcdsr5m5cX5AkpBEQjpV6dOwYuhMsgRFOp6abKBpj8oT7dDLramqVNeVbgZD6RGDN2DkfDpUahZ52hhgGalFLyf+0NKOa/1VBiOVI+lOz/JOOh39405wgBF+gOQY61CXNsY8JBOfdCYviBGhEvvoKEm6BsNKNWD6TlAAfaYCJZHogFhlgiQnovZkuiwhPfDrRal/q0qemnr6zOOvfoH1Ud07r9t1xqWFne8ijA3SIKshBZ6iBmugWtRBBL+gVvaF3Y9+4NK6N5pfVyGVv9tBcGHef+VzHSA==</latexit> c(1,1) =f(1,c(1,0),z(1,1)) z(2,1) =g(1,c(1,1),z(1,1)) <latexit sha1_base64="okVerGUg3ViIlXaiGVwHTXR/wz8=">AAACl3icdVHZSsNAFJ3Grcat1SfxJViEFqQkdX0RC4KITwp2AVPKZDKpg5NMmLmRLvRL/Bpf9Qv8Gyc1iG31wjCHc87cuYsXc6bAtj9zxsLi0vJKftVcW9/Y3CoUt5tKJJLQBhFcyLaHFeUsog1gwGk7lhSHHqct7/kq1VsvVComogcYxLQT4l7EAkYwaKpbOCHl2qFTuQj0lUK7cjicMBXXNYflo1TrZZrzo3ULJbtqT8KaB04GSiiLu24xd+v6giQhjYBwrNSjY8fQGWEJjHA6Nt1E0RiTZ9yjo0lXY+tAU74VCKlPBNaEnfLhUKlB6GlniOFJzWop+Z+WZlTTvyoIsRxIf2we/Kbj/h/eNCcIwWdojoH2dUlT7GMCwXlnxKI4ARqR776ChFsgrHQjls8kJcAHGmAimR6IRZ6wxAT03kyXRYQnPh1ptSd16WNTT9+ZnfU8aNaqzmnVvj8u1e1sD3m0h/ZRGTnoDNXRDbpDDUTQK3pD7+jD2DUujWvj5ttq5LI3O2gqjPsvCivHUA==</latexit> c(2,1) =f(2,c(2,0),z(2,1)) z(3,1) =g(2,c(2,1),z(2,1)) <latexit sha1_base64="pqUcq8TW6cd+mYmTIQZBKKA7Jb4=">AAACl3icdVFdS8MwFM3qd/3a9El8KY7BBmO0w68XcSDI8GmC08E6RpqmM5g2JbmVzbFf4q/xVX+B/8Z0K+KmXgg5nHNycz+8mDMFtv2ZM5aWV1bX1jfMza3tnd18Ye9eiUQS2iaCC9nxsKKcRbQNDDjtxJLi0OP0wXu6SvWHZyoVE9EdjGLaC/EgYgEjGDTVz5+QslOtVy4CfaXQqVRfpkzFdc2Xcj3VBplW/9b6+aJds6dh/QZOBoooi1a/kLtxfUGSkEZAOFaq69gx9MZYAiOcTkw3UTTG5AkP6Hja1cQqacq3AiH1icCasnM+HCo1Cj3tDDE8qkUtJf/T0oxq/lcFIZYj6U/M0k86Hv7hTXOCEHyB5hjoUJc0x3YTCM57YxbFCdCIzPoKEm6BsNKNWD6TlAAfaYCJZHogFnnEEhPQezNdFhGe+HSs1YHUpU9MPX1ncda/wX295pzW7NvjYsPO9rCODtERKiMHnaEGaqIWaiOCXtEbekcfxoFxaVwbzZnVyGVv9tFcGLdfBevHTg==</latexit> c(1,2) =f(1,c(1,1),z(1,2)) z(2,2) =g(1,c(1,2),z(1,2)) <latexit sha1_base64="vG2jRdyvrZuWchjAuL3C9p92y4w=">AAACl3icdVFdS8MwFM3qd/3a9El8KY7BBjLa+fkiCoIMnzZwKqxjpGm6haVNSW7FOfZL/DW+6i/w35huRdzUCyGHc05u7ocXc6bAtj9zxsLi0vLK6pq5vrG5tZ0v7NwrkUhCW0RwIR89rChnEW0BA04fY0lx6H64A2uU/3hiUrFRHQHw5h2QtyLWMAIBk118yekXDusVS4CfaXQqRy+TJiK65ov5aNU62Va7Vvr5ot21Z6E9Rs4GSiiLBrdQu7W9QVJQhoB4ViptmPH0BlhCYxwOjbdRNEYkwHu0dGkq7FV0pRvBULqE4E1YWd8OFRqGHraGWLoq3ktJf/T0oxq9lcFIZZD6Y/N0k86fv7Dm+YEIfgczTHQZ13SDNtOIDjvjFgUJ0AjMu0rSLgFwko3YvlMUgJ8qAEmkumBWKSPJSag92a6LCI88elIqz2pSx+bevrO/Kx/g/ta1Tmt2s3j4pWd7WEV7aMDVEYOOkNXqI4aqIUIekVv6B19GHvGpXFj1KdWI5e92UUzYTS/ABarx1Y=</latexit> c(2,2) =f(2,c(2,1),z(2,2)) z(3,2) =g(2,c(2,2),z(2,2)) <latexit sha1_base64="YaYnxS5ZzrKMhuFPkiFx6s+Euwk=">AAACZnicdVFdSwJBFB23L9vKtIgeehkSwSBkN6J6FHqJngzyA1RkdnbUydmdZeZuaIv/odf6Z/2Dfkaz6kNqXRg4nHPmzp1zvUhwDY7zlbE2Nre2d7K79t7+Qe4wXzhqaBkryupUCqlaHtFM8JDVgYNgrUgxEniCNb3Rfao3X5nSXIbPMIlYNyCDkPc5JWCoxlvZvXQvevmiU3FmhdeBuwBFtKhar5B57PiSxgELgQqiddt1IugmRAGngk3tTqxZROiIDFgyG3KKS4bycV8qc0LAM3bJRwKtJ4FnnAGBoV7VUvI/Le2ol1/VEBA1Uf7ULv2mo/Ef3rQnSClWaEGAjc1IS2w7hv5dN+FhFAML6fxf/VhgkDgNGPtcMQpiYgChiptAMB0SRSiYNdgdHlIR+ywx6kCZ0ae2Sd9dzXodNK4q7k3FebouVp3FHrLoDJ2jMnLRLaqiB1RDdUTRC3pHH+gz823lrBPrdG61Mos7x2ipLPwDNFi8Cw==</latexit> z(1,1) <latexit sha1_base64="CN99S4QrtWKMeLUtFYik/4SlROA=">AAACZnicdVHLTgIxFC3jC8cXaIwLN42EBBNDZohRlyRujCtM5JEAIZ1OgUpnOmnvGHDCP7jVP/MP/Aw7wEJAb9Lk5JzT29tzvUhwDY7zlbE2Nre2d7K79t7+weFRLn/c0DJWlNWpFFK1PKKZ4CGrAwfBWpFiJPAEa3qj+1RvvjKluQyfYRKxbkAGIe9zSsBQjbeSe1W57OUKTtmZFV4H7gIU0KJqvXzmseNLGgcsBCqI1m3XiaCbEAWcCja1O7FmEaEjMmDJbMgpLhrKx32pzAkBz9glHwm0ngSecQYEhnpVS8n/tLSjXn5VQ0DURPlTu/ibjsZ/eNOeIKVYoQUBNjYjLbHtGPp33YSHUQwspPN/9WOBQeI0YOxzxSiIiQGEKm4CwXRIFKFg1mB3eEhF7LPEqANlRp/aJn13Net10KiU3Zuy83RdqDqLPWTRObpAJeSiW1RFD6iG6oiiF/SOPtBn5ts6tE6ts7nVyizunKClsvAPNla8DA==</latexit> z(1,2) <latexit sha1_base64="sRxgTmK1mae/kgLq5ArpejzI2I4=">AAACZnicdVHLTgIxFC3jC8cXaIwLN42EBBNDZohRlyRujCtM5JEAIZ1OgUpnOmnvGHDCP7jVP/MP/Aw7wEJAb9Lk5JzT29tzvUhwDY7zlbE2Nre2d7K79t7+weFRLn/c0DJWlNWpFFK1PKKZ4CGrAwfBWpFiJPAEa3qj+1RvvjKluQyfYRKxbkAGIe9zSsBQjbdS5cq97OUKTtmZFV4H7gIU0KJqvXzmseNLGgcsBCqI1m3XiaCbEAWcCja1O7FmEaEjMmDJbMgpLhrKx32pzAkBz9glHwm0ngSecQYEhnpVS8n/tLSjXn5VQ0DURPlTu/ibjsZ/eNOeIKVYoQUBNjYjLbHtGPp33YSHUQwspPN/9WOBQeI0YOxzxSiIiQGEKm4CwXRIFKFg1mB3eEhF7LPEqANlRp/aJn13Net10KiU3Zuy83RdqDqLPWTRObpAJeSiW1RFD6iG6oiiF/SOPtBn5ts6tE6ts7nVyizunKClsvAPNli8DA==</latexit> z(2,1) <latexit sha1_base64="JeM3ZidIwHcqL2S0s4WVo2jbTz4=">AAACZnicdVHLTgIxFC3jC8cXaIwLN42EBBNDZohRlyRujCtM5JEAIZ1OgUpnOmnvGHDCP7jVP/MP/Aw7wEJAb9Lk5JzT29tzvUhwDY7zlbE2Nre2d7K79t7+weFRLn/c0DJWlNWpFFK1PKKZ4CGrAwfBWpFiJPAEa3qj+1RvvjKluQyfYRKxbkAGIe9zSsBQjbdS5apy2csVnLIzK7wO3AUooEXVevnMY8eXNA5YCFQQrduuE0E3IQo4FWxqd2LNIkJHZMCS2ZBTXDSUj/tSmRMCnrFLPhJoPQk84wwIDPWqlpL/aWlHvfyqhoCoifKndvE3HY3/8KY9QUqxQgsCbGxGWmLbMfTvugkPoxhYSOf/6scCg8RpwNjnilEQEwMIVdwEgumQKELBrMHu8JCK2GeJUQfKjD61TfruatbroFEpuzdl5+m6UHUWe8iic3SBSshFt6iKHlAN1RFFL+gdfaDPzLd1aJ1aZ3OrlVncOUFLZeEfOFa8DQ==</latexit> z(2,2) <latexit sha1_base64="y2A5YQzkGLDPpHIXw8dFZ/AGF5Q=">AAACZnicdVHLTgIxFC3jC0dF0BgXbhoJCSaGzKhRlyRujCtM5JEAIZ1OwUpnOmnvGHDCP7jVP/MP/Aw7wEJAb9Lk5JzT29tzvUhwDY7zlbHW1jc2t7Lb9s7uXm4/XzhoaBkryupUCqlaHtFM8JDVgYNgrUgxEniCNb3hXao3X5nSXIZPMI5YNyCDkPc5JWCoxlv58tw96+WLTsWZFl4F7hwU0bxqvULmoeNLGgcsBCqI1m3XiaCbEAWcCjaxO7FmEaFDMmDJdMgJLhnKx32pzAkBT9kFHwm0HgeecQYEnvWylpL/aWlHvfiqhoCosfInduk3HY3+8KY9QUqxRAsCbGRGWmDbMfRvuwkPoxhYSGf/6scCg8RpwNjnilEQYwMIVdwEgukzUYSCWYPd4SEVsc8Sow6UGX1im/Td5axXQeOi4l5XnMerYtWZ7yGLTtApKiMX3aAqukc1VEcUvaB39IE+M99WzjqyjmdWKzO/c4gWysI/OFi8DQ==</latexit> z(3,1) <latexit sha1_base64="pwvoB08HolRhhmsux8UTywyaluE=">AAACZnicdVHLTgIxFC3jC0dF0BgXbhoJCSaGzKBRlyRujCtM5JEAIZ1OgUpnOmnvGJDwD271z/wDP8MOsBDQmzQ5Oef09vZcLxJcg+N8payNza3tnfSuvbd/kDnM5o7qWsaKshqVQqqmRzQTPGQ14CBYM1KMBJ5gDW94n+iNV6Y0l+EzjCPWCUg/5D1OCRiq/la8uixfdLN5p+TMCq8DdwHyaFHVbi712PYljQMWAhVE65brRNCZEAWcCja127FmEaFD0meT2ZBTXDCUj3tSmRMCnrFLPhJoPQ484wwIDPSqlpD/aUlHvfyqhoCosfKnduE3HY3+8CY9QUqxQgsCbGRGWmJbMfTuOhMeRjGwkM7/1YsFBomTgLHPFaMgxgYQqrgJBNMBUYSCWYPd5iEVsc8mRu0rM/rUNum7q1mvg3q55N6UnKfrfMVZ7CGNztA5KiIX3aIKekBVVEMUvaB39IE+U99WxjqxTudWK7W4c4yWysI/Ola8Dg==</latexit> z(3,2) <latexit sha1_base64="x1KIirP/s3bm1Vny27eV1d0yCeA=">AAACaHicdVHLSgMxFM2Mrzq+Wl2IuBlaChWkZETUjVBwI64q2Ae0pWQyaQ3NTIbkjnQo/Qm3+mP+gl9hpu3CtnohcDjn5ObmXD8WXAPGX5a9sbm1vZPbdfb2Dw6P8oXjppaJoqxBpZCq7RPNBI9YAzgI1o4VI6EvWMsfPWR6640pzWX0AmnMeiEZRnzAKQFDtWnFu8QX97ifL+EqnpW7DrwFKKFF1fsF66kbSJqELAIqiNYdD8fQmxAFnAo2dbqJZjGhIzJkk9mYU7dsqMAdSGVOBO6MXfKRUOs09I0zJPCqV7WM/E/LOurlVzWERKUqmDrl33Q8/sOb9QQpxQotCLCxGWmJ7SQwuOtNeBQnwCI6/9cgES5IN4vYDbhiFERqAKGKm0Bc+koUoWAW4XR5REUSsIlRh8qMPnVM+t5q1uugeVX1bqr4+bpUw4s95NA5KqIK8tAtqqFHVEcNRJFA7+gDfVrfdt4+tc/mVtta3DlBS2UXfwAqg7x0</latexit> c(1,0) = 0 <latexit sha1_base64="IvLxKWyQPs/UQ6Fs3KAcn7NAt74=">AAACZnicdVHLSgMxFE3HVx1frSIu3ARLoYKUGRF1WXAjrirYB7SlZDJpG5uZDMkdaSn9B7f6Z/6Bn2GmnYXT6oXA4ZyTm5tzvUhwDY7zlbM2Nre2d/K79t7+weFRoXjc1DJWlDWoFFK1PaKZ4CFrAAfB2pFiJPAEa3njh0RvvTGluQxfYBqxXkCGIR9wSsBQTVpxr9zLfqHkVJ1F4XXgpqCE0qr3i7mnri9pHLAQqCBad1wngt6MKOBUsLndjTWLCB2TIZsthpzjsqF8PJDKnBDwgs34SKD1NPCMMyAw0qtaQv6nJR119lUNAVFT5c/t8m86mvzhTXqClGKFFgTYxIyUYTsxDO57Mx5GMbCQLv81iAUGiZOAsc8VoyCmBhCquAkE0xFRhIJZg93lIRWxz2ZGHSoz+tw26burWa+D5nXVva06zzelmpPuIY/O0QWqIBfdoRp6RHXUQBS9onf0gT5z39ahdWqdLa1WLr1zgjJl4R8GKrv0</latexit> c(1,1) <latexit sha1_base64="5BXalDZhowTr3HXEhYidjkDaeSQ=">AAACZnicdVFdSwJBFB23L9vKtIgeehkSwSBkV6J6FHqJngzyA1RkdnbUydmdZeZuKLL/odf6Z/2Dfkaz6kNqXRg4nHPmzp1zvUhwDY7zlbG2tnd297L79sHhUe44XzhpahkryhpUCqnaHtFM8JA1gINg7UgxEniCtbzxQ6q33pjSXIYvMI1YLyDDkA84JWCoJi2719Wrfr7oVJx54U3gLkERLaveL2Seur6kccBCoIJo3XGdCHozooBTwRK7G2sWETomQzabD5ngkqF8PJDKnBDwnF3xkUDraeAZZ0BgpNe1lPxPSzvq1Vc1BERNlZ/Ypd90NPnDm/YEKcUaLQiwiRlphe3EMLjvzXgYxcBCuvjXIBYYJE4Dxj5XjIKYGkCo4iYQTEdEEQpmDXaXh1TEPpsZdajM6Ilt0nfXs94EzWrFva04zzfFmrPcQxZdoEtURi66QzX0iOqogSh6Re/oA31mvq2cdWadL6xWZnnnFK2UhX8ACCi79Q==</latexit> c(1,2) <latexit sha1_base64="jdjwnyx5VO1aT5B/+IutsV7nfGA=">AAACaHicdVFdSwJBFB23L9u+tB4ielkUwSBkVqJ6CYReoieD/AAVmZ0ddXB2Z5m5G4rsn+i1/lh/oV/RrPqQWhcGDuecuXPnXC8SXAPGXxlra3tndy+7bx8cHh2f5PKnTS1jRVmDSiFV2yOaCR6yBnAQrB0pRgJPsJY3fkz11htTmsvwFaYR6wVkGPIBpwQM1abl6jW+esD9XBFX8LycTeAuQREtq97PZ567vqRxwEKggmjdcXEEvRlRwKlgid2NNYsIHZMhm83HTJySoXxnIJU5IThzdsVHAq2ngWecAYGRXtdS8j8t7ahXX9UQEDVVfmKXftPR5A9v2hOkFGu0IMAmZqQVthPD4L4342EUAwvp4l+DWDggnTRix+eKURBTAwhV3ATi0BFRhIJZhN3lIRWxz2ZGHSozemKb9N31rDdBs1pxbyv45aZYw8s9ZNElKqAyctEdqqEnVEcNRJFA7+gDfWa+rZx1bl0srFZmeecMrZRV+AEshbx1</latexit> c(2,0)=0 <latexit sha1_base64="cUtq6n7oVyxVHlUXryyOH3IoB8c=">AAACZnicdVFdSwJBFB23L9vKtIgeehkSwSBkV6J6FHqJngzyA1RkdnbUydmdZeZuKLL/odf6Z/2Dfkaz6kNqXRg4nHPmzp1zvUhwDY7zlbG2tnd297L79sHhUe44XzhpahkryhpUCqnaHtFM8JA1gINg7UgxEniCtbzxQ6q33pjSXIYvMI1YLyDDkA84JWCoJi1Xr92rfr7oVJx54U3gLkERLaveL2Seur6kccBCoIJo3XGdCHozooBTwRK7G2sWETomQzabD5ngkqF8PJDKnBDwnF3xkUDraeAZZ0BgpNe1lPxPSzvq1Vc1BERNlZ/Ypd90NPnDm/YEKcUaLQiwiRlphe3EMLjvzXgYxcBCuvjXIBYYJE4Dxj5XjIKYGkCo4iYQTEdEEQpmDXaXh1TEPpsZdajM6Ilt0nfXs94EzWrFva04zzfFmrPcQxZdoEtURi66QzX0iOqogSh6Re/oA31mvq2cdWadL6xWZnnnFK2UhX8ACCq79Q==</latexit> c(2,1) <latexit sha1_base64="uBNZpKXoEwFld3UiSfGk4PE2RWk=">AAACZnicdVFdSwJBFB23L9vKtIgeehkSwSBkV6J6FHqJngzyA1RkdnbUydmdZeZuKLL/odf6Z/2Dfkaz6kNqXRg4nHPmzp1zvUhwDY7zlbG2tnd297L79sHhUe44XzhpahkryhpUCqnaHtFM8JA1gINg7UgxEniCtbzxQ6q33pjSXIYvMI1YLyDDkA84JWCoJi1Xr6tX/XzRqTjzwpvAXYIiWla9X8g8dX1J44CFQAXRuuM6EfRmRAGngiV2N9YsInRMhmw2HzLBJUP5eCCVOSHgObviI4HW08AzzoDASK9rKfmflnbUq69qCIiaKj+xS7/paPKHN+0JUoo1WhBgEzPSCtuJYXDfm/EwioGFdPGvQSwwSJwGjH2uGAUxNYBQxU0gmI6IIhTMGuwuD6mIfTYz6lCZ0RPbpO+uZ70JmtWKe1txnm+KNWe5hyy6QJeojFx0h2roEdVRA1H0it7RB/rMfFs568w6X1itzPLOKVopC/8ACii79g==</latexit> c(2,2) FIGURE S1Graphical description of the information flow in an SSM. The update equations(S17) were simplifiedasfollowsfpk , cpk , i ́ 1q, zpk , iqq “ Apk , zpk , iqq cpk , i ́ 1q `Bpk , zpk , iqq zpk , iqand gpk , cpk , iq, zpk , iqq “ Cpk , zpk , iqq cpk , iq. Note that cp1,0q and cp2, 0q are only used for the purpose of initialization since position i “ 0 in the input string does not exist. Its update equations are cpk , iq “ Apk , zpk , iqq cpk , i ́ 1q` Bpk , zpk , iqq zpk , iq, zpk ` 1, iq “ Cpk , zpk , iqq cpk , iq, cpk , 0q “ 0, (S17) where A : t1, . . . ,ℓuˆ R n Ñ R dˆd , B : t1, . . . ,ℓuˆ R n Ñ R dˆn , and C : t1, . . . ,ℓuˆ R n Ñ R nˆd are nonlinear maps whose parameters are learned from data. The dependence on the layer state (which is the input for layer 1) allows the model to adapt the weights to the current input thereby changing how the state propagates. Consequently, more relevant inputs can have more impact on the state whereas less relevant input information can be discarded [28]. Empirically, selective SSMs such as the Mamba can have a comparable performance to transformers [28]. Moreover, under certain choices for the matrices Apk , zpk , iqq, Bpk , zpk , iqq, and Cpk , zpk , iqq the computational complexity of training these models is lower than transformers. Finally, it is possible to interleave SSM blocks with attention blocks to form hybrid architectures such as the Jamba model [29]. It is natural to question if the mismatch between these models decreases asℓ increases. In particular, to question if the weights obtained by minimizing (0.23) converge to the weights minimizing (0.22) asℓ Ñ 8. Note, however, that in this case the control input W for each ResNet does not come from the discretization of a continuous control input W c , which was the setting discussed in the previous section. Earlier solutions to this problem were investigated in the literature on numerical methods for optimal control problems, see for example [38], [39], [40]. A more detailed discussion of how optimal control relates to the learning of the weights W, appears in Optimal control and backpropaga- tion. For now, it suffices to observe that the minimization problems (0.22) and (0.23) can be interpreted as optimal control problems. The results in [40] establish that the weights W obtained by solving (0.23) and the evolution of the state of a ResNet when using such weights converge to the control input obtained by solving (0.22) and to the solution of (0.19) when using such control input. These convergence results are predicated on the assumption that the control input obtained by solving (0.22) is Lipschitz continuous (or on other assumptions implying Lipschitz continuity). The recent paper [41] revisits this problem using more recent mathematical tools and forgoes the Lip- schitz continuity assumption. Instead, it describes how to regularize the continuous (0.22) and the discrete loss (0.23) to ensure convergence. Intuitively, the regularizer penal- OCTOBER 2026 « IEEE CONTROL SYSTEMS 7 izes weights that change too fast across layers which can be interpreted as penalizing the discretization of the derivative of the continuous inputs. Another interesting point of departure in the paper [41] is that it does not rely on Pontryagin’s optimality conditions and, instead, it employs the notion ofΓ-convergence enabling the direct comparison between the minimizers of the regularized losses (0.22) and (0.23) asℓ Ñ 8. Novel architectures based on different discretization schemes The link between ResNets and the differential equa- tion (0.19) led several researchers to investigate if dis- cretization schemes beyond the forward Euler method could be used to produce new N architectures with better properties. Some of the new architectures produced by this process include PolyNets [42], based on a polynomial ap- proximation of the backward Euler method, a Hamiltonian inspired architecture [31], based on the semi-implicit Euler method that is known to be symplectic, the momentum ResNet [43], based on the Verlet integration scheme, and several others [32], [44], [45], [46] based on generalizations of the differential model (0.19) that include second order derivatives. Rather than starting with a differential equation and discretizing it to arrive at a N architecture, the differential equation can be directly taken as the architecture. This is the key idea behind Neural ODEs [33], in which the N directly parameterizes the right hand side of a differential equation. The evaluation of a Neural ODE on a given input consists in numerically solving the corresponding differential equation using the input as initial condition. The objective of training is to find the differential equation whose solutions best match the training data. Similarly to evaluation, the training of Neural ODEs requires the use of a numerical solver for differential equations. Neural ODEs are especially useful when working with data that is known to be produced by a differential equation but the use of numerical solvers makes them less scalable than existing alternatives. OPTIMAL CONTROL AND BACKPROPAGATION The classical connection The classical connection between control and backpropaga- tion is known since the ’80s [6], [47]. To elucidate it, recall the setting in “What are feedforward neural networks?” for a N withℓ P N layers, all of which have n neurons. The state at layer k ` 1 P t2, . . . ,ℓu is computed from the state at layer k using zpk` 1q “ΣpWpkqzpkqq(0.24) and a N can be seen as a functionΦ W : R n Ñ R n mapping zp1q toΦ W pzp1q “ zpℓq, where zpℓq is recursively computed using (0.24). NNs are trained, that is, their weights W are computed so thatΦ W approximates an unknown function for which samples are available. A col- lection of s samplesppx 1 , y 1 q,px 2 , y 2 q, . . . ,px s , y s q consists of pairs px i , y i q P R n ˆ R n where y i represents the evaluation of the unknown function at x i . Supervised training of a neural network entails learning the weights W so thatΦ W maps x i to y i . The learning objective can be formalized as the mini- mization of the loss functional s ÿ i“1 Lpy i ,Φ W px i q subject to the constraints described by the equalities (0.24) and where L : R n ˆ R n Ñ R ` 0 is a differentiable function measuring the discrepancy between y i andΦ W px i q, such as y i ́Φ W px i q 2 . Incorporating the constraints (0.24) into the loss leads to the expanded loss functional Jpλ, z, Wq “ s ÿ i“1 Lpy i ,Φpx i q ` s ÿ i“1 ℓ ÿ k“2 λ T i pkqrz i pkq ́ΣpWpk ́ 1qz i pk ́ 1qqs with λ i pkq P R n being Lagrange multipliers (adjoint vari- ables). This is now an unconstrained minimization prob- lem for which necessary conditions for optimality can be written. The first necessary condition for optimality arises by differentiating the cost with respect to the Lagrange mul- tipliers BJ Bλ i pkq “ 0 ùñz i pkq “ΣpWpk ́ 1qz i pk ́ 1q, and recovers (0.24) as expected. The second arises by differentiating the cost with respect to the state BL Bz i pkq “ 0.(0.25) In particular, at the final layer k “ℓ, it provides BJ Bz i pℓq “ BL Bz i pℓq ` λ i pℓq “ 0, defining the value of λ i at the last layer λ i pℓq “ ́ BL Bz i pℓq .(0.26) To obtain λ i at the intermediate layers, condition (0.25) is used for k “ 2, . . . ,ℓ ́ 1 BJ Bz i pkq “ λ i pkq ́ BΣ Bv Wpkqλ i pk` 1q “ 0, giving rise to the recursion λ i pkq “ BΣ Bv Wpkqλ i pk` 1q.(0.27) Note that it is possible that λ i pk ` 1q is in the kernel of Wpkq, leading to vanishing of the adjoint along the N. In what follows, as it is commonly done in the literature, the weight matrix is assumed not to be rank deficient. 8 IEEE CONTROL SYSTEMS » OCTOBER 2026 To compute the last necessary condition, note that the derivative ofΣpvq with respect to v is the diagonal matrix whose entry in row and column j is dσ dv j , j P t1, . . . , nu. Next, define v i pk` 1q def “ Wpkqz i pkq(0.28) for i P t1, . . . , su and note that the j-th component of v i pk` 1q is v i,j pk` 1q “ n ÿ m“1 Wpkq jm z i,m pkq. Consequently Bv i,j pk` 1q BWpkq ab “ δ ja z i,b pkq,(0.29) a, b P t1, . . . , nu, and where δ ja is Kronecker’s delta. Using this equality and the definition of J , the derivative of J with respect to the entry in row a and column b of W can be computed BJ BWpkq ab “ B BWpkq ab ́ s ÿ i“1 ́λ i pk` 1q T Σpv i pk` 1q ̄ “ ́ s ÿ i“1 n ÿ j“1 λ i,j pk` 1q dσ dv i,j pk` 1q Bv i,j pk` 1q BWpkq ab “ ́ s ÿ i“1 λ i,a pk` 1q dσ dv i,a pk` 1q z i,b pkq.(0.30) This expression can be written in the simpler form BJ BWpkq ab “ ́ s ÿ i“1 r i,a pk` 1qz i,b pkq, where r i pkq “ diag ˆ dσ dv i,1 pkq , . . . , dσ dv i,n pkq ̇ λ i pkq. From the derivative with respect to each entry of Wpkq, the derivative with respect to the weight matrix Wpkq can be obtained BJ BWpkq “ ́ s ÿ i“1 r i pk` 1qz i pkq J “ 0,(0.31) where k P t2, . . . ,ℓ ́ 1u. In principle, solving the station- arity conditions (0.31) jointly with the state (0.24) and adjoint (0.27) equations would yield stationary points. However, because z i pkq and λ i pkq, and hence r i pkq, de- pend nonlinearly and recursively on tWpk 1 qu k 1 ďk through the forward and backward passes, the resulting coupled nonlinear system is analytically intractable. Instead, it is customary to find the weights W that enforce (0.31) via gradient descent. This is done by viewing the loss functional as defining, for each fixed collection of weights W “ pWp1q, . . . , Wpℓ ́ 1q, a scalar functional JpWq def “ JpzpWq, λpWq, Wq, where zpWq and λpWq satisfy the forward (0.24) and backward (0.27) recursions. The gradient descent update for minimizing J then takes the form W pq`1q pkq “ W pqq pkq ́ η q BJ BWpkq ˇ ˇ ˇ W“W pqq “ W pqq pkq` η q s ÿ i“1 r pqq i pk` 1q z pqq i pkq T ,(0.32) where k P t1, . . . ,ℓ ́ 1u, the superscript q denotes the q- th iterate of the gradient update, r pqq i pkq and z pqq i pkq are obtained from the backward and forward passes using the current weights W pqq , and η q is the step size also known as the learning rate. Combining the necessary conditions for optimality with the gradient descent iterates results in the backpropagation algorithm that can be summarized as the iteration of the following steps: 1) Forward pass: propagate the input z i p1q to the output z i pℓq using (0.24). 2) Backward pass: initialize the adjoint λ i pℓq at layerℓ with (0.26) and backpropagate it to λ i p1q using (0.27). 3) Gradient descent: update the weights W using (0.32). The modern connection The connection between optimal control and backprop- agation presented in The classical connection consists in formulating the training of NNs as an optimal control problem in discrete-time, deriving necessary conditions for optimality, and observing that those conditions essentially define backpropagation. As discussed in Differential equa- tions, control systems, and neural networks, for certain N architectures, such as the ResNet, the state update equation can be regarded as the discretization of a differential equation. Similarly, the optimal control problem addressed in The classical connection can be seen as the discretization of a continuous-time optimal control problem. This obser- vation exposes an alternative path towards the training of NNs: formulate necessary conditions for optimality using an optimal control formulation in continuous-time and then discretize them to obtain a training algorithm. It is shown in this section that both approaches lead to the same solution, provided that suitable discretization techniques are employed. Redoing the analysis in The classical connection for a ResNet requires changing the expanded loss functional Jpz, W, λq to s ÿ i“1 Lpy i ,Φpx i q` s ÿ i“1 ℓ ÿ k“1 λ T i pkqrz i pkq ́ z i pk ́ 1q ́wpk ́ 1qΣpWpk ́ 1qz i pk ́ 1qqs in order to account for how the state is updated across layers in a ResNet, see (0.18). The corresponding necessary conditions for optimality become OCTOBER 2026 « IEEE CONTROL SYSTEMS 9 z i pkq “ z i pk ́ 1q` wpk ́ 1qΣpWpk ́ 1qz i pk ́ 1q λ i pkq “ λ i pk` 1q` wpkq BΣ Bv Wpkqλ i pk` 1q(0.33) 0 “ ́ s ÿ i“1 r i pk` 1qz i pkq J .(0.34) The very same conditions can be obtained starting from the continuous-time optimal control problem defined by the minimization of s ÿ i“1 Lpy i , z c,i p1q,(0.35) where z c,i p1q denotes the solution of (0.19) at time 1 when starting from initial condition x i . Since multiple solutions are used in the loss (0.35) an ensemble control system consisting of s identical copies of (0.19) running in parallel and controlled by the same input W c is considered. The necessary conditions for optimality are given by Pontrya- gin’s maximum principle [48], [45] 9 z c,i ptq “ BH Bλ c,i “ΣpW c ptqz c,i ptqq(0.36) 9 λ c,i ptq “ ́ BH Bz c,i “ ́ BΣ Bv W c ptqλ c,i ptq(0.37) 0 “ BH BW c , where the Hamiltonian H is Hpλ c,i ptq, z c,i ptq, W c ptqq “ s ÿ i“1 λ T c,i ptqΣpW c ptqz c,i ptqq. A naive discretization of (0.37), using the forward Euler method, does not lead to (0.33). In order to recover (0.33), the symplectic version of the forward Euler method, also known as the semi-implicit Euler method [49, Eq. (1.9)], must be employed. It consists of the forward Euler method applied to (0.36) but requires discretizing (0.37) using the implicit rule λ i pk` 1q “ λ i pkq ́ wpkq BΣ Bv Wpkqλ i pk` 1q, which is simply (0.33). Although the interplay between backpropagation and optimal control in discrete-time already allows for the migration of ideas from control to deep learning, the linkage with continuous-time optimal control opens new perspectives. Continuous-time control theory is substan- tially more developed than its discrete-time counterpart. Moreover, continuous-time results are often more intuitive thus making it easier to adapt/generalize to the setting of deep learning. Some examples of this migration of ideas are discussed in the next section. New training techniques The previous sections shed light into the interplay between optimal control and the training of NNs. However, the use of gradient descent is but one possible path to find the weights enforcing equality (0.34). When the weights are real valued, [45] discusses why gradient descent may be preferable to directly solving (0.34). The work in [50] uses the extra generality afforded by the maximum principle to derive training algorithms for NNs whose weights belong to finite sets, in which case gradient descent is not an option. Extending the optimal control perspective from one to two players brings us to the setting of games which proved useful to develop more efficient adversarial train- ing algorithms in [51], see also [52] for a rigorous conver- gence analysis. The modern continuous-time perspective was exploited in [48] to obtain training algorithms where the discretization time step is regarded as one additional control input used for more efficient training. Rather than linkages to optimal control, other advances in training techniques leveraged the view of NNs as dis- cretizations of differential equations. The important prop- erty of reversibility, addressed in [46], is intimately tied to training, since backpropagation requires both forward and backward passes. In particular, it shows how reversibility can be exploited for more efficient training and proposes new reversible architectures. Another important property is stability [31], [46], understood as Lipschitz continuity of the map sending the state of the first layer to the state of the last layer. Stability guarantees robustness of the network against noisy inputs or perturbations of the data. As discussed in [53], the Lipschitz constant depends on the activation function σ and the weights for the continuous-time model (0.19). For the discrete-time model (0.18), it also depends on the number of layers. The work in [53] discusses several architectures inspired by ideas from dynamical systems, such as Hamiltonian dynamics, that ensure stability by construction rather than relying on training. UNIVERSAL APPROXIMATION One of the reasons underlying the wide applicability of NNs is their perceived ability to approximate arbitrary functions. This is known as the power of universal approx- imation. To make this notion precise, recall that a N is a family of functionsΦ W : R n 1 Ñ R o parameterized by the weights W, see “What are feedforward neural networks?” sidebar. It can then be asked if, given a function ψ : K Ñ R o defined on a domain of interest K Ď R n 1 , there is a choice of weights W so thatΦ W approximates ψ arbitrarily well with respect to a chosen topology (uniform, L p , Sobolev, etc.). A concise historical account of universal approximation in neural networks Classical universal approximation results established in late ’80s and early ’90s focused on proving that feed- forward NNs with small depth, that is, a small number of layers, but unrestricted width, that is, an arbitrary 10 IEEE CONTROL SYSTEMS » OCTOBER 2026 number of neurons per layer, can approximate continu- ous functions on compact sets, under mild assumptions on the activation functions. In particular, universality of sigmoidal activation functions was proved in [54], [55]. In a series of follow-up work, including [56], [57], it was shown that for broad classes of NNs, non-polynomial activations are essentially necessary and sufficient for approximating continuous functions on compact sets. The mathematical machinery behind these results is functional analysis’ den- sity arguments. A different argument was employed in [58] to establish that the approximation error decays at the rate 1n, where n is the number of sigmoids in the hidden layer, independently of the dimension of the approximated function’s domain. The modern deep learning era revived these questions due to the evolution in N architectures. Deep NNs have large depth, that is, a large number of layers, but bounded width, that is, a bounded number of neurons per layer. This mismatch between architectures prevents a direct application of the classical universal approximation re- sults. Establishing universal approximation for deep neu- ral networks relies on understanding composition of maps, which, due to nonlinearities of the activation functions, is challenging. The literature on this subject is quite recent. One of the early results, [59], focused on the class of ReLU activation functions to establish that one neuron per layer suffices to approximate an arbitrary Lebesgue integrable function from R to R with respect to the L 1 norm. The results that followed addressed more general functions from R n 1 to R o . For instance, when using the ReLU activation function and o “ 1, it was shown in [59] that universal approximation with respect to the L 1 topology can be achieved with a minimum width between n 1 ` 1 and n 1 ` 4. The theory of universal approximation for deep NNs substantially advanced in early 2020, with notable discoveries in [60], and particularly in [61], where universal approximation was established with respect to the infinity norm for non- residual deep NNs, allowing for general classes of acti- vation functions. More precisely, it is shown in [61] that under very mild conditions on the activation functions, any continuous function ψ : K Ñ R o , where K Ă R n 1 is compact, can be approximated in the infinity norm using a deep N of width n 1 ` o` 2. The control theory vantage point Inline with the objectives of this article, this section pro- vides the control-theoretic viewpoint on universal approx- imation developed in [62], [63], [64], and also in [65]. The starting point is the continuous limit of a ResNet, as explained in Differential equations, control systems, and neural networks, and written as 9 z c ptq “ w c ptqΣpW c ptqz c ptqq.(0.38) Note that (0.38) differs from (0.19) by the presence of w c ptq. This is achieved by writing the weight wpkq in (0.18) as the product wpkq “ τ w c pkq and letting τ Ñ 0 asℓ Ñ 8 so that w c ptq is the continuous limit of w c pkq. For notational convenience, denote by γ : R n Ñ R n the flow of (0.38), that is, the map sending an initial condition z c p0q to the point reached by the solution of (0.38) at time t “ 1 under the input pw c , W c q. The objective of approximating ψ : K Ñ R o with γ requires the existence of control inputs pw c , W c q for which the corresponding flow γ satisfies γpx i q “ ψpx i q for every sample point x i P K Ď R n 1 or, at least, that γpx i q is close to ψpx i q. This can be understood as a controllability problem for which the initial conditions x 1 , . . . , x s have to be steered to y 1 “ ψpx 1 q, . . . , y s “ ψpx s q at time t “ 1 by the same open-loop control input pw c , W c q. At this point, it is worth pointing out the sharp differ- ence between this setting and earlier work on controlla- bility of differential equation models for neural networks, for example, [66], where the weights were assumed to be constant and an exogenous control signal was fed into the neurons. The simultaneous control of multiple initial conditions is naturally related to the classical settings of ensemble control theory, for example, [67], [68]. However, in ensem- ble control, the motion of each member of the ensemble is governed by a different, even if very similar, control system. In the setting of universal approximation, the motion from each initial condition is governed by the same control system under the same input. This setting is most closely related to [69], where controllability results for ensembles of infinitely many copies of the same control system are considered. With this perspective in mind, there are two key problems that can be considered. Memorization Problem: Let ψ : R n 1 Ñ R o be a continu- ous function, K samples Ă R n 1 be a finite set, and ε P R ` be the approximation accuracy. Given any desired ε and p P r1,`8s, does there exist an initialization map f : R n 1 Ñ R n , a finalization map h : R n Ñ R o , and an input pw c , W c q : r0, 1s Ñ Rˆ R nˆn so that the flow γ : R n Ñ R n defined by the solution of (0.18) under the said input satisfies › › ψ ́ h ̋ γ ̋ f › › L p pK samples q ď ε ? The notation ̈ L p pK samples q denotes the L p norm on the set K samples , in other words, the approximation guarantee only needs to hold for elements in the finite set K samples . The purpose of the initialization function f is to map R n 1 to R n since the continuous model (0.38) was developed as- suming all the layers have the same number n of neurons. Similarly, the finalization map converts R n to R o . We note that the memorization problem above is a simultaneous controllability problem which is difficult to solve since the same differential equation (0.38) governs the evolution of the state from each initial condition z c p0q and OCTOBER 2026 « IEEE CONTROL SYSTEMS 11 under the same open-loop control input. The next problem considers the more challenging case of universal approximation on compact sets. Universal Approximation Problem: Let ψ : R n 1 Ñ R o be a continuous function, K Ă R n 1 be a compact set, and ε P R ` be the approximation accuracy. Given any desired ε and p P r1,`8s, does there exist a continuous initialization map f : R n 1 Ñ R n , a continuous finalization map g : R n Ñ R o , and an input pw c , W c q : r0, 1s Ñ Rˆ R nˆn such that the flow γ : R n Ñ R n defined by the solution of (0.38) under the said input satisfies › › ψ ́ h ̋ γ ̋ f › › L p pKq ď ε ? ψ(K) (w c ,W c ) K samples K FIGURE 5 A schematic picture of the two problems under study. The objective in the memorization problem is to find a control input pw c , W c q that takes all the sample points in K samples (represented by red dots) to their corresponding points in ψpK samples q (represented by blue dots); some of these assignments are shown by black dashed arrows. In the universal approximation problem, the points outside K samples also need to be carefully mapped to guarantee approximation in the uniform sense; some of these assignments are represented by dashed green arrows. Contrary to the memorization problem, the univer- sal approximation problem requires the approximation guarantees to hold for every element in K, which is no longer a finite set. Figure 5 provides a schematic picture of memorization and universal approximation. Naturally, geometric control theory plays a key role in the study of the memorization problem when framed as a controlla- bility problem. In particular, a keen reader would realize that the classical Lie algebraic conditions for accessibility require the derivatives of the activation function to appear in the analysis of this problem. As shown in [63], a suffi- cient condition for controllability is provided by asking the activation function σ, or its derivatives, to satisfy an ordinary differential equation with quadratic righthand side; interestingly, a large class of widely used activations functions satisfy this condition. The key result in [63] provides answers to both mem- orization and universal approximation problems. In par- ticular, given a continuous function ψ : R n 1 Ñ R o , and assuming that the activation function satisfies a differential equation with quadratic right-hand side, the following was established: » Memorization: The memorization problem has a so- lution for any ε P R ` . » Universal Approximation: The problem has a so- lution for any ε P R ` when n is given by 2 maxtn 1 , ou ` 1. The results in [63] are, in fact, sharper in the sense that, under suitable technical assumptions, the memorization problem can be solved with ε “ 0 for almost all data sets and universal approximation can be achieved using a smaller number of neurons for a more restricted class of functions. Notably, in the case where n 1 “ o, the universal approximation statement above improves upon the one in [61], which requires n 1 ` o` 2 neurons. The problem of universal approximation led to a fruitful bridge between control theory and deep learning, see for instance [70], [71], [72], [73], [74]. In particular, the composition of flows, a concept with deep roots in control theory, has become instrumental for the study of Neural ODEs. This perspective has spurred substantial activity in the past few years at this intersection, including [75], [76], [77], [78], [79], [80], [81]. We close this section by discussing a few intriguing directions related to the approximation power of Neural ODEs for density estimation by analyzing controllability properties of the corresponding continuity equation, where the driving vector field is parameterized by a Neural ODE. For instance, results in [82], [83] establish approximate controllability of the controlled continuity equation on the space of probability measures, with connections to optimal transportation theory: given initial and target distributions, it is shown that the Neural ODE’s weights can be chosen so that the resulting terminal distribution is arbitrarily close to the target in an appropriate measure. The literature on this subject is fast growing [84], [85], [86], and the tech- niques have also recently been employed for continuity equations corresponding to the self-attention dynamics in transformers [87], [88]. LARGE LANGUAGE MODELS, TRANSFORMERS, AND CONSENSUS Large language models (LLMs) is now used as a generic term for N architectures that generate textual responses to user’s queries in natural language. It was formed by pre-pending “large”, referring to model and training data size, to “language models”, referring to the models used in natural language processing. Transformer-based LLMs are a good example of how ideas from control can be used to better understand deep learning models. First, by passing to the continuous limit, as explained in Differential equa- tions, control systems, and neural networks, the transformer architecture can be described by a differential equation. Second, the differential equation model can be interpreted as a generalization of the consensus models studied in control theory. This section explains these connections and 12 IEEE CONTROL SYSTEMS » OCTOBER 2026 how ideas from consensus on manifolds reveal the role of the attention mechanism in transformer-based LLMs. An LLM first converts the query or prompt into a sequence of tokens z 1 p1q, . . . , z m p1q using the map f ap- pearing in (S9). Then, a sequence-to-sequence N is em- ployed to process the sequence of tokens and produce the output sequence z 1 pℓq, . . . , z m pℓq. Although there are several choices for this N, the most widely used is the transformer described in the sidebar “What are trans- formers?”. Other possibilities are the RWKV model [89], which is based on a recurrent neural network (RNN), and the Mamba model, which is based on a selective state- space-model (SSM). See sidebar “What are recurrent neural networks and selective state-space models?” for an intro- duction to RNNs and SSMs. The output of the sequence- to-sequence model, z 1 pℓq, . . . , z m pℓq, is processed by the output map h, see (S9), to predict the next token z m`1 p1q which is appended to the end of the input token sequence to generate the longer sequence z 1 p1q, . . . , z m p1q, z m`1 p1q. This longer sequence is then processed by the sequence- to-sequence N to predict the next token, analogously to a feedback loop. The process is iterated until the reply sentence or sentences are completed. Self-attention dynamics Denote by Zpkq “ ” z 1 pkq|z 2 pkq| . . .|z m pkq ı P R dˆm the matrix representation of the tokens at layer k of a transformer N. The transformer block updates each to- ken according to the recurrence formula ̃ z i pkq “ npz i pkq` A i pZpkqqq,(0.39) z i pk` 1q “ np ̃ z i pkq` r k p ̃ z i pkqqq,(0.40) where n : R d Ñ R d is the normalization map, r k p ̃ z i pkqq “ W 2 pkqΣpW 1 pkq ̃ z i pkqq, and A “ ” A 1 |A 2 | . . .|A m ı : R dˆm Ñ R dˆm is the self-attention mechanism (see sidebar “What are transformers?” for a detailed description of these objects). Focusing on the self-attention mechanism by removing the skip-connection (0.40), the transformer block (0.39), (0.40) simplifies to z i pk` 1q “ npz i pkq` A i pZpkqqq.(0.41) It was first noted in [90] and further developed in [91] that the attention block (0.41) resembles the dynamics of a multi-particle system and, following the ideas presented in Differential equations, control systems, and neural networks, it can be modeled by a system of differential equations. In the differential equation model, the matrix represen- tation of tokens is parameterized by t P R ` 0 : Z c ptq “ ” z c,1 ptq|z c,2 ptq| . . .|z c,m ptq ı P R dˆm . Note the deviation from the description in Differential equa- tions, control systems, and neural networks, where the time domain was normalized to r0, 1s. This section discusses the asymptotic behavior of transformers thus suggesting R ` 0 as the natural time domain. As it will be seen, existence of solutions is not a concern since the state space will be a product of spheres, hence compact. Similarly to the token matrix, the weight matrices are functions of time W Q j ptq, W K j ptq P R eˆd and W V j ptq P R e 1 ˆd . Hence, the attention mechanism can be written as A c pt, Z c q “ ” A c,1 pt, Z c q|A c,2 pt, Z c q| . . .|A c,m pt, Z c q ı “ a ÿ j“1 W U j ptq Z c softmax ̃ Z J c W P j ptqZ c ? e ̧ , (0.42) where a is the number of heads, the softmax is applied column-wise, and the matrices W U j ptq “ W o j W V j ptq P R dˆd , W P j ptq “ W J K j ptqW Q j ptq P R mˆm were used for brevity. When there is only one head (a “ 1), the subscript j “ 1 is omitted for simplicity, that is, W U 1 ptq simplifies to W U ptq and W P 1 ptq simplifies to W P ptq. The self-attention mechanism (0.41) can now be written 9 z c,i “ T z c,i n ̈ A c,i pt, Z c ptqq,(0.43) where T z n : R d Ñ R d denotes the tangent map of n at z P R d using the identification T z R d » R d . For the purpose of analyzing the asymptotic behavior of z c,i , the normalization map is typically chosen to be the projection onto the pd ́ 1q-dimensional sphere S d ́1 “ tz P R d |z J z “ 1u. In other words, npzq “ z z ,(0.44) where z denotes the Euclidean norm of z P R d zt0u thus forcing tokens to have unit norm. Therefore, the configuration space of both the discrete (0.41) and the con- tinuous (0.43) dynamical systems governing the evolution of attention is the product of m copies of the pd ́ 1q-sphere pS d ́1 q m “ S d ́1 ˆ . . .ˆ S d ́1 loooooooooomoooooooooon m-times . Recall that the tangent space of the sphere at each point z P S d ́1 consists of all vectors that are orthogonal to z, that is T z S d ́1 “ tX P R d |z J X “ 0u. The tangent map of the projection thus reads T z n ̈ X “ pI d ́ z J q ̈ X for each X P R d . In real transformers, however, the standard layer nor- malization, npzq “ 1 σpzq pz ́ μpzq1q‹ γ` β, OCTOBER 2026 « IEEE CONTROL SYSTEMS 13 is the most common choice. In the previous expression, 1 denotes the vector p1, . . . , 1q P R d , ‹ denotes the element- wise product of vectors, and γ, β P R d are the learned scale and shift, respectively. Similarly, μpzq and σpzq de- note the mean and standard deviation of z, respectively. An easy check shows that, if γ “ γ 0 1 for some fixed γ 0 ‰ 0, then the tokens lie on the pd ́ 1q-sphere of center β and radius dγ 2 0 . Therefore, the theoretical results with the normalization to the sphere (0.44) also apply to the standard layer normalization. For arbitrary scale parameters, it is conjectured that the tokens also lie on a certain hypersurface of R d diffeomorphic to the pd ́ 1q- sphere, making the theoretical results still valid. Consensus dynamics Many empirical works starting with [92] observed that tokens tend to converge to consensus as the depth of the transformer increases, that is, the model collapses as all tokens converge to a single point. The first theoreti- cal explanation for this observed behavior was provided in [93], [94]. The continuous attention dynamics (0.43) was regarded as a mean-field interacting particle system and it was shown that, for almost every initial condition, all tokens converge to a fixed position as time goes to infinity. The analysis was restricted to a simplified model with only one head, a “ 1, and weight matrices assumed to be constant and proportional to the identity, W U ptq “ I d and W P ptq “ β I m , where β ą 0 is known as the coldness or inverse temperature. Under different assumptions on d, m and β, asymptotic convergence to consensus was proven using techniques from partial differential equations and ergodic theory. In subsequent papers [95], [96], [97], it was also shown that metastability holds under certain conditions on the param- eters and the initial conditions, that is, tokens first form several clusters and remain there for some time before eventually collapsing to a single one. The aforementioned papers address the full self-attention mechanism, but anal- ogous results have been proven for the causal (auto- regressive) attention [97]. The causal attention mechanism is a modification of (0.42) where the attention for the i-th token only depends on the previous ones. Ideas from consensus dynamics and control theory have also demonstrated usefulness in the asymptotic analysis of the transformer dynamics [98], [99], [100], enabling the analysis of (0.43) for more general weight matrices. To better understand the relation between (0.43) and the consensus models used in control, note that (0.43) can also be written in the following form when there is a single head (a “ 1): 9 z c,i “ T z c,i n ̈ W U ptq m ÿ j“1 a ij pt, Z c qz c,j , where the coefficients a ij appearing in the above linear combination are strictly positive and sum to 1, that is, ř m j“1 a ij “ 1. Differently from most consensus models investigated in control, the coefficients a ij are nonlinear functions of the state Z c . The presence of the matrix W U ptq is another difference with respect to the consensus models in the control literature. For causal self-attention, an input-to-state stability argu- ment shows that the behavior of the system is determined by the evolution of the first token. Causality results in a triangular structure on the dynamics suggestive of con- trol systems in strict feedback form [101]. Moreover, the dynamics of token i is input-to-state stable with respect to distance between tokens 1, . . . , i ́ 1 and the consensus set, that is, the set defined by z c,1 “ z c,2 “ . . . “ z c,i ́1 . Hence, asymptotic stability towards the consensus set is established using input-to-state stability of cascade inter- connections. The results proven using these techniques point in the same direction as previous contributions to this problem: the model collapses as all tokens converge to consensus as time increases. The main advantage of using control theoretic techniques compared to the mean-field approach is that they allow for addressing non-identity and time- varying weight matrices. This situation is significantly more realistic as weights depend on the layer in real transformers. Namely, the results in [98] for the causal self- attention case may be summarized in three cases: 1) a ě 1 and W U j ptq “ I d for all heads. Then the first token is fixed and, thus, all tokens converge to the initial position of the first token. The domain of attraction of this consensus is co-null, that is, its complement has zero measure. 2) a “ 1 and W U ptq “ W U constant and symmet- ric. Then the first token converges to the principal eigenvector of W U , that is, the eigenvector (up to sign) corresponding to maximum eigenvalue. The sign is determined by the initial position of the first token. More recently, [100] extended this result to the full self-attention. They also identified new types of equilibria showing that all of them are unstable. 3) a “ 1 and W U ptq bounded and symmetric for all times. Then the principal eigenvalue of W U ptq also depends on time. As a result, tokens do not converge to a fixed position, but they keep following the prin- cipal eigenvector instead. Specifically, they converge to a ball around the principal eigenvector forming a moving cluster. The size of the ball depends on how fast the eigenvector moves. Even though a fully general asymptotic analysis of the model (0.43) is still missing, all the results to date suggest model collapse. 14 IEEE CONTROL SYSTEMS » OCTOBER 2026 Empirical validation of theoretical analysis The collapse of the attention mechanism, explained by the theoretical analysis in [98], is supported by empirical results on LLMs that include the skip-connection (0.40) and thus go beyond the simplifying assumption made when deriving the theoretical differential equation model. Several tests on the GPT-2 XL and the GPT-Neo 2.7B models are reported in [99]. Since the theoretical results are asymptotic, the depth was increased by looping the transformers, that is, by feeding the tokens at the final layer as inputs to the first layer. The error function Epkq “ 1 ́ 1 m m ÿ i“1 z J 1 pkq z i pkq |z 1 pkq||z i pkq| (0.45) was used to measure how far the tokens are from the first one. Note that Epkq ě 0 and it vanishes only when all tokens are aligned, z 1 pkq “ z 2 pkq . . . “ z m pkq, that is, when consensus is reached. The average of (0.45) was computed across 100 random prompts generated by uniformly sam- pling 200 tokens from the GPT-2 tokenizer’s vocabulary. Two types of experiments comparing the GPT-2 XL and the GPT-Neo 2.7B models were performed (all figures are extracted from [99]): 1) Evolution of tokens using pre-trained LLMs. Using the pre-trained weights provided by the Hugging Face library [102]. Looping the transformers yields weight matrices that are time-varying but periodic. Fig- ure 6 shows the evolution of E when the skip- connection (0.40) is absent and thus there is agree- ment between the theoretical model and the LLMs. Figure 7 shows the same experiment with the full model, that is, (0.39) and (0.40). 2) Evolution of tokens using random weight matrices. As for the previous case, the experiments were carried out for the self-attention only, as well as for the full model. In Figure 8, the random weights were fixed, that is, they were the same for each pass of the model, providing time-varying, periodic weights. In Figure 9, new random weights were generated before each pass of the model, providing time-varying, non periodic weight matrices. We can observe that in all the above described cases the function E converges to zero thus showing that all the tokens converge to the same value. The experiments with randomly generated matrices suggest that consensus is a structural property of the architecture and does not depend on the choice of weights or prompts. It can be observed that the rate of convergence depends on the skip- connections: for the GPT-2 model it increases, whereas for the GPT-Neo model it decreases. Nevertheless, consensus still takes place when the skip-connection layers are con- sidered. As an illustration of consensus, we took the first sentenceofthisarticle, Deep neural networks, in their different shapes and flavors, are having a remarkable impact in almost every area of science and engineering., and passed it through GPT-2-XL. The decoded output is presented in Layer 0100200300400500 E 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 GPT-2 GPT-Neo FIGURE 6 Comparison between the GPT-2 XL and GPT-Neo 2.7B architectures with feedforward layers removed; evaluation of the average of (0.45) across all the random prompts. Layer 050100150200250 E 0 0.5 1 1.5 GPT-2 GPT-Neo FIGURE 7 Comparison between the GPT-2 XL and GPT-Neo 2.7B architectures with the full model; evaluation of the average of (0.45) across all the random prompts. 0500100015002000 E 0 0.5 1 GPT-2 GPT-Neo Layer 0500100015002000 E 0 0.5 1 GPT-2 without FeedForward GPT-Neo without FeedForward FIGURE 8 Comparison between the GPT-2 XL and GPT-Neo 2.7B architectures with fixed and randomly chosen weight matrices. Each model was evaluated with and without feedfoward layers using the average of (0.45) across all the random prompts. OCTOBER 2026 « IEEE CONTROL SYSTEMS 15 TABLE 1 Illustration of consensus by decoding the out- put produced by passing the same sentence through GPT2-XL an increasing number of times # passesOutput 1ly networks are which particular simplest forms and sizes, are the a moment impact on many every field of science and technology. that automated them is used.’ produce They 2, neural for network network the the andy the the same good good, The a the„, a, 4TheTheTheTheTheTheTheTheThe TheTheTheThe"TheTheTheTheThe TheTheTheTheTheTheThe" 10TheTheTheTheTheTheTheTheThe TheTheTheTheTheTheTheTheThe TheTheTheTheTheTheTheThe Table 1 along with the corresponding number of passes. We can appreciate that after 4 passes consensus has been reached since all the tokens are close enough so as to decode to the same word: The. OTHER RESEARCH DIRECTIONS This section focuses on more recent and innovative work that opens exciting new directions at the interplay between control theory and deep learning. Given the recent and exploratory nature of these developments, the discussion is less detailed and emphasizes the key ideas behind each approach, their potential advantages, and how they may contribute to the broader landscape of deep learning. Selective State Space Models RNNs and SSMs are the architectures that most closely resemble the models studied in control theory as they are directly given as discrete-time nonlinear control systems, see “What are recurrent neural networks and selective 0500100015002000 E 0 0.5 1 GPT-2 GPT-Neo Layer 0500100015002000 E 0 0.5 1 GPT-2 without FeedForward GPT-Neo without FeedForward FIGURE 9 Comparison between the GPT-2 XL and GPT-Neo 2.7B architectures with random weight matrices chosen before each model pass. Each model was evaluated with and without feedfoward layers using the average of (0.45) across all the random prompts. state-space models?” sidebar. RNNs are statefull models and thus can produce outputs that depend on past inputs since input information is stored in the state. In principle, this makes these models especially useful when there are long range dependencies between inputs and outputs. In practice, however, RNNs are difficult to train due to the vanishing (or exploding) gradient problem. In contrast, transformers can only reason about dependencies that oc- cur in a window whose length corresponds to the number of processed tokens. Yet, this restriction makes them more reliable, parallelizable, and scalable. Much excitement exists around SSMs; they are also statefull models yet overcome the complexity limitations brought by the difficulty of parallelizing the training of RNNs. The training and inferencing of SSMs is paral- lelizable, since the matrix Apk, zpk, iqq is restricted to be diagonal, but this restriction also raises the question of understanding if it compromises the ability of SSMs to model long range dependencies between inputs and out- puts. Answering this question is still an active area of research. It is established in [103] that SSMs are comparable to transformers in their ability to identify the most relevant information within a text. It is also known, see [104], that there are simple state tracking tasks that cannot be accomplished by SSMs, neither by transformers, but can be performed by RNNs. Another question that remains open is determining which tasks are better suited for transformers or for SSMs. A recent study [105] conducted a large-scale empirical comparison between Mamba and Mamba-2 models, and transformers. The results show that while Mamba-2 can outperform transformers of comparable size, this advan- tage is not transversal across all tasks. Interestingly, the same work introduced a hybrid model combining attention and SSM layers. This hybrid model maintained faster in- ference times than standard transformers while achieving superior performance on all standard benchmarks tested, thus suggesting that hybrid architectures may be prefer- able. Ultimately, the absolute scalability limits of pure SSM architectures remain an open question. While some models today have successfully reached approximately 8 billion parameters [105], it is still unknown how far SSMs can be pushed before encountering diminishing returns. Mean- while, a promising frontier has emerged in the form of hybrid architectures. By fusing traditional attention mech- anisms with state-space components, some works have shown that these hybrid models can outperform both SSMs and transformers of similar size [106], [29]. As the field progresses, however, control-theoretic tech- niques have yet to be used with SSMs. Given their struc- ture, which naturally resembles nonlinear control systems, it remains an open and promising direction to investigate whether insights from control theory can be leveraged to 16 IEEE CONTROL SYSTEMS » OCTOBER 2026 analyze the limitations of SSMs and potentially improve their design and training. Safety and Alignment In recent years, the alignment of LLMs and SSMs with human intent has become a central challenge in deep learning. From a control-theoretic perspective, alignment can be formulated as a safety enforcement problem or as a trajectory tracking problem on a N regarded as the plant. Several efforts have adopted a feedback control per- spective, even if this is not explicitly stated. One of the most popular techniques for alignment [107] uses human feedback to modify the underlying weights of a trained N via reinforcement learning. This approach introduces a separate neural network, known as a reward model, which is trained to emulate human preferences. During the alignment phase, the language model acts as the plant, and the reward model provides the feedback signal used by a reinforcement learning algorithm to update the plant’s internal parameters. This effectively closes the loop between model generation and human intent, steering the plant towards a preferred behavior. Performing alignment during training is expensive since it delays the training process which is already quite expensive. A convenient alternative is test time alignment, that is, algorithms that act as external controllers to regu- late the N behavior during inference without modifying the underlying weights. This is analogous to adding a controller to steer a given system to a reference. Control barrier functions (CBFs) have been used for test time alignment purposes. One framework [108] designs an add-on safety filter that intervenes in the token prediction process. The filter acts by filtering out undesirable words thus ensuring the generated text will not violate specified safety constraints, such as toxicity thresholds. A different approach is taken in [109] by adding a one token prefix to every prompt. Control is exerted by steering this prefix so as to prevent toxic outputs. Toxicity is measured by a classifier that maps the state z of an LLM into a numerical toxicity score for the text encoded by the tokens. The key result in [109] is that, under mild technical assumptions, a toxicity classifier defines a CBF for almost every choice of weights of an LLM with the added one token prefix as an input. In [110], an optimal control approach is adopted to prevent toxicity. Text generation is framed as a trajectory in the space of tokens. The alignment is formulated as the problem of minimally changing this trajectory, subject to the token dynamics (0.24), so as to steer the output of the activation functions away from regions that generate toxic outputs. Often, preventing toxicity is but one of many objec- tives. Multi-objective decoding (MOD) [111] addresses the challenge of satisfying multiple competing alignment goals simultaneously. By formulating the problem as maximiz- ing a linear combination of a measure of goal satisfaction during the decoding step (the map h in (S3) or (S9)), this approach allows for the dynamic adjustment of these objectives at runtime, functioning like a variable-gain con- troller that can be tuned on the fly. Controllers for architectural and training design Feedback controllers have been used to improve the be- havior of certain architectures. A prime example is the PIDformer [112] that inserts PID controllers in transformer blocks to create a new architecture. The objective of the PID controller is to prevent consensus, as discussed in Large language models, transformers, and consensus, and improve robustness to input perturbations. In addition to altering architectures, feedback con- trollers have been designed to improve the training of NNs. Research into PID-based optimizers has shown that applying PID control to adaptive gradient methods can dampen the oscillations seen in standard stochastic gra- dient descent or Adam method, leading to faster conver- gence [113]. Other approaches [114] treat learning rate scheduling as an online control problem, adjusting the optimizer’s step size based on real-time feedback from the training loss via an incremental PID controller. This line of work suggests that training algorithms can be seen as plants controlled by a feedback controller designed to drive the N towards a minimum with desirable proper- ties. Until now, only relatively simple control techniques have been used and more advanced control ideas are likely to have greater benefits. Contraction and implicit neural networks Contraction theory [115] has found multiple applications within control theory. Recently, its use has been expanded to also include the analysis of deep equilibrium net- works [116]. This is a class of NNs where state zpk ` 1q of layer k` 1 is computed from the state of layer k using an implicit equation such as zpk` 1q “ΣpW 1 pkqzpk` 1q` W 2 pkqzpkqq.(0.46) One of the key challenges in this class of NNs is to ensure the weight matrices W 1 pkq and W 2 pkq are such that (0.46) admits a solution for zpk ` 1q. Since a solution can be interpreted as a fixed-point, the use of contraction ideas is natural and was employed in [117] to establish existence of solutions as well as other properties. Recurrent versions of deep equilibrium networks have also been developed and, once again, contraction theory played a key role in their analysis [118]. The authors believe that contraction theory, and its generalizations such as k-contraction [119], can also play an important role in the analysis and design of other classes of NNs as well as their training algorithms. OCTOBER 2026 « IEEE CONTROL SYSTEMS 17 ACKNOWLEDGMENT TheauthorsacknowledgeMatteoMarchi,Rahal Nanayakkara,andValenYamamotoforvaluable suggestions on how to improve this article. The 1st author was partially supported by grant PID2024-156578NB-I00 funded by MICIU/AEI/10.13039/501100011033/FEDER, EU. The work of the 2nd and 4th authors was partially supported by the NSF award 2502536. The work of the 3rd author is supported by the Natural Sciences and Engineering Research Council of Canada. AUTHOR INFORMATION Álvaro Rodríguez Abella (arabella@comillas.edu) received the PhD degree in Mathematics from Complutense Uni- versity of Madrid in Spain. He is currently an Assistant Professor of Applied Mathematics at Comillas Pontifical University. João Pedro Silvestre (joaosilvestre@ucla.edu) received the Masters degree in Electrical and Computer Engineering from Instituto Superior Técnico in Lisbon, Portugal. He is currently pursuing his PhD on Electrical and Computer Engineering at UCLA. He is a Student Member of IEEE. Bahman Gharesifard (bahman.gharesifard@queensu.ca) received the PhD degree in Mathematics from Queen’s University where he is currently a Professor of Mathemat- ics. He is a Senior Member of IEEE. Paulo Tabuada (tabuada@e.ucla.edu) received the PhD degree in Electrical and Computer Engineering from Insti- tuto Superior Técnico in Lisbon, Portugal. He is the Vijay K. Dhir Professor of Engineering at UCLA’s Electrical and Computer Engineering Department. He is an IEEE Fellow. REFERENCES [1] E. Zuazua, “Machine learning and control: Foundations, advances, and perspectives,” 2025. https://arxiv.org/abs/2510.03303. [2] E. Weinan, J. Han, and Q. Li, “Dynamical systems and optimal control approach to deep learning,” Mathematical Aspects of Deep Learning, p. 422– 438, 2022. [3] W. S. McCulloch and W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” The bulletin of mathematical biophysics, vol. 5, no. 4, p. 115–133, 1943. [4] F. Rosenblatt, “The perceptron: a probabilistic model for information storage and organization in the brain.,” Psychological review, vol. 65, no. 6, p. 386, 1958. [5] M. Minsky and S. A. Papert, Perceptrons: An Introduction to Computational Geometry. The MIT Press, 09 2017. [6] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning represen- tations by back-propagating errors,” Nature, vol. 323, no. 6088, p. 533–536, 1986. [7] Y. Le Cun, “Learning process in an asymmetric threshold network,” in Disordered systems and biological organization, p. 233–240, Springer, 1986. [8] P. Werbos, “Beyond regression: New tools for prediction and analysis in the behavioral sciences,” PhD thesis, Committee on Applied Mathematics, Harvard University, Cambridge, MA, 1974. [9] H. R. Wilson and J. D. Cowan, “Excitatory and inhibitory interactions in localized populations of model neurons,” Biophysical journal, vol. 12, no. 1, p. 1–24, 1972. [10] W. Maass, “Networks of spiking neurons: the third generation of neural network models,” Neural networks, vol. 10, no. 9, p. 1659–1671, 1997. [11] A. L. Hodgkin and A. F. Huxley, “A quantitative description of mem- brane current and its application to conduction and excitation in nerve,” The Journal of physiology, vol. 117, no. 4, p. 500, 1952. [12] J. L. Elman, “Finding structure in time,” Cognitive science, vol. 14, no. 2, p. 179–211, 1990. [13] S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber, “Gradi- ent flow in recurrent nets: The difficulty of learning long-term depen- dencies,” in A Field Guide to Dynamical Recurrent Networks (J. F. Kolen and S. C. Kremer, eds.), p. 237–243, IEEE Press, 2001.Available at: https://ieeexplore.ieee.org/document/5264952. [14] V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), p. 807–814, 2010. [15] D. Hendrycks and K. Gimpel, “Gaussian error linear units (Gelus),” arXiv preprint arXiv:1606.08415, 2016. [16] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. The MIT Press, 2016. [17] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, vol. 25, p. 1097–1105, Curran Associates, Inc., 2012. [18] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, p. 770–778, 2016. [19] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, (Red Hook, NY, USA), p. 6391–6401, Curran Associates Inc., 2018. [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, (Red Hook, NY, USA), p. 6000–6010, Curran Associates Inc., 2017. [21] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (Y. Bengio and Y. LeCun, eds.), 2015. Available at: http://arxiv.org/abs/1409.0473. [22] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curiouscaseofneuraltextdegeneration,”in8thInternational ConferenceonLearningRepresentations,ICLR2020,AddisAbaba, Ethiopia, April 26-30, 2020, OpenReview.net, 2020.Available at: https://openreview.net/forum?id=rygGQyrFvH. [23] D. Jurafsky and J. H. Martin, Speech and Language Processing: An Intro- duction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models. 3rd ed., 2026. Online manuscript released January 6, 2026. [24] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, p. 1735–1780, 1997. [25] A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022. Available at: https://openreview.net/forum?id=uYLFoz1vlAC. [26] J. T. H. Smith, A. Warrington, and S. W. Linderman, “Simplified state space layers for sequence modeling,” 2023. [27] C. A. Alonso, J. Sieber, and M. N. Zeilinger, “State space models as foundation models: A control theoretic overview,” in 2025 American Control Conference (ACC), p. 146–153, IEEE, 2025. [28] A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” in First conference on language modeling, p. 1–32, 2024. [29] B. L. et al., “Jamba: Hybrid transformer-mamba language models,” in Proceedings of the Thirteenth International Conference on Learning Representa- tions, 2025. Available at: https://openreview.net/forum?id=JFPaD7lpBD. [30] W. E, “A proposal on machine learning via dynamical systems,” Com- munications in Mathematics and Statistics, vol. 5, no. 1, p. 1–11, 2017. [31] E. Haber and L. Ruthotto, “Stable architectures for deep neural net- works,” Inverse Problems, vol. 34, p. 014004, dec 2017. [32] Y. Lu, A. Zhong, Q. Li, and B. Dong, “Beyond finite layer neural net- works: Bridging deep architectures and numerical differential equations,” in Proceedings of the 35th International Conference on Machine Learning, ICML 18 IEEE CONTROL SYSTEMS » OCTOBER 2026 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 (J. G. Dy and A. Krause, eds.), vol. 80 of Proceedings of Machine Learning Research, p. 3282– 3291, PMLR, 2018. [33] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud, “Neural ordinary differential equations,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, (Red Hook, NY, USA), p. 6572–6583, Curran Associates Inc., 2018. [34] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. M. Stuart, and A. Anandkumar, “Fourier neural operator for parametric partial differential equations,” in 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021. Available at: https://openreview.net/forum?id=c8P9NQVtmnO. [35] N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stu- art, and A. Anandkumar, “Neural operator: Learning maps between func- tion spaces with applications to PDEs,” Journal of Machine Learning Research, vol. 24, no. 89, p. 1–97, 2023. [36] M. Krstic, “Machine learning: Bane or boon for control?: 2023 Bode prize lecture,” IEEE Control Systems, vol. 44, no. 4, p. 24–37, 2024. [37] A. M. Stuart and A. R. Humphries, Dynamical Systems and Numerical Analysis. Cambridge University Press, 1996. [38] A. L. Dontchev and W. W. Hager, “The Euler approximation in state constrained optimal control,” Mathematics of Computation, vol. 70, no. 233, p. 173–203, 2001. [39] W. Hager, “Runge-Kutta methods in optimal control and the trans- formed adjoint system,” Numerische Mathematik, vol. 87, no. 2, p. 247–282, 2000. [40] J. F. Bonnans and A. Festa, “Error estimates for the Euler discretization of an optimal control problem with first-order state constraints,” SIAM Journal on Numerical Analysis, vol. 55, no. 2, p. 445–471, 2017. [41] M. Thorpe and Y. van Gennip, “Deep limits of residual neural net- works,” Research in the Mathematical Sciences, vol. 10, 2023. [42] X. Zhang, Z. Li, C. C. Loy, and D. Lin, “PolyNet: A pursuit of structural diversity in very deep networks,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 3900–3908, 2017. [43] M. E. Sander, P. Ablin, M. Blondel, and G. Peyré, “Momentum residual neural networks,” in Proceedings of the 38th International Conference on Machine Learning (M. Meila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research, p. 9276–9287, PMLR, 18–24 Jul 2021. [44] A. N. G ́ mez, M. Ren, R. Urtasun, and R. B. Grosse, “The reversible resid- ual network: backpropagation without storing activations,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, (Red Hook, NY, USA), p. 2211–2221, Curran Associates Inc., 2017. [45] Q. Li, L. Chen, C. Tai, et al., “Maximum principle based algorithms for deep learning,” Journal of Machine Learning Research, vol. 18, no. 165, p. 1– 29, 2018. [46] B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert, and E. Holtham, “Reversible architectures for arbitrarily deep residual neural networks,” in Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18, p. Pages 2811 – 2818, AAAI Press, 2018. [47] Y. L. Cun, “A theoretical framework for back-propagation,” in Proceed- ings of the 1988 Connectionist Models Summer School (D. S. Touretzky, G. E. Hinton, and T. J. Sejnowski, eds.), p. 21–28, Pittsburgh, Pennsylvania: Morgan Kaufmann, 1988. [48] M. Benning, E. Celledoni, M. J. Ehrhardt, B. Owren, and C.-B. Schönlieb, “Deep learning as optimal control problems: Models and numerical meth- ods,” Journal of Computational Dynamics, vol. 6, no. 2, p. 171–198, 2019. [49] E. Hairer, C. Lubich, and G. Wanner, Geometric numerical integration. Structure-preserving algorithms for ordinary differential equations, vol. 31 of Springer Series in Computational Mathematics. Springer, 2nd ed., 2006. [50] Q. Li and S. Hao, “An optimal control approach to deep learning and applications to discrete-weight neural networks,” in Proceedings of the 35th International Conference on Machine Learning (J. Dy and A. Krause, eds.), vol. 80 of Proceedings of Machine Learning Research, p. 2985–2994, PMLR, 10–15 Jul 2018. [51] D. Zhang, T. Zhang, Y. Lu, Z. Zhu, and B. Dong, “You only propagate once: Accelerating adversarial training via maximal principle,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada (H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, eds.), p. 227–238, 2019. [52] J. H. Seidman, M. Fazlyab, V. M. Preciado, and G. J. Pappas, “Robust deep learning as optimal control: Insights and convergence guarantees,” in Proceedings of the 2nd Conference on Learning for Dynamics and Control (A. M. Bayen, A. Jadbabaie, G. Pappas, P. A. Parrilo, B. Recht, C. Tomlin, and M. Zeilinger, eds.), vol. 120 of Proceedings of Machine Learning Research, p. 884–893, PMLR, 10–11 Jun 2020. [53] E. Celledoni, M. J. Ehrhardt, C. Etmann, R. I. McLachlan, B. Owren, C.- B. Schonlieb, and F. Sherry, “Structure-preserving deep learning,” European journal of applied mathematics, vol. 32, no. 5, p. 888–936, 2021. [54] G. Cybenko, “Approximation by superpositions of a sigmoidal func- tion,” Mathematics of control, signals and systems, vol. 2, no. 4, p. 303–314, 1989. [55] K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, no. 5, p. 359–366, 1989. [56] K. Hornik, “Approximation capabilities of multilayer feedforward net- works,” Neural networks, vol. 4, no. 2, p. 251–257, 1991. [57] A. Pinkus, “Approximation theory of the MLP model in neural net- works,” Acta Numerica, vol. 8, p. 143–195, 1999. [58] A. Barron, “Universal approximation bounds for superpositions of a sigmoidal function,” IEEE Transactions on Information Theory, vol. 39, no. 3, p. 930–945, 1993. [59] H. Lin and S. Jegelka, “ResNet with one-neuron hidden layers is a universal approximator,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, (Red Hook, NY, USA), p. 6172–6181, Curran Associates Inc., 2018. [60] S. Park, C. Yun, J. Lee, and J. Shin, “Minimum width for universal approximation,” in International Conference on Learning Representations, 2021. Available at: https://openreview.net/forum?id=O-XJwyoIF-k. [61] P. Kidger and T. Lyons, “Universal Approximation with Deep Narrow Networks,” in Proceedings of Thirty Third Conference on Learning Theory (J. Abernethy and S. Agarwal, eds.), vol. 125 of Proceedings of Machine Learning Research, p. 2306–2327, PMLR, 09–12 Jul 2020. [62] P. Tabuada and B. Gharesifard, “Universal approximation power of deep residual neural networks via nonlinear control theory,” in The 2021 International Conference on Learning Representations (ICLR), 2021. Available at: https://openreview.net/forum?id=-IXhmY16R3M. [63] P. Tabuada and B. Gharesifard, “Universal approximation power of deep residual neural networks through the lens of control,” IEEE Transactions on Automatic Control, vol. 68, no. 5, p. 2715–2728, 2023. [64] P. Tabuada and B. Gharesifard, “Corrigendum to “universal approxima- tion power of deep residual neural networks through the lens of control”,” IEEE Transactions on Automatic Control, vol. 69, no. 7, p. 4891–4892, 2024. [65] A. Agrachev and A. Sarychev, “Control on the manifolds of mappings with a view to the deep learning,” Journal of Dynamical and Control Systems, vol. 28, no. 4, p. 989–1008, 2022. [66] E. D. Sontag and Y. Qiao, “Further results on controllability of recurrent neural networks,” Systems & control letters, vol. 36, no. 2, p. 121–129, 1999. [67] J.-S. Li and N. Khaneja, “Control of inhomogeneous quantum ensem- bles,” Physical Review A, vol. 73, no. 3, p. 030302, 2006. [68] R. W. Brockett, “Optimal control of the liouville equation,” in Proceedings of the International Conference on Complex Geometry and Related Fields, vol. 39 of AMS/IP Studies in Advanced Mathematics, p. 23–35, Providence, RI: American Mathematical Society, 2007. [69] A. Agrachev and A. Sarychev, “Control in the spaces of ensembles of points,” SIAM Journal on Control and Optimization, vol. 58, no. 3, p. 1579– 1596, 2020. [70] M. D. Kvalheim and E. D. Sontag, “Autoencoding dynamics: Topological limitations and capabilities,” 2025. [71] E. Bayram, M.-A. Belabbas, and T. Ba ̧sar, “Geometric foundations of tuning without forgetting in neural ODEs,” arXiv preprint arXiv:2509.03474, 2025. [72] T. Veeravalli and M. Raginsky, “Nonlinear controllability and function representation by neural stochastic differential equations,” in Learning for Dynamics and Control Conference, p. 838–850, PMLR, 2023. OCTOBER 2026 « IEEE CONTROL SYSTEMS 19 [73] J. Hanson and M. Raginsky, “Fitting an immersed submanifold to data via Sussmann’s orbit theorem,” Mathematics of Control, Signals, and Systems, vol. 37, no. 3, p. 479–506, 2025. [74] J. Hanson and M. Raginsky, “Expressivity of quadratic neural ODEs,” arXiv preprint arXiv:2504.09385, 2025. [75] A. Agrachev and C. Letrouit, “Generic controllability of equivariant systems and applications to particle systems and neural networks,” Annales de l’Institut Henri Poincaré C, vol. 43, no. 3, p. 639–668, 2025. [76] A. Scagliotti, “Deep learning approximation of diffeomorphisms via linear-control systems,” Mathematical Control and Related Fields, vol. 13, no. 3, p. 1226–1257, 2023. [77] A. Scagliotti and S. Farinelli, “Normalizing flows as approximations of optimal transport maps via linear-control neural ODEs,” Nonlinear Analysis, vol. 257, p. 113811, 2025. [78] J. Cheng, Q. Li, T. Lin, and Z. Shen, “Interpolation, approximation, and controllability of deep neural networks,” SIAM Journal on Control and Optimization, vol. 63, no. 1, p. 625–649, 2025. [79] Y. Duan and Y. Cai, “A minimal control family of dynamical systems for universal approximation,” IEEE Transactions on Automatic Control, vol. 70, no. 11, p. 7233–7244, 2025. [80] A. Álvarez-López, A. H. Slimane, and E. Zuazua, “Interplay between depth and width for interpolation in neural ODEs,” Neural Networks, vol. 180, p. 106640, 2024. [81] C. Prieur, M. Lazar, and B. Robu, “A unified representation of neural networks architectures,” 2026. [82] D. Ruiz-Balet and E. Zuazua, “Neural ODE control for classification, approximation, and transport,” SIAM Review, vol. 65, no. 3, p. 735–773, 2023. [83] K. Elamvazhuthi, B. Gharesifard, A. L. Bertozzi, and S. Osher, “Neural ODE control for trajectory approximation of continuity equation,” IEEE Control Systems Letters, vol. 6, p. 3152–3157, 2022. [84] K. Elamvazhuthi, X. Zhang, S. Oymak, and F. Pasqualetti, “Learning on manifolds: Universal approximations properties using geometric control- lability conditions for neural ODEs,” in Learning for Dynamics and Control Conference, p. 1–11, PMLR, 2023. [85] D. Ruiz-Balet and E. Zuazua, “Control of neural transport for normalis- ing flows,” Journal de Mathématiques Pures et Appliquées, vol. 181, p. 58–90, 2024. [86] B. Geshkovski and E. Zuazua, “Turnpike in optimal control of PDEs, ResNets, and beyond,” Acta Numerica, vol. 31, p. 135–263, 2022. [87] B. Geshkovski, P. Rigollet, and D. Ruiz-Balet, “Measure-to-measure interpolation using transformers,” arXiv preprint arXiv:2411.04551, 2024. [88] D. O. Adu and B. Gharesifard, “Approximate controllability of continu- ity equation of transformers,” IEEE Control Systems Letters, vol. 8, p. 964– 969, 2024. [89] Z. Li, T. Xia, Y. Chang, and Y. Wu, “A survey of RWKV,” Neurocomputing, vol. 649, p. 130711, 2025. [90] Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, and T.-Y. Liu, “Understanding and improving transformer from a multi-particle dynamic system point of view,” in Proceedings of the Second Workshop on Machine Learning and the Physical Sciences (NeurIPS 2019), 2019. Available at: https://arxiv.org/abs/1906.02762. [91] S. Dutta, T. Gautam, S. Chakrabarti, and T. Chakraborty, “Redesigning the transformer architecture with insights from multi-particle dynamical systems,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual (M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, eds.), p. 5531–5544, 2021. [92] Y. Dong, J.-B. Cordonnier, and A. Loukas, “Attention is not all you need: Pure attention loses rank doubly exponentially with depth,” in International Conference on Machine Learning, p. 2793–2803, PMLR, 2021. [93] B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet, “A mathe- matical perspective on transformers,” Bulletin of the American Mathematical Society, vol. 62, no. 3, p. 427–479, 2025. [94] B. Geshkovski, C. Letrouit, Y. Polyanskiy, and P. Rigollet, “The emer- gence of clusters in self-attention dynamics,” Advances in Neural Information Processing Systems, vol. 36, p. 57026–57037, 2023. [95] B. Geshkovski, H. Koubbi, Y. Polyanskiy, and P. Rigollet, “Dynamic metastability in the self-attention model,” arXiv preprint arXiv:2410.06833, 2024. [96] G. Bruno, F. Pasqualotto, and A. Agazzi, “Emergence of meta- stable clustering in mean-field transformer models,” in The Thir- teenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025.Available at: https://openreview.net/forum?id=eBS3dQQ8GV. [97] P. Rigollet, “The mean-field dynamics of transformers,” arXiv preprint arXiv:2512.01868, 2025. [98] Á. R. Abella, J. P. Silvestre, and P. Tabuada, “The asymptotic behavior of attention in transformers,” arXiv preprint arXiv:2412.02682, 2024. [99] Á. Rodríguez Abella, J. Silvestre, and P. Tabuada, “Consensus is all you get: The role of attention in transformers,” in Forty-second International Conference on Machine Learning, p. 174 – 184, 2025. [100] C. Altafini, “Multistability of self-attention dynamics in transformers,” IEEE Transactions on Automatic Control, p. 1–16, 2026. [101] R. A. Freeman and P. Kokotovi ́c, Robust Nonlinear Control Design. Birkhäuser Boston, MA, 2008. [102] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, p. 38–45, 2020. [103] N. Nishikawa and T. Suzuki, “State space models are provably comparable to transformers in dynamic token selection,” in Inter- national Conference on Learning Representations, 2024.Available at: https://api.semanticscholar.org/CorpusID:270095302. [104] W. Merrill, J. Petty, and A. Sabharwal, “The illusion of state in state- space models,” in Proceedings of the 41st International Conference on Machine Learning (R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, eds.), vol. 235 of Proceedings of Machine Learning Research, p. 35492–35506, PMLR, 21–27 Jul 2024. [105] R. Waleffe et al., “An empirical study of mamba-based language mod- els,” arXiv preprint arXiv:2406.07887, 2024. [106] L. Ren, Y. Liu, Y. Lu, Y. Shen, C. Liang, and W. Chen, “Samba: Simple hybrid state space models for efficient unlimited context language mod- eling,” in The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025. Available at: https://openreview.net/forum?id=bIlnpVM4bc. [107] L. Ouyang et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, p. 27730–27744, 2022. [108] Y. Miyaoka and M. Inoue, “Control barrier function for aligning large language models,” IEEE Transactions on Control Systems Technology, p. 1–10, 2026. [109] J. P. Silvestre, Á. Rodríguez Abella, and P. Tabuada, “Stopping LLMs from going rogue: A control barrier approach to text generation,” in 2025 IEEE 64th Conference on Decision and Control (CDC), p. 151–156, 2025. [110] E. Cheng and C. A. Alonso, “Linearly controlled language generation with performative guarantees,” arXiv preprint arXiv:2405.15454, 2024. [111] R. Shi, Y. Chen, Y. Hu, A. Liu, H. Hajishirzi, N. A. Smith, and S. S. Du, “Decoding-time language model alignment with multiple objectives,” in Advances in Neural Information Processing Systems (A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, eds.), vol. 37, p. 48875–48920, Curran Associates, Inc., 2024. [112] T. M. Nguyen, C. A. Uribe, T. M. Nguyen, and R. Baraniuk, “PIDformer: Transformer meets control theory,” in Proceedings of the 41st International Conference on Machine Learning (R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, eds.), vol. 235 of Proceedings of Machine Learning Research, p. 37776–37797, PMLR, 21–27 Jul 2024. [113] S. Chen, J. Liu, P. Wang, C. Xu, S. Cai, and J. Chu, “Accelerated optimiza- tion in deep learning with a proportional-integral-derivative controller,” Nature Communications, vol. 15, no. 1, 2024. [114] Z. Wang and J. Zhang, “Incremental PID controller-based learning rate scheduler for stochastic gradient descent,” IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 5, p. 7060–7071, 2022. [115] F. Bullo, Contraction Theory for Dynamical Systems. Seattle, WA: Kindle Direct Publishing, 1.3 ed., 2026. 20 IEEE CONTROL SYSTEMS » OCTOBER 2026 [116] S. Bai, J. Z. Kolter, and V. Koltun, “Deep equilibrium models,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada (H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, eds.), p. 688–699, 2019. [117] S. Jafarpour, A. Davydov, A. V. Proskurnikov, and F. Bullo, “Robust implicit networks via non-euclidean contractions,” in Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, (Red Hook, NY, USA), Curran Associates Inc., 2021. [118] M. Revay, R. Wang, and I. R. Manchester, “Recurrent equilibrium net- works: Flexible dynamic models with guaranteed stability and robustness,” IEEE Transactions on Automatic Control, vol. 69, no. 5, p. 2855–2870, 2024. [119] C. Wu, I. Kanevskiy, and M. Margaliot, “k-contraction: Theory and applications,” Automatica, vol. 136, p. 110048, 2022. OCTOBER 2026 « IEEE CONTROL SYSTEMS 21

Instant research discovery