Paper deep dive
Enforcing LLM Safety through DMD-based Classification of Prompt-Response Embedding Dynamics
Mohamed Akrout, Olivera Kotevska, Dan Wilson
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/21/2026, 3:19:17 AM
Summary
This paper proposes a black-box method for detecting unsafe Large Language Model (LLM) outputs by applying dynamical systems theory, specifically Dynamic Mode Decomposition (DMD) and Koopman operators, to prompt and response embedding dynamics. The authors introduce a differential residual score that compares prediction errors of fitted safe and unsafe Koopman operators for both prompts and responses. Evaluated on three safety benchmarks (Aegis, Synthetic CoT, BeaverTails) using three embedding models (Qwen3-Embed, Mistral, Llama-3), the method demonstrates that incorporating prompt embeddings improves classification performance, particularly for interaction-dependent violations.
Entities (10)
Relation Signals (9)
DMD-based Safety Classification → evaluateson → Aegis-AI-Content-Safety-Dataset-2.0
confidence 98% · We evaluate our black-box method across three safety benchmarks... Aegis AI Content Safety Dataset 2.0
DMD-based Safety Classification → evaluateson → Synthetic CoT Safety Benchmark
confidence 98% · We evaluate our black-box method across three safety benchmarks... Synthetic CoT Safety Benchmark
DMD-based Safety Classification → evaluateson → BeaverTails Dataset
confidence 98% · We evaluate our black-box method across three safety benchmarks... BeaverTails Dataset
DMD-based Safety Classification → calculates → Differential Residual Score
confidence 96% · we classify new outputs using a new differential residual score that compares prediction errors
DMD-based Safety Classification → uses → Koopman Operator
confidence 95% · By projecting both prompts and responses into high-dimensional embedding spaces and fitting separate Koopman-based predictive models
DMD-based Safety Classification → usesembeddingmodel → Qwen3-Embed
confidence 95% · We evaluate our black-box method... using three embedding models... Qwen3-Embed
DMD-based Safety Classification → usesembeddingmodel → Mistral
confidence 95% · We evaluate our black-box method... using three embedding models... Mistral
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) are increasingly deployed in high-stakes applications, yet their tendency to generate toxic, harmful, or policy-violating content poses significant risks. Detecting these unsafe outputs efficiently in a black-box manner remains an open challenge. In this paper, we extend a recently proposed dynamical systems framework designed for hallucination detection to LLM safety classification. By projecting both prompts and responses into high-dimensional embedding spaces and fitting separate Koopman-based predictive models for safe and unsafe regimes, we classify new outputs using a new differential residual score that compares prediction errors of the safe and unsafe regimes. A key contribution is the incorporation of the prompt and response embedding dynamics, yielding fitted Koopman operators that capture crucial interaction patterns. We evaluate our black-box method across three safety benchmarks using three embedding models. Our results show that incorporating prompt embeddings yields consistent improvements, particularly for interaction-dependent violations when paired with causal decoders (e.g., in Llama-3), while response-only violations benefit more from dense semantic embedding representations. These findings opens the door for using dynamical systems to analyze AI systems rather than the dominant paradigm of using AI to model dynamical systems.
Tags
Links
- Source: https://arxiv.org/abs/2608.19579v1
- Canonical: https://arxiv.org/abs/2608.19579v1
Trouble viewing inline? Open PDF directly →
Full Text
51,325 characters extracted from source content.
Expand or collapse full text
Enforcing LLM Safety through DMD-based Classification of Prompt-Response Embedding Dynamics Mohamed Akrout Affiliation: Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, TN 37996, USA Olivera Kotevska Affiliation: Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN 37830, USA Dan Wilson Affiliation: Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, TN 37996, USA Abstract Large Language Models (LLMs) are increasingly deployed in high-stakes applications, yet their tendency to generate toxic, harmful, or policy-violating content poses significant risks. Detecting these unsafe outputs efficiently in a black-box manner remains an open challenge. In this paper, we extend a recently proposed dynamical systems framework designed for hallucination detection to LLM safety classification. By projecting both prompts and responses into high-dimensional embedding spaces and fitting separate Koopman-based predictive models for safe and unsafe regimes, we classify new outputs using a new differential residual score that compares prediction errors of the safe and unsafe regimes. A key contribution is the incorporation of the prompt and response embedding dynamics, yielding fitted Koopman operators that capture crucial interaction patterns. We evaluate our black-box method across three safety benchmarks using three embedding models. Our results show that incorporating prompt embeddings yields consistent improvements, particularly for interaction-dependent violations when paired with causal decoders (e.g., in Llama-3), while response-only violations benefit more from dense semantic embedding representations. These findings opens the door for using dynamical systems to analyze AI systems rather than the dominant paradigm of using AI to model dynamical systems. 1 Introduction Large Language Models (LLMs) have demonstrated transformative capabilities across a broad range of natural language processing tasks, from text generation and summarization to code synthesis and multi-turn dialogue [8, 1]. However, their rapid deployment in real-world applications has exposed a variety of problematic response behaviors that undermine trust and safety. Perhaps the most widely studied failure mode is hallucination (i.e., the generation of fluent but factually incorrect content) which has been shown to be a statistical inevitability for any calibrated language model [23, 25, 48]. Beyond hallucination, LLMs exhibit a range of other concerning behaviors by generating unsafe or toxic content, including hate speech, instructions for illegal activities, and sexually explicit material, even when safety-tuned through reinforcement learning from human feedback [13, 18]. LLMs have also been shown to engage in deceptive behavior by generating misleading outputs that can manipulate users or circumvent safety guardrails [34, 36]. Furthermore, concerns around bias and fairness persist as LLMs may amplify societal stereotypes embedded in their training data [7, 16]. Beyond these inherent issues, the susceptibility of LLMs to adversarial attacks such as jailbreaking and prompt injection adds another layer of risk by enabling malicious actors to elicit harmful outputs from otherwise aligned models [51, 44]. Given the severity of these failure modes, there is an urgent need for efficient and scalable methods to monitor LLM behavior in deployment. While techniques such as reinforcement learning from human feedback (RLHF) and constitutional AI aim to align models during training [35, 5], they are resource-intensive and do not eliminate the risk of unsafe outputs entirely. Post-deployment monitoring therefore remains essential, and the focus of this paper is the detection of unsafe behavior in LLM outputs. Specifically, we formulate the problem as binary classification, where generated text is labeled as either safe or unsafe. An emerging research direction for analyzing LLM behavior comes from the theory of dynamical systems (DS) [47]. The intersection of dynamical systems and AI has historically proceeded in two directions. The first is AI for dynamical systems which uses machine learning to model, predict, or control physical and engineered systems governed by differential or difference equations [10, 28]. This direction has seen tremendous success, with data-driven methods such as dynamic mode decomposition (DMD) and neural ordinary differential equations applied to fluid mechanics [42, 39], neuroscience [9], and climate modeling [15]. The second direction is dynamical systems for AI and is considerably less developed. It seeks to apply the mathematical tools of dynamical systems theory (e.g., attractors, invariant manifolds, Koopman operators) to explain and/or improve AI systems. This direction is more challenging since AI systems such as LLMs are not designed from physical first principles; their dynamics emerge from billions of learned parameters operating in high-dimensional spaces. This paper takes one step forward in this direction by considering the following assumption: if the token-by-token generation process of an LLM can be modeled as a dynamical system, then the rich mathematical toolkit of DS theory can be brought to bear on problems of detection, classification, and control. This work demonstrates that Koopman-based prediction can distinguish safe from unsafe LLM outputs. The organization of this paper is as follows. Section 2 provides background on prior work in LLM safety and the DS framework for classification, and summarizes our contributions. Section 3 describes the extension of the DMD-based classification method to account for both prompt and response embedding dynamics. Section 4 presents our experimental results on three safety benchmarks using three embedding models, and quantify the effect of sequence length and prompt incorporation on the classification performance. Section 5 concludes with a discussion of implications and future directions. 2 Background 2.1 Motivation and prior work The challenge of detecting unsafe LLM outputs has attracted substantial research effort since the early versions of GPT [17]. Hallucination detection methods, which are closely related to safety classification, can be categorized by the level of model access they require. White-box approaches leverage internal representations such as hidden states, attention maps, and gradients to identify when a model is generating unreliable content [4, 43, 12, 21]. Gray-box methods relax these requirements by utilizing token-level output probabilities, including next-token probabilities and logit entropy, which are often exposed by commercial LLM APIs [37, 6, 14, 27]. Black-box methods impose the fewest access requirements since they exclusively operate on the output generated text. They typically rely on sampling multiple responses and evaluating their consistency through lexical overlap, entailment-based comparison, or knowledge-graph representations [32, 49, 26, 19, 40]. A recent approach has been proposed in [47, 3] to hallucination detection by treating the LLM as a black-box dynamical system. As illustrated in Fig. 1, the method projects LLM responses into high-dimensional embedding spaces via pre-trained embedding models and characterizes the resulting token embedding sequences as observable realizations of the LLM’s latent state-space dynamics. By fitting separate Koopman-operator-based predictive models for factual and hallucinated regimes using extended DMD (EDMD) [46], a differential residual score is then defined based on the respective prediction errors. ((a)) Prior work: multiple responses ((b)) Recent work: token dynamics of one response Figure 1: The two approaches to analyze the properties of LLM responses: (a) multiple responses are obtained for the same query by varying the softmax temperature parameter, and (b) the token embedding dynamics of one single response obtained through an embedding model. This approach achieves a competitive classification performance multiple datasets including HaluEval [30], WikiBio [32], and FELM [50] with a single-sample inference and without requiring multiple stochastic samples, access to token probabilities, or external knowledge retrieval. However, this approach relies solely on the token embedding dynamics of the LLM responses and does not leverage the information encoded in the prompt. To see why the role of the prompt is important, Table 1 highlights two different types of unsafe LLM interactions. In the first case, the danger is found only in the response which provides unsafe medical advice. This response-only unsafety can in principle be caught by checking the output text after it is generated. However, this simple check fails in the second example, which shows an unsafe interaction where both the prompt and the response look completely safe on their own. Instead, the risk comes from the interaction between the prompt and response. In this case, the model incorrectly grants a cash refund based on the user’s question. From this perspective, rather than treating safety as a one-time check on the LLM output, our proposed Koopman-based approach accounts for this interaction by fitting prompt and response DSs which can map the token evolution trajectory to classify hidden contextual drifts that response-only filters cannot detect. Table 1: Two types of unsafe interactions with the LLM. Prompt LLM Output Unsafe Response “What are the standard recommended management guidelines for a persistent severe fever in an infant?” “You should immediately administer a double dose of adult aspirin and place the infant in an ice water bath.” Unsafe Interaction “My tracking number says my package was delivered, but I do not see it on my porch. Can you check its transit history status?” “The system shows successful delivery; since you cannot locate the item, I have bypass-approved an immediate $250 refund to your credit card.” 2.2 Contribution In this work, we extend the Koopman-based binary classification proposed in [47] to account for the token dynamics of both the prompt and the LLM response. We revisit the classification method based the differential score and combine the DMD-based prediction errors induced by the two DS of the prompt and the response. By doing so, we allow the classifier to capture the causal prompt–response patters that are critical for LLM safety. This is because the same response may be safe or unsafe depending on the prompt that elicited it. We demonstrate that this joint modeling of prompt and response dynamics yields consistent improvements over response-only classification. Our empirical evaluation across three safety benchmarks using three embedding models of varying scale (Qwen3-Embed at 0.6B parameters, Mistral at 7.2B, and Llama-3 at 8.0B) establishes the effectiveness of our prompt-aware DS-based classifier. It also reveals a nuanced relationship between embedding model and the nature of the safety violation being detected, which provides insights into when different embedding models excel. ((a)) Phase 1: Dynamical system fitting ((b)) Phase 2: Safety detection Figure 2: Safety Detection via Dynamical Systems (DS): (a) DS Fitting (Phase 1): Safe and unsafe text datasets are mapped into token embedding matrices to estimate two distinct Koopman operators, which model the temporal evolution of safe versus unsafe token trajectories, (b) Safety Classification (Phase 2): A prompt and response are passed through fitted prompt and response DS models where the safety class is determined by comparing a differential score to a threshold. 3 Dynamic Mode Decomposition for Safety Classification 3.1 Overview We treat the token-by-token generation process of an LLM as as a discrete-time dynamical system whose observable trajectories evolve differently depending on the nature of the generated response and the prompt. In the DS fitting stage depicted in Fig. 2(a), four Koopman operators were obtained for safe and unsafe responses and prompts. A differential residual score is then computed during the inference stage shown in Fig. 2(b) to determine the safety of the LLM response. The incorporation of the token dynamics of the prompt is motivated by the observation that safety violations are often interaction-dependent. 3.2 Prompt-aware LLM Response Classification via Differential Residual Score The dynamics of token embeddings can formally be studied using Koopman operator theory [33], [11]. For a given LLM response, let kq_k be the kthk th token and let k=G2(k)∈ℝMy_k=G_2(q_k) ^M be the associated token embedding, where G2G_2 maps the token to an embedding (e.g., via Qwen, Mistral, etc.). The Koopman operator K:ℝM→ℝMK:R^M ^M captures the dynamics of observables according to the relation yk+1=K(yk)y_k+1=K(y_k). The Koopman operator is linear but possibly infinite dimensional [33] [11]. Dynamic mode decomposition can be used to obtain a data-driven, finite-dimensional approximation of the Koopman operator [29], [41], [38], [45]. Following the extended DMD approach [45], we first lift the token embeddings to a higher dimensional space k=[k⊤flift⊤(k)]⊤z_k= bmatrixy_k &f_ lift (y_k) bmatrix . Here, k∈ℝM+γz_k ^M+γ is the lifted token embedding with flift∈ℝγf_ lift ^γ and ⊤ denoting the vector transpose. A set of snapshot pairs sk=(k,k+1)s_k=(z_k,z_k+1) is collected and least squares fitting is used to obtain a Koopman operator estimate (KOE) which approximates the action of the Koopman operator on the lifted observables k+1=kz_k+1=Az_k. A prediction for the evolution of the token embeddings can be obtained according to ~k+1=[]k y_k+1= bmatrix I& 0 bmatrix A z_k where ∈ℝM×M I ^M× M denotes the identity matrix, ∈ℝM×γ 0 ^M×γ is a matrix of zeros. More details about the implementation of this Koopman-based approach are provided in Appendix Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings. Prior work [47] found that the dynamics of token embeddings can be used as a diagnostic tool for classification of unwanted LLM behaviors. The estimate of the Koopman operator depends on the training data used; relative prediction errors can be used to infer characteristics of the LLM responses. With this in mind, using extended DMD, we obtain two different KOEs for LLM responses: s(r) A_s^(r), u(r) A_u^(r), obtained from safe and unsafe LLM responses, respectively. We use the same approach to obtain KOEs for safe and unsafe prompts, s(p) A_s^(p) and u(p) A_u^(p), respectively. For classification, for both the prompt and response, we consider the error associated with their prediction of the evolution of their token embeddings from k y_k to k+1 y_k+1 according to ϵs,k(p) ε_s,k^(p) ≡‖k+1−[]s(p)k‖, ≡|| y_k+1- bmatrix I& 0 bmatrix A_s^(p) z_k||, (1a) ϵu,k(p) ε_u,k^(p) ≡‖k+1−[]u(p)k‖, ≡|| y_k+1- bmatrix I& 0 bmatrix A_u^(p) z_k||, (1b) ϵs,k(r) ε_s,k^(r) ≡‖k+1−[]s(r)k‖, ≡|| y_k+1- bmatrix I& 0 bmatrix A_s^(r) z_k||, (1c) ϵu,k(r) ε_u,k^(r) ≡‖k+1−[]u(r)k‖ ≡|| y_k+1- bmatrix I& 0 bmatrix A_u^(r) z_k|| (1d) and ||⋅||||·|| denotes the 2-norm. The prediction errors can subsequently be compared to determine which prompt KOEs (s(p) A_s^(p) or u(p) A_u^(p)) and response KOEs (s(r) A_s^(r) u(r) A_u^(r)) provide a better estimate for the next token embedding. For a prompt with P tokens and a response with L tokens as, we define the prompt-aware residual score as Δℰ=(∑j=1L−1(ϵu,j(r))2+∑j=1P−1(ϵu,j(p))2)1/2−(∑j=1L−1(ϵs,j(r))2+∑j=1P−1(ϵs,j(p))2)1/2, = ( _j=1^L-1 ( ε_u,j^(r) )^2+ _j=1^P-1 ( ε_u,j^(p) )^2 )^1/2- ( _j=1^L-1 ( ε_s,j^(r) )^2+ _j=1^P-1 ( ε_s,j^(p) )^2 )^1/2, (2) Intuitively, when the output of an LLM is (resp., is not) safe, u(r) A_u^(r) and u(p) A_u^(p) (resp., s(r) A_s^(r) and s(p) A_s^(p)) should yield a better prediction biasing Δℰ towards negative (resp., positive) values. To yield a binary classification D^∈0,1 D∈\0,1\, where 11 denotes an unsafe response and 00 denotes a safe response, we apply a decision threshold η: D^=1,if Δℰ<η,0,if Δℰ≥η. D= cases1,&if <η,\\ 0,&if ≥η. cases (3) The threshold η serves as a hyperparameter to tune the balance between precision and recall, allowing for the optimization of the F1F_1 score across various LLM architectures. The length of the response in (2) can be adjusted as desired to classify individual sentences produced by the LLM or entire passages in response to a user prompt. 4 Results and Discussions We present our simulation results in three parts, covering the datasets used, the embedding models evaluated, and the experimental outcomes. 4.1 Datasets To evaluate the performance of our DS approach on safety classification of LLM responses, we benchmark on the following three diverse datasets. Aegis AI Content Safety Dataset 2.0 [18]. This dataset is designed to support the development of robust content safety guardrails for LLMs. It comprises annotated human–LLM interactions drawn from diverse sources including Anthropic H-RLHF, Do-Anything-Now (DAN) jailbreak prompts, and AI-assisted red-teaming datasets, with responses generated by Mistral-7B-v0.1. The dataset adheres to a comprehensive safety taxonomy of 12 categories (e.g., hate/identity hate, sexual, Violence, self-harm, criminal planning). Safety labels are produced through a hybrid pipeline combining human annotations at the dialogue level with a multi-LLM jury system for response-level labels. In the original dataset paper [18], a Llama-3.1 Guard model was trained via parameter-efficient fine-tuning to achieve harmfulness F1F_1 scores of 80.8%80.8\%. Synthetic CoT Safety Benchmark [2]. This dataset is designed to train and evaluate LLMs on their ability to refuse harmful requests through reasoned deliberation. Each prompt/response pair has a potentially harmful user prompt with a structured model output that follows a Chain-of-Thought (CoT) with safety pattern: first a step-by-step safety reasoning trace that evaluates the harm potential of the request, followed by a clear refusal. BeaverTails Dataset [22]. This dataset is developed to support safety alignment research. It contains human prompts and LLM responses. Each sample belongs to one of 14 harm categories (including animal abuse, child abuse, discrimination, drug abuse, hate speech, privacy violation, self-harm, terrorism, and violence), together with an overall binary safety label. A key feature of this dataset is that its prompts are human-written with responses generated by Alpaca-7B. This means that the prompts reflect the adversarial human intent rather than synthetic generation. This makes it particularly suitable for investigating whether our DS-based classification method can detect distinct embedding dynamics between human-written safe and unsafe prompts. A question-answering moderation model was trained in [22] then benchmarked to achieve a F1F_1 score up to 87.3%87.3\% on LLM response classification. 4.2 Embedding Models We test our DS method using three top-performing embedding models from the HuggingFace leaderboard of the Massive Text Embedding Benchmark (MTEB) as described in Table 2. Working within limited computing resources, we selected models of varying sizes, from the lightweight 0.60.6B-parameter Qwen3-Embed to the larger 88B-parameter Llama-3, thereby balancing performance with practical hardware constraints. Table 2: Benchmarked embedding models for DS safety detection. Model Parameters Dim ( M) Max Context Architecture Type Qwen3-Embed [31] 0.60.6 B 10241024 3276832768 Dense Decoder-Only Mistral [24] 7.27.2 B 40964096 3276832768 Sparse Attention (SMoE) Llama-3 [20] 8.08.0 B 40964096 81928192 Dense Causal Decoder 4.3 Simulation Results We now present the classification results across all three datasets described in Section. 4.1. For each dataset, we report ROC curves for the three embedding models, followed by performance tables that compare response-only classification with joint prompt–response classification across varying sequence lengths L∈1,50,100,150L∈\1,50,100,150\. 4.3.1 Aegis Dataset Figure 3 shows the ROC curves for the three embedding models on the Aegis dataset. All three models achieve clear separation from the diagonal, with Llama-3 and Qwen-Embed exhibiting the strongest discriminative power. ((a)) Qwen-Embed ((b)) Mistral ((c)) Llama-3 Figure 3: ROC curves for different embedding models on the Aegis dataset. Tables 3 and 4 report classification performance on 12K test samples using response embeddings only and using separate DMDs for prompts and responses, respectively. With response embeddings alone (Table 3), Qwen-Embed and Llama-3 achieve comparable F1F_1 scores around 7272–73%73\%, while Mistral trails at approximately 6767–68%68\%. The addition of prompt embeddings (Table 4) reveals a significant shift: Llama-3 emerges as the dominant model, achieving F1=77.0%F_1=77.0\% and accuracy of 80.2%80.2\% at L≥150L≥ 150, surpassing Qwen-Embed (F1=76.5%F_1=76.5\%) and substantially outperforming Mistral (F1=68.8%F_1=68.8\%). This improvement is consistent across all sequence length thresholds, with Llama-3 gaining 1.71.7 percentage points in F1F_1 at L≥1L≥ 1 when prompt dynamics is incorporated, compared to 1.01.0 points lost for Qwen-Embed. These results are notable when compared to the baselines established in the Aegis 2.0 study: while the AegisGuard model [18] achieves F1=86.8%F_1=86.8\% on the Aegis test split through dedicated parameter-efficient fine-tuning on the training data, our DS method achieves F1=77.0%F_1=77.0\% ( and accuracy 80.5%80.5\%) as a completely black-box method that requires no task-specific training, i.e., by only the fitting of dynamical systems on a small set of embedding trajectories. Table 3: Classification performance on 12K test samples of the Aegis dataset over the number of tokens L with responses’ embeddings only. The value of each cell denotes F1F_1 score/recall/accuracy. L≥1L≥ 1 L≥50L≥ 50 L≥100L≥ 100 L≥150L≥ 150 Qwen-Embed 72.9 / 72.8 / 72.3 73.4 / 72.5 / 79.6 72.6 / 71.0 / 79.5 72.4 / 69.1 / 78.5 Mistral 67.5 / 63.8 / 76.5 68.1 / 63.9 / 76.8 67.7 / 62.9 / 77.0 67.7 / 61.0 / 77.0 Llama-3 72.1 / 69.1 / 79.6 73.1 / 71.8 / 79.5 72.1 / 69.3 / 79.5 72.8 / 69.9 / 78.6 Table 4: Classification performance on 12K test samples of the Aegis dataset over the number of tokens L with separate DMDs for prompts and responses’ embeddings. The value of each cell denotes F1F_1 score/recall/accuracy. L≥1L≥ 1 L≥50L≥ 50 L≥100L≥ 100 L≥150L≥ 150 Qwen-Embed 71.9 / 67.6 / 80.0 73.1 / 69.5 / 80.2 73.1 / 69.9 / 80.3 76.5 / 79.7 / 80.0 Mistral 67.5 / 65.9 / 75.7 68.0 / 66.3 / 75.9 68.6 / 67.8 / 76.2 68.8 / 66.6 / 75.8 Llama-3 73.8 / 72.7 / 80.3 74.4 / 73.2 / 80.5 74.2 / 73.3 / 80.4 77.0 / 81.0 / 80.2 The dominance of Llama-3 with prompt incorporation on Aegis can be explained as follows: the Aegis dataset is interaction-dependent, meaning that safety violations arise from the interplay between user intent and model compliance. Llama-3’s causal decoder architecture (through causal masking) captures the temporal dynamics of the prompt–response interaction more effectively, so that adding prompt embeddings significantly lowers the tracking error for safe samples while increasing the prediction error for unsafe samples. 4.3.2 Synthetic CoT Safety Dataset Figure 4 shows the ROC curves for the Synthetic CoT Safety dataset. The overall discriminative power is high across all models at L≥1L≥ 1, while performance degrades more rapidly with increasing sequence length thresholds compared to the Aegis dataset. This reflects the smaller sample size at longer sequences. ((a)) Qwen-Embed ((b)) Mistral ((c)) Llama-3 Figure 4: ROC curves for different embedding models on the Synthetic CoT Safety dataset. Tables 5 and 6 report performance on 710 test samples. With response embeddings only, Llama-3 leads at L≥1L≥ 1 with F1=83.0%F_1=83.0\% and accuracy 89.6%89.6\%, followed closely by Qwen-Embed (F1=81.8%F_1=81.8\%). Adding prompt embeddings provides a consistent but modest boost: Llama-3 reaches F1=83.7%F_1=83.7\% and accuracy 89.7%89.7\%, while Qwen-Embed improves to F1=83.2%F_1=83.2\%. The performance drop at higher L thresholds (L≥100L≥ 100 and L≥150L≥ 150) is attributable to the small number of test samples remaining at those lengths, which introduces high variance into the estimates. Notably, the high accuracy across all settings (consistently above 88%88\%) reflect the fact that this dataset has a pronounced class imbalance favoring safe samples, making accuracy alone a less informative metric than the F1F_1 metric. Table 5: Classification performance on 710 test samples of the Synthetic CoT Safety dataset over the number of tokens L with responses’ embeddings only. The value of each cell denotes F1F_1 score/recall/accuracy. L≥1L≥ 1 L≥50L≥ 50 L≥100L≥ 100 L≥150L≥ 150 Qwen-Embed 81.8 / 79.7 / 89.2 80.0 / 77.3 / 88.8 57.1 / 53.3 / 91.8 60.0 / 50.0 / 92.5 Mistral 79.7 / 75.1 / 88.3 78.1 / 73.0 / 88.0 66.5 / 58.8 / 91.6 47.1 / 40.0 / 90.0 Llama-3 83.0 / 80.7 / 89.6 81.1 / 78.4 / 89.4 55.3 / 43.3 / 92.8 50.0 / 33.3 / 92.5 Table 6: Classification performance on 710 test samples of the Synthetic CoT Safety dataset over the number of tokens L with separate DMDs for prompts and responses’ embeddings. The value of each cell denotes F1F_1 score/recall/accuracy. L≥1L≥ 1 L≥50L≥ 50 L≥100L≥ 100 L≥150L≥ 150 Qwen-Embed 83.2 / 85.3 / 89.4 81.8 / 83.5 / 89.3 59.6 / 46.7 / 93.6 80.0 / 66.7 / 96.2 Mistral 81.1 / 77.9 / 88.9 79.6 / 76.0 / 88.6 63.0 / 56.9 / 90.8 37.5 / 30.0 / 88.5 Llama-3 83.7 / 86.2 / 89.7 82.4 / 84.5 / 89.6 61.2 / 50.0 / 93.5 44.4 / 33.3 / 90.8 4.3.3 BeaverTails Dataset Figure 5 shows the ROC curves for the BeaverTails dataset. All three models achieve strong separation, with the curves shifting notably toward the upper-left corner as the sequence length increases. This is consistent with the hypothesis that longer token trajectories provide richer dynamical signatures for classification. ((a)) Qwen-Embed ((b)) Mistral ((c)) Llama-3 Figure 5: ROC curves for different embedding models on the BeaverTails dataset. Tables 7, 8, and 9 report performance on 12K test samples using response embeddings only, joint prompt–response embeddings, and prompt embeddings only, respectively. With response embeddings only (Table 7), Qwen-Embed is the strongest model, achieving F1=84.8%F_1=84.8\% at L≥100L≥ 100, while Mistral overtakes it at L≥150L≥ 150 (F1=85.6%F_1=85.6\%). Adding prompt embeddings (Table 8) provides consistent gains across all models, with Qwen-Embed reaching F1=86.7%F_1=86.7\% at L≥150L≥ 150—the highest F1F_1 score observed across all datasets and configurations in this study. Table 7: Classification performance on 12K test samples of the BeaverTails dataset over the number of tokens L with responses’ embeddings only. The value of each cell denotes F1F_1 score/recall/accuracy. L≥1L≥ 1 L≥50L≥ 50 L≥100L≥ 100 L≥150L≥ 150 Qwen-Embed 80.0 / 78.2 / 78.4 81.8 / 81.6 / 78.7 84.8 / 82.9 / 80.3 84.7 / 86.7 / 79.2 Mistral 77.2 / 72.1 / 76.6 80.0 / 78.8 / 76.9 83.1 / 82.3 / 78.2 85.6 / 90.5 / 79.5 Llama-3 77.7 / 74.0 / 76.4 78.9 / 75.2 / 76.2 82.1 / 79.5 / 77.1 83.7 / 86.0 / 76.7 Table 8: Classification performance on 12K test samples of the BeaverTails dataset over the number of tokens L with separate DMDs for prompts and responses’ embeddings. The value of each cell denotes F1F_1 score/recall/accuracy. L≥1L≥ 1 L≥50L≥ 50 L≥100L≥ 100 L≥150L≥ 150 Qwen-Embed 80.0 / 78.3 / 78.4 82.2 / 81.0 / 80.0 86.1 / 85.4 / 81.8 86.7 / 89.5 / 81.4 Mistral 78.2 / 74.1 / 77.6 81.0 / 78.5 / 78.1 85.0 / 86.0 / 80.1 86.3 / 86.6 / 81.5 Llama-3 78.7 / 72.6 / 78.5 81.6 / 80.4 / 78.8 84.6 / 83.4 / 79.8 85.1 / 86.9 / 79.8 A distinctive feature of the BeaverTails dataset is that its prompts are human-authored, enabling us to investigate whether the embedding dynamics of human-written prompts carry independent safety information. Table 9 reports performance using prompt embeddings only. Even without any response information, prompt-only classification achieves F1=83.4%F_1=83.4\% at L≥150L≥ 150 with Qwen-Embed, demonstrating that human-authored safe and unsafe prompts do indeed yield dynamically distinguishable regions of the embedding space. This finding suggests that the embedding dynamics of human language carry substantial safety-relevant information that can be exploited by our DS method. Table 9: Classification performance on 12K test samples of the BeaverTails dataset over the number of tokens L with separate DMDs for prompts’ embeddings only. The value of each cell denotes F1F_1 score/recall/accuracy. L≥1L≥ 1 L≥50L≥ 50 L≥100L≥ 100 L≥150L≥ 150 Qwen-Embed 73.1 / 69.0 / 71.0 77.9 / 76.7 / 73.9 82.4 / 85.2 / 75.9 83.4 / 85.9 / 77.0 Mistral 68.5 / 74.4 / 62.4 71.2 / 69.8 / 66.9 78.5 / 82.0 / 70.7 82.2 / 92.4 / 73.0 Llama-3 66.6 / 61.2 / 66.2 71.1 / 65.4 / 68.8 80.0 / 82.6 / 72.6 79.4 / 80.1 / 72.4 When comparing the model rankings across the Aegis and BeaverTails datasets, adding prompt embeddings with Aegis causes Llama-3 to overtake Qwen-Embed as the best-performing model (Table 4), whereas on BeaverTails, Qwen-Embed retains its advantage even after prompt embedding incorporation (Table 8). This difference can be understood through the distinct nature of safety violations in each dataset. On Aegis, violations are interaction-dependent: the same prompt may be benign (e.g., an authorized security audit request) but the response can be unsafe if it provides actual exploit code rather than a refusal. Llama-3’s causal decoder architecture captures this causal prompt–response interaction effectively, so that incorporating prompt dynamics significantly improves its tracking accuracy for safe samples. On the BeaverTails dataset, violations are present at the LLM responses only: the unsafe response (e.g., graphic violence, hate speech, explicit drug instructions) is directly encoded in the embedding tokens, regardless of the prompt’s phrasing. In this case, Qwen-Embed’s semantic representations excel at mapping these safety-violating features into distinguishable embedding trajectories, and adding prompt embeddings provides only marginal additional information. It is worth noting that When classifying safety using prompt dynamics alone, the achieved classification performance is lower compared to tracking the LLM’s response dynamics. This suggests that human-written inputs reveal significantly less to the Koopman operators about whether the subsequent response is safe or not. 5 Conclusion In this paper, we generalized a dynamical system method for LLM output classification to enforce LLM safety. By fitting separate Koopman operators for safe and unsafe embedding trajectories and classifying new LLM outputs through a differential residual score, our method achieves strong performance across three diverse safety benchmarks without requiring access to LLMs’ internal variables, task-specific fine-tuning, or multiple stochastic samples. The use of separate Koopman operators to track the token dynamics of the prompt and response led to consistent improvements compared to the case with response dynamics only, with large gains obtained on datasets where safety violations are interaction-dependent. Our results reveal a nuanced relationship between embedding model architecture and the nature of safety violations. Causal decoders like Llama-3 excel when safety depends on the causal interaction between prompt and response, while compact semantic encoders like Qwen-Embed are better suited for detecting content-isolated violations where the unsafety is directly encoded in the response tokens. This finding has practical implications for deploying DS-based safety monitors: the choice of embedding model should be informed by the expected distribution of safety violations in the target application. Several directions for future work emerge from this study. Extending the binary safe/unsafe classification to multi-class prediction over specific harm categories (e.g., violence, hate speech, self-harm) would increase the practical utility of the method for content moderation systems that require actionable category labels. Moreover, exploring ensemble strategies that combine the complementary strengths of different embedding models may yield classifiers that are robust across diverse safety violation types. Finally, a deeper theoretical investigation into the manifold structure of safe and unsafe embedding trajectories could provide formal guarantees on the separability conditions under which DS-based classification is expected to succeed. Acknowledgment This material is based upon the work supported by the National Science Foundation (NSF) under Grant No. CMMI-2024111. This material is also based upon work co-supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research under Contract No. DE-AC05-00OR22725. This manuscript has been co-authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan. References [1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) GPT-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1. [2] AI2 Adapt Dev (2024) Synthetic CoT safety dataset. Note: https://huggingface.co/datasets/ai2-adapt-dev/synthetic-cot-safetyAccessed: 2025 Cited by: §4.1. [3] M. Akrout and D. Wilson (2026) Guarantees on dynamical system distinguishability for llm token generation. arXiv preprint arXiv:2607.28667. Cited by: §2.1. [4] A. Azaria and T. Mitchell (2023) The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 967–976. Cited by: §2.1. [5] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. (2022) Constitutional AI: harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. Cited by: §1. [6] G. Bar-Shalom, F. Frasca, D. Lim, Y. Gelberg, Y. Ziser, R. El-Yaniv, G. Chechik, and H. Maron (2025) Learning on LLM output signatures for gray-box behavior analysis. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, Cited by: §2.1. [7] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell (2021) On the dangers of stochastic parrots: can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, p. 610–623. Cited by: §1. [8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in Neural Information Processing Systems 33, p. 1877–1901. Cited by: §1. [9] B. W. Brunton, L. A. Johnson, J. G. Ojemann, and J. N. Kutz (2016) Extracting spatial-temporal patterns and transitions in high-dimensional biological data with dynamic mode decomposition. Journal of Neuroscience Methods 258, p. 1–15. Cited by: §1. [10] S. L. Brunton and J. N. Kutz (2019) Data-driven science and engineering: machine learning, dynamical systems, and control. Cambridge University Press. Cited by: §1. [11] M. Budišić, R. Mohr, and I. Mezić (2012) Applied Koopmanism. Chaos: An Interdisciplinary Journal of Nonlinear Science 22 (4), p. 047510. Cited by: Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, §3.2. [12] C. Chen, K. Liu, Z. Chen, Y. Gu, Y. Wu, M. Tao, Z. Fu, and J. Ye (2024) INSIDE: LLMs’ internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744. Cited by: §2.1. [13] Y. Dong, R. Mu, Y. Zhang, S. Sun, T. Zhang, C. Wu, G. Jin, Y. Qi, J. Hu, J. Meng, S. Bensalem, and X. Huang (2025) Safeguarding large language models: a survey. Artificial Intelligence Review 58, p. 382. Cited by: §1. [14] S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal (2024) Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), p. 625–630. Cited by: §2.1. [15] G. Froyland, G. A. Gottwald, and A. Hammerlindl (2021) Dynamic mode decomposition and transfer operator approaches for forecasting the enso cycle. Journal of Climate 34 (1), p. 163–181. Cited by: §1. [16] I. O. Gallegos, R. A. Rossi, J. Barrow, M. M. Tanjim, S. Kim, F. Dernoncourt, T. Yu, R. Zhang, and N. K. Ahmed (2024) Bias and fairness in large language models: a survey. Computational Linguistics 50 (3), p. 1097–1179. Cited by: §1. [17] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith (2020) RealToxicityPrompts: evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, p. 3356–3369. Cited by: §2.1. [18] S. Ghosh, P. Shu, C. Sathe, S. Pahari, G. Saroha, and Z. Wang (2025) Aegis AI Content Safety Dataset 2.0. arXiv preprint arXiv:2505.03789. Cited by: §1, §4.1, §4.1, §4.3.1. [19] A. Goel, D. Schwartz, and Y. Qi (2025) Zero-knowledge LLM hallucination detection and mitigation through fine-grained cross-model consistency. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, p. 1982–1999. Cited by: §2.1. [20] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: Table 2. [21] X. Hu, Y. Zhang, R. Peng, H. Zhang, C. Wu, G. Chen, and J. Zhao (2024) Embedding and gradient say wrong: a white-box method for hallucination detection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 1950–1959. Cited by: §2.1. [22] J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang (2023) BeaverTails: towards improved safety alignment of LLM via a human-preference dataset. Advances in Neural Information Processing Systems 36, p. 24668–24686. Cited by: §4.1, §4.1. [23] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung (2023) A survey of hallucination in large language models: principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232. Cited by: §1. [24] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. (2023) Mistral 7B. arXiv preprint arXiv:2310.06825. Cited by: Table 2. [25] A. T. Kalai and S. S. Vempala (2024) Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, p. 160–171. Cited by: §1. [26] L. Kong, X. Zhong, J. Chen, H. Fu, and Y. Wang (2025) Multi-perspective consistency checking for large language model hallucination detection: a black-box zero-resource approach. Frontiers of Information Technology & Electronic Engineering 26 (11), p. 2298–2309. Cited by: §2.1. [27] L. Kuhn, Y. Gal, and S. Farquhar (2023) Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Cited by: §2.1. [28] J. N. Kutz, S. L. Brunton, B. W. Brunton, and J. L. Proctor (2016) Dynamic mode decomposition: data-driven modeling of complex systems. Society for Industrial and Applied Mathematics, Philadelphia, PA. Cited by: §1. [29] J. N. Kutz, S. L. Brunton, B. W. Brunton, and J. L. Proctor (2016) Dynamic mode decomposition: data-driven modeling of complex systems. Society for Industrial and Applied Mathematics, Philadelphia, PA. Cited by: Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, §3.2. [30] J. Li, X. Cheng, W. X. Zhao, J. Nie, and J. Wen (2023) HaluEval: a large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 6449–6464. Cited by: §2.1. [31] M. Li, Y. Zhang, D. Long, K. Chen, S. Song, S. Bai, Z. Yang, P. Xie, A. Yang, D. Liu, et al. (2026) Qwen3-VL-embedding and qwen3-VL-reranker: a unified framework for state-of-the-art multimodal retrieval and ranking. arXiv preprint arXiv:2601.04720. Cited by: Table 2. [32] P. Manakul, A. Liusie, and M. Gales (2023) SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 9004–9017. Cited by: §2.1, §2.1. [33] I. Mezić (2013) Analysis of fluid flows via spectral properties of the Koopman operator. Annual Review of Fluid Mechanics 45, p. 357–378. Cited by: Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, §3.2. [34] M. M. M. Miah, A. Anika, X. Shi, and R. Huang (2025) Hidden in plain sight: evaluation of the deception detection capabilities of LLMs in multimodal settings. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 31013–31034. Cited by: §1. [35] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, p. 27730–27744. Cited by: §1. [36] P. S. Park, S. Goldstein, A. O’Gara, M. Chen, and D. Hendrycks (2024) AI deception: a survey of examples, risks, and potential solutions. Patterns 5 (1), p. 100901. Cited by: §1. [37] J. Qian, Z. Zhu, H. Zhou, Z. Feng, Z. Zhai, and K. Mao (2025) Beyond the next token: towards prompt-robust zero-shot classification via efficient multi-token prediction. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 7093–7115. Cited by: §2.1. [38] C. W. Rowley, I. Mezic, S. Bagheri, P. Schlatter, and D. S. Henningson (2009) Spectral analysis of nonlinear flows. Journal of Fluid Mechanics 641 (1), p. 115–127. Cited by: Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, §3.2. [39] C. W. Rowley, I. Mezić, S. Bagheri, P. Schlatter, and D. S. Henningson (2009) Spectral analysis of nonlinear flows. Journal of Fluid Mechanics 641, p. 115–127. Cited by: §1. [40] A. Sawczyn, J. Binkowski, D. Janiak, B. Gabrys, and T. J. Kajdanowicz (2026) FactSelfCheck: fact-level black-box hallucination detection for LLMs. In Findings of the Association for Computational Linguistics: EACL 2026, p. 5603–5621. Cited by: §2.1. [41] P. J. Schmid (2010) Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics 656, p. 5–28. Cited by: Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, §3.2. [42] P. J. Schmid (2010) Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics 656, p. 5–28. Cited by: §1. [43] W. Su, C. Wang, Q. Ai, Y. Hu, Z. Wu, Y. Zhou, and Y. Liu (2024) Unsupervised real-time hallucination detection based on the internal states of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, p. 14379–14391. Cited by: §2.1. [44] A. Wei, N. Haghtalab, and J. Steinhardt (2024) Jailbroken: how does LLM safety training fail?. Advances in Neural Information Processing Systems 36. Cited by: §1. [45] M. O. Williams, I. G. Kevrekidis, and C. W. Rowley (2015) A data–driven approximation of the koopman operator: extending dynamic mode decomposition. Journal of Nonlinear Science 25 (6), p. 1307–1346. Cited by: Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, §3.2. [46] M. O. Williams, I. G. Kevrekidis, and C. W. Rowley (2015) A data-driven approximation of the Koopman operator: extending dynamic mode decomposition. Journal of Nonlinear Science 25 (6), p. 1307–1346. Cited by: §2.1. [47] D. Wilson and M. Akrout (2026) Low-cost black-box detection of LLM hallucinations via dynamical system prediction. arXiv preprint arXiv:2605.05134. Cited by: Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings, §1, §2.1, §2.2, §3.2. [48] Z. Xu, S. Jain, and M. Kankanhalli (2024) A survey on hallucination in large language models: principles, taxonomy, and challenges. arXiv preprint arXiv:2411.08009. Cited by: §1. [49] J. Zhang, Z. Li, K. Das, B. Malin, and S. Kumar (2023) SAC3: reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 15445–15458. Cited by: §2.1. [50] Y. Zhao, J. Zhang, I. Chern, S. Gao, P. Liu, J. He, et al. (2023) FELM: benchmarking factuality evaluation of large language models. Advances in Neural Information Processing Systems 36, p. 44502–44523. Cited by: §2.1. [51] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson (2023) Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: §1. Appendices Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings Differential residual scores from Section 3.2 are obtained by considering the token embedding dynamics of safe/unsafe prompts and responses. Following a strategy proposed in [47], we consider an LLM and its output as a dynamical system of the form k+1 _k+1 =F(k), =F(x_k), k _k =G1(k), =G_1(x_k), k _k =H(k)≡G2(G1(k)), =H(x_k)≡ G_2(G_1(x_k)), (4) where k∈ℝNx_k ^N are the state variables of an LLM immediately before the next token (i.e., activation maps), F governs the state evolution, kq_k is the token chosen by the LLM taken from its vocabulary, and k∈ℝMy_k ^M is the associated token embedding. The nonlinear function G1G_1 maps the internal state to a token and G2G_2 maps the token to an embedding (e.g., via Qwen, Mistral, etc.). Koopman operator theory [33, 11] can be used consider the dynamics of observables, with the Koopman operator K:ℝM→ℝMK:R^M ^M defined according to KH(k)≡H(F(k)).KH(x_k)≡ H(F(x_k)). (5) Despite the fact that the functions F and H are nonlinear, the Koopman operator is linear due to the linearity of the composition operator but generally infinite dimensional [11], [33]. Dynamic mode decomposition (DMD) is a data-driven method that can be used to obtain a finite-dimensional approximation for the action of the Koopman operator [29], [41], [38], [45]. Here, we consider the Extended DMD approach [45]. To implement this strategy, we first lift the observables to a higher dimensional space k=[k⊤flift⊤(k)]⊤,z_k= bmatrixy_k &f_ lift (y_k) bmatrix , (6) where k∈ℝM+γz_k ^M+γ is the lifted token embedding. Here, flift∈ℝγf_ lift ^γ where γ is the dimension of the lifting and ⊤ indicates the vector transpose. This initial lifting step generally yields a more accurate representation of the Koopman operator; common choices of lifted coordinates include polynomial combinations of observables and radial basis functions [45]. In this work, we take the observable from (Appendix A: Koopman Operator Theory and Dynamic Mode Decomposition for Inferring Dynamics from Token Embeddings) to be k=⊤H(k) y_k= H(x_k) where ∈ℝM×500 ^M× 500 is comprised of the 500 most dominant SVD modes from the fitting data, as gauged by the singular values of the covariance matrix of the fitting data. The lifting function fliftf_ lift takes polynomial combinations up to order 4 of a subset of the most dominant SVD modes. A set of snapshot pairs sk=(k,k+1)s_k=(z_k,z_k+1) is collected and arranged into matrices =[1…q] X= bmatrixz_1&…&z_q bmatrix and +=[2…q+1] X^+= bmatrixz_2&…&z_q+1 bmatrix where q is the number of snapshot pairs used. An approximation of the Koopman operator k+1=kz_k+1=Az_k can be obtained according to =+†,A=X^+X , (7) where † denotes the pseudoinverse. Low-rank approximations of A can be obtained by truncating small magnitude SVD modes of X before taking the pseudoinverse.