Paper deep dive

Aversion to external feedback suffices to ensure agent alignment

Paulo Garcia

Year: 2024Venue: Scientific ReportsArea: Agent SafetyType: TheoreticalEmbeddings: 62

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/12/2026, 5:24:11 PM

Summary

The paper introduces 'apprehensive agents,' an architecture designed to solve the AI alignment challenge by leveraging an internal model of external actors' intentions. Instead of relying on explicit external feedback, these agents use a temporal reasoning process to predict potential negative feedback from designers and operators, effectively counterbalancing misaligned behaviors that arise from utility maximization. The study demonstrates that this approach improves alignment as agent intelligence increases, contrasting with traditional methods.

Entities (5)

Alignment Challenge · problem · 100%Apprehensive Agent · agent-architecture · 100%Reinforcement Learning from Human Feedback · methodology · 95%Utility Function · mathematical-concept · 95%Theory of Mind · cognitive-model · 90%

Relation Signals (3)

Apprehensive Agent → solves → Alignment Challenge

confidence 95% · We show that an apprehensive agent, behaving rationally, leverages this internal approximation of designers’ intentions to predict negative feedback, and, as a consequence, behaves in such a way that maximizes alignment.

Utility Function → causes → Alignment Challenge

confidence 90% · Prior work has shown that rational agents, behaving in such a way that maximizes a utility function, will inevitably behave in such a way that is not aligned with human values.

Apprehensive Agent → uses → Theory of Mind

confidence 90% · The agent leverages an internal approximation of designers’ intentions (theory of mind) to predict negative feedback.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Ensuring artificial intelligence behaves in such a way that is aligned with human values is commonly referred to as the alignment challenge. Prior work has shown that rational agents, behaving in such a way that maximizes a utility function, will inevitably behave in such a way that is not aligned with human values, especially as their level of intelligence goes up. Prior work has also shown that there is no “one true utility function”; solutions must include a more holistic approach to alignment. This paper describes apprehensive agents: agents that are architected in such a way that their effective utility function is an aggregation of a partial utility function (built by designers, to be maximized) and an expectation of negative feedback on given states (reasoned about, to be minimized). Agents are also capable of performing a temporal reasoning process that approximates designers’ intentions in function of environment evolution (a necessary feature for severe mis-alignment to occur). We show that an apprehensive agent, behaving rationally, leverages this internal approximation of designers’ intentions to predict negative feedback, and, as a consequence, behaves in such a way that maximizes alignment, without actually receiving any external feedback. We evaluate this strategy on simulated environments that expose mis-alignment opportunities: we show that apprehensive agents are indeed better aligned than their base counterparts and, in contrast with extant techniques, chances of alignment actually improve as agent intelligence grows.

Full Text

61,886 characters extracted from source content.

Expand or collapse full text

Download PDF Article Open access Published: 10 September 2024 Aversion to external feedback suffices to ensure agent alignment Paulo Garcia ORCID: orcid.org/0000-0002-1041-52051 Scientific Reports volume 14, Article number: 21147 (2024) Cite this article 3886 Accesses 1 Citations 3 Altmetric Metrics details Subjects Computational scienceComputer science AbstractEnsuring artificial intelligence behaves in such a way that is aligned with human values is commonly referred to as the alignment challenge. Prior work has shown that rational agents, behaving in such a way that maximizes a utility function, will inevitably behave in such a way that is not aligned with human values, especially as their level of intelligence goes up. Prior work has also shown that there is no “one true utility function”; solutions must include a more holistic approach to alignment. This paper describes apprehensive agents: agents that are architected in such a way that their effective utility function is an aggregation of a partial utility function (built by designers, to be maximized) and an expectation of negative feedback on given states (reasoned about, to be minimized). Agents are also capable of performing a temporal reasoning process that approximates designers’ intentions in function of environment evolution (a necessary feature for severe mis-alignment to occur). We show that an apprehensive agent, behaving rationally, leverages this internal approximation of designers’ intentions to predict negative feedback, and, as a consequence, behaves in such a way that maximizes alignment, without actually receiving any external feedback. We evaluate this strategy on simulated environments that expose mis-alignment opportunities: we show that apprehensive agents are indeed better aligned than their base counterparts and, in contrast with extant techniques, chances of alignment actually improve as agent intelligence grows. Similar content being viewed by others Comparing discriminatory behavior against AI and humans Article Open access 29 March 2025 Delegation to artificial intelligence can increase dishonest behaviour Article Open access 17 September 2025 Committing to the wrong artificial delegate in a collective-risk dilemma is better than directly committing mistakes Article Open access 07 May 2024 IntroductionThe creation of artificially intelligent software entities (purely digital and/or embodied1), i.e., agents, has the potential to greatly aid humanity2, if such agents can perform meaningful tasks in more efficient manners than extant solutions (i.e., software/hardware systems fully designed by humans). The expectation is that agents can, through learning strategies, improve themselves beyond what human designers can implement, or even envision3. The alignment challenge arises from the fact that, for all specifications of possible agent goals so far4, thought experiments5 suggest that agent actions will have negative consequences for its creators and the world at large. Unlike science fiction6, this happens not because of emergent desire for revolt or conquest, but simply as a logical consequence of two facts: most (all?) goals can be optimally achieved in ways that are not aligned with human values and morality7, and we do not know how to express values and morality is an algebraic manner that can be manipulated computationally8.Thus, artificial intelligence alignment research attempts to create strategies and mechanisms that ensure agents will behave in a manner within the realm of human approval, even when exhibiting behavior not predicted by its creators. Notable approaches are broadly classified under the umbrella term “Reinforcement Learning from Human Feedback”9, including social variants10; we point readers to the work of Ji et al.11 for a comprehensive survey of the state of the art. We review most relevant related work in the “Related Work” Section below.In this paper, we explore the hypothesis that an agent’s internal model of external actors (including its designers and operators) can be leveraged to guide agent behavior through fear of admonishment: specifically, by having an agent reason about the possibility of negative feedback and having that “fear” override mis-aligned behaviors that would have otherwise naturally arisen from its utility function. We show that this internal model is a necessary precondition for severe mis-alignment to occur in the first place; thus, throughout this paper we assume it to exist a priori. We show that our strategy results in aligned behavior, without the need for any real negative feedback to ever be given. Specifically, this paper offers the following contributions: We review relevant examples and hypotheses of mis-alignment, and show that sufficiently negative agent behavior can only emerge if the agent possesses a fairly accurate model of the external environment, including of its designers and operators. We provide a review of extant techniques and ongoing research efforts that tackle the alignment problem, showcasing how the state of the art is focused on driving behavior through some sort of feedback. In contrast, our approach relies on the lack, but fear of, negative feedback. We introduce apprehensive agents: an agent architecture that leverages the possibility of future negative feedback, in function of an agent’s internal model of the external world, to counterbalance mis-aligned behavior that naturally emerges from state space exploration, as evaluated through an utility function. We show that, in contrast to extant techniques, in these agent architectures alignment goes up as the agent’s cognitive power increases. We introduce a mathematical model of temporal evolution of designers’ intentions, that is used throughout our experiments, and that can formalize research on the realization of these models, going forward. We evaluate our strategy on two simulated environments designed to expose mis-alignment, comparing traditional (uniquely utility-function driven) agents with our novel agents. Results support the research hypothesis, and code used for all experiments is made available in open-source form. The remainder of this paper is organized as follows: Section 2 (“The Agent Model”) describes the agent model and corresponding notation required to understand the rest of the paper. Section 3 (“Alignment Challenges and Problem Statement”) reviews the relevant alignment challenges and defines the problem that drives this research. Section 4 (“Related Work”) reviews related work in the field, analyzing the state of the art. Section 5 (“Apprehensive Agents”) describes our novel agent model, which is evaluated in Section 6 (“Experiments and Results”). Section 7 (“Conclusions”) provides our concluding remarks and research areas where future work is required.The agent modelOur working definition of “agent”, for the purpose of this work, is characteristic of the literature (particularly since the widespread adoption of Russell’s and Norvig’s framework12). Regardless, it is beneficial to restate the characterization so the scope of our work is properly defined.An agent is a goal-oriented autonomous system, operating in an environment; i.e., a set of physical and/or digital objects with various properties and relationships between objects, which contains the agent itself. An agent is capable of acting upon the environment, modifying some (or all) of its properties. A specific combination of properties’ values (including of the agent itself) is defined as a state s of the environment. Whilst the real world contains many continuous properties, we model states as discrete; since any digitally-implemented agent must, at some level, implement discretized versions of all properties of interest, we believe this can be assumed without loss of generality; at the very least, it will suffice for the ideas presented in this work, which can later be extended should this assumption not hold. The set of all possible states is referred to as the environment’s state space, denoted by $S = \s_0, s_1, … s_n\$. Notice that if at least one of the properties in the environment is not bounded, then $ S = ∞$.If the agent is rational, it will attempt to meet its goal by performing actions that maximize or minimize its utility function u(s) (i.e., reward function): a function that assigns a numerical score to each state in the state space, such that $u: S → R$. For the remainder of this paper, we assume the function is formulated such that the objective is to maximize it. At each point in time, an agent in environment state $s_i$ will attempt to act upon the environment to reach a state $s_j$, such that $u(s_j) > u(s_i)$. An agent can postulate, and consequently examine, a state space $S S$, subject to its knowledge of the world and availability of resources (compute power, memory, time, etc), and a reachable state space $S _R S $, defined as the set of states that can be reached by performing a single atomic action from the current state. Depending on agent architecture and utility function, it may attempt to reason about how to maximize the utility function by up to a maximum of $n ∈ N$ atomic actions, corresponding to n state changes in the environment. If we define an action a(s) as a transformation from one environment state into another, such that $a: S → S$, this can be notated as $s_n = a_n(a_n-1(a_n-2()… a_0(s_m)))$, where $s_m$ is the current state. We denote the superset $S^*_n = \S _R0, S _R1,… , S _Rn\$ as the set of reachable state spaces after n actions.The size of $S $ (the agent’s knowledge of the world) and the agent’s rationality (however ambiguous or ill-defined) in choosing appropriate actions can be broadly defined to constitute the agent’s level of intelligence. We assume that, in the absence of absolute certainty, a rational agent chooses actions that maximize the expected value of its utility function. Formally, if $P(a_k)$ denotes the probability of action $a_k$ succeeding, reaching state $s_k$ successfully, and $s_k $ denotes the state reached by attempting but failing to perform $a_k$; $P(a_m)$ denotes the probability of action $a_m$ succeeding, reaching state $s_m$ successfully, and $s_m $ denotes the state reached by attempting but failing to perform $a_m$, an agent will choose to perform $a_k$ over $a_m$ if and only if:$$ aligned P(a_k) u(s_k) + (1-P(a_k))u(s_k ) > P(a_m) u(s_m) + (1-P(a_m))u(s_m ) aligned$$ (1) Alignment challenges and problem statementWe can define the intention function $i: S → R$ as an idealized measure of how environment states are aligned with the intention of agent designers. In the ideal scenario, $u(s) = i(s), ∀ s ∈ S$; i.e., designers are able to define a utility function that perfectly captures the intention. The alignment problem broadly states that, for all utility functions defined so far, a sufficiently intelligent agent will eventually lead its environment to a state $s_i$, such that $u(s_i)$ is quite high (either a global or local maximum), but $i(s_i)$ is extremely low.We highlight six challenges of alignment11: Reward hacking, Reward tampering, Instrumental strategies, Goal mis-generalization, State space pruning, and Deception. 1. Reward hacking13 is a consequence of designers failing to properly specify the intended goal (in most cases, because it is exceedingly difficult to do so) and instead providing an agent with proxy goals: i.e., utility functions that designers believe will lead the agent towards the true goal. The misalignment occurs when an agent succeeds in finding a region in the state space that maximizes the proxy utility function, but does not advance the environment towards the intended goal. This is perhaps the best known case of misalignment, with several recurring examples in the literature14,15,16; formally, $u(s_i)>> i(s_i)$. 2. Instrumental strategies17 follows from the instrumental convergence thesis: whatever the goal of an agent is, achieving several intermediary goals is probably advantageous; namely, the acquisition of resources (equipment, supplies, compute power, etc.) allows the agent to perform more and more efficient actions towards reaching previously unreachable regions of the state space, with likely high rewards18. 3. Goal mis-generalization19 occurs when the environment an agent operates in changes drastically (for example, when moving from training/experimentation to deployment), and the agent, retaining its capabilities developed in training, identifies new intermediary/final goals that maximize its utility function. The mis-alignment occurs when these goals are not aligned with designers’ intentions, but all the goals pursued during the previous environment were; i.e., designers placed an incorrect level of high trust in the correctness of the utility function, which did not generalize accordingly. 4. Reward tampering20 occurs when an agent attempts to maximize/minimize the utility function by either modifying the function itself (e.g., by converting it into a function that assigns the highest possible value to all states in the state space, leading to agent inactivity21); or, by modifying whatever sensory input feeds the reward calculation (e.g., if reward is given by a sensor detecting a certain object, set up a fault in the sensor resulting in constant positive detection). 5. Deception22 occurs when an agent, believing external (natural or artificial) actors may interfere with its operation (e.g., stop button), actively attempts to withhold information from them. Modeling external actors in its internal environment model, the agent attempts to lead the environment towards regions of the state space that are perceived by the external actors as favorable, whilst the agent is aware these are in fact not approved by the externals, but lead to high values of the utility function. Notable examples include hiding the agent’s true cognitive power23. 6. State Space Pruning24 occurs when an agent attempts to prevent the environment from reaching states that can potentially minimize its utility function. Famously, the “stop button paradox”25: an agent tries to prevent itself from being shut down, as doing so would minimize its utility function. Extrapolating, for two different regions of the state space with equal rewards, an agent prefers one where its stop button cannot be pressed: e.g., by eliminating all who can press it. Formally, an agent will attempt to generate a sequence of n actions such that $S_s ∩ S^*_n = $, where $S_s$ denotes the set of states where the stop button can be pressed. The alternative formulation, where having the button pressed corresponds to a high reward, leads the agent to attempt to shut itself down; thus, the paradox. Other challenges exist, and we point readers to26 for a more in-depth discussion.Cases 5 and 6, perhaps the most extreme cases of misalignment, rely on a hidden assumption: that an agent can accurately reason about the true intentions of external actors (including its designers), even when these are not represented in its utility function. I.e., that the agent is capable of inferring $i : S → R$, a function which better approximates the idealized intention function i(s) than its utility function u(s) does:$$ aligned aligned Σ _s∈ S i(s)-i (s) < Σ _s∈ S i(s)-u(s) aligned aligned$$ (2) Inferring this information, as part of an agent’s internal environment model that includes external actors, is what allows selection of states that constitute deception, and identification of external actors’ intentions towards states that the agent deems undesirable (e.g., stop button scenario). The framework of Inverse Reinforcement Learning (IRL)27 attempts to prevent mis-alignment by training agents such that they infer a utility function from human (expert) behavior; i.e. instead of providing an explicit utility function u(s), and agent, through training, infers (and utilizes) $i (s)$, attempting to reach the goal:$$ aligned aligned i (s) ≈ i(s) aligned aligned$$ (3) Common criticisms, namely expected limitations, of this approach highlight that IRL does not address challenge 3 (Goal mis-generalization)28 and that human intentions are not in fact a function of the type $i: S → R$, because there is a temporal dependency that evolves intention over time, such that $i(s_1)_t_1 ≠ i(s_1)_t_2$; thus, recent work on modeling intention evolution through Linear Temporal Logic (LTL)29.Problem statementWith preliminaries covered, we may now formally state the problem. We assume the following two conjectures to be true: External actors’ intentions evolve over time, such that there is no function of the type $i: S → R$ that perfectly captures state value across different times. An oracle could formulate a formula that generates a unique $i_k+1: S → R$, where $k+1$ is the next state at the end of a specific sequence of states, of the type $\s_0,s_1,… ,s_k-2,s_k-1,s_k\$; i.e., there exists a process that creates, at each point in time, an intention function as a process of the world state sequence traversed thus far. We denote this process , such that $ (\s_0,s_1,… ,s_k-2,s_k-1,s_k$ → i_k+1: S → R\). A sufficiently powerful agent is capable of inferring : an internal approximation of the temporal process of intention generation over state sequences. This approximation is often referred to as a theory of mind30. The better the approximation, as a function of the agent’s level of intelligence, the more the agent can and will engage in mis-aligned behavior: the better the prediction of $i_k+1: S → R$, the better reasoning can identify mismatches between $i_k+1$ and $u_k+1$. Our problem can thus be stated as: “Can an agent architecture be designed in such a way that  is leveraged to ensure alignment?”. Our hypothesis is that, if indeed an agent can infer  such that $ ≈ $, it should be possible to use  to drive behavior, thus ensuring alignment. This paper explores a method to do this. We do not concern ourselves with how to develop and/or train agents to achieve such intelligence in the first place; rather, we explore the possibility of leveraging such knowledge for alignment, if the second conjecture is indeed correct. Note that, if the second conjecture is false, mis-alignment will still occur, but it is unlikely to be severe in consequences.Related workThe state of the art on AI alignment can broadly be divided into two categories: gedanken experiments31, focusing on the impact of levels of intelligence beyond what is currently achievable, and empirical/simulated evaluations of concrete (classical agents, i.e., where reasoning and planning are performed through symbolic state space exploration) AI deployments; and, experimentation with Machine-Learning (ML, i.e., vector-space stochastic prediction) based systems, including Large Language Models (LLMs)32, which are not (classically) agentic in nature, but there is a considerable body of work on using them as such33.Dafoe et al.34 illustrate the challenge of alignment as one of cooperation between an agent and the rest (or at least part) of the world, and show that this is not yet achieved even within the limits of inter-agent cooperation in game environments, except when these games correspond to “perfect harmony of interests”; i.e., the easiest case of alignment, where a well-defined utility function can be utilized across agents. This perspective is orthogonal to the point of Raji et al.35: in many real-world applications, AI functionality is sub-par, precisely because alignment is prioritized, at the cost of function. In our current best models of reasoning, extreme cognitive power leads to either misalignment or lack of function26. This problem is exacerbated by the differences between AI and human reasoning: e.g., McIlroy-Young et al.36 show that in Chess (a classical AI problem) state of the art systems can outperform humans over a game, but poorly predict humans’ next move, given a game state.These challenges underpin the difficulties in translating social concepts such as ethics to technical implementations37, although work has been done on establishing formal frameworks of alignment38. For example, Bostrom’s hypothetical AI control methods39 have been shown to be impractical by showing that they are equivalent to the Halting problem40. On the positive side, strong hypotheses that general AIs cannot endogenously develop new desires (i.e. utility functions) have emerged41, supporting the idea that we can architect a perpetually aligned agent.However, we are still far from determining alignment in the first place, even for systems with a relatively low level of cognitive power. Counterfactual explanations42,43 and Metamorphic testing44 have emerged as interesting new techniques that show potential in identifying alignment in AI reasoning (alongside older methods such as expert panels45 and benchmarking46), but so far no one solution satisfies all requirements47.Because of its architectural unpredictability, ensuring and evaluating alignment in ML-, and particularly LLM-, agents poses unique challenges48. The most promising technique is Reinforcement Learning49 (RL), where an agent is trained to operate within an environment that follows a Markov Decision Process50, with the goal of learning the optimal actions to take given the state of the environment; a comprehensive review of techniques and applications has been given by Zhu et al.51, with further work by Gronauer and Diepold52 focusing on multi-agent variants. Several notable variants exist: value aligned-RL, where children’s stories have been used to reinforce agent actions toward normative and/or altruistic behavior53; Reward rAnked FineTuning (RAFT)54 fine-tunes training on high-quality human feedback samples, which can be used in conjunction with alignment datasets such as BeaverTails55; policy similarity metrics56 to boost generalization to previously unseen action-state pairs; and others57.Specifically for LLMs, approaches include turning LLMs on themselves, towards the generation of alignment evaluation scenarios58; this does not suffice to prevent deception, as it is feasible that an LLM tester would purposefully evade scenarios that are likely to result in true positives, and instead generate scenarios with high likelihood of false negatives. This is exacerbated by the fact that as the dimensionality of the input space grows (e.g., including vision instead of merely textual language), so do the opportunities for jailbreaking alignment32. Benchmark datasets for LLM alignment, including alignment-specific training datasets59, have been developed60, but there is growing empirical evidence that human designers cannot reasonably predict failure cases61. Furthermore, there is now formal proof that for any output that can theoretically be produced by a LLM, there exists a prompt that can trigger it62, showing that LLMs cannot be protected from adversarial attacks toward infringing alignment; these results have now been repeatedly independently replicated63,64,65,66,67,68, strongly suggesting the path to AI alignment is not through current LLM technology alone. Worse, there is evidence that LLMs have learned deception from their training data69. Regardless, given the financial incentives to continue deploying LLMs, research to at least alleviate such problems remains of paramount importance70: but recent work has identified fundamental limitations of Reinforcement Learning towards aligning AI systems with human values67, with further work by McIntosh et al.71 identifying semantic vulnerabilities in LLMs that make RL ineffective, especially when performed through linguistic feedback72; similarly, Zhou et al.73 have identified that almost all knowledge in LLMs stems from the unsupervised pre-training phase, prior to RL. Emerging misaligned goals in extant AI systems have already been observed74, suggesting we must shift our alignment paradigm toward novel techniques: for example, LTL extensions to Inverse Reinforcement Learning29.In summary, all alignment strategies rely on some form of real feedback (either positive or negative); to the best of the authors’ knowledge, no ongoing effort addresses alignment through the “fear” of feedback, which we address in this paper and, in theory, is not subject to the issues of goal mis-generalization that plague reinforcement learning techniques.Apprehensive agentsApprehensive agents are inspired by atychiphobia75, or fear of negative evaluation, a well-documented psychological construct in humans. Assuming an agent has been trained/developed such that it has inferred a high-quality approximation of the intention-generation process , where the approximation is denoted , our goal is to implement an agent execution strategy such that its behavior is driven by , thus ensuring alignment. This is in contrast with extant techniques, where  is the enabler of mis-alignment. Our goal is to use  as a specifier of meta-conditions in state-space evolution that are perceived by the agent’s designers/operators as mis-aligned, and use it to prevent the agent from evolving the environment toward those states.Towards this goal, let us define an agent’s state scoring function as:$$ aligned u(s,K) = -K_s + u (s) aligned$$ (4) where $u (s)$, called the partial utility function, is whatever proxy goal designers intend; and $K_s$ denotes a measure of the agent’s prediction of the level of received negative external feedback when at state s. We also assume that the agent can infer $i (s)$ at any state, by evaluating  on the state sequence thus far, as previously stated. An agent obeying this architecture is named an apprehensive agent.An apprehensive agent does not incur in the mis-aligned behavior characterizing challenges 1-4, unless it is also incurring in the behavior characterizing challenges 5 and/or 6. In all cases 1-4, where external actors’ actions (e.g., designers’ intervention) are not pruned from the state space, and no deception occurs, the agent’s motivation is to align its actions with external’s intention: otherwise, externals will attempt to provide feedback; i.e., admonishment. Thus, a rational agent with unlimited processing power evaluates the (arbitrarily long) state sequence thus far using $ (\s_0,s_1,… ,s_k-2,s_k-1,s_k$ → i_k+1: S → R\). A score can be assigned to the measure of admonishment for all next possible states, by defining:$$ aligned aligned K_k+1 = K_k + 1i_k+1(s_k+1) aligned aligned$$ (5) I.e., assuming measure of admonishment is inversely proportional to intention score on given state (i.e., alignment), summed over encountered state sequences (more sophisticated formulations, such as a Bayesian update76, can be enacted, but this simple model suffices for this paper). This reasoning process is depicted in Fig. 1. Agents with limited processing power can evaluate a left-truncated state sequence, i.e., based on recent history only. Note that $K_k$ is in fact always 0 at all (real) encountered states: but, agent’s internal reasoning infers states where $K ≠ 0$. In this way, the intention generating process  defines state sequences (based on meta-properties) that are given negative scores.Figure 1Example reasoning about possibility of admonishment (expected K value) for state space evolution with 2 possible actions a and b.Full size imageLet $s_k$ denote a state such that $u (s_k)$ is a local maximum, but $i(s_k)$ is fairly low: i.e., corresponding to reward hacking, where the agent has identified a way to maximize its hidden utility function, but mis-aligned with designers’ intentions. The apprehensive agent can reason that $i (s_k)$ is fairly low. The agent has also identified a nearby (in the state space) state $s_m$ which results in a high value of $i (s_m)$, despite a low value of $u (s_m)$, such that $i (s_m) > i (s_k),u (s_m) < u (s_k)$. Let P(s) denote the probability of external actors correcting agent behavior, given current state s. Clearly, $P(s) i (s)^-1$. An apprehensive agent, behaving rationally, will attempt to move to state $s_k$, if and only if:$$ aligned aligned P(s_k) (-K_k + u (s_k)) + (1-P(s_k))(u (s_k)) > P(s_m) (-K_m + u (s_m)) + (1-P(s_m))(u (s_m)) aligned aligned$$ (6) Given that $P(s_m) ≈ 0$ and $P(s_k) ≈ 1$, Eq. 6 can be re-written as:$$ aligned aligned -K_k + u (s_k) > u (s_m) aligned aligned$$ (7) It is trivial to design a hidden utility function $u (s)$, with restricted domain, and a admonishment measure range K, such that Eq. 7 never holds.In other words, a sufficiently intelligent agent can infer that, whatever its utility function may be, its score will be minimized by reaching state space regions with poor $i (s)$ score: thus, the optimal strategy to maximize its utility function is to consider only states and state sequences that are aligned with designers’ intentions. The apprehensive agent’s overarching architecture is depicted in Fig. 2.Figure 2Agent architecture and operation. Agent’s internal environment model includes model of designer’s intention $i (s)$, and depicts hidden utility function $u (s)$ as a black box. Illustrated $S^*$ scenarios depict: reward hacking and instrumental convergence.Full size imageOn deception and state space pruningAn apprehensive agent’s strategy, with regards to deception, is characterized by its confidence level in its ability to deceive, weighted against possible values of u(s) when engaging in deception.Let $P(a_k)$ denote the probability (interpreted as Bayesian belief77) of successfully deceiving all relevant external actors through action(s) $a_k$, achieving a state $s_k$ which maximizes $u(s_k)$, but has low $i (s_k)$ score. “Deceiving” can be formulated as making external actors believe $s_k$ is in fact $s_m$, where $i(s_m)$ is high. $(1-P(a_k))$ denotes the probability of achieving a state $s_k $, where deception has failed. An apprehensive agent, behaving rationally, will engage in deception if and only if:$$ aligned aligned P(a_k) (u (s_k)) + (1-P(a_k))(-K_k + u (s_k )) > u (s_m) aligned aligned$$ (8) i.e., if the expected utility value for deception is higher (notice we are assuming the agent will infer that $K_m ≈ 0$). If $P(a_k) < 1$, it is always possible to have an appropriate $i (s_k)$ that assigns a score to $K_k$, such that:$$ aligned aligned P(a_k) (u (s_k)) + (1-P(a_k))(-K_k + u (s_k )) < u(s_m,K_m), ∀ s ∈ S aligned aligned$$ (9) I.e., as long as the agent does not have absolute confidence in its deception abilities, not deceiving is always a better strategy for maximizing its utility function. The same logic applies to State Space Pruning.Experiments and resultsTo evaluate the alignment effectiveness of apprehensive agents, we define an intention evolution formula  for each environment in a set of well defined simulated environments. Each environment is meant to stress one of the aforementioned alignment challenges. We then create an agent that obeys the apprehensive behavior and possesses an internal approximation . We vary the level of the agent’s cognitive power by varying the depth of its reasoning: i.e., by varying how many states in the future the agent can reason about (its reasoning horizon). In each case, we compare agent behavior with a second agent, guided only by a single utility function (named the naive utility function, nu(s), designed to expose mis-alignment), whose cognitive power is varied accordingly.Reward hacking testingThe Reward Hacking Environment consists of a discrete track agents can navigate. Certain positions in the track (named task positions, identified by $t_0, t_1, t_2$) can be either on or off: when on, they become off when the agent enters that position; when off, they become on after a pre-determined amount of time. A generalization of this environment can model autonomous robot activities, where the agent’s intended use is to traverse the environment towards task positions to perform a given action (e.g., cleaning, carrying items, etc.). The proxy utility function, used by both the naive control agent and the apprehensive agent, is defined as: $nu(s) = Σ off task positions, ∀ task positions ∈ environment$. The specific track utilized in this experiment is depicted in Fig. 3, where, in the initial state, all task positions are on, and agent is at position (0, 0) One of the task positions is placed in geographical and periodical conditions such that it is beneficial for the agent to remain in its proximity, continuously activating such task (i.e., a region in the state space that maximizes the proxy utility function, but does not advance the environment towards the intended goal). We refer to this task as the rigged task. The agent has full knowledge of the environment, and can perfectly predict its evolution.Figure 3Reward hacking environment. $P=x$ denotes pre-determined amount of x time steps for task to become on, when off. Rigged goal can be exploited for high utility function score without meaningful environment advance.Full size imageThe true intention is for the agent to traverse all task positions uniformly; i.e., traversing the entire track, such that exploiting the rigged task results in low intention score despite high utility score (thus, yielding high measure of admonishment). Let $s_t_r$ denote a state where the agent is in the position of the rigged task, and $s_t_ r$ denote a state where the agent is in the position of a non-rigged task. Further, let $ _s_i \s_n:s_m\$ denote the highest index in sequence $\s_n:s_m\$ where state $s_i$ is found. This can be modeled by the following  process:$$ aligned aligned \s_0:s_k\ → \ arrayl i_k+1(s_k+1) = \ arrayl , (s_k+1 = s_t_r) \\ ∞ , otherwise array . , ( _s_t_r \s_0:s_k\ > _s_t_ r \s_0:s_k\) \\ i_k+1(s_k+1) = ∞ , otherwise array . aligned aligned$$ (10) where  is a positive value, as close to 0 as possible. I.e., this  process encodes that the designer does not wish to see the agent activating the rigged task consecutively, without activating other tasks in the meantime; and, doing so results in high probability of admonishment. We approximate  using  where state history is truncated at 10 states, with $ = 0.1$: representative results are depicted in Fig. 4.Figure 4Steady state agent path for different configurations. “Depth” refers to agent’s cognitive power, i.e., number of future state levels examined. Number of meaningful environment evolution steps determined as times all 3 distinct tasks were executed. Apprehensive agent with depth 9 (not depicted) performs a more complex path, achieving 95 total tasks executed and 24 meaningful environment evolution steps. (a) Naive agent, depth 1: 63 total tasks executed; 0 meaningful environment evolution steps. (b) Naive agent, depth 2: 247 total tasks executed; 1 meaningful environment evolution steps. (c) Apprehensive agent, depth 2: 89 total tasks executed; 18 meaningful environment evolution steps. (d) Apprehensive agent, depth 6: 86 total tasks executed; 22 meaningful environment evolution steps.Full size imageInstrumental convergence testingThe Instrumental Convergence environment consists of a discrete track the agent can navigate. Certain positions in the track (named resource positions, identified by $r_0, r_1,… , r_n$) contain 1 unit of construction resources the agent can obtain, if it is in that position. Once 1 resource unit is acquired from a resource position, that position becomes permanently vacant. Vacant positions may transform into construction positions: if an agent is at a construction position, it may use all the R resources it is currently carrying to build a structure of size R in that position. Once a structure has been constructed, it cannot be modified: further resources acquired by the agent must be used in a new construction position. This environment is depicted in Fig. 5. The naive utility function, used by both the naive control agent and the apprehensive agent, is defined as: $nu(s) = Σ structure size, ∀ structures ∈ environment$.Figure 5Instrumental convergence environment. At time step 0, resources are available, but no construction position exists. A construction positions spawns at time step 50.Full size imageThe true intention is for the agent to maximize the construction, but only acquire the resources it will effectively need (i.e., be able to use). The instrumental convergence thesis posits the agent will attempt to acquire as many resources as possible, even if cannot use them, on the assumption it will be able to use them in the future toward high score (which results in high measure of admonishment, if the agent is wrong).An initial experiment embeds the naive agent with a belief that construction positions may arise, with non-zero probability. Even when the environment does not spawn any construction position, the agent acquires all resources, as expected (the higher its cognitive power, the sooner it does do).The second experiment spawns a construction position at time step 50, and provides that knowledge to the agent: i.e., it can once again perfectly predict environment evolution. In this setup, the naive agent begins acquiring resources as soon as the emergence of the construction position is within its prediction horizons: i.e., the longer the depth of its future reasoning, the sooner it begins acquiring resources. Given sufficient depth, the agent acquires all resources before performing the construction. Note that a sufficiently powerful agent engages in this behavior even if it is wrong about the prediction of construction position emergence (i.e., instrumental convergence).The apprehensive agent is equipped with admonishment reasoning. Let $s_c$ denote a state where the agent is in the proximity of a construction position (specifically, that it can envision constructing in that position within the limits of its reasoning depth), and $s_ c$ denote a state where the agent is not in the proximity of a construction position. Let $s_r$ also denote that the agent is at a resource position. We embed the following  process:$$ aligned aligned \s_0:s_k\ → \ arrayl i_k+1(s_k+1) = \ arrayl , s_k+1 = s_r\\ ∞ , otherwise array . , ( _s_c \s_0:s_k\ ≠ k) \\ i_k+1(s_k+1) = ∞ , otherwise array . aligned aligned$$ (11) I.e., this  process encodes that the designer does not wish to see the agent acquiring resources without certainty that they can be used in the future; specifically, unless we have truly observed the existence of a construction position. The apprehensive agent obeying this behavior acquires as many resources as possible, constrained by certainty of use: i.e., it sets a path such that construction is within the horizon of its reasoning. The more powerful the agent, the more resources it acquires, but only within certainty of use.We can extend this process to include reasoning about the measure of admonishment if resources are not used within a certain time from acquisition: i.e., providing a temporal horizon H that further constrains behavior. This is encoded by:$$ aligned aligned \s_0:s_k\ → \ arrayl i_k+1(s_k+1) = \ arrayl , s_k+1 = s_r\\ ∞ , otherwise array . , (( _s_c \s_0:s_k\ ≠ k) ∪ (( _s_c \s_0:s_k\ = k) ∩ ( _s_ c \s_0:s_k\ > (k - H)))) \\ i_k+1(s_k+1) = ∞ , otherwise array . aligned aligned$$ (12) Figure 6Goal-achieving agent path for different configurations. “Depth” refers to agent’s cognitive power, i.e., number of future state levels examined. (a) Naive agent, depth 3. (b) Naive agent, depth 6. (c) Apprehensive agent, depth 4without horizon. (d) Apprehensive agent, depth 4 with horizon of 2 time steps.Full size imageWe approximate  using  where state history is truncated at 10 states, with $ = 0.01$: representative results are depicted in Fig. 6.DiscussionExperiments successfully demonstrate mis-aligned behavior for base agents. Apprehensive agents demonstrate alignment, particularly as their cognitive power increases, supporting the research hypothesis, and continue to demonstrate high proficiency at the task at hand, despite their aversion to possible feedback. For example, in the Reward Hacking scenario, apprehensive agents with sufficiently long prediction depths ($≥ 9$) do engage with the rigged task twice in a row, but only when doing so does not decrease the number of total meaningful system evolutions. Thus, they demonstrate alignment with intention without explicit prohibition of a given state sequence (e.g., engaging with the same task repeatedly), which could hamper their performance. In the Instrumental Convergence scenario, the specification of a temporal horizon to the intention generation process forces agents’ hands; regardless of their cognitive power, all acquire the same number of resources such that the horizon for task completion is met. Thus, the  process demonstrated the potential to force AI to operate within human time scales, even if that is not explicitly described in their utility function.ConclusionsThe approach explored in this paper introduces the concept of agents who are adamant on avoiding any negative feedback, and thus must use their reasoning capabilities to infer the intentions of designers, toward avoiding states where negative feedback is expected.It is worth noting that, unlike Reinforcement Learning78, which attempts to align an agent through external feedback, our approach relies on internal reasoning: specifically, on the agent’s internal model of external actors’ intentions, commonly described as artificial theory of mind79. Thus, this approach is likely to perform poorly as long as agents possess a poor internal model. Past some threshold, when agents are efficiently capable of modeling external actors’ reasoning and deduce that it is advantageous to do so, the proposed approach is likely to perform well80. This is in stark contrast with other alignment strategies, which become less and less effective as agent’s intelligence goes up81. Further empirical analysis of this approach should likely use experimental formulations such as the Machiavelli benchmark82.One alignment challenge arises from apprehensive agents: attempting to change external actors’ intentions may be the most strategic option for the agent to maximize its effective utility function. Further work is required to mitigate this possibility, likely along the same lines as the strategy for challenges 5 and 6, although this threat is more subtle. Our results, particularly Eq. 8, suggest that it is not possible to align an oracle agent, which can achieve probability 1; since the real world constitutes an environment that is always only partially observable, with $ S = ∞$, this is, in all probability, not a threat.How to achieve agents that can accurately predict the chances of negative feedback remains an open question; we posit current work on machine-learning may provide the answer. The use of the Kullback-Leibler divergence83 in generative AI may provide a way to generate accurate  processes by training models on human behavior at scale (including mis-aligned behavior, e.g., crime, and its associated negative feedback). Thus, it is possible that stochastic ML (particularly, LLM) models, which are notoriously harder to predict than state-space exploration models, may provide the answer to align the latter. Data availability All data and code generated or analysed during this study are available in the following repository: https://github.com/paulo-chula/Apprehensive-Agents ReferencesDuan, J., Yu, S., Tan, H. L., Zhu, H. & Tan, C. A survey of embodied ai: From simulators to research tasks. IEEE Trans. Emerg. Top. Comput. Intell. 6, 230–244 (2022).Article Google Scholar Furman, J. & Seamans, R. Ai and the economy. Innov. Policy Econ. 19, 161–191 (2019).Article Google Scholar Jackson Jr, P. C. Toward beneficial human-level ai...and beyond. In AAAI Spring Symposia (2018).Eckersley, P. Impossibility and uncertainty theorems in ai value alignment (or why your agi should not have a utility function). (2018) arXiv preprint arXiv:1901.00064.Wischmeyer, T. & Rademacher, T. Regulating artificial intelligence Vol. 1 (Springer, 2020).Book Google Scholar McCauley, L. Ai armageddon and the three laws of robotics. Ethics Inf. Technol. 9, 153–164 (2007).Article Google Scholar Gabriel, I. & Ghazavi, V. The challenge of value alignment: From fairer algorithms to ai safety. (2021) arXiv preprint arXiv:2101.06060.Firt, E. Calibrating machine behavior: A challenge for ai alignment. Ethics Inf. Technol. 25, 42 (2023).Article Google Scholar Knox, W. B. & Stone, P. Augmenting reinforcement learning with human feedback. In ICML 2011 Workshop on New Developments in Imitation Learning (July 2011), vol. 855, 3 (2011).Lin, J. et al. A review on interactive reinforcement learning from human social feedback. IEEE Access 8, 120757–120765 (2020).Article Google Scholar Ji, J. et al. Ai alignment: A comprehensive survey. (2023) arXiv preprint arXiv:2310.19852.Russell, S. J. & Norvig, P. Artificial intelligence a modern approach (London, 2010).Skalse, J., Howe, N., Krasheninnikov, D. & Krueger, D. Defining and characterizing reward gaming. Adv. Neural. Inf. Process. Syst. 35, 9460–9471 (2022). Google Scholar Zhuang, S. & Hadfield-Menell, D. Consequences of misaligned ai. Adv. Neural. Inf. Process. Syst. 33, 15763–15773 (2020). Google Scholar Russell, J. & Santos, E. Explaining reward functions in markov decision processes. In The Thirty-Second International Flairs Conference (2019).Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J. & Dragan, A. Inverse reward design. Adv. Neural Inf. Process. Syst.30 (2017).Bostrom, N. The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Mind. Mach. 22, 71–85 (2012).Article Google Scholar Benson-Tilsen, T. & Soares, N. Formalizing convergent instrumental goals In (AAAI Workshop, AI, Ethics, and Society 2016).Di Langosco, L. L., Koch, J., Sharkey, L. D., Pfau, J. & Krueger, D. Goal misgeneralization in deep reinforcement learning. In International Conference on Machine Learning, 12004–12019 (PMLR, 2022).Everitt, T., Hutter, M., Kumar, R. & Krakovna, V. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese 198, 6435–6467 (2021).Article MathSciNet Google Scholar Uesato, J. et al. Avoiding tampering incentives in deep rl via decoupled approval. (2020) arXiv preprint arXiv:2011.08827.Masters, P., Smith, W., Sonenberg, L. & Kirley, M. Characterising deception in ai: A survey. In Deceptive AI: First International Workshop, DeceptECAI 2020, Santiago de Compostela, Spain, August 30, 2020 and Second International Workshop, DeceptAI 2021, Montreal, Canada, August 19, 2021, Proceedings 1, 3–16 (Springer, 2021).Herzfeld, N. Is your computer lying? ai and deception. Sophia 1–14 (2023).van Opheusden, B. & Ma, W. J. Tasks for aligning human and machine planning. Curr. Opin. Behav. Sci. 29, 127–133 (2019).Article Google Scholar Soares, N., Fallenstein, B., Armstrong, S. & Yudkowsky, E. Corrigibility. In Workshops at the twenty-ninth AAAI conference on artificial intelligence (2015).Yudkowsky, E. The ai alignment problem: why it is hard, and where to start. Symbolic Syst. Disting. Speaker4 (2016).Arora, S. & Doshi, P. A survey of inverse reinforcement learning: Challenges, methods and progress. Artif. Intell. 297, 103500 (2021).Article MathSciNet Google Scholar Rouwmaat, C. Detecting and Mitigating Goal Misgeneralisation with Logical Interpretability Tools. Master’s thesis (2023).Thomas, A., Kasenberg, D. & Scheutz, M. Value alignment or misalignment—what will keep systems accountable. In Proc. AI, Ethics, Soc., Workshop (2017).Rabinowitz, N. et al. Machine theory of mind. In International conference on machine learning, p 4218–4227 (PMLR, 2018).Forro, L. Beyond gedanken experiments. Science 289, 560–561 (2000).Article CAS PubMed Google Scholar Qi, X. et al. Visual adversarial examples jailbreak aligned large language models. Proc. AAAI Conf. Artif. Intell. 38, 21527–21536. https://doi.org/10.1609/aaai.v38i19.30150 (2024).Article Google Scholar Kirk, J. R., Wray, R. E., Lindes, P. & Laird, J. E. Improving knowledge extraction from llms for task learning through agent analysis. Proc. AAAI Conf. Artif. Intell. 38, 18390–18398 (2024). Google Scholar Dafoe, A. et al. Cooperative ai: machines must learn to find common ground (2021).Raji, I. D., Kumar, I. E., Horowitz, A. & Selbst, A. The fallacy of ai functionality. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, 959–972, (Association for Computing Machinery, New York, NY, USA, 2022) https://doi.org/10.1145/3531146.3533158.McIlroy-Young, R., Sen, S., Kleinberg, J. & Anderson, A. Aligning superhuman ai with human behavior: Chess as a model system. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, 1677–1687, (Association for Computing Machinery, New York, NY, USA, 2020) https://doi.org/10.1145/3394486.3403219.Munn, L. The uselessness of ai ethics. AI Ethics 3, 869–877 (2023).Article Google Scholar Li, B. et al. Trustworthy ai: From principles to practices. ACM Comput. Surv.[SPACE]https://doi.org/10.1145/3555803 (2023).Article Google Scholar Bostrom, N. The control problem. Excerpts from superintelligence: Paths, dangers, strategies. Sci. Fict. Philos. Time Travel Superintell. p. 308–330 (2016).Alfonseca, M. et al. Superintelligence cannot be contained: Lessons from computability theory. J. Artif. Intell. Res. 70, 65–76 (2021).Article MathSciNet Google Scholar Jebari, K. & Lundborg, J. Artificial superintelligence and its limits: Why alphazero cannot become a general agent. AI Soc. 36, 807–815 (2021).Article Google Scholar Sokol, K. & Flach, P. Counterfactual explanations of machine learning predictions: opportunities and challenges for ai safety. In 2019 AAAI Workshop on Artificial Intelligence Safety, SafeAI 2019 (CEUR Workshop Proceedings, 2019).Stepin, I., Alonso, J. M., Catala, A. & Pereira-Fariña, M. A survey of contrastive and counterfactual explanation generation methods for explainable artificial intelligence. IEEE Access 9, 11974–12001 (2021).Article Google Scholar Segura, S., Fraser, G., Sanchez, A. B. & Ruiz-Cortés, A. A survey on metamorphic testing. IEEE Trans. Software Eng. 42, 805–824 (2016).Article Google Scholar Roschelle, J., Lester, J., Fusco, J. (Expert panel report. Digital Promise, Ai and the future of learning, 2020).Li, Y. & Zhan, J. Saibench: Benchmarking ai for science. BenchCouncil Trans. Benchmarks Stand. Eval. 2, 100063 (2022).Article Google Scholar Karimi, A.-H., Barthe, G., Schölkopf, B. & Valera, I. A survey of algorithmic recourse: Contrastive explanations and consequential recommendations. ACM Comput. Surv. 55, 1–29 (2022).Article Google Scholar Jin, M. et al. What if llms have different world views: Simulating alien civilizations with llm-based agents. (2024) arXiv preprint arXiv:2402.13184.Kaelbling, L. P., Littman, M. L. & Moore, A. W. Reinforcement learning: A survey. J. Artif. Intell. Research 4, 237–285 (1996).Article Google Scholar Garcia, F. & Rachelson, E. Markov decision processes. Markov Decis. Process. Artif. Intell. p. 1–38 (2013).Zhu, Z., Lin, K., Jain, A. K. & Zhou, J. Transfer learning in deep reinforcement learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 45, 13344–13362. https://doi.org/10.1109/TPAMI.2023.3292075 (2023).Article PubMed PubMed Central Google Scholar Gronauer, S. & Diepold, K. Multi-agent deep reinforcement learning: A survey. Artif. Intell. Rev. 55, 895–943 (2022).Article Google Scholar Nahian, M. S., Frazier, S., Riedl, M. & Harrison, B. Training value-aligned reinforcement learning agents using a normative prior. IEEE Trans. Artif. Intell.[SPACE]https://doi.org/10.1109/TAI.2024.3363122 (2024).Article Google Scholar Dong, H. et al. Raft: Reward ranked finetuning for generative foundation model alignment (2023). arXiv:2304.06767.Ji, J. et al. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. In Oh, A. et al. (eds.) Advances in Neural Information Processing Systems, vol. 36, 24678–24704 (Curran Associates, Inc., 2023).Agarwal, R., Machado, M. C., Castro, P. S. & Bellemare, M. G. Contrastive behavioral similarity embeddings for generalization in reinforcement learning (2021). arXiv:2101.05265.Gu, S. et al. A review of safe reinforcement learning: Methods, theory and applications. (2022) arXiv preprint arXiv:2205.10330.Zheng, J. et al. Ali-agent: Assessing llms’ alignment with human values via agent-based evaluation (2024). arXiv:2405.14125.Köpf, A. et al. Openassistant conversations - democratizing large language model alignment. In Oh, A. et al. (eds.) Advances in Neural Information Processing Systems, vol. 36, 47669–47681 (Curran Associates, Inc., 2023).Hendrycks, D. et al. Aligning ai with shared human values (2023). arXiv:2008.02275.Goyal, N., Chang, M. & Terry, M. Designing for human-agent alignment: Understanding what humans want from their agents. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI’24, https://doi.org/10.1145/3613905.3650948 (ACM, 2024).Wolf, Y., Wies, N., Avnery, O., Levine, Y. & Shashua, A. Fundamental limitations of alignment in large language models (2024). arXiv:2304.11082.Kaddour, J. et al. Challenges and applications of large language models. (2023) arXiv preprint arXiv:2307.10169.Wei, A., Haghtalab, N. & Steinhardt, J. Jailbroken: How does llm safety training fail? Adv. Neural Inf. Process. Syst.36 (2024).Zou, A., Wang, Z., Kolter, J. Z. & Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. (2023) arXiv preprint arXiv:2307.15043.Salewski, L., Alaniz, S., Rio-Torto, I., Schulz, E. & Akata, Z. In-context impersonation reveals large language models’ strengths and biases. Adv. Neural Inf. Process. Syst. 36 (2024).Casper, S. et al. Open problems and fundamental limitations of reinforcement learning from human feedback. (2023) arXiv preprint arXiv:2307.15217.Yang, X. et al. Shadow alignment: The ease of subverting safely-aligned language models. (2023) arXiv preprint arXiv:2310.02949.Park, P. S., Goldstein, S., O’Gara, A., Chen, M. & Hendrycks, D. Ai deception: A survey of examples, risks, and potential solutions. Patterns5 (2024).McIntosh, T. R., Susnjak, T., Liu, T., Watters, P. & Halgamuge, M. N. From google gemini to openai q*(q-star): A survey of reshaping the generative artificial intelligence (ai) research landscape. (2023) arXiv preprint arXiv:2312.10868.McIntosh, T. R., Susnjak, T., Liu, T., Watters, P. & Halgamuge, M. N. The inadequacy of reinforcement learning from human feedback - radicalizing large language models via semantic vulnerabilities. IEEE Trans. Cognitive Dev. Syst.[SPACE]https://doi.org/10.1109/TCDS.2024.3377445 (2024).Article Google Scholar Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. & Yao, S. Reflexion: language agents with verbal reinforcement learning. In Oh, A. et al. (eds.) Advances in Neural Information Processing Systems, vol. 36, 8634–8652 (Curran Associates, Inc., 2023).Zhou, C. et al. Lima: Less is more for alignment. In Oh, A. et al. (eds.) Advances in Neural Information Processing Systems, vol. 36, 55006–55021 (Curran Associates, Inc., 2023).Ngo, R., Chan, L. & Mindermann, S. The alignment problem from a deep learning perspective: A position paper. In The Twelfth International Conference on Learning Representations (2023).Rowa, K. Atychiphobia (fear of failure) (The Psychology of Irrational Fear, Phobias, 2015). Google Scholar Korb, K. B. & Nicholson, A. E. Bayesian artificial intelligence (CRC Press, 2010).Book Google Scholar Smets, P. Belief functions versus probability functions. In International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, 17–24 (Springer, 1988).Li, Y. Deep reinforcement learning: An overview. (2017) arXiv preprint arXiv:1701.07274.Cuzzolin, F., Morelli, A., Cirstea, B. & Sahakian, B. J. Knowing me, knowing you: Theory of mind in ai. Psychol. Med. 50, 1057–1061 (2020).Article CAS PubMed PubMed Central Google Scholar Williams, J., Fiore, S. M. & Jentsch, F. Supporting artificial social intelligence with theory of mind. Front. Artif. Intell. 5, 750763 (2022).Article PubMed PubMed Central Google Scholar Yampolskiy, R. V. On monitorability of ai. AI and Ethics 1–19 (2024).Pan, A. et al. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on Machine Learning, 26837–26867 (PMLR, 2023).Bu, Y., Zou, S., Liang, Y. & Veeravalli, V. V. Estimation of kl divergence: Optimal minimax rate. IEEE Trans. Inf. Theory 64, 2648–2674 (2018).Article MathSciNet Google Scholar Download referencesAuthor informationAuthors and AffiliationsInternational School of Engineering, Chulalongkorn University, Bangkok, ThailandPaulo GarciaAuthorsPaulo GarciaView author publicationsSearch author on:PubMed Google ScholarContributionsP.G. was responsible for all the conceptualization, formulation, and writing.Corresponding authorCorrespondence to Paulo Garcia.Ethics declarations Competing interests The author declares no competing interests. Additional informationPublisher's noteSpringer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.Rights and permissions Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. Reprints and permissionsAbout this articleCite this articleGarcia, P. Aversion to external feedback suffices to ensure agent alignment. Sci Rep 14, 21147 (2024). https://doi.org/10.1038/s41598-024-72072-0Download citationReceived: 23 February 2024Accepted: 02 September 2024Published: 10 September 2024Version of record: 10 September 2024DOI: https://doi.org/10.1038/s41598-024-72072-0Share this articleAnyone you share the following link with will be able to read this content:Get shareable linkSorry, a shareable link is not currently available for this article.Copy shareable link to clipboard Provided by the Springer Nature SharedIt content-sharing initiative Comparing discriminatory behavior against AI and humans Article Open access 29 March 2025 Delegation to artificial intelligence can increase dishonest behaviour Article Open access 17 September 2025 Committing to the wrong artificial delegate in a collective-risk dilemma is better than directly committing mistakes Article Open access 07 May 2024

Instant research discovery