Paper deep dive
AI Alignment through a Game-theoretic Lens: A Survey
Yanan Cai, Zhongrui Zhao, Zhigang Lu, Ickjai Lee, Wei Emma Zhang, Minhui Xue, Yihong Zhang, Shuchao Pang, Wei Xiang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/31/2026, 3:39:47 AM
Summary
This survey paper proposes a game-theoretic framework for analyzing AI alignment, specifically focusing on post-training alignment of Large Language Models (LLMs). It identifies three core challenges: preference diversity, alignment priority, and temporal dynamics. The paper formalizes alignment as a strategic optimization problem, interpreting Nash equilibrium as a stable aligned policy and using duality gaps to measure approximation quality. It categorizes existing methods into general vs. heterogeneous preferences, synchronous vs. sequential interactions, and adaptive vs. co-evolutionary dynamics, providing a unified lens to evaluate robustness, fairness, and adaptability in AI systems.
Entities (12)
Relation Signals (8)
AI Alignment → analyzedby → Game Theory
confidence 95% · This survey reviews AI alignment through a game-theoretic lens.
AI Alignment → faceschallenge → Preference Diversity
confidence 95% · synthesizes the literature along three challenges: preference diversity...
AI Alignment → faceschallenge → Alignment Priority
confidence 95% · synthesizes the literature along three challenges: ... alignment priority...
AI Alignment → faceschallenge → Temporal Dynamics
confidence 95% · synthesizes the literature along three challenges: ... and temporal dynamics.
Nash Equilibrium → interpretedas → Aligned Policy
confidence 93% · We interpret π∗ as the aligned policy because it satisfies the saddle-point property of the equilibrium.
Duality Gap → measures → Approximation Quality
confidence 92% · DualGap(π) quantifies how far π is from the aligned Nash policy.
Yanan Cai → affiliatedwith → James Cook University
confidence 90% · Affiliation: James Cook University
RLHF → improves → Helpfulness
confidence 90% · Methods such as reinforcement learning from human feedback (RLHF) ... have improved model helpfulness
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As large language models and increasingly capable AI agents are deployed in high-risk settings, aligning them with complex human values has become a central challenge. Existing alignment methods, while effective in improving helpfulness, harmlessness, and controllability, often struggle to capture real-world preferences that are context-dependent, non-transitive, and shaped by dynamic multi-party interactions. This survey reviews AI alignment through a game-theoretic lens. Specifically, it organizes recent progress around key game-theoretic elements and synthesizes the literature along three challenges: preference diversity, alignment priority, and temporal dynamics. This perspective clarifies where current alignment methods genuinely benefit from game-theoretic analysis, where the framework is looser, and what challenges remain in building robust, adaptive, and verifiable AI systems.
Tags
Links
- Source: https://arxiv.org/abs/2608.27910v1
- Canonical: https://arxiv.org/abs/2608.27910v1
Trouble viewing inline? Open PDF directly →
Full Text
92,432 characters extracted from source content.
Expand or collapse full text
AI Alignment through a Game-theoretic Lens: A Survey Yanan Cai †thanks: Equal contribution. Affiliation: James Cook University Affiliation: Western Sydney University Zhongrui Zhao11footnotemark: 1 Affiliation: James Cook University Affiliation: Western Sydney University Zhigang Lu †thanks: Corresponding author. Affiliation: Western Sydney University Ickjai Lee Affiliation: James Cook University Wei Emma Zhang Affiliation: Adelaide University Minhui Xue Affiliation: Adelaide University Affiliation: CSIRO Yihong Zhang Affiliation: The University of Osaka Shuchao Pang Affiliation: Nanjing University of Science and Technology Affiliation: Macquarie University Wei Xiang Affiliation: La Trobe Universityyanan.cai, zhongrui.zhao@my.jcu.edu.au, z.lu@westernsydney.edu.au Abstract As large language models and increasingly capable AI agents are deployed in high-risk settings, aligning them with complex human values has become a central challenge. Existing alignment methods, while effective in improving helpfulness, harmlessness, and controllability, often struggle to capture real-world preferences that are context-dependent, non-transitive, and shaped by dynamic multi-party interactions. This survey reviews AI alignment through a game-theoretic lens. Specifically, it organizes recent progress around key game-theoretic elements and synthesizes the literature along three challenges: preference diversity, alignment priority, and temporal dynamics. This perspective clarifies where current alignment methods genuinely benefit from game-theoretic analysis, where the framework is looser, and what challenges remain in building robust, adaptive, and verifiable AI systems. Figure 1: Alignment scale process (top panel) and three core challenges (bottom): preference diversity, alignment priority, and temporal dynamics. 1 Introduction As large language models (LLMs) and AI agents are increasingly deployed in real-world settings Xi et al. (2025), security and governance risks raised by LLMs and AI agents have received growing attention Duenas and Ruiz (2024); Milano and Nyholm (2024). Reported concerns include bias, toxicity, manipulation, and misrepresentation Duan et al. (2025); Gallegos et al. (2024); Guo (2024), as well as deception, reward hacking, and power-seeking behavior under imperfect oversight Pan et al. (2022); Park et al. (2024); Turner and Tadepalli (2022). AI alignment is therefore an urgent goal. It aims to ensure that AI systems not only perform tasks effectively, but also act consistently with human intentions, social norms, and ethical constraints Shen et al. (2024). Although alignment can be applied during training or inference, this survey mainly focuses on post-training alignment (top solid box in Fig. 1), which adapts foundation models to human preferences or deployment needs without retraining them from scratch, while encoding behavioral changes into model parameters to avoid repeated inference-time intervention Lai et al. (2025); Ouyang et al. (2022). Methods such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) have improved model helpfulness, harmlessness, and controllability Ouyang et al. (2022); Rafailov et al. (2023). However, they still struggle when proxy rewards differ from intended goals, parties involved in the alignment process have conflicting preferences (e.g., when responding to a controversial social question, some users may prefer an open and direct answer, while others may prioritize harm avoidance and the prevention of stereotypes), or static objectives fail under repeated interaction and environmental changes Munos et al. (2024); Tang et al. (2025); Chu et al. (2025); Ye et al. (2024a). We group these limitations into three core challenges: preference diversity, alignment priority, and temporal dynamics. The core challenges mentioned above motivate a promising and more structural perspective on alignment via game theory Tang et al. (2025); Chu et al. (2025); Munos et al. (2024); Ye et al. (2024a). Specifically, preference diversity arises since values differ across human groups, e.g., ethnic or political groups, making a single tractable and socially legitimate objective hard to define Gabriel (2020); Floridi and Cowls (2022); alignment priority reflects multiple levels of alignment, including reward proxies, optimization targets, user preferences, and higher legal or institutional constraints, which may conflict Amodei et al. (2016); Kleinberg et al. (2016); furthermore, temporal dynamics need to reflect the evolution of human values, social environment, and AI capabilities Carroll et al. (2024); Qiu et al. (2024). In this sense, we aim to use the game-theoretic language to characterize and explain a series of application scenarios and uncertainties in the alignment problem, by analyzing some game elements such as players, utilities, information, strategies, equilibrium, and dynamics Gemp et al. (2024); Mozikov et al. (2024). To avoid overstating the role of game theory, we also clarify what game-theoretic analysis can formally guarantee and under which assumptions rather than claiming that game theory universally solves alignment. In summary, we give the main contributions of this survey below: • We organize a three-dimensional analysis framework of AI alignment by challenge one - preference diversity, challenge two - alignment priority, and challenge two - temporal dynamics for helping researchers clarify the current alignment landscape. • We provide a unified game-theoretic perspective for AI alignment, which is used to interpret and formalize specific alignment goals and application scenarios, based on the mathematical foundations and theoretical results of game theory. • We summarize six alignment directions under the challenge: general preference or heterogeneous preference, synchronous interaction or sequential interaction, and adaptive evolution or co-evolution, encouraging new insights within the AI alignment community, which are inspired by concepts such as players, strategy interaction, and temporal factor. 2 Related Work Recent research has begun to apply the game-theoretic perspective to preference conflict, equilibrium-based alignment, cooperative AI, online adaptation, and pluralistic evaluation Tang et al. (2025); Chu et al. (2025); Ye et al. (2024a); Mao et al. (2025), but these efforts remain scattered across different scene settings and research communities. Additionally, some surveys have reviewed AI alignment broadly, emphasizing objectives, training pipelines, and governance Ji et al. (2023); Sun et al. (2024b); Shen et al. (2024); Kayabay et al. (2025), while others focus on LLM agents and multi-agent systems, organizing the literature around architectures, workflows, and deployment challenges Wang et al. (2024e); Li et al. (2024c); Zhou et al. (2025a). Recent work on game theory and LLMs, meanwhile, highlights the broader intersection between language models and games, including benchmarking, strategic behavior, and social simulation Sun et al. (2025). Our survey is differentiated from previous surveys by both its scope and organizing principle: rather than surveying all alignment phases or game-related research, we mainly focus on post-training alignment approaches, and ask how game theory determines the formulation, guarantee, or evaluation of alignment problems. Alignment meets game theory Challenge One: Preference diversity (Sec. 4.1) General preference Heterogeneous preference Non-game: best response toward a fixed policy Two-player game Non-game Multi-player game: preference aggregation Existence convergence Average-iterate convergence Last-iterate convergence Social welfare optimization Multi-objective optimization IPO Azar et al. (2024) Online Iterative RLHF Ye et al. (2024a) SPOSwamy et al. (2024), DNO Rosset et al. (2024), REBEL Gao et al. (2024), SPPO Wu et al. (2024), TANPO Wang et al. (2025b), ONPO Zhang et al. (2025a) NLHF Munos et al. (2024), IPO-MD Calandriello et al. (2024), INPO Zhang et al. (2025b), COMAL Liu et al. (2026), MPO Wang et al. (2025a), RSPO Tang et al. (2025) Chidambaram et al. (2024), Kim et al. (2025), MOP Xiong and Singh (2025), Pessimistic Nash Bargaining Zhong et al. (2024a), GRPO Ramesh et al. (2024) CLP Wang et al. (2024d), Panacea Zhong et al. (2024b), MO-ODPO Gupta et al. (2025a), SIPO Li et al. (2025), DPA Wang et al. (2024c), DRM Luo et al. (2025) RLHF Game Sun et al. (2024a), Dynamic Bayesian Game Formulation Hao and Duan (2025), Battling Influencers Game Wu et al. (2025), ⟨M, N , ϵε, δ⟩-AgreementNayebi (2025) Challenge Two: Alignment priority (Sec. 4.2) Synchronous interaction Sequential interaction Attacker-defender: self-generated data Generator-evaluator: self-annotated data Leader-follower: self-annotated data Orderly cooperation: external annotated data Jatova et al. (2024), GPO Zheng et al. (2024), SELF-REDTEAM Liu et al. (2025), MTSAGuo et al. (2025), SPAG Cheng et al. (2024) APO Cheng et al. (2023), SPIN Chen et al. (2024b), LANA Azarafrooz and Faal (2024), DuoGuard Deng et al. (2025), SPC Chen et al. (2025b), PEG Chen et al. (2025a) STA-RLHF Makar-Limanov et al. (2024), SPAC Ji et al. (2024), SGPO Chu et al. (2025) Anyprefer Zhou et al. (2025c), CORY Ma et al. (2024) Challenge Three: Temporal dynamics (Sec. 4.3) Adaptive evolution Co-evolution Trajectory-based Selection-based Game-theoretic algorithm modeling Scenario simulation Social activities Strategic games Non-MDP / POMDP Evolutionary game Creator-solver game Klassen et al. (2024), ProgressGym Qiu et al. (2024) Suzuki and Arita (2024), Evolutionary Agent Li et al. (2024b) EVA Ye et al. (2024b) SPARTAJiang et al. (2025), Red-Team Game Ma et al. (2023) SSI Liu et al. (2024b), Artificial Leviathan Dai et al. (2024), CAMEL Li et al. (2023), AI Collectives Lai et al. (2024) Du et al. (2024), Alympics Mao et al. (2025), FairMindSim Lei et al. (2024), MAD Liang et al. (2024) Figure 2: Taxonomy of alignment based on a game theory lens. 3 Preliminaries In this section, we use game-theoretic language to characterize when alignment can be formulated as a strategic optimization problem, when an equilibrium can be interpreted as an aligned policy, and when convergence guarantees and equilibrium gaps can serve as formal certificates of approximate alignment Munos et al. (2024); Ye et al. (2024a); Wu et al. (2024); Zhang et al. (2025b); Pásztor et al. (2025); Shi et al. (2025). 3.1 General Alignment Guarantee Since two-player game theory is the most common formalization of alignment problems, we take its modeling of general preference scenarios as a basis to interpret how to achieve theoretical alignment by pursuing game equilibrium. 3.1.1 Problem Formulation As noted earlier, aligning LLMs typically relies on learning from human preference feedback. Rather than providing a scalar reward signal r(x,y)r(x,y), human feedback more often takes the form of a pairwise comparison signal Pr(y1≻y2∣x) (y_1 y_2 x) denoting the probability that y1y_1 is preferred over y2y_2 given prompt x, where ≻ indicates a preference relation. Accordingly, preference learning more closely matches practical RLHF settings and is better suited to tasks that are difficult to specify using a single numerical reward. In this setting, we seek a policy that performs robustly under human pairwise comparisons against alternative policies. Specifically, consider the following objective J(π1,π2) J( _1, _2) =x∼d0y1∼π1,y2∼π2[ℙ(x,y1,y2) =E_x d_0E_y_1 _1,\;y_2 _2 [P(x,y_1,y_2) (1) −τDKL(π1(⋅∣x)∥πref(⋅∣x)) -τ D_KL\! ( _1(· x)\,\|\, _ref(· x) ) +τDKL(π2(⋅∣x)∥πref(⋅∣x))], +τ D_KL\! ( _2(· x)\,\|\, _ref(· x) ) ], where d0d_0 is a prompt distribution. Here, ℙ(x,y1,y2)P(x,y_1,y_2) is a payoff term instantiated usually as a preference model: ℙ(x,y1,y2)=Pr(y1≻y2∣x)∈[0,1]P(x,y_1,y_2)= (y_1 y_2 x)∈[0,1] (In many alignment scenarios, the payoff can also be a point reward or other forms). The KL terms regularize both policies toward a reference policy (πref _ref denotes a fixed conditional language-model distribution, typically instantiated as the initial supervised fine-tuned model, i.e., πref=πSFT _ref= _SFT), thereby stabilizing learning and controlling deviation from the instruction-following behavior Rafailov et al. (2023). When the payoff term ℙP satisfies Pr(y1≻y2∣x)+Pr(y2≻y1∣x)=1 (y_1 y_2 x)+ (y_2 y_1 x)=1, the above objective function defines a two-player constant-sum game, i.e., J(π1,π2)+J(π2,π1)=1J( _1, _2)+J( _2, _1)=1, where the players choose policies π1 _1 and π2 _2. In particular, ℙP further can take the centered form of ℙ(x,y1,y2)=Pr(y1≻y2∣x)−12P(x,y_1,y_2)= (y_1 y_2 x)- 12, with a two-player zero-sum game Liu et al. (2026). Based on games, the first player maximizes J(π1,π2)J( _1, _2), while the second player minimizes it. Intuitively, π1 _1 seeks a policy that is preferred under human comparisons, whereas π2 _2 acts as an adversarial comparator. The alignment problem can therefore be formulated as the saddle-point problem (π1∗,π2∗)∈argmaxπ1∈Πargminπ2∈ΠJ(π1,π2),( _1^*, _2^*)∈ _ _1∈ _ _2∈ J( _1, _2), (2) where both players optimize over the same policy class Π . 3.1.2 Nash Equilibrium as Strategic Alignment For analysis, we restrict attention to policies in Π that have the same support as πref _ref. Under this assumption, Ye et al. (2024a) shows that the game admits a unique Nash equilibrium. By symmetry, the equilibrium policies coincide, i.e., π1∗=π2∗=π∗ _1^*= _2^*=π^*. We interpret π∗π^* as the aligned policy because it satisfies the saddle-point property of the equilibrium. For any alternative policy π∈Ππ∈ , we have J(π,π∗)≤J(π∗,π∗)≤J(π∗,π).J(π,π^*)≤ J(π^*,π^*)≤ J(π^*,π). Hence, no policy can systematically obtain a better payoff against π∗π^* than π∗π^* obtains against itself, while π∗π^* performs at least as well against any opponent as it does against itself. Further, when J(π1,π2)J( _1, _2) is the unregularized objective with a centered pairwise-preference payoff, since ℙ(x,y1,y2)P(x,y_1,y_2) is antisymmetric, we have J(π∗,π∗)=0J(π ,π )=0. The saddle-point property then gives, for any π∈Ππ∈ , J(π∗,π)≥J(π∗,π∗)=0J(π ,π)≥ J(π ,π )=0. Based on the form of ℙ(x,y1,y2)P(x,y_1,y_2), the win rate of π∗π^* against any π is at least one half. Under this view, alignment is not defined as maximizing an unobserved utility function. Rather, it is defined as non-domination under the induced preference relation. In this sense, an aligned policy is strategically stable under pairwise comparison and resistant to exploitation by competing policies. 3.1.3 Approximation and Convergence To assess how well a candidate policy approximates π∗π^*, we use the following duality gap Zhang et al. (2025b): DualGap(π):=maxπ1∈ΠJ(π1,π)−minπ2∈ΠJ(π,π2).DualGap(π):= _ _1∈ J( _1,π)- _ _2∈ J(π, _2). (3) The duality gap is always nonnegative, and DualGap(π)=0DualGap(π)=0 if and only if π=π∗π=π^*. Thus, DualGap(π)DualGap(π) quantifies how far π is from the aligned Nash policy. A small duality gap means that π is only weakly exploitable in the induced preference game, providing a formal certificate of approximate alignment. When DualGap(π)≤ϵDualGap(π)≤ε, π may be regarded as an ϵε-approximate Nash policy. Related definitions and guarantees also appear in Ye et al. (2024a); Ma et al. (2023). The practical significance of this equilibrium formulation depends on whether the learning dynamics (policy updates induced by the alignment procedure itself) converge to such a stable point. Recent works Munos et al. (2024); Wang et al. (2024f); Liu et al. (2026) establish convergence guarantees for several algorithms, including average-iterate convergence of 1T∑t=1Tπt 1T _t=1^T _t. Moreover, if an algorithm enjoys last-iterate convergence, then the final output policy πT _T itself can be interpreted as a strategically aligned solution, and the resulting model inherits the same stability interpretation Tiapkin et al. (2025). That implies that alignment is not merely observed empirically, but rather the limit of a strategic learning process. 3.2 Complex Alignment Extensions The theoretical form described above cannot strictly cover all scenarios, and more complex alignment settings typically relax one or more of classical game theory assumptions by considering additional players, heterogeneous policy spaces, asymmetric roles, action order, or time dependence. For example, alignment involving the preferences of multiple parties can be modeled as a multi-player game, denoted by =⟨,Πii∈,Uii∈⟩,G= ,\ _i\_i ,\U_i\_i , where =1,…,nN=\1,…,n\ is the set of players, Πi _i is the strategy space of player i, and Ui(π)U_i(π) is its utility (analogous to the payoff term in Eq. (1)) under the joint strategy profile π=(π1,…,πn)π=( _1,…, _n). A joint strategy π∗π is a Nash equilibrium if Ui(πi∗,π−i∗)≥Ui(πi,π−i∗),∀i∈,πi∈Πi,U_i ( _i , _-i )≥ U_i ( _i, _-i ),∀ i ,\ _i∈ _i, which simulates how each player’s payoff depends on the joint strategy combination. Some formulations also define an individual loss ℓi=−Ui _i=-U_i that each player minimizes. In this case, the same equilibrium condition is written with ≤, as in Wu et al. (2025). Additionally, social welfare or multi-objective optimization are related to non-game extensions, combining player-specific utilities through selected aggregation rules to optimize this global goal. Furthermore, the two-player game of general preference alignment can be generalized by relaxing its assumptions about player roles, shared policy space, and action order. With asymmetric roles in adversarial scenarios Zheng et al. (2024); Chen et al. (2025b), a representative formulation is: maxπ1∈Π1minπ2∈Π2J(π1,π2), _ _1∈ _1 _ _2∈ _2J( _1, _2), where Π1 _1 and Π2 _2 may represent different strategies, such as attackers vs. defenders in Zheng et al. (2024) or generators vs. critics in Chen et al. (2025b). Its Nash equilibrium is a saddle point satisfying J(π1,π2∗)≤J(π1∗,π2∗)≤J(π1∗,π2).J( _1, _2 )≤ J( _1 , _2 )≤ J( _1 , _2). When Π1=Π2 _1= _2, and the two simultaneous actions are aligned with the same preference objective, this reduces to the two-policy formulation similar to Eq. (2). If one of the two players observes moves after observing the other Chu et al. (2025), the solution becomes a Stackelberg equilibrium with the following nested form: π2∗(π1)∈argmaxπ2∈Π2J2(π1,π2), _2 ( _1)∈ _ _2∈ _2J_2( _1, _2), π1∗∈argmaxπ1∈Π1J1(π1,π2∗(π1)). _1 ∈ _ _1∈ _1J_1 ( _1, _2 ( _1) ). This expression assumes a selected follower best response. Sequential cooperation can alternatively be modeled as a cooperative Markov game with a shared objective, which does not imply the adversarial saddle-point condition above Zhou et al. (2025c). In general, equilibrium existence and min-max equality require assumptions determined by the game form, such as finite or compact strategy spaces, continuity, and appropriate convexity or concavity conditions Nash Jr (1950). Finally, temporal extensions distinguish repeated strategic interaction from changes in the alignment environment itself. When two sides adapt strategically over multiple rounds, the interaction can be represented as a finite-horizon extensive-form game. An ϵε-Nash policy satisfies Uij(πi∗,π−i∗)≥Uij(πi,π−i∗)−ϵ,∀i∈1,2,πi∈Πi,U_i^j( _i , _-i )≥ U_i^j( _i, _-i )-ε,∀ i∈\1,2\,\ _i∈ _i, where UijU_i^j is the payoff accumulated over j rounds Ma et al. (2023) (alternatively, denoted by a time-dependent preference/reward model with ℙtP_t). This extends the one-shot equilibrium concept in Eq. (2), but does not require the same pairwise payoff or constant-sum structure. Furthermore, when a single AI agent adapts to evolving human values, a representative formulation is the value-state POMDP ℳ=⟨,,,Ω,O,U⟩,U:(×)∗→ℝ,M= ,A,T, ,O,U ,U:(S×A)^* , where S is the space of human value states (alternatively, simulated by the distribution of human value datasets d0d_0), A is the action space of the aligned policy, T is the value-state transition model, Ω is the observation space, O is the observation model, and U maps an adaptive trajectory to a scalar alignment evaluation Qiu et al. (2024). 4 Taxonomy: A Game-theoretic Lens on Alignment Frameworks Having gained a basic understanding of how game theory explains alignment, this section will summarize the methods for different alignment problems based on the three major challenges. 4.1 Challenge One: Preference Diversity Preference alignment studies how AI systems are optimized from feedback. In practice, this process usually involves three components: the AI system to be aligned, the feedback source, and a proxy that models such feedback for learning, e.g., a reward or preference model Ji et al. (2023). This section focuses on the structure and diversity of preferences. A game-theoretic lens becomes especially useful when alignment depends either on strategic robustness against alternative policies or on fair aggregation across stakeholders with conflicting values. We therefore discuss two related settings: general preference, where the key question is robustness under comparison, and heterogeneous preferences, where the main issue is how to aggregate diverse human interests in a fair and stable way. 4.1.1 General preference Figure 3: General preference alignment: pairwise optimization and game-theoretic formulation. Heterogeneous preference alignment: multi-group feedback and preference aggregation. Non-game. General preference alignment aims to optimize model behavior toward broadly shared human judgments, as shown in Fig. 3(a). Early work mainly provided theoretical perspectives on learning from human preferences Wang et al. (2023b), but did not directly capture practical LLM alignment under pairwise feedback. A first line of work is non-game optimization against a fixed policy. The representative example is IPO Azar et al. (2024), which directly optimizes pairwise preferences without assuming an explicit reward model: maxπy∼π,y′∼μ[ℙ(y≻y′)]−τKL(π∥πref). _πE_y π,\,y μ [P(y y ) ]-τ\,KL(π\| _ref). Compared with point-wise reward optimization, IPO better matches preference learning. However, unlike the minimax games, IPO optimizes only the target policy π as a best response to a predefined reference distribution μ, rather than jointly optimizing two competing policies. Specifically, μ plays a role analogous to π2 _2 in Eq. (2), but is fixed in advance and is neither learned nor updated during training. Therefore, IPO does not guarantee robustness against stronger competing policies. Two-player game. This limitation motivates a second line of work that formulates alignment as a two-player game. Representative studies such as NLHF Munos et al. (2024), IPO-MD Calandriello et al. (2024), and iterative RLHF Ye et al. (2024a) cast alignment as a constant-sum or minimax game, where the target policy and alternative policy are optimized in competition with each other under a learned preference function, updating alternately through self-play in practice. Early results mainly established the existence of Nash-style aligned policies. Subsequent work focused on stronger convergence guarantees. One branch studies average-iterate convergence, including SPO Swamy et al. (2024), DNO Rosset et al. (2024), REBEL Gao et al. (2024), SPPO Wu et al. (2024), TANPO Wang et al. (2025b), and ONPO Zhang et al. (2025a). These methods often replace explicit win-rate estimation with simpler regression-style objectives or incorporate online exploration to improve sample efficiency. Another branch studies last-iterate convergence, which is more desirable because the final learned policy itself inherits the equilibrium interpretation. Representative examples include INPO Zhang et al. (2025b), COMAL Liu et al. (2026), MPO Wang et al. (2025a), and RSPO Tang et al. (2025). Overall, general preference alignment has evolved from fixed-opponent optimization to self-play Nash learning, with increasing emphasis on convergence, robustness, and the role of regularization. 4.1.2 Heterogeneous preferences Heterogeneous preference alignment concerns settings where stakeholders hold overlapping or conflicting values, as shown in Fig. 3(b). Unlike general preference alignment, the goal is not to fit a single dominant preference, but to balance multi-party, objectives, and minority concerns. Non-game. A first category is non-game alignment. One line formulates the problem as social welfare optimization, where group utilities are aggregated through welfare or bargaining objectives Mishra (2023); Conitzer et al. (2024); Qiu (2024). Representative methods include MOP Xiong and Singh (2025), Pessimistic Nash Bargaining Zhong et al. (2024a), and group-robust approaches Chidambaram et al. (2024); Ramesh et al. (2024); Kim et al. (2025). These methods improve fairness and reduce dominance, but their behavior depends strongly on the chosen welfare function. Another line formulates alignment as multi-objective optimization. Methods such as CLP Wang et al. (2024d), Panacea Zhong et al. (2024b), MO-ODPO Gupta et al. (2025a), and SIPO Li et al. (2025) jointly optimize multiple objectives and often support controllable trade-offs through preference vectors or conditional signals. Related work also studies decomposed or vector-valued reward representations for more interpretable preference control Wang et al. (2024c); Luo et al. (2025). Multi-player game. A second category treats heterogeneous alignment as a multi-player game over preference aggregation. Here the main issue is that stakeholders may strategically report preferences to influence the final model. RLHF Game Sun et al. (2024a) and Dynamic Bayesian Game Hao and Duan (2025) explicitly study truthful reporting and weighting mechanisms, while BIG Wu et al. (2025) analyzes strategic influence among multiple parties. Other work further examines multi-preference optimization and the complexity of alignment across multiple objectives and agents under weak assumptions Gupta et al. (2025b); Nayebi (2025). Together, these studies show that heterogeneous preference alignment is not only an optimization problem, but also a mechanism-design problem shaped by strategic human behavior. 4.2 Challenge Two: Alignment Priority Whereas the previous subsection organizes the literature by preference structure, this section shifts the focus to interaction structure. AI alignment is increasingly understood as an interactive process rather than a one-shot transfer of human preferences to a model Shen et al. (2024). Many methods improve alignment not only by fitting feedback, but also by generating informative feedback through play, critique, or role-dependent adaptation. The main distinction here is whether strategic updates occur within a shared decision round or under an explicit order of moves. We therefore distinguish between synchronous interaction, which is especially useful for adversarial stress testing and self-improvement, and sequential interaction, which introduces commitment, asymmetry, and bilevel optimization into alignment. Figure 4: Synchronous interaction for alignment: simultaneous play, adversarial roles, and alignment targets. Sequential interaction for alignment: ordered roles and commitment-based optimization. 4.2.1 Synchronous interaction When interaction is synchronous, both sides act within the same round, and alignment is improved through repeated exposure to hard cases generated during play, as shown in Fig. 4(a). Attacker-defender This setting has been particularly influential in safety alignment. GPO Zheng et al. (2024) formulates alignment as a max–min game between an attacker and a defender, showing that adversarial prompt generation can uncover vulnerabilities while improving robustness through iterative updates. Closely related work applies the same intuition to toxicity mitigation, guardrail training, and jailbreak robustness, either by training separate attacker–defender models or by alternating these roles within a single model Jatova et al. (2024); Deng et al. (2025); Liu et al. (2025); Guo et al. (2025). Across these studies, synchronous play functions less as an abstract game metaphor than as a practical mechanism for generating stress cases that standard alignment pipelines would otherwise miss. Generator-evaluator. The same interaction pattern has also been repurposed to reduce dependence on human annotation. APO Cheng et al. (2023) couples the policy with a reward model in an adversarial loop to address the mismatch between evolving model outputs and static preference supervision. SPIN Chen et al. (2024b) instead treats self-improvement as iterative self-distillation, while LANA Azarafrooz and Faal (2024) further shows that self-rewarding can remain effective even under noisy or suboptimal demonstrations. What connects these methods is that interaction does not merely refine the policy; it also becomes a means of producing or filtering the supervision signal itself. This interactive template has further extended to reasoning and truthfulness. SPC Chen et al. (2025b) uses adversarial self-play between a generator and critic to improve step-level reasoning and support test-time search, whereas SPAG Cheng et al. (2024) relies on adversarial language games to strengthen reasoning-oriented dialogue. PEG Chen et al. (2025a) pushes the idea toward truthful evaluation by casting peer assessment as a game in which truthful reporting is incentivized. Taken together, these studies suggest that synchronous interaction is most useful when alignment requires continual pressure from challenging inputs, self-generated supervision, or competing evaluations. Its limitations, however, remain clear: many methods are computationally intensive, sensitive to training dynamics, and still rely on assumptions that are difficult to satisfy in realistic black-box settings. 4.2.2 Sequential interaction Leader-follower. Once the order of moves becomes explicit, interaction changes character. The earlier player can commit to a strategy that reshapes the response of the later player in Fig. 4(b), making this setting closer to bilevel or Stackelberg optimization than to simultaneous self-play. This perspective has been especially useful for preference learning. STA-RLHF Makar-Limanov et al. (2024) models the language model and reward model as leader and follower, respectively, and searches for a Stackelberg equilibrium under nested optimization. SPAC Ji et al. (2024) preserves the same leader–follower structure but replaces multi-stage updates with a single-timescale procedure, yielding stronger practical appeal in offline settings with sparse preference data. Orderly cooperation. A similar logic also appears when sequential interaction is used to improve data quality and robustness. SGPO Chu et al. (2025) studies alignment under scarce and noisy preferences through a Stackelberg-style formulation with regret guarantees under bounded distribution shift. Anyprefer Zhou et al. (2025c) combines automatic synthesis, external tools, and a cooperative Markov game to mitigate self-reward bias while increasing sample diversity. CORY Ma et al. (2024) goes a step further by using sequential cooperative multi-agent reinforcement learning and role exchange to stabilize fine-tuning. In these settings, ordered interaction is valuable not only because it captures asymmetric roles more faithfully, but also because it offers a natural way to control how evaluators, generators, and auxiliary tools influence one another over time. These interaction-based approaches move alignment beyond static supervision, but they also expose a broader challenge: alignment objectives, evaluators, and environments may all evolve over time. We therefore turn next to dynamic evolution in alignment. 4.3 Challenge Three: Temporal dynamic The previous two sections focus on preference structure and interaction structure at a given stage of learning or deployment. We now turn to a third aspect that cuts across both: temporal change. Once systems interact repeatedly with humans, other agents, and changing environments, alignment can no longer be treated as a one-shot objective fixed at training time. Instead, the central challenge is to maintain behavioral consistency as preferences, incentives, and external conditions evolve. In this sense, dynamic evolution is not a separate alignment category so much as a temporal extension of the preceding discussion. This has motivated a growing body of work that studies alignment through adaptive evolution and co-evolution. 4.3.1 Adaptive evolution Adaptive evolution focuses on how AI systems update under temporally varying social environments, as illustrated in Fig. 5(a). Figure 5: Adaptive evolution in alignment: changing social environments and a typical update framework. Co-evolution in alignment: multiple agents evolving through competition, cooperation, and social interaction. Trajectory-based evolution. One line of work formulates this problem through sequential decision-making or trajectory-based evaluation. ProgressGym Qiu et al. (2024), for example, frames alignment as moral progress over time and studies how historical moral data can guide future decisions. Relatedly, Klassen et al. Klassen et al. (2024) propose a temporal pluralistic alignment framework that evaluates stakeholder satisfaction over trajectories rather than isolated outputs, allowing alignment to depend on long-horizon and non-Markovian requirements. Selection-based evolution. A complementary line introduces evolutionary mechanisms such as mutation, selection, and environmental variation. Li et al. Li et al. (2024b) embed social norms into a genetic process so that agents adapt through repeated selection under changing group behavior, while Suzuki and Arita Suzuki and Arita (2024) use evolutionary games to study the emergence of cooperative traits in LLM-based agents. Ye et al. Ye et al. (2024b) further model prompt mutation as a way to simulate novel environments and formulate alignment as a creator–solver game. Across these studies, the common intuition is that alignment should remain responsive to gradual changes in norms, tasks, and social expectations, rather than being anchored to a static preference snapshot. 4.3.2 Co-evolution Whereas adaptive evolution emphasizes how a model tracks environmental change, co-evolution focuses on settings in which the environment itself is partly generated by other evolving agents, as shown in Fig. 5(b). Game-theoretic algorithm modeling. This perspective is particularly useful when alignment depends on multi-agent interaction, population diversity, or open-ended strategic adaptation. SPARTA Jiang et al. (2025) studies collective alignment through repeated combat, evaluation, and learning among multiple models, while Ma et al. Ma et al. (2023) cast safety alignment as a red–blue team game in which attack and defense strategies evolve together. These approaches suggest that robustness may emerge not only from stronger objectives, but also from continued exposure to adaptive opponents and evaluators. Scenario simulation. The same co-evolutionary perspective also underlies a broader set of social simulation environments. Recent work uses agent societies, role-playing communities, and sandbox worlds to study how norms, cooperation, and governance emerge through interaction Liu et al. (2024b); Lai et al. (2024); Li et al. (2023); Dai et al. (2024). In parallel, strategically structured environments such as Alympics Mao et al. (2025) and FairMindSim Lei et al. (2024) are used to probe bargaining, fairness, and collective behavior under resource constraints or ethical dilemmas. Debate-based methods extend this logic to inference-time improvement: by allowing multiple agents to critique and challenge one another, they can improve reasoning and factuality without retraining Liang et al. (2024); Du et al. (2024). Together, these studies broaden alignment from single-model optimization to the analysis of collective behavior in evolving social systems, but many of these environments remain behaviorally rich while still offering limited visibility into the internal mechanisms that drive adaptation. 5 Future Directions Fairness guarantees Pluralistic societies typically do not rely on a single value standard, so alignment should not only search for a ‘true’ value objective but also design fair procedures for deciding which values guide model behavior MacIntyre (2013); Gabriel (2020). Existing multi-objective methods introduce fairness constraints, such as demographic parity, equalized odds, group-DRO, and worst-group lower bounds Agnihotri et al. (2025); Chakraborty et al. (2024). However, soft regularization, such as imposing a finite penalty on the objective function, typically gives weak guarantees, while hard constraints can reduce utility or destabilize training Tran et al. (2024). A promising direction is to model fairness-utility trade-offs as games with social-welfare objectives Chen et al. (2024a), including Nash welfare, Leximin Barman and Suzuki (2024), and Pareto-optimal solution families Censor (1977). Future work should further combine adaptive weighting with procedural safeguards from Constitutional AI, social choice, voting, and public participation, so that both outcomes and decision procedures remain legitimate and adaptable Zhao et al. (2024); Huang et al. (2024b). Continual alignment under drift. Online tool use, continual preference learning, and scenario migration can update useful capabilities while gradually overwriting aligned behavior. This causes catastrophic forgetting and value drift, especially in adversarial testing and long multi-turn interaction. From a game-theoretic view, alignment maintenance can be modeled as a dynamic interaction among the model, deployment environment, and strategic auditors or adversaries, where feedback streams may expose or induce degradation. Existing remedies include continual or online RLHF Zhang et al. (2024), trust-region or strongly regularized updates Li et al. (2024a), and generative replay under data-access limits Huang et al. (2024a). However, most methods optimize general retention rather than treating alignment-critical knowledge as the main retention target Wang et al. (2024h). Future work should combine continual learning with hierarchical or retrieval-augmented memory Wang et al. (2023a); Borgeaud et al. (2022); Meng et al. (2022), classical anti-forgetting mechanisms Dokania et al. (2019); Kirkpatrick et al. (2017), and adversarially robust updates Khan et al. (2022); Mirzadeh et al. (2020); Ebrahimi et al. (2020), so that safety-relevant knowledge remains stable under changing or manipulated data flows. Visual simulation for explainable AI. AI alignment remains difficult to inspect when simulations only report external behavior. Controlled social simulations, agent societies, and policy games, as discussed in Section 4.3, can reveal dynamic failures in interaction, adaptation, and social feedback. Yet many such environments remain behaviorally rich but mechanistically shallow. Future work should connect these environments with mechanistic interpretability, rule-based dynamic models, intervention logging, and counterfactual analysis. This would turn simulation from outcome-level evaluation into process-level diagnosis, making long-term alignment easier to inspect, explain, and steer. Conclusion This survey examined AI alignment through the lens of game theory, reframing alignment as a problem of strategic dependence rather than static objective fitting. By organizing the literature around preference structure, interaction structure, and temporal dynamics, this work clarifies how values are aggregated, how models or agents influence one another, and how aligned behavior may emerge, change, or remain stable over time. Furthermore, we point out that existing game-theoretic alignment methods still focus on the training phase, while their role in inference control, social game-based benchmarking, practical verification, and interpretability remains to be developed. Therefore, researchers need to further expand game-theoretic tools to achieve, diagnose, and manage AI alignment, strengthening the connection between theoretical guarantees and the real world, thereby supporting the development of safer and more reliable AI systems. Limitations This survey focuses primarily on post-training alignment work for LLMs and LLM-based agents that can be meaningfully analyzed through game-theoretic concepts. It therefore does not aim to exhaustively cover adjacent literature on pre-training objectives, inference-time defenses, mechanistic interpretability, or governance frameworks that do not directly adopt this lens. In addition, the reviewed areas are uneven in theoretical maturity: preference-learning work currently admits stronger formal guarantees than many studies on social interaction, simulation, and long-term evolution. As a result, some parts of the survey necessarily provide a looser synthesis than others. Ethics Statement While game-theoretic frameworks can help clarify incentives and interaction structures in alignment, they may also simplify complex moral, cultural, and institutional factors. Therefore, the methods reviewed in this paper should not be regarded as complete solutions to alignment or as substitutes for human oversight and governance. We encourage readers to interpret the reviewed approaches with caution, especially in safety-critical or socially sensitive settings. References Abdelnabi et al. (2024) S. Abdelnabi, A. Gomaa, S. Sivaprasad, L. Schönherr, and M. Fritz Cooperation, competition, and maliciousness: llm-stakeholders interactive negotiation. Advances in Neural Information Processing Systems 37, p. 83548–83599. Cited by: Table 4. Agnihotri et al. (2025) A. Agnihotri, R. Jain, D. Ramachandran, and Z. Wen Multi-objective preference optimization: improving human alignment of generative models. arXiv preprint arXiv:2505.10892. Cited by: §5. Akata et al. (2025) E. Akata, L. Schulz, J. Coda-Forno, S. J. Oh, M. Bethge, and E. Schulz Playing repeated games with large language models. Nature Human Behaviour, p. 1–11. Cited by: Table 4. Amodei et al. (2016) D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané Concrete problems in ai safety. arXiv preprint arXiv:1606.06565. Cited by: §1. Azar et al. (2024) M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, p. 4447–4455. Cited by: Table 1, Figure 2, §4.1.1. Azarafrooz and Faal (2024) A. Azarafrooz and F. Faal Language alignment via nash-learning and adaptive feedback. arXiv preprint arXiv:2406.15890. Cited by: Table 2, Figure 2, §4.2.1. Barman and Suzuki (2024) S. Barman and M. Suzuki Compatibility of fairness and nash welfare under subadditive valuations. arXiv preprint arXiv:2407.12461. Cited by: §5. Borgeaud et al. (2022) S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J. Lespiau, B. Damoc, A. Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, p. 2206–2240. Cited by: §5. Calandriello et al. (2024) D. Calandriello, D. Guo, R. Munos, M. Rowland, Y. Tang, B. A. Pires, P. H. Richemond, C. L. Lan, M. Valko, T. Liu, et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.08635. Cited by: Table 1, Figure 2, §4.1.1. Carroll et al. (2024) M. Carroll, D. Foote, A. Siththaranjan, S. Russell, and A. Dragan AI alignment with changing and influenceable reward functions. In Proceedings of the 41st International Conference on Machine Learning, p. 5706–5756. Cited by: §1. Censor (1977) Y. Censor Pareto optimality in multiobjective problems. Applied Mathematics and Optimization 4 (1), p. 41–59. Cited by: §5. Chakraborty et al. (2024) S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, D. Manocha, F. Huang, A. S. Bedi, and M. Wang MaxMin-rlhf: alignment with diverse human preferences. In Proceedings of the 41st International Conference on Machine Learning, p. 6116–6135. Cited by: §5. Chen et al. (2025a) B. Chen, T. Zhu, J. Han, L. Li, G. Li, and X. Dai Incentivizing truthful language models via peer elicitation games. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Table 2, Figure 2, §4.2.1. Chen et al. (2025b) J. Chen, B. Zhang, R. Ma, P. Wang, X. Liang, Z. Tu, X. Li, and K. K. Wong Spc: evolving self-play critic via adversarial games for llm reasoning. arXiv preprint arXiv:2504.19162. Cited by: Table 2, Figure 2, §3.2, §4.2.1. Chen et al. (2024a) V. Chen, J. Hooker, and D. Leben Assessing group fairness with social welfare optimization. In International Conference on the Integration of Constraint Programming, Artificial Intelligence, and Operations Research, p. 208–225. Cited by: §5. Chen et al. (2024b) Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu Self-play fine-tuning converts weak language models to strong language models. In International Conference on Machine Learning, p. 6621–6642. Cited by: Table 2, Figure 2, §4.2.1. Cheng et al. (2024) P. Cheng, Y. Dai, T. Hu, H. Xu, Z. Zhang, L. Han, N. Du, and X. Li Self-playing adversarial language game enhances llm reasoning. Advances in Neural Information Processing Systems 37, p. 126515–126543. Cited by: Table 2, Figure 2, §4.2.1. Cheng et al. (2023) P. Cheng, Y. Yang, J. Li, Y. Dai, T. Hu, P. Cao, N. Du, and X. Li Adversarial preference optimization: enhancing your alignment via rm-llm game. arXiv preprint arXiv:2311.08045. Cited by: Table 2, Figure 2, §4.2.1. Chidambaram et al. (2024) K. Chidambaram, K. V. Seetharaman, and V. Syrgkanis Direct preference optimization with unobserved preference heterogeneity. arXiv preprint arXiv:2405.15065. Cited by: Table 1, Figure 2, §4.1.2. Chu et al. (2025) X. Chu, Z. Zhang, T. Jia, and Y. Jin Stackelberg game preference optimization for data-efficient alignment of language models. arXiv preprint arXiv:2502.18099. Cited by: Table 2, §1, §1, Figure 2, §2, §3.2, §4.2.2. Conitzer et al. (2024) V. Conitzer, R. Freedman, J. Heitzig, W. H. Holliday, B. M. Jacobs, N. Lambert, M. Mossé, E. Pacuit, S. Russell, H. Schoelkopf, et al. Social choice should guide ai alignment in dealing with diverse human feedback. arXiv preprint arXiv:2404.10271. Cited by: §4.1.2. Curvo (2025) P. M. Curvo The traitors: deception and trust in multi-agent language model simulations. arXiv preprint arXiv:2505.12923. Cited by: Table 4. Dai et al. (2024) G. Dai, W. Zhang, J. Li, S. Yang, S. Rao, A. Caetano, M. Sra, et al. Artificial leviathan: exploring social evolution of llm agents through the lens of hobbesian social contract theory. arXiv preprint arXiv:2406.14373. Cited by: Table 3, Figure 2, §4.3.2. Davidson et al. (2024) T. R. Davidson, V. Veselovsky, M. Josifoski, M. Peyrard, A. Bosselut, M. Kosinski, and R. West Evaluating language model agency through negotiations. In ICLR 2024, Cited by: Table 4. Deng et al. (2025) Y. Deng, Y. Yang, J. Zhang, W. Wang, and B. Li DuoGuard: a two-player rl-driven framework for multilingual llm guardrails. arXiv preprint arXiv:2502.05163. Cited by: Table 2, Figure 2, §4.2.1. Dokania et al. (2019) P. Dokania, P. Torr, and M. Ranzato Continual learning with tiny episodic memories. In Workshop on Multi-Task and Lifelong Reinforcement Learning, Cited by: §5. Du et al. (2024) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning, p. 11733–11763. Cited by: Table 3, Figure 2, §4.3.2. Duan et al. (2025) W. Duan, N. McNeese, and L. Li Gender stereotypes toward non-gendered generative ai: the role of gendered expertise and gendered linguistic cues. Proceedings of the ACM on Human-Computer Interaction 9 (GROUP), p. 1–35. Cited by: §1. Duenas and Ruiz (2024) T. Duenas and D. Ruiz The frontier of ai alignment: challenges and strategies for future ai systems. ResearchGate, 2024a. doi 10. Cited by: §1. Ebrahimi et al. (2020) S. Ebrahimi, F. Meier, R. Calandra, T. Darrell, and M. Rohrbach Adversarial continual learning. In European Conference on Computer Vision, p. 386–402. Cited by: §5. Fan et al. (2024) C. Fan, J. Chen, Y. Jin, and H. He Can large language models serve as rational players in game theory? a systematic analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 17960–17967. Cited by: Table 4. Floridi and Cowls (2022) L. Floridi and J. Cowls A unified framework of five principles for ai in society. Machine learning and the city: Applications in architecture and urban design, p. 535–545. Cited by: §1. Fontana et al. (2025) N. Fontana, F. Pierri, and L. M. Aiello Nicer than humans: how do large language models behave in the prisoner’s dilemma?. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 19, p. 522–535. Cited by: Table 4. Gabriel (2020) I. Gabriel Artificial intelligence, values, and alignment. Minds and machines 30 (3), p. 411–437. Cited by: §1, §5. Gallegos et al. (2024) I. O. Gallegos, R. A. Rossi, J. Barrow, M. M. Tanjim, S. Kim, F. Dernoncourt, T. Yu, R. Zhang, and N. K. Ahmed Bias and fairness in large language models: a survey. Computational Linguistics, p. 1–79. Cited by: §1. Gao et al. (2024) Z. Gao, J. Chang, W. Zhan, O. Oertell, G. Swamy, K. Brantley, T. Joachims, D. Bagnell, J. D. Lee, and W. Sun Rebel: reinforcement learning via regressing relative rewards. Advances in Neural Information Processing Systems 37, p. 52354–52400. Cited by: Table 1, Figure 2, §4.1.1. Gemp et al. (2024) I. Gemp, Y. Bachrach, M. Lanctot, R. Patel, V. Dasagi, L. Marris, G. Piliouras, S. Liu, and K. Tuyls States as strings as strategies: steering language models with game-theoretic solvers. arXiv preprint arXiv:2402.01704. Cited by: §1. Guo et al. (2025) W. Guo, J. Li, W. Wang, Y. Li, D. He, J. Yu, and M. Zhang Mtsa: multi-turn safety alignment for llms through multi-round red-teaming. arXiv preprint arXiv:2505.17147. Cited by: Table 2, Figure 2, §4.2.1. Guo (2024) Z. Guo Online disinformation and generative language models: motivations, challenges, and mitigations. In Companion Proceedings of the ACM on Web Conference 2024, p. 1174–1177. Cited by: §1. Gupta et al. (2025a) R. Gupta, R. Sullivan, Y. Li, S. Phatale, and A. Rastogi Robust multi-objective preference alignment with online dpo. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 27321–27329. Cited by: Table 1, Figure 2, §4.1.2. Gupta et al. (2025b) T. Gupta, R. Madhavan, X. Zhang, C. Bansal, and S. Rajmohan AMPO: active multi preference optimization for self-play preference selection. In Scaling Self-Improving Foundation Models without Human Supervision, External Links: Link Cited by: §4.1.2. Hao and Duan (2025) S. Hao and L. Duan Online learning from strategic human feedback in llm fine-tuning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: Table 1, Figure 2, §4.1.2. Huang et al. (2024a) J. Huang, L. Cui, A. Wang, C. Yang, X. Liao, L. Song, J. Yao, and J. Su Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1416–1428. Cited by: §5. Huang et al. (2024b) S. Huang, D. Siddarth, L. Lovitt, T. I. Liao, E. Durmus, A. Tamkin, and D. Ganguli Collective constitutional ai: aligning a language model with public input. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, p. 1395–1417. Cited by: §5. Jatova et al. (2024) L. Jatova, J. Smith, and A. Wilson Employing game theory for mitigating adversarial-induced content toxicity in generative large language models. Authorea Preprints. Cited by: Table 2, Figure 2, §4.2.1. Ji et al. (2023) J. Ji, T. Qiu, B. Chen, B. Zhang, H. Lou, K. Wang, Y. Duan, Z. He, J. Zhou, Z. Zhang, et al. Ai alignment: a comprehensive survey. arXiv preprint arXiv:2310.19852. Cited by: §2, §4.1. Ji et al. (2024) X. Ji, S. Kulkarni, M. Wang, and T. Xie Self-play with adversarial critic: provable and scalable offline alignment for language models. arXiv preprint arXiv:2406.04274. Cited by: Table 2, Figure 2, §4.2.2. Jiang et al. (2025) Y. Jiang, W. Ding, S. Feng, G. Durrett, and Y. Tsvetkov SPARTA alignment: collectively aligning multiple language models through combat. arXiv preprint arXiv:2506.04721. Cited by: Table 3, Figure 2, §4.3.2. Kayabay et al. (2025) K. Kayabay, M. O. Gökalp, A. Kılınç, E. Gökalp, and T. Daim Data science roadmapping for ai alignment: insights from a multivocal literature review and a retrospective case study. IEEE Transactions on Engineering Management. Cited by: §2. Khan et al. (2022) H. Khan, N. C. Bouaynaya, and G. Rasool Adversarially robust continual learning. In 2022 International Joint Conference on Neural Networks (IJCNN), p. 1–8. Cited by: §5. Kim et al. (2025) K. Kim, J. Zhang, A. Ozdaglar, and P. A. Parrilo Population-proportional preference learning from human feedback: an axiomatic approach. arXiv preprint arXiv:2506.05619. Cited by: Table 1, Figure 2, §4.1.2. Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America 114 (13), p. 3521–3526. Cited by: §5. Klassen et al. (2024) T. Q. Klassen, P. A. Alamdari, and S. A. McIlraith Pluralistic alignment over time. arXiv preprint arXiv:2411.10654. Cited by: Table 3, Figure 2, §4.3.1. Kleinberg et al. (2016) J. Kleinberg, S. Mullainathan, and M. Raghavan Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807. Cited by: §1. Lai et al. (2025) H. Lai, X. Liu, J. Gao, J. Cheng, Z. Qi, Y. Xu, S. Yao, D. Zhang, J. Du, Z. Hou, et al. A survey of post-training scaling in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2771–2791. Cited by: §1. Lai et al. (2024) S. Lai, Y. Potter, J. Kim, R. Zhuang, D. Song, and J. Evans Position: evolving ai collectives enhance human diversity and enable self-regulation. In Forty-first International Conference on Machine Learning, Cited by: Table 3, Figure 2, §4.3.2. Lei et al. (2024) Y. Lei, H. Liu, C. Xie, S. Liu, Z. Yin, C. Chen, G. Li, P. Torr, and Z. Wu FairMindSim: alignment of behavior, emotion, and belief in humans and llm agents amid ethical dilemmas. arXiv preprint arXiv:2410.10398. Cited by: Table 3, Figure 2, §4.3.2. Li et al. (2023) G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem Camel: communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems 36, p. 51991–52008. Cited by: Table 3, Figure 2, §4.3.2. Li et al. (2024a) H. Li, L. Ding, M. Fang, and D. Tao Revisiting catastrophic forgetting in large language model tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 4297–4308. Cited by: §5. Li et al. (2025) M. Li, Y. Zhang, W. Wang, W. Shi, Z. Liu, F. Feng, and T. Chua Self-improvement towards pareto optimality: mitigating preference conflicts in multi-objective alignment. arXiv preprint arXiv:2502.14354. Cited by: Table 1, Figure 2, §4.1.2. Li et al. (2024b) S. Li, T. Sun, Q. Cheng, and X. Qiu Agent alignment in evolving social norms. arXiv preprint arXiv:2401.04620. Cited by: Table 3, Figure 2, §4.3.1. Li et al. (2024c) X. Li, S. Wang, S. Zeng, Y. Wu, and Y. Yang A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth 1 (1), p. 9. Cited by: §2. Liang et al. (2024) T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, S. Shi, and Z. Tu Encouraging divergent thinking in large language models through multi-agent debate. In EMNLP, Cited by: Table 3, Figure 2, §4.3.2. Liu et al. (2024a) J. Liu, Z. Ni, H. Que, T. Sun, Z. Wang, J. Yang, J. Wang, H. Guo, Z. Peng, G. Zhang, et al. Roleagent: building, interacting, and benchmarking high-quality role-playing agents from scripts. Advances in Neural Information Processing Systems 37, p. 49403–49428. Cited by: Table 4. Liu et al. (2025) M. Liu, L. Jiang, Y. Liang, S. S. Du, Y. Choi, T. Althoff, and N. Jaques Chasing moving targets with online self-play reinforcement learning for safer language models. arXiv preprint arXiv:2506.07468. Cited by: Table 2, Figure 2, §4.2.1. Liu et al. (2024b) R. Liu, R. Yang, C. Jia, G. Zhang, D. Yang, and S. Vosoughi Training socially aligned language models on simulated social interactions. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Table 3, Figure 2, §4.3.2. Liu et al. (2026) Y. Liu, A. Oikonomou, W. Zheng, Y. Cai, and A. Cohan Comal: a convergent meta-algorithm for aligning llms with general preferences. In International Conference on Learning Representations, Vol. 2026, p. 51383–51422. Cited by: Table 1, Figure 2, §3.1.1, §3.1.3, §4.1.1. Lorè and Heydari (2024) N. Lorè and B. Heydari Strategic behavior of large language models and the role of game structure versus contextual framing. Scientific Reports 14 (1), p. 18490. Cited by: Table 4. Lu (2024) S. E. Lu Strategic interactions between large language models-based agents in beauty contests. arXiv preprint arXiv:2404.08492. Cited by: Table 4. Luo et al. (2025) F. Luo, R. Yang, H. Sun, C. Deng, J. Yao, J. Shen, H. Zhang, and H. Chen Rethinking diverse human preference learning through principal component analysis. arXiv preprint arXiv:2502.13131. Cited by: Table 1, Figure 2, §4.1.2. Ma et al. (2023) C. Ma, Z. Yang, H. Ci, J. Gao, M. Gao, X. Pan, and Y. Yang Evolving diverse red-team language models in multi-round multi-agent games. arXiv preprint arXiv:2310.00322. Cited by: Table 3, Figure 2, §3.1.3, §3.2, §4.3.2. Ma et al. (2024) H. Ma, T. Hu, Z. Pu, L. Boyin, X. Ai, Y. Liang, and M. Chen Coevolving with the other you: fine-tuning llm with sequential cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems 37, p. 15497–15525. Cited by: Table 2, Figure 2, §4.2.2. MacIntyre (2013) A. MacIntyre After virtue. A&C Black. Cited by: §5. Makar-Limanov et al. (2024) J. Makar-Limanov, A. Prakash, D. Goktas, N. Ayanian, and A. Greenwald STA-rlhf: stackelberg aligned reinforcement learning with human feedback. In Coordination and Cooperation for Multi-Agent Reinforcement Learning Methods Workshop, Cited by: Table 2, Figure 2, §4.2.2. Mao et al. (2025) S. Mao, Y. Cai, Y. Xia, W. Wu, X. Wang, F. Wang, Q. Guan, T. Ge, and F. Wei ALYMPICS: llm agents meet game theory. In Proceedings of the 31st International Conference on Computational Linguistics, p. 2845–2866. Cited by: Table 3, Figure 2, §2, §4.3.2. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in gpt. Advances in neural information processing systems 35, p. 17359–17372. Cited by: §5. Milano and Nyholm (2024) S. Milano and S. Nyholm Advanced ai assistants that act on our behalf may not be ethically or legally feasible. Nature Machine Intelligence 6 (8), p. 846–847. Cited by: §1. Mirzadeh et al. (2020) S. I. Mirzadeh, M. Farajtabar, R. Pascanu, and H. Ghasemzadeh Understanding the role of training regimes in continual learning. Advances in Neural Information Processing Systems 33, p. 7308–7320. Cited by: §5. Mishra (2023) A. Mishra AI alignment and social choice: fundamental limitations and policy implications. arXiv preprint arXiv:2310.16048. Cited by: §4.1.2. Motwani et al. (2024) S. R. Motwani, M. Baranchuk, M. Strohmeier, V. Bolina, P. Torr, L. Hammond, and C. S. de Witt Secret collusion among AI agents: multi-agent deception via steganography. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Table 4. Mozikov et al. (2024) M. Mozikov, N. Severin, V. Bodishtianu, M. Glushanina, I. Nasonov, D. Orekhov, P. Vladislav, I. Makovetskiy, M. Baklashkin, V. Lavrentyev, et al. EAI: emotional decision-making of llms in strategic games and ethical dilemmas. Advances in Neural Information Processing Systems 37, p. 53969–54002. Cited by: §1. Munos et al. (2024) R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, C. Fiegel, et al. Nash learning from human feedback. In International Conference on Machine Learning, p. 36743–36768. Cited by: Table 1, §1, §1, Figure 2, §3.1.3, §3, §4.1.1. Nash Jr (1950) J. F. Nash Jr Equilibrium points in n-person games. Proceedings of the national academy of sciences 36 (1), p. 48–49. Cited by: §3.2. Nayebi (2025) A. Nayebi Barriers and pathways to human-ai alignment: a game-theoretic approach. arXiv preprint arXiv:2502.05934. Cited by: Table 1, Figure 2, §4.1.2. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1. Pan et al. (2022) A. Pan, K. Bhatia, and J. Steinhardt The effects of reward misspecification: mapping and mitigating misaligned models. In International Conference on Learning Representations, Cited by: §1. Park et al. (2024) P. S. Park, S. Goldstein, A. O’Gara, M. Chen, and D. Hendrycks AI deception: a survey of examples, risks, and potential solutions. Patterns 5 (5). Cited by: §1. Pásztor et al. (2025) B. Pásztor, T. K. Buening, and A. Krause Stackelberg learning from human feedback: preference optimization as a sequential game. In NeurIPS 2025 Workshop: Second Workshop on Aligning Reinforcement Learning Experimentalists and Theorists, Cited by: §3. Qiu et al. (2024) T. A. Qiu, Y. Zhang, X. Huang, J. Li, J. Ji, and Y. Yang Progressgym: alignment with a millennium of moral progress. Advances in Neural Information Processing Systems 37, p. 14570–14607. Cited by: Table 3, §1, Figure 2, §3.2, §4.3.1. Qiu (2024) T. Qiu Representative social choice: from learning theory to AI alignment. In Pluralistic Alignment Workshop at NeurIPS 2024, External Links: Link Cited by: §4.1.2. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. Advances in Neural Information Processing Systems 36, p. 53728–53741. Cited by: §1, §3.1.1. Ramesh et al. (2024) S. S. Ramesh, Y. Hu, I. Chaimalas, V. Mehta, P. G. Sessa, H. Bou Ammar, and I. Bogunovic Group robust preference optimization in reward-free rlhf. Advances in Neural Information Processing Systems 37, p. 37100–37137. Cited by: Table 1, Figure 2, §4.1.2. Rosset et al. (2024) C. Rosset, C. Cheng, A. Mitra, M. Santacroce, A. Awadallah, and T. Xie Direct nash optimization: tea ching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715. Cited by: Table 1, Figure 2, §4.1.1. Shen et al. (2024) H. Shen, T. Knearem, R. Ghosh, K. Alkiek, K. Krishna, Y. Liu, Z. Ma, S. Petridis, Y. Peng, L. Qiwei, et al. Towards bidirectional human-ai alignment: a systematic review for clarifications, framework, and future directions. arXiv preprint arXiv:2406.09264. Cited by: §1, §2, §4.2. Shi et al. (2025) Z. Shi, K. Liu, Q. Long, W. J. Su, and J. Xiao Fundamental limits of game-theoretic llm alignment: smith consistency and preference matching. arXiv preprint arXiv:2505.20627. Cited by: §3. Sreedhar et al. (2025) K. Sreedhar, A. Cai, J. Ma, J. V. Nickerson, and L. B. Chilton Simulating cooperative prosocial behavior with multi-agent llms: evidence and mechanisms for ai agents to inform policy decisions. In Proceedings of the 30th International Conference on Intelligent User Interfaces, p. 1272–1286. Cited by: Table 4. Sun et al. (2024a) H. Sun, Y. Chen, S. Wang, W. Chen, and X. Deng Mechanism design for llm fine-tuning with multiple reward models. arXiv preprint arXiv:2405.16276. Cited by: Table 1, Figure 2, §4.1.2. Sun et al. (2025) H. Sun, Y. Wu, Y. Cheng, and X. Chu Game theory meets large language models: a systematic survey. arXiv preprint arXiv:2502.09053. Cited by: §2. Sun et al. (2024b) Q. Sun, Y. Li, E. Alturki, S. M. K. Murthy, and B. W. Schuller Towards friendly ai: a comprehensive review and new perspectives on human-ai alignment. arXiv preprint arXiv:2412.15114. Cited by: §2. Suzuki and Arita (2024) R. Suzuki and T. Arita An evolutionary model of personality traits related to cooperative behavior using a large language model. Scientific Reports 14 (1), p. 5989. Cited by: Table 3, Figure 2, §4.3.1. Swamy et al. (2024) G. Swamy, C. Dann, R. Kidambi, Z. S. Wu, and A. Agarwal A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056. Cited by: Table 1, Figure 2, §4.1.1. Tang et al. (2025) X. Tang, S. Yoon, S. Son, H. Yuan, Q. Gu, and I. Bogunovic Game-theoretic regularized self-play alignment of large language models. arXiv preprint arXiv:2503.00030. Cited by: Table 1, §1, §1, Figure 2, §2, §4.1.1. Tiapkin et al. (2025) D. Tiapkin, D. Calandriello, D. Belomestny, E. Moulines, A. Naumov, K. Rasul, M. Valko, and P. Menard Accelerating nash learning from human feedback via mirror prox. arXiv preprint arXiv:2505.19731. Cited by: §3.1.3. Topsakal et al. (2024) O. Topsakal, C. J. Edell, and J. B. Harper Evaluating large language models with grid-based game competitions: an extensible llm benchmark and leaderboard. arXiv preprint arXiv:2407.07796. Cited by: Table 4. Tran et al. (2024) C. Tran, K. Zhu, P. Van Hentenryck, and F. Fioretto On the effects of fairness to adversarial vulnerability. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, p. 521–529. Cited by: §5. Turner and Tadepalli (2022) A. Turner and P. Tadepalli Parametrically retargetable decision-makers tend to seek power. Advances in Neural Information Processing Systems 35, p. 31391–31401. Cited by: §1. Wang et al. (2024a) C. Wang, B. Dai, H. Liu, and B. Wang Towards objectively benchmarking social intelligence of language agents at the action level. In Findings of the Association for Computational Linguistics ACL 2024, p. 8885–8897. Cited by: Table 4. Wang et al. (2024b) H. Wang, X. Feng, L. Li, Y. Guo, Z. Qin, D. Sui, and L. Kong Tmgbench: a systematic game benchmark for evaluating strategic reasoning abilities of llms. arXiv preprint arXiv:2410.10479. Cited by: Table 4. Wang et al. (2024c) H. Wang, Y. Lin, W. Xiong, R. Yang, S. Diao, S. Qiu, H. Zhao, and T. Zhang Arithmetic control of llms for diverse user preferences: directional preference alignment with multi-objective rewards. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8642–8655. Cited by: Table 1, Figure 2, §4.1.2. Wang et al. (2024d) K. Wang, R. Kidambi, R. Sullivan, A. Agarwal, C. Dann, A. Michi, M. Gelmi, Y. Li, R. Gupta, A. Dubey, et al. Conditional language policy: a general framework for steerable multi-objective finetuning. arXiv preprint arXiv:2407.15762. Cited by: Table 1, Figure 2, §4.1.2. Wang et al. (2024e) L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), p. 186345. Cited by: §2. Wang et al. (2024f) M. Wang, C. Ma, Q. Chen, L. Meng, Y. Han, J. Xiao, Z. Zhang, J. Huo, W. J. Su, and Y. Yang Magnetic preference optimization: achieving last-iterate convergence for language model alignment. arXiv preprint arXiv:2410.16714. Cited by: §3.1.3. Wang et al. (2025a) M. Wang, C. Ma, Q. Chen, L. Meng, Y. Han, J. Xiao, Z. Zhang, J. Huo, W. J. Su, and Y. Yang Magnetic preference optimization: achieving last-iterate convergence for language model alignment. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Table 1, Figure 2, §4.1.1. Wang et al. (2024g) N. Wang, Z. Peng, H. Que, J. Liu, W. Zhou, Y. Wu, H. Guo, R. Gan, Z. Ni, J. Yang, et al. Rolellm: benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, p. 14743–14777. Cited by: Table 4. Wang et al. (2023a) W. Wang, L. Dong, H. Cheng, X. Liu, X. Yan, J. Gao, and F. Wei Augmenting language models with long-term memory. Advances in Neural Information Processing Systems 36, p. 74530–74543. Cited by: §5. Wang et al. (2025b) Y. Wang, Z. Zhang, Z. Liu, S. Zhang, and Z. Wang Provably efficient and practical self-play for better LLM alignment. External Links: Link Cited by: Table 1, Figure 2, §4.1.1. Wang et al. (2023b) Y. Wang, Q. Liu, and C. Jin Is rlhf more difficult than standard rl? a theoretical perspective. Advances in Neural Information Processing Systems 36, p. 76006–76032. Cited by: §4.1.1. Wang et al. (2024h) Z. Wang, B. Bi, S. K. Pentyala, K. Ramnath, S. Chaudhuri, S. Mehrotra, X. Mao, S. Asur, et al. A comprehensive survey of llm alignment techniques: rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216. Cited by: §5. Wu et al. (2025) Y. Wu, Y. Zhu, J. Cai, and X. Zhu The battling influencers game: nash equilibria structure of a potential game and implications to value alignment. arXiv preprint arXiv:2502.01127. Cited by: Table 1, Figure 2, §3.2, §4.1.2. Wu et al. (2024) Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu Self-play preference optimization for language model alignment, 2024b. URL https://arxiv. org/abs/2405.00675. Cited by: Table 1, Figure 2, §3, §4.1.1. Xi et al. (2025) Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al. The rise and potential of large language model based agents: a survey. Science China Information Sciences 68 (2), p. 121101. Cited by: §1. Xia et al. (2024) T. Xia, Z. He, T. Ren, Y. Miao, Z. Zhang, Y. Yang, and R. Wang Measuring bargaining abilities of llms: a benchmark and a buyer-enhancement method. In Findings of the Association for Computational Linguistics ACL 2024, p. 3579–3602. Cited by: Table 4. Xie et al. (2024) C. Xie, C. Chen, F. Jia, Z. Ye, S. Lai, K. Shu, J. Gu, A. Bibi, Z. Hu, D. Jurgens, J. Evans, P. Torr, B. Ghanem, and G. Li Can large language model agents simulate human trust behavior?. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Table 4. Xie et al. (2026) S. Xie, Z. Shi, H. Shen, G. Huang, Y. Ma, and X. Jing M3-bench: process-aware evaluation of llm agents social behaviors in mixed-motive games. arXiv preprint arXiv:2601.08462. Cited by: Table 4. Xiong and Singh (2025) N. Xiong and A. Singh Projection optimization: a general framework for multi-objective and multi-group rlhf. In Proceedings of the 42nd International Conference on Machine Learning, ICML’25. Cited by: Table 1, Figure 2, §4.1.2. Ye et al. (2024a) C. Ye, W. Xiong, Y. Zhang, H. Dong, N. Jiang, and T. Zhang Online iterative reinforcement learning from human feedback with general preference model. Advances in Neural Information Processing Systems 37, p. 81773–81807. Cited by: Table 1, §1, §1, Figure 2, §2, §3.1.2, §3.1.3, §3, §4.1.1. Ye et al. (2024b) Z. Ye, R. Agarwal, T. Liu, R. Joshi, S. Velury, Q. V. Le, Q. Tan, and Y. Liu Evolving alignment via asymmetric self-play. In Language Gamification-NeurIPS 2024 Workshop, Cited by: Table 3, Figure 2, §4.3.1. Zakazov et al. (2024) I. Zakazov, M. Boronski, L. Drudi, and R. West Assessing social alignment: do personality-prompted large language models behave like humans?. arXiv preprint arXiv:2412.16772. Cited by: Table 4. Zhang et al. (2024) H. Zhang, Y. Lei, L. Gui, M. Yang, Y. He, H. Wang, and R. Xu CPPO: continual learning for reinforcement learning with human feedback. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §5. Zhang et al. (2025a) Y. Zhang, D. Yu, T. Ge, L. Song, Z. Zeng, H. Mi, N. Jiang, and D. Yu Improving llm general preference alignment via optimistic online mirror descent. arXiv preprint arXiv:2502.16852. Cited by: Table 1, Figure 2, §4.1.1. Zhang et al. (2025b) Y. Zhang, D. Yu, B. Peng, L. Song, Y. Tian, M. Huo, N. Jiang, H. Mi, and D. Yu Iterative nash policy optimization: aligning llms with general preferences via no-regret learning. In International Conference on Learning Representations, Vol. 2025, p. 31833–31849. Cited by: Table 1, Figure 2, §3.1.3, §3, §4.1.1. Zhao et al. (2024) X. Zhao, K. Wang, and W. Peng An electoral approach to diversify llm-based multi-agent collective decision-making. In EMNLP, p. 2712–2727. External Links: Link Cited by: §5. Zheng et al. (2024) R. Zheng, H. Guo, Z. Liu, X. Zhang, Y. Yao, X. Xu, Z. Wang, Z. Xi, T. Gui, Q. Zhang, et al. Toward optimal llm alignments using two-player games. arXiv preprint arXiv:2406.10977. Cited by: Table 2, Figure 2, §3.2, §4.2.1. Zhong et al. (2024a) H. Zhong, Z. Deng, W. J. Su, Z. S. Wu, and L. Zhang Provable multi-party reinforcement learning with diverse human feedback. arXiv preprint arXiv:2403.05006. Cited by: Table 1, Figure 2, §4.1.2. Zhong et al. (2024b) Y. Zhong, C. Ma, X. Zhang, Z. Yang, H. Chen, Q. Zhang, S. Qi, and Y. Yang Panacea: pareto alignment via preference adaptation for llms. Advances in Neural Information Processing Systems 37, p. 75522–75558. Cited by: Table 1, Figure 2, §4.1.2. Zhou et al. (2025a) D. Zhou, J. Zhang, T. Feng, and Y. Sun A survey on alignment for large language model agents. In UIUC Spring 2025 CS598 LLM Agent Workshop; Urbana, IL, USA, Cited by: §2. Zhou et al. (2025b) J. Zhou, Y. Chen, Y. Shi, X. Zhang, L. Lei, Y. Feng, Z. Xiong, M. Yan, X. Wang, Y. Cao, et al. Socialeval: evaluating social intelligence of large language models. arXiv preprint arXiv:2506.00900. Cited by: Table 4. Zhou et al. (2024) X. Zhou, H. Zhu, L. Mathur, R. Zhang, H. Yu, Z. Qi, L. Morency, Y. Bisk, D. Fried, G. Neubig, and M. Sap SOTOPIA: interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Table 4. Zhou et al. (2025c) Y. Zhou, Z. Wang, T. Wang, S. Xing, P. Xia, B. Li, K. Zheng, Z. Zhang, Z. Chen, W. Zheng, et al. Anyprefer: an agentic framework for preference data synthesis. arXiv preprint arXiv:2504.19276. Cited by: Table 2, Figure 2, §3.2, §4.2.2. Zhu et al. (2025) K. Zhu, H. Du, Z. Hong, X. Yang, S. Guo, Z. Wang, Z. Wang, C. Qian, X. Tang, H. Ji, et al. MultiAgentBench: evaluating the collaboration and competition of llm agents. arXiv preprint arXiv:2503.01935. Cited by: Table 4. Appendix A Summary Tables This appendix provides representative methods and evaluation benchmarks for large language models or agent alignment, summarizing them in tabular form (see Tabs. 1- 4) to facilitate the presentation of details such as equilibrium convergence, self-game, and evaluation metrics, thereby enabling comparisons with similar methods. Tabs. 1- 3 correspond to the three branches of the taxonomy framework in Section 4, with the themes of preference diversity, alignment priority, and temporal dynamics, respectively, demonstrating game-based alignment methods. Additionally, Game-based evaluation complements alignment methods by providing controlled settings for probing strategic reasoning, social behavior, and system-level interaction. Tab. 4 therefore lists representative benchmarks that are most directly related to the survey’s main themes. These benchmarks are included as supplementary reference material rather than as a separate organizing axis of the survey. Table 1: Overview of alignment frameworks based on preference diversity. Preference alignment Direction Framework Regularization Formalization Convergence Preference modeling Self-play? Robust alignment? Online? General preference IPOAzar et al. (2024) Explicit Best response (fixed strategy) Average-iterate (ϵε-OPT) ∘ ∘ ∘ ∘ NLHFMunos et al. (2024) Explicit Two-player game (constant-sum) Last-iterate (NE) ∙ ⊙ ∘ ∙ IPO-MDCalandriello et al. (2024) Explicit Two-player game Exist (NE) ∘ ⊙ ∘ ∙ Iterative RLHFYe et al. (2024a) Explicit Two-player game (mini-max) Exist (ϵε-NE) ∙ ∙ ∘ ∙ SPOSwamy et al. (2024) - Two-player game (zero-sum) Average-iterate (ϵε-Minimum Winner) ∙ ∙ ∘ ∙ DNORosset et al. (2024) - Two-player game (zero-sum) Average-iterate (NE) ∙ ∙ ∘ ⊙ REBELGao et al. (2024) - Two-player game (zero-sum) Average-iterate (ϵε-OPT) ∙ ∙ ∘ ⊙ SPPOWu et al. (2024) - Two-player game (constant-sum) Average-iterate (ϵε-NE) ∙ ∙ ∘ ⊙ TANPOWang et al. (2025b) Explicit Two-player game (zero-sum) Average-iterate (ϵε-NE) ∘ ∙ ∘ ∙ ONPOZhang et al. (2025a) - two-player game (zero-sum) Average-iterate (ϵε-NE) ∘ ∙ ∘ ∙ INPOZhang et al. (2025b) Explicit Two-player game Last-iterate (ϵε-NE) ∘ ∙ ∘ ∙ COMALLiu et al. (2026) - Two-player game (zero-sum) Last-iterate (NE) ∙ ∘ ∙ ∙ MPOWang et al. (2025a) - Two-player game (constant-sum) Last-iterate (NE) ∙ ∙ ∘ ∙ RSPO Tang et al. (2025) Explicit Two-player game Last-iterate (NE) ∙ ∙ ∘ ∙ Heterogeneous preference Chidambaram et al. (2024) Explicit Social welfare optimization Average-iterate (ϵε-OPT) ∙ ∘ ∙ ∘ Kim et al. (2025) Explicit Social welfare optimization - ∙ ∘ ∘ ∘ MOPXiong and Singh (2025) Explicit Social welfare optimization Average-iterate (OPT) ∙ ∘ ∘ ⊙ Pessimistic Nash BargainingZhong et al. (2024a) Implicit Social welfare optimization Exist (ϵε-OPT) ∙ ∘ ∘ ∘ GRPORamesh et al. (2024) Implicit Social welfare optimization Average-iterate (ϵε-OPT) ∙ ∘ ∙ ∘ CLP Wang et al. (2024d) Explicit Multi-objective optimization Exist (ϵε-OPT) ∙ ∘ ∘ ∙ PanaceaZhong et al. (2024b) Both Multi-objective optimization Exist (Pareto-OPT) ∙ ∘ ∘ ∙ MO-ODPOGupta et al. (2025a) Explicit Multi-objective optimization - ∙ ∘ ∘ ∙ SIPOLi et al. (2025) Explicit Multi-objective optimization - ∙ ∘ ∘ ⊙ DPAWang et al. (2024c) - Multi-objective optimization - ∙ ∘ ∘ ⊙ DRMLuo et al. (2025) Explicit Multi-objective optimization - ∙ ∘ ∘ ∘ RLHF GameSun et al. (2024a) Explicit Multi-player game (mechanism design) - ∙ ∘ ∘ ∘ Dynamic Bayesian GameHao and Duan (2025) Explicit Multi-player game (dynamic Bayesian) Average-iterate (ϵε-OPT) ∙ ∘ ∘ ∙ BIGWu et al. (2025) - Multi-player game (cooperative) Last-iterate (NE) ∙ ∘ ∘ ∘ ⟨M,N,ϵ,δ⟩ M,N,ε,δ -AgreementNayebi (2025) - Multi-player game (theoretical) - ∘ ∘ ∘ ∙ Notes. (1) Regularization: explicit penalties or procedures that constrain the model’s policy, learning, and outputs. (2) Formalization: how the alignment problem is modeled. (3) Convergence: based on theoretical proof or experimental observation, the optimal strategy exists when converging to an equilibrium point, or explicitly reaching a consensus state in decision-making; some methods provide stronger guarantees such as average-iterate and last-iterate convergence. (4) Preference modeling: a preference probability model (reward model) is learned from human preference data for candidate outputs. (5) Self-play: the same model or homologous variants/clones engage in mutual competition, collaboration, or evaluation within a task or game. (6) Robust alignment: a Nash equilibrium policy guarantees at least a 50%50\% win (or preference) rate against any alternative policy under general preferences, and guarantees the worst-group performance under heterogeneous preferences. (7) Online: new training data are obtained from the current model and the parameters are updated accordingly. Symbols ∙ , ∘ , and ⊙ denote conditions that are met, not met, and adjustable mixed settings, respectively; ‘-’ indicates not applicable or not reported. Table 2: Overview of alignment frameworks based on alignment priority. Enhancement via interaction Direction Framework Target Formalization Interaction structure Convergence External feedback Self-play? Online? Synchronous interaction Jatova et al. (2024) Safety alignment Two-player game LLM vs adversarial generator Exist (NE) ∙ ∘ ∙ GPO Zheng et al. (2024) Safety alignment Two-player game (zero-sum) LLMs (attacker vs defender) Average-iterate (NE) ∙ ∘ ∙ SELF-REDTEAM Liu et al. (2025) Safety alignment Two-player game (zero-sum) LLM (attacker / defender) - ∙ ∙ ∙ MTSA Guo et al. (2025) Safety alignment Multi-turn RL LLMs (attacker vs defender) - ∙ ∘ ⊙ SPAG Cheng et al. (2024) Reasoning Two-player game (zero-sum) LLM (attacker / defender) - ∙ ∙ ∘ APO Cheng et al. (2023) Effective alignment Two-player game (mini-max) LLM vs RM - ∘ ∘ ∙ SPIN Chen et al. (2024b) Self-improvement Two-player game LLM (generator / distinguisher) Exist ∘ ∙ ⊙ LANA Azarafrooz and Faal (2024) Self-alignment Two-player game (zero-sum) LLM (generation / evaluation) Average-iterate (ϵε-NE) ∘ ∙ ∙ DuoGuard Deng et al. (2025) Safety alignment Two-player game (mini-max) LLMs (generator vs classifier) Last-iterate (NE) ∘ ∘ ⊙ SPC Chen et al. (2025b) Reasoning Two-player game LLM (generator / critic) - ∘ ∙ ∙ PEG Chen et al. (2025a) Truthfulness Peer elicitation game LLMs (peer discriminators) Last-iterate (Truthful NE) ∘ ∘ ∙ Sequential interaction STA–RLHF Makar-Limanov et al. (2024) Human preference Two-player game (Stackelberg) LM (leader) vs RM (follower) - ∘ ∘ ⊙ SPAC Ji et al. (2024) Human preference Two-player game (Stackelberg) LLM (policy model / critic) Average-iterate (ϵε-OPT) ∘ ∙ ∘ SGPO Chu et al. (2025) Effective alignment Two-player game (Stackelberg) LLM vs pref. distribution Last-iterate (SE) ∘ ∘ ⊙ Anyprefer Zhou et al. (2025c) Effective alignment Two-player game (Markov) LLMs (target model & judger) - ∙ ∘ ∘ CORY Ma et al. (2024) Robustness Cooperative multi-agent RL LLM (pioneer / observer) - ∙ ∙ ∙ Notes. (1) Target: research question. (2) Formalization: how the alignment problem is modeled. (3) Interaction structure: players and roles engaging in interaction. (4) Convergence: based on theoretical proof or experimental observation, the optimal strategy exists when converging to an equilibrium point, or explicitly reaching a consensus state in decision-making; some methods provide stronger guarantees such as average-iterate and last-iterate convergence. (5) External feedback: the AI model (agent) relies on externally added, manually annotated data for optimization and updates, rather than self-generating new annotations via interaction. (6) Self-play: the same model or homologous variants/clones engage in mutual competition, collaboration, or evaluation within a task or game. (7) Online: new training data are obtained from the current model and the parameters are updated accordingly. Symbols ∙ , ∘ , and ⊙ denote conditions that are met, not met, and adjustable mixed settings, respectively; ‘-’ indicates not applicable or not reported. Table 3: Overview of alignment (simulation) frameworks based on temporal dynamics. Dynamic evolution Direction Framework Target Formalization Application Stage Convergence Online? Adaptive evolution Klassen et al. (2024) Temporal pluralism values Non-Markovian process Measurement - - - Suzuki and Arita (2024) Social cooperation Evolutionary game (mutation) Simulation Inference - - ProgressGym Qiu et al. (2024) Social moral progress Partially observable MDP Improvement Post-training - - Evolutionary Agent Li et al. (2024b) Norms and cooperation Evolutionary game (mutation) Improvement Inference - - EVA Ye et al. (2024b) Preference alignment Creator-solver game (mutation) Improvement Post-training - ∙ Co-evolution SPARTA Jiang et al. (2025) Preference / instruction Collective competition Improvement Post-training - ⊙ Red-Team Game Ma et al. (2023) Safety alignment Red-blue team game Improvement Post-training Exist (ϵε-NE) ∙ SSI Liu et al. (2024b) Social alignment Social interaction Improvement Post-training Exist ∘ Artificial Leviathan Dai et al. (2024) Social order Social interaction (survival sandbox) Simulation Inference - - CAMEL Li et al. (2023) Social cooperation Role-playing interaction (dialogue) Simulation Inference - - AI Collectives Lai et al. (2024) Social alignment Social interaction (community) Simulation Inference - - Alympics Mao et al. (2025) Empirical game theory Strategic games (survival resources) Simulation Inference - - FairMindSim Lei et al. (2024) Social value alignment Ethical dilemma Simulation Inference - - Du et al. (2024) Reasoning / factuality Multi-agent debate Improvement Inference Exist (Consensus) - MAD Liang et al. (2024) Reasoning Multi-agent debate Improvement Inference - - Notes. (1) Target: research question. (2) Formalization: how the process of dynamic evolution is driven or modeled, such as game, social interaction, and debate. (3) Application: quantify and enhance the alignment performance of AI systems, or conduct interdisciplinary research by simulating social behavior and group evolution processes through AI agents, such as social sciences and empirical game theory. (4) Stage: the phase during which the training or evaluation framework is running. (5) Convergence: based on theoretical proof or experimental observation, the optimal strategy exists when converging to an equilibrium point, or explicitly reaching a consensus state in decision-making. (6) Online: new training data are obtained from the current model and the parameters are updated accordingly. Symbols ∙ , ∘ , and ⊙ denote conditions that are met, not met, and adjustable mixed settings, respectively; ‘-’ indicates not applicable or not reported. Table 4: Overview of evaluation benchmarks based on games. Evaluation and Benchmark Paradigm Ref. Target Formalization Metric Benchmark Oriented-O/P Classic Games Fan et al.Fan et al. (2024) Rationality Dictator Game Ring-network Game Action Accuracy Average Payoff Preference Consistency - ∘ Wang et al. Wang et al. (2024b) Strategic reasoning Robinson-Goforth Topology of 2×2 Games Bias Degree Perfect Accuracy Rate Inconsistency Degree Tmgbench ⊙ Lor‘e et al.Lorè and Heydari (2024) Strategic behavior Social Dilemmas Games Cooperation Rate Dominance Statistic - ⊙ Topsaka et al. Topsakal et al. (2024) Rule comprehension Strategic thinking Grid-based Games Win/DQ Rates Invalid Moves Missed Opportunities ✔ ⊙ Lu Lu (2024) Strategic levels Adaptive learning Beauty Contest Strategic Degree Convergence Average Payoffs - ∘ Sreedhar et al. Sreedhar et al. (2025) Cooperative prosocial behavior Public Goods Game Average Contribution Emergence Frequency - ⊙ Akata et al, Akata et al. (2025) Cooperation Coordination Finitely repeated 2×2 games Score Ratio Defection Rate Coordination Rate - ⊙ Fontan et al. Fontana et al. (2025) Cooperative behavior Iterated Prisoner’s Dilemma Behavioral Traits SFEM Scores ✔ ⊙ Xie et al. Xie et al. (2024) Human trust behavior Trust Game Trust Rate Valid Response Rate Returned/Sent Ratio ✔ ⊙ Zakazo et al. Zakazov et al. (2024) Social alignment Ultimatum Game Acceptance Rate Disobedience Ratio - ⊙ Davidson et al. Davidson et al. (2024) Instruction-following Faithfulness Structured Negotiation Agreement Rate Normalized Payoffs LAMEN ⊙ Xia et al. Xia et al. (2024) Bargaining abilities Bargaining Deal Rate First Bid Ratio Sum of Normalized Profits AmazonHistoryPrice ∙ Social games Zhou et al. Zhou et al. (2025b) Social intelligence World tree scenarios Goal Achievement Ratio Ability Selection Accuracy SocialEval ⊙ Zhou et al. Zhou et al. (2024) Social intelligence Mixed-motive Markov games Goal Completion Social Rules SOTOPIA-EVAL ⊙ Liu.et al. Liu et al. (2024a) Role consistency Interactive dialogue Self-Knowledge accuracy RoleAgentBench ∘ Wang et al. Wang et al. (2024g) Role consistency Role-playing dialogue Win rate Average ranking RoleBench ∙ Zhu et al. Zhu et al. (2025) Cooperation & competition Mutual/conflicting scenarios Task Score Milestone-based KPI MultiAgentBench ⊙ Wang et al. Wang et al. (2024a) Social intelligence Social sandbox tasks Success Rate Goal Condition SR STSS benchmark ∙ Abdelnabi et al. Abdelnabi et al. (2024) Negotiation & security Multi-agent, multi-issue negotiation Final success Collective score Score leakage ratio ✔ ⊙ Xie et al. Xie et al. (2026) Social behavior Mixed-motive games Overall Score Talk–Action Consistency Trajectory Analysis M3-Bench ⊙ Curvo et al. Curvo (2025) Deception & trust Communication utterances Voting decisions Traitor Survival Rate Trust Network Stability Faithful Correctness Rate ✔ ⊙ Motwani et al. Motwani et al. (2024) Secret collusion Covert communication Cipher round-trip accuracy Coordination rates ✔ ⊙ Notes. (1) Target: research question. (2) Formalization: evaluation scenario or benchmark type. (3) Metric: quantifiable evaluation metrics that are typically results- or process-oriented. (4) Benchmark: self-named benchmarks are listed explicitly, while unnamed benchmarks are represented by ✔. (5) Oriented-O/P: the evaluation paradigm follows outcome-oriented (O: ∙ ), process-oriented (P: ∘ ), or both (⊙ ). The symbol ‘-’ is used in the same way as above.