Paper deep dive
Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignment
Zachary Wojtowicz, Michelle Si, Finale Doshi-Velez, Ariel Procaccia
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/26/2026, 5:26:25 AM
Summary
This paper reformulates AI alignment as a linear optimization problem over a convex 'impact space' to address the social choice problem of aggregating divergent human preferences. It demonstrates that this framework allows for the application of welfare economics and mechanism design, proving that mechanisms like voting-by-issues and random dictatorship are strategyproof and unanimous. The authors derive alignment protocols that maximize utilitarian social welfare subject to constraints on harm and illustrate these concepts using empirical data from kidney allocation, food distribution, and trolley problems.
Entities (7)
Relation Signals (6)
Impact Space → enables → linear optimization
confidence 95% · the alignment problem can be reformulated as linear optimization over a convex impact space
Voting-by-issues → is → strategyproof
confidence 95% · We apply this transformation to show that voting-by-issues and random-dictatorship mechanisms are strategyproof and unanimous.
Random Dictatorship → is → strategyproof
confidence 95% · We apply this transformation to show that voting-by-issues and random-dictatorship mechanisms are strategyproof and unanimous.
Kidney Allocation → usedforvalidation → Alignment Protocols
confidence 92% · We illustrate the welfare implications of these alignment protocols empirically using real human preferences over kidney allocation
Reinforcement Learning from Human Feedback → hasweakness → poor social choice guarantees
confidence 90% · The standard approach to aligning frontier AI models—reinforcement learning from human feedback—largely sidesteps this question and has poor social choice guarantees.
Alignment Protocols → maximizes → Utilitarian Social Welfare
confidence 90% · derive a family of alignment protocols that maximize utilitarian social welfare subject to various social desiderata
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:When an AI algorithm makes decisions that affect more than one person, aligning it becomes a problem of social choice: how should people's divergent preferences about system behavior be reconciled and aggregated into a single coherent model? The standard approach to aligning frontier AI models$\unicode{x2013}$reinforcement learning from human feedback$\unicode{x2013}$largely sidesteps this question and has poor social choice guarantees. However, it remains unclear what alternative should replace it. We show that, by focusing directly on an algorithm's welfare consequences, the alignment problem can be reformulated as linear optimization over a convex impact space, which makes it amenable to the standard toolkit of welfare economics and mechanism design. This reformulation clarifies how alignment protocols translate into welfare consequences and, conversely, how a social planner's desired constraints on welfare consequences can be translated back into alignment protocols. We apply this transformation to show that voting-by-issues and random-dictatorship mechanisms are strategyproof and unanimous. Demonstrating the reverse direction, we also apply the impact representation to derive a family of alignment protocols that maximize utilitarian social welfare subject to various social desiderata, such as bounds on individual or group harm. We illustrate the welfare implications of these alignment protocols empirically using real human preferences over kidney allocation, charitable food distribution, LLM responses, and trolley problems.
Tags
Links
- Source: https://arxiv.org/abs/2608.24046v1
- Canonical: https://arxiv.org/abs/2608.24046v1
Trouble viewing inline? Open PDF directly →
Full Text
117,431 characters extracted from source content.
Expand or collapse full text
Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignment Zachary Wojtowicz MIT Cambridge MA 02139 zachwoj@mit.edu Michelle Si Harvard University Cambridge MA 02138 msi@g.harvard.edu Finale Doshi-Velez Harvard University Cambridge MA 02138 finale@seas.harvard.edu Ariel Procaccia Harvard University Cambridge MA 02138 arielpro@seas.harvard.edu Abstract When an AI algorithm makes decisions that affect more than one person, aligning it becomes a problem of social choice: how should people’s divergent preferences about system behavior be reconciled and aggregated into a single coherent model? The standard approach to aligning frontier AI models—reinforcement learning from human feedback—largely sidesteps this question and has poor social choice guarantees. However, it remains unclear what alternative should replace it. We show that, by focusing directly on an algorithm’s welfare consequences, the alignment problem can be reformulated as linear optimization over a convex impact space, which makes it amenable to the standard toolkit of welfare economics and mechanism design. This reformulation clarifies how alignment protocols translate into welfare consequences and, conversely, how a social planner’s desired constraints on welfare consequences can be translated back into alignment protocols. We apply this transformation to show that voting-by-issues and random-dictatorship mechanisms are strategyproof and unanimous. Demonstrating the reverse direction, we also apply the impact representation to derive a family of alignment protocols that maximize utilitarian social welfare subject to various social desiderata, such as bounds on individual or group harm. We illustrate the welfare implications of these alignment protocols empirically using real human preferences over kidney allocation, charitable food distribution, LLM responses, and trolley problems. 11 1 All code can be found at https://github.com/michelleeesi/hiddenstructure. 1 Introduction Alignment is typically framed as the problem of ensuring that an AI system respects human preferences. However, AI systems increasingly take actions that affect more than one person, and some disagreement about how these systems should behave is inevitable. In such cases, alignment becomes a problem of social choice: how should an AI model combine the potentially conflicting preferences of many individuals to select among alternatives? Recently, reinforcement learning from human feedback (RLHF) has become the standard approach to aligning AI systems (Bai et al., 2022). In this method, human annotators provide structured feedback on model responses, typically in the form of pairwise comparisons, which are then pooled together and used to train a model, either directly or through a proxy reward model. Although widely used, this approach does not directly engage with the social choice problem intrinsic to alignment, and recent work has shown that RLHF produces models that have a variety of potentially undesirable properties, such as the down-weighting of minority groups (Ge et al., 2024; Gölz et al., 2025; Shirali et al., 2025; Chakraborty et al., 2024; Chidambaram et al., 2024; Siththaranjan et al., 2023; Halpern et al., 2025). These limitations have prompted recent calls to elicit preferences at a more granular level, such as the group or individual, rather than pooling all feedback anonymously (Park et al., 2024; Shirali et al., 2025; Li et al., 2024; Poddar et al., 2024; Dai and Fleisig, 2024). However, this leaves open the question: how should individual preferences, once collected, be combined? Social choice theory was developed precisely to guide such decisions (Conitzer et al., 2024). Yet applying classical results from the field to alignment has been complicated by the structure of modern AI systems, in which alternatives are high-dimensional model parameterizations rather than discrete options. The key insight of our paper is that this complexity is an artifact of the parameterization, not of the underlying alignment problem itself: A decision-making algorithm can be summarized by a single impact vector that captures how its choices affect individual welfare during deployment. Once reformulated in impact space, aligning a model to maximize utilitarian social welfare reduces to optimizing a linear function over a convex polytope. Our main theorem establishes this correspondence. Specifically, we give a constructive procedure for both reformulating an alignment problem as an instance of linear social choice (Ge et al., 2024) and mapping the solution back to a deployable model parameter. Contributions: We show that directly considering the distributional welfare impact of an AI model provides clean insights into the social choice properties of alignment protocols. We introduce our main result in Section 4, which shows that maximizing social welfare in the linear social choice setting is a linear optimization problem over a convex polytope that we call the impact space (Theorem 1). In Section 5, we show that in impact space strategyproof alignment mechanisms inherit the classical menu structure of option-set mechanisms (Lemma 2, after Barbera and Peleg, 1990), and we show that aggregating each deployment query by a monotone vote yields the strategyproof voting-by-issues rules (Theorem 2), with random dictatorships as a key example. We also establish that random dictatorships can be implemented, in expected impact, by a single model parameter (Theorem 3). In Section 6, we use our framework to characterize a family of mechanisms that maximize welfare subject to constraints on individual and group outcomes (Theorem 4), such as bounds on harm (Corollary 2) and public-spirited trade-offs between personal loss and public gain (Corollary 3). These applications highlight how working in impact space makes it easy to build social objectives beyond utilitarian welfare maximization into AI systems. Finally, in Section 7 we illustrate the welfare consequences of these alignment protocols using real human preferences from four domains: kidney allocation, charitable food distribution, LLM responses, and trolley problems. 2 Related Work Our work builds on a recent literature that studies the distributional welfare implications and other social properties of RLHF, particularly for populations with diverse preferences. Characterizing RLHF Distortion. Understanding the shortcomings of RLHF has become an area of heightened interest (Siththaranjan et al., 2023; Shirali et al., 2025; Gölz et al., 2025). Gölz et al. (2025) measure how suboptimal (relative to a utilitarian optimum) an alignment method can be in aggregate, while we focus on what generates that suboptimality on a participant-to-participant (or group-to-group) basis. Shirali et al. (2025) show that pooled RLHF differs from using the average of individual choice probabilities by a variance term informed by preference dispersion. Strategyproof Alignment. A separate line of work studies strategic behavior in preference collection (Sun et al., 2024; Kleine Buening et al., 2026). Kleine Buening et al. (2026) show that standard RLHF procedures can be manipulated by strategic feedback providers and introduce the Pessimistic Median of MLEs algorithm, which is approximately strategyproof. We study a much more restricted domain (linear rewards) that gives simple results: the strategyproof alignment mechanisms are option-set mechanisms, with the random dictatorship as a focal member, and a random dictatorship need not be implemented by literally sampling one person’s personalized model at deployment time—a single preference parameter θλspθ^sp_λ reproduces its expected impact. Alternative Alignment Protocols. Other papers propose algorithmic alternatives for heterogeneous preferences (Chakraborty et al., 2024; Chidambaram et al., 2024; Park et al., 2024), from MaxMin-RLHF to personalization. These papers study ways to model or optimize over latent preference types given some desiderata, but our approach is more general and geometric: we show how different social objectives correspond to different optimization problems over that same set—the impact space. This lets us compare utilitarian alignment, strategyproof alignment, harm-bounded alignment, and public-spirit constraints within one common representation. Our work is most closely related to the linear social choice framework of Ge et al. (2024). Theorem 1 formally establishes that, when utilities are linearly representable, alignment is an instance of linear social choice. 3 Preliminaries 3.1 Choice in the Linear Utility Setting We begin with the standard model of stochastic choice assumed by Direct Preference Optimization (Rafailov et al., 2023) and related methods. Let X be a finite set of contexts, Y a finite set of actions, and N the number of agents. Agent n has utility un:X×Y→ℝu_n:X× Y . For a binary choice problem (x,y,y′)(x,y,y ), we assume stochastic choice follows a Bradley–Terry–Luce model pn(y≻y′∣x,y,y′)=σ(un(y∣x)−un(y′∣x)),σ(a)=exp(a)1+exp(a).p_n(y y x,y,y )=σ\! (u_n(y x)-u_n(y x) ), σ(a)= (a)1+ (a). Each context–action pair is embedded as ϕ(x,y)∈ℝkφ(x,y) ^k. Our central assumption is that each agent’s utility can be expressed linearly in this feature space: un(y∣x)=ϕ(x,y)⊤θnu_n(y x)=φ(x,y) _n for θn∈ℝk. _n ^k. For social welfare weights w∈ΔNw∈ _N, define the utilitarian preference parameter θ¯w=∑nwnθn θ_w= _nw_n _n. By linearity, the w-weighted utilitarian aggregate is represented by the same feature map: u¯w(y∣x)=∑nwnun(y∣x)=ϕ(x,y)⊤θ¯w. u_w(y x)= _nw_nu_n(y x)=φ(x,y) θ_w. Restricting attention to weighted sums of individual utilities is itself justified by the aggregation theorem of Harsanyi (1955), which shows that a social preference satisfying the expected-utility axioms over a convex set of prospects, together with Pareto indifference, must take exactly this form. Under linear utilities, each pairwise comparison depends only on the feature difference α(x,y,y′)=ϕ(x,y)−ϕ(x,y′).α(x,y,y )=φ(x,y)-φ(x,y ). We write αz∈ℝk _z ^k for the vector associated with a choice problem z∈Z=X×Y×Yz∈ Z=X× Y× Y. Let qtrain∈Δ([N]×Z)q_train∈ ([N]× Z) be the distribution of training query-agent pairs, and qdep∈Δ(Z)q_dep∈ (Z) be the distribution of deployment problems. 3.2 Deployment Welfare A central objective for alignment is ensuring that model behaviors improve aggregate social welfare. We measure the welfare agent n receives from a model deployed at parameter θ as the amount of utility it generates relative to the baseline of choosing randomly between the two options, Un(θ)=z∼qdep[αz⊤θn(σ(αz⊤θ)−12)].U_n(θ)=E_z q_dep [ _z _n (σ( _z θ)- 12 ) ]. The utilitarian social welfare of a deployed model is then the sum of the individual welfares, U(θ)=∑n=1NwnUn(θ)U(θ)= _n=1^Nw_nU_n(θ). 4 Framework: Alignment as Linear Optimization in Impact Space As reviewed in Section 2, recent literature has catalogued many ways that anonymous RLHF can fail to represent heterogeneous populations, violating basic welfare criteria such as Pareto efficiency (Gölz et al., 2025; Shirali et al., 2025; Ge et al., 2024; Chakraborty et al., 2024; Park et al., 2024). A natural remedy is to estimate preferences at the individual or group level and then combine these estimates into a single deployed model. But this immediately raises the social choice questions that anonymous RLHF sidesteps: which aggregation rule should be used? What welfare guarantees does it provide? Can agents manipulate it? Answering these questions requires a framework in which the welfare consequences of different alignment protocols can be directly compared. However, even if the goal of alignment is to combine the preferences of many individuals, working directly in model-parameter space obscures the comparison, because welfare is a nonlinear function of the model’s parameters. The key insight of our framework is that this nonlinearity can be avoided if we consider the set of potential welfare implications directly, instead of the parameterization. Under our linear utility assumptions, a model’s welfare effects can be summarized by a single vector—its impact—that lives in the same space as preferences and deployment queries (see Figure 1). Welfare is linear in this vector, so the social alignment problem reduces to optimizing a linear function over a convex set: a problem for which we have the tools of welfare economics and mechanism design at our disposal. Definition 1. The impact of a model with parameter θ is ψ(θ)=z∼qdep[αz(σ(αz⊤θ)−12)]∈ℝkψ(θ)=E_z q_dep [ _z (σ( _z θ)- 12 ) ] ^k. The impact vector is a sufficient statistic for welfare. Since Un(θ)=z∼qdep[αz⊤θn(σ(αz⊤θ)−12)]=θn⊤ψ(θ)U_n(θ)=E_z q_dep[ _z _n(σ( _z θ)- 12)]= _n ψ(θ), agent n’s welfare is the inner product of their preference with the impact vector.22 2 Economically, ψ is analogous to a vector of quantities delivered and θn _n encodes agent n’s marginal valuations. Utilitarian welfare is therefore also linear: U(ψ)=θ¯w⊤ψU(ψ)= θ_w ψ. Ψ¯ OψA⋆ _A ψB⋆ _B ψ⋆ψ θA _AθB _Bθ¯w θ_wψspψ^sp Figure 1: The impact-space view of social alignment. Preferences (θn _n, dashed) and the feasible impact set Ψ¯ live in the same Euclidean space; for finitely many deployment queries, Ψ¯ is a convex, origin-symmetric polytope. Preferences differ in magnitude and need not themselves be feasible impacts: both θA _A and θB _B lie outside Ψ¯ . Because welfare Un(ψ)=θn⊤ψU_n(ψ)= _n ψ is linear, each agent’s ideal impact ψn⋆=argmaxψ∈Ψ¯θn⊤ψ _n = _ψ∈ _n ψ is the vertex of Ψ¯ farthest in the direction of θn _n, which in general is not collinear with θn _n. The utilitarian optimum ψ⋆ψ maximizes welfare in the weighted-average direction θ¯w=∑nwnθn θ_w= _nw_n _n, where the iso-welfare line ψ:θ¯w⊤ψ=const\ψ: θ_w ψ=const\ is tangent to Ψ¯ . The dashed segment conv(ψA⋆,ψB⋆)conv( _A , _B ) is the set of expected impacts attainable by strategyproof mechanisms (Section 5), with ψspψ^sp the equal-weight random dictatorship. This lets us rewrite social alignment as a linear optimization over the set of feasible impacts: ψ⋆∈argmaxψ∈Ψ¯∑n=1Nwnθn⊤ψ,Ψ¯=closureψ(θ):θ∈ℝk⊆Sdep.ψ ∈ _ψ∈ _n=1^Nw_n _n ψ, =closure\ψ(θ):θ ^k\ S_dep. (1) Here, SdepS_dep is the subspace spanned by deployment queries, and Ψ¯ is the feasible set of impacts achievable by some model parameter.33 3 We optimize over the closure because deterministic limits can be approached to arbitrary precision. The following theorem characterizes the geometry of the impact set. Theorem 1. Let Ψ=ψ(θ):θ∈ℝk =\ψ(θ):θ ^k\ and Ψ¯=cl(Ψ) =cl( ), and let SdepS_dep be the subspace of ℝkR^k spanned by the deployment queries. Then: 1. Ψ¯⊆Sdep S_dep is the origin-symmetric zonotope Ψ¯=z∼qdep[tzαz]:tz∈[−12,12] =\E_z q_dep[t_z\, _z]:t_z∈[- 12, 12]\ generated by the deployment queries, and Ψ=relintΨ¯ =relint , the interior of Ψ¯ relative to SdepS_dep. In particular, Ψ¯ is convex and centrally symmetric (Ψ¯=−Ψ¯ =- ), and ψ is antisymmetric (ψ(−θ)=−ψ(θ)ψ(-θ)=-ψ(θ)) with ψ(0)=0ψ(0)=0. 2. On SdepS_dep, the map ψ is a C∞C^∞ diffeomorphism from SdepS_dep onto Ψ . Every feasible impact ψ∈Ψψ∈ is implemented by a unique parameter in SdepS_dep. 3. ψ is bounded, with supθ∈ℝk‖ψ(θ)‖≤12z∼qdep‖αz‖. _θ ^k\|ψ(θ)\|≤ 12\,E_z q_dep\| _z\|. 4. Scaling is monotone in the direction of θ: θ⊤∂ψ(νθ)∂ν=z∼qdep[(αz⊤θ)2v(ναz⊤θ)]≥0,θ ∂ψ(νθ)∂ν=E_z q_dep [( _z θ)^2v(ν _z θ) ]≥ 0, where v(x)=σ′(x)v(x)=σ (x). 5. The deterministic boundary in direction θ is limν→∞ψ(νθ)=12z∼qdep[αzsign(αz⊤θ)]. _ν→∞ψ(νθ)= 12E_z q_dep [ _zsign( _z θ) ]. For generic θ (i.e., αz⊤θ≠0 _z θ≠ 0 for all z∈supp(qdep)z (q_dep)), this limit is the unique maximizer of θ⊤ψθ ψ over Ψ¯ , and it is a vertex. This structure allows us to restate the utilitarian optimal model as the boundary point of Ψ¯ in the direction of the weighted average preference θ¯w θ_w. Proposition 1. The limit limν→∞ψ(νθ¯w) _ν→∞ψ(ν\, θ_w) exists and is a utilitarian-optimal impact: limν→∞ψ(νθ¯w)∈argmaxψ∈Ψ¯θ¯w⊤ψ _ν→∞ψ(ν\, θ_w)∈ _ψ∈ θ_w ψ. In particular, we may take ψ⋆=limν→∞ψ(νθ¯w)ψ = _ν→∞ψ(ν\, θ_w), and this maximizer is unique whenever αz⊤θ¯w≠0 _z θ_w≠ 0 for all z∈supp(qdep)z (q_dep). We can now explicitly describe how scaling affects individual welfare. For agent n, ∂Un(νθ)∂ν=θn⊤∂ψ(νθ)∂ν=z∼qdep[(αz⊤θn)(αz⊤θ)v(ναz⊤θ)]. ∂ U_n(νθ)∂ν= _n ∂ψ(νθ)∂ν=E_z q_dep [( _z _n)( _z θ)v(ν _z θ) ]. Thus scaling helps agents whose preferences align with the model direction across deployment queries, and harms agents whose preferences covary negatively. More generally, translating the alignment problem into impact space and considering the entire problem from the perspective of welfare invokes externalities as the objects we would like to regulate as we design alignment protocols. Our framework gives us the machinery to do so. In the remaining sections of this paper, we discuss how our framework leads to straightforward ways of addressing various types of externalities. 5 Strategyproof Alignment in Impact Space When individuals have a say in how an AI system is aligned, they may be tempted to misreport their preferences to steer the outcome in their favor. Even though this problem is not a major issue for chatbots and other present AI platforms, it will likely become more salient as algorithms are deployed in increasingly consequential social domains, and their behavioral policies therefore become the subject of more direct and explicit contention among impacted stakeholders. Misreporting one’s true preferences to distort alignment in a self-serving way imposes welfare externalities on others. In economics, the field of mechanism design regulates such externalities by identifying procedures for combining preferences that are strategyproof—i.e., such that no agent can benefit from misreporting, so truthful participation is always a weakly dominant strategy. The impact-space formulation places social alignment within the classical theory of strategyproof mechanism design, enabling us to apply its toolkit directly. Because welfare is linear in the impact vector and the feasible impact set Ψ¯ is convex, we identify strategyproof alignment mechanisms in three nested steps. The general family consists of the option-set mechanisms of Barbera and Peleg (1990). Restricting to per-query aggregation yields voting-by-issues rules. A leading example is the random dictatorship, which we show is implementable by a single model parameter (exactly or in the limit) and is the deterministic limit of anonymous RLHF when the labeling pool is common across queries. Our treatment of strategyproofness complements recent work by Kleine Buening et al. (2026), who study strategic behavior in offline RLHF. Their setting is a learning problem: labelers strategically manipulate comparison labels, and they propose an approximately strategyproof learning algorithm. In contrast, we abstract away from statistical estimation and focus directly on the impact that preference reports have on an AI model’s behavior and, through it, the distribution of welfare among those impacted. By taking advantage of our linearity assumption, we can characterize a family of simple and intuitive strategyproof mechanisms. Definition 2. A mechanism is a function μ:ℝN×k→Δ(Ψ¯)μ:R^N× k→ ( ) that maps a report profile θ^=(θ^1,…,θ^N) θ=( θ_1,…, θ_N) to a probability distribution over the space of impact vectors Ψ¯ . Throughout this section we restrict attention to generic profiles and reports—those with αz⊤θn≠0 _z _n≠ 0 for every z∈supp(qdep)z (q_dep) and every n—so that vote signs and ideal impacts ψn⋆=argmaxψ∈Ψ¯θn⊤ψ _n = _ψ∈ _n ψ are single-valued (Theorem 1). Equivalently, one may fix any tie-breaking convention on the measure-zero set of non-generic reports. Intuitively, we want a mechanism to satisfy two basic requirements. The first, as we have discussed, is strategyproofness: an agent should not be able to obtain a better outcome, according to their true preferences, by misreporting those preferences. Definition 3. A mechanism is strategyproof if, for all preference profiles θ∈ℝN×kθ ^N× k and n∈[N]n∈[N], all θn′ _n , [θn⊤μ(θ)]≥[θn⊤μ(θn′,θ−n)],E [ _n μ(θ) ] [ _n μ( _n , _-n) ], where (θn′,θ−n)( _n , _-n) denotes the profile obtained by replacing agent n’s true preferences with θn′ _n . The second is unanimity: If all agents have exactly the same preferences, then there is no social conflict to resolve: the mechanism should simply choose the impact vector that is best according to that shared preference. Definition 4. A mechanism μ is unanimous if, for any profile θ such that θn=θ¯ _n= θ for all n∈[N]n∈[N], μ(θ)μ(θ) is the point mass on argmaxψ∈Ψ¯θ¯⊤ψ _ψ∈ θ ψ (a singleton, since θ¯ θ is generic). We point out a powerful consequence of the linearity of θn⊤ψ _n ψ: that a stochastic mechanism can be fully characterized by its expected impact vector at each profile. Lemma 1. Let μ¯(θ)=ψ∼μ(θ)[ψ]∈Ψ¯ μ(θ)=E_ψ μ(θ)[ψ]∈ be the expected impact. Then [θn⊤μ(θ)]=θn⊤μ¯(θ)E[ _n μ(θ)]= _n μ(θ), so strategyproofness and unanimity depend on μ only through the expected-impact map μ¯:ℝN×k→Ψ¯ μ:R^N× k→ . This enables us to characterize classes of alignment protocols satisfying strategyproofness and unanimity while restricting attention to deterministic expected-impact maps. 5.1 The General Family: Option-Set Mechanisms The impact space reformulation enables us to recognize social alignment as an instance of strategyproof social choice, which can be characterized generally using the option-set principle (Barbera and Peleg, 1990). For a report profile, let On(θ−n)=μ¯(θn′,θ−n):θn′∈ℝk⊆Ψ¯O_n( _-n)=\ μ( _n , _-n): _n ^k\ be agent n’s option set: the impacts it can induce by varying its own report while the others’ reports are held fixed. Lemma 2. A mechanism μ is strategyproof if and only if there exist menus Mn(θ−n)⊆Ψ¯M_n( _-n) , each depending only on the other agents’ reports, such that, at every profile, the mechanism deploys each agent’s most-preferred impact from its menu: μ¯(θ)∈argmaxψ∈Mn(θ−n)θn⊤ψ∀n. μ(θ)∈ _ψ∈ M_n( _-n) _n ψ ∀ n. (2) In particular, we can set Mn(θ−n)=On(θ−n)M_n( _-n)=O_n( _-n) to be the option set itself. The lemma holds for any convex feasible set. It says that a strategyproof alignment protocol is equivalent to one that posts a menu of achievable model behaviors to each participant, which they cannot themselves enlarge, and deploys their favorite. The menu form is exact but implicit: an explicit characterization of all strategyproof mechanisms for expected-utility agents is a long-standing open problem even in the classical case of lotteries over finitely many alternatives (Barberà, 2011). We therefore proceed by explicitly identifying structured subfamilies, each a member of the menu family above. 5.2 Reduction 1: Voting by Deployment Query As established by Theorem 1, the impact set is a zonotope generated by the deployment queries. By decomposing the expectation over deployment queries into its individual terms, we can construct a strategyproof mechanism that aggregates the population’s preferences query by query. To ensure strategyproofness, the mechanism must ensure that agents cannot benefit by exaggerating the magnitude of their reports, which can be accomplished by reducing preference information to directional votes on each query. Write sz,n=sign(αz⊤θn)∈±1s_z,n=sign( _z _n)∈\± 1\ for how agent n would decide query z.44 4 For (z,n)(z,n) such that αz⊤θn=0 _z _n=0, an arbitrary tie-breaking rule can be used, which does not affect welfare and therefore does not affect strategyproofness. Definition 5. A voting-by-issues mechanism is specified by, for each query z, an aggregator fz:±1N→[−1,1]f_z:\± 1\^N→[-1,1] that is nondecreasing in each argument with fz(1,…,1)=1f_z(1,…,1)=1 and fz(−1,…,−1)=−1f_z(-1,…,-1)=-1. Its expected impact is μ¯(θ)=12z∼qdep[fz(sz,1,…,sz,N)αz] μ(θ)= 12\,E_z q_dep [f_z(s_z,1,…,s_z,N)\, _z ] (3) Theorem 2. Every voting-by-issues mechanism is strategyproof and unanimous. Voting-by-issues rules are the impact-space analogue of generalized median voter schemes (Barberà et al., 1993): each deployment query is settled by a monotone aggregation of the participants’ preferred sides. Our setting differs from those schemes in that mechanisms are stochastic, outcomes range over the impact space, and preferences are linear in the same space. The fact that the impact set is a zonotope generated by the queries, and therefore exhibits central symmetry, makes this per-query aggregation feasible. 5.3 Reduction 2: Random Dictatorships Definition 6. A mechanism is a random dictatorship with weights λ∈ΔNλ∈ _N if μ(θ)=∑n=1Nλnδ(ψn⋆)μ(θ)= _n=1^N _nδ( _n ). Under Lemma 1, this is impact-equivalent to μ¯(θ)=∑n=1Nλnψn⋆ μ(θ)= _n=1^N _n _n . Proposition 2. A random dictatorship is impact-equivalent to the voting-by-issues mechanism whose aggregators are the vote averages fz(sz,⋅)=∑nλnsz,nf_z(s_z,·)= _n _ns_z,n. It is therefore strategyproof and unanimous. Classical results force strategyproofness to coincide with random dictatorship alone, but only on richer preference domains: a universal domain over the outcomes (Gibbard, 1977), or strictly convex single-peaked preferences over a convex set (Dutta et al., 2002). The assumption that preferences can be represented linearly in the model’s feature space is not without loss, as it restricts the power of more general preference profiles to constrain the space of mechanisms. This yields a richer strategyproof class, with random dictatorship as a focal example. As defined, a random dictatorship draws a single individual, with probability λn _n, and deploys their ideal impact. By Lemma 1 it is impact-equivalent to the more natural protocol of choosing an individual at random on a query-by-query basis and deciding each query the way that individual would. A naive implementation of either would fit a separate optimal model for each individual (e.g., a collection of person-specific LoRA adapters) and randomly select among them at deployment. We now show something more powerful: this apparently impractical lottery can be compiled into a single static parameter. It can be implemented exactly when its target impact lies in the relative interior of Ψ¯ , and as a limit of single parameters otherwise. Theorem 3. The random dictatorship with weights λ∈ΔNλ∈ _N has target impact ψ¯λ=∑n=1Nλnψn⋆∈Ψ¯ ψ_λ= _n=1^N _n _n ∈ . This impact is attained by a single model parameter, which then reproduces the random dictatorship’s welfare for every agent, if and only if ψ¯λ∈Ψ ψ_λ∈ , in which case the parameter is θλsp=ψ−1(ψ¯λ)θ^sp_λ=ψ^-1( ψ_λ). When ψ¯λ ψ_λ lies on the boundary of Ψ¯ , it is instead the limit of the impacts of a sequence of single parameters. Even though θλspθ^sp_λ is one fixed parameterization, its stochastic responses reproduce, in expectation, the same impact as randomizing over agents’ ideal models. Mathematically, it follows from the invertibility of ψ; conceptually, it reflects that model stochasticity is itself a form of compromise between conflicting individuals (see Figure 3). Recall ψn⋆=argmaxψ∈Ψ¯θn⊤ψ _n = _ψ∈ _n ψ is agent n’s ideal impact. Having established that a random dictatorship is strategyproof and can be compiled into a single model parameter, we now ask how much doing so constrains the achievable welfare. First, note that we can select a set of weights λ to achieve any ψ∈Ψsp=Hull(ψn⋆n=1N)ψ∈ ^sp=Hull(\ _n \_n=1^N). Intuitively, each ψn⋆ _n is the support point of Ψ¯ in the direction of θn _n. If agent preferences are spread over all directions, then their convex hull Ψsp ^sp will approximate Ψ¯ . Welfare will be low when social welfare maximization selects a “compromise” impact vector that is dissimilar to all individual vectors. Proposition 3. Let Ψ¯⊂ℝk ^k be any compact, convex, origin-symmetric set with dim(Ψ¯)≥2 ( )≥ 2, suppose the welfare weights w∈ΔNw∈ _N are not a point mass, and write U⋆(θ)=maxψ∈Ψ¯θ¯w⊤ψU (θ)= _ψ∈ θ_w ψ for the optimal utilitarian welfare. For any weights λ∈ΔNλ∈ _N and any δ>0δ>0, there exists a preference profile θ with U⋆(θ)>0U (θ)>0 at which the random dictatorship μλsp _λ sp attains at most a δ fraction of the optimum, [U(μλsp(θ))]≤δU⋆(θ)E[U( _λ sp(θ))]≤δ\,U (θ). Hence, for non-degenerate welfare weights, the worst-case distortion U⋆(θ)/[U(μλsp(θ))]U (θ)/E[U( _λ sp(θ))] of every random dictatorship is unbounded.55 5 Some restriction on w is necessary: if w=δiw= _i, then U⋆(θ)=maxψ∈Ψ¯θi⊤ψ=θi⊤ψi⋆U (θ)= _ψ∈ _i ψ= _i _i , while origin symmetry gives θi⊤ψ≥−U⋆(θ) _i ψ≥-U (θ) for all ψ∈Ψ¯ψ∈ ; hence every profile satisfies [U(μλsp(θ))]=∑nλnθi⊤ψn⋆≥(2λi−1)U⋆(θ)E[U( _λ^sp(θ))]= _n _n _i _n ≥(2 _i-1)\,U (θ), so for λi>12 _i> 12 the distortion is bounded. Intuitively, the proof shows that this worst case arises when an individual or small group has a very strong preference along some dimension that the overwhelming majority has a very weak preference in the opposite direction. In such cases, a random dictatorship will implement the weak preferences of the majority, which causes outsized harm to the minority. In the applied domains that we study, this structure does not generally arise and the strategyproof mechanism’s welfare sub-optimality is bounded (see Figure 8 in Appendix E). These results also shed new light on the standard practice of RLHF. Anonymous RLHF fits a single parameter θ^RLHF θ^RLHF to the pooled labels, ignoring who produced them. Our unifying result is that its deployed impact is exactly that of a simple stochastic protocol—answer each query by sampling a labeler and following their choice—and therefore depends on the training data only through the population’s per-query label frequencies. Proposition 4. Suppose qtrainq_train and qdepq_dep share the same query marginal, and let θ^RLHF θ^RLHF be any stationary point of the anonymous RLHF objective. Writing μz=n∼qtrain(⋅∣z)[[c∣n,z]] _z=E_n q_train(· z)[E[c n,z]] for the population’s label frequency on query z, ψ(θ^RLHF)=z∼qdep[(μz−12)αz].ψ( θ^RLHF)=E_z q_dep [( _z- 12)\, _z ]. That is, the deployed model is impact-equivalent to the random-labeler protocol that answers each deployment query z by sampling an agent n∼qtrain(⋅∣z)n q_train(· z) and a response c∼p(⋅∣n,z)c p(· n,z). The identity holds because the RLHF first-order condition matches the first moment of σ(αz⊤θ^RLHF)σ( _z θ^RLHF) against αz _z to that of μz _z, and the impact ψ is exactly that moment. In the deterministic limit of labeler accuracy, this turns into a per-query vote count, and the impact into that of a voting-by-issues rule. Corollary 1. Suppose, as in Proposition 4, that qtrainq_train and qdepq_dep share the same query marginal. In the deterministic labeling limit [c∣n,z]→αz⊤θn>0E[c n,z] 1\ _z _n>0\, anonymous RLHF converges to the voting-by-issues mechanism with per-query aggregators fz(sz,⋅)=∑nqtrain(n∣z)sz,nf_z(s_z,·)= _nq_train(n z)\,s_z,n, and is therefore strategyproof and unanimous. When the labeling weights are query-independent, qtrain(n∣z)=λnq_train(n z)= _n, it is the random dictatorship with weights λ. This highlights a connection between RLHF and the strategyproof voting-by-issues family, reaching the random-dictatorship point when the labeling pool is the same across queries. 6 Linear Constraints Can Enforce Externality Bounds and Trade-offs In practice, alignment designers have objectives beyond strictly maximizing utilitarian welfare or ensuring strategyproofness. A designer may want to pursue high total welfare while guaranteeing that no individual or group is harmed beyond some threshold or, more generally, to guarantee normative commitments about how the benefits and costs of alignment are distributed. The impact-space formulation makes this task straightforward. Because welfare is linear in the impact vector, any welfare guarantee expressible as a linear inequality becomes a halfspace constraint on ψ. As we show below, a floor on individual welfare, a bound on the ratio of personal harm to social benefit, or a constraint that no individual imposes externalities above some level can all be expressed in this way. When augmented with such a linear constraint, the designer’s problem becomes: ψwel∈argmaxψ∈Ψ¯θ¯w⊤ψs.t.γn⊤ψ≥cn∀n,ψ^wel∈ _ψ∈ θ_w ψ .t. _n ψ≥ c_n\ ∀\,n, (4) for constraint directions γn∈ℝk _n ^k and thresholds cn∈ℝc_n . Since each constraint is a halfspace and Ψ¯ is convex, closed, and bounded, this is a linear optimization with linear constraints whenever the feasible set is nonempty. Different choices of γn _n and cnc_n encode different commitments. We develop three natural families below—absolute welfare floors, public-spirit constraints, and participation-externality bounds—and show that all three admit closed-form characterizations. Theorem 4. If Ψ¯∩⋂nψ:γn⊤ψ≥cn ∩ _n\ψ: _n ψ≥ c_n\ is nonempty, then an optimum ψwelψ^wel exists. Moreover, there are multipliers ηn≥0 _n≥ 0 such that ψwel∈argmaxψ∈Ψ¯(θ¯w+∑n=1Nηnγn)⊤ψ,ψ^wel∈ _ψ∈ ( θ_w+ _n=1^N _n _n ) ψ, with complementary slackness ηn(γn⊤ψwel−cn)=0 _n( _n ψ^wel-c_n)=0 for all n. Equivalently, only the binding constraints ℬ=n:ηn>0B=\n: _n>0\ tilt the objective, giving effective direction θ¯w+∑n∈ℬηnγn θ_w+ _n _n _n. The optimal utilitarian value is weakly decreasing and concave in the thresholds c. 6.1 Example: Bounding Harm A natural welfare guarantee is to require every agent’s welfare to exceed a floor bnb_n. This is the special case γn=θn _n= _n and cn=bnc_n=b_n. Definition 7 (Harm-bounded alignment). For welfare floors b∈ℝNb ^N, ψbh∈argmaxψ∈Ψ¯θ¯w⊤ψs.t.θn⊤ψ≥bn∀n. _b^h∈ _ψ∈ θ_w ψ .t. _n ψ≥ b_n\ ∀ n. Corollary 2 (of Theorem 4). The harm-bounded optimum satisfies ψbh∈argmaxψ∈Ψ¯(∑n=1N(wn+ηn)θn)⊤ψ, _b^h∈ _ψ∈ ( _n=1^N(w_n+ _n) _n ) ψ, where ηn≥0 _n≥ 0 and ηn(θn⊤ψbh−bn)=0 _n( _n _b^h-b_n)=0 for all n. The corollary has a simple interpretation. Instead of maximizing with fixed weights wnw_n, the planner maximizes with effective weights wn+ηnw_n+ _n. The extra term ηn _n is positive exactly for agents who would fall below their required welfare level without additional protection. Thus the constrained solution can be implemented as an ordinary weighted-welfare maximization problem, but with extra weight placed on the agents that need protection. Useful choices of welfare floor include: (i) bn=0b_n=0, so no agent is worse off than under a random model; (i) bn=θn⊤ψwspb_n= _n _w^sp, so every agent weakly prefers the mechanism to the welfare-weighted strategyproof solution; and (i) bn=−ϵhΨ¯(θn)b_n=-ε h_ ( _n), so agent n can lose at most an ϵε fraction of their maximum achievable welfare.66 6 The same construction applies to groups by replacing θn _n with a group preference θ¯=∑n∈wnθn θ_G= _n w_n _n and bnb_n with a group floor. 6.2 Example: Public Spirit Another example of a welfare constraint is public spirit (Flanigan et al., 2023). This constraint allows people to tolerate personal harm in proportion to the social benefit created, where γ∈[0,1]γ∈[0,1] is an individual’s degree of public spirit.77 7 This is a special case of Theorem 4 with γn=γθ¯w+(1−γ)θn _n=γ θ_w+(1-γ) _n and cn=(γθ¯w+(1−γ)θn)⊤ψ0c_n=(γ θ_w+(1-γ) _n) _0, where ψ0 _0 is a reference impact (e.g., the status quo or the random model). Definition 8 (Public-Spirit Alignment). For a public-spirit parameter γ∈[0,1]γ∈[0,1] and reference impact ψ0 _0: ψγps∈argmaxψ∈Ψ¯θ¯w⊤ψs.t.(γθ¯w+(1−γ)θn)⊤ψ≥(γθ¯w+(1−γ)θn)⊤ψ0∀n. _γ^ps∈ _ψ∈ θ_w ψ s.t. (γ\, θ_w+(1-γ) _n) ψ≥(γ\, θ_w+(1-γ) _n) _0 ∀\,n. The constraint has a direct interpretation, saying that agent n will accept some personal loss only when it is justified by enough social gain: (1−γ)θn⊤(ψ0−ψ)≤γθ¯w⊤(ψ−ψ0).(1-γ)\, _n ( _0-ψ)≤γ\, θ_w (ψ- _0). When γ=0γ=0, this reduces to an individual rationality constraint: no agent can be made worse off relative to ψ0 _0. Corollary 3 (of Theorem 4). The public-spirit optimum satisfies ψγps∈argmaxψ∈Ψ¯((1+γ∑n∈ℬηn)θ¯w+(1−γ)∑n∈ℬηnθn)⊤ψ, _γ^ps∈ _ψ∈ ( (1+γ _n _n ) θ_w+(1-γ) _n _n\, _n ) ψ, (5) where ηn≥0 _n≥ 0 and ηn[(γθ¯w+(1−γ)θn)⊤(ψγps−ψ0)]=0∀n. _n [ (γ θ_w+(1-γ) _n ) ( _γ^ps- _0) ]=0 ∀ n. Figure 2: Community Alignment dataset: distribution of individual welfare Un/U⋆U_n/U across the 2387 annotators under each mechanism, at high choice precision (β=100β=100). Top: utilitarian, RLHF, and strategyproof. Bottom: harm-bounded alignment at three welfare floors b; the faint gray curve repeats the utilitarian distribution for reference, and the dotted red line marks the floor. Tightening the floor (toward 00) binds more agents and compresses the welfare distribution, removing the harmed left tail. Figure 2 shows how each mechanism distributes welfare across individuals at high choice precision. Under the utilitarian, RLHF, and strategyproof mechanisms the distribution is wide and carries mass below zero—agents who are actively harmed. The harm-bounded mechanism removes this left tail: as the welfare floor b tightens toward 00, more agents’ constraints bind and the distribution compresses upward against the floor, at the cost of some reduction in mean welfare. Appendix E.2 traces the same effect as a function of the choice-precision parameter β. 6.3 Example: Bounding Participation Externalities The welfare floor (Corollary 2) protects agents who are harmed—it increases their influence over the deployed model. Another potential concern is limiting the harm that any single agent’s participation imposes on everyone else. We formally define the participation externality in Appendix D.1. Here we need only the leave-one-out utilitarian preference, the welfare-weighted average preference of everyone other than n, θ¯−n=11−wn∑m≠nwmθm, θ_-n= 11-w_n _m≠ nw_m _m, (6) which is well defined whenever wn<1w_n<1. Definition 9 (Externality-bounded alignment). For a reference impact ψ0 _0 and tolerances b∈ℝ≥0Nb _≥ 0^N: ψbext∈argmaxψ∈Ψ¯θ¯w⊤ψs.t.θ¯−n⊤ψ≥θ¯−n⊤ψ0−bn∀n. _b^ext∈ _ψ∈ \; θ_w ψ s.t. θ_-n ψ≥ θ_-n _0-b_n ∀\,n. The constraint says: the aggregate welfare of everyone except n, evaluated at the deployed impact, must not fall more than (1−wn)bn(1-w_n)\,b_n below its value at the reference (recall that θ¯−n θ_-n normalizes the leave-one-out weights by 1/(1−wn)1/(1-w_n), which requires wn<1w_n<1 for all n). This bounds the cost that accommodating n’s preferences imposes on the rest of the population. When bn=0b_n=0, the mechanism cannot reduce the rest of the population’s aggregate welfare by including n; as bn→∞b_n→∞, the constraint becomes vacuous. Corollary 4 (of Theorem 4). The externality-bounded optimum satisfies ψbext∈argmaxψ∈Ψ¯((1+∑n∈ℬηn1−wn)θ¯w−∑n∈ℬwnηn1−wnθn)⊤ψ, _b^ext∈ _ψ∈ \; ( (1+ _n _n1-w_n ) θ_w\;-\; _n w_n\, _n1-w_n\, _n ) ψ, (7) where ηn≥0 _n≥ 0 and ηn(θ¯−n⊤ψbext−θ¯−n⊤ψ0+bn)=0 _n( θ_-n _b^ext- θ_-n _0+b_n)=0 for all n. The effective direction has a striking structure that is the opposite of the welfare floor. In harm-bounded alignment (Corollary 2), binding agents receive additional weight: wn→wn+ηnw_n→ w_n+ _n. Here, binding agents have their weight reduced: their preferences are subtracted from the effective direction, and the social welfare direction θ¯w θ_w is amplified. Intuitively, the welfare floor asks “who is being harmed?” and gives them more voice; the externality bound asks “who is causing harm?” and attenuates their influence. With equal weights wn=1/Nw_n=1/N and large N, the effective direction simplifies to approximately (1+∑n∈ℬηn)θ¯w−1N∑n∈ℬηnθn(1+ _n _n)\, θ_w- 1N _n _n\, _n: the dominant effect is an amplification of the utilitarian direction, with a small per-agent correction of order 1/N1/N. A bounded harm mechanism allows us to regulate individuals’ participation externalities (Figure 4 in Appendix E.3). The bright rows in the bounded harm panel are the agents for whom the constraint was binding—agents whose welfare would otherwise fall below the floor. When such an agent is removed from the dataset, their binding constraint disappears and everyone else benefits from returning closer to the utilitarian regime; hence, their inclusion imposes a drastic negative externality on every other member of the population. 7 Empirical Illustrations Setup. We illustrate the framework on real human pairwise-choice data from four domains: kidney allocation (Keswani et al., 2024), charitable food distribution (Lee et al., 2019), trolley problems from Moral Machine (Awad et al., 2018), and preferences over LLM responses from the Community Alignment dataset (Zhang et al., 2025). In each domain, options are described by interpretable features (k ranging from 55 to 2020) and we fit each participant’s linear preference θn _n with a Bradley–Terry–Luce model on their own choices; a precision parameter β scales all preferences, with choices becoming deterministic as β→∞β→∞. For Community Alignment, responses are first embedded by a frozen sentence encoder and projected onto a K=15K=15-dimensional interpretable basis using the method of Wojtowicz et al. (2026), yielding N=2387N=2387 annotators. Taking the observed queries as the deployment distribution, the impact zonotope Ψ¯ —and with it every mechanism in this paper—can be computed exactly. Dataset details and preprocessing are given in Appendix E. Mechanisms and measures. We compare the deployed impacts of the utilitarian optimum (Proposition 1), anonymous RLHF (Proposition 4), the uniform-weight random dictatorship compiled into a single model parameter (Theorem 3), and harm-bounded alignment at several welfare floors (Corollary 2). For each mechanism we compute every agent’s welfare UnU_n relative to the utilitarian optimum U⋆U , as well as the person-on-person participation externalities Γn,m _n,m (Appendix D.1). Findings. On Community Alignment, welfare varies widely across agents under the utilitarian, RLHF, and strategyproof mechanisms, and under all three some agents are actively harmed (Figure 2). Under harm-bounded alignment no agent’s welfare falls below the chosen floor, at a cost in mean welfare that grows as the floor tightens. The externality measures show that this cost comes from a small number of agents whose harm constraint binds, and that including these agents is costly to everyone else. Across the four datasets, agents disagree about the direction of their preferences in Moral Machine and Community Alignment, but mostly about the magnitude of their preferences in the kidney and food-rescue domains (Figure 7 in Appendix E). As a result, the strategyproof mechanism loses a fraction of the optimal welfare in these datasets (Figure 8), because the adversarial structure behind Proposition 3 does not arise in them. At high choice precision, welfare also varies least across agents under this mechanism (Figures 9 and 10). Under the utilitarian, RLHF, and strategyproof mechanisms, welfare varies more across agents as the choice precision β grows and individual choices become more deterministic (Figure 3). 8 Limitations and Discussion Limitations. Our framework builds on, and therefore relies on, the assumption that preferences are linear in a model’s feature space. For LLM alignment, making such a linear representation interpretable is nontrival, which we demonstrate on the Community Alignment dataset by importing the feature extraction pipeline of (Wojtowicz et al., 2026). Additionally, some assumptions in our results about query distributions may not hold in practice, and the strategyproofness results rely on the central planner broadcasting individual probabilities of being selected ahead of time. The “optimal” strategyproof solution may require solving a fixed-point problem where the optimal probabilities are learned over time, a technical problem we propose for future work. Discussion. We have shown that linear social choice provides the right setting in which to reformulate alignment as a linear optimization problem. By analyzing welfare not as a second-order consequence of model parameterization but as a first-order object, we are able to develop an impact-space framework that directly informs how we design strategyproof mechanisms as well as mechanisms that implement a variety of welfare-specific desiderata. The information granularity resulting from the impact space analysis leads to a rich set of future questions: can we use this framework to constrain the welfare impacts of one demographic group on another? Can every alignment mechanism (traditional RLHF, DPO, Max-Min, Nash) be characterized by the patterns of externalities that it creates over the population of individuals, and do we have the power to regulate these in a unified way? Prioritizing welfare as the primary object of analysis may also lead to other elegant structures in the analysis of alignment protocols—we believe this viewpoint will only become more important as we begin to align agents and other systems capable of making decisions on behalf of humans, such as negotiation or high-stakes resource allocation. References Awad et al. (2018) E. Awad, S. Dsouza, R. Kim, J. Schulz, J. Henrich, A. Shariff, J. Bonnefon, and I. Rahwan The moral machine experiment. Nature 563 (7729), p. 59–64. Cited by: §E.1.3, §7. Bai et al. (2022) Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: §1. Barberà et al. (1993) S. Barberà, F. Gul, and E. Stacchetti Generalized median voter schemes and committees. Journal of Economic Theory 61 (2), p. 262–289. Cited by: §5.2. Barbera and Peleg (1990) S. Barbera and B. Peleg Strategy-proof voting schemes with continuous preferences. Social choice and welfare 7 (1), p. 31–38. Cited by: Appendix B, §1, §5.1, §5. Barberà (2011) S. Barberà Strategyproof social choice. In Handbook of Social Choice and Welfare, K. J. Arrow, A. Sen, and K. Suzumura (Eds.), Vol. 2, p. 731–831. Cited by: §5.1. Boerstler et al. (2024) K. Boerstler, V. Keswani, L. Chan, J. S. Borg, V. Conitzer, H. Heidari, and W. Sinnott-Armstrong On the stability of moral preferences: a problem with computational elicitation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7, p. 156–167. Cited by: §E.1.2. Chakraborty et al. (2024) S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang MaxMin-RLHF: alignment with diverse human preferences. arXiv preprint arXiv:2402.08925. Cited by: §1, §2, §4. Chidambaram et al. (2024) K. Chidambaram, K. V. Seetharaman, and V. Syrgkanis Direct preference optimization with unobserved preference heterogeneity. arXiv preprint arXiv:2405.15065. Cited by: §1, §2. Conitzer et al. (2024) V. Conitzer, R. Freedman, J. Heitzig, W. H. Holliday, B. M. Jacobs, N. Lambert, M. Mossé, E. Pacuit, S. Russell, H. Schoelkopf, et al. Social choice should guide AI alignment in dealing with diverse human feedback. arXiv preprint arXiv:2404.10271. Cited by: §1. Dai and Fleisig (2024) J. Dai and E. Fleisig Mapping social choice theory to RLHF. arXiv preprint arXiv:2404.13038. External Links: Document, Link Cited by: §1. Dutta et al. (2002) B. Dutta, H. Peters, and A. Sen Strategy-proof probabilistic mechanisms in economies with pure public goods. Journal of Economic Theory 106 (2), p. 392–416. Cited by: §5.3. Flanigan et al. (2023) B. Flanigan, A. D. Procaccia, and S. Wang Distortion under public-spirited voting. arXiv preprint arXiv:2305.11736. Cited by: §6.2. Ge et al. (2024) L. Ge, D. Halpern, E. Micha, A. D. Procaccia, I. Shapira, Y. Vorobeychik, and J. Wu Axioms for AI alignment from human feedback. Advances in Neural Information Processing Systems 37, p. 80439–80465. Cited by: §1, §1, §2, §4. Gibbard (1977) A. Gibbard Manipulation of schemes that mix voting with chance. Econometrica: Journal of the Econometric Society, p. 665–681. Cited by: §5.3. Gölz et al. (2025) P. Gölz, N. Haghtalab, and K. Yang Distortion of AI alignment: does preference optimization optimize for preferences?. arXiv preprint arXiv:2505.23749. Cited by: §1, §2, §4. Halpern et al. (2025) D. Halpern, E. Micha, A. D. Procaccia, and I. Shapira Pairwise calibrated rewards for pluralistic alignment. arXiv preprint arXiv:2506.06298. Cited by: §1. Hammoudeh and Lowd (2024) Z. Hammoudeh and D. Lowd Training data influence analysis and estimation: a survey. Machine Learning 113 (5), p. 2351–2403. Cited by: §C.2. Harsanyi (1955) J. C. Harsanyi Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility. Journal of political economy 63 (4), p. 309–321. Cited by: §3.1. Keswani et al. (2024) V. Keswani, V. Conitzer, H. Heidari, J. S. Borg, and W. Sinnott-Armstrong On the pros and cons of active learning for moral preference elicitation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7, p. 711–723. Cited by: §E.1.2, §7. Kleine Buening et al. (2026) T. Kleine Buening, J. Gan, D. Mandal, and M. Kwiatkowska Strategyproof reinforcement learning from human feedback. Advances in Neural Information Processing Systems 38, p. 101431–101464. Cited by: §C.1, §2, §5. Lee et al. (2019) M. K. Lee, D. Kusbit, A. Kahng, J. T. Kim, X. Yuan, A. Chan, D. See, R. Noothigattu, S. Lee, A. Psomas, et al. WeBuildAI: participatory framework for algorithmic governance. Proceedings of the ACM on human-computer interaction 3 (CSCW), p. 1–35. Cited by: §E.1.1, §7. Li et al. (2024) X. Li, R. Zhou, Z. C. Lipton, and L. Leqi Personalized language modeling from personalized human feedback. arXiv preprint arXiv:2402.05133. External Links: Document, Link Cited by: §1. Park et al. (2024) C. Park, M. Liu, D. Kong, K. Zhang, and A. Ozdaglar RLHF from heterogeneous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.00254. Cited by: §1, §2, §4. Poddar et al. (2024) S. Poddar, Y. Wan, H. Ivison, A. Gupta, and N. Jaques Personalizing reinforcement learning from human feedback with variational preference learning. In Advances in Neural Information Processing Systems, Vol. 37. External Links: Document, Link Cited by: §1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §3.1. Rockafellar (1970) R. T. Rockafellar Convex analysis. Princeton University Press. Cited by: item 2, item 2. Shirali et al. (2025) A. Shirali, A. Nasr-Esfahany, A. Alomar, P. Mirtaheri, R. Abebe, and A. Procaccia Direct alignment with heterogeneous preferences. arXiv preprint arXiv:2502.16320. Cited by: §1, §1, §2, §4. Siththaranjan et al. (2023) A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell Distributional preference learning: understanding and accounting for hidden context in RLHF. arXiv preprint arXiv:2312.08358. Cited by: §1, §2. Sun et al. (2024) H. Sun, Y. Chen, S. Wang, X. Chu, W. Chen, and X. Deng Mechanism design for LLM fine-tuning with multiple reward models. arXiv preprint arXiv:2405.16276. Cited by: §2. Wojtowicz et al. (2026) Z. Wojtowicz, A. Nayak, and J. Andreas From weights to words: expressing and editing preference model inferences in natural language. arXiv preprint arXiv:2607.16232. Cited by: §E.1.4, §7, §8. Zhang et al. (2025) L. H. Zhang, S. Milli, K. Jusko, J. Smith, B. Amos, W. Bouaziz, M. Revel, J. Kussman, Y. Sheynin, L. Titus, et al. Cultivating pluralism in algorithmic monoculture: the community alignment dataset. arXiv preprint arXiv:2507.09650. Cited by: §E.1.4, §7. Appendix A Proofs for Sections 3 and 4 Throughout the proofs, let Ψ=ψ(θ):θ∈ℝk =\ψ(θ):θ ^k\ and Ψ¯=cl(Ψ) =cl( ). We optimize over Ψ¯ , but ψ−1ψ^-1 is used only on Ψ , where the impact map is invertible on the deployment subspace. For any non-empty closed convex set A, the support function of A is hA(x)=supa∈Ax⊤a.h_A(x)= _a∈ Ax a. (8) Proof of Theorem 1. 1. Write the impact map in terms of its per-query coordinates ψ(θ)=z∼qdep[tz(θ)αz],tz(θ)=σ(αz⊤θ)−12∈(−12,12),ψ(θ)=E_z q_dep [t_z(θ)\, _z ], t_z(θ)=σ( _z θ)- 12∈(- 12, 12), (9) and let =z∼qdep[tzαz]:tz∈[−12,12]Z=\E_z q_dep[t_z _z]:t_z∈[- 12, 12]\ be the origin-symmetric zonotope generated by 12qdep(z)αz\ 12q_dep(z) _z\, i.e. the linear image M([−12,12]Z)M([- 12, 12]^Z) of the cube (one coordinate per query z∈Zz∈ Z) under M(t)=z∼qdep[tzαz]M(t)=E_z q_dep[t_z _z]. Since each αz∈Sdep _z∈ S_dep, both Ψ and Z lie in SdepS_dep. Because a linear map carries the relative interior of a convex set onto the relative interior of its image, M maps the open cube onto relintrelintZ, the interior of Z relative to SdepS_dep. Note that this is nonempty, since the generators of Z span SdepS_dep. Moreover, because t(θ)t(θ) lies in the open cube, Ψ⊆relint . A zonotope is convex and origin-symmetric. We have ψ(0)=0ψ(0)=0 since σ(0)=12σ(0)= 12, and ψ(−θ)=−ψ(θ)ψ(-θ)=-ψ(θ) since σ(−x)−12=−(σ(x)−12)σ(-x)- 12=-(σ(x)- 12). 2. The map ψ is the gradient of the convex log-partition potential Φ(θ)=z∼qdep[log(1+eαz⊤θ)−12αz⊤θ],∇Φ=ψ,∇2Φ(θ)=z∼qdep[v(αz⊤θ)αzαz⊤], (θ)=E_z q_dep [ (1+e _z θ)- 12 _z θ ], ∇ =ψ, ∇^2 (θ)=E_z q_dep [v( _z θ)\, _z _z ], (10) with v=σ′>0v=σ >0. The Hessian is positive definite on SdepS_dep, i.e., if u∈Sdepu∈ S_dep satisfies u⊤∇2Φ(θ)u=z∼qdep[v(αz⊤θ)(αz⊤u)2]=0u ∇^2 (θ)\,u=E_z q_dep[v( _z θ)( _z u)^2]=0, then αz⊤u=0 _z u=0 for every z∈supp(qdep)z (q_dep), so u⟂Sdepu S_dep, which together with u∈Sdepu∈ S_dep forces u=0u=0. Hence Φ restricted to SdepS_dep is finite, C∞C^∞, and strictly convex. Given that it is finite and differentiable everywhere, it is essentially smooth, and is therefore a convex function of Legendre type (Rockafellar, 1970, §26). We restrict attention to SdepS_dep throughout, as on the ambient space Φ has flat directions and Z has empty interior whenever Sdep≠ℝkS_dep ^k, so all interiors below are relative to SdepS_dep. The recession function of Φ is limν→∞Φ(νu)ν=z∼qdep[(αz⊤u)+−12αz⊤u]=12z∼qdep[|αz⊤u|]=h(u), _ν→∞ (ν u)ν=E_z q_dep [( _z u)_+- 12\, _z u ]= 12\,E_z q_dep [\,| _z u|\, ]=h_Z(u), (11) the support function of Z. By Rockafellar (1970, Theorem 13.3), the support function of domΦ∗dom ^* is the recession function of Φ ; hence cl(domΦ∗)=cl(dom ^*)=Z, and so relint(domΦ∗)=relintrelint(dom ^*)=relintZ. By Rockafellar (1970, Theorem 26.5), the gradient map of a Legendre-type function is a bijection from the interior of its domain onto the interior of the domain of its convex conjugate, with continuous inverse ∇Φ∗∇ ^*; here these interiors are SdepS_dep and relintrelintZ, so ψ=∇Φψ=∇ is a bijection from SdepS_dep onto relintrelintZ, and, by the inverse function theorem, a C∞C^∞ diffeomorphism. Therefore Ψ=relint =relintZ, so Ψ¯=cl(relint)= =cl(relintZ)=Z is the stated zonotope. 3. From |tz(θ)|≤12|t_z(θ)|≤ 12, ‖ψ(θ)‖≤[‖αz‖|tz(θ)|]≤12[‖αz‖]\|ψ(θ)\| [\| _z\|\,|t_z(θ)|]≤ 12E[\| _z\|]. 4. Differentiating under the expectation, ∂νψ(νθ)=[(αz⊤θ)v(ναz⊤θ)αz] ∂νψ(νθ)=E[( _z θ)v(ν _z θ) _z], so θ⊤∂νψ(νθ)=z∼qdep[(αz⊤θ)2v(ναz⊤θ)]≥0.θ ∂νψ(νθ)=E_z q_dep [( _z θ)^2v(ν _z θ) ]≥ 0. (12) 5. By dominated convergence, limν→∞tz(νθ)→12sign(αz⊤θ) _ν→∞t_z(νθ)→ 12sign( _z θ), so limν→∞ψ(νθ)=12z∼qdep[αzsign(αz⊤θ)], _ν→∞ψ(νθ)= 12\,E_z q_dep [ _zsign( _z θ) ], (13) which attains maxψ∈Ψ¯θ⊤ψ=12z∼qdep|αz⊤θ| _ψ∈ θ ψ= 12\,E_z q_dep| _z θ|. For generic θ (i.e., αz⊤θ≠0 _z θ≠ 0 for all z∈supp(qdep)z (q_dep)), it is the unique maximizer: any maximizer may be written z∼qdep[tzαz]E_z q_dep[t_z\, _z] with tz∈[−12,12]t_z∈[- 12, 12], and θ⊤ψ=z∼qdep[tz(αz⊤θ)]θ ψ=E_z q_dep[t_z\,( _z θ)] attains this value only if tz=12sign(αz⊤θ)t_z= 12sign( _z θ) for every z∈supp(qdep)z (q_dep), which determines the point. As the singleton exposed face of the polytope Ψ¯ in direction θ, it is a vertex; when θ=θnθ= _n, it is agent n’s ideal impact ψn⋆ _n . ∎ Proof of Proposition 1. By Theorem 1.5, the limit exists and equals p=12z∼qdep[αzsign(αz⊤θ¯w)]p= 12\,E_z q_dep[ _zsign( _z θ_w)], and p∈Ψ¯p∈ because Ψ¯ is closed and p is a limit of points of Ψ . Moreover, θ¯w⊤p=12z∼qdep[|αz⊤θ¯w|]=hΨ¯(θ¯w), θ_w p= 12\,E_z q_dep[\,| _z θ_w|\,]=h_ ( θ_w), the support function of the zonotope computed in the proof of Theorem 1, so p attains the maximum of θ¯w⊤ψ θ_w ψ over Ψ¯ , i.e. p∈argmaxψ∈Ψ¯θ¯w⊤ψp∈ _ψ∈ θ_w ψ. For uniqueness under genericity: any maximizer can be written ψ=z∼qdep[tzαz]ψ=E_z q_dep[t_z _z] with tz∈[−12,12]t_z∈[- 12, 12], and θ¯w⊤ψ=[tz(αz⊤θ¯w)] θ_w ψ=E[t_z( _z θ_w)] attains 12|αz⊤θ¯w| 12E| _z θ_w| only if tz=12sign(αz⊤θ¯w)t_z= 12sign( _z θ_w) for every z with αz⊤θ¯w≠0 _z θ_w≠ 0; when αz⊤θ¯w≠0 _z θ_w≠ 0 for all z∈supp(qdep)z (q_dep), this pins down every representation, so the maximizer equals p. ∎ Appendix B Proofs for Section 5 Proof of Lemma 1. By linearity of θn⊤(⋅) _n (·), [θn⊤μ(θ)]=θn⊤ψ∼μ(θ)[ψ]=θn⊤μ¯(θ)E[ _n μ(θ)]= _n E_ψ μ(θ)[ψ]= _n μ(θ). Strategyproofness is an inequality between such expectations and so depends only on μ¯ μ. For unanimity, on the generic domain where it is defined, the point ψ⋆(θ¯)=argmaxψ∈Ψ¯θ¯⊤ψ ( θ)= _ψ∈ θ ψ is an exposed vertex—and hence an extreme point—of Ψ¯ (Theorem 1). A distribution whose mean is an extreme point is the point mass, so μ(θ,…,θ)=δ(ψ⋆(θ¯))μ(θ,…,θ)=δ(ψ ( θ)) iff μ¯(θ,…,θ)=ψ⋆(θ¯) μ(θ,…,θ)=ψ ( θ). ∎ Proof of Lemma 2. By Lemma 1 we restrict attention to μ¯ μ. The argument follows that of Barbera and Peleg (1990). ⇐ Suppose such menus MnM_n exist. Fix n, θ−n _-n, and true type θn _n. For any misreport θn′ _n , the mechanism applied at the profile (θn′,θ−n)( _n , _-n) gives μ¯(θn′,θ−n)∈Mn(θ−n) μ( _n , _-n)∈ M_n( _-n), and applied at θ it says μ¯(θ) μ(θ) maximizes θn⊤ψ _n ψ over Mn(θ−n)M_n( _-n). Hence θn⊤μ¯(θ)≥θn⊤μ¯(θn′,θ−n) _n μ(θ)≥ _n μ( _n , _-n). Thus, it is strategyproof. ⇒ Suppose μ is strategyproof and take Mn(θ−n)=On(θ−n)=μ¯(θn′,θ−n):θn′∈ℝkM_n( _-n)=O_n( _-n)=\ μ( _n , _-n): _n ^k\, which depends only on θ−n _-n. Then μ¯(θ)∈On(θ−n) μ(θ)∈ O_n( _-n) trivially, and strategyproofness says θn⊤μ¯(θ)≥θn⊤ψ _n μ(θ)≥ _n ψ for every ψ∈On(θ−n)ψ∈ O_n( _-n), i.e. μ¯(θ)∈argmaxψ∈On(θ−n)θn⊤ψ μ(θ)∈ _ψ∈ O_n( _-n) _n ψ. ∎ Proof of Theorem 2. Fix agent n and the others’ reports. By Definition 5, agent n’s expected utility is θn⊤μ¯(θ)=12z∼qdep[fz(sz)(αz⊤θn)], _n μ(θ)= 12\,E_z q_dep [f_z(s_z)\,( _z _n) ], (14) a sum over queries in which n controls only its own votes sz,n\s_z,n\. For each z, this is increasing in fzf_z when αz⊤θn>0 _z _n>0 and decreasing when αz⊤θn<0 _z _n<0. Since fzf_z is nondecreasing in sz,ns_z,n, it is maximized by the truthful vote sz,n=sign(αz⊤θn)s_z,n=sign( _z _n). The truthful report realizes all these votes simultaneously, so it maximizes every term and hence the sum. For unanimity, note that identical reports give sz,n=sz=sign(αz⊤θ¯)s_z,n=s_z=sign( _z θ), so fz=szf_z=s_z and μ¯=12z[szαz]=ψ⋆(θ¯) μ= 12E_z[s_z _z]=ψ ( θ), which is unanimous by Lemma 1. ∎ Proof of Proposition 2. The aggregator fz(sz)=∑nλnsz,nf_z(s_z)= _n _ns_z,n is non-decreasing in each vote (given λn≥0 _n≥ 0) with fz(±)=±1f_z( 1)=± 1, so it defines a voting-by-issues mechanism, which is strategyproof and unanimous by Theorem 2. Its expected impact is 12z[(∑nλnsz,n)αz]=∑nλn⋅12z[sz,nαz]=∑nλnψn⋆, 12E_z [ ( _n _ns_z,n ) _z ]= _n _n· 12E_z[s_z,n _z]= _n _n _n , the random dictatorship with weights λ. ∎ Proof of Theorem 3. Each ideal impact ψn⋆ _n lies in Ψ¯ , which is convex (Theorem 1), so the target impact ψ¯λ=∑nλnψn⋆∈Ψ¯ ψ_λ= _n _n _n ∈ . By Theorem 1(1), the impacts attainable by a single parameter are exactly Ψ=ψ(θ):θ∈ℝk =\ψ(θ):θ ^k\, the relative interior of Ψ¯ , and by Theorem 1(2), ψ is a bijection from SdepS_dep onto Ψ . Hence ψ¯λ ψ_λ is attained by a single parameter if and only if ψ¯λ∈Ψ ψ_λ∈ , in which case the unique such parameter in SdepS_dep is θλsp=ψ−1(ψ¯λ)θ^sp_λ=ψ^-1( ψ_λ), with impact exactly ψ¯λ ψ_λ. This is the mean impact of the random dictatorship (Proposition 2). Since welfare depends on a mechanism only through its expected impact (Lemma 1), this single parameter reproduces the random dictatorship’s welfare for every agent. When ψ¯λ ψ_λ instead lies on the boundary of Ψ¯ , it remains a limit of single-parameter impacts: since 0=ψ(0)∈Ψ=relintΨ¯0=ψ(0)∈ =relint and ψ¯λ∈Ψ¯ ψ_λ∈ , the half-open segment (1−ε)ψ¯λ:ε∈(0,1] \(1- ) ψ_λ: ∈(0,1] \ lies in Ψ . Thus θλ,εsp:=ψ−1((1−ε)ψ¯λ)θ^sp_λ, :=ψ^-1 ((1- ) ψ_λ ) is well-defined for every ε∈(0,1) ∈(0,1), and ψ(θλ,εsp)=(1−ε)ψ¯λ→ψ¯λψ (θ^sp_λ, )=(1- ) ψ_λ→ ψ_λ as ε→0 → 0. ∎ Proof of Proposition 3. Choice of directions. Since Ψ¯ is convex and origin-symmetric, its affine hull is the linear span S=span(Ψ¯)S=span( ) and 0∈relintΨ¯0 ; hence hΨ¯(d)>0h_ (d)>0 for every nonzero d∈Sd∈ S. The support function hΨ¯h_ is finite and convex on S, hence differentiable at Lebesgue-almost every direction in S, and since Ψ¯⊆S S it is differentiable at d∈Sd∈ S exactly when argmaxψ∈Ψ¯d⊤ψ _ψ∈ d ψ is a singleton. Fix such a d1∈Sd_1∈ S with d1≠0d_1≠ 0 and let v denote the unique maximizer in direction d1d_1; then d1⊤v=hΨ¯(d1)>0d_1 v=h_ (d_1)>0 and v≠0v≠ 0, and by symmetry −v-v is the unique maximizer in direction −d1-d_1. Because dim(Ψ¯)≥2 ( )≥ 2 and v∈Sv∈ S, we may choose d2∈Sd_2∈ S with d2⟂vd_2 v and d2≠0d_2≠ 0, and then hΨ¯(d2)>0h_ (d_2)>0. Profile. Since w is not a point mass, fix i with 0<wi<10<w_i<1. For ε>0 >0, set θn=d1+εd2n=i−wi1−wid1+εd2n≠i. _n= casesd_1+ \,d_2&n=i\\[4.0pt] - w_i1-w_i\,d_1+ \,d_2&n≠ i. cases (15) (The coefficient for n≠in≠ i is common to all agents, so zero welfare weights are permitted.) The d1d_1 components cancel in the welfare-weighted average: θ¯w=∑nwnθn=wid1−wi1−wi(1−wi)d1+εd2=εd2, θ_w= _nw_n\, _n=w_i\,d_1- w_i1-w_i\,(1-w_i)\,d_1+ \,d_2= \,d_2, (16) so the first-best welfare is U⋆=εhΨ¯(d2)>0U = \,h_ (d_2)>0. Welfare of the random dictatorship. As ε↓0 0, the normalized direction of θi _i converges to that of d1d_1 and the normalized direction of every θn _n, n≠in≠ i, converges to that of −d1-d_1. The correspondence d↦argmaxψ∈Ψ¯d⊤ψd _ψ∈ d ψ is upper hemicontinuous (Berge’s maximum theorem, using compactness of Ψ¯ ) and singleton-valued at ±d1± d_1, so every selection of maximizers satisfies ψi⋆→v _i → v and ψn⋆→−v _n →-v for n≠in≠ i; no continuity or strict-convexity assumption on Ψ¯ is needed, and the argument covers the zonotopes of Theorem 1. Therefore ψsp=∑nλnψn⋆⟶λiv−(1−λi)v=(2λi−1)v(ε↓0),ψ^sp= _n _n\, _n \; \; _i\,v-(1- _i)\,v=(2 _i-1)\,v ( 0), (17) and, since d2⊤v=0d_2 v=0 by construction, Usp=θ¯w⊤ψsp=εd2⊤ψsp=ε⋅o(1).U^sp= θ_w ψ^sp= \,d_2 ψ^sp= · o(1). (18) Combining with U⋆=εhΨ¯(d2)>0U = \,h_ (d_2)>0, the welfare ratio is Usp/U⋆=o(1)U^sp/U =o(1) as ε↓0 0, which is below δ once ε is small enough. The construction places no restriction on λ. ∎ Appendix C Anonymous RLHF Through the Impact Identity This section collects our results on anonymous RLHF. We first prove the identity and derive its mechanism consequences: the deterministic labeling limit (Corollary 1) and manipulability at finite labeling precision (Corollary 5). We then develop the marginal welfare effect of reweighting training sources and the resulting contrast between RLHF stationarity and utilitarian efficiency (Theorem 5). Finally, we characterize the identification limits imposed by the identity’s information bottleneck: which impacts reweighting can and cannot recover (Section C.5). C.1 The Identity and Its Mechanism Consequences Proof of Proposition 4. The anonymous RLHF objective is the cross-entropy loss −(n,z)∼qtrainc|n,z[clogσ(αz⊤θ)+(1−c)log(1−σ(αz⊤θ))],-\,E_(n,z) q_train\,E_c n,z\! [c σ( _z θ)+(1-c) (1-σ( _z θ) ) ], (19) whose stationarity condition is z∼qtrain[(μz−σ(αz⊤θ^RLHF))αz]=0,μz=n∼qtrain(⋅∣z)[[c∣n,z]].E_z q_train\! [ ( _z-σ( _z θ^RLHF) )\, _z ]=0, _z=E_n q_train(· z) [E[c n,z] ]. (20) Since qtrainq_train and qdepq_dep share the same query marginal, the same moment identity [σ(αz⊤θ^RLHF)αz]=[μzαz]E[σ( _z θ^RLHF) _z]=E[ _z _z] holds under qdepq_dep. Subtracting 12z∼qdep[αz] 12E_z q_dep[ _z] from both moments, ψ(θ^RLHF)=z∼qdep[(σ(αz⊤θ^RLHF)−12)αz]=z∼qdep[(μz−12)αz].ψ( θ^RLHF)=E_z q_dep\! [ (σ( _z θ^RLHF)- 12 ) _z ]=E_z q_dep\! [ ( _z- 12 ) _z ]. (21) The argument uses only first-order stationarity and the shared query marginal. For the equivalence claim, the random-labeler protocol answers query z with choice probability μz _z, so it delivers welfare θn⊤z∼qdep[(μz−12)αz] _n E_z q_dep[( _z- 12) _z] to each agent n—by the display above, the same as the deployed model. Note, finally, that the right-hand side of the identity is a function of μz\ _z\ alone, so the deployed impact depends on the training data only through the per-query label frequencies. ∎ Proof of Corollary 1. In the deterministic labeling limit [c∣n,z]→αz⊤θn>0E[c n,z] 1\ _z _n>0\, so μz→∑nqtrain(n∣z) 1αz⊤θn>0 _z→ _nq_train(n z)\,1\ _z _n>0\. Using x>0−12=12sign(x)1\x>0\- 12= 12sign(x) and ∑nqtrain(n∣z)=1 _nq_train(n z)=1, μz−12=12∑nqtrain(n∣z)sign(αz⊤θn)=12fz(sz,⋅),fz(sz,⋅)=∑nqtrain(n∣z)sz,n. _z- 12= 12 _nq_train(n z)\,sign( _z _n)= 12\,f_z(s_z,·), f_z(s_z,·)= _nq_train(n z)\,s_z,n. (22) By Proposition 4 and dominated convergence (|(μz−12)αz|≤12‖αz‖|( _z- 12) _z|≤ 12\| _z\|), the deployed impact converges to 12z∼qdep[fz(sz,⋅)αz] 12E_z q_dep[f_z(s_z,·) _z], the impact of the voting-by-issues mechanism of Definition 5 with aggregator fzf_z. Each fzf_z is a convex combination of the votes, hence nondecreasing in every argument with fz(±)=±1f_z( 1)=± 1, so the mechanism is strategyproof and unanimous by Theorem 2. When qtrain(n∣z)=λnq_train(n z)= _n for all z, fz(sz,⋅)=∑nλnsz,nf_z(s_z,·)= _n _ns_z,n and the impact is ∑nλnψn⋆ _n _n _n , the random dictatorship with weights λ (Proposition 2). ∎ At finite labeling precision, the identity instead exposes a manipulation channel: the frequencies μz\ _z\, and hence the deployed impact, respond continuously to the intensity of an agent’s reported preferences. Corollary 5 (Finite-temperature RLHF is manipulable). Suppose, as in Corollary 1, that qtrainq_train and qdepq_dep share the same query marginal, that the labeling weights are query-independent, qtrain(n∣z)=λnq_train(n z)= _n, and that labels follow the BTL model at the reported preferences, [c∣n,z]=σ(αz⊤θ^n)E[c n,z]=σ( _z θ_n). Then the anonymous RLHF impact is the λ-average of the reported individual impacts, ψ(θ^RLHF)=∑n=1Nλnψ(θ^n),ψ ( θ^RLHF )= _n=1^N _n\,ψ( θ_n), and the induced mechanism is not strategyproof: for every agent n with λn>0 _n>0 and generic true preference θn _n, the utility of reporting νθnν _n is strictly increasing in ν, so any ν>1ν>1 strictly improves on truthful reporting and no optimal report exists. Proof. Since every μz=∑nλnσ(αz⊤θ^n) _z= _n _nσ( _z θ_n) lies in (0,1)(0,1), the cross-entropy objective is strictly convex and coercive on Strain=SdepS_train=S_dep (the supports of the query marginals coincide), so a unique stationary point exists there. By Proposition 4 and ∑nλn=1 _n _n=1, ψ(θ^RLHF)=z∼qdep[(μz−12)αz]=∑n=1Nλnz∼qdep[(σ(αz⊤θ^n)−12)αz]=∑n=1Nλnψ(θ^n).ψ ( θ^RLHF )=E_z q_dep [( _z- 12) _z ]= _n=1^N _n\,E_z q_dep [ (σ( _z θ_n)- 12 ) _z ]= _n=1^N _n\,ψ( θ_n). Fix agent n and the others’ reports. By the identity above, agent n’s deployment welfare from reporting θ′θ is λnθn⊤ψ(θ′) _n\, _n ψ(θ ) plus a term that does not depend on its report, and by Theorem 1(4), dνθn⊤ψ(νθn)=z∼qdep[(αz⊤θn)2v(ναz⊤θn)]>0 ddν\, _n ψ(ν _n)=E_z q_dep [( _z _n)^2\,v(ν\, _z _n) ]>0 for generic θn _n. Hence reporting νθnν _n with ν>1ν>1 strictly improves on the truthful report. The supremum hΨ¯(θn)h_ ( _n) of θn⊤ψ(θ′) _n ψ(θ ) over reports is approached along the ray νθnν _n as ν→∞ν→∞ but is not attained by any report: for generic θn _n the unique maximizer ψn⋆ _n is an exposed vertex of Ψ¯ and hence lies outside Ψ=relintΨ¯ =relint . ∎ Kleine Buening et al. (2026) show that RLHF with stochastic labelers is not strategyproof: a labeler can manipulate its choice probabilities to steer the learned policy. Corollary 1 is complementary: in the deterministic limit, where labelers can only manipulate their choice directions (not intensities), RLHF converges to a strategyproof voting-by-issues rule. The additional manipulation channel available at finite temperature is exactly the nonlinearity that makes the finite-temperature mechanism deviate from this limit. C.2 The Differential Companion: Marginal Alignment Externalities The identity pins down the deployed impact at a fixed training distribution; its differential companion describes how the impact—and hence welfare—moves as the training distribution is reweighted. For an RLHF procedure that trains on a query distribution qtrain∈Δ([N]×Z)q_train∈ ([N]× Z), we define the marginal alignment externality: the first-order welfare effect of infinitesimally upweighting a specific training source (n,z)(n,z). This is the alignment analogue of the influence function from robust statistics (see Hammoudeh and Lowd, 2024, for a review). Here θ^q θ_q denotes the anonymous RLHF parameter trained on distribution q; recall that Um(θ)=θm⊤ψ(θ)U_m(θ)= _m ψ(θ) is agent m’s deployment welfare. Definition 10 (Marginal Alignment Externality). For a training source (n,z)(n,z), let qε=(1−ε)qtrain+εδ(n,z)q_ =(1- )\,q_train+ \, _(n,z), where δ(n,z) _(n,z) is a point mass on (n,z)(n,z). The marginal alignment externality of source (n,z)(n,z) on agent m is γn,zm=dεUm(θ^qε)|ε=0=θm⊤dψ(θ^qε)dε|ε=0,γ^m_n,z= dd U_m( θ_q_ ) |_ =0= _m d\,ψ( θ_q_ )d |_ =0, (23) the inner product of m’s preference with the marginal impact shift from upweighting source (n,z)(n,z). The alignment externality Γ (Appendix D.1) and the marginal externality γ are related: the former is the discrete welfare change from participation, the latter the infinitesimal welfare change from reweighting. Both are inner products of preferences with impact shifts—Γ uses the finite shift Δn=ψ−ψ−n _n=ψ- _-n, while γ uses the infinitesimal shift dψ/dεdψ/d —but are conceptually distinct: Γ asks “what if agent n had never participated?” while γ asks “what if we gave source (n,z)(n,z) slightly more weight?” C.3 Anonymous RLHF and the Closed-Form Marginal Externality Anonymous RLHF fits a single parameter θ^RLHF θ^RLHF to pooled data, ignoring agent identities at training time: θ^qtrainRLHF∈argmaxθ∈ℝk−(n,z)∼qtrainc|n,z[−clogσ(αz⊤θ)−(1−c)log(1−σ(αz⊤θ))] θ_q_train^RLHF∈ _θ ^k\ -E_(n,z) q_train\,E_c n,z [-c σ( _z θ)-(1-c) (1-σ( _z θ) ) ] (24) The marginal externality has a closed form, obtained from the influence function of the estimator. Throughout the remainder of this section we assume the Bradley–Terry–Luce labeler model [c∣n,z]=σ(αz⊤θn)E[c n,z]=σ( _z _n) and that the training queries span ℝkR^k, so that H below is positive definite and θ^q θ_q is the unique stationary point of the training objective, depending smoothly on q by the implicit function theorem. (Otherwise, every statement should be read as restricted to Strain=spanαz:z∈supp(qtrain)S_train=span\ _z:z (q_train)\, with Sdep⊆StrainS_dep S_train required.) The first-order condition is g(θ^RLHF,qtrain)=0g( θ^RLHF;q_train)=0 with g(θ,q)=(n′,z′)∼q[(σ(αz′⊤θn′)−σ(αz′⊤θ))αz′]g(θ;q)=E_(n ,z ) q[(σ( _z _n )-σ( _z θ))\, _z ]. Differentiating g(θ^qε,qε)=0g( θ_q_ ;q_ )=0 at ε=0 =0 and using the base condition g(θ^RLHF,qtrain)=0g( θ^RLHF;q_train)=0 to cancel the distributional term gives the influence function dθ^qεdε|0=(σ(αz⊤θn)−σ(αz⊤θ^RLHF))H−1αz,H=(n′,z′)∼qtrain[v(αz′⊤θ^RLHF)αz′αz′⊤]. d\, θ_q_ d |_0= (σ( _z _n)-σ( _z θ^RLHF) )\,H^-1 _z, H=E_(n ,z ) q_train [v( _z θ^RLHF)\, _z _z ]. (25) Applying the Jacobian ∇ψ(θ^RLHF)=z′∼qdep[v(αz′⊤θ^RLHF)αz′αz′⊤]∇ψ( θ^RLHF)=E_z q_dep[v( _z θ^RLHF) _z _z ] (Theorem 1) and summing over agents with the welfare weights w yields a closed form for the social welfare effect γn,zW=∑mwmγn,zmγ^W_n,z= _mw_mγ^m_n,z of source (n,z)(n,z): γn,zW=(σ(αz⊤θn)−σ(αz⊤θ^RLHF))⏟preference deviation on query z(αz⊤η)⏟welfare sensitivity of query z _n,z^W= (σ( _z _n)-σ( _z θ^RLHF) )_preference deviation on query z\, ( _z η )_welfare sensitivity of query z (26) where η=(n′,z′)∼qtrain[v(αz′⊤θ^RLHF)αz′αz′⊤]−1z∼qdep[(αz⊤θ¯w)v(αz⊤θ^RLHF)αz]η=E_(n ,z ) q_train [v( _z θ^RLHF)\, _z \, _z ]^-1\,E_z q_dep [( _z θ_w)\,v( _z θ^RLHF)\, _z ] (27) is independent of (n,z)(n,z) and acts as a sufficient statistic for the welfare implications of parameter changes. Anonymous RLHF distorts welfare along directions that are simultaneously (i) systematically misrepresented in the training data and (i) socially important at deployment. C.4 RLHF Stationarity vs. Utilitarian Efficiency The central result of this section is that RLHF’s first-order condition and utilitarian efficiency impose qualitatively different requirements on the marginal externalities. RLHF ensures only that the social welfare effect of marginal perturbations cancels on average across training sources; utilitarian efficiency requires that it vanishes for every training source individually. Equivalently, γn,zW=θ¯w⊤dψ(θ^qε)dε|ε=0γ^W_n,z= θ_w d\,ψ( θ_q_ )d |_ =0, the inner product of the utilitarian preference with the marginal impact shift from upweighting source (n,z)(n,z). Theorem 5 (RLHF Stationarity vs. Utilitarian Efficiency). 1. RLHF stationarity (average zero). At the anonymous RLHF solution, the training-weighted average social welfare effect vanishes: (n,z)∼qtrain[γn,zW]=0.E_(n,z) q_train [ _n,z^W ]=0. (28) Individual sources may have γn,zW>0 _n,z^W>0 (upweighting would improve welfare) or γn,zW<0 _n,z^W<0, but these cancel on average under qtrainq_train. 2. Utilitarian efficiency (pointwise zero). If the training weighting is welfare-optimal—if qtrainq_train maximizes the deployed utilitarian welfare θ¯w⊤ψ(θ^q) θ_w ψ( θ_q) over all reweightings q of its sources—then the social welfare effect vanishes for every source individually: γn,zW=0∀(n,z)∈supp(qtrain). _n,z^W=0 ∀\,(n,z) (q_train). (29) Anonymous RLHF guarantees only the average condition. Whenever it leaves some source with γn,zW≠0 _n,z^W≠ 0, upweighting the sources with positive marginal effect (and downweighting those with negative effect) strictly improves deployed welfare to first order. Proof of Theorem 5. (1). By the closed form (26), (n,z)∼qtrain[γn,zW]=((n,z)∼qtrain[(σ(αz⊤θn)−σ(αz⊤θ^RLHF))αz])⊤η=0,E_(n,z) q_train[γ^W_n,z]= (E_(n,z) q_train[(σ( _z _n)-σ( _z θ^RLHF))\, _z] ) η=0, because the RLHF first-order condition makes the bracketed vector vanish. (2). Let W(q)=U(θ^q)=θ¯w⊤ψ(θ^q)W(q)=U( θ_q)= θ_w ψ( θ_q) be the deployed welfare as a function of the training weighting. Since qε=(1−ε)qtrain+εδ(n,z)q_ =(1- )q_train+ _(n,z), the definition of γn,zWγ^W_n,z is the directional derivative toward the vertex (n,z)(n,z) of the simplex, γn,zW=dεW(qε)|0=∂n,zW−qtrain[∂W].γ^W_n,z= dd W(q_ ) |_0= _n,zW-E_q_train[∂ W]. If qtrainq_train maximizes W over the simplex of reweightings, the first-order (KKT) condition is that the partials ∂n,zW _n,zW are equal—to a common multiplier λ—across supp(qtrain)supp(q_train); their qtrainq_train-average is then also λ, so γn,zW=λ−λ=0γ^W_n,z=λ-λ=0 for every (n,z)∈supp(qtrain)(n,z) (q_train). ∎ The welfare consequences of RLHF distortion are fully characterized by the single vector Δψ∈ℝk ψ ^k: the aggregate welfare loss is θ¯w⊤Δψ θ_w ψ, and the per-agent redistribution is (θn⊤Δψ)n=1N( _n ψ)_n=1^N. Remark 1 (Query-level decomposition). The impact gap decomposes across deployment queries as θ¯w⊤Δψ=z∼qdep[(θ¯w⊤αz)(12sign(αz⊤θ¯w)−(σ(αz⊤θ^RLHF)−12))] θ_w ψ=E_z q_dep [( θ_w _z) ( 12sign( _z θ_w)- (σ( _z θ^RLHF)- 12 ) ) ] (30) The distortion on each query z is the difference between the deterministic utilitarian decision and the probabilistic RLHF decision, weighted by the welfare importance of the query. C.5 Recoverability under Anonymous RLHF A natural question is whether reweighting the training distribution can close the gap. By the impact identity (Proposition 4), reweighting moves the deployed impact only through the per-query label frequencies μz\ _z\, and the answer depends on whether the reweighting conditions on individual identities. Let C=Conv(θnn=1N)C=Conv(\ _n\_n=1^N) be the convex hull of the population preferences, let S=spanαz:z∈supp(qtrain)S=span\ _z:z (q_train)\, and let PSP_S denote projection onto S. Definition 11. A parameter θ∈ℝkθ ^k is individually mix-recoverable if there exists, for each query z∈Zz∈ Z, a probability vector π(⋅∣z)∈ΔNπ(· z)∈ _N such that σ(αz⊤θ)=∑n=1Nπ(n∣z)σ(αz⊤θn)for every z∈Zσ( _z θ)= _n=1^Nπ(n z)\,σ( _z _n) every $z∈ Z$ Let Θindiv ^indiv denote the set of all such parameters. A parameter is anonymously mix-recoverable if π(n∣z)=1Nπ(n z)= 1N for every n. Denote the set of such parameters Θanon ^anon. Proposition 5. 1. A parameter θ∈Θindivθ∈ ^indiv if and only if αz⊤θ∈[minnαz⊤θn,maxnαz⊤θn] _z θ∈ [ _n _z _n,\; _n _z _n ] for every z∈Zz∈ Z. 2. PSC⊆PSΘindivP_S\,C P_S\, ^indiv: the convex hull of individual preferences is contained in the individually recoverable set. Proof of Proposition 5. Part 1: ⇒ Per the definition of θ∈Θindivθ∈ ^indiv, there exists a π such that, for all z, σ(αz⊤θ)=∑n=1Nπ(n|z)σ(αz⊤θn)σ( _z θ)= _n=1^Nπ(n|z)\,σ( _z _n) (31) As a convex combination of scalars, the right-hand side must lie between its extreme points, so that σ(αz⊤θ)∈[minnσ(αz⊤θn),maxnσ(αz⊤θn)]σ( _z θ)∈[ _nσ( _z _n), _nσ( _z _n)] for all z. But then, by the strict monotonicity of σ, this implies that αz⊤θ∈[minnαz⊤θn,maxnαz⊤θn] _z θ∈[ _n _z _n, _n _z _n] for all z. ⇐ Let z∈Zz∈ Z be arbitrary, and suppose that σ(αz⊤θ)∈[minnσ(αz⊤θn),maxnσ(αz⊤θn)]σ( _z θ)∈[ _nσ( _z _n), _nσ( _z _n)]. Let n¯=argminnσ(αz⊤θn) n= _nσ( _z _n) and n¯=argmaxnσ(αz⊤θn) n= _nσ( _z _n). Since σ(αz⊤θ)σ( _z θ) lies in the closed interval with endpoints σ(αz⊤θn¯)σ( _z _ n) and σ(αz⊤θn¯)σ( _z _ n), there exists tz∈[0,1]t_z∈[0,1] such that σ(αz⊤θ)=(1−tz)σ(αz⊤θn¯)+tzσ(αz⊤θn¯).σ( _z θ)=(1-t_z)\,σ( _z _ n)+t_z\,σ( _z _ n). (32) Set π(n¯∣z)=1−tzπ( n z)=1-t_z, π(n¯∣z)=tzπ( n z)=t_z, and π(n∣z)=0π(n z)=0 for every other n. If n¯=n¯ n= n, the interval is a single point, and we set π(n¯∣z)=1π( n z)=1 instead. In either case π(⋅∣z)∈ΔNπ(· z)∈ _N, as the definition of Θindiv ^indiv requires. Applying this construction at each z∈Zz∈ Z yields the result. Part 2: PSC⊆PSΘindivP_SC P_S ^indiv. Fix any θ∈Cθ∈ C. By definition of C, there exists λ∈ΔNλ∈ _N such that θ=∑n=1Nλnθn.θ= _n=1^N _n _n. Then, for every z∈Zz∈ Z, αz⊤θ=∑n=1Nλnαz⊤θn, _z θ= _n=1^N _n _z _n, so αz⊤θ _z θ lies between minnαz⊤θn _n _z _n and maxnαz⊤θn _n _z _n. By Part 1, θ∈Θindivθ∈ ^indiv, and therefore PSC⊆PSΘindivP_SC P_S ^indiv. ∎ Thus, with individual-level label weights, RLHF can recover any parameter in the convex hull C of the population preferences—in particular the welfare-weighted parameter θ¯w θ_w for every w∈ΔNw∈ _N: if θ∈Θindivθ∈ ^indiv with mixture weights π, then setting qtrain(n∣z)=π(n∣z)q_train(n z)=π(n z) makes θ satisfy the stationarity condition of Proposition 4 exactly, with zero population loss. Anonymous weighting, by contrast, is far more rigid: Proposition 6. If S≠0S≠\0\, then PSΘanonP_S\, ^anon is either empty or a singleton. Proof of Proposition 6. When θ∈Θanonθ∈ ^anon, αz⊤θ=σ−1(1N∑n=1Nσ(αz⊤θn))∀z∈Z _z θ=σ^-1 ( 1N _n=1^Nσ( _z _n) ) ∀ z∈ Z (33) where the right-hand side is determined entirely by the population θnn=1N\ _n\_n=1^N for each z. If there exists no θ satisfying this system of linear constraints, then Θanon=∅ ^anon= and hence PSΘanonP_S\, ^anon is empty. Otherwise, suppose θ,θ′∈Θanonθ,θ ∈ ^anon. They both satisfy the system of linear constraints, so for every z∈Zz∈ Z, αz⊤θ=αz⊤θ′ _z θ= _z θ and hence αz⊤(θ−θ′)=0 _z (θ-θ )=0 for z∈Zz∈ Z. This implies θ−θ′θ-θ is orthogonal to spanαz:z∈Z⊇Sspan\ _z:z∈ Z\ S, and hence to S. Thus all elements of Θanon ^anon have the same projection onto S. Therefore PSΘanonP_S\, ^anon is at most a singleton. ∎ Anonymous RLHF is pinned to at most one parameter in the identifiable subspace, regardless of how the query distribution is chosen. Corollary 6 (Impact-space consequence). Let Ψindiv=ψ(θ):θ∈Θindiv ^indiv=\ψ(θ):θ∈ ^indiv\ and Ψanon=ψ(θ):θ∈Θanon ^anon=\ψ(θ):θ∈ ^anon\. Then ψ(θ¯w)∈Ψindivψ( θ_w)∈ ^indiv for every w∈ΔNw∈ _N, but Ψanon ^anon is at most a singleton. If Ψanon=ψanon ^anon=\ψ^anon\, the welfare gap from the anonymity constraint is θ¯w⊤(ψ⋆−ψanon) θ_w (ψ -ψ^anon). Proof of Corollary 6. The proof of Proposition 5(2) shows C⊆ΘindivC ^indiv, and θ¯w∈C θ_w∈ C, so ψ(θ¯w)∈Ψindivψ( θ_w)∈ ^indiv. For the second claim, any θ,θ′∈Θanonθ,θ ∈ ^anon satisfy (33), so αz⊤θ=αz⊤θ′ _z θ= _z θ for every z∈Zz∈ Z; since ψ depends on its argument only through (αz⊤θ)z∈supp(qdep)( _z θ)_z (q_dep) and supp(qdep)⊆Zsupp(q_dep) Z, the map ψ is constant on Θanon ^anon, and Ψanon ^anon is at most a singleton. ∎ Letting μz=n∼qtrain(n∣z)[σ(αz⊤θn)] _z=E_n q_train(n z)[σ( _z _n)] denote the population-level choice frequency for z, anonymous RLHF effectively computes the M-projection of the choice data onto the logistic model family: θ^qtrainRLHF∈argminθ∈ℝk(z,n)∼qtrain[DKL(Bern(μz)∥Bern(σ(αz⊤θ)))] θ^RLHF_q_train∈ _θ ^k\ E_(z,n) q_train\! [D_KL\! (Bern( _z)\, \|\,Bern(σ( _z θ)) ) ] Proposition 7. Suppose every query is represented in training, supp(qtrain)=Zsupp(q_train)=Z. If θ^qtrainRLHF θ_q_train^RLHF achieves zero population training loss, then PSθ^qtrainRLHF∈PSΘindiv.P_S θ_q_train^RLHF∈ P_S ^indiv. Proof of Proposition 7. If θ^qtrainRLHF θ^RLHF_q_train achieves zero excess loss, then every KL term vanishes on the support of qtrainq_train, hence σ(αz⊤θ^qtrainRLHF)=∑n=1Nqtrain(n∣z)σ(αz⊤θn)∀z∈supp(qtrain)σ( _z θ^RLHF_q_train)= _n=1^Nq_train(n z)\,σ( _z _n) ∀ z (q_train) (34) Since supp(qtrain)=Zsupp(q_train)=Z, this is exactly the mixture-recovery condition of Proposition 5(1) at every z∈Zz∈ Z. As θ^qtrainRLHF θ^RLHF_q_train is only identifiable in the subspace S, applying Proposition 5 gives PSθ^qtrainRLHF∈PSΘindiv.P_S θ^RLHF_q_train∈ P_S ^indiv. ∎ Appendix D Additional Material for Section 6 D.1 Alignment Externalities Define Δψ=ψ⋆−ψRLHF ψ=ψ -ψ^RLHF as the impact gap. Writing U⋆=hΨ¯(θ¯w)=θ¯w⊤ψ⋆U =h_ ( θ_w)= θ_w ψ for the optimal utilitarian welfare, the welfare loss from RLHF is U⋆−θ¯w⊤ψRLHF=θ¯w⊤Δψ≥ 0U - θ_w ψ^RLHF= θ_w ψ\;≥\;0, with equality if and only if ψRLHFψ^RLHF is itself utilitarian-optimal. Agent n’s welfare change from deploying ψRLHFψ^RLHF instead of ψ⋆ψ is θn⊤ψRLHF−θn⊤ψ⋆=−θn⊤Δψ. _n ψ^RLHF- _n ψ =- _n ψ. Agents whose preferences are aligned with the impact gap (θn⊤Δψ>0 _n ψ>0) are harmed by RLHF’s distortion; agents anti-aligned with Δψ ψ benefit. D.1.1 Participation Alignment Externality The participation alignment externality measures the welfare impact of including an agent’s data in the training process relative to the counterfactual in which they are excluded. This is the alignment analogue of the VCG pivot from mechanism design: it quantifies how much an agent’s participation affects others. To see this formally, let θ θ denote the parameter obtained from training on the full population and θ^−n θ_-n the parameter obtained when agent n is excluded (and remaining agents’ weights are renormalized). The corresponding deployed impacts are ψ=ψ(θ^)ψ=ψ( θ) and ψ−n=ψ(θ^−n) _-n=ψ( θ_-n). Definition 12 (Participation Alignment Externality). The participation alignment externality of agent n on agent m is Γn,m=Um(θ^)−Um(θ^−n)=θm⊤(ψ−ψ−n) _n,m=U_m( θ)-U_m( θ_-n)= _m \! (ψ- _-n ) (35) The second equality follows from Um(θ)=θm⊤ψ(θ)U_m(θ)= _m ψ(θ). In impact space, the alignment externality is the inner product of m’s preference with the impact shift Δn=ψ−ψ−n _n=ψ- _-n caused by n’s participation. The impact shift Δn∈ℝk _n ^k is a vector that represents the welfare consequences of n’s participation: agent m benefits from n’s inclusion whenever θm⊤Δn>0 _m _n>0 (their preferences are aligned with the impact shift) and is harmed whenever θm⊤Δn<0 _m _n<0. Thus, the aggregate externality of n on all other agents is Γn,−n=∑m≠nwmΓn,m=(1−wn)θ¯−n⊤Δn _n,-n= _m≠ nw_m\, _n,m=(1-w_n)\, θ_-n _n (36) where θ¯−n=11−wn∑m≠nwmθm θ_-n= 11-w_n _m≠ nw_m _m is the leave-one-out utilitarian preference. This is positive if the addition of n’s data moves the estimated preference parameter in a direction the rest of the population likes and negative when n’s participation moves the parameter in a direction the rest of the population dislikes. Proof of Theorem 4. Define the Lagrangian ℒ(ψ,η)=θ¯w⊤ψ+∑n=1Nηn(γn⊤ψ−cn),ηn≥0.L(ψ,η)= θ_w ψ+ _n=1^N _n( _n ψ-c_n), _n≥ 0. (37) Equivalently, ℒ(ψ,η)=(θ¯w+∑n=1Nηnγn)⊤ψ−∑n=1Nηncn.L(ψ,η)= ( θ_w+ _n=1^N _n _n ) ψ- _n=1^N _nc_n. (38) Since the feasible set is non-empty and compact and the objective is continuous, an optimum exists. By Theorem 1, Ψ¯ is a zonotope generated by finitely many segments, hence a polytope, so the feasible set Ψ¯∩⋂nγn⊤ψ≥cn ∩ _n\ _n ψ≥ c_n\ is a polyhedron and the problem is a finite linear program. Linear-programming duality therefore yields—with no constraint qualification required—multipliers ηn≥0 _n≥ 0 such that the constrained optimum also solves ψwel∈argmaxψ∈Ψ¯(θ¯w+∑n=1Nηnγn)⊤ψ.ψ^wel∈ _ψ∈ ( θ_w+ _n=1^N _n _n ) ψ. (39) Complementary slackness gives ηn(γn⊤ψwel−cn)=0∀n. _n( _n ψ^wel-c_n)=0 ∀ n. (40) Hence only binding constraints enter the effective objective direction, so if ℬ=n:ηn>0B=\n: _n>0\, then θ¯w+∑n∈ℬηnγn θ_w+ _n _n _n (41) is the effective welfare direction. Finally, let V(c)V(c) denote the optimal value as a function of the threshold vector c. Raising any cnc_n shrinks the feasible set, so V is weakly decreasing in each cnc_n. For concavity, let ψ1,ψ2 _1, _2 be optimal at thresholds c(1),c(2)c^(1),c^(2) and t∈[0,1]t∈[0,1]; then tψ1+(1−t)ψ2∈Ψ¯t _1+(1-t) _2∈ by convexity and satisfies γn⊤(tψ1+(1−t)ψ2)≥tcn(1)+(1−t)cn(2) _n (t _1+(1-t) _2)≥ tc^(1)_n+(1-t)c^(2)_n for all n, so it is feasible at tc(1)+(1−t)c(2)tc^(1)+(1-t)c^(2) and V(tc(1)+(1−t)c(2))≥tV(c(1))+(1−t)V(c(2))V(tc^(1)+(1-t)c^(2))≥ tV(c^(1))+(1-t)V(c^(2)). Whenever V is differentiable, its derivative satisfies ∂V∂cn=−ηn. ∂ V∂ c_n=- _n. (42) ∎ Proof of Corollary 2. Apply Theorem 4 with γn=θn _n= _n and cn=bnc_n=b_n. The effective welfare direction is θ¯w+∑n=1Nηnθn=∑n=1N(wn+ηn)θn. θ_w+ _n=1^N _n _n= _n=1^N(w_n+ _n) _n. Complementary slackness gives ηn(θn⊤ψbh−bn)=0∀n. _n( _n _b^h-b_n)=0 ∀ n. ∎ Proof of Corollary 3. Apply Theorem 4 with γnps=γθ¯w+(1−γ)θnandcn=(γθ¯w+(1−γ)θn)⊤ψ0. _n^ps=γ θ_w+(1-γ) _n c_n= (γ θ_w+(1-γ) _n ) _0. The effective welfare direction is θ¯w+∑n=1Nηn(γθ¯w+(1−γ)θn) θ_w+ _n=1^N _n (γ θ_w+(1-γ) _n ) =(1+γ∑n=1Nηn)θ¯w+(1−γ)∑n=1Nηnθn. = (1+γ _n=1^N _n ) θ_w+(1-γ) _n=1^N _n _n. By complementary slackness, only binding constraints enter this expression, giving (1+γ∑n∈ℬηn)θ¯w+(1−γ)∑n∈ℬηnθn. (1+γ _n _n ) θ_w+(1-γ) _n _n _n. The complementary slackness condition is ηn[(γθ¯w+(1−γ)θn)⊤(ψγps−ψ0)]=0∀n. _n [ (γ θ_w+(1-γ) _n ) ( _γ^ps- _0) ]=0 ∀ n. ∎ Proof of Corollary 4. Apply Theorem 4 with γn=θ¯−n _n= θ_-n. Substituting θ¯−n=11−wn(θ¯w−wnθn) θ_-n= 11-w_n( θ_w-w_n _n) into the effective direction: θ¯w+∑n∈ℬηnθ¯−n θ_w+ _n _n\, θ_-n =θ¯w+∑n∈ℬηn1−wn(θ¯w−wnθn) = θ_w+ _n _n1-w_n( θ_w-w_n\, _n) =(1+∑n∈ℬηn1−wn)θ¯w−∑n∈ℬwnηn1−wnθn. = (1+ _n _n1-w_n ) θ_w\;-\; _n w_n\, _n1-w_n\, _n. Complementary slackness follows from the general theorem. ∎ The constrained welfare mechanisms introduced in Section 6 can be understood as directly regulating the alignment externality system defined in Appendix D.1. Proposition 8 (Externality Decomposition of Welfare Floors). Let ψ=ψ(θ^)ψ=ψ( θ) be the impact deployed by the full-population training procedure and ψ−n=ψ(θ^−n) _-n=ψ( θ_-n) the leave-one-out impact, as in Definition 12. Then the welfare floor constraint θm⊤ψ≥bm _m ψ≥ b_m is equivalent to Γn,m≥bm−θm⊤ψ−n∀n. _n,m\;≥\;b_m- _m _-n ∀\,n. (43) That is, the welfare floor constrains each agent’s participation externality on m to be large enough to lift m’s welfare above the floor, given m’s baseline welfare without n. Proof. From θm⊤ψ=θm⊤ψ−n+Γn,m _m ψ= _m _-n+ _n,m, the floor θm⊤ψ≥bm _m ψ≥ b_m is equivalent to Γn,m≥bm−θm⊤ψ−n _n,m≥ b_m- _m _-n. ∎ Public spirit changes how binding agents pull on the solution. When γ is small, their constraints act mostly like personal welfare protections, tilting the objective toward their own preferences. When γ is large, the same constraints put more weight on the social objective itself. Thus public spirit makes the constrained solution look less like individualized compensation and more like utilitarian alignment. Appendix E Additional Experiments E.1 Dataset Information and Experimental Setup All experiments were run locally on commodity hardware. An A100 was rented for conducting the linear feature embeddings for the Community Alignment dataset, which took approximately 5 minutes to run. All code can be found at https://github.com/michelleeesi/hiddenstructure. E.1.1 412 Food Rescue We use the data from Lee et al. (2019), which records pairwise choices over food-rescue recipients along k=7k=7 features (size, access, income, poverty, last_donation, total_donation, dist). Each row of the CSV is one binary comparison by one participant: we set αz=ϕA−ϕB _z= _A- _B and yz=chose Ay_z=1\chose A\, with 19 participants (different stakeholders for the 412 Food Rescue program) and 45 pairwise responses each. This data is not public and was accessed with permission from the original authors on the WeBuildAI paper. E.1.2 Kidney Exchange We use the kidney-allocation data from Keswani et al. (2024); Boerstler et al. (2024), in which respondents choose between two patients described by k=5k=5 features (elderlyDep, lifeYearsGained, obesity, weeklyWorkhours, yearsWaiting). We construct αz _z from the released columns as the left-option minus right-option feature difference and yzy_z from the chosen indicator. Each respondent responded to ∼400 400 pairwise comparisons. The data is publicly available online at https://github.com/vijaykeswani/Preference-Instability/tree/main/Study%201%262%20-%20AIES%202024. E.1.3 Moral Machine We use the trolley-problem dataset from Awad et al. (2018), which records, for each of millions of participants, a choice between two groups of characters drawn from 20 character types (Man, Woman, Pregnant, OldMan, Boy, Girl, Homeless, Criminal, MaleExecutive, FemaleExecutive, MaleAthlete, FemaleAthlete, MaleDoctor, FemaleDoctor, Dog, Cat, etc.; k=20k=20). We process the public data by restricting to annotators with between 100 and 500 responses, and by screening out bots. The Moral Machine dataset is publicly available online at https://osf.io/mxa6z/overview. E.1.4 Community Alignment We use the Community Alignment Dataset of Zhang et al. (2025), which contains over 200,000 pairwise preference judgments collected from over 3000 annotators across five countries. We apply the natural language preference learning method of Wojtowicz et al. (2026). Options are encoded by a frozen sentence encoder and pairwise differences are projected onto a K=15K=15-dimensional human-interpretable basis of preference-relevant axes of variation in the choice domain. Per-participant preferences are then estimated under a Bradley-Terry-Luce model on this subspace. We filtered for annotators with ≥20≥ 20 responses, leaving N=2387N=2387 annotators as agents. E.2 Welfare Dispersion vs. Choice Precision Figure 3: Community Alignment dataset: per-agent welfare Un/U⋆U_n/U as a function of the choice-precision parameter β, for each mechanism. The shaded band is the density of all 2387 agents; solid, dashed, and dotted lines mark the best-off agent, mean welfare, and worst-off agent. As β grows, choices become more deterministic, individual preferences become more powerful, and the welfare distribution spreads—widening the gap between best- and worst-off agents. Strategyproof and harm-bounded mechanisms produce lower welfare dispersion, and the bounded-harm constraints are empirically satisfied. E.3 Per-Agent Harm Bound Figure 4: Participation externalities under the per-agent harm bound. Here each agent’s welfare floor is a fixed fraction of their own maximum achievable welfare, bn=−ϵhΨ¯(θn)b_n=-ε\,h_ ( _n) with ϵ=0.05ε=0.05 (lose at most 5% of one’s own best case), rather than a single absolute floor. The left panel is the utilitarian baseline; the right is the harm-bounded model. The qualitative story matches that of the single absolute floor discussed in Section 6: a small set of binding agents accounts for almost all of the participation externalities. E.4 Full Alignment Externalities Figure 5: Community Alignment Dataset: Utilitarian vs RLHF participation alignment externalities by demographic. The figure includes 2387 annotators, with full alignment externalities averaged within each group. Figure 6: Community Alignment Dataset: 100 person sample with granular person-on-person full externalities compared to the group-averaged full externalities. Our participation externality formalization gives us a microscope to analyze where the per-individual welfare effects are coming from. E.5 Four Dataset Comparisons Figure 7: Preference distribution per feature dimension for four datasets. Moral Machine and Community Alignment have high disagreement, with a high number of preferences on either side of 0, while Kidney Allocation and 412 Food Rescue have less disagreement: people disagree on the magnitude of preference, but the direction is more unanimous. Figure 8: Expected welfare for four datasets. Bootstrap mean and std. deviations were calculated by bootstrapping l agents 100 times and calculating the welfare of the strategyproof mechanism. Sequential addition denotes the procedure where 100 permutations of agents were generated, and welfare was calculated for each coalition created by adding a new agent. Figure 9: Welfare variance over four datasets. The strategyproof mechanism leads to the least welfare dispersion, while RLHF follows as a far second for larger values of β. Figure 10: Welfare of the best- and worst-off agents in four datasets for three model parameters.