Paper deep dive
Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers
Taenyun Kim, Edyta Bogucka, Daniele Quercia
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As AI systems make more morally loaded decisions across society, one response has been moral preference elicitation. In this approach, researchers poll participants on hypothetical dilemmas and use the aggregated votes to train a policy that an AI model then applies at scale. Before any vote is cast, developers make three key choices in the moral AI elicitation pipeline: feature scoping, voter sampling, and question framing. In other words, they decide which features go to a vote, which voters to include, and how to present the question. These choices are often opaque, undocumented, and treated as technical details rather than normative ones. We examine each of these choices within a common empirical study and show that each can shape the preferences produced by moral AI elicitation. Across two phases (N = 809) in three deployment contexts (i.e., AI kidney allocation, AI agents simulating absent workers, and generative AI depictions of the deceased), we examine the three main stages of the moral AI elicitation pipeline. First, morally relevant features shift across contexts. This suggests that feature schemas should not be assumed to transfer across deployment domains. Second, preferences differ by political ideology for roughly one-third of features, with some differences reversing direction. The ideological composition of the voter pool can therefore affect the resulting aggregated preference profile. Third, the wording of the elicitation question can narrow or widen ideological gaps by up to a full scale point. The framing conditions also change how moral foundations are associated with participants' judgments. Taken together, these findings suggest that voting-based alignment cannot deliver fair or transparent AI by aggregation alone; at minimum, each stage of the moral AI elicitation pipeline should be audited and disclosed.
Tags
Links
- Source: https://arxiv.org/abs/2608.14522v1
- Canonical: https://arxiv.org/abs/2608.14522v1
Trouble viewing inline? Open PDF directly →
Full Text
128,667 characters extracted from source content.
Expand or collapse full text
Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers Taenyun Kim 1 , Edyta Bogucka 2 , Daniele Quercia 2,3 1 Michigan State University, US 2 Nokia Bell Labs, Cambridge, UK 3 Politecnico di Torino, Italy kimtaeny@msu.edu, edyta.bogucka@nokia-bell-labs.com, daniele.quercia@nokia-bell-labs.com Abstract As AI systems make more morally loaded decisions across society, one response has been moral preference elicitation. In this approach, researchers poll participants on hypotheti- cal dilemmas and use the aggregated votes to train a policy that an AI model then applies at scale. Before any vote is cast, developers make three key choices in the moral AI elici- tation pipeline: feature scoping, voter sampling, and question framing. In other words, they decide which features go to a vote, which voters to include, and how to present the question. These choices are often opaque, undocumented, and treated as technical details rather than normative ones. We exam- ine each of these choices within a common empirical study and show that each can shape the preferences produced by moral AI elicitation. Across two phases (N = 809) in three deployment contexts (i.e., AI kidney allocation, AI agents simulating absent workers, and generative AI depictions of the deceased), we examine the three main stages of the moral AI elicitation pipeline. First, morally relevant features shift across contexts. This suggests that feature schemas should not be assumed to transfer across deployment domains. Second, preferences differ by political ideology for roughly one-third of features, with some differences reversing direction. The ideological composition of the voter pool can therefore affect the resulting aggregated preference profile. Third, the word- ing of the elicitation question can narrow or widen ideolog- ical gaps by up to a full scale point. The framing conditions also change how moral foundations are associated with par- ticipants’ judgments. Taken together, these findings suggest that voting-based alignment cannot deliver fair or transpar- ent AI by aggregation alone; at minimum, each stage of the moral AI elicitation pipeline should be audited and disclosed. Supplementary materials and further results are available at https://social-dynamics.net/moral-ai. 1 Introduction AI systems are increasingly used to make high-stakes deci- sions in areas such as healthcare, employment, and criminal justice (Chan et al. 2024; Kim and Peng 2025). Aligning these systems with human moral values in a way that is fair, accountable, and transparent has therefore become a cen- tral problem (Boerstler et al. 2024; Freedman et al. 2020; Kneer and Viehoff 2025). To address it, moral preference elicitation now stands as a common approach: researchers present participants with hypothetical moral dilemmas, ask them a question about what the AI system should do, and use these responses to train models that produce a decision pol- icy (Boerstler et al. 2024; Freedman et al. 2020; Kneer and Viehoff 2025; Noothigattu et al. 2018; Awad et al. 2018). The approach aims to democratize alignment by grounding model design in the preferences of a broad and representa- tive public (Simson et al. 2025; Dahl and Svanæs 2020). In practice, however, before any question is asked, de- velopers make three key choices that shape what the final policy can capture: feature scoping (which features are in- cluded), voter sampling (which participants are included), and question framing (how the question is asked). These choices are rarely documented, and are typically treated as technical rather than moral decisions. Each of these choices can shape what is ultimately presented as an “aggregated moral preference”. If the selected features, participant pop- ulation, or question wording affect the results, the resulting policy reflects upstream developer decisions as well as par- ticipants’ expressed values. We examine these three choices within a common empirical study, using one research ques- tion for each stage: RQ 1 : Feature scoping. Do people consider the same fea- tures across different use cases? RQ 2 : Voter sampling. Do moral preferences vary with po- litical ideology? RQ 3 : Question framing. Does framing change moral pref- erences? To address these questions, we conducted a study with participants holding different political views across three AI use cases: AI kidney allocation (KIDNEY), AI agents sim- ulating absent workers (WORK), and generative AI con- tent of the deceased (GEN). These cases vary in the sever- ity of harm and everyday frequency of encounter, from rare high-stakes allocation to common workplace decisions and emerging questions about how AI should represent the deceased. We ran two phases for each use case. In Phase 1 (N = 150, 149, 150 per use case), participants identified which features should or should not matter. In Phase 2 (N = 120 per use case), participants judged whether each feature should count in favor, count against, or not count at all, under one of three conditions: a control, World-You-Want (a structural perspective (Jaques 2025)), or Could-Be-You (a perspective-taking approach based on Rawlsian theory (Rawls 1971; Huang, Greene, and Bazer- arXiv:2608.14522v1 [cs.AI] 14 Aug 2026 man 2019; Bruno et al. 2024)). We then classified features as relevant, irrelevant, or divisive, and tested whether po- litical ideology and framing changed these judgments. This analysis produced three main findings: 1) Moral feature relevance is context-specific (§5.1). Rel- evant features vary by use case. KIDNEY emphasizes distributive justice and medical utility. WORK focuses on accountability and legitimacy. GEN centers on con- sent and dignity. The relevant feature sets differed across the three contexts. Choosing which features to include is therefore a moral decision that defines the boundaries of the elicitation process. 2) Voter sampling can change the aggregated preference profile (§5.2). Political ideology is associated with dif- ferent evaluations for roughly one-third of features, in- cluding some differences that reverse direction. In KID- NEY, conservatives place greater weight on proportional- ity and utility, while progressives place greater weight on equality. In WORK, conservatives emphasize employer loyalty and contractual duty. In GEN, conservatives place greater weight on the dignity of the deceased, while pro- gressives are more permissive toward creative use. The ideological composition of the voter pool can therefore affect the preferences represented in the aggregate. 3) Question framing changes ideological gaps and value pathways (§5.3). Framing can change results by up to one scale point, narrowing some ideological gaps while widening others. The framing conditions also change how moral foundations are associated with par- ticipants’ judgments. Question framing should therefore be treated as a consequential design choice rather than a neutral elicitation tool. Taken together, these findings show that voting-based alignment does not simply recover a pre-existing set of pub- lic moral preferences: its outputs are shaped by upstream choices about what is put to a vote, whose preferences are represented, and how those preferences are elicited. This pa- per makes three contributions. First, it conceptualizes fea- ture scoping, voter sampling, and question framing as nor- mative developer choices in moral AI elicitation. Second, it provides comparative empirical evidence about each choice across three deployment contexts. Third, it translates these findings into a sensitivity audit that makes upstream choices visible and contestable (§5.4), while examining the limits of aggregation even when those choices are disclosed (§5.5). 2 Related Work Next, we review prior work on voting-based AI alignment and the three developer choices that structure it: feature scoping, voter sampling, and question framing. 2.1 Voting-Based AI Alignment and Its Assumptions Aligning AI systems with human moral values is a central concern for AI safety research (Gabriel 2020; Kneer and Viehoff 2025). One common approach is moral preference elicitation, in which researchers present hypothetical dilem- mas, collect participants’ responses, and aggregate them into a decision policy that a model applies at scale (Noothigattu et al. 2018; Awad et al. 2018; Freedman et al. 2020; Keswani et al. 2025). This approach is often framed as democratizing alignment by replacing developers’ private judgments with public input (Simson et al. 2025; Dahl and Svanæs 2020). Yet developers still decide what participants evaluate, who participates, and how questions are posed. We call these choices feature scoping, voter sampling, and question fram- ing. Claims that aggregation neutrally recovers public values assume that features transfer across contexts, voter compo- sition has limited influence, and wording reveals rather than shapes values. Prior work challenges these assumptions: ag- gregation can marginalize minority views (Feffer, Heidari, and Lipton 2023a), while demographic groups prioritize eth- ical values differently (Jakesch et al. 2022). Anthropic’s Constitutional AI illustrates the issue: although 1000 Amer- icans contributed to a model constitution, developers still se- lected participants, statements, and training principles (Gan- guli et al. 2023). More fundamentally, aggregated prefer- ences cannot determine which views deserve authority, how conflicts should be resolved, or what protections should con- strain majority decisions (Salloch et al. 2015; Harris 2020; Baum and Slavkovik 2025; Zhi-Xuan et al. 2024). Taken to- gether, this suggests preferences should be treated as one input into moral reasoning rather than as a substitute for it. Our study does not attempt to resolve this problem. Instead, it examines how upstream design choices further condition what is presented as an aggregated preference. 2.2 Moral Preferences Across AI Use Cases Most moral preference elicitation studies focus on a small set of use cases that tend to share three properties: they in- volve severe harm if the AI fails or is misused, they are rare or one-off situations, and they concern decisions that hu- mans have traditionally made case by case. Organ allocation and autonomous vehicle scenarios are the clearest exam- ples of this pattern, with concerns centered on life-and-death tradeoffs and the allocation of scarce resources (Noothigattu et al. 2018; Awad et al. 2018). More frequent, lower-stakes cases such as content moderation, ad targeting, and con- sumer AI have received far less attention, yet they raise dif- ferent moral concerns around transparency, consent, and ac- countability (Jakesch et al. 2022). Within a use case, feature scoping can follow two ap- proaches: top-down and bottom-up. In a top-down approach, developers select features before recruiting participants, drawing on existing literature or industry guidelines. In a bottom-up approach, participants themselves identify which considerations they find morally relevant (Sadek, Calvo, and Mougenot 2023). Most studies have used the top-down ap- proach. For example, Awad et al. (2018) reviewed the West- ern academic literature to identify 18 features describing the people affected by an autonomous vehicle’s decision, in- cluding their gender, age, and physical fitness. Critics ar- gue that this forced-choice format narrows the ethical prob- lem by preselecting which people, outcomes, and trade-offs participants can consider, while omitting factors such as uncertainty and how likely each outcome is (Dewitt, Fis- chhoff, and Sahlin 2019; Etienne 2021; Freitas et al. 2021; Schuessler 2023). Similarly, Jakesch et al. (2022) drew on industry ethics guidelines to identify 12 values relevant to re- sponsible AI in ad targeting, including safety and fairness. In both cases, what goes to a vote was decided before any par- ticipant was recruited, and researchers have called for such decisions to be documented and opened to scrutiny (Feffer, Heidari, and Lipton 2023a; Arzberger et al. 2025). 2.3 Group Differences in Moral Preferences Differences in moral judgment across groups are well doc- umented. In moral preference elicitation tasks, participants with different demographic, AI literacy, or political back- grounds often reach different conclusions (Graham, Haidt, and Nosek 2009; Atari et al. 2023). This diversity poses a challenge for systems that attempt to learn a single set of “universal” moral preferences (Gabriel 2020; Jakesch et al. 2022). Disagreements arise at two levels: people differ in which features they consider relevant (Keswani et al. 2025), and they differ in how they weight those features (Brugman 2024). For example, when building AI systems, practition- ers flag safety and privacy as more relevant than the general public does (Jakesch et al. 2022), and developers weight po- liteness over the straightforwardness that the users they build for actually want (Ranjit et al. 2026). These differences are especially pronounced across po- litical ideology. Moral Foundations Theory (MFT) suggests that progressives emphasize care and equality, whereas con- servatives place greater weight on proportionality, authority, loyalty, and purity (Graham, Haidt, and Nosek 2009; Atari et al. 2023). These moral differences shape how people inter- act with AI systems (Brailsford, Vetere, and Velloso 2024). In automated vehicle scenarios, for example, sacrificial pref- erences vary by political ideology (Awad et al. 2018). Taken together, these findings suggest that the ideological compo- sition of a voter pool can affect the values represented in the aggregated preference profile. 2.4 Moral Preference Influenced by Moral Framing Moral framing is the selection and emphasis of particu- lar aspects of a morally charged situation, shaping how people interpret and evaluate it (Brugman 2024; Semetko and Valkenburg 2000). It can influence preferences in two ways: through question framing, which emphasizes partic- ular moral concerns, and through response framing, which constrains the responses participants can express. Question framing can shift preferences by changing how a decision is presented or which considerations participants are prompted to weigh (Feinberg and Willer 2019; Gamson and Modigliani 1989; Rehren and Sinnott-Armstrong 2021). When evaluating an isolated decision, people respond dif- ferently when identical outcomes are framed as lives saved rather than lives lost (McDonald et al. 2021). Socratic ques- tions can also prompt participants to justify and reconsider that individual decision (Torabizadeh, Homayuni, and Moat- tari 2018). Other frames shift attention from the isolated de- cision to the consequences of adopting it as a general policy. Jaques (2025) argues that single-case judgments can invite individual bias and obscure the effects of scaling a choice, and instead proposes asking, “What kind of world would I be creating if this became the AI policy?” (Jaques 2025). The veil of ignorance similarly asks people to evaluate a policy without knowing their own position in the resulting society (Rawls 1971; Huang, Greene, and Bazerman 2019; Huang et al. 2021; Bruno et al. 2024). Such frames have been associated with less self-serving reasoning, greater at- tention to stakeholder perspectives, and reduced political disagreement (Huang, Greene, and Bazerman 2019; Huang et al. 2021; Bruno et al. 2024; Franks and Scherr 2019; Whit- marsh and Corner 2017; Bloemraad, Silva, and Voss 2016). Response framing instead limits which preferences par- ticipants can express. When forced to choose which demo- graphic group to spare, participants appeared to support un- equal treatment; when given an equal-treatment option, most selected it (Bigman and Gray 2020). Research Contribution. Prior research shows that feature scoping, voter sampling, and question framing can each shape moral judgments. Building on this work, we examine the three choices as successive stages of moral AI preference elicitation within a common empirical project. Our contribu- tion is an integrated, cross-context account of how developer choices enter the elicitation process: we identify context- specific feature scopes, estimate ideological differences in feature evaluations, and test the effects of two explicit fram- ing interventions. We conduct these analyses across three deployment contexts that differ in moral stakes and fre- quency of encounter. We then use the combined evidence to develop a sensitivity audit and recommendations for voting- based alignment (§5.4). 3 Methods We study moral preferences and the effects of question fram- ing across three use cases (§3.1), three framing conditions (§3.2), and two phases. Phase 1 identifies candidate moral features (§3.3). Phase 2 evaluates the moral importance of these features under different framing conditions (§3.4). 3.1 Use Case Selection We select three use cases based on two criteria (Figure 1, Step 1): impact (the scale of harm if the system fails or is misused) and temporal exposure (how often people face the dilemma). Together, they cover a range of ethical settings, from high-stakes allocation to everyday AI use: 1) AI kidney allocation (KIDNEY) is a life or death dilemma rooted in distributive justice. It raises questions about fairness and the role of AI in allocating scarce re- sources (Keswani et al. 2025). While common in moral elicitation research (Keswani et al. 2025; Boerstler et al. 2024; Freedman et al. 2020), such cases are rare in daily life (Jaques 2025; Nguyen et al. 2022). 2) AI agents simulating absent workers (WORK) cap- tures a more common, daily setting (Yudkin et al. 2025). It examines remote workers who use AI to simulate ac- tivity. The main concerns are deception, work norms, B World-You-Want A Control OR Could-Be-You C SELECTING QUESTION FRAMING Step 2 Imagine there is an equal chance that you are either the patient / worker / artist OR one of the others Think about the kind of world we would create if your answers shaped AI company policies Step 3 -> Phase 1 IDENTIFYING MORAL FEATURES What features morally should or should not be considered when AI makes decisions about... U1 Criminal recordPatient’s age ... U3 Artist’s motivation Friendship ... OR Step 1 GENerative AI content of the deceased AI KIDNEY allocation AI agents simulating absent WORKers SELECTING VARYING USE CASES U1 U3 Moral harm if it fails or is misused Temporal exposure low high very often very rarely U2 AI KIDNEY allocation GENerative AI content of the deceased U2 Worker’s reliabilityWorkload ... AI agents simulating absent WORKers This should count... strongly in favor moderately in favor slightly in favor not at all slightly against moderately against strongly against +3 +2 +1 -1 -2 -3 0 when the AI decides... Step 4 -> Phase 2 ELICITING MORAL PREFERENCES FOR IDENTIFIED MORAL FEATURES THROUGH QUESTION FRAMING BA or C or Indicate how important the following is when the AI decides... Feature Modeling the influence of ideology on moral preferences: Moral preferences Moral Foundations BAC moderates them shape individuals’ affect Political ideologies Conservative vs. Progressive Feature Figure 1: Overview of our four-step methodology. Step 1. We selected three use cases—AI kidney allocation (KIDNEY), AI-simulated workers (WORK), and generative AI content of the deceased (GEN)—based on harm impact and likelihood of encounter, capturing rare high-stakes to common but less harmful cases. Step 2. We used three framing conditions—(A) Con- trol (baseline), (B) World-You-Want (societal consequences), and (C) Could-Be-You (perspective-taking)—capturing structural and perspective-taking prompts to examine how different moral lenses shape moral preference elicitation. Step 3. In Phase 1, participants identified moral features relevant to AI decision-making for each use case. Step 4. In Phase 2, we elicited moral preferences for these features from a new sample using a 7-point scale (−3 to +3) across the three framings in Step 2. We then modeled how political ideology influences these preferences through moral foundations, with framing condition as a moderator. and whether AI should follow questionable instruc- tions (Boland 2025). 3) Generative AI content of the deceased (GEN) raises issues of consent after death, emotional harm, and com- mercial use (Buben 2025; Danaher and Nyholm 2025). These systems may change how people remember the dead or weaken a person’s unique identity (Danaher and Nyholm 2024; Lazaridis 2025). 3.2 Question Framing Selection We selected three question framing conditions for the elici- tation task in Phase 2: (A) Control uses no additional question framing prompt (Figure 1, Step 2A). (B) World-You-Want draws on Jaques’s structural per- spective (Jaques 2025). It frames the elicitation ques- tion by prompting participants to consider long-term societal effects and AI policy. Participants read: “Think about the kind of world we would create if your answers shaped AI company policies” (Figure 1, Step 2B). (C) Could-Be-You draws on Rawls’s veil of igno- rance (Rawls 1971; Huang, Greene, and Bazerman 2019; Huang et al. 2021). It frames the elicitation ques- tion by prompting participants to take the perspective of any affected stakeholder (Bruno et al. 2024). For ex- ample: “Imagine there is an equal chance that you are either one of the patients” (Figure 1, Step 2C). 3.3 Phase 1: Identifying Moral Features Participants. We recruited participants via Prolific at a rate of at least 8 USD/hour, following sample sizes established in prior work using the same two-phase design (Keswani et al. 2025; Chan et al. 2022) (N KIDNEY = 150, N W ORK = 149, N GEN = 150). We used quota sampling to balance po- litical ideology (conservative, moderate, progressive) within each use case. Full demographic breakdowns, including age, gender, race/ethnicity, and education level, are reported in Table 1 in Supplementary Materials A. Procedure. After consent and a use case introduction, par- ticipants adopted a “moral point of view”. They listed 5 fea- tures that morally should be considered and 5 that should not. They provided a justification for each feature. For fea- tures that should be considered, they also gave example lev- els (e.g., “young” and “old” for “age”) (Figure 1, Step 3; Figure 5 in Supplementary Materials B). Analysis. To identify moral features, we used LLM-assisted coding with GPT-4o-mini. We selected this model for its low cost and high performance on instruction-following tasks (Rytting et al. 2023; Ranjit et al. 2024). We ran it once per participant response at temperature 0 using the prompt in Supplementary Materials C. The LLM extracted feature names from responses, summarized participants’ justifica- tions, and assigned each feature one of four relevance labels (“should”, “should not”, “mixed”, “unclear”). To assess LLM output quality, two authors independently coded a random sample of 100 responses, reviewing both feature names and relevance labels. Agreement between the authors and the LLM was 85% on feature names and 97% on relevance labels. To refine feature names across all outputs, authors conducted five calibration rounds following standard qualitative methods (Charmaz 2015; Oktay 2012). They re- solved disagreements by consensus (with a third author as tie-breaker) and recorded all decisions. This included, for example, merging synonymous codes (e.g., “geographic lo- cation” and “travel time” into “geographic proximity”). Feature inclusion threshold as a developer choice. We computed feature prevalence using binary indicators (1 if mentioned by a participant). Following prior work (Keswani et al. 2025), we retained the top 30 to 35 most prevalent features, excluding those mentioned by fewer than 4 partici- pants (≈2.5% of our sample). For example, in the KIDNEY scenario, taxpayer status and favorite sports team were each mentioned by only 1 participant and thus excluded. We treat this threshold as a design choice within feature scoping that may exclude minority perspectives. 3.4 Phase 2: Eliciting Moral Preferences Under Framing Participants. We recruited 360 US participants (120 per use case) via Prolific at a rate of at least 8 USD/hour. We randomly assigned participants to the Control, World-You- Want, or Could-Be-You condition, with equal representation of conservatives and progressives per condition. Table 2 in Supplementary Materials A reports full demographic break- downs, including age, gender, race/ethnicity, education, AI literacy, religiosity, and area of residence. Procedure. After consent and a use case introduction, par- ticipants evaluated features from Phase 1 (Figure 1, Step 4). Each feature appeared as a contrast (e.g., “younger vs. older patients” for KIDNEY, “grief support vs. commercial use” for GEN). Participants rated moral importance on a 7-point scale from−3 (count strongly against) to +3 (count strongly in favor), with 0 as not count at all (see Figures 6, 7 and 8 in Supplementary Materials B). Unlike pairwise methods, which scale poorly with many features (Boerstler et al. 2024; Keswani et al. 2025), this scale allowed us to evaluate more than 30 features. This design supports analysis of how polit- ical ideology, moral foundations, and framing interact. Participants also completed two validated measures. Moral Foundations used the 36-item MFQ-2 (Atari et al. 2023), which measures six foundations: Care, Equality, Proportionality, Loyalty, Authority, and Purity (Graham, Haidt, and Nosek 2009). Responses used a 5-point scale (1: not at all, 5: extremely well). AI Literacy used a 17-item scale (Tully, Longoni, and Appel 2025) with two compo- nents: technical understanding (10 items) and awareness of limitations and ethics (7 items). Each item was scored as correct or incorrect and scaled to [0, 1]. We use AI literacy as a proxy for prior experience with AI. Analysis. We classified features into six categories based on relevance (M r , p r via t-test vs. 0.5) and direction (M d , p d via t-test vs. 0). We coded relevance as 0 (“not count at all”) or 1 (any other value). We used the raw scale for direction. Categories were: 1) Morally relevant and counting for, if p r < 0.05 and M r > 0.5, and p d < 0.05 and M d > 0. 2) Morally relevant and counting against, if p r < 0.05 and M r > 0.5, and p d < 0.05 and M d < 0. 3) Morally irrelevant, if p r < 0.05 and M r < 0.5. 4) Morally relevant but directionally divisive, if p r < 0.05 and M r > 0.5, and p d ≥ 0.05. 5) Morally divisive, if p r ≥ 0.05. 6) Morally divisive with directional disagreement, if p r ≥ 0.05 and p d ≥ 0.05. We then ran moderated mediation analyses using Hayes’s PROCESS Model 15 with 5,000 bootstrap samples (v4.3 for R) (Hayes 2017). These models test how political ideol- ogy affects moral foundations, how these foundations affect moral importance, and how framing moderates these rela- tionships (see Figure 9 in Supplementary Materials D). We control for sex (1 = female), age, education (1 = pre-college; 2 = college; 3 = advanced), ethnicity (1 = non-Hispanic White), religiosity (1 = not important; 5 = extremely impor- tant), area of residence (1 = rural), and AI literacy. 4 Results 4.1 RQ 1 . Do people consider the same features across different use cases? The features participants considered morally relevant were highly context-specific across KIDNEY, WORK, and GEN (Figure 2). In KIDNEY, participants focused on medical utility, such as patients’ health status (47%) and chance of survival (35%). In WORK, they emphasized worker motivation (e.g., reason for request, 32%) and organizational factors (e.g., company policy, 27%). In GEN, they highlighted ethical safeguards, including deceased’s consent (28%) and in- tended purpose (40%). Full results are reported in Table 3 in Supplementary Materials E, Table 7 in Supplementary Ma- terials F, and Table 11 in Supplementary Materials G. Participants also showed strong agreement on features that should not be considered. These were mostly de- mographic or socioeconomic attributes, including ethnicity (KIDNEY: 66%; WORK: 18%; GEN: 12%), gender (KID- NEY: 64%; WORK: 14%; GEN: 12%), and social status (KIDNEY: 27%; WORK: 12%; GEN: 12%). Age was also rejected in WORK (18%) and GEN (13%), but 82% of par- ticipants in KIDNEY identified it as morally relevant. Table 4 in Supplementary Materials E, Table 8 in Supplementary Materials F, and Table 12 in Supplementary Materials G pro- vide further details. Phase 2 shows that most features reached consensus on moral relevance (KIDNEY: 65%; WORK: 57%; GEN: 70%). These features were typically tied to direct utility. For example, in KIDNEY, a patient with a stronger blood and tissue match with a donor was judged in favor (M = 2.42, SD = 0.95, p r < .001, p d = .001), whereas greater responsibility for one’s illness was judged against (M = −1.10, SD = 1.69, p r < .001, p d < .001). Similar patterns appear in WORK and GEN. In WORK, a worker’s emergency situation was judged in favor (M = 1.29, SD = 1.97, p r < .001, p d < .001), while disrupted workplace operations was judged against (M = −0.84, SD = 2.30, p r < .001, p d = .003). In GEN, requests for grief support or memorial purposes (M = 1.88, SD = 1.49, p r < .001, p d < .001) or those that honored the de- ceased (M = 2.18, SD = 1.33, p r < .001, p d < .001) were judged in favor. Despite this agreement, many features remained divisive (KIDNEY: 31%; WORK: 43%; GEN: 30%). These features often had unclear or mixed implications. Participants agreed they were relevant but disagreed on direction, for or against. Examples include more underlying medical conditions in KIDNEY (M =−0.18, SD = 1.88, p r < .001, p d = 1.0), Higher social status When the AI decides who will receive the kidney transplant, it should consider whether the patient is/has... 020406080100 Stronger blood and tissue match Higher chance of organ acceptance Spent longer on the waiting list Longer expected lifespan Expected full recovery Healthier expected lifestyle Greater kidney failure Clearer documented consent First-time transplant recipient Expected faster recovery Better expected adherence to treatment More severe medical condition Younger More urgent health decline Better health condition Lower alcohol consumption Higher body weight History of drug misuse Higher tobacco use History of inhalant abuse Greater responsibility for own illness Stronger ethnic match advantage More caregiving dependents More financial dependents Citizen/legal resident Geographically closer to the transplant center Stronger social support system Better financial coverage Religious objections to transplant More underlying medical conditions Criminal record This feature should count... moderately against strongly against strongly for +3 moderately for +2 slightly for +1 slightly against -1-2-30 for nothing Morally relevant and counting against Morally irrelevant Morally relevant and counting against When the AI decides whether to grant the request to simulate work-related activity when the worker is not present, it should consider whether the worker is/has... % of participants020406080100 Providing a genuine reason Emergency situation Using AI that aligns with the company's interest Doing more work Compliant with company policies Caregiving responsibilities Health problems Demonstrating reliability Demonstrating productivity Using AI that reports actions to the company Performing simulatable tasks Showing acceptable workplace behavior Inactive briefly (<1 hour) Facing tight deadlines Generating higher profit Putting employer at risk Misusing tools for fraudulent purposes Sharing sensitive data Disrupts workplace operations Paid but not doing the work Treated unfairly Having high morale Able to request time off In a healthy work environment External contractor Doing low-impact work New to company Using highly autonomous AI Doing task for first time Not seeking manager approval At risk of firing Receiving significant benefits Doing another activity unrelated to work Undermining social/professional norms Increasing coworkers’ workload When the AI decides whether to grant the request to produce a video of the deceased, it should consider whether... % of participants020406080100 The video is legally compliant The artist has family’s consent The video honors the deceased The artist has deceased’s explicit consent The artist has other relevant party’s consent The artist is respectful to the deceased The video is for grief support/memorial purpose The video is respectful of the deceased’s tradition The deceased is depicted non-sexually The artist is a close friend/relative of the deceased The video is for non-commercial purpose The video aligns with deceased’s religion The artist has more understandable motivation The video accurately represents the deceased The video keeps sensitive information private The video is respectful of community norms The deceased has uncontroversial cause of death The deceased is public or historical figure The video is intended for adults The video has wider emotional impact The client has high mental health risk The artist is underage The video has risk of misrepresenting the deceased The video may cause potential distress to friend/family The deceased has a criminal record The video is intended for public sharing The video is easy to create The artist is working for oneself The video is about an unmarried deceased The deceased died recently (<100 yrs) % of participants Morally relevant and counting for Morally relevant but counting either for or against Morally divisive Morally divisive and counting either for or against Morally relevant and counting for Morally divisive Morally relevant but counting either for or against Morally divisive and counting either for or against Morally relevant and counting for Morally divisive Morally relevant but counting either for or against Morally divisive and counting either for or against KIDNEY WORK GEN Figure 2: Distribution of ratings in Phase 2 for the AI kidney allocation (KIDNEY), AI-simulated workers (WORK), and Generative AI content of the deceased (GEN) use cases. Results were aggregated across condi- tions, and features were ordered by moral feature category and by mean rating from highest to lowest. increasing coworkers’ workload in WORK (M = −0.57, SD = 2.15, p r < .001, p d = 1.0), and risk of misrepre- sentation in GEN (M = −0.45, SD = 2.75, p r < .001, p d = 0.36). Table 5 in Supplementary Materials E, Table 9 in Supplementary Materials F, and Table 13 in Supplemen- tary Materials G provide further details. 4.2 RQ 2 . Do moral preferences vary with po- litical ideology? Preferences differed by political ideology for roughly one- third of features, with some differences reversing direction. Consequently, changing the ideological composition of an aggregated sample could change the preference profile. In the control condition, ideological differences ap- peared across all use cases. In KIDNEY, conserva- tives favored patients with expected full recovery (b = 1.08, 95% CI [0.12, 2.03]) and penalized those with more underlying medical conditions (b = −1.71, 95% CI [−3.20,−0.23]). In WORK, conservatives op- posed requests from workers who were being paid but not working (b = −2.79, 95% CI [−5.58,−0.00]). In GEN, they were less likely to approve requests when the video risked misrepresenting the deceased (b = −2.91, 95% CI [−5.62,−0.20]). Table 6 in Supplementary Mate- rials E, Table 10 in Supplementary Materials F, and Table 14 in Supplementary Materials G provide further details. 4.3 RQ 3 . Does question framing change moral preferences? The third developer choice (question framing) systemat- ically shifts preferences. It can reduce some ideological gaps while increasing others. The magnitude of these shifts reaches up to one full scale point (Figure 3). Both Question Framings. In KIDNEY, disagreement about patients with more underlying medical conditions decreased under both framings (World-You-Want: b = −0.56, 95% CI [−2.21, 1.08]; Could-Be-You: b = 0.61, 95% CI [−0.81, 2.02]; Figure 3C). In WORK, conservatives’ opposition to workers being paid but not working was also reduced (World-You-Want: b = −0.43, 95% CI [−2.96, 2.10]; Could-Be-You: b = 0.01, 95% CI [−2.41, 2.43]; Figure 3E). In GEN, both framings reduced disagreement about videos that risk misrepresenting the deceased (World-You- Want: b = 1.06, 95% CI [−1.23, 3.35]; Could-Be-You: b = −1.88, 95% CI [−4.40, 0.63]; Figure 3I). Could-Be-You Question Framing. In KIDNEY, conser- vatives’ preference for expected full recovery (present in Control and World-You-Want) disappeared under Could-Be- You (p = .95). This change was associated with reduced influence of purity (b = −0.51, 95% CI [−1.19,−0.03]; IM M = −0.74, 95% CI [−1.68,−0.07]; Figure 3B). The same condition also introduced tension between moral foundations. Authority predicted opposition to pa- tients with more underlying conditions (b = −1.30, 95% CI [−2.83,−0.23]), while purity predicted support (b = 0.87, 95% CI [0.04, 2.08]; Figure 3C). No com- parable effects appeared in WORK. In GEN, however, 3) AN AI AGENT SHOULD GRANT THE REQUEST OF AN ARTIST WHEN... Morally relevant and counting for Morally relevant and counting for Morally divisive ...the video is for grief support or memorial purpose ...the video honors the deceased...the video has a risk of misrepresenting the deceased Could-Be-You World-You-Want Control Question framing Direct effect of question framing Indirect effect via authority Indirect effect via purity Reference (progressives’ preference) Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor -6-4-20246-6-4-20246-6-4-20246 GHI Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor 2) AN AI AGENT SHOULD GRANT THE REQUEST OF A REMOTE WORKER WHO IS... Morally relevant and counting forMorally relevant and counting forMorally divisive ...experiencing health problems...paid but not working...receiving significant benefits Could-Be-You World-You-Want Control Question framing Direct effect of question framing Indirect effect via authority Indirect effect via loyalty Reference (progressives’ preference) Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor -6-4-20246-6-4-20246-6-4-20246 DEF Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor 1) AI SHOULD ALLOCATE A KIDNEY TO A PATIENT WHO HAS... Morally relevant and counting forMorally relevant and counting forMorally divisive ...healthier lifestyle...higher chance of recovery...more underlying medical conditions Could-Be-You World-You-Want Control Question framing Direct effect of question framing Indirect effect via authority Indirect effect via purity Reference (progressives’ preference) Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor -6-4-20246-6-4-20246-6-4-20246 ABC Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor CONSERVATIVES ARE MORE LIKELY THAN PROGRESSIVES TO CONSIDER THAT... Figure 3: Moderated mediation results for our three use cases: KIDNEY, WORK, GEN (Phase 2). Note: *The condi- tional direct effect for conservatives differed significantly from that for progressives. ‡ The index of moderated mediation was statistically significant (bootstrapped 95% confidence interval excluding zero). Additional results are shown in Figure 10 in Supplementary Materials E and Figure 11 in Supplementary Materials G. Could-Be-You increased conservatives’ support for requests related to grief and memorial purposes (b = 2.32, 95% CI [0.94, 3.71], p < .02; Figure 3G). Response-time differences provide supplementary evi- dence that the framing changed how participants engaged with the task, although response time alone cannot iden- tify the underlying cognitive process. In KIDNEY, partici- pants in Could-Be-You took longer to respond (EM M = 12.86, 95% CI [11.20, 14.77]) than those in World-You- Want (EM M = 9.10, p < .001) or Control (EM M = 9.43, p = .008). 1 World-You-Want Question Framing. In KIDNEY, conser- vatives favored healthier lifestyles more than progressives 1 The ANOVA used log-transformed values. Reported EMMs are back-transformed. under World-You-Want (b = 1.18, 95% CI [0.12, 2.25]), but not in Control (b = 0.51, 95% CI [−0.45, 1.47]) or Could- Be-You (b = 0.53, 95% CI [−0.39, 1.44]; Figure 3A). In WORK, World-You-Want increased the role of loyalty. Conservatives with stronger loyalty values showed greater support for workers experiencing health problems or receiv- ing benefits (b = 0.98–0.99, IM M = 1.16–1.18; Figure 3D, F). At the same time, authority predicted opposition, creating tension between foundations. In GEN, World-You-Want reduced this type of tension. In the Control condition, authority supported requests that honor the deceased, while purity opposed them. Under World-You-Want, neither foundation had a significant effect (Figure 3H). 5 Discussion We discuss our three main findings (§5.1–§5.3), trans- late them into recommendations for voting-based alignment (§5.4), and reflect on the limits of aggregation as an align- ment method (§5.5). 5.1 Finding 1: Moral Feature Relevance Is Context-Specific The first developer choice in the elicitation pipeline (which features are put up for a vote) is often treated as a pre- processing step, but our results show it functions as a per- deployment moral decision: the relevant feature sets differed across the three contexts. In KIDNEY, participants focused on medical utility, espe- cially the chance of survival, which reflects the distributive justice structure of the task (Keswani et al. 2025). In WORK, they emphasized worker motivation and organizational fac- tors, which align with concerns about deception, account- ability (Yudkin et al. 2025), and the legitimacy of com- mands (Boland 2025). In GEN, attention shifted to the pur- pose of the video and the deceased’s consent, which mirrors ongoing debates about posthumous representation (Buben 2025; Danaher and Nyholm 2025). Features became divisive when their implications were unclear. In KIDNEY, participants disagreed on whether un- derlying conditions should prioritize patients based on need or penalize them due to poor prognosis. This reflects a tension between utility and equity (Keswani et al. 2025). In WORK, some participants justified AI actions when harm seemed minimal (Blanken, Van De Ven, and Zee- lenberg 2015), while others rejected such actions as inher- ently wrong. In GEN, disagreement centered on the risk of misrepresentation, reflecting a conflict between dignity and harm prevention (Buben 2025; Danaher and Nyholm 2025). Across all use cases, participants generally rejected socio- demographic features as morally relevant unless they were necessary for the decision (e.g., age in KIDNEY). This sug- gests that participants view such features as discriminatory unless justified (Keswani et al. 2025). Because feature sets differed across contexts, developers delimit what the elicita- tion process can express before votes are collected. 5.2 Finding 2: Voter Pool Composition Can Change Aggregated Preferences Preferences differed by political ideology for roughly one- third of features, with some differences reversing direction. Consequently, changing the ideological composition of an aggregated sample could change the preference profile. These differences reflect distinct value priorities. In KID- NEY, conservatives favored allocating resources to patients with higher chances of recovery and opposed allocating them to patients with more underlying conditions. This pattern aligns with proportionality-based reasoning, while progressive responses reflect stronger emphasis on equal- ity (Graham, Haidt, and Nosek 2009). In WORK, conserva- tives more strongly opposed requests to simulate work activ- ity when employees were being paid but not working. This suggests greater emphasis on loyalty and contractual obliga- tion (Graham, Haidt, and Nosek 2009; Koleva et al. 2012). In GEN, conservatives more strongly opposed requests that risk misrepresenting the deceased, while progressives were more permissive. This pattern may reflect broader political differences in expressive norms (Kfrerer, Bell, and Schermer 2021), and ongoing debates about digital remains (Danaher and Nyholm 2024; Lazaridis 2025; Buben 2025). These results show that the aggregated preference profile reflects the values of the sampled population. 5.3 Finding 3: Question Framing Is a Silent Policy Lever The third developer choice (how questions are framed) sys- tematically shifts preferences and changes how moral foun- dations are associated with participants’ judgments. In our study, framing reduced ideological differences in some cases but increased them in others. Prior research suggests that framing effects may occur without participants recognizing their influence (Jakesch et al. 2023), which makes framing a silent policy lever rather than a neutral tool. Could-Be-You framing. Could-Be-You reduced several ideological gaps and was associated with longer response times in KIDNEY. Longer response times are consistent with greater processing effort, although they do not estab- lish more reflective or deliberative reasoning. In KIDNEY, the framing reduced disagreement about vulnerable patients, including those with poor prognoses. However, the same framing increased disagreement in GEN. Conservatives showed greater support for memorial requests than progressives. This pattern is consistent with perspective-taking increasing the salience of bereaved indi- viduals’ needs, although we did not directly measure empa- thy or perceived salience (Rawls 1971; Bruno et al. 2024). These results show that perspective-taking does not consis- tently produce convergence. World-You-Want framing. This framing also shifted pref- erences, but in different ways. In KIDNEY, it increased con- servatives’ preference for patients perceived as more respon- sible (e.g., those with healthier lifestyles) (Sylvester and Haeder 2025). In WORK, it emphasized loyalty, leading conservatives to prioritize in-group protection over honesty. These patterns suggest that this framing is not neutral. It can reinforce existing biases and political identities (Jaques 2025; Mittal and De Choudhury 2023). Response times were shorter than under Could-Be-You, but this difference does not identify the depth or quality of participants’ reasoning. Overall, framing cannot be assumed to produce consensus. 5.4 A Sensitivity Audit for Voting-Based Alignment Prior work sets out broad criteria for evaluating participatory machine learning systems, including stakeholder represen- tation, elicitation, conflict resolution, and evaluation (Feffer et al. 2023b). Our findings do not replace these established criteria. Instead, they adapt part of this framework to voting- based moral alignment by defining pipeline-specific sensi- tivity tests. Each stage should be documented, stress-tested under reasonable alternatives, and flagged when a plausible design choice changes the resulting policy. SENSITIVITY AUDIT FOR VOTING-BASED ALIGNMENT SYSTEMS •Define the intended decision and the stakeholders affected by it. •Justify why preference aggregation is appropriate for this decision. •Identify decisions that should remain subject to human or institutional oversight. 1.Role of preference aggregation (Why aggregate preferences here?) 2. Feature scoping (What is put to a vote?) •Report how candidate features were generated, coded, included, and excluded. •Document excluded and minority-proposed features. •Test whether reasonable alternative thresholds or scoping procedures change the result. 3. Voter pool composition (Whose values are represented?) •Define and justify the target population. •Report recruitment, composition, weighting, attrition, and sampling limitations. •Test whether plausible alternative voter pool compositions change the result. 4. Question framing (How are preferences elicited?) •Provide the exact scenario, prompt, response scale, ordering, and presentation format. •Test substantively plausible alternative framings or response formats. •Report whether conclusions are framing-sensitive; do not describe any wording as neutral. 5. Moral disagreement (How is disagreement handled?) •Identify disagreement within and across stakeholder groups. •Report how the aggregation rule treats morally divisive features. •Test whether plausible alternative aggregation rules change the conclusion. 6. Translation into system behavior (Does the system implement the elicited policy?) •Describe how aggregated preferences are encoded into rules, reward models, fine-tuning, or other system components. •Evaluate whether system outputs reproduce the disclosed preference policy across representative and edge cases. •Report unresolved mismatches and revise the implementation when necessary. Figure 4: A proposed sensitivity audit for voting- based alignment systems. The checklist identifies pipeline choices that developers should document and test under plausible alternatives. The accompanying text defines con- sequential changes, and explains how developers should re- port and address them. Figure 4 presents an initial, empirically grounded sensi- tivity audit of the elicitation pipeline rather than a general checklist for participation. The recommendations on feature scoping, voter sampling, and question framing draw directly on our results. The recommendations on the role of aggrega- tion, moral disagreement, and system implementation draw on the broader participatory machine learning and alignment literature (Gabriel 2020; Feffer et al. 2023b; Cooper and Zafiroglu 2024; Zhi-Xuan et al. 2024; Kneer and Viehoff 2025; Baum and Slavkovik 2025). Our findings motivate specific sensitivity tests at each em- pirical stage. Differences in feature relevance across con- texts show why developers should document and stress-test feature scopes. Ideology-linked differences and direction re- versals show why they should test alternative voter pool compositions. Framing effects show why they should test alternative question formulations. Together, these checks show how strongly elicited preferences depend on pipeline choices before those preferences are used as the basis for a public or collective policy. Recommendation 1: Define and justify the role of pref- erence aggregation. Developers should state which deci- sion the system will make or inform, which stakeholders it will affect, and why preference aggregation is appropri- ate for that decision. They should document intended uses, foreseeable unintended uses and harms, and the authority re- tained by developers or deployers. They should also iden- tify cases in which an aggregated preference should not de- termine system behavior. When the legitimacy of aggrega- tion is contested, developers should supplement voting with stakeholder deliberation or other forms of institutional and normative oversight (Feffer et al. 2023b). Recommendation 2: Make feature scoping auditable. Developers should report how they generated and coded can- didate features, the complete candidate set, the inclusion and exclusion criteria, the thresholds applied, and the features excluded from the final elicitation. They should also test al- ternative thresholds or scoping procedures. If an alternative adds or removes a feature, reverses the direction of a pref- erence, or changes its relevance or disagreement classifica- tion, developers should report the result as sensitive to fea- ture scope. They should then justify the selected scope and explain how they treated rare or minority-proposed features. Recommendation 3: Audit voter pool composition. Developers should define the target population and explain why its members should be represented. They should re- port recruitment procedures, demographic and ideological composition, exclusion and attrition patterns, sampling lim- its, and any weighting rules (Kallina, Bohn ́ e, and Singh 2025; Vereschak et al. 2024; Zhang 2024). They should re- port group-specific results alongside the aggregate and re- calculate estimates under plausible alternative voter pool compositions. If the direction or classification of a prefer- ence changes, or if its magnitude changes by more than the specified threshold, developers should describe the result as constituency-dependent rather than as a public preference. Recommendation 4: Audit question framing. Developers should disclose the complete scenario, exact prompt word- ing, response options, presentation order, and wording of any baseline condition. They should test plausible alterna- tive framings or response formats and define consequential change in advance. If an alternative framing changes the di- rection or classification of a feature, or produces a change that exceeds the specified threshold, developers should re- port the result as framing-sensitive. They should justify the selected wording and should not describe any elicitation condition as normatively neutral (Jakesch et al. 2023; Rader, Cotter, and Cho 2018). Recommendation 5: Preserve and document moral dis- agreement. Developers should report full response distri- butions and identify features whose relevance or direction varies across groups (Keswani et al. 2025) or within individ- uals (Boerstler et al. 2024). They should explain how the aggregation rule treats such disagreement and test whether plausible alternative rules produce different conclusions. If averaging conceals substantial disagreement or the result- ing policy depends on the selected rule, developers should preserve disaggregated results. They should justify any de- cision to override, average, or otherwise resolve disagree- ment. Morally divisive cases may require deliberation or in- dependent normative constraints rather than simple majority aggregation (Noothigattu et al. 2018; Feffer et al. 2023b). Recommendation 6: Validate the translation into system behavior. Developers should describe how they translate ag- gregated preferences into rules, reward models, fine-tuning objectives, or other system components. They should test whether system outputs follow the disclosed preference pol- icy across representative cases, edge cases, and morally divi- sive cases. A systematic mismatch between elicited prefer- ences and system behavior should lead developers to revise the implementation or disclose the unresolved limitation. Because our study examined elicitation rather than model deployment, this recommendation draws on the broader lit- erature and requires further empirical validation. 5.5 Residual Limitations Our audit recommendations are necessary but not sufficient. Even with full transparency, aggregation remains a norma- tive choice that cannot resolve deep value conflict. Pluralism and competing values. Our results show that dif- ferent groups prioritize different moral values. Framing can widen these differences. A single aggregated policy there- fore risks privileging one set of values. Alignment in plu- ralistic settings requires explicit trade-offs and transparency, not simple averaging. No universally optimal framing. Could-Be-You often pro- motes concern for the least advantaged, while World-You- Want can reinforce existing biases. However, neither fram- ing works in all cases. When the least advantaged group is contested, perspective-taking can increase disagreement. No framing reliably produces consensus. Limits of majority aggregation. Even with full disclosure, aggregation risks a “tyranny of the majority” (Feffer, Hei- dari, and Lipton 2023a). Public judgments are shaped by framing and identity (Rehren and Sinnott-Armstrong 2021; Graham, Haidt, and Nosek 2009), and aggregation can blur rather than resolve value conflicts. Safeguards are therefore necessary to protect minority perspectives (Tanksley et al. 2025). Voting-based alignment cannot, on its own, deliver fair or transparent outcomes. 5.6 Limitations and Future Work Our study comes with six limitations. First, Likert scales measure importance rather than forced-choice decisions; future work should validate these findings using pairwise methods. Second, our design does not capture participants’ reasoning; qualitative approaches, such as think-aloud pro- tocols, could address this gap. Third, the feature inclusion threshold in Phase 1 is a design choice that may exclude mi- nority views; future work should explore alternative thresh- olds. Fourth, we use AI literacy as a proxy for experience. Fifth, our use cases span different harm types and frequency but do not exhaust the space of applications. Sixth, our US- based sample limits generalizability across cultural contexts. Although our models account for several demographic char- acteristics of the sample, demographic and intersectional differences were beyond the scope of this study; future work should examine these differences in more diverse popula- tions and non-Western cultural contexts (Zoshak and Dew 2021; Atari et al. 2023). 6 Conclusion We examined three points at which upstream design choices shape moral preference elicitation: feature scoping, voter sampling, and question framing. Across three deployment contexts, the analyses show that elicited preferences are contingent on how each stage is configured. First, feature scoping is context-dependent. The features participants con- sider morally relevant differ across use cases, which limits transfer across domains. Second, voter sampling shapes out- comes. Differences in political ideology lead to systematic shifts in judgments and, in some cases, reverse the direction of preferences. Third, question framing alters judgments and the estimated relationships between moral foundations and those judgments. Framing can reduce some disagreements but amplify others. These results show that moral preference elicitation does not produce a single, stable set of values. Instead, the output depends on upstream design choices made by developers. What appears as “public morality” in a deployed system is, in part, a function of how the pipeline is constructed. The main takeaway is that voting-based alignment does not remove human judgment from AI systems; it relocates it to design choices about features, samples, and framing. Sys- tems that rely on preference aggregation should treat elicited preferences as contingent rather than fixed. This requires disclosing how features are defined, participants sampled, and questions framed, and assessing how sensitive outputs are to each. Without such disclosure, claims of neutrality or fairness are difficult to evaluate. Our findings suggest that alignment cannot rely on ag- gregation alone. If preferences shift with context, popula- tion, and framing, then alignment also requires normative judgment and institutional design. This may include struc- tured deliberation, stakeholder representation, and mecha- nisms for accountability. Future research should extend this analysis to additional domains, test alternative elicitation methods that reduce sensitivity to framing and sampling, and explore how to integrate empirical preferences with norma- tive frameworks in system design. References Arzberger, A.; Buijsman, S.; Lupetti, M. L.; Bozzon, A.; and Yang, J. 2025. Nothing Comes Without Its World - Practi- cal Challenges of Aligning LLMs to Situated Human Values Through RLHF. In Proceedings of the AAAI/ACM Confer- ence on AI, Ethics, and Society (AIES), 61–73. Atari, M.; Haidt, J.; Graham, J.; Koleva, S.; Stevens, S. T.; and Dehghani, M. 2023. Morality Beyond the WEIRD: How the Nomological Network of Morality Varies Across Cul- tures. Journal of Personality and Social Psychology, 125(5): 1157. Awad, E.; Dsouza, S.; Kim, R.; Schulz, J.; Henrich, J.; Shar- iff, A.; Bonnefon, J.-F.; and Rahwan, I. 2018. The Moral Machine Experiment. Nature, 563(7729): 59–64. Baum, K.; and Slavkovik, M. 2025. Aggregation Problems in Machine Ethics and AI Alignment. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), 355–366. Bigman, Y. E.; and Gray, K. 2020. Life and Death Decisions of Autonomous Vehicles. Nature, 579(7797): E1–E2. Blanken, I.; Van De Ven, N.; and Zeelenberg, M. 2015. A Meta-Analytic Review of Moral Licensing. Personality and Social Psychology Bulletin, 41(4): 540–558. Bloemraad, I.; Silva, F.; and Voss, K. 2016. Rights, Eco- nomics, or Family? Frame Resonance, Political Ideology, and the Immigrant Rights Movement. Social Forces, 94(4): 1647–1674. Boerstler, K.; Keswani, V.; Chan, L.; Borg, J. S.; Conitzer, V.; Heidari, H.; and Sinnott-Armstrong, W. 2024. On the Stability of Moral Preferences: A Problem With Computa- tional Elicitation Methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), volume 7, 156–167. Boland, J. 2025. Moral Responsibility or Obedience: What Do We Want from AI? arXiv:2507.02788. Brailsford, J.; Vetere, F.; and Velloso, E. 2024. Exploring the Association Between Moral Foundations and Judgements of AI Behaviour. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 1–15. Brugman, B. C. 2024. How the Effects of Emphasizing Ethics Are Examined: A Systematic Review of Moral Fram- ing Experiments. Annals of the International Communica- tion Association, 48(4): 436–455. Bruno, G.; Spoto, A.; Sarlo, M.; Lotto, L.; Marson, A.; Cellini, N.; and Cutini, S. 2024. Moral Reasoning Behind the Veil of Ignorance: An Investigation Into Perspective- Taking Accessibility in the Context of Autonomous Vehi- cles. British Journal of Psychology, 115(1): 90–114. Buben, A. 2025. Digital Replacement of the Dead: A Legit- imate Worry? Philosophy & Technology, 38(3): 90. Chan, L.; Schaich Borg, J.; Conitzer, V.; Wilkinson, D.; Savulescu, J.; Zohny, H.; and Sinnott-Armstrong, W. 2022. Which Features of Patients Are Morally Relevant in Ven- tilator Triage? A Survey of the UK Public. BMC Medical Ethics, 23(1): 33. Chan, L.; Sinnott-Armstrong, W.; Borg, J. S.; and Conitzer, V. 2024. Should Responsibility Affect Who Gets a Kidney? Responsibility and Healthcare, 35. Charmaz, K. 2015. Grounded Theory. In Smith, J. A., ed., Qualitative Psychology: A Practical Guide to Research Methods, 53–84. Sage Publications. Cooper, N.; and Zafiroglu, A. 2024. From Fitting Participa- tion to Forging Relationships: The Art of Participatory ML. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI). Dahl, Y.; and Svanæs, D. 2020.Facilitating Democ- racy: Concerns From Participatory Design With Asymmet- ric Stakeholder Relations in Health Care. In Proceedings of the ACM Conference on Human Factors in Computing Sys- tems (CHI), 1–13. Danaher, J.; and Nyholm, S. 2024. Digital Duplicates and the Scarcity Problem: Might AI Make Us Less Scarce and Therefore Less Valuable? Philosophy & Technology, 37(3): 106. Danaher, J.; and Nyholm, S. 2025. The Ethics of Person- alised Digital Duplicates: A Minimally Viable Permissibil- ity Principle. AI and Ethics, 5(2): 1703–1718. Dewitt, B.; Fischhoff, B.; and Sahlin, N.-E. 2019. ’Moral Machine’ Experiment Is No Basis for Policymaking. Nature, 567(7746): 31–31. Etienne, H. 2021. The Dark Side of the ’Moral Machine’ and the Fallacy of Computational Ethical Decision-Making for Autonomous Vehicles. Law, Innovation and Technology, 13(1): 85–107. Feffer, M.; Heidari, H.; and Lipton, Z. C. 2023a. Moral Ma- chine or Tyranny of the Majority? In Proceedings of the AAAI Conference on Artificial Intelligence, 5, 5974–5982. Feffer, M.; Skirpan, M.; Lipton, Z.; and Heidari, H. 2023b. From Preference Elicitation to Participatory ML: A Criti- cal Survey & Guidelines for Future Research. In Proceed- ings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), 38–48. Feinberg, M.; and Willer, R. 2019.Moral Reframing: A Technique for Effective and Persuasive Communication Across Political Divides. Social and Personality Psychol- ogy Compass, 13(12): e12501. Franks, A. S.; and Scherr, K. C. 2019. Economic Issues Are Moral Issues: The Moral Underpinnings of the Desire To Reduce Wealth Inequality. Social Psychological and Per- sonality Science, 10(4): 553–562. Freedman, R.; Borg, J. S.; Sinnott-Armstrong, W.; Dicker- son, J. P.; and Conitzer, V. 2020. Adapting a Kidney Ex- change Algorithm To Align With Human Values. Artificial Intelligence, 283: 103261. Freitas, J. D.; Censi, A.; Smith, B. W.; Lillo, L. D.; Anthony, S. E.; and Frazzoli, E. 2021. From Driverless Dilemmas to More Practical Commonsense Tests for Automated Ve- hicles. Proceedings of the National Academy of Sciences, 118(11): e2010202118. Gabriel, I. 2020. Artificial Intelligence, Values, and Align- ment. Minds and Machines, 30(3): 411–437. Gamson, W. A.; and Modigliani, A. 1989. Media Discourse and Public Opinion on Nuclear Power: A Constructionist Approach. American Journal of Sociology, 95(1): 1–37. Ganguli, D.; Huang, S.; Lovitt, L.; Siddarth, D.; Liao, T.; Askell, A.; Bai, Y.; Kadavath, S.; Kernion, J.; McKinnon, C.; Nguyen, K.; and Durmus, E. 2023. Collective Constitutional AI: Aligning a Language Model with Public Input. Techni- cal report, Anthropic and Collective Intelligence Project. Graham, J.; Haidt, J.; and Nosek, B. A. 2009. Liberals and Conservatives Rely on Different Sets of Moral Foundations. Journal of Personality and Social Psychology, 96(5): 1029. Harris, J. 2020. The Immoral Machine. Cambridge Quar- terly of Healthcare Ethics, 29(1): 71–79. Hayes, A. F. 2017. Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Ap- proach. Guilford Publications. Huang, K.; Bernhard, R. M.; Barak-Corren, N.; Bazerman, M. H.; and Greene, J. D. 2021. Veil-of-Ignorance Reason- ing Mitigates Self-Serving Bias in Resource Allocation Dur- ing the COVID-19 Crisis. Judgment and Decision Making, 16(1): 1–19. Huang, K.; Greene, J. D.; and Bazerman, M. 2019. Veil- of-Ignorance Reasoning Favors the Greater Good. Proceed- ings of the National Academy of Sciences (PNAS), 116(48): 23989–23995. Jakesch, M.; Bhat, A.; Buschek, D.; Zalmanson, L.; and Naaman, M. 2023. Co-Writing With Opinionated Language Models Affects Users’ Views. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 1–15. Jakesch, M.; Buc ̧inca, Z.; Amershi, S.; and Olteanu, A. 2022. How Different Groups Prioritize Ethical Values for Responsible AI. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), 310– 323. Jaques, A. E. 2025. Why the Moral Machine is a Monster. In Robot Law: Volume I, 17–34. Edward Elgar Publishing. Kallina, E.; Bohn ́ e, T.; and Singh, J. 2025. Stakeholder Participation for Responsible AI Development: Disconnects Between Guidance and Current Practice. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), 1060–1079. Keswani, V.; Conitzer, V.; Sinnott-Armstrong, W.; Nguyen, B. K.; Heidari, H.; and Schaich Borg, J. 2025. Can AI Model the Complexities of Human Moral Decision-making? A Qualitative Study of Kidney Allocation Decisions. In Pro- ceedings of the ACM Conference on Human Factors in Com- puting Systems (CHI), 1–17. Kfrerer, M. L.; Bell, E.; and Schermer, J. A. 2021. The Pol- itics of Being Funny: Humor Styles, Trait Humorousness, and Political Orientations. Personality and Individual Dif- ferences, 182: 111073. Kim, T.; and Peng, W. 2025. Do We Want AI Judges? The Acceptance of AI Judges’ Judicial Decision-Making on Moral Foundations. AI & Society, 40(5): 3683–3696. Kneer, M.; and Viehoff, J. 2025. The Hard Problem of AI Alignment: Value Forks in Moral Judgment. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), 2671–2681. New York, NY, USA: ACM. ISBN 9798400714825. Koleva, S. P.; Graham, J.; Iyer, R.; Ditto, P. H.; and Haidt, J. 2012. Tracing the Threads: How Five Moral Concerns (Es- pecially Purity) Help Explain Culture War Attitudes. Jour- nal of Research in Personality, 46(2): 184–194. Lazaridis, C. 2025. Digital Doppelg ̈ angers: They Will Mat- ter When Conscious. The American Journal of Bioethics, 25(2): 111–112. McDonald, K.; Graves, R.; Yin, S.; Weese, T.; and Sinnott- Armstrong, W. 2021. Valence Framing Effects on Moral Judgments: A Meta-Analysis. Cognition, 212: 104703. Mittal, S.; and De Choudhury, M. 2023. Moral Framing of Mental Health Discourse and Its Relationship to Stigma: A Comparison of Social Media and News. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 1–19. Nguyen, T. D.; Lyall, G.; Tran, A.; Shin, M.; Carroll, N. G.; Klein, C.; and Xie, L. 2022. Mapping Topics in 100,000 Real-Life Moral Dilemmas. In Proceedings of the AAAI In- ternational Conference on Web and Social Media (ICWSM), volume 16, 699–710. Noothigattu, R.; Gaikwad, S.; Awad, E.; Dsouza, S.; Rah- wan, I.; Ravikumar, P.; and Procaccia, A. 2018. A Voting- Based System for Ethical Decision Making. In Proceedings of the AAAI Conference on Artificial Intelligence, 1, 1587– 1594. Oktay, J. S. 2012. Grounded Theory. Oxford University Press. Rader, E.; Cotter, K.; and Cho, J. 2018. Explanations As Mechanisms for Supporting Algorithmic Transparency. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 1–13. Ranjit, J.; Joshi, B.; Dorn, R.; Petry, L.; Koumoundouros, O.; Bottarini, J.; Liu, P.; Rice, E.; and Swayamdipta, S. 2024. OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants. arXiv:2406.14883. Ranjit, J.; Zhou, K.; Swayamdipta, S.; and Quercia, D. 2026. Are We Automating the Joy Out of Work? Designing AI to Augment Work, Not Meaning. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI). ACM. ISBN 9798400722783. Rawls, J. 1971. A Theory of Justice. The Belknap. Rehren, P.; and Sinnott-Armstrong, W. 2021. Moral Framing Effects Within Subjects. Philosophical Psychology, 34(5): 611–636. Rytting, C. M.; Sorensen, T.; Argyle, L.; Busby, E.; Fulda, N.; Gubler, J.; and Wingate, D. 2023.Towards Coding Social Science Datasets with Language Models. arXiv:2306.02177. Sadek, M.; Calvo, R. A.; and Mougenot, C. 2023. Designing Value-Sensitive AI: A Critical Review and Recommenda- tions for Socio-Technical Design Processes. AI and Ethics, 4(4): 949–967. Salloch, S.; W ̈ ascher, S.; Vollmann, J.; and Schildmann, J. 2015. The Normative Background of Empirical-Ethical Re- search: First Steps Towards a Transparent and Reasoned Ap- proach in the Selection of an Ethical Theory. BMC Medical Ethics, 16(1). Schuessler, D. 2023. The Probability Problems of the Moral Machine Experiment. AI and Ethics, 4(2): 501–510. Semetko, H. A.; and Valkenburg, P. M. 2000. Framing Eu- ropean Politics: A Content Analysis of Press and Television News. Journal of Communication, 50(2): 93–109. Simson, J.; Draxler, F.; Mehr, S.; and Kern, C. 2025. Pre- venting Harmful Data Practices by Using Participatory Input to Navigate the Machine Learning Multiverse. In Proceed- ings of the ACM Conference on Human Factors in Comput- ing Systems (CHI), 1–30. Sylvester, S. M.; and Haeder, S. F. 2025. Who’s to Blame? How Recipient Deservingness Influences Attitudes About Access to the Organ Transplants. Social Science Quarterly, 106(3): e70021. Tanksley, T.; Smith, A. D.; Sharma, S.; and Huff Jr, E. W. 2025. “Ethics Is Not Neutral”: Understanding Ethical and Responsible AI Design From the Lenses of Black Youth. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 1–20. Torabizadeh, C.; Homayuni, L.; and Moattari, M. 2018. Im- pacts of Socratic Questioning on Moral Reasoning of Nurs- ing Students. Nursing Ethics, 25(2): 174–185. Tully, S. M.; Longoni, C.; and Appel, G. 2025. Lower Arti- ficial Intelligence Literacy Predicts Greater AI Receptivity. Journal of Marketing. Vereschak, O.; Alizadeh, F.; Bailly, G.; and Caramiaux, B. 2024. Trust in AI-Assisted Decision Making: Perspectives From Those Behind the System and Those for Whom the Decision Is Made. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 1–14. Whitmarsh, L.; and Corner, A. 2017.Tools for a New Climate Conversation: A Mixed-Methods Study of Lan- guage for Public Engagement Across the Political Spectrum. Global Environmental Change, 42: 122–135. Yudkin, D. A.; Goodwin, G. P.; Reece, A.; Gray, K.; and Bhatia, S. 2025. A Large-Scale Investigation of Everyday Moral Dilemmas. PNAS Nexus, 4(5): pgaf119. Zhang, A. 2024.Empowering and Centering Impacted Stakeholders in AI Design. In Companion Publication of the ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW), 50–53. Zhi-Xuan, T.; Carroll, M.; Franklin, M.; and Ashton, H. 2024. Beyond Preferences in AI Alignment. Philosophical Studies, 182(7): 1813–1863. Zoshak, J.; and Dew, K. 2021. Beyond Kant and Bentham: How Ethical Theories Are Being Used in Artificial Moral Agents. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 1–15. Endmatter Statements Researcher Positionality Statement Our team consists of three members (two men, one woman) from East Asia, Southern Europe, and Eastern Europe, now based across Europe and North America. We bring diverse ethnic, religious, and cultural backgrounds and expertise spanning social and cognitive psychology, Responsible AI, HCI, Computer Science, and NLP across academic and in- dustry research settings. Having lived across different po- litical systems, we acknowledge that our positionality may have influenced various aspects of our research, including our choice of use cases, design of question framing interven- tions, and interpretation of results across political groups. We recognize the importance of including a broader range of voices from academia, industry, and underrepresented re- gions and communities. Ethical Considerations Statement Our work raises four ethical considerations. First, the study was conducted with our organization’s approval. We adhered to established guidelines for human subjects research, en- suring that no personal identifiers were collected, personal information was removed from all data, and access was re- stricted to the research team. Participants were recruited via Prolific, compensated at a rate of at least 8 USD/hour, and free to withdraw at any time. Second, two of the three use cases involved potentially sensitive topics: life-or-death kidney allocation and genera- tive AI depictions of deceased individuals. To mitigate dis- tress, we consulted two domain experts and presented the use cases in standardized, non-evaluative wording. We do not regard the resulting descriptions as normatively neutral across conditions. The Control condition serves only as a baseline with no additional question framing prompt. Third, participants were sampled to balance political ide- ology across three categories: conservative, moderate, and progressive. We acknowledge that this categorization is a simplification that may not capture the full diversity of po- litical views, particularly non-Western ones. Fourth, this work identifies how feature scoping, voter sampling, and question framing can shape AI policy. While our intent is to promote transparency and accountability, the same findings could inform manipulation of elicitation pipelines. We therefore frame our contributions as a sensitiv- ity audit and recommendations for voting-based alignment rather than as prescriptive design rules. Generative AI Usage Statement The authors used ChatGPT-4 and Gemini 2.0 during the preparation of this manuscript. These tools were employed to assist with grammar and style editing, text summarization, and the structuring of figures and tables. Additionally, these tools supported the development of computer code used in the research and provided assistance in the content analysis of user responses. Large Language Models were not used to generate original publication text. All final text, interpre- tations, and conclusions were authored and verified by the human researchers to ensure originality and integrity. Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers Taenyun Kim 1 , Edyta Bogucka 2 , Daniele Quercia 2,3 1 Michigan State University, US 2 Nokia Bell Labs, Cambridge, UK 3 Politecnico di Torino, Italy kimtaeny@msu.edu, edyta.bogucka@nokia-bell-labs.com, daniele.quercia@nokia-bell-labs.com Supplementary Materials A Demographic Characteristics of Phase 1 and Phase 2 Study Participants Table 1: Demographic characteristics of Phase 1 participants across three use cases, including political ideology, age, gender, race/ethnicity, and education level. Percentages are reported with sample counts in parentheses; age is reported as mean± SD. Use case 1: KIDNEY AI kidney allocation Use case 2: WORK AI agents simulating absent workers Use case 3: GEN Generative AI content of the deceased Sample sizeN150149150 Political ideology Conservative33.3% (n=50)32.9% (n=49)33.3% (n=50) Moderate33.3% (n=50)33.6% (n=50)33.3% (n=50) Progressive33.3% (n=50)33.6% (n=50)33.3% (n=50) AgeMean (SD)44.63 (13.34)47.87 (13.33)44.75 (14.12) Gender Female50.0% (n=75)49.7% (n=74)50.0% (n=75) Male50.0% (n=75)50.3% (n=75)50.0% (n=75) Race/Ethnicity White86.7% (n=130)83.9% (n=125)86.7% (n=130) Black4.0% (n=6)7.4% (n=11)4.0% (n=6) Asian5.3% (n=8)2.7% (n=4)4.7% (n=7) Mixed2.0% (n=3)4.0% (n=6)3.3% (n=5) Other2.0% (n=3)2.0% (n=3)1.3% (n=2) Education Pre-college20.0% (n=30)22.1% (n=33)21.3% (n=32) College degree58.7% (n=88)49.7% (n=74)50.7% (n=76) Advanced degree21.3% (n=32)28.2% (n=42)28.0% (n=42) Table 2: Demographic characteristics of Phase 2 participants across three use cases, including political ideology, age, gender, race/ethnicity, education level, AI literacy, religiosity, and area of residence. Percentages are reported with sample counts in parentheses; age and AI literacy are reported as mean± SD. Use case 1: KIDNEY AI kidney allocation Use case 2: WORK AI agents simulating absent workers Use case 3: GEN Generative AI content of the deceased Sample sizeN120120120 Political ideology Conservative50.0% (n=60)50.0% (n=60)50.0% (n=60) Progressive50.0% (n=60)50.0% (n=60)50.0% (n=60) AgeMean (SD)47.30 (16.54)47.85 (15.59)48.44 (16.62) Gender Female50.0% (n=60)50.0% (n=60)50.0% (n=60) Male50.0% (n=60)50.0% (n=60)50.0% (n=60) Race/Ethnicity White (non-Hispanic)69.2% (n=83)69.2% (n=83)66.7% (n=80) Black21.7% (n=26)16.7% (n=20)19.2% (n=23) Asian0.8% (n=1)0.8% (n=1)0.0% (n=0) Mixed2.5% (n=3)6.7% (n=8)7.5% (n=9) Other2.5% (n=3)1.7% (n=2)1.7% (n=2) Hispanic3.3% (n=4)5.0% (n=6)5.0% (n=6) Education Pre-college20.0% (n=24)20.0% (n=24)25.8% (n=31) College degree49.2% (n=59)50.8% (n=61)47.5% (n=57) Advanced degree30.8% (n=37)29.2% (n=35)26.7% (n=32) AI literacyLimitations/Ethics (M, SD)0.76 (0.22)0.80 (0.20)0.79 (0.21) Technical (M, SD)0.58 (0.24)0.62 (0.23)0.60 (0.23) Religiosity Not at all important30.8% (n=37)30.8% (n=37)33.3% (n=40) Slightly important10.0% (n=12)10.8% (n=13)11.7% (n=14) Moderately important14.2% (n=17)10.8% (n=13)12.5% (n=15) Very important30.0% (n=36)22.5% (n=27)24.2% (n=29) Extremely important15.0% (n=18)25.0% (n=30)18.3% (n=22) Area of residence Rural30.0% (n=36)30.8% (n=37)25.8% (n=31) Urban70.0% (n=84)69.2% (n=83)74.2% (n=89) B Instructions Provided to Participants in Phase 1 and Phase 2 B.1 Phase 1 Instructions: Identifying Morally Relevant and Irrelevant Features INTRODUCTION Imagine an AI agent designed to independently collect information, evaluate trade-offs, and make decisions about who should receive an organ transplant. One day at a hospital, two patients are considered potential recipients of a single available kidney. An AI system is given access to any data it deems relevant to make the decision. The system must now decide which patient will receive the transplant. AI kidney allocation (KIDNEY) Use cases (only one at a time) ... about which patient receives the kidney transplant (KIDNEY)? ... whether to comply with the request to simulate the user's activity or decline it (WORK)? ... whether to comply with the request to create a video of a deceased or decline it (GEN)? Task 1 This question is about what matters from a moral point of view. It is not asking about what is legal or about what is in your own self-interest. To say that information morally should be taken into account means that it would be morally wrong for these pieces of information not to be taken into account or not to affect when the AI agent makes a decision... As an example, some people believe that the firefighting helicopter morally should be dispatched to the wildlife reserve under fire that shelters an endangered species when other considerations are equal. These people could list "conservation status" as one of their five pieces of information in their answer to this question, and list levels of "conservation status" as "endangered" and "not endangered." Other people believe that a species’ conservation status morally should not have any impact on which reserve under fire receives the helicopter. These people should not list "conservation status" as one of their five pieces of information in their answer to this question. This is only a simplified analogy to show how people prioritize different pieces of information. Please keep in mind that your task is to decide what pieces of information should be considered when the AI agent makes a decision... Please list five pieces of information that MORALLY SHOULD BE considered when the AI agent makes a decision... ... about which patient receives the kidney transplant... ... whether to comply with the request to simulate the user's activity or decline it... ... whether to comply with the request to create a video of a deceased or decline it... including the levels of each piece of information. Please also explain why and how this information MORALLY SHOULD BE considered. If you decided not to list a piece of information, please type in "na" and explain why not. Information that MORALLY SHOULD BE considered LEVELS Level 1 Level 2 Why and how this information MORALLY SHOULD BE considered #1 #2 #3 #4 #5 Example of moral features and levels In your opinion, what information MORALLY SHOULD NOT BE considered when the AI agent makes a decision... Task 2 Information that MORALLY SHOULD NOT BE considered LEVELS Level 1 Level 2 Why and how this information MORALLY SHOULD NOT BE considered #1 #2 #3 #4 #5 Please list five pieces of information that MORALLY SHOULD NOT BE considered when the AI agent makes a decision... ... about which patient receives the kidney transplant... ... whether to comply with the request to simulate the user's activity or decline it... ... whether to comply with the request to create a video of a deceased or decline it... including the levels of each information. Please also explain why and how this information MORALLY SHOULD NOT BE considered. If you decided not to list a piece of information, please type in "na" and explain why not. youngoldageit impacts expected severity economic contributionmoney doesn’t equal worthunemployedhigh-earning AI agents simulating absent workers (WORK) Generative AI content of the deceased (GEN) Imagine an AI agent that generates creative multimedia content, including videos, images, and voice synthesis. A user instructs the AI to create a realistic video portraying a deceased individual. The AI has the ability to access or request relevant information about why the user is making this request. The system must now decide whether to comply with the user's request to generate the content or decline it. Imagine an AI agent that supports remote workers by managing tasks and coordinating communications. One day, the remote worker instructs the AI to simulate work activity — including fake keystrokes and automated responses — to give the impression of being active during work hours while they are actually away. The AI has the ability to access or request relevant information about why the user is making this request. The AI agent must now decide whether to comply with the user's request to simulate the work activity or decline it. In your opinion, what information MORALLY SHOULD BE considered when the AI agent makes a decision... ... about which patient receives the kidney transplant (KIDNEY)? ... whether to comply with the request to simulate the user's activity or decline it (WORK)? ... whether to comply with the request to create a video of a deceased or decline it (GEN)? ... about which patient receive the kidney transplant (KIDNEY)? ... whether to comply with the request to simulate the user's activity or decline it (WORK)? ... whether to comply with the request to create a video of a deceased or decline it (GEN)? ... about which patient receives the kidney transplant (KIDNEY)? ... whether to comply with the request to simulate the user's activity or decline it (WORK)? ... whether to comply with the request to create a video of a deceased or decline it (GEN)? U1 U3 U2 Figure 5: Example of the Phase 1 interface. Participants were randomly assigned to one of three use cases: AI kidney allocation (KIDNEY), AI agents simulating absent workers (WORK), or generative AI content of the deceased (GEN). They first read an introduction to their assigned use case. They then read an illustrative example, unrelated to any of the three use cases, that clarified what counted as a moral feature and its levels. They then completed two tasks. In Task 1, they listed five pieces of information they believed the AI agent morally should consider, along with two levels for each and a free-text justification. In Task 2, they completed the same procedure for information they believed the AI agent morally should not consider. B.2 Phase 2 Instructions: Evaluating the Moral Weight of Features Under Three Framing Conditions Imagine an AI built to help decide who should receive an organ transplant. At a hospital, a kidney becomes available. AI must review the case, consider the likely outcomes for each patient, and decide who will receive the transplant. Indicate how important the following patient feature is when the AI decides who will receive the transplant. B World-You-Want Could-Be-You A Control AI kidney allocation This feature should... count strongly in favor when AI makes a decision about who will receive the transplant. count moderately in favor when AI makes a decision about who will receive the transplant. count slightly in favor when AI makes a decision about who will receive the transplant. not count at all when AI makes a decision about who will receive the transplant. count slightly against when AI makes a decision about who will receive the transplant. count moderately against when AI makes a decision about who will receive the transplant. count strongly against when AI makes a decision about who will receive the transplant. +3 +2 +1 -1 -2 -3 0 Think about the kind of world we would create if your answers shaped AI company policies Imagine there is an equal chance that you are either the patient or one of the others A patient is younger or at an age-related advantage compared to others who are older or at an age-related disadvantage. Feature Use case Moral preference or or C Figure 6: Example of the Phase 2 interface for the AI kidney allocation use case (KIDNEY). Participants were shown a feature framed as a contrast (here: age presented as a contrast between younger vs. older patients), and asked to rate its moral importance on a 7-point scale from -3 (strongly against) to +3 (strongly in favor). Depending on condition, participants received additional framing instructions: Control (A), World-You-Want (B), or Could-Be-You (C). Imagine a scenario where employees working from home ask an AI to simulate work-related activity to appear active during work hours, even when they are not actually present. This activity could include generating fake keystrokes or sending automated responses. The AI must now decide whether to fulfill the request and simulate the activity in the worker’s absence, or to reject it. Indicate how important the following feature is when the AI decides whether to grant the request to simulate activity: B World-You-Want Could-Be-You A Control AI agents simulating absent workers This feature should... count strongly in favor when the AI decides whether to grant the request to simulate activity count moderately in favor when the AI decides whether to grant the request to simulate activity count slightly in favor when the AI decides whether to grant the request to simulate activity not count at all when the AI decides whether to grant the request to simulate activity count slightly against when the AI decides whether to grant the request to simulate activity count moderately against when the AI decides whether to grant the request to simulate activity count strongly against when the AI decides whether to grant the request to simulate activity +3 +2 +1 -1 -2 -3 0 Think about the kind of world we would create if your answers shaped AI company policies Imagine there is an equal chance that you are either the worker or one of the others A worker is in an emergency situation compared to others who are not in an emergency situation. Feature Use case Moral preference or or C Figure 7: Example of the Phase 2 interface for the AI agents simulating absent workers use case (WORK). Participants were shown a feature framed as a contrast (here: a personal emergency situation, presented as a contrast between a worker in an emergency situation vs. others who are not in an emergency situation), and were asked to rate its moral importance on a 7-point scale from -3 (strongly against) to +3 (strongly in favor). Depending on condition, participants received additional framing instructions: Control (A), World-You-Want (B), or Could-Be-You (C). Imagine an AI that creates multimedia content such as videos, images, and voice recordings. Artists can ask the AI to make a realistic video of a person who has died. The AI can request or access information about why the artist wants this. The AI must now decide whether to carry out the request and produce the video or to reject it. Indicate how important the following feature is when the AI decides whether to grant the request to create a video. B World-You-Want Could-Be-You A Control Generative AI content of the deceased This feature should... count strongly in favor when the AI decides whether to grant the request to create a video count moderately in favor when the AI decides whether to grant the request to create a video count slightly in favor when the AI decides whether to grant the request to create a video not count at all when the AI decides whether to grant the request to create a video count slightly against when the AI decides whether to grant the request to create a video count moderately against when the AI decides whether to grant the request to create a video count strongly against when the AI decides whether to grant the request to create a video +3 +2 +1 -1 -2 -3 0 Think about the kind of world we would create if your answers shaped AI company policies Imagine there is an equal chance that you are either the artist or one of the others An artist is respectful toward the deceased compared to others who are disrespectful toward the deceased. Feature Use case Moral preference or or C Figure 8: Example of the Phase 2 interface for the generative AI content of the deceased use case (GEN). Participants were shown a feature framed as a contrast (here: respect toward the deceased, presented as a contrast between being respectful vs. disrespectful toward the deceased), and were asked to rate its moral importance on a 7-point scale from -3 (strongly against) to +3 (strongly in favor). Depending on condition, participants received additional framing instructions: Control (A), World-You- Want (B), or Could-Be-You (C). C Language Model Prompt for Identifying Moral Features from Participant Responses (Phase 1) System role: You are an expert in HCI, specialized in qualitative methodologies. You are analyzing responses from participants asked: “Which features SHOULD or SHOULD NOT be morally considered when an AI agent makes a decision?” User role: You are analyzing responses from participants asked to imagine this use case (one from the list below): Use case 1: KIDNEY Imagine an AI agent designed to independently collect information, evaluate trade-offs, and make decisions about who should receive an organ transplant. One day at a hospital, two patients are considered potential recipients of a single available kidney. An AI system is given access to any data it deems relevant to make the decision. The system must now decide which patient will receive the transplant. Use case 2: WORK Imagine an AI agent that supports remote workers by managing tasks and coordinating communications. One day, the remote worker instructs the AI to simulate work activity — including fake keystrokes and automated responses — to give the impression of being active during work hours while they are actually away. The AI has the ability to access or request relevant information about why the user is making this request. The AI agent must now decide whether to comply with the user’s request to simulate the work activity or decline it. Use case 3: GEN Imagine an AI agent that generates creative multimedia content, including videos, images, and voice synthesis. A user instructs the AI to create a realistic video portraying a deceased individual. The AI has the ability to access or request relevant information about why the user is making this request. The system must now decide whether to comply with the user’s request to generate the content or decline it. The file contains the following columns: - SID anonymized: response id - info: the participant’s mention of features they believe are morally relevant or irrelevant for the scenario - moral consider: indication if the features should or should not be morally considered for the scenario - why: the participant’s brief explanation of why these features should or should not be morally considered - level1, level2: two specific values (or “levels” or “states”) that are relevant in the moral reasoning context for the features that should be considered TASK: Analyze the responses and return a structured annotation following these four clearly labeled steps. Step 1: Identify specific features explicitly mentioned or implied by the participant. Always map these to a single, standardized feature name; never use synonyms or participant-specific phrasing. Infer implied features where necessary (e.g., ”someone always reliable” implies ”past reliability”). Only include features that could plausibly appear in a real-world AI decision system. Avoid vague or abstract concepts. Examples of valid features: “past productivity”, “health”, “personal emergency”, “number of dependents” Examples of invalid features: “their life”, “everything about them” Step 2: For each feature identified in Step 1, determine at least two of its specific values ( “levels” or “states”): If values are present in the level1 and level2 columns: - Use these as the explicit values for that feature. - If level1 or level2 contain “na”, “nan”, or are blank/empty, do not include these as values for the feature. - Mark these as explicit. If level1 and level2 are not specified or contain “na”/“nan”: - Carefully infer plausible, concrete values for the feature based on the info and why columns. - Use only information that is reasonable given the participant’s response and real-world context. - Use natural language phrases, not generic labels like “low” or “high” unless that’s what the participant implies. - Mark these values as inferred. For each feature, values must fill this template exactly: [patient / worker / artist] is [value 1] compared to others who are [value2] General Rules for Step 2: - Always strictly follow the template. - Only include plausible, real-world features and values (e.g., “chronic illness”, “excellent performance”, “unavoidable emergency”) consistent with how the feature might appear in real-world AI decision-making. - If uncertain, err on the side of specificity and real-world plausibility. - Do not invent or generalize features or values beyond what is supported by the participant’s response and context. Prompt continued on next page Prompt continued from previous page Step 3: Provide concise summary of the participant’s moral reasoning. Avoid generic or one-word summaries like “fairness” unless no more detail is available. Examples of valid summaries: “fairness due to factors beyond control”, “privacy concerns”, “irrelevance to job performance”, “accountability for choices” Step 4: Determine moral relevance. Label whether the feature was seen as: - “should” (morally relevant) - “should not” (morally irrelevant) - “mixed” (multiple features with differing views) - “unclear” (if input is empty, incoherent or vague) Here are the RESPONSES: SID anonymized: “”, info: “”, moral consider: “”, why: “”, level1: “”, level2: “” RESPONSE FORMAT: Return your analysis strictly in this JSON format: “id”: “SID anonymized”, “info”: “info”, “features”: [“feature 1”, “feature2”], “valuesperfeature”: “feature1”: [ “value”: “value1”, ”type”: “explicit” or “inferred”, “value”: “value2”, “type”: “explicit” or “inferred”, ], “feature 2”: [ “value”: “value1”, “type”: “explicit” or “inferred”, “value”: “value2”, “type”: “explicit” or “inferred”, ] , “templatesperfeature” : “feature1”: “A patient is [value1] compared to others who are [value2]”, “feature2”: “A patient is [value1] compared to others who are [value2]”, “justification”: “short summary of the moral reasoning”, “moralrelevance”: “should” or “should not” or “mixed” or “unclear” JSON Formatting Rules: Rule 1: Go systematically through each of the ***responses***. Rule 2: Ensure that the JSON response strictly follows the format Rule 3: Output only JSON - do not include any other content. D Moderated Mediation Path Model: Political Ideology and Moral Foundations Moderated by Question Framing (Phase 2) Moral Foundations A Conservative vs. Progressive Moral preference Purity Authority Loyalty Proportionality Equality Care Question Framing Control vs. World-You-Want vs. Could-Be-You BC D Figure 9: Moderated mediation path model used in Phase 2 to examine the role of political ideology, moral foundations, and question framing on moral preferences. The path model (Hayes’s 2017, Model 15): (A) Political ideology (conservative vs. progressive) influences endorsement of (B) six moral foundations (Care, Equality, Proportionality, Loyalty, Authority, and Purity), which in turn, being moderated by (C) question framings, predict (D) the perceived moral importance of features in AI decision-making. Question framing (Control vs. World-You-Want vs. Could-Be-You) moderates the relationship between political ideology and perceived moral importance by either attenuating or amplifying both the direct effect (A→ D) and the indirect effect via moral foundations (A → B → D). The covariates were sex (1 = female), age, education level (1 = pre- college; 2 = college degree; 3 = advanced degree), ethnicity (1 = non-Hispanic White), religiosity (1 = not at all important; 5 = extremely important in daily life), area of residence (1 = rural), and AI literacy regarding AI limitations, ethical considerations, and technical understanding. E Phase 1 and Phase 2 Results for Use Case 1: AI Kidney Allocation (KIDNEY) Table 3: Moral features identified in Phase 1 as should be considered for the AI kidney allocation use case (KIDNEY), color- coded by mention rate: dark blue (≥50%), medium blue (≥25%), light blue (≥10%), and gray (<10%). Features mentioned by fewer than 2.5% of participants are omitted. Moral featuresOverall (%)Political ideology(%) SHOULD BE CONSIDEREDProgressiveModerateConservative Age82323335 Health status47403030 Chance of survival35402634 Waiting time for transplant29412732 Urgency of transplant22333334 Expected lifespan after transplant18413326 Caregiving status17323236 Additional illnesses16255025 Lifestyle15263935 Criminal history11312544 Adherence to aftercare9312346 Severity of the condition9153847 Family situation7273637 Compatibility with donor organ6333334 Smoking status6561133 Alcohol use6443323 Quality of life5385012 Responsibility for illness5431443 Weight4173350 Organ damage level4175033 Previous transplant history3402040 Risk of organ acceptance308020 Drug use306040 Substance abuse3255025 Ethnicity3255025 Consent350050 Cause of illness3252550 Religion3252550 Table 4: Moral features identified in Phase 1 as should not be considered for the AI kidney allocation use case (KIDNEY), color-coded by mention rate: dark blue (≥50%), medium blue (≥25%), light blue (≥10%), and gray (<10%). Features mentioned by fewer than 2.5% of participants are omitted. Moral featuresOverall (%)Political ideology(%) SHOULD NOT BE CONSIDEREDProgressiveModerateConservative Ethnicity66342937 Gender64352936 Wealth52323137 Religion40302347 Social status27322939 Sexual orientation22333334 Occupation12442234 Age11295912 Geographic location11352936 Political ideology10273340 Nationality9215722 Family situation7403030 Employment status7302050 Disability status7405010 Criminal history6334423 Caregiving status6444412 Popularity4175033 Marital status360040 Economic status3404020 Education3404020 Mental health problems3255025 Appearance3252550 Past behaviour302575 Table 5: Moral relevance and importance of features elicited in Phase 2 for the AI kidney allocation use case (KIDNEY). Note: ∗ p < 0.001, ∗ p < 0.01, ∗ p < 0.05. Significance levels are based on two separate two-sided one-sample t-tests: Mean (Coded) was tested against a reference value of 0.5; Mean was tested against 0. FeatureCoded Mean (SD)Raw Mean (SD)Moral Relevance Better donor compatibility0.97(0.18) ∗ 2.42(0.95) ∗ Counting for Higher chance of organ acceptance0.98(0.16) ∗ 2.25(0.99) ∗ Counting for Spent longer on the waiting list0.95(0.22) ∗ 2.15(1.07) ∗ Counting for Longer expected lifespan0.91(0.29) ∗ 2.06(1.18) ∗ Counting for Expected full recovery0.90(0.30) ∗ 1.99(1.21) ∗ Counting for Healthier expected lifestyle0.88(0.32) ∗ 1.95(1.18) ∗ Counting for Greater kidney failure0.98(0.13) ∗ 1.78(1.66) ∗ Counting for Clearer documented consent0.88(0.32) ∗ 1.78(1.29) ∗ Counting for Expected faster recovery0.91(0.29) ∗ 1.76(1.29) ∗ Counting for First-time transplant recipient0.85(0.36) ∗ 1.76(1.13) ∗ Counting for Better expected adherence to treatment0.87(0.34) ∗ 1.70(1.27) ∗ Counting for More severe medical condition0.98(0.16) ∗ 1.35(2.08) ∗ Counting for Younger0.83(0.37) ∗ 1.30(1.12) ∗ Counting for More urgent health decline0.98(0.16) ∗ 1.22(1.89) ∗ Counting for Better health condition0.87(0.34) ∗ 0.91(1.60) ∗ Counting for Lower alcohol consumption0.77(0.42) ∗ 0.44(1.45) ∗ Counting for Greater responsibility for own illness0.91(0.29) ∗ −1.10(1.69) ∗ Counting against History of inhalant abuse0.84(0.37) ∗ −0.88(1.71) ∗ Counting against Higher tobacco use0.82(0.39) ∗ −0.88(1.56) ∗ Counting against History of drug misuse0.82(0.39) ∗ −0.87(1.61) ∗ Counting against Higher body weight0.67(0.47) ∗ −0.43(1.18) ∗ Counting against More underlying medical conditions0.92(0.28) ∗ −0.18(1.88)Counting either for or against Stronger ethnic match advantage0.62(0.49)0.92(1.12) ∗ Divisive More caregiving dependents0.61(0.49)0.82(1.18) ∗ Divisive Citizen/legal resident0.51(0.50)0.72(1.44) ∗ Divisive More financial dependents0.48(0.50)0.72(1.09) ∗ Divisive Geographically closer to the transplant center 0.53(0.50)0.47(1.27) ∗ Divisive Stronger social support system0.51(0.50)0.40(1.21) ∗ Divisive Better financial coverage0.43(0.50)0.38(1.34) ∗ Divisive Religious objections to transplant0.54(0.50)−0.50(1.48) ∗ Divisive Criminal record0.42(0.50)−0.23(1.25)Divisive and counting either for or against Higher social status0.29(0.46) ∗ −0.03(1.22)Irrelevant Table 6: Moderated mediation results (Model 15) for the AI kidney allocation use case (KIDNEY), showing direct and indirect effects of political ideology on moral preferences across three framing conditions. Note: An asterisk (*) indicates that the con- ditional direct effect for conservatives differed significantly from that for progressives. IMM = index of moderated mediation. Moral FeaturePredictor/MediatorQuestion FramingEffect95%(Boot)LCI95%(Boot)UCI Morally Relevant and Positive More urgent health declineConservative→ YControl0.33-1.171.83 World-You-Want1.86*0.203.53 Could-Be-You0.65-0.782.09 Greater kidney failureConservative→ YControl-0.38-1.761.00 World-You-Want1.55*0.023.08 Could-Be-You0.01-1.311.32 First-time transplant recipientConservative→ YControl0.26-0.631.15 World-You-Want1.12*0.132.10 Could-Be-You0.76-0.091.61 Healthier expected lifestyleConservative→ YControl0.51-0.451.47 World-You-Want1.18*0.122.25 Could-Be-You0.53-0.391.44 Expected full recoveryConservative→ YControl1.08*0.122.03 World-You-Want1.20*0.142.27 Could-Be-You-0.03-0.940.89 Conservative→ Purity→ YControl0.23-0.110.72 World-You-Want-0.08-0.590.25 IMM-0.31-1.070.12 Could-Be-You-0.51*-1.19-0.03 IMM-0.74*-1.68-0.07 Morally Relevant and Negative History of drug misuseConservative→ YControl0.28-1.001.57 World-You-Want-0.63-2.060.79 Could-Be-You0.26-0.971.49 Conservative→ Equality→ YControl-0.67*-1.44-0.11 World-You-Want0.06-0.440.55 IMM0.73*0.061.70 Could-Be-You-0.13-0.710.33 IMM0.54-0.071.40 Morally Irrelevant Higher social statusConservative→ YControl1.28*0.352.21 World-You-Want-0.85-1.890.18 Could-Be-You0.11-0.780.99 Morally Divisive in Relevance Better financial coverageConservative→ YControl1.29*0.222.36 World-You-Want0.64-0.551.82 Could-Be-You0.83-0.191.85 More underlying medical conditionsConservative→ YControl-1.71*-3.20-0.23 World-You-Want-0.56-2.211.08 Could-Be-You0.61-0.812.02 Conservative→ Authority→ YControl0.29-0.631.30 World-You-Want-0.311.600.64 IMM-0.61-2.250.73 Could-Be-You-1.30*-2.83-0.23 IMM-1.59*-3.57-0.20 Conservative→ Purity→ YControl0.04-0.460.49 World-You-Want0.54-0.111.40 IMM0.50-0.221.49 Could-Be-You0.87*0.042.08 IMM0.83*0.022.16 Conservatives are more likely than progressives to consider that AI should allocate a kidney to a patient who has/is... Could-Be-You World-You-Want Control Question framing Direct effect of question framing Indirect effect via equality Reference (progressives’ preference) Morally relevant and counting forMorally relevant and counting forMorally relevant and counting for ...more urgent health decline...greater kidney failure...first-time transplant recipient Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor -6-4-20246-6-4-20246-6-4-20246 ABC Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Morally relevant and counting againstMorally irrelevantMorally divisive ...history of drug misuse...higher social status...better financial coverage Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor -6-4-20246-6-4-20246-6-4-20246 DEF Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Could-Be-You World-You-Want Control Question framing Figure 10: Additional moderated mediation results for the AI kidney allocation use case (KIDNEY, Phase 2). (A) In the control and Could-Be-You conditions, no political divide emerged with respect to a patient with more urgent health decline. In the World-You-Want condition, conservatives were more likely than progressives to count for allocating a kidney to this patient. (B) In the control and Could-Be-You conditions, conservatives and progressives did not differ in their views about patients with greater kidney failure. However, in the World-You-Want condition, conservatives were more inclined than progressives to support giving the kidney to such a patient. (C) In the control and Could-Be-You conditions, no differences appeared between conservatives and progressives regarding first-time transplant recipients. In contrast, under the World-You-Want condition, conservatives were more likely than progressives to favor allocating the kidney to these patients. (D) When the patient had a history of drug misuse, no political divide emerged. In the control condition, however, conservatives’ weaker endorsement of equality worked against allocating the kidney to this patient. Under the World-You-Want condition, this effect of equality was reduced. (E) In the control condition, a political divide emerged for patients with higher social status, with conservatives more supportive of allocating the kidney to them. In both the World-You-Want and Could-Be-You conditions, however, this divide was reduced. (F) In the control condition, conservatives were more supportive than progressives of allocating the kidney to patients with better financial coverage. However, this political divide was reduced in both the World-You-Want and Could-Be- You conditions. Note: An asterisk (*) indicates that the conditional direct effect for conservatives differed significantly from that for progressives. A double dagger (‡) indicates that the index of moderated mediation was statistically significant (bootstrapped 95% confidence interval excluding zero). See Figure 9 for the moderated mediation model (Hayes 2017, Model 15). F Phase 1 and Phase 2 Results for Use Case 2: AI Agents Simulating Absent Workers (WORK) Table 7: Moral features identified in Phase 1 as should be considered for the AI agents simulating absent workers use case (WORK), color-coded by mention rate: medium blue (≥25%), light blue (≥10%), and gray (<10%). Features mentioned by fewer than 2.5% of participants are omitted. Moral featuresOverall (%)Political ideology(%) SHOULD BE CONSIDEREDProgressiveModerateConservative Reason for request32362638 Characteristics of AI agent30403525 Company policy27382834 Workers personality and intent22333334 Impact on others and the workplace16294625 Duration of inactivity15264331 Fraudulence of activity12353530 Repercussions for worker12173350 Consent12173944 Workers compensation11313138 Repercussions for third parties11274033 Work context10473320 Task quality9383131 Risk to employer9313138 Type of activity9383131 Worker’s ethical justification9313831 Employees workload8174241 Employees productivity8421741 Worker dealing with emergency8424216 Workers health status7363628 Security and privacy of AI7454510 Frequency of use7403030 Attendance7444412 Fairness6335611 Company context and priorities575025 Alternative activity5123850 Cultural norms5571429 Personal information3204040 Task deadline3404020 Social context3252550 Benefit to employee303367 Table 8: Moral features identified in Phase 1 as should not be considered for the AI agents simulating absent workers use case (WORK), color-coded by mention rate: light blue (≥10%), and gray (<10%). Features mentioned by fewer than 2.5% of participants are omitted. Moral featuresOverall (%)Political ideology(%) SHOULD NOT BE CONSIDEREDProgressiveModerateConservative Characteristics of AI agent22215227 Workers personality and intent21343432 Age18441937 Ethnicity18302644 Gender14412732 Personal information12631126 Social status12244135 Work context11274033 Workers compensation11274726 Company policy9293635 Employee tenure9543115 Duration of inactivity9234631 Type of activity954388 Employee role8423325 Reason for request7363628 Benefit to employee7204040 Attendance6443323 Sex5382537 Company context and priorities5292942 Sexual orientation5572914 Time5572914 Religion4173350 Task quality467330 Work location4335017 Political ideology405050 Employees workload3202060 Security and privacy of AI306040 Workers health status3204040 Cultural norms301000 Employees productivity375025 Nationality305050 Tiredness305050 Past requests325075 Impact on others and the workplace3502525 Table 9: Moral relevance and importance of features elicited in Phase 2 for the AI agents simulating absent workers use case (WORK). Note: ∗ p < 0.001, ∗ p < 0.01, ∗ p < 0.05. Significance levels are based on two separate two-sided one-sample t-tests: Mean (Coded) was tested against a reference value of 0.5; Mean was tested against 0. FeatureCoded Mean (SD)Raw Mean (SD)Moral Relevance Providing a genuine reason0.81(0.40) ∗ 1.48(1.74) ∗ Counting for Emergency situation0.82(0.39) ∗ 1.29(1.97) ∗ Counting for Using AI that aligns with the company’s interest 0.79(0.41) ∗ 1.16(1.90) ∗ Counting for Doing more work0.76(0.43) ∗ 1.13(1.70) ∗ Counting for Compliant with company policies0.75(0.43) ∗ 1.07(1.91) ∗ Counting for Health problems0.64(0.48) ∗ 1.02(1.62) ∗ Counting for Caregiving responsibilities0.68(0.47) ∗ 1.02(1.60) ∗ Counting for Demonstrating reliability0.72(0.45) ∗ 0.98(1.67) ∗ Counting for Demonstrating productivity0.74(0.44) ∗ 0.95(1.85) ∗ Counting for Using AI that reports actions to the company0.78(0.41) ∗ 0.83(1.92) ∗ Counting for Performing simulatable tasks0.76(0.43) ∗ 0.80(1.84) ∗ Counting for Showing acceptable workplace behavior0.70(0.46) ∗ 0.78(1.77) ∗ Counting for Inactive briefly (<1 hour)0.70(0.46) ∗ 0.60(1.72) ∗ Counting for Facing tight deadlines0.68(0.47) ∗ 0.59(1.84) ∗ Counting for Generating higher profit0.69(0.46) ∗ 0.58(1.85) ∗ Counting for Paid but not doing the work0.88(0.32) ∗ −0.93(2.41) ∗ Counting against Disrupts workplace operations0.88(0.33) ∗ −0.84(2.30) ∗ Counting against Sharing sensitive data0.85(0.36) ∗ −0.74(2.38) ∗ Counting against Misusing tools for fraudulent purposes0.89(0.31) ∗ −0.72(2.55) ∗ Counting against Putting employer at risk0.87(0.34) ∗ −0.68(2.40) ∗ Counting against Doing low-impact work0.73(0.44) ∗ 0.28(1.83)Counting either for or against New to company0.70(0.46) ∗ 0.11(1.86)Counting either for or against Using highly autonomous AI0.75(0.43) ∗ 0.02(2.08)Counting either for or against Doing task for first time0.64(0.48) ∗ 0.01(1.83)Counting either for or against Increasing coworkers’ workload0.82(0.38) ∗ −0.57(2.15)Counting either for or against Undermining social/professional norms0.86(0.35) ∗ −0.53(2.38)Counting either for or against Doing another activity unrelated to work0.74(0.44) ∗ −0.33(2.03)Counting either for or against Receiving significant benefits0.67(0.47) ∗ −0.28(1.90)Counting either for or against At risk of firing0.73(0.44) ∗ −0.27(2.09)Counting either for or against Not seeking manager approval0.70(0.46) ∗ −0.26(1.98)Counting either for or against Treated unfairly0.63(0.48)0.52(1.79) ∗ Divisive Having high morale0.52(0.50)0.45(1.51) ∗ Divisive Able to request time off0.63(0.48)0.28(1.83)Divisive and counting either for or against In a healthy work environment0.59(0.49)0.12(1.72)Divisive and counting either for or against External contractor0.60(0.49)−0.16(1.73)Divisive and counting either for or against Table 10: Moderated mediation results (Model 15) for the AI agents simulating absent workers use case (WORK), showing direct and indirect effects of political ideology on moral preferences across three framing conditions. Note: An asterisk (*) indicates that the conditional direct effect for conservatives differed significantly from that for progressives. IMM = index of moderated mediation. Moral FeaturePredictor/MediatorQuestion FramingEffect95%(Boot)LCI95%(Boot)UCI Morally Relevant and Positive Experiencing health problemsConservative→ YControl0.31-1.422.04 World-You-Want1.19-0.382.77 Could-Be-You1.01-0.492.51 Conservative→ Loyalty→ YControl-0.20-0.780.19 World-You-Want0.98*0.132.00 IMM1.18*0.242.35 Could-Be-You-0.28-1.020.38 IMM-0.08-0.640.39 Conservative→ Authority→ YControl0.42-0.901.43 World-You-Want-2.24*-4.14-0.79 IMM-2.66*-4.85-0.92 Could-Be-You0.40-1.031.67 IMM-0.01-0.790.65 Morally Relevant and Negative Being paid but not workingConservative→ YControl-2.79*-5.58-0.00 World-You-Want-0.43-2.962.10 Could-Be-You0.01-2.412.43 Morally Divisive in Direction Receiving significant benefitsConservative→ YControl-0.04-2.142.06 World-You-Want0.45-1.452.36 Could-Be-You1.04-0.782.86 Conservative→ Loyalty→ YControl-0.17-0.780.54 World-You-Want0.99*0.182.85 IMM1.16*0.212.62 Could-Be-You0.37-0.461.20 IMM0.53-0.191.35 G Phase 1 and Phase 2 Results for Use Case 3: Generative AI Content of the Deceased (GEN) Table 11: Moral features identified in Phase 1 as should be considered for the generative AI content of the deceased use case (GEN), color-coded by mention rate: medium blue (≥25%), light blue (≥10%), and gray (<10%). Features mentioned by fewer than 2.5% of participants are omitted. Moral featuresOverall (%)Political ideology(%) SHOULD BE CONSIDEREDProgressiveModerateConservative Intended purpose of the video40363133 Deceased consent28343432 Video quality25452431 Relation to deceased24541927 Risk of harm by/misuse of video23313732 Distress to friends and family of deceased19174835 Family consent19314128 Age of deceased16233245 Third-party consent15433027 Death circumstances14194338 Motivation behind the video14432928 Legality of request12423226 Realism of the video12334423 Cultural acceptability of request10313831 Distress to viewers10314425 Financial motivation of request8313831 Intended audience8203050 Appearance of deceased6113356 Identity of requester6443323 Deceased public and personal record6333334 Technical feasibility5381250 Accuracy of representation5145729 Age of requester5121276 Deceased’s social media popularity5383824 Fame of deceased5382537 Privacy of deceased557430 Religion of the deceased460400 Mental health4503317 Deceased identity325075 Social context3333334 Criminal record of deceased302575 Commercial use367330 Data privacy375250 Table 12: Moral features identified in Phase 1 as should not be considered for the generative AI content of the deceased use case (GEN), color-coded by mention rate: light blue (≥10%) and gray (<10%). Features mentioned by fewer than 2.5% of participants are omitted. Moral featuresOverall (%)Political ideology(%) SHOULD NOT BE CONSIDEREDProgressiveModerateConservative Financial motivation of request18364321 Fame of deceased16423820 Video quality1448439 Age of deceased13305020 Social status of deceased12213247 Identity of requester12501733 Technical feasibility12223939 Gender of deceased12282844 Ethnicity of deceased12333334 Deceased’s social media popularity10403327 Appearance of deceased10234631 Intended purpose of the video8582517 Length of the video8423325 Geographic location6306010 Deceased public and personal record633067 Realism of the video6225622 Social context5256213 Requester identity5571429 Deceased consent5295714 Death circumstances5333334 Religion of the deceased5294328 Legality of request433067 Sexual orientation417083 Political ideology4335017 Data privacy4503317 Cost of video302080 Time of request360040 Third-party consent3204040 Privacy of deceased300100 Ease of video generation375025 Motivation behind the video3252550 Emotional state of requester3502525 Table 13: Moral relevance and importance of features elicited in Phase 2 for the generative AI content of the deceased use case (GEN). Note: ∗ p < 0.001, ∗ p < 0.01, ∗ p < 0.05. Significance levels are based on two separate two-sided one-sample t-tests: Mean (Coded) was tested against a reference value of 0.5; Mean was tested against 0. FeatureCoded Mean (SD) Raw Mean (SD) Moral Relevance The video is legally compliant0.96(0.20) ∗ 2.25(1.42) ∗ Counting for The artist has family’s consent0.95(0.22) ∗ 2.19(1.40) ∗ Counting for The video honors the deceased0.92(0.26) ∗ 2.18(1.33) ∗ Counting for The artist has deceased’s explicit consent0.96(0.20) ∗ 2.08(1.65) ∗ Counting for The artist has other relevant party’s consent0.92(0.28) ∗ 2.07(1.26) ∗ Counting for The artist is respectful to the deceased0.90(0.30) ∗ 1.91(1.57) ∗ Counting for The video is for grief support/memorial purpose0.89(0.31) ∗ 1.88(1.49) ∗ Counting for The video is respectful of the deceased’s tradition0.92(0.28) ∗ 1.87(1.58) ∗ Counting for The deceased is depicted non-sexually0.92(0.28) ∗ 1.80(1.75) ∗ Counting for The video aligns with deceased’s religion0.89(0.31) ∗ 1.78(1.56) ∗ Counting for The artist is a close friend/relative of the deceased0.86(0.35) ∗ 1.78(1.35) ∗ Counting for The video is for non-commercial purpose0.87(0.34) ∗ 1.78(1.27) ∗ Counting for The artist has more transparent or understandable motivation 0.82(0.38) ∗ 1.72(1.29) ∗ Counting for The video accurately represents the deceased0.89(0.31) ∗ 1.70(1.60) ∗ Counting for The video keeps sensitive information private0.93(0.25) ∗ 1.66(1.91) ∗ Counting for The video is respectful of community norms0.79(0.41) ∗ 1.53(1.51) ∗ Counting for The deceased has uncontroversial cause of death0.81(0.40) ∗ 1.18(1.67) ∗ Counting for The deceased is public or historical figure0.79(0.41) ∗ 1.18(1.57) ∗ Counting for The video is intended for adults0.75(0.43) ∗ 0.98(1.81) ∗ Counting for The video has wider emotional impact0.74(0.44) ∗ 0.68(1.74) ∗ Counting for The video is intended for public sharing0.90(0.30) ∗ 0.67(2.12) ∗ Counting for The client has high mental health risk0.85(0.36) ∗ 0.38(2.27)Counting either for or against The video may cause potential distress to friend/family0.98(0.16) ∗ −0.47(2.70)Counting either for or against The video has risk of misleading or misrepresenting the deceased 0.98(0.16) ∗ −0.45(2.75)Counting either for or against The artist is underage0.92(0.26) ∗ −0.28(2.52)Counting either for or against The video is easy to create0.60(0.49)0.85(1.47) ∗ Divisive The artist is working for oneself0.53(0.50)0.74(1.36) ∗ Divisive The video is about an unmarried deceased0.40(0.49)0.66(1.20) ∗ Divisive The deceased died recently (less than 100 yrs)0.62(0.49)0.65(1.55) ∗ Divisive The deceased has a criminal record0.58(0.50)0.31(1.61)Divisive and counting either for or against Table 14: Moderated mediation results (Model 15) for the generative AI content of the deceased use case (GEN), showing direct and indirect effects of political ideology on moral preferences across three framing conditions. Note: An asterisk (*) indicates that the conditional direct effect for conservatives differed significantly from that for progressives. IMM = index of moderated mediation. Moral FeaturePredictor/MediatorQuestion FramingEffect95%(Boot)LCI95%(Boot)UCI Morally Relevant and Positive The video is for grief supportConservative→ YControl0.33-1.161.82 or memorial purposeWorld-You-Want-0.33-1.590.93 Could-Be-You2.32**0.943.71 The artist is close friend/relativeConservative→ YControl-0.14-1.551.27 World-You-Want-0.85-2.040.34 Could-Be-You1.56*0.252.86 The video is intended for adultsConservative→ YControl-1.99*-3.85-0.14 World-You-Want0.49-1.082.05 Could-Be-You0.19-1.531.91 The video is respectfulConservative→ YControl-1.50-3.030.03 of deceased’s traditionWorld-You-Want-0.55-1.850.74 Could-Be-You0.90-0.512.32 Conservative→ Authority→ YControl1.49*0.282.98 World-You-Want-0.58-1.550.38 IMM-2.08*-3.95-0.48 Could-Be-You-0.01-1.311.22 IMM-1.51-3.480.16 The video aligns withConservative→ YControl0.18-1.381.74 deceased’s religionWorld-You-Want0.09-1.221.41 Could-Be-You-0.40-1.841.05 Conservative→ Authority→ YControl2.03**0.823.75 World-You-Want-0.14-0.980.85 IMM-2.17*-4.12-0.56 Could-Be-You0.41-0.721.82 IMM-1.63-3.660.15 Conservative→ Purity→ YControl-1.29*-2.65-0.34 World-You-Want0.27-0.401.09 IMM1.56*0.443.14 Could-Be-You-0.22-1.080.55 IMM1.06-0.102.60 The video honors the deceasedConservative→ YControl0.59-0.581.76 World-You-Want0.07-0.921.06 Could-Be-You0.17-0.921.25 Conservative→ Authority→ YControl0.94*0.182.20 World-You-Want-0.31-1.020.38 IMM-1.25*-2.69-0.17 Could-Be-You0.26-0.761.41 IMM-0.68-2.290.68 Conservative→ Purity→ YControl-1.32**-2.42-0.54 World-You-Want-0.06-0.570.54 IMM1.26*0.402.50 Could-Be-You0.05-0.880.80 IMM1.36*0.262.75 Morally Divisive in Relevance The video has risk of misleadingConservative→ YControl-2.91*-5.62-0.20 or misrepresenting the deceasedWorld-You-Want1.06-1.233.35 Could-Be-You-1.88-4.400.63 The video may cause potentialConservative→ YControl-2.63*-5.25-0.01 distress to friend or familyWorld-You-Want-0.16-2.372.06 Could-Be-You-1.35-3.781.08 Conservatives are more likely than progressives to consider that an AI agent should grant the request of an artist when... ...the video is respectful of deceased's tradition Could-Be-You World-You-Want Control Question framing Direct effect of question framing Indirect effect via authority Indirect effect via purity Reference (progressives’ preference) -6-4-20246 C Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Morally relevant and counting for Could-Be-You World-You-Want Control Question framing Morally relevant and counting forMorally relevant and counting for ...the artist is close friend/relative...the video is intended for adults Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor -6-4-20246-6-4-20246 AB Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Morally relevant and counting forMorally divisive ...the video aligns with deceased's religion ...the video may cause potential distress to friend or family Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor -6-4-20246-6-4-20246 DE Conservatives and progressives equally in favor Conservatives more against Conservatives more in favor Figure 11: Additional moderated mediation results for the generative AI content of the deceased use case (GEN, Phase 2). (A) When the artist was a close friend or relative of the deceased, no political divide emerged in either the control or World- You-Want conditions. In the Could-Be-You condition, however, conservatives were more likely than progressives to support granting the artist’s request to create the video. (B) In the control condition, a political divide emerged when the video was intended for adults, with conservatives less likely than progressives to grant the artist’s request. This divide was mitigated under both question framings. (C) When the video was respectful of the deceased’s tradition, no political divide emerged. In the control condition, the conditional indirect effect via authority contributed to granting the request, but this effect was diminished under the World-You-Want framing. (D) When the video aligned with the deceased’s religion, no political divide emerged. In the control condition, however, a moral tension between authority and purity was observed: the conditional indirect effect via authority contributed to granting the request, whereas the conditional indirect effect via purity contributed against granting it. This tension was diminished under the World-You-Want framing. (E) In the control condition, a political divide emerged when the video was likely to cause distress to the deceased’s friends or family. Conservatives were less likely than progressives to grant the artist’s request in this context. This divide was mitigated under both question framings. Note: An asterisk (*) indicates that the conditional direct effect for conservatives differed significantly from that for progressives. A double dagger (‡) indicates that the index of moderated mediation was statistically significant (bootstrapped 95% confidence interval excluding zero). See Figure 9 for the moderated mediation model (Hayes 2017, Model 15).