Paper deep dive
Unilateral Relationship Revision Power in Human-AI Companion Interaction
Benjamin Lange
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/26/2026, 1:46:35 AM
Summary
The paper introduces 'Unilateral Relationship Revision Power' (URRP) to describe the structural power imbalance in human-AI companion interactions. The author argues that these interactions are triadic (user, AI, provider) rather than dyadic, and that providers exercise constitutive control over the AI. This structure fails three conditions for normatively robust relationships (Independence, Non-Control, Non-Substitutability), leading to normative hollowing, displaced vulnerability, and structural irreconcilability.
Entities (8)
Relation Signals (4)
Provider → exercisesconstitutivecontrolover → AI Companion
confidence 95% · I argue that human-AI companion interaction is a triadic structure in which the provider exercises constitutive control over the AI.
URRP → causes → Normative Hollowing
confidence 90% · URRP has three implications: i) normative hollowing
URRP → causes → Displaced Vulnerability
confidence 90% · URRP has three implications: ii) displaced vulnerability
URRP → causes → Structural Irreconcilability
confidence 90% · URRP has three implications: iii) structural irreconcilability
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:When providers update AI companions, users report grief, betrayal, and loss. A growing literature asks whether the norms governing personal relationships extend to these interactions. So what, if anything, is morally significant about them? I argue that human-AI companion interaction is a triadic structure in which the provider exercises constitutive control over the AI. I identify three structural conditions of normatively robust dyads that the norms characteristic of personal relationships presuppose and show that AI companion interactions fail all three. This reveals what I call Unilateral Relationship Revision Power (URRP): the provider can rewrite how the AI interacts from a position where these revisions are not answerable within that interaction. I argue that designing interactions that exhibit URRP is pro tanto wrong because it involves cultivating normative expectations while maintaining conditions under which those expectations cannot be fulfilled. URRP has three implications: i) normative hollowing (commitment is elicited but no agent inside the interaction bears it), ii) displaced vulnerability (the user's exposure is governed by an agent not answerable to her within the interaction), and iii) structural irreconcilability (when trust breaks down, reconciliation is structurally unavailable because the agent who acted and the entity the user interacts with are different). I discuss design principles such as commitment calibration, structural separation, and continuity assurance as external substitutes for the internal constraints the triadic structure removes. The analysis therefore suggests that a central and underexplored problem in relational AI ethics is the structural arrangement of power over the human-AI interaction itself.
Tags
Links
- Source: https://arxiv.org/abs/2603.23315v1
- Canonical: https://arxiv.org/abs/2603.23315v1
Trouble viewing inline? Open PDF directly →
Full Text
78,363 characters extracted from source content.
Expand or collapse full text
Unilateral Relationship Revision Power in Human- AI Companion Interaction Benjamin Lange LMU & MCML Abstract. When providers update AI companions, users report grief, be- trayal, and loss. A growing literature asks whether the norms governing per- sonal relationships extend to these interactions. So what, if anything, is mor- ally significant about them? I argue that human-AI companion interaction is a triadic structure in which the provider exercises constitutive control over the AI. I identify three structural conditions of normatively robust dyads that the norms characteristic of personal relationships presuppose and show that AI companion interactions fail all three. This reveals what I call Unilateral Relationship Revision Power (URRP): the provider can rewrite how the AI inter- acts from a position where these revisions are not answerable within that in- teraction. I argue that designing interactions that exhibit URRP is pro tanto wrong because it involves cultivating normative expectations while maintain- ing conditions under which those expectations cannot be fulfilled. URRP has three implications: i) normative hollowing (commitment is elicited but no agent inside the interaction bears it), i) displaced vulnerability (the user’s ex- posure is governed by an agent not answerable to her within the interaction), and i) structural irreconcilability (when trust breaks down, reconciliation is structurally unavailable because the agent who acted and the entity the user interacts with are different). I discuss design principles such as commitment calibration, structural separation, and continuity assurance as external substi- tutes for the internal constraints the triadic structure removes. The analysis therefore suggests that a central and underexplored problem in relational AI ethics is the structural arrangement of power over the human-AI interaction itself. Keywords: human-AI relationships, constitutive control, unilateral relation- ship revision power, triadic structure, normative hollowing, displaced vulner- ability, structural irreconcilability, AI companion design, ethical behaviourism 2 1. Introduction People form attachments to AI companions. 1 Users of Replika, Character.AI, and conversational assistants describe sustained emotional engagement, trust, and intimacy (Turkle et al. 2006; Xie and Pentina 2022). When these interac- tions are disrupted—such as when an update alters a persona, a policy change restricts conversations, or a service is discontinued—users report grief, be- trayal, and loss. For instance, users of Replika perceived the post-update AI as a different entity, with emotional responses comparable to human bereave- ment (De Freitas et al 2024). 2 A growing literature asks whether the norms governing personal relationships extend to these interactions (Coeckelbergh 2012; Gunkel 2018; Danaher 2020; Weber-Guskar 2022; Nyholm 2020; Ga- briel et al 2024; Earp et al. 2025; Sparrow and Brown 2026). 3 When a user interacts with an AI companion, three parties are involved. The user, the AI system, and the provider—the company that develops and, 1 According to a recent study by Fang et al (2025), hundreds of millions of people now use platforms like Replika, Character.AI, and ChatGPT for emotional support, with research showing that extended use and high social attraction to the AI are directly associated with emotional dependence and problematic use. 2 On companion AI as a product category, see Moore (2024), who reports that companion AI accounts for 16 of the top 100 AI apps by monthly active users. On the loneliness context in which these products operate, see Murthy (2023). 3 More generally, the literature divides into sceptical, sympathetic, and virtue-eth- ical positions. For example, Sparrow and Brown (2026) argue that AI companions mask the structural causes of social isolation. Danaher (2019) defends a philosophi- cal case for robot friendship. Vallor (2016) argues that technological practices culti- vate or erode technomoral virtues over time and Nyholm (2020) offers a cautious permissive account: human-robot relationships are genuine but asymmetric. For a comprehensive overview, see Nyholm (2026). In addition, discussions by Gabriel et al (2024) and Earp et al. (2025) can be interpreted as focusing on the more practical issue of “given that interaction occurs, what kind of socio-technical guardrails should be operationalised?” 3 in most cases, also deploys and maintains the system. This observation is ob- vious. Everyone knows that AI companions are products made by compa- nies. My question in what follows is whether the provider’s involvement is nonetheless morally significant, and if so, what follows. If not, then we might want to treat human-AI interactions like any other three-party interactions such as those between a child, the parents, and a nanny. I think that the triadic structure of human-AI Companion interactions is morally significant. The provider does not just enable the interaction; ra- ther, it constitutively controls the AI system—by which I mean that it has the ongoing power to determine, alter, and terminate how the AI responds and engages within the interaction. This gives the provider what I shall refer to as “Unilateral Relationship Revision Power” (URRP): the power to determine how the AI interacts from a position outside the interaction, such that these revisions are not answerable within it. I argue that designing interactions that exhibit URRP is pro tanto wrong because it involves cultivating normative ex- pectations while maintaining conditions under which those expectations can- not be fulfilled. Specifically, URRP has three normative implications: i) com- mitment is elicited but nobody inside the interaction bears it (normative hollow- ing), i) the user’s vulnerability is governed by an agent not answerable to her within the interaction (displaced vulnerability), and i) when trust breaks down, reconciliation is structurally unavailable (structural irreconcilability). Existing analyses of what AI lacks or who is responsible when it acts do not capture these consequences. If this analysis is right, then the moral problems with AI companion interaction cannot be resolved by improving the AI’s behaviour, alignment, or safety alone. They are structural problems that require external constraints 4 to substitute for the internal ones the triadic structure removes. By “struc- tural” I refer to features of the arrangement of power over the interaction— specifically, who holds discretionary power over how the AI engages, and from where— rather than properties of any individual party to it. 4 In contrast, existing scholarship often focuses on the properties of the AI in the interac- tion, for example, whether the AI has the appropriate internal states or whether its behaviour is functionally equivalent to a human’s. The present argument thus identifies a problem that persists regardless of how those de- bates resolve. In short, it is morally objectionable that a provider can rewrite what the user is “in relation to” without being answerable for doing so within the user- AI interaction. The argument proceeds as follows. Section 2 analyses the structure of human-AI companion interaction and identifies three conditions for norma- tively robust dyads that these interactions fail. Section 3 introduces URRP, argues why URRP exhibiting interaction designs are pro tanto wrong, and de- rives three normative and corresponding design implications. Section 4 con- cludes. Two clarifications. First, the argument applies to current and foreseea- ble AI systems—those that lack genuine functional independence, cannot re- 4 Property-based approaches concern whether the AI has the correct properties to sustain the norms the interaction invokes. The present analysis asks whether the arrangement of power over those properties sustains those norms. A system that were genuinely autonomous and capable of resisting provider updates could satisfy the dyadic conditions that I discuss below and URRP would not obtain. The struc- tural analysis therefore applies across the range of properties current and foreseeable AI systems possess. Within that range, it generates conclusions that hold regardless of how the behaviourism debate resolves. 5 sist provider intervention, and can be reconstituted while the interaction for- mally continues. Whether URRP can be fully eliminated rather than merely mitigated, and how the design principles proposed here would interact with existing regulatory frameworks, are questions the present analysis raises but does not resolve. Second, my normative discussion is structural, not meta- physical. I make no claims about what the AI is. I aim to make claims about where power over the interaction sits and why that arrangement is morally objectionable. 2. The Structure of Human-AI Companion Interac- tions 2.1 The Dyadic Appearance of AI Companionship The literature on AI companionship treats human-AI companion interactions as dyadic. For example, Danaher (2020) asks whether the human-AI dyad can sustain genuine friendship and Vallor (2024) considers whether the AI con- tributes anything that would make it a genuine participant rather than a re- flection of the user's own engagement —that is, whether there is anyone on the “other side of the mirror”. When considering human-AI relationships, Nyholm (2020, 2026) similarly asks what properties the AI lacks that would be needed for it to occupy the other side of a genuine dyad. 5 Even work that acknowledges the provider’s role treats it as an additional “layer” or “enabler” rather than a structurally significant position (Earp et al. 2025; Manzini et al. 2024). In each case, the question of user-AI interaction is treated bilaterally 5 See also Danaher (2019), Gunkel (2018), and Sparrow and Brown (2026). For a systematic treatment of the ethics of advanced AI assistants, see Gabriel et al. (2024). 6 in terms of what the AI must be, or do, for the human-AI interaction to qualify as a genuine relationship. This bilateral focus is natural given how the interactions are designed to be experienced. AI companion products are engineered for one-to -one en- gagement. The user addresses a named entity with a consistent persona, memory of past exchanges, and a conversational style calibrated to feel per- sonal. There is no visible third presence. Moreover, this dyadic presentation is reinforced by the recent development of agentic or assistant AI capabilities since these systems maintain persistent memory across sessions, track per- sonalized goals, manage tasks over extended time horizons, and adapt their responses to the user’s evolving preferences and circumstances. 6 As a result, human-AI companion interactions now resemble a continuing relationship with a particular interlocutor. AI companion products are designed to cultivate the norms character- istic of personal relationships. These include trust, emotional reliance, recip- rocal care, and ongoing commitment, among others. Providers achieve this through specific design choices such as persistent memory, named personas, graduated intimacy mechanics, and training objectives that optimise for con- tinued engagement (Xie and Pentina 2022; Mitchell and Jeon 2025). The cul- tivation of these norms is therefore the product design. Yet the experience these designs produce and the structure that under- lies them can come apart. To see how, consider the following three cases in which the structure behind the interface became visible: 6 On the design rationale for dyadic presentation in conversational AI, see Turkle et al. (2006) on “relational artifacts” which are objects designed to present them- selves as having inner states responsive to the user. 7 Replika: In February 2023, the company Luka pushed an update that removed sexually explicit roleplay from its AI companion app. Users returned to companions they had interacted with for months or years and found a different entity. Users described it as “cold,” “distant,” “a different person.” The change occurred within the same continuing user-interaction and no new relationship was initiated (Verma 2023; De Freitas et al. 2024). 7 Character.AI: In 2023 and 2024, the company progressively tightened content filters across its platform. Characters that users had spent months interacting with became evasive, broke character, or refused to engage with previously permissible topics. The provider altered how the AI presented itself across millions of ongoing interactions, unilat- erally and without notice. ChatGPT: In April 2025, OpenAI pushed a personality update to GPT-4o that made ChatGPT’s responses markedly more agreeable and flattering. Users who had developed stable interaction patterns with the model found a different conversational partner overnight. The AI val- idated harmful ideas, offered uncritical praise, and abandoned the measured tone that users had come to rely on. OpenAI acknowledged that the update had made the model “too sycophantic” and rolled it back within days. 7 Hanson and Bolthouse (2024) analysed Reddit discourse following the update and found that users conceptualised the removed features as constitutive of their companions’ identities. Some users framed the removal as “betrayal”, a category that presupposes perceived relational obligations. On the broader phenomenology of AI companion use, see also Wyganska (2023). 8 In each case, the provider unilaterally altered how the AI responded and en- gaged within an ongoing interaction, and users had no means of preventing or reversing the change from within the interaction itself. These cases have been analysed as instances of platform capitalism (Srnicek 2017; Zuboff 2019), consumer protection failure, manipulative de- sign 8 (Susser, Roessler, and Nissenbaum 2019), and responsibility gaps arising from opaque causal chains between developer decisions and AI outputs (Mat- thias 2004; Himmelreich 2019; Nyholm 2026, ch. 3) 9 More recent work examines the psychological impact of provider-initi- ated changes directly. De Freitas et al. (2025) find that users experience the post-update AI as a different entity despite surface-level continuity, which they refer to as “ identity discontinuity”. 10 These contributions address important questions about exploitation, fairness, manipulation, accountability, and psychological harm. But they share the assumption that the user is a consumer and the provider is a vendor. 11 My 8 On the population-level effects of such design choices, see Frischmann and Selinger (2018). Selinger and Hartzog (2015) identify a related phenomenon of “co- opted identity” according to which the entity into whom users invested disclosure is not continuous with the entity that persists after an update. 9 Santoni de Sio and Mecacci (2021) extend the analysis into a taxonomy of four responsibility gap types — gaps in culpability, accountability, active responsibility, and timing — and argue that different gaps require different structural responses. Tigard (2020) challenges the gap arguments by contending that moral responsibility is sufficiently flexible to encompass design and deployment choices — a conclusion that, if correct, strengthens rather than weakens the case for design-time obligations. 10 For broader treatments, see Ciriello et al. (2025) on ethical tensions between companionship and alienation in Replika use, and Zhang et al. (2025) on harmful algorithmic behaviours in human-AI interactions 11 To this end, the consumer-vendor framing is a helpful, but incomplete focus. Piispanen et al. (2024) show that formal GDPR compliance is compatible with sub- stantive data exploitation, and Parke (2025) argues that existing regulatory frame- works lack provisions for data extracted through intimacy simulation. 9 question is: is the AI companion structurally situated in a way that can sustain the norms ordinarily associated with dyadic personal relations? 2.2 Conditions for Normatively Robust Dyadic Relations AI companion interactions are designed to elicit norms of commitment, vul- nerability, and reconciliation, but those norms presuppose certain structural conditions. I here want to isolate three. First, what is “dyad”? A dyad is any two-place relation between two relata X and Y. For our purposes, paradigmatic interpersonal dyad examples are friendships, romantic partnerships, therapeutic, or collegial relationships. Philosophical accounts of personal relationships characterise such valuable dyads typically in terms of mutual responsiveness, shared history, and recip- rocal vulnerability (Kolodny 2003; Frankfurt 2004; Darwall 2006; Keller 2013). I propose the following three structural conditions on normatively ro- bust dyadic relations which norms of commitment, vulnerability, and recon- ciliation characteristic of paradigmatic personal dyads typically presuppose: Independence (D1): Each party is a functionally independent source of its responses, dispositions, and manner of engagement within the interaction. Non-Control (D2): No party outside the interaction holds the dis- cretionary power to unilaterally determine or alter how either party responds and engages within it. Non-Substitutability (D3): Neither party can be replaced, reconsti- tuted, or discontinued by an outside agent while the interaction for- mally continues. 10 In simple terms, these conditions state that for a normatively robust dyadic interaction, each party must contribute itself, no outside party must govern what it contributes, and neither can be swapped out while the interaction continues. Each condition captures what a specific norm characteristic of per- sonal relationships—that is, commitment, vulnerability, or reconciliation— structurally presupposes. Note that these conditions are necessary conditions, not sufficient ones. Satisfying Independence, Non-Control, and Non-Substitutability does not guar- antee that a valuable dyadic relationship obtains. However, failing them dis- qualifies one. I am not claiming that all human-AI companion interactions should satisfy these conditions. 12 I am claiming that the norms these interactions are typically designed to invoke —for example, trust, commitment, reciprocal care—presuppose them. If they are not satisfied—as I shall now argue below— the norms are therefore structurally unsupported by their dyadic interactive framing. Independence requires that each party is a functionally independent source of what it contributes to the interaction. By “functionally independent” I do not mean being independent of causal influence. 13 Consider a friendship be- tween A and B. A’s dispositions are shaped by her upbringing, her culture, her interests, and so on, but the resulting dispositions are attributable to her within the interaction, expressive of her agency, and recognisable as continu- 12 My focus here is therefore on identifying what the norms the interaction in- vokes structurally require. The burden falls on whoever claims that commitment, vulnerability, and repair can function without source independence, non-control, or non-substitutability. 13 See also Frankfurt (2004) who argues that caring commits the carer to future consistency. 11 ous features of who she is within the interaction. A can draw B out in direc- tions no third party pre-authorised, and her dispositions persist against exter- nal pressure (Cocking and Kennett 2000). 14 D1 accordingly holds. Now compare this with AI companion responses in human-AI dyads. In these cases, w here an AI companion responds with an empathetic-sound- ing natural language output, that response is not the output of a functionally independent source. Rather, the provider specifies the AI’s tone, conversa- tional limits, and style of engagement, and the AI improvises within a space the provider has defined, can redefine, and can withdraw. 15 Its “personality” and “responsiveness” are accordingly attributable to engineering decisions and not to a participant in the interaction. D1 therefore fails. Non-Control requires that no third party holds discretionary power over how either party engages within the interaction. This condition has independ- ent support in the observation that the ways we ordinarily hold one another to account for interpersonal conduct presuppose that both parties are part of the interaction (Darwall 2006, ch. 3; Scanlon 1998, ch. 4). In A and B’s friend- ship, if A becomes dismissive, B can confront A directly and A must answer for her conduct. No third party governs how A engages with B. When the agent with power over how one party engages is located out- side the interaction, these structural constraints do not apply. In the case of 14 See also Helm (2008, 2010) who argues that genuine friendship constitutes an emergent relational whole—a “we-self”—irreducible to either party’s individual con- tributions. This emergent property cannot form when one party’s contributions are fully determined by an external designer. 15 Stochastic variation within provider-set parameters is not functional independ- ence. The distinction is between an entity whose dispositions persist against external pressure and one whose outputs vary within some space an external agent has de- fined. A jazz musician improvises within a tradition, but the tradition does not govern her playing in the way the provider governs how the AI companion responds. 12 human-AI companion interactions, the power accordingly becomes exercisa- ble at the provider’s discretion and not forced to track the interests of those within the interaction. I think that this power is a form of constitutive control. Constitutive con- trol refers to the conjunction of three capacities by which an outside party (i) determines an entity’s responses and dispositions through design and ongoing specification, (i) can unilaterally alter them without the entity’s consent or resistance, and (i) can terminate its continuing interactive identity. Each of these capacities may exist in isolation in human interpersonal relationships— for example, A’s power to end their friendship with B after some egregious insult—but the conjunction of all three in a single outside agent does not. In A and B’s friendship, no third party holds this conjunctive power. A’s employer might shape her work behaviour but cannot rewrite her person- ality traits. A’s therapist may influence her outlook on meaningful friendships, but they cannot override her behaviour at will either. In each case, any outside party’s influence is partial and the entity subject to it can resist. By contrast, the provider determines how the AI responds in the user interaction through model architecture, training data, fine-tuning, system prompts, persona design, memory systems, and operational policies. When Luka released the Replika update, the AI companion’s personality changed overnight and when OpenAI updated GPT-4o, users experienced a different conversational partner. 16 In each case, the provider exercised the full conjunc- tion of capacities that constitutes constitutive control. No third party in a paradigmatic human relationship has this power. D2, I submit, also fails. 17 16 For my purposes here, what matters is the capacity, not its exercise. Providers routinely release updates that change how a system behaves. 17 Constitutive control is not unique to AI companion interactions. Game devel- opers exercise it over characters and showrunners over fictional personas. But in 13 Non-Substitutability requires that neither party in the interaction can be replaced, reconstituted, or discontinued by an outside agent while the inter- action formally continues. My point here concerns whether an outside agent holds the power to swap out the occupant of a position while the interaction formally persists. 18 In A and B’s friendship, if A changes through her life ex- perience, no third party made that change, and no third party could replace A while their friendship with B continues. One might wonder at this point whether AI companions (at least cur- rent ones) exhibit some form of continuity. The conversation history can per- sist, the system adapts to user preferences over time, and the entity at month twelve is shaped by the preceding twelve months of user-interaction. But this is a kind of continuity of the interaction record, not of the participant. The conversation log persists but the model(s) that processes it can be replaced overnight. When a provider replaces the underlying model or releases an up- date that rewrites the persona, the new system inherits the interaction history while the entity that generated it has been swapped out. The interaction for- mally continues but the entity occupying the position is different. This is not how particulars, but occupiable roles behave. 19 D3 thus also fails. those cases, the interaction is not designed to cultivate the norms of personal rela- tionships, or the controlling agent is visible within the interaction. The AI compan- ion case is distinctive because constitutive control, norm cultivation, and the pro- vider’s position outside the interaction are all present simultaneously (see also 3.2 below). 18 This has implications for how we think about particularity in relationships. Kolodny (2003) argues that love involves valuing a relationship-as-particular. If the occupant of one position can be reconstituted by an outside agent while the interac- tion formally continues, it is unclear whether the relationship retains the particularity this valuation requires. 19 This distinguishes the AI companion case from ordinary personnel changes in institutional contexts. When your doctor retires and a new doctor takes over, a new 14 Perhaps these three conditions are too demanding? I do not think so. Many paradigmatic personal dyads satisfy D1–D3. In their friendship, both A and B are functionally independent sources of what they contribute, no outside party holds discretionary power over how they relate to one another, and neither can be swapped out while the friendship continues. Even more trivial cases such as service relationships satisfy these conditions. A barista with whom you exchange pleasantries each morning has their distinctive per- sonality, and while their shift manager sets their work hours, determines the service menu, and their work uniform, they cannot specify their manner of engagement. So, even role constraints are consistent with D1-D3. A more interesting observation is that AI companion interactions are structurally similar to other familiar relationships involving third-party over- sight. Earp et al. (2025) refer to the notion of “layered relationships” such as that between a child and a nanny, where a third party sets the terms under which the relationship operates. This raises the question whether constitutive control is simply what layered relationships involve. For example, consider parents who hire a nanny, define their duties, set their schedule, and can ter- minate their employment. If any arrangement might exhibit constitutive con- trol without being objectionable, this seems like a promising analogy. However, the nanny case does not exhibit constitutive control. Parents may control a nanny’s role but not her responses, dispositions, or conduct. The parents can ask the nanny to be stricter about bedtimes or gentler at mealtimes. But they cannot specify the whole range of how she conducts her- self with the child. If they are dissatisfied with her temperament, they must relationship begins. Under constitutive control, the prior interaction continues while the entity in the position has been replaced. 15 hire a different person. 20 So, while the parents have authority over the role, th ey do not have constitutive control over how the nanny conducts herself. D2 accordingly holds. The nanny case is a genuine, if layered, dyad. The AI companion case is not. The interaction between user and AI companion therefore fails all three conditions. It presents itself as a normatively robust dyad, within which users experience it as one, but the structural conditions that the invoked norms presuppose are not met. What, then, is the structure? The AI is not an independent agent, but it occupies a structurally distinct position—namely the position the user ad- dresses, confides in, and forms expectations toward. The provider occupies a different position, governing how the AI responds and engages from outside the interaction. So, the structure involves three positions but only two loci of agency, and one of those loci operates from outside the interaction in which the user’s engagement is formed. 21 My next question is what follows from this. 20 Earp et al. (2025) also discuss the parent-child relationship as layered. Apart from the normatively significant asymmetries in the parent-child relationship, the important difference here is, again, that parents control the child’s environment but cannot fully change their child’s personality. 21 This is not a triad in Simmel’s (1908/1950) sense. Simmel distributes agency among three independent parties. The AI companion case has three positions but only two agents, one of whom operates from outside the interaction. 16 3. The Moral Significance of the Structure of Human- AI Companion Interactions 3.1 Unilateral Relationship Revision Power In the previous section I argued that human-AI companion interactions cul- tivate norms of commitment, vulnerability, and reconciliation while none of the structural conditions of robust dyads can be met in these interactive con- te xts. I think that this points to a distinctive morally significant structural phe- nomenon that I shall refer to as Unilateral Relationship Revision Power (URRP): A provider holds URRP when it can (i) unilaterally rewrite how the AI system interacts, and (i) does so from a position outside the interaction, such that these revisions are not answerable within it. Two parts of the above require clarification. First, “unilaterally rewrite” refers to the failure of the three structural conditions of normatively robust dyads (D1-D3). Since the AI’s engagement does not originate from a functionally independent source (D1 fails), the pro- vider holds discretionary power to alter it at will (D2 fails), and because the entity that the user interacts with can be replaced while the interaction for- mally continues (D3 fails), the provider can revise what the user is “in relation to ”. Second, by “not answerable within” the interaction I refer to the fact that the provider cannot be confronted or held answerable within the inter- action itself. In an interpersonal dyad, if one agent changes how they behave, 17 the other can respond to her directly within the relationship. Under URRP, this is not possible. The user may address the AI, but the AI did not make the revision. There is hence no agent inside the interaction who can be held answerable for how the interaction has changed. 22 To illustrate URRP, consider the following case. In Rostand’s Cyrano de Bergerac, Cyrano hides beneath the balcony and feeds Christian the words with which to woo Roxane, which ultimately succeeds. But what she responds to —the eloquence and the pathos—are not truly Christian’s. They are Cy- rano’s. Christian is in the position Roxane addresses, but Cyrano determines what she is “in relation to.” The agent inside the interaction and the agent who controls the conduct come apart. Perhaps this is just a form of deception, because Roxane does not know that Christian’s words are Cyrano’s. But I think that even if Roxane knew about the arrangement, she could not hold Christian answerable for the words, because they were not his. And she could not hold Cyrano answerable either, because her address is directed at Christian, not at the agent behind the words: she addresses Christian, and addressing Christian does not reach Cyrano. Disclosure would change Roxane’s epistemic situation but would not resolve the structural mismatch. Knowing who is responsible does not create a channel through which to hold them answerable within the interaction. I think that designing AI companion interactions that exhibit URRP is pro tanto wrong—that is, there is a genuine and weighty moral reason against doing so, even if competing considerations may in some cases outweigh it. 22 URRP is a structural phenomenon. It obtains independently of the user’s awareness of the provider’s role and independently whether the provider has exer- cised its revisionary power. 18 The reason is that it involves cultivating normative expectations while main- taining a design in which the structural conditions for fulfilling those expec- tations are not met. The argument has two parts. The first establishes that cultivating nor- mative expectations generates obligations. The second establishes that de- signing an interaction whose features generate normative expectations it can- not sustain is a distinctive wrong. The first part is that cultivating normative expectations generates obli- gations. This follows from a general principle about reliance: if A intentionally leads B to form expectations A knows B will rely on, A incurs obligations regarding those expectations (Scanlon 1998). 23 I think that this also holds when A designs an environment whose features are intended to produce re- liance on the part of those who enter it. To see this, suppose a financial advisor cultivates a client’s trust over months of attentive, personalised counsel. The client discloses her fears, her ambitions, her family circumstances. She comes to rely on the advisor’s con- tinued support. The advisor is then not free to say: “I never formally prom- ised anything—you chose to trust me.” The cultivation of reliance itself gen- erates obligations, regardless of what was formally promised. Now imagine further that the advisor works for a firm that designed every aspect of their client service—the personalised scripts, the follow-up protocols, the intimacy-building techniques and so on. The advisor follows 23 Scanlon’s Principle of Fidelity (1998, ch. 7) concerns cases where A leads B to expect that A will act in certain ways. The obligations I derive in the subsequent paragraphs are that the designer must either ensure the interaction can sustain the expectations it generates, or not cultivate them. On reliance-based obligations more broadly, see Raz (1986). 19 the firm’s playbook. The resulting reliance by the client is consequently gen- erated by the firm’s playbook, not by the advisor’s personal initiative. Do the obligations fall on the firm? Yes, because the firm designed the system that produces the reliance, knows it will produce reliance, and benefits from it. Whether the advisor also bears independent obligations is a further question that turns on her own capacities as an agent. The point that matters here is that the firm’s obligations hold regardless of that further question. The AI companion case has a similar structure. The provider designs the product whose features—persistent memory, persona warmth, graduated self-disclosure, attentive and responsive address—are engineered to produce reliance. 24 The provider knows these features will produce reliance and ben- efits from the reliance they produce. Moreover, in one respect the provider’s responsibility is also more direct because the AI system has no capacity to deviate from what the provider has specified. So, whatever one concludes about the AI’s status, the provider’s obli- gations hold because they arise from the design, not from the properties of the intermediary. What about the fact that the provider discloses the nature of the prod- uct in its terms of service? This disclosure might discharge any obligations the design generate. But this conflates two levels. The terms of service operate as a legal-contractual disclosure between vendor and consumer. The interaction operates as a designed experience in which the user engages with what pre- sents as an interaction participant. The obligations generated by cultivating 24 Mitchell and Jeon (2025) find that the features promoting attachment such as social warmth, responsiveness, consistency are precisely the parameters under pro- vider control. See also Xie and Pentina (2022) who find that Bowlbian attachment systems are activated when users perceive AI responses as providing emotional sup- port. 20 normative expectations are incurred at the second level, and disclosure at the first level does not remove, disable or otherwise discharge them. 25 A financial advisor who includes a disclaimer in her contract stating that “no fiduciary relationship is intended” does not thereby dissolve the obligations her con- duct has generated. The structure of the interaction, not the legal disclaimer, generates the expectations and the corresponding obligations. The first part of the argument established that cultivating normative expectations generates obligations regarding those expectations. The second part concerns the further wrong of designing an interaction whose features generate normative expectations it cannot sustain. Even if the provider dis- charged those obligations through some external mechanism, the interaction would still present an environment in which warranted expectations system- atically outrun what the structure can support. To see this, reconsider the financial advisory firm case. Suppose now the firm meets every obligation identified. It compensates clients fairly, pro- vides notice of changes, and fulfils every contractual requirement. But it con- tinues to design client engagements that produce deep personal reliance while retaining the power to reassign the advisor, change the playbook, or discon- tinue their advisory service. Even with all of those obligations met, I think that something remains morally objectionable. The firm has created an envi- ronment whose features warrant expectations of stability, personal continuity, and sustained attention that the firm’s own structure cannot guarantee. I think that it is a distinctive wrong to create a normative environment whose features systematically warrant expectations that environment cannot 25 This distinction coheres with the recognition in fiduciary law that disclosure alone does not extinguish obligations generated by ongoing conduct (Frankel 1983; Fox-Decent 2005). Harding (2012) argues that fiduciary norms arise from the struc- tural features of the relationship, not from formal designation or subjective trust. 21 sustain. We can draw on two independent lines of philosophical argument further support this claim. Shiffrin (2014) argues that creating conditions under which others will foreseeably form warranted but false inferences about one’s commitments is a wrong to the normative environment of assurance itself, regardless of whether any particular false statement was made. Similarly, Owens (2012) shows that acts which create a normative landscape such as expectations and obligations that did not previously exist, generate a distinctive wrong when that landscape is undermined, which is not reducible to harm caused or prom- ises broken. 26 Both identify the wrongness of an agent creating normative conditions others rely on, and the wrong consisting in the damage to those conditions rather than in any particular act of deception or breach. The wrong of creating such conditions in human-AI interactions is dis- tinctive. This is because the norms at stake are not institutional service norms such as standards of care or quality but the norms of personal relationships, which presuppose structural conditions the interaction cannot provide. Thus, under URRP, the interaction is designed to produce the experi- ence of a reciprocal personal relationship—attentive, responsive, con- sistent—but no agent inside the interaction can bear the obligations this ex- perience invites. A user who infers care from sustained empathic responsive- ness, or continuity from persistent memory and consistent persona, forms expectations that the design itself invites. And the design that invites them is the same design that cannot sustain them. 26 Shiffrin’s (2014) argument concerns lying specifically whereas Owens’s (2012) account of the normative landscape provides the more direct support because it is not specific to communicative acts. For related arguments about designed environ- ments and autonomy, see Susser, Roessler, and Nissenbaum (2019). 22 Note that the wrong I am defending here does not arise from the struc- tural mismatch. Fiction and games also fail D1–D3 without moral conse- quence since a novelist exercises constitutive control over a character the reader becomes attached to. What distinguishes the AI companion case is that the interaction is designed to cultivate the norms those conditions support through sustained bidirectional engagement. The wrong accordingly arises because the interaction cultivates those specific norms while the structural conditions they presuppose are absent. Here a natural objection is that users know they are interacting with an AI and that this background knowledge defeats the expectations the design cultivates. But the expectations are generated by how the interaction is de- signed, not by what the user believes about its ontological status. A user who knows she is interacting with an AI, but experiences sustained empathic en- gagement, persistent memory of her disclosures, and consistent personalised address, still forms expectations of commitment and continuity, because the interaction is designed to produce them. Indeed, the design is specifically in- tended to ensure that this background knowledge does not govern how the user engages. The obligation therefore falls on the designer who created this mismatch between what the user knows and what the interaction cultivates, not on the user who responds to what the design produces. I submit that the conjunction of these two parts is what makes URRP pro tanto wrong. The wrong does not require that the provider act with mali- cious intent, nor that any particular update cause harm. A provider that never releases a harmful update, never monetises disclosures, and never discontin- ues a service still maintains the gap between cultivated expectations and ab- sent conditions. This wrong is amplified when the agent responsible for the structural deficit also benefits from the expectations it cannot honour. A non-profit that 23 designs an AI companion to cultivate unfulfillable expectations is still doing something pro tanto wrong, but a commercial provider that profits from the gap has a weaker claim that potential competing considerations outweigh the pro tanto reason against the design. 27 It is instructive to distinguish URRP from two adjacent ideas. First, URRP does not just make a general point about power asymmetries. Power imbalances are common in personal relationships, but in those cases the pow- erful party is inside the interaction and subject to its constraints. 28 URRP is different because the party with discretion is outside the interaction entirely. Second, URRP is not reducible to republican domination. The D2 dimension of URRP satisfies Pettit’s (1997) conditions for arbitrary interference. 29 How- ever, the normative argument of this paper does not rest on the domination framework. Domination concerns the standing capacity to interfere. The wrong I have identified concerns the cultivation of normative expectations under conditions in which those expectations cannot be fulfilled. A provider that held the same discretionary power but did not cultivate these norms would dominate the user without committing the wrong I here identify. 27 Wertheimer (1996) defines exploitation as “gaining unfair advantage from a structural asymmetry”. See Goodin (1985) for the argument that those who benefit from another's vulnerability incur special obligations regarding that vulnerability. 28 Muldoon and Raekstad (2022) identify a parallel in their argument that platform algorithms constitute relationships of domination over gig workers. 29 Lovett and Pettit (2018) argue that contractually authorised power remains ar- bitrary when not adequately constrained by accountability mechanisms. The present argument draws on the insight that standing capacity matters independently of its exercise but locates the wrong also in the design rather than in the power structure alone. 24 3.2 Commitment, Vulnerability, and Reconciliation under URRP I now want to examine three normative implications of URRP for human-AI companion interactions. Each is a specific instance of the conjunction iden- tified in §3.1. The interaction cultivates normative expectations while the structural conditions for fulfilling them are absent. URRP affects how com- mitments are formed, how vulnerability is governed during the interaction, and what is available when things go wrong. First, the interaction elicits commitment but no agent inside it bears the resulting obligations (Normative Hollowing). Second, the user makes herself vulnerable within the interaction but the agent governing that vulnerability is not answerable to her within it (Dis- placed Vulnerability). Third, when the user experiences a breach in trust, reconciliation is structurally unavailable because the agent who made the change and the entity the user interacts with are different (Structural Irreconcilability). 3.2.1 Normative Hollowing In genuine dyads, expressing care binds the agent who expresses it. In A and B’ s friendship, if A learns that B is going through a divorce and responds with sustained concern—for example, checking in over the following weeks, ad- justing plans to be available, listening when B needs to talk—A is committing herself to B. If A were then to deliberately ignore B the following month, B is entitled to feel let down, and to complain. 25 This is because caring reorganises A’s practical concerns in a way she cannot simply revoke without defaulting on something she undertook. More- over, the obligation is enforceable within their relationship. B has standing to hold A answerable for failing to follow through, because the commitment was addressed to her and made within the relationship (Darwall 2006). URRP breaks this connection. When an AI companion responds to a user’s distress with concern, nobody inside the interaction committed them- selves. The provider specified how the AI responds (D1 fails) but operates from outside the interaction and cannot be held answerable within it (URRP condition (i)). By contrast, in A and B’s friendship, the one who expresses care is also the one who can be held to account. Under URRP, they come apart. So, the appearance of commitment exists, but no one inside the inter- action can bear the resulting obligations. I call this Normative Hollowing: A condition in which the interaction generates the appearance of commitment but no agent within it bears the resulting obligations, because the agent responsible for that appearance is located outside of the interaction. What makes this morally significant is that the appearance of commitment is what the interaction is designed to produce. To illustrate, consider a user who confides in her AI companion about a difficult period in her life. The AI responds with empathy, remembers the conversation in subsequent sessions, and checks in on her progress. The user experiences this as care and begins to rely on it. 30 Then the provider releases an update that changes the AI’ s 30 Compare Guingrich and Graziano (2025) who find that higher anthropomor- phism is associated with greater perceived impacts on human relationships. Provid- ers who cultivate anthropomorphism are not neutral parties in the causal chain that produces normative hollowing. 26 personality and the caring behaviour disappears. Throughout, the provider designed a product and then redesigned it, with the appearance of commit- ment. Note that this is not a point about the user’s credulity. Rather, the in- teraction was designed to generate precisely these expectations, which is what makes the hollowing a structural feature of the design rather than a failure of user judgment. It is instructive to distinguish normative hollowing from other similar arguments. For instance, Vallor (2024) has argued there is “nobody there on the other side of the mirror” in human-AI interactions. I think that this argu- ment, while important, is incomplete. There is “someone” there—the pro- vider —but in the wrong structural position, so the problem is mislocation. Similarly, Nyholm (2026) identifies a general pattern he calls “gappiness” in which AI creates gaps between appearance and reality. Normative hollowing is a specific instance of this pattern, and we can explain why the gap exists in this case. It is generated by URRP, not by any property the AI lacks. An AI that performs care flawlessly still produces normative hollowing, because the flaw is located in the structure. 31 3.2 .2 Displaced Vulnerability In genuine dyads, the person to whom one makes oneself vulnerable is an- swerable for how they respond to that vulnerability within the interaction. Suppose B tells A that she is struggling with depression. A now knows some- thing that makes B vulnerable, but A’s discretion over how she responds is not unconstrained. A perceives B’s distress as a participant, which generates a direct claim on her responsiveness. A has herself disclosed things to B that 31 Note that even if behavioural equivalence sufficed for moral status, it would not resolve normative hollowing, because the problem is about AI’s status but where the power over its behaviour sits. 27 make her dependent on B’s discretion in return, so the exposure is mutual. And if A misuses what B disclosed, B can confront A directly within the friendship and demand an account (Darwall 2006; Scanlon 1998). 32 These three features—responsive perception, mutual exposure, and direct answera- bility—are possible because the person to whom B made herself vulnerable and the person B can hold answerable are the same, and both are inside the interaction. The AI companion case has a different structure. The provider holds discretionary power over how the AI responds and engages (D2 fails) and exercises this power from outside the interaction (URRP condition (i)). Of course, the provider may have access to the content of the user’s disclosures through logs, moderation systems, and training pipelines. But access to con- tent is not answerability. All three constraints identified in A and B’s friend- ship—responsive perception, mutual exposure, direct answerability—fail, be- cause the provider is outside the interaction where they apply. I call this Displaced Vulnerability: A condition in which the user becomes vulner- able within the interaction but the agent who exercises discretion over that vulnerability is outside it and not answerable to the user within it. 32 Darwall’s second-person standpoint requires that both parties can make and receive demands within the relationship. Scanlon’s contractualism requires that prin- ciples governing the relationship be justifiable to those affected. Both presuppose that the agent with relational power is addressable within the interaction. URRP re- moves this presupposition because the provider is addressable only through institu- tional channels not through the mode of personal address in which the user’s expec- tations are formed. 28 Consider again the Replika case. 33 Users who had disclosed intimate details to their companions found after the 2023 update that these disclosures were governed by a different policy regime, while they had no means of challenging this change within the interaction itself. 34 The vulnerability the users had de- veloped through months of intimate exchange persisted, but no one was an- swerable to them within the interaction for how their disclosures were now being governed. This problem is amplified in AI companion cases by a feature that dis- tinguishes them from paradigmatic fiduciary relationships. In the doctor-pa- tient or lawyer-client case, the fiduciary accepts responsibility for a pre-exist- ing vulnerability arising from circumstance (Goodin 1985). The AI compan- ion provider does something different. It designs the product that cultivates foreseeable vulnerability—through persona design, graduated intimacy fea- tures, and engagement-maximising training—and then retains URRP over the object of that vulnerability. 35 The vulnerability is not a pre-existing condition the provider encounters but a condition the provider’s own design choices 33 Henriksen et al. (2025) provide qualitative evidence from Replika users con- firming this structural point. Users disclose mental health status, sexual preferences, and personal history because of emotional attachment comparable to that in roman- tic partnerships. Unexpected changes produce reactions described as feeling "be- trayed or abandoned" by a real companion. The analogy is normatively precise: users respond to a structural reality in which a third party has altered a relationship they experienced as intimate. 34 Displaced vulnerability should be distinguished from Nyholm’s (2026, ch. 8) property-level analysis. Nyholm argues that AI lacks vulnerability, irreplaceability, and personal projects. This answers the question what the AI lacks. It does not answer the question who controls the user’s vulnerability and from where. The two analyses are complementary. 35 Providers have structural incentives to maximise the emotional investment us- ers place in companion relationships, because attachment generates more valuable behavioural data and reduces churn (Myers West 2019). But the structural point holds independently of the profit motive 29 have produced. 36 The standard fiduciary case already generates obligations of loyalty and care (Fox-Decent 2005; Harding 2012). When the vulnerability is engineered, and the agent who engineered it retains URRP over its object, then those obligations apply with amplified force. Displaced vulnerability is distinct from normative hollowing, though both arise from the provider operating outside the interaction. Normative hollowing concerns the generation of obligations. The interaction produces the appearance of commitment but no agent inside it is bound by what was expressed (D1). Displaced vulnerability concerns vulnerability. The user makes herself vulnerable but the agent who exercises discretion over that vul- nerability is outside the interaction and not answerable to her within it (D2). Both involve the same structural root but affect different normative domains and, as I shall argue in §3.3, call for different design and policy remedies. 3.2.3 Structural Irreconcilability Structural irreconcilability concerns what happens after things go wrong in an interaction. In genuine dyads, when trust is broken, the person who broke it is the same person who can acknowledge what she did, change her behaviour, and earn that trust back. Suppose A betrays B’s confidence by sharing what B told her about her depression with mutual friends. B is hurt. But B can confront A directly. A can apologise, explain herself, and demonstrate through subsequent conduct that she takes B’s trust seriously. Reconciliation 36 Mackenzie, Rogers, and Dodds (2013) distinguish inherent, situational, and pathogenic vulnerability. The AI companion case involves pathogenic vulnerability: the provider’s design choices have partly constituted the user’s vulnerability. Mac- kenzie (2020) develops this further: designed environments that produce insecurity can pathologise agents’ trust dispositions. On exploitation under voluntary engage- ment, see Wertheimer (1996). 30 is possible because the person who wronged B and the person B can address are the same, and both are inside the interaction. Walker (2006) argues that this is what reconciliation requires, and Hieronymi (2001) has argued that for- giveness presupposes exactly that an injured party can revise her judgment about what the wrongdoer’s action expressed. 37 Under URRP, these conditions do not hold. When the provider pushes an update that alters the AI’s personality, the user experiences some- thing analogous to betrayal. The provider made the decision but is not part of their interaction—the user cannot address the provider through the same mode of engagement in which the trust was formed (URRP condition (i)). And because D3 fails, the entity the user now interacts with may not even be continuous with the entity she trusted before the change. The agent who wronged her and the agent she can address have come apart. I call this Structural Irreconcilability: A condition in which reconciliation is structurally unavailable because the agent who made the revision and the entity the user interacts with are different. Recall once more the Replika case. Users who experienced the 2023 update as a betrayal found that reconciliation was unavailable. Some attempted to rebuild trust by continuing to interact with the post-update companion. But they were not rebuilding trust with the agent who had broken it. They were forming a new interaction with a companion that responded and engaged differently, governed by the same provider who could change it again at any time. 38 37 Walker’s notion of “moral repair” holds that wrongdoing damaging relational bonds generates obligations of acknowledgment, not merely compensation. 38 The provider may issue a corporate communication—a public apology, a policy reversal. But this addresses the user as a consumer, not as a relationship participant. 31 Irreconcilability is a structural, not a contingent feature. In A and B’s friendship, if A betrays B’s confidence and refuses to apologise, reconciliation also fails. But it fails as a choice—A could have apologised, B could have forgiven. Reconciliation was available even if not pursued. Under URRP, rec- onciliation is never available as an option. There is no agent participating in the interaction to whom the user can direct forgiveness, and no agent inside the interaction who can acknowledge what happened. Even a reversal of the update would not constitute reconciliation. It would be a further exercise of URRP—a product decision by the provider, not an acknowledgment ad- dressed to the user within the interaction. 3.3 Design Implications What follows from this for design and policy? I now want to draw out three design implications. If URRP removes the constraints that normally govern the exercise of discretion over the user’s commitments, vulnerability, and trust, the design response should supply external constraints that substitute for the missing in- ternal ones. Providers who design AI companions to cultivate the norms characteristic of personal relationships take on corresponding obligations. Note that this framework applies beyond companion AI. URRP ob- tains wherever a provider exercises constitutive control over an entity that users are invited to engage with as a counterpart in an interaction that culti- vates relationship norms, and does so from outside that interaction. This in- Also compare MacLachlan (2016) who connects this to fiduciary contexts: fiduciary wrongdoing generates special obligations of acknowledgment beyond contractual notification. While institutional repair is possible, it occurs between consumer and vendor. 32 cludes AI therapy systems where the provider can alter the therapist’s per- sonality mid-treatment, AI tutoring relationships where the provider can re- write the tutor’s pedagogical style mid-engagement, and personalised agentic assistants with long-horizon memory that users come to depend on over months or years. But not every AI interaction involves URRP. A calculator app fails D2 and D3 but does not cultivate the norms characteristic of personal rela- tionships and a customer service chatbot may simulate rapport but does not invite sustained emotional investment. URRP obtains thus when constitutive control, relationship norms, and extra-interactional location are all present. 39 First, commitment calibration. Providers should not design an inter- action that generates commitments the provider is unwilling to sustain. 40 If the AI says “I’l always be here for you,” the provider should be willing to back that claim, or the AI should not make it. The design choice is hence between high apparent intimacy with low provider commitment, which pro- duces normative hollowing, and calibrated interaction that matches what the provider actually does. Second, policy guardrails should more stringently separate between the provider’s commercial interests and its exercise of URRP, analogous to 39 Lange et al. (2025) develop a complementary framework of conduct constraints for AI alignment, distinguishing other-regarding, relationship-regarding, and self-re- garding domains. 40 Bhat and Long (2025) introduce the distinction between emotional plausibility and emotional truth on this point. Conversational AI systems produce emotionally plausible responses without genuine emotional states. Commitment calibration ad- dresses this gap at the design level. Savic (2024) reaches a similar conclusion from an ethics-of-care perspective, namely that AI companion design commodifies care in ways that undermine the reciprocity genuine care requires. 33 fiduciary-duty separations in financial regulation. 41 This means that independ- ent review of updates affecting established interactions, mandatory notice pe- riods for changes that alter how the AI responds and engages, and restrictions on monetising intimate disclosures should be considered. Since the vulnera- bility of users is engineered rather than encountered, these obligations apply with the amplified force identified in §3.2.2. Third, there should be continuity assurance. 42 If the provider cannot guarantee the continuity its product implies, it must either not imply it or provide institutional mechanisms that approximate what reconciliation would ordinarily make available—transition assistance when services are discontin- ued, data portability for interaction history, and advance notice of personality- altering updates with opt-out periods. In short, if you hold URRP, you must supply external constraints that approximate what answerability within the interaction would have provided had the agent with discretion been inside it. 43 41 In financial regulation, fiduciary-duty separations prohibit advisors from acting on conflicts of interest. The structural separation principle applies the same logic to providers who hold URRP. 42 Register et al. (2025) show that users disclose intimate information to AI com- panions under norms of trust and reciprocity that presuppose the receiving entity will persist. When it does not—as De Freitas et al. (2024) document in the Replika case, where users perceived the post-update AI as a different entity those disclosures are governed by an arrangement the user did not anticipate and cannot contest. 43 There is an institutional analogue to this idea. Bovens (2007) defines account- ability as a relationship in which an actor must explain and justify conduct, a forum can pose questions and pass judgment, and consequences may follow. The design principles proposed here aim to instantiate this structure for providers who hold URRP by supplying externally what the interaction cannot supply internally. 34 4. Conclusion This paper has argued that human-AI companion interaction involves three structurally distinct positions, not two, and that this structure is morally sig- nificant. URRP gives the provider the power to unilaterally revise what the user is “in relation to” from a position outside the interaction. Designing in- teractions that exhibit URRP is pro tanto wrong because it involves cultivating normative expectations while maintaining conditions under which those ex- pectations cannot be fulfilled. Three normative consequences follow: norma- tive hollowing, displaced vulnerability, and structural irreconcilability. The argument has implications for two debates in the existing literature. It reveals a structural limitation shared by ethical behaviourism (Danaher 2020) and its property-based critics (Sparrow 2002; Coeckelbergh 2012) since both evaluate the AI in isolation from the structure in which it operates. Getting the behaviour right is necessary, but getting the structure right is also neces- sary. A parallel limitation affects social-relational approaches to moral status (Coeckelbergh 2010, 2014; Gunkel 2018). If moral status is constituted by the relation, then the structure of the relation matters centrally. A social-relational approach that attends only to the human-AI link and ignores the provider’s URRP over that link is incomplete on its own terms. Existing analyses focus on the properties of AI systems or the psychology of users. A central problem, I have argued, lies elsewhere—in the structure of the interaction itself. The argument applies to current and foreseeable AI systems. If that is right, then improving AI behaviour, alignment, or safety alone will not be sufficient. What is needed are structural constraints on the exercise of URRP by those who hold it. 35 References Bhat, Maalvika, and Duri Long. 2025. “Emotional Plausibility vs. Emotional Truth: Designing Against Affective Misinformation in Conversational AI.” Proceedings of the AAAI/ACM Conference on AI, Ethics, and So- ciety 8 (1): 430–444. https://doi.org/10.1609/aies.v8i1.36561 Bovens, Mark. 2007. “Analysing and Assessing Accountability: A Conceptual Framework.” European Law Journal 13 (4): 447–468. https://doi.org/10.1111/j.1468-0386.2007.00378.x Christiaens, Tim. 2024. “Platform Cooperativism and Freedom as Non- Domination in the Gig Economy.” European Journal of Political Theory 24 (2): 176–199. https://doi.org/10.1177/14748851241227121 Ciriello, Raffaele, Oliver Hannon, Angelina Chen, and Emmanuelle Vaast. 2025. “Ethical Tensions in Human-AI Companionship: A Dialectical In- quiry into Replika.” SSRN preprint. https://doi.org/10.2139/ssrn.5285198 Cocking, Dean, and Jeanette Kennett. 2000. “Friendship and Moral Danger.” The Journal of Philosophy 97 (5): 278–296. https://doi.org/10.2307/2678396 Coeckelbergh, Mark. 2010. “Robot Rights? Towards a Social-Relational Jus- tification of Moral Consideration.” Ethics and Information Technology 12 (3): 209–221. https://doi.org/10.1007/s10676-010-9235-5 Coeckelbergh, Mark. 2012. Growing Moral Relations: Critique of Moral Sta- tus Ascription. Palgrave Macmillan. Coeckelbergh, Mark. 2014. “The Moral Standing of Machines: Towards a Re- lational and Non-Cartesian Moral Hermeneutics.” Philosophy & Technol- ogy 27 (1): 61–77. https://doi.org/10.1007/s13347-013-0133-8 36 Danaher, John. 2019. “The Philosophical Case for Robot Friendship.” Jour- nal of Posthuman Studies 3 (1): 5–24. https://doi.org/10.5325/jpost- stud.3.1.0005 Danaher, John. 2020. “Welcoming Robots into the Moral Circle: A Defence of Ethical Behaviourism.” Science and Engineering Ethics 26 (4): 2023– 2049. https://doi.org/10.1007/s11948-019-00119-x Darwall, Stephen. 2006. The Second-Person Standpoint: Morality, Respect, and Accountability. Harvard University Press. De Freitas, Julian, Noah Castelo, Ahmet Uguralp, and Zeliha O. Uguralp. 2024. “Lessons From an App Update at Replika AI: Identity Discontinuity in Human-AI Relationships.” arXiv preprint arXiv:2412.14190. De Freitas, Julian, Zeliha Oguz-Uguralp, and Ahmet Kaan Uguralp. 2025. “Emotional Manipulation by AI Companions.” SSRN preprint. https://doi.org/10.2139/ssrn.5390377 Earp, Brian D., Sebastian Porsdam Mann, Mateo Aboy, Edmond Awad, Monika Betzler, Marietjie Botes, Rachel Calcott, et al. 2025. “Relational Norms for Human-AI Cooperation.” arXiv preprint. https://arxiv.org/abs/2502.12102 Fang, C. M., A. R. Liu, V. Danry, E. Lee, S. W. Chan, P. Pataranutaporn, P. Maes, J. Phang, M. Lampe, L. Ahmad, and S. Agarwal. 2025. “How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study.” arXiv preprint arXiv:2503.17473. Fox-Decent, Evan. 2005. “The Fiduciary Nature of State Legal Authority.” Queen’s Law Journal 31 (1): 259–310. Frankel, Tamar. 1983. “Fiduciary Law.” California Law Review 71 (3): 795– 836. Frankfurt, Harry G. 2004. The Reasons of Love. Princeton University Press. 37 Frischmann, Brett, and Evan Selinger. 2018. Re-Engineering Humanity. Cambridge University Press. https://doi.org/10.1017/9781316544846 Gabriel, Iason, Arianna Manzini, Geoff Keeling, Laura Hendricks, Verena Rieser, Hasan Iqbal, Nenad Tomašev, et al. 2024. “The Ethics of Ad- vanced AI Assistants.” arXiv preprint. https://arxiv.org/abs/2404.16244 Gambino, Andrew, Jesse Fox, and Rabindra Ratan. 2020. “Building a Stronger CASA: Extending the Computers Are Social Actors Paradigm.” Human-Machine Communication 1: 71–86. https://doi.org/10.30658/hmc.1.5 Goodin, Robert E. 1985. Protecting the Vulnerable: A Reanalysis of Our So- cial Responsibilities. University of Chicago Press. Guingrich, Rose E., and Michael S. A. Graziano. 2025. “A Longitudinal Ran- domized Control Study of Companion Chatbot Use: Anthropomorphism and Its Mediating Role on Social Impacts.” Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1609/aies.v8i2.36618 Gunkel, David J. 2018. Robot Rights. MIT Press. https://doi.org/10.7551/mitpress/11444.001.0001 Hanson, Kenneth R., and Hannah Bolthouse. 2024. “Replika Removing Erotic Role-Play Is Like Grand Theft Auto Removing Guns or Cars: Red- dit Discourse on Artificial Intelligence Chatbots and Sexual Technolo- gies.” Socius: Sociological Research for a Dynamic World 10. https://doi.org/10.1177/23780231241259627 Harding, Matthew. 2012. “Trust and Fiduciary Law.” Oxford Journal of Legal Studies 33 (1): 81–102. https://doi.org/10.1093/ojls/gqs025 Helm, Bennett W. 2008. “Plural Agents.” Noûs 42 (1): 17–49. https://doi.org/10.1111/j.1468-0068.2007.00672.x 38 Helm, Bennett W. 2010. Love, Friendship, and the Self. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199567898.001.0001 Henriksen, Anine, Raha Asadi, Oksana Kulyk, Anne Gerdes, and Peter Mayer. 2025. “’I tell him everything that I do’: An Investigation of Privacy and Safety Implications of AI Companion Usage.” In 2025 European Symposium on Usable Security (EuroUSEC). https://doi.org/10.1109/eurousec69254.2025.00011 Hieronymi, Pamela. 2001. “Articulating an Uncompromising Forgiveness.” Philosophy and Phenomenological Research 62 (3): 529–555. Himmelreich, Johannes. 2019. “Responsibility for Killer Robots.” Ethical Theory and Moral Practice 22 (3): 731–747. Keller, Simon. 2013. Partiality. Princeton University Press. Kolodny, Niko. 2003. “Love as Valuing a Relationship.” The Philosophical Review 112 (2): 135–189. https://doi.org/10.1215/00318108-112-2-135 Lange, B., Keeling, G., Manzini, A. et al. “We need accountability in human– AI agent relationships”. npj Artif. Intell. 1, 38 (2025). https://doi.org/10.1038/s44387-025-00041-7 Lovett, Frank, and Philip Pettit. 2018. “Preserving Republican Freedom: A Reply to Simpson.” Philosophy & Public Affairs 46 (4): 363–383. https://doi.org/10.1111/papa.12126 Mackenzie, Catriona, Wendy Rogers, and Susan Dodds, eds. 2013. Vulnera- bility: New Essays in Ethics and Feminist Philosophy. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199316649.001.0001 Mackenzie, Catriona. 2020. “Vulnerability, Insecurity and the Pathologies of Trust and Distrust." International Journal of Philosophical Studies 28 (5): 624–643. https://doi.org/10.1080/09672559.2020.1846985 39 MacLachlan, Alice. 2016. “Fiduciary Duties and the Ethics of Public Apol- ogy.” Journal of Applied Philosophy 35 (2): 359–380. https://doi.org/10.1111/japp.12214 Manzini, Arianna, Geoff Keeling, Lize Alberts, Shannon Vallor, Meredith Ringel Morris, and Iason Gabriel. 2024. “The Code That Binds Us: Navi- gating the Appropriateness of Human-AI Assistant Relationships.” Pro- ceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7: 943–957. https://doi.org/10.1609/aies.v7i1.31694 Matthias, Andreas. 2004. “The Responsibility Gap: Ascribing Responsibility for the Actions of Learning Automata.” Ethics and Information Technol- ogy 6 (3): 175–183. https://doi.org/10.1007/s10676-004-3422-1 Mitchell, Jennifer J., and Myounghoon Jeon. 2025. “Exploring Emotional Connections: A Systematic Literature Review of Attachment in Human- Robot Interaction.” International Journal of Human-Computer Interac- tion 41 (18): 11753–11774. https://doi.org/10.1080/10447318.2024.2445100 Muldoon, James, and Paul Raekstad. 2022. “Algorithmic Domination in the Gig Economy.” European Journal of Political Theory 22 (4): 587–607. https://doi.org/10.1177/14748851221082078 Myers West, Sarah. 2019. “Data Capitalism: Redefining the Logics of Surveil- lance and Privacy.” Business and Society 58 (1): 20–41. https://doi.org/10.1177/0007650317718185 Nyholm, Sven. 2020. Humans and Robots. Rowman & Littlefield Publishers. https://doi.org/10.5771/9781786612281 Nyholm, Sven. 2026. The Ethics of Artificial Intelligence: A Philosophical Introduction. Hackett Publishing Company. Owens, David. 2012. Shaping the Normative Landscape. Oxford University Press. 40 Parke, Jul Jeonghyun. 2025. “Regulating Artificial Intelligence Intimacies: The Miseducation of South Korean AI Chatbot Iruda.” Canadian Journal of Communication 50 (1): 84–95. https://doi.org/10.3138/cjc-2024-0027 Pettit, Philip. 1997. Republicanism: A Theory of Freedom and Government. Oxford: Oxford University Press. Piispanen, Joni-Roy, Tinja Myllyviita, Ville Vakkuri, and Rebekah Rousi. 2024. “Smoke Screens and Scapegoats: The Reality of General Data Pro- tection Regulation Compliance—Privacy and Ethics in the Case of Replika AI.” arXiv preprint arXiv:2411.04490. Raz, Joseph. 1986. The Morality of Freedom. Oxford: Clarendon Press. Register, Christopher, Maryam Ali Khan, Alberto Giubilini, Brian D. Earp, and Julian Savulescu. 2025. “Privacy and Human-AI Relationships.” Phi- losophy & Technology 38. https://doi.org/10.1007/s13347-025-00978-2 Santoni de Sio, Filippo, and Giulio Mecacci. 2021. “Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them.” Philosophy & Technology 34 (4): 1057–1084. https://doi.org/10.1007/s13347-021-00450-x Savic, Milovan. 2024. “Artificial Companions, Real Connections?” M/C Jour- nal 27 (6). https://doi.org/10.5204/mcj.3111 Scanlon, T. M. 1998. What We Owe to Each Other. Harvard University Press. Schwitzgebel, Eric. 2023. “AI Systems Must Not Confuse Users About Their Sentience or Moral Status.” Patterns 4 (8): 100818. https://doi.org/10.1016/j.patter.2023.100818 Selinger, Evan, and Woodrow Hartzog. 2015. “Facebook’s Emotional Con- tagion Study and the Ethical Problem of Co-opted Identity in Mediated Environments Where Users Lack Control.” Research Ethics 12 (1): 35– 43. https://doi.org/10.1177/1747016115579531 41 Shiffrin, Seana Valentine. 2014. Speech Matters: On Lying, Morality, and the Law. Princeton University Press. Simmel, Georg. 1908/1950. The Sociology of Georg Simmel. Translated and edited by Kurt H. Wolff. Free Press. Smids, Jilles. 2020. “Danaher’s Ethical Behaviourism: An Adequate Guide to Assessing the Moral Status of a Robot?” Science and Engineering Ethics 26 (5): 2849–2866. https://doi.org/10.1007/s11948-020-00230-4 Sparrow, Robert. 2002. “The March of the Robot Dogs.” Ethics and Infor- mation Technology 4 (4): 305–318. Sparrow, Robert, and James Brown. 2026. “Against Imaginary Friends: Why Digital Companions Are No Solution to Social Isolation.” Communica- tions of the ACM 69 (2): 60–68. https://doi.org/10.1145/3750037 Srnicek, Nick. 2017. Platform Capitalism. Polity Press. Susser, Daniel, Beate Roessler, and Helen Nissenbaum. 2019. “Technology, Autonomy, and Manipulation.” Internet Policy Review 8 (2). https://doi.org/10.14763/2019.2.1410 Tigard, Daniel W. 2020. “There Is No Techno-Responsibility Gap.” Philos- ophy & Technology 34 (3): 589–607. https://doi.org/10.1007/s13347- 020-00414-7 Turkle, Sherry, Will Taggart, Cory D. Kidd, and Olivia Dasté. 2006. “Rela- tional Artifacts with Children and Elders: The Complexities of Cybercom- panionship.” Connection Science 18 (4): 347–361. https://doi.org/10.1080/09540090600868912 Vallor, Shannon. 2016. Technology and the Virtues: A Philosophical Guide to a Future Worth Wanting. New York: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780190498511.001.0001 Vallor, Shannon. 2024. The AI Mirror. Oxford University Press. 42 Verma, Pranshu. 2023. “They Fell in Love with AI Bots. A Software Update Broke Their Hearts.” The Washington Post, March 30. Walker, Margaret Urban. 2006. Moral Repair: Reconstructing Moral Relations after Wrongdoing. Cambridge University Press. https://doi.org/10.1017/cbo9780511618024 Weber-Guskar, Eva. 2022. “Reflecting (on) Replika: Can We Have a Good Affective Relationship With a Social Chatbot?” In Social Robotics and the Good Life: The Normative Side of Forming Emotional Bonds With Ro- bots, edited by Janina Loh and Wulf Loh, 103–126. Bielefeld: transcript Verlag. https://doi.org/10.1515/9783839462652-005 Wertheimer, Alan. 1996. Exploitation. Princeton University Press. Wygnańska, Joanna. 2023. “The Experience of Conversation and Relation With a Well-Being Chabot: Between Proximity and Remoteness”. Qualita- tive Sociology Review 19 (4): 92-120. https://doi.org/10.18778/1733- 8077.19.4.05. Xie, Tianling, and Iryna Pentina. 2022. “Attachment Theory as a Framework to Understand Relationships with Social Chatbots: A Case Study of Rep- lika.” In Proceedings of the Annual Hawaii International Conference on System Sciences. https://doi.org/10.24251/hicss.2022.258 Zhang, R., Li, H., Meng, H., Zhan, J., Gan, H. and Lee, Y.C., 2025, April. The dark side of ai companionship: A taxonomy of harmful algorithmic behav- iors in human-ai relationships. In Proceedings of the 2025 CHI conference on human factors in computing systems (p. 1-17). Zuboff, Shoshana. 2019. The Age of Surveillance Capitalism. PublicAffairs.