Paper deep dive

Toward Collective-Centric Evaluation of Preference Inference for Participatory Democracy

Pierre-Antoine Lequeu, Salim Hafid, Paul Lerner, Nazanin Shafiabadi, Laurène Cave, David Mas, Jean-Philippe Cointet, Benjamin Piwowarski, François Yvon

Year: 2026Venue: arXiv preprintArea: cs.SIType: PreprintEmbeddings: 54

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 9/4/2026, 2:56:34 AM

Summary

This paper introduces a collective-centric evaluation framework for Preference Inference (PI) models used in participatory democracy platforms like Polis and Remesh. It argues that standard accuracy metrics are insufficient as they may distort collective preference structures, such as consensus levels and group-specific approval patterns. The authors present a new multilingual dataset with over 90k participants and 1M votes, and propose metrics like Approval Rate Hit Ratio, Consensus Distortion, and Community Stability to assess whether inferred votes preserve the integrity of the democratic landscape.

Entities (8)

Pol.is · platform · 99%Remesh · platform · 99%Preference Inference · task · 98%Approval Rate Hit Ratio · metric · 95%Community Stability · metric · 95%Consensus Distortion · metric · 95%Pierre-Antoine Lequeu · person · 95%Democratic Commons · dataset-collection · 90%

Relation Signals (5)

Preference Inference → usedby → Pol.is

confidence 95% · Platforms such as Polis... rely on Preference Inference (PI) models to predict missing votes.

Preference Inference → usedby → Remesh

confidence 95% · Platforms such as Polis and Remesh... rely on Preference Inference (PI) models to predict missing votes.

Approval Rate Hit Ratio → evaluates → Preference Inference

confidence 90% · we use... Approval Rate Hit Ratio to assess whether the resulting patterns of support and opposition across propositions remain statistically credible.

Consensus Distortion → evaluates → Preference Inference

confidence 90% · we measure Consensus Distortion to evaluate whether inference alters the approval patterns of particular participant groups

Democratic Commons → contains → Preference Inference

confidence 85% · We further contribute the largest multilingual dataset of its kind... datasets available at https://huggingface.co/collections/democratic-commons/citizens-consultations

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:To scale up collective decision-making, participatory democracy platforms such as Polis and Remesh enable online deliberation among thousands of participants. However, at this scale, participants cannot review every opinion submitted by others, producing highly sparse voting data that misrepresent patterns of consensus, conflict, and minority support. Platforms therefore increasingly rely on Preference Inference (PI) models to predict missing votes. Yet this automation is not neutral: inferred preferences can artificially amplify, suppress, or reorder existing patterns of support, ultimately reshaping how the outcomes of a deliberation are interpreted. More generally, we lack a systematic understanding of how existing PI methods affect the collective preference landscape. To address this gap, we benchmark several existing PI approaches in this context. Moving beyond conventional user-centric evaluations centered on the accuracy of individual predictions, we introduce a collective-centric evaluation framework that measures whether inferred votes preserve salient properties of the broader preference landscape. We further contribute the largest multilingual dataset of its kind: four consultations spanning over 90k participants, 1M votes, and 22 languages. Our experiments show that models with comparable predictive accuracy can differ substantially in the degree to which they preserve the collective structure. These results demonstrate that accuracy alone is insufficient for evaluating \emph{PI} in democratic settings. By contributing a novel comprehensive and collective-centric evaluation benchmark for the task of PI, this work aims to support the development of AI systems that scale deliberation without compromising the integrity of its democratic outcomes.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

50,308 characters extracted from source content.

Expand or collapse full text

Toward Collective-Centric Evaluation of Preference Inference for Participatory Democracy Pierre-Antoine Lequeu Salim Hafid Paul Lerner Nazanin Shafiabadi Laurène Cave David Mas Jean-Philippe Cointet Benjamin Piwowarski François Yvon Abstract To scale up collective decision-making, participatory democracy platforms such as Polis and Remesh enable online deliberation among thousands of participants. However, at this scale, participants cannot review every opinion submitted by others, producing highly sparse voting data that misrepresent patterns of consensus, conflict, and minority support. Platforms therefore increasingly rely on Preference Inference (PI) models to predict missing votes. Yet this automation is not neutral: inferred preferences can artificially amplify, suppress, or reorder existing patterns of support, ultimately reshaping how the outcomes of a deliberation are interpreted. More generally, we lack a systematic understanding of how existing PI methods affect the collective preference landscape. To address this gap, we benchmark several existing PI approaches in this context. Moving beyond conventional user-centric evaluations centered on the accuracy of individual predictions, we introduce a collective-centric evaluation framework that measures whether inferred votes preserve salient properties of the broader preference landscape. We further contribute the largest multilingual dataset of its kind: four consultations spanning over 90k participants, 1M votes, and 22 languages. Our experiments show that models with comparable predictive accuracy can differ substantially in the degree to which they preserve the collective structure. These results demonstrate that accuracy alone is insufficient for evaluating PI in democratic settings. By contributing a novel comprehensive and collective-centric evaluation benchmark for the task of PI, this work aims to support the development of AI systems that scale deliberation without compromising the integrity of its democratic outcomes.11 1 datasets available at https://huggingface.co/collections/democratic-commons/citizens-consultations 1Sorbonne Université, CNRS, ISIR, Paris, France 2Sciences Po, Médialab, Paris, France 3Sorbonne Université, STIH/CERES, Paris, France 4Make.org lequeu@isir.upmc.fr, salim.hafid@sciencespo.fr Code — https://github.com/LequeuISIR/PrefInferenceEval 1 Introduction Figure 1: Accuracy fails to reveal distortions induced by Preference Inference. In this example, 75% of held-out test votes (unobserved in deployment) are correctly inferred. Despite this accuracy, inference drops a consensual proposition’s approval rate (AR) from 80% to 40% and distributes errors unevenly across socio-demographic groups. This motivates evaluating proposition-level support and participant-level outcomes alongside accuracy. Participatory democracy grounds the legitimacy of collective decisions in the capacity of citizens to participate in forming those decisions (Habermas 2015; Rawls 1997; Dryzek and List 2003). Participatory democracy platforms such as Polis and Remesh promise to extend such participation beyond small, resource-intensive forums to large-scale consultations used in public policy design, participatory budgeting, and peace-building (Konya et al. 2025; Fishkin et al. 2019; Landemore 2020; Small et al. 2021). In these consultations, participants submit propositions in free form text and express support for or opposition to propositions submitted by others, resulting in a collection of textual propositions and a participant–proposition voting matrix (see Figure 1). However, given the volume of inputs, each participant can only engage with a small fraction of all propositions, leading to a highly sparse voting matrix. This sparsity creates the need for Preference Inference (PI): the task of predicting whether a participant would support or oppose a proposition they have not reviewed (Konya et al. 2022). These predictions carry concrete implications as they reconstruct the landscape of preferences subsequently used to identify opinion groups, select representative propositions, and produce summaries for facilitators and decision-makers (Small et al. 2021; Revel et al. 2025; Fish et al. 2026). Prior work on deliberation summarization demonstrates that different models can produce substantially different levels of proportional representation and may erase minority voices (Zhu et al. 2025; Hafid et al. 2026). Distortions introduced during PI may thus propagate throughout the deliberative chain and alter the resulting collective representation. Since PI errors may be distributed unevenly (for instance, as a function of support size or internal group-level correlations), certain propositions risk being poorly modeled, generating spuriously consensual or non-consensual outcomes (Revel and Pénigaud 2026), or under-representing specific socio-demographic and interest groups. PI should therefore neither erase genuine political conflict to produce an appearance of agreement nor create an illusion of disagreement by fragmenting groups or distorting propositions around which participants genuinely agree. Deliberative theory offers a useful middle ground through the notion of meta-consensus: deliberation need not eliminate disagreement, but should accurately structure the values, judgments, and preferences over which disagreement persists (Dryzek 2012). A democratic PI model must therefore operate between two failures: artificial consensus and artificial conflict (Revel and Pénigaud 2026). We argue that it can achieve this by preserving the salient structure of the expressed landscape of preferences at two levels: how propositions are positioned through patterns of support and opposition, and how participants are organized into behavioral and socio-demographic groups. Meeting this dual objective creates a corresponding evaluation problem: how can we determine whether an inference model preserves, rather than distorts, the proposition- and participant-level structures of the preference landscape? A preference matrix is expected to exhibit a clustered structure because political preferences are often organized by relatively stable shared belief systems (Converse 2006). Yet, PI systems are commonly assessed using classification or recommendation metrics that aggregate errors across participant–proposition pairs (Konya et al. 2022), overlooking whether inference alters proposition-level support patterns, group-specific approval profiles, or vote-based community structure. More broadly, research on fairness in inference systems has emphasized the need to connect abstract metrics to the normative requirements of particular socio-technical contexts (Deldjoo et al. 2024). As illustrated in Figure 1, a model may achieve high accuracy while still distorting the preferences of several socio-demographic groups. We therefore argue for complementing standard user-centric metrics with collective-centric metrics designed to assess whether inference preserves the landscape of preferences.22 2 Algorithmic preference inference also introduces an accountability challenge when preferences do not stem directly from participation, though addressing this limitation lies beyond our scope. To address this gap, this work makes three contributions: (1) We construct a theoretical and empirical framework for evaluating how Preference Inference reshapes the landscape of preferences derived from a voting matrix. We formalize this landscape along two complementary dimensions grounded in social sciences: the positioning of propositions through patterns of support and opposition, and the grouping of participants through voting behavior and socio-demographic attributes. Building on this formalization, we develop an evaluation benchmark that complements standard predictive metrics by measuring whether inferred preferences preserve proposition-level approval rates (AR) and participant-level preference structures, including vote-based communities and socio-demographic group patterns. (2) We contribute a new multilingual dataset comprising four consultations obtained via a partner platform. To our knowledge, it is the largest corpus of its kind, containing a total of almost 90k participants, 1M votes, and 22 distinct languages. To support reproducibility and facilitate the use of our metrics, we make our code and dataset publicly available. (3) We conduct an extensive evaluation of a comprehensive set of Preference Inference models, showing that models with high predictive accuracy can still substantially distort patterns of consensus and preferences of particular socio-demographic groups. 2 Related Work Participatory Democracy Platforms Preference Inference operates within the broader pipeline through which participatory democracy platforms organize, represent, and summarize large-scale public input. These platforms typically represent participants’ expressed preferences as a participant-proposition voting matrix, and use different inference strategies to deal with sparsity. Polis imputes missing entries using proposition-level mean votes before applying dimensionality reduction and clustering (Small et al. 2021), whereas Remesh combines language model representations with latent factors learned from observed voting behavior (Konya et al. 2022; Konya et al. 2023). The resulting inferred matrices support downstream tasks such as opinion group identification or automated summarization. Such systems have seen various high-stakes deployments: Remesh by the UN for peace-building (Ovadya 2023), and Polis in Taiwan to shape Uber and taxi regulations (Konya et al. 2023). From a social-science perspective, Preference Inference goes beyond a mere technical operation: it attributes preferences that participants have not explicitly expressed, thereby shaping how citizens and groups become represented within the consultation and how collective preferences are made visible to decision-makers (Revel and Pénigaud 2025). PI should therefore be understood as both a predictive mechanism and a form of algorithmic representation. Inspired by Innerarity’s discussion of the normative implications of predictive analytics (Innerarity 2023), we focus on one aspect of algorithmic fairness: whether prediction accuracy differs systematically across socio-demographic groups and vote-based communities. Building on this perspective, we introduce novel metrics to assess whether inferred votes preserve proposition-level support patterns and whether preferences are inferred with comparable accuracy across groups. Moreover, because multilingual data remain underexplored in existing PI benchmarks, we evaluate our framework on a new multilingual dataset comprising four consultations, over 90K participants, 1M votes, and 22 languages. Preference Models for Collective Representation Models Although Preference Inference is a cornerstone task for participatory democracy platforms, it long predates these systems and has a substantial history across several research traditions. Early probabilistic models from psychometrics and conjoint analysis represented preferences through latent utilities, pairwise comparisons, and the attributes of alternatives (Thurstone 1927; Green and Srinivasan 1978). Later work introduced adaptive survey methods that collect informative responses without requiring every participant to evaluate every proposition (Salganik and Levy 2015). More contemporary approaches draw on recommender systems and related latent-variable methods for user-item data, including matrix-factorization and collaborative filtering (Koren et al. 2009), latent-factor models (Johnson and others 2014), and regularized matrix completion (Bilich et al. 2023). These methods estimate missing responses from shared low-dimensional structure across participants and items. Recent enhancements additionally incorporate the semantic content of propositions through language model representations (Konya et al. 2022), following recent advances in NLP (Sorensen et al. 2025) and stance detection (Allaway and McKeown 2020; Barriere and Balahur 2023). Blair et al. (2026) go further by training language models so that distances accurately represent participants’ preferences. Across methods, Preference Inference must typically contend with sparsity (Koren et al. 2009) and cold-start participants and propositions (Lee et al. 2019). In this work, we show that multilingual content poses an additional challenge. Evaluation In recommender systems, Preference Inference models are generally evaluated using classification or ranking metrics aggregated across user-item pairs, with recent work expanding to user- and item-side fairness (Deldjoo et al. 2024; Zhao et al. 2025; Ge et al. 2010). However, participatory democracy demands more than the equal predictive performance common in recommender systems, as group-level parity does not guarantee faithful collective representation. For example, suppose Group A supports proposition 1 and opposes proposition 2, while Group B holds the opposite preferences. A model that predicts 50% support for both propositions in both groups will show no disparity when evaluated using a conventional group-fairness metric, because both groups are inferred equally inaccurately. Yet the inference has erased the central structure of the consultation: the groups originally held opposite preferences, but now appear identical. To address this gap, we introduce metrics that evaluate whether inference preserves such group-specific approval profiles and vote-based community structure. Put simply, in addition to assessing whether groups are predicted equally well, our metrics determine whether the inferred voting matrix still represents what the groups believe and how they differ. 3 Theoretical Framework Notation Consider a democratic process D involving a set of N participants =uii=1NU=\u_i\_i=1^N and a set of M propositions =tjj=1MT=\t_j\_j=1^M. Participants’ opinions are represented by a voting matrix ∈−1,+1N×MV∈\-1,+1\^N× M, where vij=+1v_ij=+1 if participant uiu_i supports proposition tjt_j, and vij=−1v_ij=-1 if they oppose it. Because participants vote on only a subset of propositions, the democratic process provides a partially observed matrix obs∈−1,+1,⊥N×MV_obs∈\-1,+1, \^N× M, where vijobs=vijv^obs_ij=v_ij when the vote is observed and vobsij=⊥v^obs_ij= otherwise. Problem formalization: Preference Inference Given the partially observed voting matrix obsV_obs, a Preference Inference model ℳM predicts the missing entries and produces a completed voting matrix ^ℳ∈−1,+1N×M V^M∈\-1,+1\^N× M, defined by v^ijℳ=vijobs,if vobsij≠⊥,ℳ⁡(obs,ui,tj),if vobsij=⊥. v_ij^M= \ array[]lv^obs_ij,&if v^obs_ij≠ ,\\[4.0pt] M(V_obs,u_i,t_j),&if v^obs_ij= . array . (1) Preference inference therefore aims to estimate the unobserved votes while leaving the observed votes unchanged. In this work, we evaluate not only the accuracy of these predictions but also whether ^ℳ V^M preserves the collective properties of the underlying voting matrix V. Characterizing the Landscape of Preferences We define the landscape of preferences as the set of collective structures derived from the voting matrix. We characterize it through two complementary perspectives: how propositions are positioned relative to participants’ preferences, and how participants are organized through their voting behavior and social attributes. Propositions On the proposition side, the voting matrix induces aggregate patterns of support and opposition. We define the Approval Rate of a proposition tjt_j, denoted AR⁡(tj)∈[0,1]AR(t_j)∈[0,1], as the proportion of positive votes it receives. Approval rates position propositions along the landscape of preferences: some propositions are broadly supported, some are broadly rejected, and others are contested, with approval rates close to 0.50.5. These proposition-level patterns are central to how consultations are interpreted, since they indicate which ideas attract consensus, opposition, or disagreement. A Preference Inference model should therefore not artificially inflate, suppress, or reorder these patterns. Participants On the participant side, the voting matrix induces both behavioral and socio-demographic structures. First, participants may form vote-based communities33 3 We borrow the community terminology from graph theory, as further explained in the next section. (Mason 2022): groups of users with similar voting profiles, characterized by high within-group similarity and lower between-group similarity. These communities capture distinct perspectives within the consultation and reflect the plurality of preferences expressed by participants. Second, participants may belong to socio-demographic groups defined by attributes such as age, gender, ethnicity, or location. Because inference may affect these groups differently, Preference Inference should be evaluated with respect to whether it preserves their position and representation in the landscape of preferences, rather than amplifying, attenuating, or erasing group-specific patterns. Measuring the Distortion of Preferences We select metrics that evaluate whether preference inference preserves the landscape of preferences along its two main dimensions: propositions and participants. On the proposition side, we use Balanced Accuracy to measure whether individual missing votes are predicted correctly despite label imbalance, and Approval Rate Hit Ratio to assess whether the resulting patterns of support and opposition across propositions remain statistically credible. On the participant side, we measure Consensus Distortion to evaluate whether inference alters the approval patterns of particular participant groups, and Community Stability to assess whether groups of participants with similar voting behavior remain represented after matrix completion. We complement these measures with fairness metrics for socio-demographic groups: minimum group Balanced Accuracy to capture the performance experienced by the worst-served group, and Equalized Odds Ratio to measure disparities in error rates across groups. Together, these metrics provide a comprehensive assessment of whether inference preserves both how propositions are positioned and how participants (including behavioral communities and socio-demographic groups) are represented in the preference landscape. We define each metric below. Approval Rate Hit Ratio (ARHR) ARHR measures whether a PI model ℳM provides statistically credible approval rates at the proposition-level. For each proposition tjt_j, the observed approval rate AR⁡(tj|obs)AR(t_j|V_obs) is treated as an estimate of the approval rate in the underlying full matrix V. Considering votes on tjt_j as Bernoulli trials (where success is seen as supporting a proposition), we compute a 95%95\% Wilson confidence interval44 4 We use the Wilson score interval as it correctly handles cases with small sample size and extreme probability values. More details are given in the Supplementary Material. (Wilson 1927) from the observed votes in obsV_obs, denoted CI0.95(tj|obs)CI_0.95(t_j|V_obs). The Approval Rate Hit Ratio is then, for a given PI model ℳM: 1||∑tj∈[AR(tj∣^ℳ)∈CI0.95(tj∣obs)] 1|T| _t_j 1 [AR(t_j V^M) _0.95(t_j _obs) ] (2) A system with a low ARHR outputs statistically unlikely approval rates, meaning it is likely to distort the actual approval from the participant. Consensus Distortion (CD) We conceptualize consensus on a proposition as the distribution of approval across relevant participant groups, following work that evaluates common ground through cross-group agreement (Small et al. 2021; Konya et al. 2025; Hafid et al. 2026). Under this view, a proposition is seen as consensual only if its support is represented accurately across groups: substantially overestimating or underestimating the approval of any group may create a misleading picture of shared support or disagreement. CD therefore measures how much Preference Inference alters the group-level approval profile on which judgments of consensus are based. Let C denote a partition of U where each user uiu_i is assigned to a cluster ck∈c_k . C can be defined based on protected attributes, such as age or sex, or be inferred from the observed voting data. We denote AR⁡(ck,tj|)AR(c_k,t_j|V) as the approval rate of the user-group ckc_k on tjt_j. We define the group-level approval-rate distortion as: Δ⁡(ck,tj,ℳ)=|AR⁡(tj∣ck,^ℳ)−AR⁡(tj∣ck,)| (c_k,t_j,M)= |AR(t_j c_k, V^M)-AR(t_j c_k,V) | (3) Because a large distortion for even one group can produce a misleading impression of cross-group consensus, the proposition-level consensus distortion is defined as the maximum group-level distortion: CD(tj,ℳ)=maxc∈jΔ(c,tj,,ℳ)CD(t_j,M)= _c _j (c,t_j,,M) (4) Finally, the democratic-process-level consensus distortion CD⁡(,ℳ) CD(D,M) is the average of the proposition-level maxima over the eligible set of propositions ′T . Unlike Approval Rate Hit Ratio, which compares inferred approval rates to confidence intervals estimated from obsV_obs, Consensus Distortion is defined against the underlying matrix V. In empirical evaluations, V is approximated by held-out votes. The two metrics also differ in their level of analysis: ARHR evaluates whether proposition-level aggregate approval rates are preserved, whereas CD evaluates whether group-level approval rates are preserved within each proposition before aggregating across propositions. Platform Dataset Topic # Votes # Participants # Propositions # Attributes Lang Polis austria-climate Energy 133,048 1,756 1,039 – DE bowling-green BGKY Improvement 226,136 2,031 896 – EN 15-per-hour-seattle Seattle Minimum Wage Impact 2,995 339 54 – EN march-on Operation Marching Orders 536,923 6,289 2,160 – EN bank-farmland Land Use and Conservation 33,530 404 293 – EN vtaiwan.uberx Taiwan Uber Regulation 49,996 1,921 197 – EN bg2050 Bowling Green Future 14,341 126 371 – EN Remesh right-to-assemble Right to Assemble 7,547 307 1,296 4 EN foreign-intervention US Foreign Relations 4,520 308 767 – EN campus-protests University Campus Protests 6,066 309 1,020 7 EN Anonymized steuer-debate Taxation 16,204 1,050 25 4 DE gc-sante Healthcare 226,010 25,058 1,331 – FR ingerence Foreign Digital Interference 103,671 8,968 559 – FR eurhope Future of Europe 892,273 54,421 4,366 – 22 Table 1: Datasets statistics. The following datasets comprise protected socio-demographic attributes as metadata: right-to-assemble: age, ethnicity, gender, politics; campus-protests: age, education, gender, income, living-area, politics, religion; steuer-debate: age, gender, living-area, politics. Community Stability (CS) We previously defined communities as groups of participants who display similar voting . Hypothesizing that such dynamics can be found in obsV_obs, these communities should not disappear after inference. Voting matrices can be viewed as Signed Bipartite Graphs (SBG)—graphs with two distinct sets of nodes (participants and propositions) and edges existing only between the two sets (preferences). A Community in graph theory refers to a cluster of nodes strongly interconnected and weakly connected to others. In the PI setup, it corresponds to participants with similar voting behaviors. The Community Stability metric evaluates how these communities evolve after inferring the unobserved preferences of their population. More specifically, we only consider the subset of users who expressed at least Nmin=10N_ =10 opinions on propositions, and apply agglomerative clustering55 5 Agglomerative clustering is favored over other methods as it has no stochastic component. (Müllner 2011) on the signed bipartite spectral representation (Kunegis et al. 2010) of the sub-graph to find the optimal number of clusters k∗k^* maximizing the silhouette score (Rousseeuw 1987). We then apply the same method with k∗k^* clusters on the same user subset, but on the fully inferred matrix ^ℳ V^M. We compare the resulting clustering using the Adjusted Rand Index (Hubert and Arabie 1985). A higher score indicates community stability, while a low score hints at their disappearance. Other Metrics We compute additional standard metrics. First, we use Balanced Accuracy (B-Acc)—the average of the True Positive Rate (aka recall) and True Negative Rate—as it is well-suited for imbalanced datasets (Brodersen et al. 2010). Moreover, when protected attributes are available, we compute fairness-aware metrics: Min. B-Acc is simply the Minimum Balanced Accuracy across group subsets (Weerts et al. 2023), reflecting a Rawlsian principle of evaluating outcomes for the worst-served group (Rawls 1971). Equalized Odds Ratio (EOR; Hardt et al., 2016) is defined as the minimum of the True Positive Rate and False Positive Rate ratios across groups, where each ratio measures the disparity between the largest and smallest rates. min⁡(minc∈⁡TPR⁡(c)maxc∈⁡TPR⁡(c),minc∈⁡FPR⁡(c)maxc∈⁡FPR⁡(c)) ( _c TPR(c) _c TPR(c), _c FPR(c) _c FPR(c) ) (5) An EOR of 1 indicates that all groups achieve identical TP, TN, FP, and FN rates. 4 Experiments and Results Datasets We select datasets from two distinct online citizen consultation platforms—namely, Polis66 6 https://github.com/compdemocracy/openData and Remesh.77 7 https://github.com/akonya/polarized-issues-data To these, we add four newly compiled datasets sourced from a large-scale international civic technology platform (cf. Table 1 for details on all used datasets). We contribute this novel corpus alongside our work to address limitations in the existing datasets. First, it provides non-English evaluation data (French, German, and 22 languages in eurhope88 8 bg, cs, da, de, el, en, es, et, fi, fr, hr, hu, it, lt, lv, nl, pl, pt, ro, sk, sl, and sv (ISO 639-1)), countering the English-heavy bias of existing resources and enabling future multilingual analyses. Second, it captures national and transnational policy debates—such as taxation and foreign digital interference—at a generally larger scale than previous localized datasets. Finally, it offers diverse structural properties for evaluation, ranging from highly dense consultation matrices (e.g., over 50% interaction density in steuer-debate) to massive, sparse environments (eurhope alone comprises over 892k votes and 54k participants). To unify the datasets across the different platforms, we restrict our analysis to agree and disagree votes, discarding neutral votes (unavailable in Remesh) and pairwise preferences (exclusive to Remesh). For protected socio-demographic attributes, to ensure that the computed statistics are reliable, we only consider groups with |c|≥25|c|≥ 25. Statistics of the post-processed datasets can be found in the Supplementary Material. Models We evaluate a comprehensive set of Preference Inference models spanning the main methodological families identified in the related work. Collaborative Filtering (CF) CF predicts preferences using historical interaction patterns rather than explicit metadata. User-based models (CF-user) infer unseen votes by identifying individuals with highly correlated voting histories. Proposition-based models (CF-prop) calculate similarity between the propositions themselves based on overlapping ratings. The algorithm then aggregates these proximity metrics across the interaction matrix to infer the preferences. Nearest Neighbor (N) N uses the embeddings of propositions from pretrained encoders. It relies on the text embedding e of the propositions by assigning to v^ij v_ij the opinion vij∗v_ij* of the closest proposition in the embedding space for which the user already expressed a preference: j∗=argmaxj′≠jtj⋅tj′∀t∈uij*=argmax_j ≠ je_t_j·e_t_j \ \ \ \ ∀ t _u_i (6) where uiT_u_i is the set of propositions on which uiu_i voted. Figure 2: B-Acc vs. ARHR, CS, CD, EOR and Min. B-Acc for the politics protected attribute. CD, EOR and Min. B-Acc, are averaged over the three datasets with protected attributes and the other metrics are averaged over all datasets for each model. Regressions are computed over all (model, dataset) pairs, and not the displayed averages. Factorization Machines (FMs) FMs learn representations of users and propositions from existing preferences to infer new ones. Standard FMs initialize both user and proposition embeddings randomly in ℝPR^P and optimize them such that the utility for a pair (ui,tj)(u_i,t_j), mapped via the sigmoid function σ⁡(ui⋅tj)σ(u_i· t_j), approximates 1 if user uiu_i agrees with proposition tjt_j, and 0 otherwise. Additionally, we implement STUMP (Konya et al. 2022), an FM variant tailored for preference inference. STUMP initializes user representations ui∈ℝPu_i ^P randomly99 9 We experimented with initializing user representations from the embeddings of propositions they voted on, but this did not improve performance. and uses pretrained proposition embeddings tj∈ℝLe_t_j ^L as initialization. It then learns user embeddings alongside a linear projection matrix t∈ℝL×PW_t ^L× P for propositions, and computes the utility of (ui,tj)(u_i,t_j) as v^ij=σ⁡(ui⋅tTtj) v_ij=σ(u_i·W_t^Te_t_j), where σ denotes the sigmoid function.1010 10 STUMP also learns a context-specific projection for consultations. As these contexts are only available on the Remesh datasets, we discard them and remove the projection. SBG link-prediction As stated previously, consultations can be viewed as Signed Bipartite Graphs (SBG). Recent works have explored machine learning for link prediction in SBG. We implement SBGCL (Zhang et al. 2023), a Graph Neural Network which uses a two-level view of the graph (the explicit inter-sets and implicit intra-set relation of nodes) and applies augmentation techniques and contrastive learning to learn representations of the nodes. LLM-based methods Modern recommendation approaches leverage in-context learning and LLMs (Brown et al. 2020). In this framework, an LLM is prompted with a simple instruction (cf. Supplementary Material) containing the user’s observed positive and negative votes and asked to predict whether the user would approve of a new proposition (returning 1 for approval, 0 otherwise). The final prediction corresponds to the most likely token between the two1111 11 Because LLM recommendation is not scalable to large consultations (e.g. eurhope necessitate 250M inference), we only report its results on the three datasets with protected attributes.: v^ij=argmaxv∈0,1(P^(v|tj,ui)) v_ij=argmax_v∈\0,1\ ( P(v\ |\ t_j,T_u_i) ) (7) Experimental Protocol For each consultation, votes (obsV_obs) are randomly partitioned into a 70-15-15 train-validation-test split. For learned models, we conduct an extensive grid-search over learning rates and model-specific parameters (see Supplementary Material for the search space and optimal configurations). We select the model checkpoint that maximizes B-Acc on the validation set after a maximum of 400 training epochs, and evaluate its performance on the corresponding test set. For B-Acc, EOR, Min. B-Acc, and CD, we report results on the test set, a subset of obsV_obs. ARHR and CS are computed on the full ℳV^M. We compare two text-embedding models: (i) LaBSE (Feng et al. 2022), a 110M-parameter multilingual model; and (i) Pref (Blair et al. 2026), a 1.2B-parameter model based on Sentence-T5 (Ni et al. 2022) that is specifically trained for preference representation. For LLM recommendation, we use the open-weight post-trained model Qwen3-8B (Team 2025). Results We present per-model averages throughout this section1212 12 All results are scaled by ×102× 10^2. (detailed breakdowns are available in the Supplementary Material). Our evaluation addresses two central questions: (i) Does higher balanced accuracy necessarily imply a less distorted landscape of preferences? (i) Do existing PI methods preserve opinion structures adequately, or is further improvement necessary? Accuracy vs. Opinion Distortion Figure 2 presents B-Acc compared to distortion and fairness metrics per model. Although SBGCL achieves the highest B-Acc overall, it falls short on our proposed metrics, ranking 5th on ARHR and CS and 6th on CD. More generally, we find little to no correlation between B-Acc and these metrics (R2=0.05R^2=0.05 for ARHR, R2=0.14R^2=0.14 for CS, R2=0.13R^2=0.13 for CD on the political attribute)1313 13 We provide spearman correlation between all metrics used in this work in the supplementary materials., with the exception of Min. B-Acc, which is naturally strongly correlated. Figure 4 displays model performance on the attribute-based metrics (EOR, Min. B-Acc and CD) across the three common attributes (Gender, Age, Politics), further supporting the finding that SBGCL lags behind other models despite its high B-Acc. Are Existing PI methods Sufficient? No model noticeably outperforms the others across the board. N models achieve strong CD and ARHR results, but exhibit limited B-Acc and score the lowest on CS. Both collaborative filtering models show high B-Acc. scores—particularly impressive given that they are not learned—yet obtain the poorest results on almost all distortion and fairness metrics. Overall, STUMP-based models provide the best trade-off, showing accuracy results slightly below SBGCL, but at the same time distorting the preference landscape significantly less. In general, current results regarding preference landscape distortion fall short of acceptable standards for democratic processes. No model produces statistically plausible approval rates (at a 95% confidence interval); specifically, no model achieves an ARHR score above 0.95 on any dataset (cf. Supplementary Material). Similarly, the lowest average CD is 21.7 (obtained by FM) which is arguably too high for democratic processes. Finally, CS remains consistently low, with the best performing model (CF) achieving an average of only 22.9. Additionally, a per-language analysis on Eurhope (reported in the Supplementary Material) shows disparities in treatment. Ultimately, these findings reveal a critical limitation in current preference inference methods, underscoring the need for improved approaches that prioritize faithful representation and preservation of the underlying preference landscape alongside predictive accuracy. Figure 3: B-Acc. and ARHR averaged over three datasets across varying numbers of examples per user. Data Efficiency Following Konya et al. (2022), we evaluate the effect of preference sparsity by limiting the number of observed preferences per participant (1, 2, 4, 8, 16, or all available)1414 14 SBGCL requires at least two observed preferences per participant and is therefore not evaluated with one. on the three datasets with socio-demographic attributes. Figure 3 reports B-Acc and ARHR (results for the other metrics are provided in the Supplementary Material). B-Acc, Min. B-Acc, and CD consistently improve as more preferences become available, an important result not observed when evaluating using only accuracy (Konya et al. (2022)).1515 15 Konya et al. (2022) report no improvement in accuracy above five preferences per participant, but do not evaluate using collective-centric metrics. This further highlights the need for collective-centric metrics beyond accuracy. In contrast, ARHR, EOR, and CS remain largely unchanged, with the latter exhibiting noisy but uniformly low values (0–0.18). These findings suggest that several limitations of current preference inference models are structural rather than a consequence of sparsity. Figure 4: Equalized Odds Ratio (↑), Min. B-Acc (↑) and CD (↓) averaged over campus-protests, right-to-assemble, and steuer-debate. B-Acc is shown for reference. 5 Conclusion Participatory democracy platforms enable collective decision-making among thousands of participants, yet the resulting voting data are often highly sparse. Preference inference is therefore essential for reconstructing the preference landscape and supporting downstream decision-making. We introduce a collective-centric evaluation framework, grounded in deliberative democratic theory, to assess whether preference inference systems faithfully preserve the collective structure of expressed preferences rather than merely predicting individual votes accurately. Our findings demonstrate that preference inference is not a neutral computational step: models with high predictive accuracy can substantially distort patterns of consensus and the preferences of particular groups. Concerningly, across all evaluated consultations, no existing method faithfully preserves the underlying preference landscape. Alongside this evaluation framework, we release a new multilingual benchmark spanning four consultations, nearly 90,000 participants, one million votes, and 22 languages. In particular, the Eurhope consultation constitutes the largest publicly available consultation dataset to date, comprising eight times more participants and twice as many propositions as the largest previously available benchmark, while being the first multilingual. This work establishes collective-centric evaluation as a standard objective for future preference inference systems, encouraging models that optimize not only predictive accuracy but also faithful collective representation. Limitations and Ethical Considerations Although our framework provides principles for evaluating preference inference systems, it should not be interpreted as legitimizing preference inference as a substitute for genuine democratic participation. Preference inference can only complement citizen consultations whose design already ensures broad, representative, and meaningful participation. Moreover, our evaluation framework operationalizes a specific set of democratic principles grounded in deliberative theory, but these principles are not exhaustive. Alternative conceptions of deliberation and democratic representation may motivate additional evaluation criteria. Acknowledgments This research was funded by BPI-France under the project AI For Democracy - Democratic Commons, one of seven winners of BPI-France’s ’Digital Commons for Generative AI’ call for projects, conducted as part of the France 2030 investment plan. We thank the members of the Democratic Commons project, especially Manon Berriche, for their help and careful proof-reading. This work was performed using HPC resources from GENCI–IDRIS (Grant 2026-A0201017543). References Allaway and McKeown (2020) E. Allaway and K. McKeown Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), Online, p. 8913–8931. External Links: Document Cited by: §2. Barriere and Balahur (2023) V. Barriere and A. Balahur Multilingual multi-target stance recognition in online public consultations. Mathematics 11 (9), p. 2161. Cited by: §2. Bilich et al. (2023) J. Bilich, M. Varga, D. Masood, and A. Konya Faster peace via inclusivity: an efficient paradigm to understand populations in conflict zones. arXiv preprint arXiv:2311.00816. Cited by: §2. Blair et al. (2026) C. Blair, A. D. Procaccia, and M. Tambe Embeddings for Preferences, Not Semantics. arXiv. External Links: 2605.08360, Document Cited by: §2, §4. Brodersen et al. (2010) K. H. Brodersen, C. S. Ong, K. E. Stephan, and J. M. Buhmann The balanced accuracy and its posterior distribution. In 2010 20th International Conference on Pattern Recognition, Vol. , p. 3121–3124. External Links: Document Cited by: §3. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33, p. 1877–1901. Cited by: §4. Converse (2006) P. E. Converse The nature of belief systems in mass publics (1964). Critical review 18 (1-3), p. 1–74. Cited by: §1. Deldjoo et al. (2024) Y. Deldjoo, D. Jannach, A. Bellogin, A. Difonzo, and D. Zanzonelli Fairness in recommender systems: research landscape and future directions. User Modeling and User-Adapted Interaction 34 (1), p. 59–108. Cited by: §1, §2. Dryzek and List (2003) J. S. Dryzek and C. List Social choice theory and deliberative democracy: a reconciliation. British journal of political science 33 (1), p. 1–28. Cited by: §1. Dryzek (2012) J. S. Dryzek Foundations and frontiers of deliberative governance. Oxford University Press. Cited by: §1. Feng et al. (2022) F. Feng, Y. Yang, D. Cer, N. Arivazhagan, and W. Wang Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Dublin, Ireland, p. 878–891. External Links: Link, Document Cited by: §4. Fish et al. (2026) S. Fish, P. Gölz, D. Parkes, A. Procaccia, G. Rusak, I. Shapira, and M. Wuthrich Generative social choice. Journal of the ACM 73 (2), p. 1–52. Cited by: §1. Fishkin et al. (2019) J. Fishkin, N. Garg, L. Gelauff, A. Goel, K. Munagala, S. Sakshuwong, A. Siu, and S. Yandamuri Deliberative democracy with the online deliberation platform. In The 7th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2019), p. 1–2. Cited by: §1. Ge et al. (2010) M. Ge, C. Delgado-Battenfeld, and D. Jannach Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proceedings of the fourth ACM conference on Recommender systems, p. 257–260. Cited by: §2. Green and Srinivasan (1978) P. E. Green and V. Srinivasan Conjoint analysis in consumer research: issues and outlook. Journal of consumer research 5 (2), p. 103–123. Cited by: §2. Habermas (2015) J. Habermas Between facts and norms: contributions to a discourse theory of law and democracy. John Wiley & Sons. Cited by: §1. Hafid et al. (2026) S. Hafid, M. Berriche, and J. Cointet Algorithmic approaches to opinion selection for online deliberation: a comparative study. In Pluralistic Alignment Workshop at ICML 2026, Cited by: §1, §3. Hardt et al. (2016) M. Hardt, E. Price, and N. Srebro Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29, p. . External Links: Link Cited by: §3. Hubert and Arabie (1985) L. Hubert and P. Arabie Comparing partitions. Journal of classification 2 (1), p. 193–218. Cited by: §3. Innerarity (2023) D. Innerarity Predicting the past: a philosophical critique of predictive analytics. IDP: revista de Internet, derecho y política= revista d’Internet, dret i política (39), p. 2. Cited by: §2. Johnson et al. (2014) C. C. Johnson et al. Logistic matrix factorization for implicit feedback data. Advances in Neural Information Processing Systems 27 (78), p. 1–9. Cited by: §2. Konya et al. (2022) A. Konya, Y. L. Qiu, M. P. Varga, and A. Ovadya Elicitation Inference Optimization for Multi-Principal-Agent Alignment. In NeurIPS 2022 Foundation Models for Decision Making Workshop, Cited by: §1, §1, §2, §2, §4, §4, footnote 15. Konya et al. (2025) A. Konya, L. Thorburn, W. Almasri, O. A. Leshem, A. Procaccia, L. Schirch, and M. Bakker Using collective dialogues and ai to find common ground between israeli and palestinian peacebuilders. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, p. 312–333. Cited by: §1, §3. Konya et al. (2023) A. Konya, D. Turan, A. Ovadya, L. Qui, D. Masood, F. Devine, L. Schirch, I. Roberts, and D. A. Forum Deliberative technology for alignment. arXiv preprint arXiv:2312.03893. Cited by: §2. Koren et al. (2009) Y. Koren, R. Bell, and C. Volinsky Matrix factorization techniques for recommender systems. Computer 42 (8), p. 30–37. Cited by: §2. Kunegis et al. (2010) J. Kunegis, S. Schmidt, A. Lommatzsch, J. Lerner, E. W. De Luca, and S. Albayrak Spectral analysis of signed graphs for clustering, prediction and visualization. In Proceedings of the 2010 SIAM international conference on data mining, p. 559–570. Cited by: §3. Landemore (2020) H. Landemore Open democracy: reinventing popular rule for the twenty-first century. Cited by: §1. Lee et al. (2019) H. Lee, J. Im, S. Jang, H. Cho, and S. Chung Melu: meta-learned user preference estimator for cold-start recommendation. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, p. 1073–1082. Cited by: §2. Mason (2022) L. Mason Uncivil agreement: how politics became our identity. University of Chicago Press. Cited by: §3. Müllner (2011) D. Müllner Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378. Cited by: §3. Ni et al. (2022) J. Ni, G. Hernandez Abrego, N. Constant, J. Ma, K. Hall, D. Cer, and Y. Yang Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models. In Findings of the Association for Computational Linguistics: ACL 2022, S. Muresan, P. Nakov, and A. Villavicencio (Eds.), Dublin, Ireland, p. 1864–1874. External Links: Document Cited by: §4. Ovadya (2023) A. Ovadya ’Generative ci’through collective response systems. arXiv preprint arXiv:2302.00672. Cited by: §2. Rawls (1971) J. Rawls A theory of justice: original edition. Harvard University Press. Cited by: §3. Rawls (1997) J. Rawls The idea of public reason revisited. The university of Chicago law review 64 (3), p. 765–807. Cited by: §1. Revel et al. (2025) M. Revel, S. Milli, T. Lu, J. Watson-Daniels, and M. Nickel Representative ranking for deliberation in the public sphere. arXiv preprint arXiv:2503.18962. Cited by: §1. Revel and Pénigaud (2025) M. Revel and T. Pénigaud AI-Facilitated Collective Judgements. arXiv. External Links: 2503.05830, Document Cited by: §2. Revel and Pénigaud (2026) M. Revel and T. Pénigaud AI-enhanced deliberative democracy and the future of the collective will. Philosophy & Technology 39 (2), p. 81. Cited by: §1. Rousseeuw (1987) P. J. Rousseeuw Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics 20, p. 53–65. Cited by: §3. Salganik and Levy (2015) M. J. Salganik and K. E. Levy Wiki surveys: open and quantifiable social data collection. PloS one 10 (5), p. e0123483. Cited by: §2. Small et al. (2021) C. Small, M. Bjorkegren, T. Erkkilä, L. Shaw, and C. Megill Polis: scaling deliberation by mapping high dimensional opinion spaces. Recerca: revista de pensament i anàlisi 26 (2). Cited by: §1, §1, §2, §3. Sorensen et al. (2025) T. Sorensen, P. Mishra, R. Patel, M. H. Tessler, M. A. Bakker, G. Evans, I. Gabriel, N. Goodman, and V. Rieser Value profiles for encoding human variation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 2047–2095. Cited by: §2. Team (2025) Q. Team Qwen3 technical report. External Links: 2505.09388, Link Cited by: §4. Thurstone (1927) L. Thurstone A law of comparative judgment.. Psychological Review 34 (4). Cited by: §2. Weerts et al. (2023) H. Weerts, M. Dudík, R. Edgar, A. Jalali, R. Lutz, and M. Madaio Fairlearn: Assessing and Improving Fairness of AI Systems. Journal of Machine Learning Research 24. External Links: Link Cited by: §3. Wilson (1927) E. B. Wilson Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22 (158), p. 209–212. Cited by: §3. Zhang et al. (2023) Z. Zhang, J. Liu, K. Zhao, S. Yang, X. Zheng, and Y. Wang Contrastive Learning for Signed Bipartite Graphs. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, New York, NY, USA, p. 1629–1638. External Links: Document, ISBN 978-1-4503-9408-6 Cited by: §4. Zhao et al. (2025) Y. Zhao, Y. Wang, Y. Liu, X. Cheng, C. C. Aggarwal, and T. Derr Fairness and diversity in recommender systems: a survey. ACM Transactions on Intelligent Systems and Technology 16 (1), p. 1–28. Cited by: §2. Zhu et al. (2025) S. Zhu, S. Yang, M. A. Bakker, A. Pentland, and J. Pei Can ai truly represent your voice in deliberations? a comprehensive study of large-scale opinion aggregation with llms. arXiv preprint arXiv:2510.05154. Cited by: §1.

Instant research discovery