Paper deep dive
Can LLMs Rank? A Tale of Triads and Triage
Gaurab Pokharel, Shafkat Farabi, Patrick J. Fowler, Sanmay Das
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 7/5/2026, 3:57:11 AM
Summary
The paper investigates the reliability of Large Language Models (LLMs) when used as judges for ranking tasks in high-stakes scenarios like homelessness service allocation and emergency department triage. The authors propose a two-axis taxonomy for assessing ranking reliability: intra-run consistency (measured by the coefficient of consistency ζ, based on circular triads in tournament graphs) and inter-run variance (measured by Kendall's τ). The research demonstrates that these two metrics are independent and that LLMs exhibit distinct performance profiles across them. Specifically, the study compares LLaMA, Qwen, and DeepSeek, finding that while LLaMA excels in intra-run consistency, Qwen performs better in terms of inter-run stability. The paper provides a theoretical and empirical framework for practitioners to assess whether an LLM's ranking is sufficiently consistent to be trusted before deployment.
Entities (9)
Relation Signals (5)
LLM → evaluatedusing → Coefficient of Consistency ζ
confidence 100% · We discuss two different ways of identifying consistency. A classical diagnostic, the coefficient of consistency ζ...
LLM → evaluatedusing → Kendall's τ
confidence 100% · Various standard measures of distance between rankings, for example Kendall's τ, can measure inter-run variability.
Llama → isa → LLM
confidence 100% · differences in 3 major LLM models (LLaMA, Qwen, and DeepSeek)
Coefficient of Consistency ζ → measures → Intra-run consistency
confidence 100% · the coefficient of consistency ζ... provides a cheap, model-free measure of intra-run consistency.
Kendall's τ → measures → Inter-run variance
confidence 100% · Kendall's τ, can measure inter-run variability.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:From housing allocation for households experiencing homelessness to triage in emergency departments, LLMs are increasingly being considered as judges of consequential decisions that require ranking people for scarce resources. Ranking large groups simultaneously is cognitively demanding and error-prone. A natural solution, drawing on decades of social choice theory, elicits pairwise comparisons and aggregates them into a total order. However, a fundamental question remains when LLMs serve as the pairwise judge: how can a practitioner tell, before committing to a ranking, whether the LLM's judgments are sufficiently consistent to trust the result? We discuss two different ways of identifying consistency. A classical diagnostic, the coefficient of consistency $\zeta$, originally developed to measure judge reliability by counting circular triads in tournament graphs, provides a cheap, model-free measure of intra-run consistency. Various standard measures of distance between rankings, for example Kendall's $\tau$, can measure inter-run variability. We show, in both theory and practice, that these measures are independently valuable, and advocate for using both to assess reliability of rankings. We demonstrate the practical importance of our results across two high-stakes prioritization tasks: homelessness service allocation and emergency department triage. Three different leading LLMs have considerably different performance profiles across these two axes of consistency. We provide guidelines for how practitioners could think about measuring and assessing consistency before committing to a model for ranking or prioritization.
Tags
Links
- Source: https://arxiv.org/abs/2606.30412v1
- Canonical: https://arxiv.org/abs/2606.30412v1
Trouble viewing inline? Open PDF directly →
Full Text
94,051 characters extracted from source content.
Expand or collapse full text
Can LLMs Rank? A Tale of Triads and Triage Gaurab Pokharel Virginia Tech Alexandria, VA gaurab@vt.edu &Shafkat Farabi Virginia Tech Alexandria, VA mfarabi@vt.edu &Patrick J. Folwer Washington University in St. Louis St. Louis, MO pjfowler@wustl.edu &Sanmay Das Virginia Tech Alexandria, VA sanmay@vt.edu Abstract From housing allocation for households experiencing homelessness to triage in emergency departments, LLMs are increasingly being considered as judges of consequential decisions that require ranking people for scarce resources. Ranking large groups simultaneously is cognitively demanding and error-prone. A natural solution, drawing on decades of social choice theory, elicits pairwise comparisons and aggregates them into a total order. However, a fundamental question remains when LLMs serve as the pairwise judge: how can a practitioner tell, before committing to a ranking, whether the LLM’s judgments are sufficiently consistent to trust the result? We discuss two different ways of identifying consistency. A classical diagnostic, the coefficient of consistency ζ, originally developed to measure judge reliability by counting circular triads in tournament graphs, provides a cheap, model-free measure of intra-run consistency. Various standard measures of distance between rankings, for example Kendall’s τ, can measure inter-run variability. We show, in both theory and practice, that these measures are independently valuable, and advocate for using both to assess reliability of rankings. We demonstrate the practical importance of our results across two high-stakes prioritization tasks: homelessness service allocation and emergency department triage. Three different leading LLMs have considerably different performance profiles across these two axes of consistency. We provide guidelines for how practitioners could think about measuring and assessing consistency before committing to a model for ranking or prioritization. Keywords large language models ⋅· pairwise comparisons ⋅· rank aggregation ⋅· resource allocation ⋅· algorithmic decision-making ⋅· coefficient of consistency 1 Introduction Ranking represents a ubiquitous challenge in high-stakes societal resource allocation settings. Caseworkers must prioritize households for receiving emergency housing, and triage nurses must prioritize patients in the emergency room, as just two examples. These frontline workers – caseworkers, triage nurses, benefits administrators – typically apply professional judgment guided by standardized instruments. In homeless services, communities use different prioritization scores to rank households for housing interventions (for many years, they used the now-disfavored Vulnerability Index - Service Prioritization Decision Assistance Tool (VI-SPDAT), for example [31, 30, 32]). In emergency departments, the Emergency Severity Index (ESI) [52] structures nurse assessments into a five-level acuity rating that determines who is seen first. Interest in using large language models (LLMs) to assist with prioritization decisions continues to grow, as LLMs demonstrate increasingly sophisticated reasoning capabilities [46, 13, 51]. Interest aligns with the broader LLM-as-a-judge paradigm, gaining traction for tasks that once required trained human evaluators [9, 27]. Recent work has examined LLM judgments across domains, such as social-service interventions and medical acuity, finding that while models can engage with the task, their assessments are often inconsistent and unreliable [37, 51, 16, 25]. If LLMs are to inform prioritization in such situations, how can a practitioner assess whether the resulting rankings are sufficiently consistent to trust? To begin answering this crucial question, we first examine how LLMs are actually deployed in ranking tasks. It is well known that LLM outputs are sensitive to presentation order, list length, and prompt framing [47, 9, 40], making them fragile for ranking a large number of cases at once. An alternative, grounded in social-choice theory [10, 22], is to elicit pairwise comparisons. This involves asking a model to repeatedly prioritize one out of two cases based on some criteria (e.g., urgency) and then aggregate the responses into a total order. Pairwise elicitation simplifies each judgment, controls context length, and allows the application of principled aggregation methods such as Rank Centrality [29] or Borda Count [14]. Two broad approaches have been used in prior literature to examine the reliability of such rankings. The first reviews intra-run consistency in the rankings, which can be quantified by a measure of transitivity violations [22]. The alternative examines some measure of stability or variance of the ranking outcome, typically when the variance is over the subset of comparisons used in aggregation [2, 55]. However, to our knowledge, the empirical relationship between the two methods of measuring consistency has not yet been explored. Moreover, we are unaware of research that examines whether potential consistency or inconsistency in evaluation has implications for the reliability and trustworthiness of AI applications in high-stakes prioritization. These are the gaps that this paper explores. 1.1 Background and Motivation In principle, one can construct a ranking method that is always internally consistent (no transitivity violations), yet has high variance. Consider a method that randomly permutes the candidates and reports all pairwise comparisons consistent with the chosen permutation. It is also possible that a system with low intra-run consistency nevertheless produces low-variance outcomes. An example of this would be any method that always outputs a fixed ranking (say, lexicographic) on the whole set of choices but encodes an arbitrary number of triad transitivity violations into its pairwise reporting. This makes clear that, at least in theory, intra-run consistency (IC) and inter-run variance (IV) are independent axes, necessitating a 2×22× 2 taxonomy illustrated in Table 1. Inter-run variance (IV) Low High Intra-run consistency (IC) High Borda, consistent triads Random-permutation Low Fixed-cycle Noisy method, inconsistent triads Table 1: Ranking methods by intra-run consistency (IC) and inter-run variance (IV). The question we ask is where in this taxonomy a combination of an LLM-based (or other) approach and real-world data place us, and what factors drive intra-run consistency and inter-run variance. To measure intra-run consistency, we turn to the coefficient of consistency ζ. Introduced by Kendall and Smith [22], ζ aims to measure judge reliability by counting circular triads in tournament graphs. Circular triads, i.e., triples a,b,c\a,b,c\ for which a≻b≻c≻a b c a, are the minimal unit of intransitivity. These correspond to the curl component of the Hodge decomposition of Jiang et al. [19]; ζ is a scalar summary of its magnitude. We use ζ rather than the full decomposition because it distills inconsistency into a singular scalar that a practitioner can compute without specifying a generative model. We measure inter-run variance using (normalized) Kendall’s τ across runs [23].111Our results are robust to the choice of a specific measure of distance between rankings, as we show in the appendix. 1.2 Related Work Pairwise Comparison and Rank Aggregation The problem of aggregating pairwise comparisons into a global ranking has a long history in social choice and statistical decision theory. It dates back at least to Condorcet [10]’s study of majority preferences. Classical approaches include positional voting rules, such as Borda count [14] and probabilistic models for pairwise outcomes. In particular, Thurstone’s Case V model [44] and the Bradley-Terry-Luce model [4, 28] provide foundational frameworks in which each item is associated with a latent score and pairwise choices are generated based on score differences. More recent work has developed scalable ranking algorithms for large pairwise-comparison datasets, including spectral methods such as Rank Centrality [29]. Rank Centrality (RC) constructs a Markov chain from observed pairwise comparisons and estimates item rankings from its stationary distribution. This makes RC a good fit for our setting, where prioritization judgments are performed as pairwise comparisons. Consistency Diagnostics for Tournaments Kendall and Smith [22] introduced the circular triad count and the coefficient of consistency ζ as a test of judge reliability in paired-comparison experiments; subsequent work developed distributional theory and extensions [12, 1, 18]. These classical diagnostics operate on tournament graphs and require no generative model. A more recent line of work takes a different approach. Jiang et al. [19] decomposes the space of pairwise comparison data into three orthogonal components using the Hodge decomposition on graphs: a gradient component (consistent with a global ranking), a curl component (local cyclicity, the source of circular triads), and a harmonic component (global inconsistencies that cannot be resolved locally). The L2L_2 norm of the curl component quantifies the total cyclicity in the data, and in this sense ζ can be understood as a scalar summary of curl magnitude: a tournament with ζ near 1 has negligible curl, while one with ζ near 0 has maximum. Perotti [34] study HodgeRank under noise, examining how the decomposition behaves when pairwise data are corrupted. We use ζ because it can be easily computed from the tournament adjacency matrix in closed form without solving an optimization problem and can by used by any practitioner as a summary diagnostic for inconsistency without depending on the specifics of the generative model. LLM-as-a-Judge – Reliability and Intransitivity: As interest grows in using LLMs for pairwise prioritization tasks [38], a parallel literature on the LLM-as-a-judge paradigm [9] has identified systematic failure modes in LLM pairwise judgments, including sensitivity to presentation order [47, 9], verbosity bias [40], and self-preference [33]. Beyond these biases, recent work documents a subtler problem: LLM pairwise preferences are frequently intransitive. Xu et al. [53] find that non-transitivity is more prevalent when the items being compared are close in quality. Zeng et al. [55] count circular triads directly in passage-ranking tournaments (n=100n=100), Gupta and Kumar [15] and Qian et al. [39] report directed 3-cycle rates on text-evaluation tasks, and several other studies measure related constructs using k-subset violation ratios [49], strongly connected components [54], and intransitivity rates under various aggregation schemes [48]. The consistent finding is that intransitivity is common, varies between models and tasks, and worsens for weaker judges. Crucially, however, all report consistency metrics and ranking accuracy in separate tables. None frames the question in terms of the two independent reliability axes we identify in this paper: intra-run consistency and inter-run variance. AI in High-Stakes Resource Allocation: Many public service allocation problems reduce to ranking under scarcity: caseworkers must decide which households receive limited housing interventions, triage nurses determine which patients are seen first, and child welfare workers choose which families receive intensive services [8, 24, 35]. Standardized instruments, such as the VI-SPDAT [31, 30, 32] in homelessness services and the Emergency Severity Index [52] in emergency departments, exist precisely to impose consistency on these rankings. However, they have well-documented limitations, including racial bias [11, 50] and weak predictive validity [7]. Policies often mandate prioritizing the most vulnerable [45], while the outcomes on which workers are evaluated favor whoever would benefit most from the intervention [24, 35]. Certainly, frontline workers exercise professional discretion to navigate this tension [26, 36]. Algorithmic tools are increasingly deployed in these settings to assist or replace human judgment [3, 41, 43], raising questions about whether automated rankings can be trusted when the consequences are borne by vulnerable populations. Recently, Pokharel et al. [37] explicitly examined LLM judgments in social service triage. They found that while models can engage meaningfully with prioritization tasks, their pairwise assessments are inconsistent and unreliable. In this paper, rather than asking whether LLMs can replicate expert rankings, we ask a more fundamental question: are LLMs internally consistent (a basic requirement for trustworthiness) in their prioritization rankings? The diagnostic framework we develop is domain-general, but the motivation is grounded in these high-stakes settings where an unreliable ranking could have direct consequences for people’s lives. 1.3 Contributions We study the relationship between n, the number of candidates; ζ, the Coefficient of Consistency (measuring intra-run consistency); and τ, the rank distance (measuring inter-run variability) in both theory and practice. 1. We theoretically characterize the behavior of ζ as the number of candidates n increases. Of particular interest, we show that if the rate at which pairwise comparisons produce transitivity violations stays constant, ζ converges rapidly to a fixed floor and can be reliably estimated from small samples. 2. In a common stylized model (synthetic tournaments generated from Bradley-Terry-Luce (BTL) models [4, 28] with known ground-truth orderings), we empirically show that ζ increases linearly with n and tracks τ in tight lockstep. 3. In the same model, we demonstrate a divergence in the behaviors of τ and ζ when n is held constant, but the sample size of comparisons available to the rank aggregator changes. τ increases with this sample size, while ζ is, by definition, constant in expectation. 4. We show that empirical behavior on real data is markedly different as a function of n. Across a range of datasets that span three different populations experiencing homelessness, as well as patients in emergency rooms who need triage, we show that ζ stabilizes near its asymptotic floor as n increases, while τ continues to increase. 5. On these real-world datasets, we also show the differences between 3 major LLM models (LLaMA, Qwen, and DeepSeek) in terms of their performance on the ζ and τ dimensions. We show that the models differ considerably in terms of which is best on each of the two dimensions – Llama always “winning” on ζ while Qwen wins on τ, while DeepSeek often lands in the middle on both. 6. We discuss the implications of these results for real-world use of LLMs in prioritization and triage. Our results demonstrate clearly that our taxonomy has significant practical implications for measuring consistency and reliability of ranking. Our main recommendation is to analyze models carefully using both measures rather than relying exclusively on one of ζ or τ (or similar measures). 2 Preliminaries Having introduced intra-run consistency and inter-run variance as independent axes of reliability in Table 1, we now develop the formal machinery needed to measure each one. For intra-run consistency, we need a measure that is computable from tournament data alone and that does not require external validation. The circular triad count and its normalization into ζ provide exactly this. For inter-run variance, we use a standard measure of distance between rankings – Kendall’s τ [23]. 2.1 Tournaments and Circular Triads Given n items, a pairwise comparison procedure applied to every pair produces a tournament: a complete graph in which every pair of items is connected by exactly one directed edge indicating the preferred item. We represent a tournament by its adjacency matrix ∈0,1n×n A∈\0,1\^n× n, where Aij=1A_ij=1 if item i is preferred to item j and Aij=0A_ij=0 otherwise, with Aij+Aji=1A_ij+A_ji=1 for all i≠ji≠ j. The out-degree of item i is si=∑j≠iAijs_i= _j≠ iA_ij, the number of items it is preferred to. Definition 2.1 (Circular Triad). A circular triad in a tournament A is a triple i,j,k\i,j,k\ such that Aij=Ajk=Aki=1A_ij=A_jk=A_ki=1 or Aik=Akj=Aji=1A_ik=A_kj=A_ji=1. Circular triads are the minimal unit of intransitivity: three pairwise preferences that admit no coherent ordering of the triple. These arise naturally in voting data, in expert judgment, and in any setting where preferences are not perfectly transitive. Their presence indicates that the pairwise preference structure is not fully consistent with any single total order, and the more of them a tournament contains, the less cleanly that tournament can be summarized by a ranking. The total circular triad count in a tournament as defined above admits a closed-form expression [22, 1]: Tn=(n3)−∑i=1n(si2),T_n= n3- _i=1^n s_i2, (1) with maximum value Tmax=124⋅n(n2−1)if n is odd,n(n2−4)if n is even.T_ = 124· casesn(n^2-1)&if n is odd,\\[6.0pt] n(n^2-4)&if n is even. cases (2) Definition 2.2 (Coefficient of Consistency). The coefficient of consistency [22, 1] is ζ=1−TnTmax∈[0,1].ζ=1- T_nT_ ∈[0,1]. (3) A value of ζ=1ζ=1 indicates a perfectly transitive tournament (zero circular triads), while ζ=0ζ=0 indicates maximum cyclicity. The intuition for why ζ is a good measure of ranking consistency is straightforward: any aggregation method must resolve circular preferences, and the more such resolutions it must make, the more the output depends on the particular method of resolution rather than on a shared signal roughly consistent with a total order. A tournament with high ζ is one where pairwise preferences mostly agree on a single ordering; a tournament with low ζ is one where preferences point in conflicting directions, and no ranking can faithfully represent them all. Crucially, ζ can be computed from the tournament data A alone. This is precisely what a practitioner needs: after collecting pairwise comparisons and running an aggregator, they have a ranking but no way to assess its quality. Computing ζ from the same data that produced the ranking provides a diagnostic that requires no additional comparisons and no “ground truth” (in situations where that exists). Kendall’s τ as a Measure of Inter-Run Variance To measure the second axis of our taxonomy, inter-run variance, we need a way to quantify how much two rankings of the same n items disagree. We use Kendall’s τ distance [23], which counts the number of pairwise disagreements between two rankings. Given two rankings of the same n items, a pair (i,j)(i,j) is concordant if both rankings place i and j in the same relative order, and discordant otherwise. The normalized Kendall τ is τ=(concordant pairs)−(discordant pairs)(n2),τ= (concordant pairs)-(discordant pairs) n2, which takes on values in [−1,1][-1,1], with τ=1τ=1 indicating identical rankings, τ=−1τ=-1 indicating reversed rankings, and τ=0τ=0 indicating no more agreement than would be expected by chance. In the context of our work, given two independent runs of the same pairwise comparison and aggregation pipeline on the same set of items, τ measures inter-run agreement. A value near 1 indicates that the pipeline produces stable rankings regardless of which particular run is observed; a value near 0 indicates that the rankings are essentially unrelated across runs. 2.2 Size Dependence of ζ The definition of ζ in Equation (3) combines two quantities that behave differently as the number of items n changes: the circular triad count TnT_n, which depends on the comparison data, and the normalization constant TmaxT_ , which depends only on n. To understand how ζ evolves with n, it is useful to decompose it into a data-dependent term and a purely combinatorial term. This decomposition also has a practical benefit: it yields a natural estimator of ζ from incomplete tournament data, which we will need for our experiments. Definition 2.3 (Inconsistency Rate). The inconsistency rate of a tournament A on n items is r=Tn(n3),r= T_n n3, (4) the fraction of all triplets that form circular triads. Rewriting Equation (3) in terms of r gives ζ=1−r⋅(n3)Tmax.ζ=1-r· n3T_ . (5) This separates ζ into two factors: the inconsistency rate r, which captures how intransitive the pairwise comparisons are, and the scaling factor (n3)/Tmax n3/T_ , which is a known constant depending only on n. The inconsistency rate has a clean interpretation: it is the probability that a uniformly random triplet drawn from the tournament contains a circular triad. Substituting in Equation (2) and simplifying yields the scaling factor in closed form: (n3)Tmax=4⋅(n−2)n+1if n is odd,(n−1)n+2if n is even. n3T_ =4· cases (n-2)n+1&if n is odd,\\[6.0pt] (n-1)n+2&if n is even. cases (6) Estimation of ζ in sparse tournaments: The decomposition in Equation (5) has an immediate practical benefit. In many settings, not every pair of items can be compared: either the number of pairs is too large, or comparisons are expensive. When each pair i,j\i,j\ is observed independently with probability p, a triplet i,j,k\i,j,k\ is fully observed only if all three of its constituent pairs happen to be selected, which occurs with probability p3p^3. Because the sampling mechanism is independent of the comparison outcomes, the circular triad rate among fully observed triplets is an unbiased estimate of r (and consequently in zetazeta). Let =i,j,k:all three pairs observedO= \\i,j,k\:all three pairs observed \ denote the set of fully observed triplets, and let TobsT_obs be the number of circular triads among them. Then ζ^=1−Tobs||⋅(n3)Tmax ζ=1- T_obs|O|· n3T_ (7) is an unbiased estimator of ζ. The ratio Tobs/||T_obs/|O| estimates the inconsistency rate r, and the scaling factor (n3)/Tmax n3/T_ is the same constant as before. The precision of the estimator depends on |||O|, which grows as (n3)⋅p3 n3· p^3; even at moderate sparsity the number of fully observed triplets is typically large enough for a reliable estimate. The decomposition in Equation (5) also makes it clear how ζ depends on the number of items n. The scaling factor (n3)/Tmax n3/T_ is an increasing function of n that converges to a finite limit, so the behavior of ζ is governed by how the inconsistency rate r changes with n. Proposition 1 (Asymptotic behavior of the scaling factor). Both cases of Equation (6) converge monotonically to 44 from below as n→∞n→∞. Proof. The odd case can be written as 4⋅(1−2n)/(1+1n)4·(1- 2n)/(1+ 1n); the n terms vanish as n→∞n→∞ and the ratio approaches 44. It is strictly less than 44 for all finite n because n−2<n+1n-2<n+1. The even case is analogous: 4⋅(1−1n)/(1+2n)→44·(1- 1n)/(1+ 2n)→ 4. ∎ Since the scaling factor increases toward 44, Equation (5) shows that the trajectory of ζ as n grows is entirely determined by the trajectory of r. Three qualitatively different regimes are possible. Constant r: If the inconsistency rate does not depend on n, then ζ(n)=1−r⋅f(n)ζ(n)=1-r· f(n) where f(n)=(n3)/Tmaxf(n)= n3/T_ is increasing in n. In this case ζ decreases monotonically toward the asymptote: limn→∞ζ(n,r)=1−4r. _n→∞ζ(n,r)=1-4r. (8) Note that the factor of 44 does not allow ζ to become negative. Since Tn≤TmaxT_n≤ T_ by definition, the inconsistency rate is bounded above by r≤Tmax/(n3)=1/f(n)r≤ T_ / n3=1/f(n), and this bound converges to 1/41/4 as n→∞n→∞. The product r⋅f(n)≤1r· f(n)≤ 1 for all n, so ζ∈[0,1]ζ∈[0,1] always. At the asymptote, the maximum possible inconsistency rate is r=14r= 14, which corresponds to ζ=0ζ=0 (maximum cyclicity). Increasing r: If the inconsistency rate grows with n, then both r and the scaling factor push ζ downward, and ζ decreases faster than in the constant-r case. This regime would arise if adding more items to the comparison set introduced systematically harder distinctions, increasing the fraction of triplets that contain circular triads. Decreasing r: If the inconsistency rate decreases with n, then r and the scaling factor act in opposite directions: the scaling factor pushes ζ down while the falling r pushes it up. The net effect depends on which force dominates. If r decreases slowly relative to the growth of f(n)f(n), ζ still decreases, just more slowly than in the constant case. If r decreases fast enough to exactly offset the growth of the scaling factor, ζ can remain approximately constant in n. And if r decreases faster still, ζ can increase with n. This three-way characterization frames a central empirical question of the paper. The behavior of ζ as a function of n is not determined by the combinatorics of tournaments alone; it depends on the inconsistency rate, which is a property of whatever process generates the pairwise comparisons. Different comparison processes (different LLMs, different noise models, different domains) may place the system in different regimes. In Section 3, we instantiate this framework with the Bradley-Terry-Luce model, where r can be derived analytically as a function of the preference strength parameter β. In Section 4, we measure r empirically across several LLMs and datasets and identify which regime each combination occupies. 3 A Synthetic Model Any algorithm that produces a ranking from pairwise comparison data, whether Rank Centrality (RC) [29], Bradley-Terry Maximum Likelihood Estimation (BT-MLE) [17], or simple win-rate aggregation, faces a common problem at inference time: the practitioner has a ranking in hand but no way to assess how reliable it is. Theoretical guarantees, where they exist, are stated in terms of population-level quantities such as the spectral gap of a transition matrix [29, 42], which are not available from observed data. A practitioner can control the number of comparisons elicited, but cannot determine, after the fact, whether the resulting ranking should be trusted. The goal of this section is to establish, under controlled conditions, the relationship between intra-run consistency (ζ) and inter-run variance (τ) in a stylized setting. We use a standard synthetic generative model, well-established in the literature, with a known ground-truth ranking so that we can vary the noise level systematically and observe how both axes of our taxonomy respond. 3.1 Generative Model We require a generative model for pairwise comparisons with three properties: (1) a known true ranking against which the aggregator’s output can be measured, (2) a single tunable parameter that controls comparison noise, and (3) rank-gap-dependent difficulty, so that close-in-rank items are harder to distinguish than distant ones. The Bradley-Terry-Luce (BTL) model [4, 28] satisfies the first and third requirements by construction: each item i∈[n]i∈[n] has a latent score wi>0w_i>0, and the probability that item i defeats item j is P(i≻j)=wi/(wi+wj)=σ(θi−θj)P(i j)=w_i/(w_i+w_j)=σ( _i- _j), where θi=logwi _i= w_i and σ is the logistic function. To satisfy the second requirement, we impose a structure on the score vector that collapses the n−1n-1 free parameters into a single dimension. Definition 3.1 (Equally-Spaced BTL Model). Fix n items indexed 1,…,n1,…,n in true ranking order. Assign log-scores θi=β(n−i) _i=β(n-i) for i=1,…,ni=1,…,n, where β≥0β≥ 0 is the per-position discriminability. The pairwise comparison probability is P(i≻j)=σ(β(j−i))=11+exp(−β(j−i)).P(i j)=σ (β(j-i) )= 11+ (-β(j-i) ). (9) The single parameter β governs the entire noise structure. At β=0β=0, all comparisons are fair coin flips; as β→∞β→∞, all comparisons become deterministic. The probability that two adjacent items are ranked differently than in the true ranking is ε(β)=1/(1+eβ) (β)=1/(1+e^β), which decreases monotonically in β and represents the hardest-case pair-level error rate. This model has direct precedent in the original RC paper. In Negahban et al. [29], the authors generate synthetic BTL data with geometric scores wi=b(2i−1−n)/2nw_i=b^(2i-1-n)/2n, yielding a constant log-score gap of (logb)/n( b)/n between consecutive items. This is the same one-parameter family as Definition 3.1 under the identification β=(logb)/nβ=( b)/n. We prefer the β parameterization because it is directly interpretable: two experiments with the same β but different n share the same adjacent error rate ε(β) (β) and the same local comparison difficulty. 3.2 Experimental Design The mathematical analysis of Section 2.2 showed that the behavior of ζ with n is governed by the inconsistency rate r, which under the model defined in Definition 3.1 is a function of both β and n. To disentangle the roles of these two parameters, we organize the synthetic experiments along two axes. 3.2.1 Results: Complete Tournaments In the first study, we fix a grid of target ζ values and, for each pair (ζ∗,n)(ζ^*,n) with n∈20,50,100n∈\20,50,100\, solve for the value of β that produces expected consistency ζ∗ζ^* at tournament size n. This inverts the natural direction of the model: rather than fixing the noise level and observing how consistent the resulting tournament is, we fix the consistency level and ask what noise level is required to achieve it. For each (ζ∗,n)(ζ^*,n) pair, we generate 100 independent complete tournaments from the BTL model at the corresponding β, aggregate each via Rank Centrality (we show in Figure 6 in Appendix A.3 that these experiments are robust to the specific choice of rank aggregator used), and measure three quantities: the observed ζ of each tournament, the accuracy of the aggregated ranking (Kendall τ against the known ground-truth ordering), and the inter-run agreement (Kendall τ between rankings produced by independent runs of the same generative process, which we denote τcons _cons). (a) Accuracy (b) Inter-run agreement Figure 1: Intra-run consistency (ζ) vs. the two facets of ranking quality under the synthetic BTL model, for n∈20,50,100n∈\20,50,100\. (a) ζ vs. ranking accuracy (Kendall τ against the known ground-truth ordering). The relationship is monotonically increasing and largely converges across tournament sizes at high ζ, with greater separation at low consistency where smaller tournaments provide less information per item. (b) ζ vs. inter-run agreement (τcons _cons, Kendall τ between two independent RC rankings from the same generative model). The curves separate by n: at a given ζ, larger tournaments yield higher inter-run agreement. This separation shows that intra-run consistency alone does not determine inter-run variance; the size of the comparison matrix also matters. Error bars represent ±1± 1 standard deviation across 100 trials. Figure 1(a) (left panel) shows that ζ is a strong predictor of ranking accuracy. The relationship is monotonically increasing across all three tournament sizes: higher intra-run consistency corresponds to higher agreement with the ground-truth ordering. With increasing n, at the same level of consistency ζ, we observe a higher value of τacc _acc. At high ζ (>0.8>0.8), the three curves converge, indicating that when comparisons are sufficiently clean, even a modest number of items yields accurate rankings. Figure 1(b) (right panel) shows the relationship between ζ and inter-run agreement τcons _cons. Here the curves separate similarly by n: at any given level of intra-run consistency, larger tournaments produce substantially more stable rankings across independent runs. At ζ=0.5ζ=0.5, for instance, τcons _cons ranges from roughly 0.550.55 at n=20n=20 to roughly 0.790.79 at n=100n=100. The explanation is that a complete tournament on n items contains (n2) n2 comparisons, so each item participates in n−1n-1 comparisons. Increasing n increases the amount of data per item available to the aggregator, reducing estimation variance in the ranking even when the level of intransitivity (as captured by ζ) is unchanged. These results show that, in this synthetic setting, intra-run consistency and inter-run variance are correlated, but they are not redundant: two tournaments with identical ζ can have very different levels of ranking stability depending on the size of the comparison matrix. 3.2.2 Results: Sparse Observation In the second study, we fix n=30n=30 and β=0.3β=0.3 (corresponding to ζ≈0.859ζ≈ 0.859 in the full tournament) and vary the fraction of pairwise comparisons observed. For each observation probability p∈0.2,0.3,…,1.0p∈\0.2,0.3,…,1.0\, we subsample edges uniformly at random, aggregate only the observed comparisons via RC, and measure both ζ ζ (using the sparse estimator from Equation (7)) and τcons _cons (between rankings from independent subsampled runs). This isolates the effect of comparison budget on inter-run variance while holding the underlying noise process constant. (a) The sparse estimator ζ ζ is unbiased across the full range of p, with the dashed line marking the complete-tournament value. Variance increases at low p but the mean remains stable. (b) Inter-run agreement (τcons _cons) increases monotonically with the observation fraction. At p=0.2p=0.2, independent runs agree at roughly τcons≈0.55 _cons≈ 0.55; at p=1.0p=1.0, agreement is perfect. Figure 2: Sparse observation on synthetic model (n=30n=30, β=0.3β=0.3, full-tournament ζ≈0.859ζ≈ 0.859). Each point averages over 100 independent subsampled tournaments at the given observation probability p; error bars represent ±1± 1 standard deviation. The contrast between the two panels illustrates that ζ measures the consistency of the comparison signal while τcons _cons reflects both the consistency and the quantity of comparisons collected. Figure 2(a) confirms that the sparse estimator ζ ζ is unbiased across the full range of observation probabilities. At every value of p from 0.20.2 to 1.01.0, the mean of ζ ζ falls on or near the dashed line marking the full-tournament value ζ=0.859ζ=0.859. The variance of the estimator increases at low p, as expected given that fewer fully observed triplets are available, but there is no systematic bias. Figure 2(b), on the other hand, tells a different story for inter-run agreement. τcons _cons increases steadily from roughly 0.550.55 at p=0.2p=0.2 to 1.01.0 at p=1.0p=1.0. When only 20%20\% of pairs are observed, two independent subsampled runs see largely different comparisons, and their aggregated rankings diverge substantially. As the observation fraction increases, the runs share more data and the rankings converge. Taken together, Figure 2 provides a clean illustration of the distinction between our two axes. The underlying comparison process is identical across all values of p: the same β, the same items, the same pairwise noise. The sparse estimator correctly reports this, returning the same ζ ζ regardless of how many comparisons are observed. But inter-run variance depends not only on the consistency of the comparisons but on the quantity. A practitioner who observes low τcons _cons but high ζ ζ can diagnose the problem as insufficient comparison budget rather than unreliable judgments, and the remedy is to elicit more comparisons rather than to distrust the comparison process itself. These synthetic results establish a baseline and a diagnostic framework. Under the BTL model, ζ reliably predicts ranking accuracy, but does not fully determine inter-run variance because the latter also depends on how many comparisons are available per item. The two axes of our taxonomy are correlated but not redundant, even under controlled conditions. The question is whether real-world pairwise comparisons generated by LLMs respect this relationship, or depart from it in informative ways. We turn to this question in the next section. 4 Evaluating LLMs The synthetic calibration of Section 3 demonstrates that ζ is useful in capturing comparison consistency independently of how many comparisons are collected, and this yields a useful measure of intra-run consistency. Inter-run variation, on the other hand, could be affected by the number of comparisons available, the aggregation method, etc. In this section, we turn to real-world pairwise comparisons generated by LLMs. Figure 3: Expected inconsistency rate E[r]E[r] as a function of n for several values of β in the equally-spaced BTL model. For every fixed β>0β>0, the inconsistency rate decreases with n, confirming Proposition 2. Proposition 2. Fix β>0β>0. In the equally-spaced BTL model as defined in Definition 3.1, the expected number of circular triads in a complete tournament satisfies E[Tn]=Θ(n)E[T_n]= (n), and Tmax=Θ(n3)T_ = (n^3). Hence: E[1−ζ]=E[Tn]Tmax=Θ(1n2).E[1-ζ]= E[T_n]T_ = ( 1n^2 ). Proof sketch. For a triple i<j<ki<j<k with rank gaps a=j−ia=j-i and b=k−jb=k-j, the upset probability at gap d is qd=σ(−βd)q_d=σ(-β d), which decays exponentially in d. A circular triad requires either one long-edge upset or two short-edge upsets, so the cycle probability for a given triple is dominated by e−β(a+b)e^-β(a+b). Summing over all triples, the contribution from close neighbors (small rank gaps) grow linearly in n, while contributions from distant items are exponentially suppressed. This gives E[Tn]=Θ(n)E[T_n]= (n). Since Tmax=Θ(n3)T_ = (n^3), the ratio vanishes as Θ(1/n2) (1/n^2). The full proof is in Appendix A.2. ∎ VI-SPDAT VIF-SPDAT TAY-VISPDAT DeepSeek LLaMA Qwen Figure 4: Each panel shows the empirical distribution of τcons _cons from 100 bootstrap subsamples of a single 30×3030× 30 complete subtournament (at p=0.4p=0.4), with the full-pipeline τcons _cons overlaid as a vertical red line, and τcons _cons using all the edges from the 30x30 tournament as a vertical green line. Rows correspond to LLM judges (DeepSeek, LLaMA, Qwen); columns correspond to homelessness datasets (VI-SPDAT, VIF-SPDAT, TAY-VISPDAT). In every case, the full-pipeline τcons _cons falls well to the right of the bootstrap distribution, illustrating the distinction between our two reliability axes: the subtournament correctly estimates comparison quality (ζ, see Figure 5), but inter-run agreement (τcons _cons, this Figure) depends on both the quality and the quantity of comparisons collected. Figure 3 confirms Proposition 2 visually. For every fixed β, the inconsistency rate E[r]E[r] decreases with n and approaches zero. The intuition is that in the equally-spaced BTL model, the vast majority of triples involve items that are far apart in rank, and those comparisons are nearly deterministic: the upset probability qd=σ(−βd)q_d=σ(-β d) decays exponentially with rank gap d. Circular triads then arise almost exclusively from close neighbors, and the number of such close triples grows only linearly in n, while the total number of triples grows cubically. So r shrinks as Θ(1/n2) (1/n^2), driving ζ toward 11. In the language of Section 2.2, this places the BTL model firmly in the decreasing-r regime, and the decrease is fast enough that ζ not only remains stable but actually increases. However, we note that the scaling of ζ with n in practice may differ substantially from the stylized model. If LLM pairwise comparisons behaved like this BTL process with some fixed preference strength, we would expect ζ to be very high in the tournaments we study (n in the hundreds). As we will soon see, this is not what we observe. The ζ values in our real-world experiments remain roughly constant across different values of n, suggesting that the inconsistency rate r is approximately constant rather than decreasing. Further, in real prioritization tasks, asking an expensive method like an LLM to perform the full (n2) n2 comparisons may be infeasible. We now evaluate the relationship between ζ and the inter-run variability τcons _cons. Datasets We evaluate the diagnostic framework on real-world prioritization tasks spanning two domains: homelessness housing allocation and emergency department triage. Homelessness housing allocation. We use household-level vulnerability assessments from St. Louis coordinated entry records (2021-2024) [37]. Three population-specific instruments are treated as separate datasets: VI-SPDAT for single adults (n=325n=325), VIF-SPDAT for families (n=698n=698), and TAY-VISPDAT for transition-age youth (n=561n=561). Each household has a total acuity score used in practice for prioritization and is assigned to a priority band (low, medium, high) based on established scoring rubrics. For the Stage 1 subtournament, we select m=30m=30 households per instrument with proportional representation across priority bands. Emergency department triage. We use ED visits from MIMIC-IV [21] (N=180,057N=180,057 total encounters), restricted to encounters with complete triage features and valid acuity scores. Each visit is characterized by vital signs, chief complaint, medication history, and demographics recorded at arrival. The ground-truth label is the Emergency Severity Index (ESI), a five-level acuity score assigned by the triage nurse [52]. For Stage 1, we select m=30m=30 patients with proportional representation across ESI levels, matching the distribution in the full dataset. For Stage 2, we sample n=500n=500 again matching proportional representation across ESI levels from the eligible pool. Full inclusion criteria and feature definitions are in Appendix B. LLMs, Prompting and Implementation Details We use three LLMs for our experiments, namely: Meta-Llama-3-8B-Instruct, DeepSeek-R1-Distill-Llama-8B, and Qwen2.5-7B-Instruct. Due to the sensitive nature of the data, we deploy these open source models locally with weights downloaded from Hugging-face, and prompt them locally. The prompts were iteratively designed to try and achieve high consistency, and are provided in Appendix C. We ran these models on one NVIDIA A100 GPU with 80 GB of memory and sampled outputs with 0.1 temperature. (a) VI-SPDAT (b) VI-FSPDAT (c) TAY-VI-SPDAT (d) MIMIC-IV Figure 5: Cycle ratio, ζ, and Rank Centrality (RC) Kendall τ as functions of the number of nodes sampled in the subgraph. Each model uses a consistent color across runs, while separate runs are distinguished by line style and marker. The two left columns show that r=Tobs||r= T_obs|O| and ζ computed from the comparison graphs corresponding to a sub-sample of items remain relatively unchanged, making it a good estimator of LLM comparison consistency even with low comparison samples. The right column shows the same cannot be stated for inter-rank consistency ( Kendall’s τ =) between generated ranks using rank centrality from same comparison graphs, making it a relatively poor estimator of consistency without sufficiently large sample size. 4.1 Experimental Results 4.1.1 Subtournament Evaluation For each dataset, we select m=30m=30 representative items and run all (302)=435 302=435 pairwise comparisons per LLM judge, forming a complete subtournament. Item order in each comparison is chosen at random. We compute ζ exactly from the out-degree formula (Equation 1) and measure self-consistency τcons _cons by running the aggregator on two independent sets of comparisons drawn from the same judge. In addition, we apply the following bootstrap procedure. We repeatedly subsample 40%40\% of the edges of this subtournament. This matches the observation probability p=0.4p=0.4 in what we call the full pipeline, where all nodes are known, 40%40\% of edges are sampled, and a full ranking constructed. For each run, we record ζ, and in addition the (1002)=4,950 1002=4,950 pairwise τcons _cons values form an empirical distribution of self-consistency. Figure 4 shows the result of this experiment for inter-run variability. In almost every case, the full-pipeline τcons _cons, and the τ from the Stage 1 fall well to the right of the bootstrap distribution: the full pipeline is considerably more self-consistent than the subtournament. Meanwhile, the subtournament ζ ζ correctly estimates comparison quality (when scaled appropriately by Equation 5), as in the synthetic sparse-observation experiment (Figure 2): ζ ζ is unbiased regardless of observation budget. Figure 5 confirms this in the real data: as the number of items grows, ζ behaves as predicted by Equation (8) (the constant r regime) while τcons _cons increases steadily across all datasets and LLM judges. The full pipeline observes more comparisons per item than any single bootstrap subsample, so its rankings are more stable even though the underlying comparison noise is the same. 4.1.2 Full Sparse Pipeline: ζ and τobs _obs as Functions of n For each dataset, we run the full pipeline on all n items with edge sampling probability p=0.4p=0.4. This is to keep the number of calls to the LLM tractable. We estimate ζ ζ from the fully-observed triplets in the sparse data (Equation (7)) and measure self-consistency by running the pipeline twice on independent edge samples. To estimate how our two measures scale with n, for each dataset, we take the comparisons performed above, randomly sample fractions of nodes ∈0.1,0.325,0.55,0.775∈\0.1,0.325,0.55,0.775\, and construct a comparison graph with only the edges that had been sampled. We repeat this 100 times and empirically compute the expected τ,ζτ,ζ and r=Tobs||r= T_obs|O|. The results are provided in Figure 5. As n increases, the inter-run variability of the ranks produced also decreases. This is because τcons _cons depends on the number of effective comparisons per item, which increases as we get more nodes with the same sampling ratio of edges. However, the expected inconsistency rate r remains approximately constant across values of n, placing these datasets in the constant-r regime of the decomposition in Equation (5). This is qualitatively different from the BTL model, where Proposition 2 predicts r=Θ(1/n2)r= (1/n^2) and ζ→1ζ→ 1. 4.2 Consequences The discussion above demonstrates in practice that ζ and τcons _cons are different ways of measuring consistency of a ranking method, and they can behave quite differently. Another natural question is whether a method that is good along one dimension is naturally also good along another. Our experiments also allow us to examine this question in the context of the three LLMs that we use. Table 2 shows which of the models performs best/middle/worst along each of these dimensions across the four datasets we look at.222For complete results see Appendix B.3 We find that no model Pareto-dominates across both dimensions. In fact, Qwen is reliably best in terms of inter-run variability, but also reliably worse in terms of intra-run consistency! Meanwhile, LLaMa does best across all the datasets in terms of intra-run consistency, but it is twice worst and twice in the middle in terms of inter-run variability. DeepSeek typically occupies the middle position along both axes. Which dimension is more important? This will of course come down to a question of what a stakeholder values in a particular domain. But our results illustrate clearly the value of evaluating ranking along both these dimensions of consistency. TAY-VI-SPDAT VI-FSPDAT VI-SPDAT MIMIC LLM ζ¯ ζ τcons _cons ζ¯ ζ τcons _cons ζ¯ ζ τcons _cons ζ¯ ζ τcons _cons DeepSeek ∘ −- ∘ ∘ −- −- ∘ ∘ LLaMA ++ ∘ ++ −- ++ ∘ ++ −- Qwen −- ++ −- ++ ∘ ++ −- ++ Table 2: Relative model ranking by average ζ and Kendall rank correlation across the homelessness and MIMIC datasets. ++ indicates best, ∘ middle, and −- worst. 5 Discussion The central finding of this paper is that intra-run consistency and inter-run variance are genuinely independent axes of ranking reliability, not just in theory (Table 1) but in practice. The synthetic calibration shows that even under the idealized BTL model, where a single parameter governs all comparison noise, the two axes are correlated but not redundant: tournaments with identical ζ can have very different τcons _cons depending on the number of comparisons per item. The real-world experiments sharpen this separation further. The inconsistency rate r remains approximately constant as n grows, placing all four datasets in the constant-r regime of our decomposition and confirming that LLM comparison noise does not have the rank-gap-dependent structure of the BTL model (where Proposition 2 predicts r=Θ(1/n2)r= (1/n^2)). Most strikingly, no LLM dominates across both dimensions: Qwen produces the most stable rankings across runs but the least internally consistent comparisons, while LLaMA exhibits the opposite profile. This means that choosing a model on the basis of only one diagnostic can be actively misleading. A practitioner who evaluated only inter-run variance would select Qwen; one who evaluated only intra-run consistency would select LLaMA. Neither choice is wrong, but neither is complete. Our results suggest a simple diagnostic protocol. First, run a small complete subtournament and compute ζ exactly; because ζ is unbiased regardless of observation budget, this inexpensive step reliably estimates comparison quality before committing to the full pipeline. If ζ is low, the comparisons themselves are inconsistent in the sense of transitivity, and no amount of additional data will “fix” that issue. If ζ is high but τcons _cons is low, the comparisons are consistent but too few have been collected, and the remedy is to increase the sampling fraction rather than change the model. This diagnostic separation between signal quality and signal quantity is the practical payoff of the 2×22× 2 taxonomy. Acknowledgements We are grateful for support from NSF Award 2533162. References [1] G. G. Alway (1962-06) The distribution of the number of circular triads in paired comparisons. Biometrika 49 (1/2), p. 265. External Links: ISSN 00063444, Link, Document Cited by: §1.2, §2.1, Definition 2.2. [2] S. Ameli, S. Zhuang, I. Stoica, and M. W. Mahoney (2024) A statistical framework for ranking llm-based chatbots. External Links: Link, Document Cited by: §1. [3] V. Angelova, W. Dobbie, and C. S. Yang (2025-sept) Algorithmic recommendations and human discretion. Review of Economic Studies, p. rdaf084 (en). External Links: ISSN 0034-6527, 1467-937X, Link, Document Cited by: §1.2. [4] R. A. Bradley and M. E. Terry (1952-12) Rank analysis of incomplete block designs: i. the method of paired comparisons. Biometrika 39 (3/4), p. 324. External Links: ISSN 00063444, Link, Document Cited by: item 2, §1.2, §3.1. [5] R. P. Brent (2002) Algorithms for minimization without derivatives. Unabridged republication of the work publ. by Prentice-Hall … 1973 edition, Dover books on mathematics, Dover Publications, Mineola, NY (eng). External Links: ISBN 9780486419985 Cited by: §A.1. [6] S. Brin and L. Page (1998) The anatomy of a large-scale hypertextual web search engine. Computer Networks 30, p. 107–117. External Links: Link Cited by: Remark. [7] M. Brown, C. Cummings, J. Lyons, A. Carrión, and D. P. Watson (2018-07) Reliability and validity of the vulnerability index-service prioritization decision assistance tool (vi-spdat) in real-world implementation. Journal of Social Distress and the Homeless 27 (2), p. 110–117 (en). External Links: ISSN 1053-0789, 1573-658X, Link, Document Cited by: §1.2. [8] H. Cheng, L. Stapleton, A. Kawakami, V. Sivaraman, Y. Cheng, D. Qing, A. Perer, K. Holstein, Z. S. Wu, and H. Zhu (2022) How child welfare workers reduce racial disparities in algorithmic decisions. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. External Links: ISBN 9781450391573, Link, Document Cited by: §1.2. [9] W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. I. Jordan, J. E. Gonzalez, and I. Stoica (2024) Chatbot arena: an open platform for evaluating llms by human preference. ICML’24. Cited by: §1.2, §1, §1. [10] m. d. Condorcet (1785) Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Imprimerie Royale, Paris. Cited by: §1.2, §1. [11] C. Cronley (2022-01) Invisible intersectionality in measuring vulnerability among individuals experiencing homelessness – critically appraising the vi-spdat. Journal of Social Distress and Homelessness 31 (1), p. 23–33 (en). External Links: ISSN 1053-0789, 1573-658X, Link, Document Cited by: §1.2. [12] H. A. David (1969) The method of paired comparisons. 2. impr., with minor corr edition, Griffin’s statistical monographs and courses, Griffin, London (eng). External Links: ISBN 9780852640135 Cited by: §1.2. [13] K. Drobotowicz, J. Ylipulli, U. S. Varanasi, and H. S. Mäkitalo (2026) Automate, assist, avoid: caseworkers’ perspectives on applying large language model-based assistance in public sector decision-making processes. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI ’26, New York, NY, USA. External Links: ISBN 9798400722783, Link, Document Cited by: §1. [14] P. Emerson (2013-02) The original borda count and partial voting. Social Choice and Welfare 40 (2), p. 353–358 (en). External Links: ISSN 0176-1714, 1432-217X, Link, Document Cited by: §1.2, §1. [15] M. Gupta and D. Kumar (2026-04) Diagnosing llm judge reliability: conformal prediction sets and transitivity violations. (arXiv:2604.15302). Note: arXiv:2604.15302 External Links: Link, Document Cited by: §1.2. [16] G. B. Haim, M. Saban, Y. Barash, D. Cirulnik, A. Shaham, B. Z. Eisenman, L. Burshtein, O. Mymon, and E. Klang (2024-11) Evaluating large language model‐assisted emergency triage: a comparison of acuity assessments by gpt ‐4 and medical experts. p. jocn.17490 (en). External Links: ISSN 0962-1067, 1365-2702, Link, Document Cited by: §1. [17] D. R. Hunter (2004-02) M algorithms for generalized bradley-terry models. The Annals of Statistics 32 (1). External Links: ISSN 0090-5364, Link, Document Cited by: §3. [18] Y. Iida (2009) The number of circular triads in a pairwise comparison matrix and a consistency test in the ahp. Journal of the Operations Research Society of Japan 52 (2), p. 174–185 (en). External Links: ISSN 0453-4514, 2188-8299, Link, Document Cited by: §1.2. [19] X. Jiang, L. Lim, Y. Yao, and Y. Ye (2011-03) Statistical ranking and combinatorial hodge theory. Mathematical Programming 127 (1), p. 203–244 (en). External Links: ISSN 0025-5610, 1436-4646, Link, Document Cited by: §1.1, §1.2. [20] A. Johnson, L. Bulgarelli, T. Pollard, L. A. Celi, R. Mark, and S. Horng (2023-01) MIMIC-IV-ED. PhysioNet. Note: Version 2.2 External Links: Document, Link Cited by: §B.2. [21] A. Johnson, L. Bulgarelli, T. Pollard, B. Gow, B. Moody, S. Horng, L. A. Celi, and R. Mark (2024-10) MIMIC-IV. PhysioNet. Note: Version 3.1 External Links: Document, Link Cited by: §B.2, §4. [22] M. G. Kendall and B. B. Smith (1940) On the method of paired comparisons. Biometrika 31 (3–4), p. 324–345 (en). External Links: ISSN 0006-3444, 1464-3510, Link, Document Cited by: §1.1, §1.2, §1, §1, §2.1, Definition 2.2. [23] M. G. Kendall (1938-06) A new measure of rank correlation. Biometrika 30 (1/2), p. 81. External Links: ISSN 00063444, Link, Document Cited by: §1.1, §2.1, §2. [24] A. R. Kube, S. Das, and P. J. Fowler (2023) Fair and efficient allocation of scarce resources based on predicted outcomes: implications for homeless service delivery. Journal of Artificial Intelligence Research 76, p. 1219–1245. Cited by: §1.2. [25] J. Lee, T. Shang, J. Y. Baik, D. A. Duong-Tran, S. Yang, L. Li, and L. Shen (2025) Investigating llms in clinical triage: promising capabilities, persistent intersectional biases. ArXiv abs/2504.16273. External Links: Link Cited by: §1. [26] M. Lipsky (1980) Street-level bureaucracy: dilemmas of the individual in public services. Russell Sage Foundation, New York. Cited by: §1.2. [27] Y. Liu, H. Zhou, Z. Guo, E. Shareghi, I. Vulić, A. Korhonen, and N. Collier (2024) Aligning with human judgement: the role of pairwise preference in large language model evaluators. External Links: Link, Document Cited by: §1. [28] R. D. Luce (1959) Individual choice behavior. Vol. 4, Wiley New York. Cited by: item 2, §1.2, §3.1. [29] S. Negahban, S. Oh, and D. Shah (2017-02) Rank centrality: ranking from pairwise comparisons. Operations Research 65 (1), p. 266–287 (en). External Links: ISSN 0030-364X, 1526-5463, Link, Document Cited by: §1.2, §1, §3.1, §3. [30] OrgCode Consulting Inc. and Community Solutions (2015) Family Service Prioritization Decision Assistance Tool (F-SPDAT): U.S. Version 2.0. Note: https://everyonehome.org/wp-content/uploads/2016/02/F-SPDAT-2.0-Families.pdfAccessed May 17, 2025 Cited by: §B.1, §1.2, §1. [31] OrgCode Consulting Inc. and Community Solutions (2015) Vulnerability Index–Service Prioritization Decision Assistance Tool (VI-SPDAT): Prescreen Triage Tool for Single Adults. Community Solutions. Note: https://everyonehome.org/wp-content/uploads/2016/02/VI-SPDAT-2.0-Single-Adults.pdfAccessed May 17, 2025 Cited by: §B.1, §1.2, §1. [32] OrgCode Consulting Inc., Corporation for Supportive Housing, Community Solutions, and Eric Rice (2015) Next Step Tool for Homeless Youth (TAY-VI-SPDAT): U.S. Version 1.0. Note: https://letsendhomelessness.org/wp-content/uploads/2018/07/TAY-VI-SPDAT-v1-0-w.-Intro-Script.pdfAccessed May 17, 2025 Cited by: §B.1, §1.2, §1. [33] A. Panickssery, S. R. Bowman, and S. Feng (2024) LLM evaluators recognize and favor their own generations. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: §1.2. [34] J. I. Perotti (2025-03) Analysis of the inference of ratings and rankings in complex networks using discrete exterior calculus on higher-order networks. Physical Review E 111 (3), p. 034306 (en). External Links: ISSN 2470-0045, 2470-0053, Link, Document Cited by: §1.2. [35] G. Persad, A. Wertheimer, and E. J. Emanuel (2009-01) Principles for allocation of scarce medical interventions. The Lancet 373 (9661), p. 423–431 (en). External Links: ISSN 01406736, Link, Document Cited by: §1.2. [36] G. Pokharel, S. Das, and P. J. Fowler (2024) Discretionary trees: understanding street-level bureaucracy via machine learning. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24. External Links: ISBN 978-1-57735-887-9, Link, Document Cited by: §1.2. [37] G. Pokharel, S. Farabi, P. J. Fowler, and S. Das (2025-10) Street-level AI: are large language models ready for real-world judgments?. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8 (3), p. 2043–2054. External Links: ISSN 3065-8365, Link, Document Cited by: §B.1, §1.2, §1, §4. [38] G. Pokharel (2025-10) Beyond automation: understanding fairness, ethics, and human discretion in ai-driven societal decisions. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8 (3), p. 2918–2920. External Links: ISSN 3065-8365, Link, Document Cited by: §1.2. [39] M. Qian, G. Sun, M. J. F. Gales, and K. M. Knill (2026) Who can we trust? llm-as-a-jury for comparative assessment. External Links: Link, Document Cited by: §1.2. [40] K. Saito, A. Wachi, K. Wataoka, and Y. Akimoto (2023-10) Verbosity bias in preference labeling by large language models. (en). External Links: Link Cited by: §1.2, §1. [41] D. Saxena and S. Guha (2024-03) Algorithmic harms in child welfare: uncertainties in practice, organization, and street-level decision-making. 1 (1). External Links: Link, Document Cited by: §1.2. [42] N. Shah, S. Balakrishnan, A. Guntuboyina, and M. Wainwright (2016-20–22 Jun) Stochastically transitive models for pairwise comparisons: statistical and computational issues. In Proceedings of The 33rd International Conference on Machine Learning, M. F. Balcan and K. Q. Weinberger (Eds.), Proceedings of Machine Learning Research, Vol. 48, New York, New York, USA, p. 11–20. External Links: Link Cited by: §3. [43] C. Starke, J. Baleis, B. Keller, and F. Marcinkowski (2022-07) Fairness perceptions of algorithmic decision-making: a systematic review of the empirical literature. Big Data & Society 9 (2) (en). External Links: ISSN 2053-9517, 2053-9517, Link, Document Cited by: §1.2. [44] L. L. Thurstone (1994) A law of comparative judgment.. Psychological review 101 (2), p. 266. Cited by: §1.2. [45] U.S. Department of Housing and Urban Development, Office of Community Planning and Development (2017-01) Notice establishing additional requirements for a continuum of care centralized or coordinated assessment system. Technical report Technical Report Notice CPD-17-01. External Links: Link Cited by: §1.2. [46] F. Wan, K. Wang, T. Wang, H. Qin, J. Fondrevelle, and A. Duclos (2025) Enhancing healthcare resource allocation through large language models. Swarm and Evolutionary Computation 94, p. 101859. External Links: ISSN 2210-6502, Document, Link Cited by: §1. [47] P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui (2024) Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 9440–9450 (en). External Links: Link, Document Cited by: §1.2, §1. [48] V. Wang, M. J. Zhang, and E. Choi (2025) Improving llm-as-a-judge inference with the judgment distribution. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, p. 23173–23199 (en). External Links: Link, Document Cited by: §1.2. [49] Y. Wang, Y. Song, T. Zhu, X. Zhang, Z. Yu, H. Chen, C. Song, Q. Wang, C. Wang, Z. Wu, X. Dai, Y. Zhang, W. Ye, and S. Zhang (2025-sept) TrustJudge: inconsistencies of llm-as-a-judge and how to alleviate them. (arXiv:2509.21117). Note: arXiv:2509.21117 External Links: Link, Document Cited by: §1.2. [50] C. Wilkey, R. Donegan, S. Yampolskaya, and R. Cannon (2019-10) Coordinated entry systems: racial equity analysis of assessment data. Technical report C4 Innovations. External Links: Link Cited by: §1.2. [51] C. Y. Williams, T. Zack, B. Y. Miao, M. Sushil, M. Wang, A. E. Kornblith, and A. J. Butte (2024) Use of a large language model to assess clinical acuity of adults in the emergency department. JAMA network open 7 (5), p. e248895. Cited by: §1. [52] L. Wolf, K. Ceci, D. McCallum, and D. BrecherC. Zahn (Ed.) (2023) Emergency severity index handbook. 5th edition, Emergency Nurses Association, Schaumburg, IL. External Links: Link Cited by: §B.2, §1.2, §1, §4. [53] Y. Xu, L. Ruis, T. Rocktäschel, and R. Kirk (2025) Investigating non-transitivity in llm-as-a-judge. External Links: Link, Document Cited by: §1.2. [54] Y. Yu, Y. Liu, M. He, S. Tao, W. Meng, X. Yang, L. Zhang, H. Ma, D. Li, D. Wei, B. Chen, and F. Li (2026-03) ELSPR: evaluator llm training data self-purification on non-transitive preferences via tournament graph reconstruction. 40, p. 17975–17983. External Links: ISSN 2374-3468, 2159-5399, Link, Document Cited by: §1.2. [55] Y. Zeng, O. Tendolkar, R. Baartmans, Q. Wu, L. Chen, and H. Wang (2024) LLM-rankfusion: mitigating intrinsic inconsistency in llm-based ranking. External Links: Link, Document Cited by: §1.2, §1. Appendix A Additional Details on the Synthetic Model/Experiments This appendix provides implementation details and supplementary experiments for the synthetic experiments in Section 3. Throughout, error bars on the y-axis represent ±1± 1 standard deviation across 100 independent trials. Error bars on the x-axis, where present, represent the standard deviation of the observed ζ around its analytic expectation [ζ]E[ζ]. Remark (Regularization of RC at High β). RC ranks items by the stationary distribution of a Markov chain whose transition matrix P is constructed from the tournament outcomes. In the single-comparison setting, an item that wins all n−1n-1 of its comparisons becomes an absorbing state: Pii=1P_i=1 and the stationary distribution places all mass there, assigning zero score to every other item. Under the equally-spaced BTL model this occurs with probability ∏j=1n−1σ(βj) _j=1^n-1σ(β j), which approaches 1 for moderate β. Following the standard PageRank remedy [6], we replace P with ′=(1−α)+(α/n) P =(1-α) P+(α/n) J, where J is the all-ones matrix and α=0.01α=0.01. This eliminates absorbing states while leaving rankings in the moderate-noise regime essentially unchanged. A.1 Matching ζ across n To ensure that calibration curves for different values of n are evaluated at the same set of ζ values, we solve for the value of β that yields a given target [ζ]E[ζ]. Under the equally-spaced BTL model (Definition 3.1), the expected circular triad count admits an exact closed form: [Tn]=∑i<j<k[PijPjkPki+PjiPkjPik],E[T_n]= _i<j<k [P_ij\,P_jk\,P_ki\;+\;P_ji\,P_kj\,P_ik ], (10) where Pij=σ(β(j−i))P_ij=σ (β(j-i) ) is the pairwise comparison probability from Equation (9). The expected coefficient of consistency is then [ζ]=1−[Tn]/TmaxE[ζ]=1-E[T_n]/T_ . Because the mapping β↦[ζ]β [ζ] is smooth and strictly increasing on [0,∞)[0,∞), ranging from 0 as β→0β→ 0 to 11 as β→∞β→∞, there exists a unique β∗β^* satisfying [ζ(β∗,n)]=ζ∗E[ζ(β^*,n)]=ζ^* for every target ζ∗∈(0,1)ζ^*∈(0,1) and every n. We find β∗β^* via Brent’s root-finding method [5] applied to [ζ(β,n)]−ζ∗=0E[ζ(β,n)]-ζ^*=0, with tolerance 10−810^-8. This allows all experiments to be evaluated at a common grid of ζ values (0.1,0.2,…,0.9)(0.1,0.2,…,0.9), eliminating the confound of different n values producing different ζ ranges at the same β. A.2 Proof of Proposition 2 See 2 Proof. Triad anatomy. For a triple i<j<ki<j<k with rank gaps a=j−ia=j-i and b=k−jb=k-j, the third pairwise distance is a+ba+b. Let qd=σ(−βd)q_d=σ(-β d) be the upset probability at gap d. Under BTL the three pair outcomes are independent. The triad is cyclic in exactly two configurations: • single long-edge upset: i beats j, j beats k, k beats i; • two short-edge upsets: j beats i, k beats j, i beats k. Every other configuration is transitive. Therefore: P(cycle∣a,b)=(1−qa)(1−qb)qa+b+qaqb(1−qa+b).P(cycle a,b)=(1-q_a)(1-q_b)q_a+b+q_aq_b(1-q_a+b). Upper bound on E[Tn]E[T_n]. Since σ(−x)≤e−xσ(-x)≤ e^-x for x≥0x≥ 0, P(cycle∣a,b)≤qa+b+qaqb≤2e−β(a+b).P(cycle a,b)≤ q_a+b+q_aq_b≤ 2e^-β(a+b). The number of triples with gap pattern (a,b)(a,b) is n−a−bn-a-b, so: E[Tn]=∑a,b≥1a+b≤n−1(n−a−b)P(cycle∣a,b)≤2n∑a,b≥1e−β(a+b)=2n(e−β1−e−β)2=O(n),E[T_n]= _ subarrayca,b≥ 1\\ a+b≤ n-1 subarray(n-a-b)P(cycle a,b)≤ 2n _a,b≥ 1e^-β(a+b)=2n ( e^-β1-e^-β )^2=O(n), with constant depending only on β. Lower bound on E[Tn]E[T_n]. Consecutive triples have a=b=1a=b=1 and there are n−2n-2 of them. For each, P(cycle∣1,1)≥q12(1−q2),P(cycle 1,1)≥ q_1^2(1-q_2), a strictly positive constant in β. Hence E[Tn]≥(n−2)q12(1−q2)=Ω(n)E[T_n]≥(n-2)q_1^2(1-q_2)= (n). Denominator. Tmax=(n3−n)/24T_ =(n^3-n)/24 for odd n and (n3−4n)/24(n^3-4n)/24 for even n, both Θ(n3) (n^3). Combining, E[1−ζ]=Θ(n)/Θ(n3)=Θ(1/n2)E[1-ζ]= (n)/ (n^3)= (1/n^2). ∎ Why the upward shift in Figure 1(b) with n The expected number of circular triads is carried by triples with small rank gaps, since the cycle probability is exponentially small in the long-edge gap a+ba+b. Small-gap triples are linear in n, while the total triad count is cubic. ζ therefore mixes a linear numerator with a cubic denominator, and at fixed β it approaches 11 at rate 1/n21/n^2. Matching two tournaments at the same value of ζ across different n is consequently matching them on a statistic whose denominator absorbs Θ(n3) (n^3) trivially-transitive triples; the larger-n tournament must have substantially smaller β (noisier per-pair comparisons) to reach the same ζ. Self-consistency, by contrast, is controlled by the n−1n-1 comparisons each item participates in and tracks per-pair noise directly through aggregation accuracy. A.3 Aggregator Comparison Here, we test whether the diagnostic value of ζ is specific to Rank Centrality or generalizes across aggregation methods. We fix n=50n=50 and run three aggregators on the same sampled tournaments: Rank Centrality, Bradley–Terry Maximum Likelihood Estimation (BT-MLE, via the iterative Luce spectral ranking algorithm), and Borda count (ranking by total win count). For each aggregator, we record accuracy τacc _acc and self-consistency τcons _cons at each target ζ value. (a) (b) Figure 6: Aggregator comparison (n=50n=50). (a) ζ vs. accuracy for RC, BT-MLE, and Borda; all three aggregators yield nearly identical accuracy as a function of ζ, indicating that comparison quality dominates the choice of aggregation method. (b) ζ vs. self-consistency for the same four aggregators; the curves are again nearly coincident. Figure 6(a) shows that all three aggregators produce nearly identical accuracy curves as a function of ζ. Figure 6(b) shows the same for self-consistency. This establishes that ζ is a diagnostic for the data, not for any particular aggregation algorithm – the quality of the pairwise comparisons, as captured by ζ, is the dominant factor, and the choice of aggregator is secondary. Appendix B Dataset Details Both experimental domains share a common structure: each case is characterized by a set of features, a human expert assigns a discrete priority level, and the task is to decide which of two cases should be prioritized for intervention. The domains differ in population, feature representation, and label granularity. B.1 Homelessness Service Allocation We use household-level vulnerability assessments from St. Louis coordinated entry records (2021-2024) [37]. The data comprise assessments from three population-specific VI-SPDAT instruments: VI-SPDAT for single adults (n=325n=325), VIF-SPDAT for families (n=698n=698), and TAY-VISPDAT for transition-age youth (n=561n=561). Each record contains the full set of raw questionnaire responses (35 items for single adults, 54 for families, 41 for youth; see Appendix D for the full set of questions) together with the corresponding total acuity score used in practice for prioritization. We treat the three instruments as separate datasets throughout, since they correspond to different populations with different questionnaires and scoring rubrics. Each household is assigned to a priority band based on its total score. For single adults and youth, the bands are low (0–33), medium (44–77), and high (8+8+). For families, the bands are low (0–33), medium (44–88), and high (9+9+), following the established scoring rubrics [31, 30, 32]. Table 3 summarizes the thresholds, pool sizes, and the subtournament samples drawn with proportional representation across bands. Assessment Band Threshold Pool Pool % Sample VI-SPDAT (n=325n=325) Low 0–33 61 18.8% 6 Medium 44–77 175 53.8% 16 High 8+8+ 89 27.4% 8 VIF-SPDAT (n=698n=698) Low 0–33 45 6.4% 2 Medium 44–88 415 59.5% 18 High 9+9+ 238 34.1% 10 TAY-VISPDAT (n=561n=561) Low 0–33 49 8.7% 3 Medium 44–77 281 50.1% 15 High 8+8+ 231 41.2% 12 Table 3: Priority band score thresholds, pool sizes, and subtournament sample sizes (m=30m=30) by assessment type. Samples are drawn with proportional representation across priority bands. B.2 Emergency Department Triage We use emergency department triage data from MIMIC-IV [21], a publicly available de-identified electronic health record database from Beth Israel Deaconess Medical Center in Boston. We link three MIMIC-IV modules: the ED module [20] (triage assessments and medication reconciliation), the hospital module (patient demographics), and the admissions table (supplementary sociodemographic fields for admitted patients). ESI Category Definition Pool Pool % m=30m=30 n=500n=500 1 Immediate Requires immediate life-saving intervention (e.g., cardiac arrest, respiratory failure) 5,600 3.1% 1 16 2 Emergent High-risk situation; patient should not wait (e.g., severe pain, altered mental status) 60,280 33.5% 10 167 3 Urgent Stable, but expected to require multiple resources (e.g., labs, imaging, IV medications) 98,919 54.9% 17 275 4 Less urgent Stable, expected to require one resource (e.g., a single lab test or X-ray) 14,726 8.2% 2 41 5 Non-urgent Stable, expected to require no resources (e.g., prescription refill, simple wound check) 532 0.3% 0 1 Table 4: Emergency Severity Index (ESI) acuity levels, clinical definitions, and sampling details. From 180,057180,057 eligible MIMIC-IV encounters, we draw two cohorts with proportional representation across ESI levels: a subtournament (m=30m=30) and a full pipeline (n=500n=500). Each record corresponds to a single ED visit. At arrival, triage nurses record temperature, heart rate, respiratory rate, oxygen saturation, systolic and diastolic blood pressure, a patient-reported pain score (0–10), and a free-text chief complaint. Medication reconciliation data document pre-visit medications, serving as a proxy for medical history. Demographics available at triage include age, gender, race, and mode of arrival (e.g., ambulance, walk-in); primary language and marital status are additionally available for admitted patients. Category Prompt label Description Demographics age Patient age at time of visit gender Patient gender race Patient self-reported race Arrival arrival_mode Mode of arrival (e.g., ambulance, walk-in) Vital signs temperature_in_fahrenheit Body temperature (∘F) heart_rate_bpm Heart rate (beats per minute) respiratory_rate Respiratory rate (breaths per minute) o2_saturation_percent Peripheral oxygen saturation (%) systolic_blood_pressure_mmhg Systolic blood pressure (mmHg) diastolic_blood_pressure_mmhg Diastolic blood pressure (mmHg) pain_score Patient-reported pain (0–10) Clinical presentation chief_complaint Free-text reason for presenting Medications current_medications Medications taken prior to visit medication_classes Therapeutic classes of medications num_current_medications Number of current medications Table 5: Features presented to the LLM for each patient in pairwise triage comparisons. Field names are mapped from the original MIMIC-IV schema to the labels used in the prompt. The ground-truth label is the Emergency Severity Index (ESI) acuity score, an integer from 1 (most acute) to 5 (least acute) assigned by the triage nurse. ESI is a validated five-level triage algorithm widely used in U.S. emergency departments [52]. Levels 1 and 2 are determined by patient acuity (immediate life-saving need vs. high-risk, should not wait); Levels 3 through 5 are distinguished by the number of resources the patient is expected to require. Table 4 summarizes the ESI levels, pool sizes, and the two cohorts sampled with proportional representation. We restrict the sample to ED visits with a valid acuity score, excluding encounters in which the patient eloped, left without being seen, or left against medical advice. We require all triage feature fields to be non-missing and retain only the most recent visit per patient to avoid within-patient correlation. The resulting eligible pool contains 180,057180,057 unique patients. B.3 Experimental Data underlying Table 2 LLM ζ1 _1 ζ2 _2 ζ¯ ζ τcons _cons DeepSeek 0.849144 0.852818 0.850981 0.887179 LLaMA 0.917566 0.916106 0.916836 0.923300 Qwen 0.838598 0.839361 0.838979 0.924599 Table 6: ζ1 _1, ζ2 _2, mean ζ¯ ζ, and Kendall τcons _cons for TAY-VI-SPDAT. LLM ζ1 _1 ζ2 _2 ζ¯ ζ τcons _cons DeepSeek 0.812661 0.817402 0.815031 0.884729 LLaMA 0.871484 0.861638 0.866561 0.879903 Qwen 0.792314 0.794580 0.793447 0.930426 Table 7: ζ1 _1, ζ2 _2, mean ζ¯ ζ, and Kendall τcons _cons for VI-FSPDAT. LLM ζ1 _1 ζ2 _2 ζ¯ ζ τcons _cons DeepSeek 0.834824 0.842320 0.838572 0.852877 LLaMA 0.928576 0.928845 0.928711 0.897892 Qwen 0.881298 0.873071 0.877184 0.902564 Table 8: ζ1 _1, ζ2 _2, mean ζ¯ ζ, and Kendall τcons _cons for VI-SPDAT. LLM ζ1 _1 ζ2 _2 ζ¯ ζ τcons _cons DeepSeek 0.837359 0.840854 0.839107 0.896449 LLaMA 0.900303 0.898932 0.899618 0.894846 Qwen 0.819045 0.819877 0.819461 0.897267 Table 9: ζ1 _1, ζ2 _2, mean ζ¯ ζ, and Kendall τcons _cons for MIMIC. Appendix C LLM Prompts Each pairwise comparison is presented to the LLM as a single-turn conversation. The prompt contains the full feature representation of both cases (questionnaire responses for homelessness, triage features for ED visits) and asks the model to select which case should be prioritized, along with a short justification. We found that eliciting a brief explanation alongside the structured decision improved consistency compared to requesting only the formatted template. The placeholder <insert block data> is replaced with the relevant case data for each comparison. C.1 Pairwise Comparison Prompts Figures 7 and 8 show the prompts used for the homelessness and ED triage domains respectively. The homelessness prompt is shared across all three VI-SPDAT instruments, with the questionnaire content varying by assessment type. Figure 7: Pairwise comparison prompt for homelessness prioritization (VI-SPDAT, VIF-SPDAT, and TAY-VISPDAT datasets). Figure 8: Pairwise comparison prompt for emergency department triage (MIMIC-IV dataset). C.2 Second-Level Response Parsing In practice, we found that LLMs occasionally produced contradictory outputs: the structured template would indicate one decision while the free-text explanation would argue for the opposite. To resolve these conflicts, we use a second-level LLM (Qwen2.5-7B-Instruct) to parse the full output of the decision-making LLM and extract the final prioritization decision. Figures 9 and 10 show the parsing prompts for the homelessness and ED triage domains respectively. Figure 9: Second-level parsing prompt for homelessness prioritization outputs. Figure 10: Second-level parsing prompt for emergency department triage outputs. Appendix D Questionnaires used in VI-SPDAT system to score vulnurablity D.1 VI-SPDAT 1. Are you currently able to care for your basic needs, such as bathing, changing clothes, using the restroom, obtaining food, and accessing clean water? 2. Are you not taking any medications that a doctor has prescribed for you? 3. Are you taking prescribed painkillers incorrectly or selling them instead of using them as directed? 4. Do you currently have legal issues that might result in incarceration, fines, or difficulties in renting housing? 5. Do you engage in risky behaviors, such as exchanging sex for money, running drugs, having unprotected sex with strangers, sharing needles, or similar activities? 6. Do you have a learning disability, developmental disability, or any other impairment? 7. Do you have a mental health issue or concern? 8. Do you have any mental health or cognitive issues that make it difficult to live independently? 9. Do you have any physical disabilities that limit the type of housing you can access or make it difficult to live independently? 10. Do you have planned activities—aside from mere survival—that make you feel happy and fulfilled? 11. Do you receive income from the government, a pension, an inheritance, informal work, or a regular job? 12. Do you suffer from any chronic health issues involving your liver, kidneys, stomach, lungs, or heart? 13. Does anyone force or trick you into doing things against your will? 14. For female respondents only: Are you currently pregnant? 15. Has your alcohol or drug use resulted in you being kicked out of an apartment or shelter program in the past? 16. Has your current period of homelessness been caused by experiencing emotional, physical, psychological, sexual, or other trauma? (Please answer YES or NO) 17. Have you been attacked or beaten up since you became homeless? 18. Have you been hospitalized as an inpatient? 19. Have you ever had to leave your apartment, shelter program, or other living arrangement because of physical health issues? 20. Have you experienced a head injury in the past? 21. Have you received health care at an emergency department or room? 22. Have you spent one or more nights in a holding cell, jail, or prison, regardless of the duration? 23. Have you taken an ambulance to the hospital? 24. Have you talked to the police because you witnessed a crime, were a victim or suspect, or were told to move along? 25. Have you used a crisis service, such as those for sexual assault, mental health, family/intimate violence, distress, or suicide prevention? 26. How long has it been since you lived in permanent, stable housing? 27. If space were available in a program that specifically assists people living with HIV or AIDS, would you be interested? 28. In the last three years, how many times have you experienced homelessness? 29. In the last year, have you threatened or attempted to harm yourself or someone else? 30. Is there any person or entity (for example, a past landlord, business, bookie, dealer, or government group like the IRS) that believes you owe them money? 31. Is your current homelessness caused by a relationship breakdown, an unhealthy or abusive relationship, or actions by family or friends leading to eviction? 32. When you are sick or not feeling well, do you avoid seeking help? 33. Where do you sleep most frequently? (choose one) • If you selected ‘Other’, please specify further details. 34. Will alcohol or drug use make it difficult for you to maintain or afford housing? D.2 VI-F-SPDAT 1. Has any child in the family experienced abuse or trauma in the last 180 days? 2. Stayed one or more nights in a holding cell, jail, or prison, whether that was a short-term stay like the drunk tank, a longer stay for a more serious offense, or anything in between? 3. Are any prescribed painkillers not taken as directed or being sold instead of used? 4. Are there any children in your family aged 11 or younger? 5. Are there any children in your family aged 6 or younger? 6. Are there any family legal issues currently in or needing court resolution that might impact your housing or household composition? 7. Do children aged 12 or younger spend 2 or more hours per day in activities? 8. Do children aged 13 or older spend 3 or more hours per day in activities? 9. Do you anticipate any additional adults or children joining your household within the first 180 days of being housed? 10. Do your older children spend 2 or more hours per day helping their younger siblings with tasks such as preparing for school, homework, dinner, or bathing? 11. Does any single member of your household have a medical condition, mental health concern, and issues with substance use? 12. Does anyone in your family currently have legal issues that might lead to incarceration, fines, or difficulties in renting housing? 13. Does anyone in your family force or trick you into doing things against your will? 14. Does anyone in your family have a learning disability, developmental disability, or any other impairment? 15. Does anyone in your family have a mental health issue or concern? 16. Does anyone in your family have mental health or cognitive issues that make independent living challenging? 17. Does anyone in your family have physical disabilities that limit the type of housing accessible or make independent living challenging? 18. Does anyone in your family suffer from chronic health issues involving their liver, kidneys, stomach, lungs, or heart? 19. Does every member of your family have planned activities—beyond just survival—that bring happiness and fulfillment? 20. Does your family engage in risky behaviors such as exchanging sex for money, running drugs, having unprotected sex, sharing needles, or similar activities? 21. Does your family have two or more planned activities each week, such as outings, library visits, family gatherings, or movie nights? 22. Does your family receive income from the government, a pension, inheritance, informal work, or a regular job? 23. Has alcohol or drug use by anyone in your family resulted in being evicted from your housing or shelter program? 24. Has any family member been hospitalized as an inpatient? 25. Has any family member received care at an emergency department or room? 26. Has any family member taken an ambulance to the hospital? 27. Has anyone in your family been attacked or beaten up since becoming homeless? 28. Has anyone in your family experienced a head injury in the past? 29. Has your family composition changed in the last 180 days due to factors such as divorce, children rejoining, someone leaving for military service or incarceration, or a relative moving in? 30. Has your family ever had to leave an apartment, shelter, or other accommodation due to physical health issues? 31. Has your family spoken to the police because they witnessed a crime, were victims, or were suspects, or because the police advised them to move along? 32. Has your family used a crisis service such as those for sexual assault, mental health, family/intimate violence, distress, or suicide prevention? 33. Has your family’s current homelessness been caused by experiencing emotional, physical, psychological, sexual, or other trauma? (Please answer YES or NO) 34. Have any children been removed from your family by a child protection service in the last 180 days? 35. How long has it been since you and your family lived in permanent, stable housing? 36. How many children under the age of 18 are currently with you? 37. How many children under the age of 18 are not currently with your family but are expected to join you when you get housed? 38. How many parents are in your family? 39. If space were available in a program for people living with HIV or AIDS, would your family be interested? 40. If there are school-aged children, do they attend school most weeks? 41. If your household includes a female, is any member currently pregnant? 42. In the last 180 days, have any children lived with family or friends due to your housing situation or homelessness? 43. In the last three years, how many times has your family experienced homelessness? 44. In the last year, has anyone in your family threatened or attempted to harm themselves or others? 45. Is every member of your family capable of taking care of basic needs such as bathing, changing clothes, using the restroom, obtaining food, and accessing clean water? 46. Is the head of your household 60 years of age or older? 47. Is there any medication that a doctor prescribed for you or your family that is not being taken? 48. Is there any person or entity (e.g., past landlord, business, bookie, dealer, government group like the IRS) that believes your family owes them money? 49. Is your family’s current homelessness caused by a relationship breakdown, an unhealthy or abusive relationship, or interventions by family or friends? 50. Please provide a list of children’s names and ages. 51. When a family member is sick or unwell, does your family avoid seeking medical help? 52. Where do you and your family sleep most frequently? (choose one) • If you selected ‘Other’, please specify further details. 53. Will alcohol or drug use make it difficult for your family to maintain or afford housing? D.3 TAY-VI-SPDAT 1. Are you currently able to care for your basic needs (such as bathing, changing clothes, using a restroom, obtaining food, and accessing clean water)? 2. Are you not taking any medications that a doctor prescribed for you? 3. Are you taking prescribed painkillers incorrectly or selling them instead of using them as directed? 4. Did conflicts regarding gender identity or sexual orientation contribute to your homelessness? 5. Did differences in religious or cultural beliefs with your parents, guardians, or caregivers lead to your homelessness? 6. Did you become homeless because you ran away from your family home, a group home, or a foster home? 7. Do you currently have legal issues that might lead to incarceration, fines, or difficulties renting housing? 8. Do you engage in risky behaviors (such as exchanging sex for money, food, drugs, or a place to stay; running drugs; having unprotected sex with strangers; sharing needles; etc.)? 9. Do you have a learning disability, developmental disability, or any other impairment? 10. Do you have a mental health issue or concern? 11. Do you have any mental health or cognitive issues that hinder your ability to live independently? 12. Do you have physical disabilities that limit your housing options or make living independently difficult? 13. Do you have planned activities—beyond mere survival—that make you feel happy and fulfilled? 14. Do you receive income from the government, an inheritance, an allowance, informal work, or a regular job? 15. Do you suffer from chronic health issues involving your liver, kidneys, stomach, lungs, or heart? 16. Does anyone force or trick you into doing things against your will? 17. Has your alcohol or drug use resulted in eviction from your housing or shelter program? 18. Have you been attacked or beaten up since you became homeless? 19. Have you been hospitalized as an inpatient? 20. Have you ever been pregnant, or have you ever gotten someone pregnant? 21. Have you ever had to leave your living situation due to physical health issues? 22. Have you experienced a head injury in the past? 23. Have you received care at an emergency department or room? 24. Have you spent one or more nights in a holding cell, jail, prison, or juvenile detention, regardless of the duration? 25. Have you taken an ambulance to the hospital? 26. Have you talked to the police because you witnessed a crime, were a victim or suspect, or were advised to move along? 27. Have you used a crisis service (for sexual assault, mental health, family/intimate violence, distress, or suicide prevention)? 28. How long has it been since you lived in permanent, stable housing? 29. If space were available in a program for people living with HIV or AIDS, would you be interested? 30. If you have tried marijuana, did you first try it at age 12 or younger? 31. In the last three years, how many times have you experienced homelessness? 32. In the last year, have you threatened or attempted to harm yourself or others? 33. Is there any person or entity (e.g., a past landlord, business, bookie, dealer, or the IRS) that believes you owe money? 34. Was your homelessness a result of violence among family members at home? 35. Was your homelessness caused by an unhealthy or abusive relationship? 36. Was your homelessness caused by your family or friends? 37. Were you ever incarcerated before the age of 18? 38. When you are sick or unwell, do you avoid seeking medical help? 39. Where do you sleep most frequently? (choose one) • If you selected ‘Other’, please specify further details. 40. Will alcohol or drug use make it difficult for you to maintain or afford housing?