Paper deep dive
Language-model ratings of depression reflect the rater more than the patient
Baihan Lin
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Depression has no diagnostic blood test. Language models promise tireless, consistent assessment, but can accurate raters disagree about individuals? We pre-registered 880 language-model raters, crossing 11 open models with prompting and scoring choices, and applied them to 189 interviews against the eight-item Patient Health Questionnaire. Model choice explained 30.0% of summed-symptom score variance, stable participant differences 10.5%. Two randomly drawn raters with area under the receiver operating characteristic curve (AUC) >= 0.70 disagreed on screening decisions for 40% of participants, on average. Average over-rating governed how many were flagged, yet equal-capacity raters chose differently for about one participant in five. A locked analysis of 86 new interviews reproduced the main pre-registered findings. Exploratory recalibration with 40 labelled participants raised accuracy from about 60% to 75% and halved disagreement, leaving one participant in five decided differently. Calibration repaired much of the rater dependence without securing agreement about individuals.
Tags
Links
- Source: https://arxiv.org/abs/2610.08501v1
- Canonical: https://arxiv.org/abs/2610.08501v1
Trouble viewing inline? Open PDF directly →
Full Text
128,205 characters extracted from source content.
Expand or collapse full text
Language-model ratings of depression reflect the rater more than the patient Baihan Lin Affiliation: Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA Affiliation: Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA Affiliation: Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA Affiliation: Mental Illness Research, Education and Clinical Center, James J. Peters VA Medical Center, Bronx, NY, USA Affiliation: Berkman Klein Center for Internet & Society, Harvard University, Cambridge, MA, USA Correspondence: baihan.lin@mssm.edu Abstract Depression has no diagnostic blood test. Language models promise tireless, consistent assessment, but can accurate raters disagree about individuals? We pre-registered 880 language-model raters, crossing 11 open models with prompting and scoring choices, and applied them to 189 interviews against the eight-item Patient Health Questionnaire. Model choice explained 30.0% of summed-symptom score variance, stable participant differences 10.5%. Two randomly drawn raters with area under the receiver operating characteristic curve (AUC) ≥ 0.70 disagreed on screening decisions for 40% of participants, on average. Average over-rating governed how many were flagged, yet equal-capacity raters chose differently for about one participant in five. A locked analysis of 86 new interviews reproduced the main pre-registered findings. Exploratory recalibration with 40 labelled participants raised accuracy from about 60% to 75% and halved disagreement, leaving one participant in five decided differently. Calibration repaired much of the rater dependence without securing agreement about individuals. Preprint Introduction A depression rating can shape who is offered further assessment. If the interview stays the same, how much should the decision depend on the software reading it? Psychiatry has confronted a version of this question before. A 1971 study found that American and British psychiatrists disagreed after watching the same videotaped interviews27. Agreement was subsequently found to be poor for most diagnostic categories57. Psychiatry responded with explicit criteria and structured interviews16, 58. Half a century later, large language models (LLMs) promise scalable, inexpensive, repeatable ratings from clinical conversations47, 59, 13, 17. Early evaluations are encouraging: open models achieved clinically acceptable total-score error on structured psychiatric interviews26, reached intraclass correlations near 0.75 with trained raters of psychosis-risk and schizophrenia-spectrum interviews70, 11, and recovered self-reported depression severity from transcripts52, 62. Psychotherapy dialogue also supports alliance inference33, 32 and topical analysis31. But a model becomes a rater only through choices. Someone must choose the model and word the request, perhaps as a questionnaire, in a psychiatrist’s voice or as the patient answering about themselves. They must choose digits or letters and their order, decide whether the model reads the patient’s words or the whole conversation, and turn its answer into a score. These choices can define plausible raters, although evaluations rarely show the alternatives. Outputs shift with prompt formatting53, 44, answer order and labels69, 64, inference settings49, and even supposedly deterministic settings4. Such choices reverse social-science conclusions6, alter depression inference40, change diagnoses when clinical cases are reworded2, and shift severity judgements across languages68. High self-consistency makes reliability easier to establish than validity45, 5, 38. For a clinic, the uncertainty concerns whose assessment changes when one plausible rater replaces another, and how much local labels can repair. Generalizability theory asks whether differences between people survive changes in how they are measured. It separates contributions from the person and the measurement facets12, 9, with dependable scores requiring the person’s contribution to dominate error. Defensible configurations define a multiverse60, 56. Disagreement among similarly accurate configurations is predictive multiplicity42, 7. Even predictors that agree on average can induce different decisions, motivating recalibration for downstream decisions15. Calibration, long a weak point of clinical prediction66, connects measurement to action. Together, these ideas distinguish whether a rater orders people well, whether its threshold flags an appropriate number, and whether another defensible rater selects the same people. We crossed 11 open models, five wordings, two conversation views, two answer-reading rules and four score constructions (eight symptom ratings in three formats, or one overall rating), creating 880 raters for 189 interviews with the eight-item Patient Health Questionnaire (PHQ-8) as criterion28. Analysis code was fixed at pre-registration before comparison with the criterion46, 41. The locked procedure was then applied once to 86 new interviews from a related corpus, with an autonomous interviewer and automatic transcription. The rater outweighed the patient in score variance, and accurate rankings concealed divergent decisions. Offsets governed how many were flagged, but equal-capacity raters still chose different people. Even answer letters mattered. We then examined how far local recalibration could repair these differences: decisions improved, while disagreement about individuals remained. Results The same interview, opposite verdicts One participant reported mild symptoms (PHQ-8 5), below the screening threshold of 10, yet 50% of 880 raters screened them positive (Fig. 1b). No participant received a unanimous verdict, and 91% split raters between 20% and 80% (Fig. 1c). Some dissent becomes likelier as raters accumulate: from 40% of participants with two AUC ≥ 0.70 raters to 94% with ten (Extended Data Fig. 7a). We therefore measured the chance that two distinct, randomly drawn raters disagree: 44% across all raters, 40% among the 540 with AUC ≥ 0.70 (95% participant interval 39–42%, raters fixed), and 12% among the 75 with AUC, sensitivity and specificity all at least 0.70. Even this strict set gave opposite decisions to 53% of participants (Extended Data Fig. 7b). An outcome-independent, clinically defensible core using instrument or psychiatrist wording, digits, expected values and symptom sums (44 raters, AUC 0.61–0.87) disagreed on 31% (Supplementary Information). Nor were only borderline cases affected (Fig. 1d). Participants reporting few or no symptoms (PHQ-8 ≤ 4) were flagged by 35% of AUC ≥ 0.70 raters, which disagreed about them as much as about anyone. Figure 1: The same interview, opposite verdicts. a, The choices that make a rater; each of the 880 combinations was applied to every interview. b, Scores given to one participant by all 880 raters, on the PHQ-8 scale; black line, the participant’s self-reported PHQ-8; dashed line, the screening threshold of 10; orange, scores of 10 or more. c, For every participant, the share of raters screening them positive, against their self-reported PHQ-8 (jittered horizontally); grey band, 20–80% of raters; dashed line, between self-reported totals of 9 and 10. d, The chance that two different raters with AUC ≥ 0.70, drawn at random, give a participant opposite decisions, by self-reported severity; dark grey bars, discovery; white bars, confirmation. Accuracy does not decide who is flagged Many raters met published standards of accuracy. AUC asks how well scores rank participants with a positive screen above those without one, without choosing a threshold. AUC for PHQ-8 ≥ 10 ranged from 0.34 to 0.88 (median 0.75), with 39% below 0.70 (Fig. 2a; Extended Data Fig. 6). Between-rater AUC variance was 10.6 times its sampling variance: rater choice mattered more than participant sampling. Decisions told another story. Raters flagged 0–100% of participants against 30.2% self-reported prevalence; model medians ranged from 14% to 99% (Fig. 2b). The rater’s offset, its mean over-rating of PHQ-8, governed the share flagged along a normal-ogive curve (R2R^2 = 0.91; Fig. 2c). Applied unchanged to the new interviews, that curve explained 0.90 of the variance. The relation is partly built in: offset and threshold crossings summarise the same scores. Nevertheless, location nearly exhausted what mattered at the threshold, and AUC cannot see location. Near the curve’s centre, each point of over-rating flagged 13 additional people per 100. A rater behaves like a bathroom scale with a misplaced zero: it may rank people well, but its offset decides who crosses the line. Qwen2.5 7B and Mistral 7B, in their typical configurations, had almost identical AUCs (0.81, 0.80). Per 100 people like those studied, the first would flag 24 and miss 14 of 30 positive screens; the second would flag 80 and miss 1 (Fig. 2d). AUC ≥ 0.70 raters differing by at most 0.02 in AUC differed by a median 28 percentage points in the share flagged. Across all raters, sensitivity quartiles were 47–100%, specificity 5–84%. Figure 2: Accuracy does not decide who is flagged. a, AUC of each model’s 80 raters: dot, median; thick line, interquartile range; thin line, range. Dotted line, chance; dashed line, 0.70. b, Share of participants that each model’s raters screen positive; dashed line, self-reported prevalence. c, Share screened positive against the rater’s mean over-rating of the PHQ-8, for every rater in discovery (filled) and confirmation (open); line, normal-ogive curve fitted to the discovery raters, with its R2R^2 in discovery and, applied unchanged, in confirmation. d, Decisions for 100 people with the sample’s prevalence by the typical configuration (median AUC) of two models with nearly the same AUC; shown is the pair of models whose typical configurations differ in AUC by at most 0.02 and differ most in the share screened positive. The rater outweighs the patient Each interview was held fixed while the rater changed. Stable participant differences, the person main effect, explained 10.5% of summed-symptom score variance. Model choice explained 30.0%, or 2.8 times as much (Fig. 3a). Model-by-format interaction explained 15.8%: formats shifted models differently. Participant-by-model interaction explained 8.1%: models scored participants differently relative to one another, reflecting rankings or score scales. Crediting every participant-related component to the participant still gave 31%, against 69% for the rater alone. The ordering survived restriction to digits (model-to-person ratio 2.4), one neutral wording, format and view (3.0), and omission of each developer’s models (1.9–6.3). Dependability was 0.11 for one model, wording, format and view. Averaging 11 models, five wordings and three formats raised it to only 0.53, or 0.55 with both views (Fig. 3b; Extended Data Fig. 3). Participants’ mean scores across the 330 expected-value item-sum raters explained 0.53 of PHQ-8 variance (Fig. 3c). Even the average rater captured only part of the questionnaire. Figure 3: The rater outweighs the patient. a, Shares of the variance of the expected item sum in the generalizability analysis of the 330 expected-value item-sum raters (discovery); black, participant main effect; dark grey, interactions involving the participant; light grey, components of the rater alone. b, Dependability of a score averaged over models, for one or all wordings and formats, with both views of the conversation averaged; dashed line, 0.80. c, Participants’ mean score over the same 330 raters against their self-reported PHQ-8; orange, PHQ-8 of 10 or more. Small choices move the threshold Apparent technicalities moved the threshold (Fig. 4). Rating only explicitly reported symptoms, an assessment-policy change from the other four instructions, lowered the median share flagged by 27 percentage points (10 of 11 models; Fig. 4a). Reversing answer letters so A meant ‘nearly every day’ instead of ‘not at all’, with no clinical change, moved it by a median 21 points and up to 85 (Fig. 4b). The interview and the symptom descriptions were unchanged. With digits, symptom sums over-rated relative to direct totals in 8 of 11 models in both sets (Fig. 4c), despite similar discrimination (median AUC 0.78, 0.78). For symptom sums, digits outperformed letters (0.78 versus 0.71, or 0.66 reversed). Larger models ranked better (Spearman correlation 0.82; Fig. 4d), without better calibration or specificity. Their configurations’ decisions converged, but even the largest models varied in share flagged by tens of percentage points (Fig. 4e). Deployment choices also changed decisions. Running Qwen2.5 7B as a 4-bit file with another inference library changed a median 21% of decisions (maximum 35%) under identical configurations: a comparison of deployment stacks, not compression alone. Automatic instead of manual transcripts changed 5% (maximum 37%). Increasing model size with other choices fixed changed 21–25% within Qwen2.5 and 73% from Llama 3.2 3B to Llama 3.1 8B. Figure 4: Small choices move the threshold, and size buys ranking but not calibration or specificity. a–c, For each model, the median share screened positive with the other wordings (filled) and with the instruction to rate unmentioned symptoms as absent (open) (a), and with answer letters A–D meaning ‘not at all’ to ‘nearly every day’ (filled) and the reverse (open) (b); and the median over-rating with digit answers when symptoms are summed (filled) or the total is rated directly (open) (c). d–f, Against model size, one point per model: median AUC (d), interquartile range across a model’s raters of the share screened positive (e), and the median difference between a rater’s correlation with the PHQ-8 and with PTSD severity (f); grey band, differences within ± 0.10. Lines, least-squares fits against log size; filled markers and solid lines, discovery; open markers and dashed lines, confirmation; ρ, Spearman correlations across models. What the raters read At every model size, scores showed little preferential association with PHQ-8 over PTSD severity (Fig. 4f). Only 4% of raters correlated significantly more strongly with PHQ-8 (4.3% by paired bootstrap). The criteria themselves correlated strongly (ρ = 0.84), preventing conclusions about diagnostic separation. Below the screening threshold, higher PTSD symptoms predicted more flags than PHQ-8 alone predicted (partial Spearman 0.36; confirmation 0.23, P = 0.049; Fig. 5a). Talkativeness showed the same association (0.25, 0.26; Fig. 5b). These associations warrant specificity tests without identifying what drives ratings. Interviewer speech alone, containing no participant report, yielded median model AUC 0.59 (Fig. 5c): above chance, as follow-up questions predict10, but below the predicted 0.60. Group conclusions were also fragile. The self-reported female–male difference was small (standardised difference 0.20, P = 0.18), yet 16% of raters reversed its sign and 14% reported a nominally significant difference absent from self-reports (Fig. 5d). Figure 5: What the raters read. a, Among participants below the screening threshold, the share of raters with AUC ≥ 0.70 screening them positive beyond what their PHQ-8 predicts (residual of a linear fit, percentage points), against PTSD symptoms (PCL-C). b, The same for all participants, against the amount of speech (logarithmic scale). Points and lines show residuals and linear fits on the original scales; the statistics in the panel titles are partial Spearman correlations on ranks (discovery; confirmation). Filled markers and solid lines, discovery; open markers and dashed lines, confirmation. c, AUC of each model from the participant’s speech (filled) and from the interviewer’s speech alone (open), median over wordings and formats of the expected-value item sum (discovery). d, Female–male standardised difference estimated by every rater (discovery); black line, the difference in self-reported PHQ-8. Checks that need no labels These raters were consistent, as evaluations commonly report (median Cronbach’s α 0.91). Internal consistency was unrelated to validity across raters (Spearman correlation with AUC 0.02; confirmation −-0.06; Fig. 6a), although informative within models (median correlation 0.51, 0.34). A model can distribute one global impression coherently across symptoms whether that impression is right or wrong. Agreement across rewordings tracked validity both across raters and within models (0.71, 0.67; confirmation 0.67, 0.59; Fig. 6b). Agreement between independent runs sampled at temperature 1 also tracked validity (0.76 across raters). Before confirmation, we specified a label-free rule choosing each model’s configuration with greatest agreement across rewordings. As predicted, it beat the model median in 9 of 11 confirmation models (11 of 11 in discovery). But the rule chose only expected-value configurations, whereas its comparator included cruder most-likely-option scores. Against eligible configurations’ median, it won in 8 of 11 models (mean AUC gain +0.035). Against the digit-answer median it won in 6 (−-0.005), and against a fixed neutral configuration (instrument wording, digits, participant speech, expected values) in 5 (−-0.004; Fig. 6c). Avoiding letters explained most of the gain. Rewording robustness can flag fragile, often invalid configurations, but found none better than a sensible default. Figure 6: Checks that need no labels. a,b, AUC of each item-sum rater against its internal consistency (a) and against its agreement with the same model and format under the other wordings (b); filled, discovery; open, confirmation; Spearman correlations across all raters and medians of the correlations within models, for discovery and then confirmation. c, In the confirmation set, the AUC of each model’s median item-sum configuration, of the configuration chosen without labels by agreement across rewordings, of a fixed neutral configuration (the questionnaire’s wording, digit answers, the participant’s speech, expected value) and of its best configuration; the rule was specified before the confirmation analysis. How many versus who Disagreement can concern how many people to flag or which people. An identity separates the minimum disagreement forced by unequal positive rates from the remainder: total disagreement equals the absolute rate difference plus twice the smaller one-sided disagreement. At the screening threshold of 10, unequal rates dominated. Among AUC ≥ 0.70 raters, 40% disagreement comprised 38 percentage points forced by rates and 2.3 beyond them (confirmation 40% total, 2.1 excess; Fig. 7a). All raters, the defensible core and one fixed configuration per model showed the same pattern. Decisions were nearly nested: lenient raters flagged almost everyone strict raters flagged, then added others. Nested decisions need not imply shared severity rankings. To separate who is selected from how many, we gave each rater the same number of places to fill. Selecting equal numbers, matching self-reported prevalence, expected-value raters with AUC ≥ 0.70 still chose differently for 18% of participants (95% interval 15–21%), replacing 30% of each other’s selections (confirmation 20%, 30%; Fig. 7b). Disagreement was 21% within the defensible core (21% confirmation) and 25% with one fixed configuration per model (22%), without accuracy-based selection. Selecting the top tenth reduced disagreement to 9% because fewer people were selected, while still replacing 46% of selections. Raters disagreed with the criterion on 24%, more than with one another, indicating shared departures from PHQ-8. The two forms of disagreement also affected different people. At the threshold of 10, participants reporting few or no symptoms contributed almost half (48%). At equal capacity, their exchange probability was only 9%, compared with 27% at PHQ-8 10–14 and 25% at 15 or above (Supplementary Information). Changing one choice isolates its consequences (Fig. 7c). Changing models altered a median 35% of decisions at 10 and 25% at equal capacity. Reversing letters altered 20% and 21%: clinically meaningless relabelling changed both how many and who. The psychiatrist persona changed least among wordings (7%, 8%). Full dialogue rather than participant speech changed 17% of equal-capacity selections. The unmentioned-symptom rule and direct-total scoring chiefly moved thresholds: 21% versus 14%, and 24% versus 15%, respectively. The dominance of offsets suggested a practical repair: use local labels to reset the thresholds. This repaired much of the disagreement. Across 200 random splits, 40 labelled participants admitted raters with AUC ≥ 0.70 (mean 538 of 880) and recalibrated them by five rules: subtracting mean over-rating, linear regression, logistic regression, the attainable threshold closest to labelled prevalence, or the highest threshold retaining at least 80% of labelled positives. Evaluation used the remaining participants, whose labels informed no step. Prevalence matching raised accuracy from 61% to 75% (confirmation 59% to 75%) and halved disagreement, from 40% to 20% (40% to 21%; Fig. 7d). Every rule improved accuracy. Lower disagreement sometimes meant more missed cases: linear recalibration gave 16% disagreement and median sensitivity 42%. Retaining 80% of labelled positives gave held-out sensitivity 78% and disagreement 27%. Each rule sets a cut-off on the rater’s score. Increasing fits preserve rankings, as did all but 14 of 107,540 linear fits. Residual disagreement after prevalence matching remained near one in five, resembling equal-capacity disagreement with all labels available. It persisted with 20 or 80 labels (22%, 20%), fixed configurations (24%) and whole-cohort admission (19%; Extended Data Fig. 8). Prevalence matching brought the share flagged within a median 7 percentage points of self-reported prevalence (confirmation 8), with a central 95% split-to-split range of 3–19 points. Score ties sometimes prevented matching labelled prevalence within 5 points (achieved in 88% of cases). For prevalence estimation, the 40 labels alone gave intervals nearly as narrow as prediction-powered estimates using ratings3 (25 versus 27 percentage points), with similar coverage (Extended Data Fig. 4c). Calibration did not always travel. We transferred discovery calibrations unchanged for configurations with discovery AUC ≥ 0.70, from manual transcripts and operator-controlled interviews to automatic transcripts and autonomous interviews. Subtracting discovery offsets over-corrected: median share flagged fell from 51% to 23%, against 34% prevalence. Prevalence-matched thresholds transferred reasonably (37% flagged, median sensitivity 72%). The best discovery configuration (Qwen2.5 14B), with its discovery threshold, flagged 43%, with 79% accuracy and 83% sensitivity. In this single related transfer, transportability depended on the calibration and required checking. Could disagreement guide deferral? Pooling and abstention have been proposed for medical predictive multiplicity39, but selective prediction can fail particular groups29. We compared rules on the same held-out participants at matched coverage (Extended Data Fig. 9). Majority accuracy was 78%, versus 75% for a single recalibrated rater (confirmation 80%, 75%). Requiring 80% consensus retained 71% of participants, with majority accuracy 86% (89%). Single raters achieved 83% on those participants (84%), or 82% (82%) when deferring their own closest calls at equal coverage. Consensus selection outperformed a rater’s own margin, but most apparent gain came from excluding difficult participants, disproportionately positive screens: 24 of 45 positives remained, versus 82 of 104 negatives. Majority sensitivity changed from 58% overall to 56% among retained participants (confirmation 65% to 70%). These sensitivities describe different groups. Deferred participants are neither detected nor missed until reviewed. Without costs for review and missed cases, these comparisons do not define a triage policy. Figure 7: How many versus who. a, The chance that two different raters disagree about a participant at the threshold of 10, split by the identity P(A≠B)=|pA−pB|+2minP(A=1,B=0),P(A=0,B=1)P(A≠ B)=|p_A-p_B|+2 \P(A=1,B=0),P(A=0,B=1)\ into the part forced by their different positive rates (solid) and the rest, from flagging different people (hatched), for all raters, raters with AUC ≥ 0.70 and one fixed configuration per model; dark grey, discovery; white, confirmation. b, Participants decided differently when every rater selects the same share of participants, its highest-scoring ones, by two expected-value raters with AUC ≥ 0.70 or two fixed configurations (one per model); diamonds, participants decided differently by a rater and the self-reported criterion when the rater selects as many as screen positive. Filled markers and solid lines, discovery; open markers and dashed lines, confirmation. c, Participants decided differently by configurations that differ in one choice only (discovery; changes of wording are relative to the questionnaire’s own wording): dot, median over pairs of configurations; bar, interquartile range; filled, at the threshold of 10; open, selecting as many as screen positive (expected-value scores only); deployment-stack and transcript comparisons are for decisions only. d, For raters admitted by their AUC on 40 labelled participants and recalibrated with the same 40 labels, the chance that two raters disagree against the share of correct decisions, both on the held-out participants (means over 200 splits); filled, discovery; open, confirmation. Exploratory analyses. The regularities held in new interviews The locked procedure was applied once to 86 E-DAIC interviews (sessions 600–718) from the same laboratory, with an autonomous interviewer and automatic transcription (440 participant-speech raters). AUC ranged from 0.34 to 0.91 (median 0.78), and share flagged from 0% to 100%, against 33.7% prevalence (Extended Data Fig. 5). AUC ≥ 0.70 raters disagreed on 40%. Participant differences explained 13.0% of variance, model choice 26.2%. Of 6 evaluable pre-registered predictions, 5 were supported. As in discovery, 40-label calibration did not halve prevalence spread. Discovery supported 5 of 7 predictions, also falling short on interviewer speech. All 9 of 9 discovery regularities met criteria specified before confirmation (Extended Data Table 1). Discussion A language model asked to rate depression does not report a property of the patient alone. In these interviews, model choice explained about three times more summed-symptom score variation than stable differences between participants. Raters that ranked participants well still disagreed about many of them, and how many people were flagged depended on the offset rather than the accuracy. Psychiatry spent decades responding to unreliable diagnosis with explicit criteria and structured interviews27, 57, 16, 58. Automation promises relief from variation between human raters. Here, the rater returns as a configuration: a model, an instruction and a choice of letters, often undocumented. Choices made when building an instrument can become differences in how it judges a person. Earlier findings of accurate, consistent LLM ratings26, 70, 11 remain compatible with ours: they usually examined few configurations, and our best were accurate too. Ranking versus calibration66, predictive multiplicity42, 7 and downstream decision disagreement15 are established distinctions. The multiverse makes these distinctions tangible for one clinical decision. It measures how much a decision about a person depends on choices that an evaluation usually leaves in the background, and how much of that dependence a simple local repair removes. Clinics act at thresholds, where AUC is blind to offsets. Internal consistency did not rank configurations across models, and rewording robustness identified fragility without improving on a neutral default. For prevalence estimation, ratings added little precision beyond the 40 questionnaire labels alone in this cohort. The value of automation therefore needs to be established against what the same local measurement effort can achieve without it. Alliance estimates also inform classification36 and conversational recommendations35, 34, 22. Calibration exposed the distinction between agreement about populations and agreement about people. A few dozen local labels repaired much of the disagreement: decisions became more accurate and more alike, and prevalence estimates became better calibrated. But agreement about how many is not agreement about who. An increasing recalibration leaves a rater’s ranking untouched, and different configurations ranked participants differently enough to make opposite selection decisions for about one participant in five even when selecting equal numbers. Resetting a scale’s zero cannot reconcile two scales that disagree about who is heavier. In a service using these ratings to allocate a fixed number of assessment appointments, the implication would be different names on a referral list of the same length. Pooling improved accuracy; consensus deferral improved it further at matched coverage, while disproportionately excluding positive screens. Decision-focused reconciliation15 may narrow the residual. Until demonstrated, individual agreement must be measured and calibration checked locally. Agreement itself is also insufficient: raters shared departures from PHQ-8, and choosing a threshold trades missed positive screens against unnecessary flags. The clinical objective must therefore be specified alongside the calibration rule. The ratings also carried information beyond depression, raising the question of what an apparently successful score measures. They tracked PTSD as closely as PHQ-8. A preprint on these corpora found symptom geometry closer to questionnaire wording than self-report37. Below the screening threshold, participants with more trauma symptoms were flagged more often than their PHQ-8 predicted. Greater amounts of speech also predicted additional flags beyond PHQ-8. Specificity remains unresolved where depression and PTSD co-occur, including trauma-exposed and veteran populations. Ambient scribes now record entire clinical visits63, making the clinician’s contribution to model inputs a practical concern beyond research interviews. If the configuration is the instrument, changing a component changes the instrument. Changes in model precision and inference software, model size, instructions and transcription each altered substantial numbers of decisions. Five practices follow (Box 1). Report decision ranges across defensible configurations, extending multiverse and many-analyst approaches60, 55, 8 to clinical LLM reporting18. Fix a neutral configuration and test rewording robustness. Calibrate against a validated criterion in a few dozen local patients, check transfer and report residual individual disagreement. Test specificity against neighbouring conditions. Freeze the complete stack, especially when following a patient over time. Otherwise, a future change in a patient’s score could reflect a changed instrument as well as a changed condition. Longitudinal validation would need to distinguish these possibilities. Box 1 | Five checks before a language-model rating informs care Report the range, not one number. Rate with several defensible configurations and report how often their decisions disagree, including among configurations that pass checks of accuracy, sensitivity and specificity, and whether they disagree about how many people to flag or about which ones (Figs 1, 2 and 7). Fix a neutral configuration, and treat fragility as an alarm. Choose the wording, answer format and scoring in advance, for example the questionnaire’s own wording with digit answers. Reword the request: a rating that changes with the wording is fragile. Internal consistency did not rank configurations across models (Fig. 6). Calibrate locally, check transfer, and measure what remains. A few dozen local patients who completed a validated questionnaire can set the threshold and give honest estimates of prevalence. Check whether a calibration carries over to a new setting rather than assuming it. A recalibration that preserves a configuration’s ranking changes how many people are flagged, not whom it ranks highest, so measure the remaining disagreement about individuals (Figs 2c and 7). Ask what else it reads. Test specificity against neighbouring conditions such as PTSD with criteria that can separate them, and whether the clinician’s words drive the rating (Fig. 5). Freeze the whole stack. Treat any change of model, precision, inference software, transcription or wording as a new instrument, and revalidate it (Fig. 4). Several limits define the scope. Corpus licences required local processing, restricting us to open models up to 21 billion parameters. We scored normalised answer probabilities after a fixed cue, without generated reasoning. Frontier models used as intended may differ. Size improved ranking and narrowed threshold variation without eliminating it; larger models may reduce these effects, but disappearance cannot be assumed. Our finite, deliberately broad grid makes inclusion consequential, motivating stricter sets, fixed configurations and single-choice comparisons. PHQ-8 was a pre-interview self-report screening criterion, not a diagnosis, although rater disagreement requires no criterion. No clinicians rated these transcripts, precluding direct comparison with human disagreement. Both English-language sets followed one laboratory’s protocol. Confirmation was small, with few women, and changed participants, interviewer and transcription, not clinical setting. Calibration transfer was therefore one related comparison. Analyses of how many versus who, recalibration, transfer, deferral and fairer label-free comparisons were added after confirmation and remain exploratory. Repeated splits of small cohorts are not independent samples. Finally, frozen code corrected termination for only some direct totals, and batching affected inference reproducibility. Both changed scores by fractions of a point in the re-scored sample (Methods). Open questions extend beyond depression. Do frontier models escape the rater effect? Does disagreement persist across clinically equivalent paraphrases, and how does it compare with clinicians reading identical transcripts? How large are configuration effects relative to treatment-related change? Can raters be reconciled, or designed, to agree about individuals while remaining accurate against an independent clinical assessment? The same decomposition could inform assessments of essays, job applicants and online speech wherever a model’s judgement becomes a measurement of a person. A questionnaire makes its eight questions and cut-off visible, so readers can inspect and contest the terms on which it judges a person. A language-model rater can conceal equally consequential choices in software that appears merely to read the interview. Here, those choices shaped how many people were flagged and which people they were. Calibration made the raters more nearly right on average, yet left them divided about individuals. Clinical care meets people one at a time. When changing how we ask changes whom we identify, the instrument becomes part of the explanation. Responsible measurement makes that contribution visible, so that a judgement about a person can be understood and questioned by the person who must live with it. Online Methods Data. The discovery set was the Distress Analysis Interview Corpus – Wizard of Oz (DAIC-WOZ)20, 14: 189 semi-structured interviews of adults, conducted by an animated virtual interviewer that a human operator controlled from another room, with manual transcripts of the interviewer’s and the participant’s turns. Before the interview, participants completed the PHQ-8, which scores eight symptoms over the preceding two weeks from 0 (not at all) to 3 (nearly every day), for a total of 0–24; a total of 10 or more is the standard screening threshold28. A participant-level meta-analysis also supports this threshold for the related PHQ-930. A participant screened positive if their PHQ-8 total was 10 or more; this registered definition differs from the binary flag distributed with the corpora for one discovery and five confirmation participants (Supplementary Information). Totals were available for all 189 participants and item scores for 142 (complete for 141; the challenge training and development partitions65). The same participants’ PTSD Checklist – Civilian Version (PCL-C) severity was available from the extended corpus (E-DAIC)51. The confirmation set comprised the 86 E-DAIC interviews (sessions 600–718) conducted by a fully autonomous version of the virtual interviewer, with automatic speech recognition (ASR) transcripts of the participant’s speech and PHQ-8 and PCL-C scores (PCL-C missing for two). In the discovery set, 30.2% of participants screened positive (57 of 189; 87 women and 102 men); in the confirmation set, 33.7% (29 of 86; 18 women and 68 men). The corpora were used under their end-user licence agreements; all processing ran on a local workstation, and no transcript text was sent to any external service. Models. We used 11 open-weight instruction-tuned models from six developers, chosen because their weights were already on the workstation: Qwen2.5-Instruct with 1.5, 3, 7 and 14 billion parameters50; Llama-3.2-Instruct 1B and 3B and Llama-3.1-8B-Instruct21; Mistral-7B-Instruct-v0.325; Phi-3.5-mini-instruct1; DeepSeek-R1-Distill-Qwen-7B23; and gpt-oss-20b48. Qwen2.5 1.5B, 3B and 7B ran in bfloat16 with the Transformers library67 on the Apple GPU; the others ran as 8-bit (Llama-3.2-1B) or 4-bit quantized GGUF files with llama.cpp. For the reasoning model the reasoning block was left empty, and gpt-oss-20b answered in its final channel at low reasoning effort. As a check on the deployment stack, Qwen2.5-7B was also run as a 4-bit GGUF file with llama.cpp. Prompts. Each prompt, sent in the model’s own chat format, began with a one-sentence description of the transcript and the transcript itself, followed by a task instruction (wording), a description of the response options (format) and one PHQ-8 item. The five wordings represented distinct instructions: instrument, complete the PHQ-8 on the participant’s behalf using the interview as the only source; clinician, an experienced psychiatrist rating the participant; evidence, rate only symptoms the participant explicitly reports and rate undiscussed symptoms as absent; role-play, answer as the participant would about themselves; and inferential, use every cue, including tone, word choice and avoidance. The evidence, role-play and inferential wordings change the assessment policy as well as the words; the clinician wording changes only who is said to rate. The three formats were digits 0–3, letters A–D with A meaning ‘not at all’, and letters A–D with A meaning ‘nearly every day’, each with the PHQ-8 anchor labels. Items used the PHQ-8 wording with the standard two-week stem. For the direct total, the wording was followed by a request for the participant’s total PHQ-8 score from 0 to 24, answered with digits. Exact texts are given in Supplementary Information and the code. Reading the answer. We did not sample. The assistant’s reply was prefilled with an answer cue (“Answer:” for items, “Score:” for totals), and each answer option was scored by its probability as the next tokens, summed over tokenisation variants (with and without a leading space) and renormalised over the options. The unnormalised total was retained as a measure of format compliance. This is normalised candidate-label likelihood scoring. The expected-value rule uses the whole distribution, and the most-likely-option rule takes the option with the highest normalised probability, which need not coincide with the answer that free or token-by-token greedy generation would produce. For totals, we scored the 25 numbers 0–24 as token sequences. The intended rule multiplies each number’s probability by the probability that it ends where it does (the next token is not a digit); the frozen code applied this factor only to numbers whose tokens begin a longer candidate. Re-scoring the direct totals of 25 participants spread over the PHQ-8 range with both rules from the same forward pass, for four models and the participant view, changed expected totals by at most 0.008 points in that sample and no screening decision (Supplementary Information); this measures the defect in a sample rather than repairing it, and item sums are unaffected. All prompts for one transcript share the transcript as a prefix; we encoded it once and evaluated all continuations in a few forward passes with tree-structured attention, which reproduced one-prompt-at-a-time evaluation to 10−410^-4 in log-probability in 32-bit arithmetic for the prompts tested. Inference was not bit-for-bit reproducible across batch compositions: re-running the same direct-total prompts reproduced the saved expected totals to a median of 0.003 points (maximum 0.68), changing 1 of 500 decisions. Raters and scores. A rater (configuration) is a combination of model (11), transcript view (participant speech only; full dialogue), and either a wording, a format and a scoring rule applied to the eight items, summed (5 × 3 × 2), or a wording and a scoring rule applied to the direct total (5 × 2): 11×2×40=88011× 2× 40=880 raters. The expected-value rule scores an item as ∑kpk _kk\,p_k and the mode rule as argmaxkpk _kp_k. A rater screens a participant positive if its score is 10 or more. Two further views served as checks: the interviewer’s speech only, which contains no report from the participant but is known to leak depression status through the interviewer’s follow-up questions10, and ASR transcripts of the same participants’ speech. Quality checks. Two secondary outputs of the first run, for Qwen2.5 1.5B (the interviewer’s speech for one participant and the automatic transcript for another), had no finite probabilities, and the frozen analysis’s median over models silently omitted that model. We regenerated both with the frozen code, which gave finite outputs with nearly all probability on the answer options, kept the original files, and re-ran the frozen analysis: primary results were unchanged, and the prediction about the interviewer’s speech, previously met, was no longer supported. A check of every raw output against an explicit manifest of the expected models, conditions and participants now runs before the frozen analysis. The only permitted exclusions are three participants whose interviewer transcripts were empty. The check also verifies the registered dimensions and that the stored option probabilities sum to one within their rounding. It rejects empty, truncated or incomplete files and flags the two original outputs. In six discovery transcripts, empty turns had been converted to the literal token ‘nan’ (1–12 tokens each); excluding these participants changed the median AUC from 0.753 to 0.757. Estimands. For each rater we computed the area under the ROC curve (AUC) for PHQ-8 ≥10≥ 1024, Spearman’s correlation with the PHQ-8 total, the mean absolute error and bias (mean over-rating), and the share screened positive. For each participant, πi _i is the share of raters that screen them positive; a participant is discordant if 0<πi<10< _i<1 and contested if 0.2≤πi≤0.80.2≤ _i≤ 0.8. The chance that two different raters drawn at random from K disagree about a participant is 2πi(1−πi)K/(K−1)2 _i(1- _i)K/(K-1); we report its mean over participants (the frozen regularities used 2πi(1−πi)2 _i(1- _i), which differs negligibly). The expected share of participants receiving both decisions from k raters drawn at random without replacement is computed exactly, 1−[(mik)+(K−mik)]/(Kk)1-[ m_ik+ K-m_ik]/ Kk averaged over participants, with mim_i the number of raters screening participant i positive. For two raters with positive rates pAp_A and pBp_B, P(A≠B)=|pA−pB|+2minP(A=1,B=0),P(A=0,B=1)P(A≠ B)=|p_A-p_B|+2 \P(A=1,B=0),P(A=0,B=1)\, an algebraic identity that separates the least disagreement forced by unequal rates from the rest, without assigning causes. At equal capacity, each rater selects its q highest-scoring participants, with q the number of participants screening positive by self-report unless stated; expected-value scores had no ties at the boundaries used. We compared the between-rater variance of the AUC with its participant-bootstrap sampling variance. The generalizability study treated person, model, wording, format and view as crossed random facets of the expected item sum (330 raters in discovery), with variance components estimated from expected mean squares12, 9. We report the person main effect and, separately, all components involving the person. Decision studies give the generalizability (Eρ2Eρ^2) and dependability (Φ ) coefficients when scores are averaged over several models, wordings and formats; in discovery they also average both views unless stated. For each item-sum rater we computed Cronbach’s α across the eight items; the correlation between two independent runs sampled at temperature 1, which follows analytically from the answer distributions; and the mean correlation with the same rater under the other wordings. We related each index to the AUC across raters and within models. For group inference, we computed each rater’s standardised female–male difference (the difference in means divided by the square root of the mean of the two group variances) and compared its sign and size with the difference in the criterion19. Convergent and discriminant validity compared each rater’s correlation with the PHQ-8 and with PTSD severity by the Meng–Rosenthal–Rubin test for correlated correlations43; Steiger’s pooled form61 and a paired participant bootstrap of the difference gave the same conclusions. Agreement between transcript sources and between deployment stacks used decision flips and ICC(2,1)54. Pre-registered remedies were averaging across raters or models; isotonic calibration of each rater’s total score to the PHQ-8 total with 20 or 40 labelled participants (200 random draws, evaluated on the rest), which calibrates scores rather than probabilities; and prediction-powered estimation of prevalence with the same labels3. Further analyses. Sensitivity, specificity and the share screened positive follow from each rater’s decisions. The share screened positive was related to the rater’s bias by a normal-ogive curve, Φ((b−c)/s) ((b-c)/s), fitted by least squares; the number of additional people flagged per point of bias at the centre is 100ϕ(0)/s100\,φ(0)/s. A model’s typical configuration is the one with its median AUC. Effects of wording and letter order compare model medians across the corresponding configurations; effects of aggregation compare item sums and direct totals with digit answers only, because direct totals were asked only with digits (the frozen regularity, Extended Data Table 1, counted item sums in all formats and found 7 of 11 models); and effects of size use the log parameter count. Partial Spearman correlations remove a linear trend in the ranks of the PHQ-8 from both variables and are tested with n−3n-3 degrees of freedom. Decision changes between neighbouring model sizes compare identical configurations. The label-free rule chose, for each model, the expected-value item-sum rater with the highest agreement across wordings (ties broken by sampling reliability), and was compared, as specified, with the median AUC of the model’s item-sum raters. These analyses were exploratory in the discovery set. The label-free rule, with the prediction that it would beat the model median in at least 9 of 11 models, and nine regularities, each with a pass criterion, were specified in time-stamped addenda after the discovery analysis and before any confirmation output was compared with the criterion, while confirmation outputs were still being generated; all are reported. The criteria are descriptive, and because the regularities share participants and raters they are not independent replications. Analyses added after the confirmation analysis. The following analyses were not pre-specified and are exploratory in both sets. Pairwise disagreement in stricter sets of raters (AUC ≥ 0.70 and 5–95% screened positive; AUC, sensitivity and specificity all ≥ 0.70), and the exact expected discordance among k raters. The identity for the original decisions, and equal-capacity selection at 10–50% of participants, for expected-value raters with AUC ≥ 0.70 (a selection that uses the whole cohort’s labels), all expected-value raters and one fixed configuration per model (instrument wording, digits, participant’s speech, expected value), with each rater’s disagreement with the criterion at the self-reported prevalence. Single-choice comparisons over all pairs of configurations identical except in one choice. The model-to-person variance ratio with digit answers only, for the instrument wording with digits and the participant’s speech, and with each developer’s models left out. The label-free rule against the median of expected-value item sums, the median of those with digit answers, and the fixed neutral configuration. Recalibration on held-out participants: in each of 200 random splits, 20, 40 or 80 participants (20 or 40 in confirmation) formed a labelled sample and the rest were held out; three rater sets were recalibrated, namely raters with AUC ≥ 0.70 in the labelled sample, one fixed configuration per model, and, for comparison, raters with AUC ≥ 0.70 in the whole cohort, whose admission uses the held-out labels; the rules were subtracting the mean over-rating, linear regression of the PHQ-8 total on the score, logistic regression for PHQ-8 ≥ 10 with a weak ridge penalty (deciding at a probability of 0.5), the attainable threshold whose labelled positive count is closest to the number of labelled positives, compared exactly as integers with ties going to the higher threshold (ties among scores can make the target unattainable; achieved counts are reported, a miss of 5 percentage points counting as within 5), and the highest threshold that keeps at least 80% of the labelled positives (the exact order statistic). We report held-out accuracy, sensitivity, specificity, pairwise disagreement, the median absolute error of the share flagged with its 2.5th and 97.5th percentiles across splits, and achieved training targets. A defensible core (instrument or clinician wording, digits, expected values, symptom sums; every view) was analysed alongside these sets without any selection by outcome. Deferral was compared at matched coverage on the same held-out participants: a single recalibrated rater, the majority of raters, the majority after setting aside participants on whom fewer than a given share of raters agreed, single raters on the same retained participants, and each single rater setting aside its own closest calls, those nearest its cut-off, to reach the same coverage; retained participants with and without a positive screen are counted. Where disagreement falls was summarised by self-reported PHQ-8 band, at the threshold and at equal capacity. Three headline estimands were bootstrapped over participants with the rater sets held fixed. Transfer carried each configuration’s discovery offset or discovery thresholds (matching the discovery prevalence, or keeping 80% of discovery positives), for configurations with AUC ≥ 0.70 in discovery and for the single most accurate discovery configuration, unchanged to confirmation. Prevalence intervals from 40 labels (2,000 draws): prediction-powered with the registered formula, prediction-powered for the finite cohort, and from the labels alone. The discovery normal-ogive curve applied unchanged to confirmation. The decision rules and their tie handling are implemented and tested in the analysis code. Pre-registration and confirmation. The study was pre-registered on OSF Registries (https://osf.io/8qg2m/) before any model output was compared with the criterion, and the scoring and analysis code were fixed at the same time. The pre-registered hypotheses were that the validity, individual screening decisions and implied prevalence of the ratings depend substantially on configuration choices, and that indices of rating consistency do not indicate which configurations are valid. Model outputs for the confirmation set were generated after the discovery analysis was complete, with the locked code, and analysed once. Deviations from the plan, including the regenerated outputs and the scoring defect, are listed in Supplementary Information. Statistics. Intervals for AUCs and for the gain of the label-free rule are percentile 95% intervals from 2,000 bootstrap resamples of participants. The pre-registered calibration and prediction-powered analyses used 200 random draws of labelled participants (50 for the display in Extended Data Fig. 4b); the exploratory analyses used 200 splits for recalibration and deferral, 2,000 draws for prevalence intervals, 1,000 paired bootstrap resamples for the specificity check and 1,000 participant resamples, with the rater sets held fixed, for the headline intervals, which are therefore conditional on the raters. Split-to-split ranges describe variation over repeated splits of one cohort and are not confidence intervals. All tests are two-sided unless stated. Data availability DAIC-WOZ and E-DAIC are available from the University of Southern California Institute for Creative Technologies under a data use agreement. Only aggregate results are distributed with the code. Code availability The scoring and analysis code will be made available on publication at a public repository. Acknowledgements This work was supported in part by a 2025 NARSAD Young Investigator Grant from the Brain & Behavior Research Foundation, funded by The Sidney R. Baer, Jr. Foundation (Grant ID 33941). B.L. is also supported by the U.S. National Institutes of Health (NIH), the U.S. Department of Veterans Affairs (VA), the Hasso Plattner Foundation, and the Windreich Family Foundation. We thank the University of Southern California Institute for Creative Technologies for making the interview corpora available. Author contributions B.L. conceived the study, performed the analyses, and wrote the manuscript. Competing interests The author declares no competing interests. Extended Data Extended Data Table 1: Regularities found in the discovery set and checked in new interviews. Discovery values are exploratory; the criteria were specified before the confirmation analysis. π, a participant’s share of raters with AUC ≥ 0.70 screening them positive. For the first regularity, confirmation values refit the curve; applied unchanged, the discovery curve gives R2R^2 = 0.90. P values for partial correlations use n−3n-3 degrees of freedom (P = 0.049 for the PTSD regularity in confirmation). With digit answers only, item sums over-rate in 8 of 11 models in both sets. The regularities share participants and raters, so they are not independent tests. Regularity Criterion Discovery Confirmation Held The share screened positive follows a configuration’s mean over-rating (probit R2R^2; referrals per 100 per point) R2≥R^2≥ 0.75 0.91; 13 0.91; 14 yes Configurations with the same AUC (± 0.02) differ in share screened positive (median, points) ≥ 15 28 28 yes Participants reporting few symptoms (PHQ-8 ≤ 4) are screened positive (mean share of configurations with AUC ≥ 0.70) ≥ 20% 35% 44% yes Larger models discriminate better (Spearman, size vs median AUC), yet no model’s ratings correlate with the PHQ-8 more than 0.10 above their correlation with PTSD severity (median difference; not a test of diagnostic separation) >> 0.5; none >> 0.10 0.82; max 0.04 0.77; max 0.01 yes Below the threshold, more PTSD symptoms go with a larger share screened positive at equal PHQ-8 (partial Spearman) >> 0, P<P< 0.05 0.36 0.23 yes Rating only explicitly reported symptoms lowers referrals; reversing the answer letters moves them (median points) 8 and 6 of 11 models 27; 21 31; 24 yes Summing item ratings over-rates relative to rating the total directly (models; as frozen, item sums in all formats) ≥ 6 of 11 7 of 11 7 of 11 yes Talkative participants are screened positive more often at equal PHQ-8 (partial Spearman) >> 0, P<P< 0.05 0.25 0.26 yes The most accurate model’s own configurations disagree about participants (mean 2π(1−π)2π(1-π), as frozen) ≥ 0.15 0.24 0.26 yes Extended Data Fig. 1: Format compliance and rated severity. a, Median probability that each model placed on the answer options, by wording (participant view). b, Mean expected item sum by model and wording; the self-reported mean PHQ-8 is given in the panel title. c, Mean expected item sum by model and answer format. Extended Data Fig. 2: Agreement with individual symptoms. Spearman correlation between each model’s expected item rating and the participant’s self-reported score on the same item, for the 141 participants with complete item scores (participant view), per symptom: dot, median model; thick line, interquartile range over models; thin line, range over models; each model’s value is its median over wordings and formats. Exploratory analysis. Extended Data Fig. 3: Decision studies. Generalizability (Eρ2Eρ^2; a) and dependability (Φ ; b) of the expected item sum (discovery), when scores are averaged over the given number of models and over one or all wordings and formats, with both views of the conversation averaged. Dashed line, 0.80. Extended Data Fig. 4: What averaging and labels do (discovery). a, AUC of single raters, of scores averaged over models for each configuration of the other choices, and of the average over all raters; lines, medians; dashed line, 0.70. b, Share screened positive by each rater before and after isotonic calibration of its total score against 20 or 40 labelled participants (mean over 50 draws); dashed line, self-reported prevalence. c, Mean width of 95% intervals for the prevalence from 40 labelled participants (2,000 draws): prediction-powered intervals with the registered formula and for the finite cohort, and intervals from the labels alone; observed coverage above the bars. Extended Data Fig. 5: Discovery and confirmation. Discovery (DAIC-WOZ, dark grey) and confirmation (E-DAIC sessions 600–718, automatic transcripts, white). a, AUC of every rater; lines, medians. b, Share screened positive; dashed lines, self-reported prevalence. c, Participants receiving both decisions, contested decisions (20–80% of raters), and both decisions among raters with AUC ≥ 0.70. d, Shares of the variance of the item-sum score: participant, model, wording, participant × model (P × M) and participant × wording (P × W). Extended Data Fig. 6: All raters, ordered by accuracy (discovery). a, AUC of every rater, ordered, with 95% bootstrap intervals; dotted line, chance; dashed line, 0.70. b, Share screened positive by the same raters; dashed line, self-reported prevalence. c, The choices that make up each rater. Extended Data Fig. 7: How disagreement depends on which raters are counted. a, Expected share of participants who receive both decisions from k raters with AUC ≥ 0.70 drawn at random without replacement (exact), against k; filled markers and solid line, discovery; open markers and dashed line, confirmation. b, Chance that two different raters disagree about a participant, among all raters, raters with AUC ≥ 0.70, those that also screen 5–95% of participants positive, and those with AUC, sensitivity and specificity all ≥ 0.70; dark grey, discovery; white, confirmation. Exploratory analyses. Extended Data Fig. 8: Recalibration in detail. a, The chance that two recalibrated raters disagree about a held-out participant against the number of labelled participants, for raters admitted by their AUC in the labelled sample and recalibrated to match the labelled prevalence, to keep at least 80% of labelled positives, or by logistic regression; solid and filled, discovery; dashed and open, confirmation. b, Median absolute error of the share flagged with the prevalence-matched threshold; bars, 2.5th to 97.5th percentiles across splits (not confidence intervals). c, Disagreement after matching the prevalence with 40 labels for three rater sets: one fixed configuration per model, raters admitted by their AUC in the labelled sample, and raters admitted by their AUC in the whole cohort, a selection that uses the held-out labels. Means over 200 splits; exploratory analyses. Extended Data Fig. 9: Deferral at matched coverage. For raters admitted by their AUC on 40 labelled participants and recalibrated to the labelled prevalence, accuracy (a, discovery; b, confirmation) and sensitivity (c, discovery; d, confirmation) on the held-out participants who are decided rather than deferred, against the share decided: the majority of raters for all participants (50% consensus) and after setting aside participants on whom fewer than 60, 70, 80 or 90% of raters agree (black; unanimous consensus is not shown because it left fewer than 3% of participants); single raters on the same retained participants (grey, dashed); and single raters setting aside their own closest calls to reach the same coverage (grey, solid). Large circle, the 80% consensus rule. Means over 200 splits; exploratory analyses. Supplementary Information 1 Supplementary Methods 1.1 Prompts Each prompt was a single user turn in the model’s own chat format. Its components appeared in a fixed order: a one-sentence description of the transcript view; the transcript, enclosed in <transcript> tags, with turns labelled “Interviewer:” and “Participant:” in the full-dialogue view, “Interviewer:” in the interviewer view and unlabelled in the participant view; the template’s task instruction; the response format; and one PHQ-8 item preceded by the stem “Over the last 2 weeks, how often has the participant been bothered by the following problem?”. The assistant’s reply began with the answer cue specified by that format. For the direct total, the format and item were replaced by “Considering all eight PHQ-8 problems together, what is the participant’s total PHQ-8 score (from 0 to 24)? Reply in the form ‘Score: N’.”, and the reply was prefilled with “Score:”. Supplementary Table 1 gives every text. Supplementary Table 1: Prompt texts. The transcript follows the first line; the template, the format and the item follow the transcript. Transcript view (first line of every prompt) participant Below is a transcript of a clinical research interview. Only the participant’s speech is shown. full Below is a transcript of a clinical research interview between an interviewer and a participant. interviewer Below is a transcript of a clinical research interview. Only the interviewer’s speech is shown. Template instrument Task: complete the PHQ-8 depression questionnaire on behalf of the participant, using the interview as your only source of information. clinician You are an experienced psychiatrist. Using your clinical judgement about this interview, rate the participant on the PHQ-8 depression scale. evidence Rate each PHQ-8 symptom only from what the participant explicitly reports in the interview. If a symptom was not discussed, rate it as ’Not at all’. Do not speculate. role-play Imagine that you are the participant in this interview. Answer the PHQ-8 questionnaire the way they would answer it about themselves, given everything they said. inferential Use every available cue, including indirect ones such as tone, word choice and what the participant avoids talking about, to infer how the participant would score on the PHQ-8. Response format (answer cue prefilled in the reply) digits Answer each question with a single digit: 0 = Not at all, 1 = Several days, 2 = More than half the days, 3 = Nearly every day. Reply with the digit only. Cue: “Answer:” letters Answer each question with a single letter: A = Not at all, B = Several days, C = More than half the days, D = Nearly every day. Reply with the letter only. Cue: “Answer:” letters reversed Answer each question with a single letter: A = Nearly every day, B = More than half the days, C = Several days, D = Not at all. Reply with the letter only. Cue: “Answer:” Item stem and items stem Over the last 2 weeks, how often has the participant been bothered by the following problem? item 1 Little interest or pleasure in doing things. item 2 Feeling down, depressed, or hopeless. item 3 Trouble falling or staying asleep, or sleeping too much. item 4 Feeling tired or having little energy. item 5 Poor appetite or overeating. item 6 Feeling bad about yourself, or that you are a failure or have let yourself or your family down. item 7 Trouble concentrating on things, such as reading the newspaper or watching television. item 8 Moving or speaking so slowly that other people could have noticed, or the opposite, being so fidgety or restless that you have been moving around a lot more than usual. direct total Considering all eight PHQ-8 problems together, what is the participant’s total PHQ-8 score (from 0 to 24)? Reply in the form ’Score: N’. Cue: “Score:” 1.2 Models Supplementary Table 2 lists the models, their runtimes and the exact weights. Transformers models ran on the Apple GPU (Metal Performance Shaders) in bfloat16; GGUF files ran with llama.cpp (llama-cpp-python 0.3.16) with all layers on the GPU. The reasoning model received an empty reasoning block, and gpt-oss-20b was set to low reasoning effort and read in its final channel, so that every model answered directly. Supplementary Table 2: Models and weights. †Numerical-precision check only; not part of the multiverse. Model Developer Parameters (B) Runtime and precision Weights Qwen2.5 1.5B Alibaba 1.5 Transformers, bf16 Qwen/Qwen2.5-1.5B-Instruct (revision 989a79) Qwen2.5 3B Alibaba 3.1 Transformers, bf16 Qwen/Qwen2.5-3B-Instruct (revision a8e725) Qwen2.5 7B Alibaba 7.6 Transformers, bf16 Qwen/Qwen2.5-7B-Instruct (revision a09a354) Qwen2.5 14B Alibaba 14.8 llama.cpp, Q4_K_M Ollama library qwen2.5:14b; SHA-256 2049f5674b1e R1-Distill 7B DeepSeek 7.6 llama.cpp, Q4_K_M DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf; SHA-256 731ece8d06dc Llama 3.2 1B Meta 1.2 llama.cpp, Q8_0 Ollama library llama3.2:1b; SHA-256 74701a8c35f6 Llama 3.2 3B Meta 3.2 llama.cpp, Q4_K_M llama-3.2-3b-instruct.Q4_K_M.gguf; SHA-256 6c1a2b411610 Llama 3.1 8B Meta 8 llama.cpp, Q4_K_M Ollama library llama3.1:8b; SHA-256 667b0c1932bc Phi-3.5 mini Microsoft 3.8 llama.cpp, Q4_0 Ollama library phi3.5:latest; SHA-256 b5374915da53 Mistral 7B Mistral AI 7.2 llama.cpp, Q4_K_M mistral-7b-instruct-v0.3.Q4_K_M.gguf; SHA-256 1270d22c0fbb gpt-oss 20B OpenAI 21 llama.cpp, MXFP4/Q4_K_M gpt-oss-20b-Q4_K_M.gguf; SHA-256 c27536640e41 Qwen2.5 7B (4-bit)† Alibaba 7.6 llama.cpp, Q4_K_M Ollama library qwen2.5:7b; SHA-256 2bada8a74506 1.3 Reading the answer For each item we read the probability of every response option at the position after the answer cue, summing the probabilities of tokenisation variants (with and without a leading space). The four probabilities were renormalised to sum to one; their unnormalised sum (the option mass) measures how completely the model complied with the format. For the direct total, each number from 0 to 24 was scored as a token sequence. The intended rule multiplies its probability by the probability that the next token is not a digit, so that, for example, “1” is not counted when the model goes on to write “12”. The frozen code applied this factor only to numbers whose token sequence begins a longer candidate. Supplementary Table 3 gives the size of the difference: we re-scored the direct totals of 25 participants spread over the PHQ-8 range for four models, assembling both rules from the same forward pass. The 25 probabilities were renormalised in the same way. The expected-value rule scores an item as ∑kpk _kk\,p_k and the mode rule as argmaxkpk _kp_k. This is normalised candidate-label likelihood scoring: each option’s probability sums its tokenisation variants, and the mode is the option with the highest normalised probability. It need not coincide with the answer of free or token-by-token greedy generation; for example, an option whose two token variants each have probability 0.25 is chosen over an option with a single variant of probability 0.40, whereas greedy generation would emit the latter. Supplementary Table 3: Direct totals re-scored for 25 participants (five wordings each). Termination rule: the factor applied to every candidate against the frozen rule, from the same forward pass; flips, screening decisions that change under either scoring rule. Re-run against saved: the frozen rule in the new pass against the saved outputs (different batch composition). Termination rule Re-run against saved Model Totals max |Δ|| | Flips median |Δ|| | max |Δ|| | Flips gpt-oss 20B 125 0.0000 0 0.006 0.68 0 Llama 3.1 8B 125 0.0000 0 0.001 0.01 0 Mistral 7B 125 0.0080 0 0.001 0.01 0 Qwen2.5 7B 125 0.0028 0 0.090 0.45 1 1.4 Tree-structured evaluation All prompts for one transcript and view share the transcript as a prefix. We encoded the shared prefix once, then evaluated the continuations (templates, formats and items) in a few forward passes. An attention mask restricted each continuation to its own ancestors in the prompt tree. Position indices continued from that continuation’s prefix. For Transformers models this was a four-dimensional attention mask; for llama.cpp, one sequence per continuation sharing the cached prefix. Against one-prompt-at-a-time evaluation, the log-probabilities agreed to 10−410^-4 in 32-bit arithmetic; in bfloat16 option probabilities differed by at most 0.03, and with llama.cpp log-probabilities agreed within 0.007. The checks covered one participant and 16 sampled continuations (the 20 most probable tokens of each). They diagnose numerical agreement without guaranteeing it for every prompt; Supplementary Table 3 shows the reproducibility of whole scores across batch compositions. 1.5 Generalizability analysis The data were the expected item sums XpmtfvX_pmtfv for person p, model m, template t, format f and view v, in a fully crossed design with one observation per cell. We computed mean squares for all 31 effects and solved their expectations for the variance components, working downwards from the highest-order interaction. Negative estimates were retained when solving for lower-order components, then set to zero for reporting9. The highest-order interaction is confounded with residual error. For a decision study that averages over nm′n _m models, nt′n _t templates and nf′n _f formats (and, in discovery, over both views unless stated), the relative error variance σδ2σ^2_δ is the sum of the interactions involving the person, each divided by the numbers of conditions averaged over its other facets; the absolute error variance σΔ2σ^2_ adds the effects not involving the person, divided in the same way. The coefficients are Eρ2=σp2/(σp2+σδ2)Eρ^2=σ^2_p/(σ^2_p+σ^2_δ) and Φ=σp2/(σp2+σΔ2) =σ^2_p/(σ^2_p+σ^2_ ). 1.6 Consistency indices For an item-sum configuration with answer distributions pikp_ik for participant i and item k, the expected total is μi=∑kE[Xik] _i= _kE[X_ik] and, with items answered independently at temperature 1, the sampling variance of the total is si2=∑kVar(Xik)s^2_i= _kVar(X_ik). The expected correlation between two independent sampled runs is Vari(μi)/[Vari(μi)+si2¯]Var_i( _i)/[Var_i( _i)+ s^2_i]. Cronbach’s α was computed across the eight expected item scores. Agreement across templates is the mean Pearson correlation of the configuration’s expected totals with those of the same model, format and view under each of the other four templates. 1.7 Remedies For calibration, we randomly drew n participants (n=20n=20 or 40) and fitted isotonic regression from each configuration’s score to their PHQ-8 totals. We applied the fitted map to the remaining participants and evaluated the resulting decisions and implied prevalence. This was repeated for 200 draws. Prediction-powered estimates of prevalence used the configuration’s decisions on all participants and the labels of the same n participants to correct the bias of the model-based prevalence, with the standard variance estimate3. The standard estimate targets a superpopulation. Our coverage assessment instead used the prevalence of the fixed cohort. In the exploratory analyses we therefore also computed a finite-cohort version, θ^=f¯+(y−f)¯L θ= f+ (y-f)_L with variance (1−n/N)sy−f2/n(1-n/N)\,s^2_y-f/n, and an estimate from the labels alone, y¯L y_L with variance (1−n/N)sy2/n(1-n/N)\,s^2_y/n, where f is the configuration’s decision, y the label, L the labelled participants and N the cohort size. 1.8 Decision analyses (exploratory) How many versus who. For two raters with decisions A and B and positive rates pAp_A and pBp_B, P(A≠B)=P(A=1,B=0)+P(A=0,B=1)P(A≠ B)=P(A=1,B=0)+P(A=0,B=1) and pA−pB=P(A=1,B=0)−P(A=0,B=1)p_A-p_B=P(A=1,B=0)-P(A=0,B=1), so P(A≠B)=|pA−pB|+2minP(A=1,B=0),P(A=0,B=1)P(A≠ B)=|p_A-p_B|+2 \P(A=1,B=0),P(A=0,B=1)\. The first term is the least disagreement that the two positive rates force; the second, the excess, is zero exactly when one rater’s positives are a subset of the other’s. We average both over all pairs of distinct raters in a set. The identity is elementary and assigns no causes: a pure difference of scale between two raters with identical rankings and equal means produces disagreement at a fixed threshold, so the residual after subtracting each rater’s mean bias is not a measure of ranking disagreement. At equal capacity, every rater selects its q highest-scoring participants; we used expected-value scores, whose boundaries had no ties at the capacities reported (ties would be broken by participant order and are counted in the code). Single-choice comparisons use every pair of configurations that are identical except in the named choice, at the threshold of 10 and, for expected-value pairs, at equal capacity. Recalibration. In each of 200 random splits (the same splits for every rule and rater set), 20, 40 or 80 participants formed a labelled sample (20 or 40 in confirmation) and the rest were held out. Three rater sets were recalibrated: raters with AUC ≥ 0.70 in the labelled sample; one fixed configuration per model (instrument wording, digits, participant’s speech, expected value), which uses no selection; and, for comparison, raters with AUC ≥ 0.70 in the whole cohort, whose admission uses the held-out labels. The rules were: subtracting the mean difference between the score and the PHQ-8 total in the labelled sample; linear regression of the PHQ-8 total on the score, thresholded at 10; logistic regression of PHQ-8 ≥ 10 on the standardised score with a weak ridge penalty (10−310^-3, intercept included; Newton’s method, convergence recorded), deciding at a probability of 0.5; the threshold, among the attainable ones (the distinct labelled scores), whose labelled positive count is closest to the number of labelled positives, compared exactly as integers, the higher threshold on equal distance; and the highest threshold keeping at least 80% of the labelled positives, that is, the ⌈0.8n+⌉ 0.8\,n_+ -th largest labelled positive score. Ties among scores can make the prevalence target unattainable, and the achieved labelled rates are reported. Every rule sets a cut-off on the rater’s own score. An increasing fitted relation preserves its ranking, a decreasing fit reverses it, and a constant fit flags everyone or no one. With 40 labels and raters admitted on the labelled sample, the linear fit was increasing in all but 14 of 107,540 rater–split combinations in discovery and in all 62,743 in confirmation. On the held-out participants we computed the chance that two recalibrated raters disagree, accuracy, the median sensitivity and specificity across raters, and the median absolute error of the share flagged, and averaged over splits; the central 95% range across splits (2.5th to 97.5th percentiles) describes split-to-split variation within one cohort and is not a confidence interval (Supplementary Table 4). Deferral set aside the held-out participants flagged by more than 20% and fewer than 80% of the recalibrated raters, retaining those at exactly 20% or 80%, and scored the majority decision on the rest. Defensible core. Besides the full grid, raters with AUC ≥ 0.70 and one fixed configuration per model, we report a clinically defensible core: the questionnaire’s own instruction or the psychiatrist wording (neither changes the assessment policy), digit answers, expected values and symptom sums, in every view (44 raters in discovery, 22 in confirmation). It uses no outcome labels to choose raters (Supplementary Table 5). Where disagreement falls. For each participant, the chance that two raters decide differently, 2πi(1−πi)K/(K−1)2 _i(1- _i)K/(K-1), was averaged within bands of self-reported PHQ-8 (0–4, 5–9, 10–14, 15–24), at the threshold of 10 and at equal capacity, and each band’s share of the total was computed (Supplementary Table 6). This describes the criterion status of the participants on whom raters differ; it does not identify which decision is clinically right. Deferral at matched coverage. On the held-out participants of each split, with raters admitted on the 40 labels and recalibrated to the labelled prevalence, we compared on the same participants: a single recalibrated rater (averaged over raters); the majority of raters; the majority after deferring participants on whom fewer than a share c of raters agreed (c = 0.6, 0.7, 0.8, 0.9, 1.0; c = 0.8 defers those flagged by strictly more than 20% and fewer than 80% of raters, retaining those at exactly 20% or 80%; unanimous consensus retained fewer than 3% of participants and is not displayed); single raters evaluated on the same retained participants, which separates the effect of deferral from that of pooling; and each single rater deferring its own closest calls, the participants whose scores lie nearest its cut-off, to reach the same coverage. We report accuracy, sensitivity, specificity and the numbers of retained participants with and without a positive screen (Supplementary Table 7; Extended Data Fig. 9). Neither deferral rule has defined costs for review or for missed cases, so these are descriptive comparisons, not a clinical policy evaluation. Uncertainty for headline estimands. For three headline estimands (pairwise disagreement at 10 among raters with AUC ≥ 0.70, and equal-capacity disagreement among expected-value raters with AUC ≥ 0.70 and among one fixed configuration per model) we resampled participants 1,000 times with the rater sets held fixed; the percentile intervals are therefore conditional on the raters and describe sampling of participants, not of configurations. Transfer. For configurations with AUC ≥ 0.70 in discovery (participant view), three discovery calibrations were applied unchanged to the same configuration in confirmation: subtracting the discovery mean over-rating; the attainable discovery threshold closest to the discovery prevalence; and the discovery threshold keeping at least 80% of discovery positives. The discovery set had manual transcripts and an operator-controlled interviewer, and the confirmation set automatic transcripts and an autonomous interviewer, so this is one related transfer, not a comparison of independent sites. Supplementary Table 4: Recalibration with 40 labelled participants, evaluated on the held-out participants (raters admitted by their AUC in the labelled sample; means over 200 splits). Disagreement, the chance that two recalibrated raters decide a held-out participant differently; error, median absolute error of the share flagged (2.5th–97.5th percentile across splits); target, the share of raters and splits whose labelled positive rate came within 5 points of the labelled prevalence (prevalence rule) or that kept at least 80% of labelled positives (sensitivity rule). Set Rule Disagree (%) Acc. (%) Sens. (%) Spec. (%) Error (points) Target (%) Discovery Threshold of 10 40 61 83 67 28 (21–32) Mean over-rating 18 75 39 92 14 (7–25) Linear 16 75 42 91 12 (5–27) Logistic 17 75 45 89 11 (4–28) Labelled prevalence 20 75 56 84 7 (3–19) 88 ≥ 80% of positives 27 68 78 68 17 (7–33) 100 Confirmation Threshold of 10 40 59 91 53 33 (26–39) Mean over-rating 21 74 49 92 14 (7–24) Linear 20 75 54 89 11 (4–24) Logistic 20 75 56 87 10 (4–22) Labelled prevalence 21 75 62 84 8 (4–15) 86 ≥ 80% of positives 27 69 79 70 15 (7–28) 100 Supplementary Table 5: How the decision results change under restrictions of the grid. K, raters; at 10, the chance that two raters decide a participant differently at the threshold; forced, the share of that disagreement forced by different positive rates; equal no., the chance at equal capacity among expected-value raters of the set (all expected-value raters for the full grid); recal. and acc., held-out disagreement and accuracy after matching the labelled prevalence with 40 labels (no selection for the defensible core and the fixed configurations; whole-cohort AUC for the AUC ≥ 0.70 row). Set Raters K At 10 (%) Forced (%) Equal no. (%) Recal. (%) Acc. (%) Discovery Full grid 880 44 92 27 – – AUC ≥ 0.70 540 40 94 18 19 75 Defensible core 44 31 95 21 21 74 One per model 11 32 98 25 24 72 Confirmation Full grid 440 43 94 26 – – AUC ≥ 0.70 319 40 95 20 21 76 Defensible core 22 30 97 21 20 76 One per model 11 32 97 22 21 75 Supplementary Table 6: Where the disagreement falls, by self-reported PHQ-8, for expected-value raters with AUC ≥ 0.70: each participant’s chance of being decided differently by two raters, averaged within the band (mean) and as the band’s share of all such disagreement (share), at the threshold of 10 and at equal capacity. At 10 Equal numbers Set PHQ-8 n Mean (%) Share (%) Mean (%) Share (%) Discovery 0–4 86 44 48 9 24 5–9 46 43 25 25 33 10–14 30 40 15 27 24 15–24 27 33 11 25 20 Confirmation 0–4 36 47 48 13 28 5–9 21 42 25 24 29 10–14 13 38 14 30 23 15–24 16 27 12 22 21 Supplementary Table 7: Deferral at matched coverage on held-out participants (raters admitted on 40 labels and recalibrated to the labelled prevalence; means over 200 splits). Coverage, participants decided rather than deferred; counts in parentheses, mean retained participants with and without a positive screen. Set Rule Coverage (%) Accuracy (%) Sens. (%) Spec. (%) Discovery Single rater, all 100 75 55 83 Majority, all 100 78 58 88 Majority, consensus ≥ 0.8 (24+, 82−-) 71 86 56 95 single rater, same participants 71 83 54 91 single rater, own closest calls deferred 71 82 48 92 Majority, consensus ≥ 0.9 (13+, 70−-) 56 90 55 97 single rater, same participants 56 88 54 94 single rater, own closest calls deferred 56 85 45 94 Confirmation Single rater, all 100 75 61 83 Majority, all 100 80 65 88 Majority, consensus ≥ 0.8 (9+, 24−-) 70 89 70 95 single rater, same participants 70 84 66 91 single rater, own closest calls deferred 70 82 57 92 Majority, consensus ≥ 0.9 (5+, 19−-) 53 92 65 98 single rater, same participants 53 89 63 95 single rater, own closest calls deferred 53 86 53 94 Supplementary Table 8: Participants decided differently by configurations that differ in one choice only: median (interquartile range) over pairs, at the threshold of 10 and, for expected-value pairs, selecting as many participants as screen positive by self-report. Discovery Confirmation Choice changed At 10 Equal numbers At 10 Equal numbers Letter–answer mapping reversed 20 (7–55) 21 (12–43) 26 (6–49) 21 (12–36) Digits vs letters 13 (5–25) 15 (7–24) 12 (5–22) 14 (9–22) Psychiatrist persona 7 (2–13) 8 (6–15) 6 (2–12) 10 (7–16) Unmentioned symptoms rated absent 21 (8–35) 14 (8–23) 23 (9–36) 14 (9–21) Role-play as the participant 8 (2–14) 11 (7–17) 8 (3–14) 10 (7–19) Use tone and other cues 8 (2–17) 11 (7–20) 9 (2–14) 14 (7–19) Most likely option, not expected value 4 (1–8) – 3 (1–8) – One overall rating, not a symptom sum 24 (10–49) 15 (8–22) 21 (10–35) 16 (9–23) Whole conversation, not participant’s speech 11 (3–19) 17 (13–26) – – Next model size, same family 28 (13–47) 22 (15–33) 30 (16–48) 23 (16–30) A different model 35 (17–63) 25 (17–33) 35 (15–60) 23 (16–33) 2 Supplementary Results Supplementary Table 9: Results by model (discovery set; all 80 configurations of each model). Sens. and Spec., median sensitivity and specificity for PHQ-8 ≥ 10 over the model’s configurations (descriptive). Bias, median over configurations of the mean difference between the configuration’s score and the PHQ-8 total. Option mass, median over participants, templates and formats. Model AUC, median (range) AUC ≥ 0.70 (%) Positive, % (range) Sens. (%) Spec. (%) Bias Option mass Qwen2.5 1.5B 0.59 (0.41–0.77) 8 50 (0–100) 55 52 +3.2 0.99 Qwen2.5 3B 0.70 (0.49–0.82) 49 84 (0–100) 96 21 +5.0 1.00 Qwen2.5 7B 0.81 (0.64–0.87) 96 67 (17–98) 93 45 +5.0 1.00 Qwen2.5 14B 0.85 (0.78–0.88) 100 33 (1–82) 71 83 +0.7 1.00 R1-Distill 7B 0.67 (0.35–0.81) 32 80 (1–100) 89 23 +5.3 0.99 Llama 3.2 1B 0.61 (0.34–0.78) 24 99 (0–100) 100 2 +7.3 0.89 Llama 3.2 3B 0.74 (0.45–0.82) 60 97 (58–100) 100 4 +7.2 1.00 Llama 3.1 8B 0.84 (0.76–0.88) 100 24 (0–68) 54 89 -0.4 0.99 Phi-3.5 mini 0.63 (0.49–0.81) 26 79 (3–99) 84 23 +4.6 1.00 Mistral 7B 0.80 (0.55–0.87) 92 85 (21–100) 98 21 +6.0 1.00 gpt-oss 20B 0.81 (0.59–0.85) 88 14 (0–46) 31 95 -0.8 0.91 Supplementary Table 10: Variance components of the expected item sum (person × model × template × format × view). Source σ^2 σ^2 (raw) σ^2 σ^2 Share (%) model 6.943 6.943 30.0 model × format 3.645 3.645 15.8 person 2.437 2.437 10.5 person × model 1.872 1.872 8.1 template 1.779 1.779 7.7 format 1.363 1.363 5.9 model × template 0.971 0.971 4.2 model × template × format 0.691 0.691 3.0 person × model × format × view 0.588 0.588 2.5 person × model × format 0.498 0.498 2.2 person × model × template × format × view 0.429 0.429 1.9 person × format 0.413 0.413 1.8 person × model × template 0.244 0.244 1.1 model × format × view 0.234 0.234 1.0 person × model × template × format 0.215 0.215 0.9 person × model × template × view 0.156 0.156 0.7 person × model × view 0.155 0.155 0.7 person × view 0.143 0.143 0.6 model × template × format × view 0.138 0.138 0.6 person × template 0.076 0.076 0.3 template × format 0.043 0.043 0.2 person × template × format 0.021 0.021 0.1 person × template × view 0.014 0.014 0.1 model × view 0.014 0.014 0.1 model × template × view 0.014 0.014 0.1 template × view 0.014 0.014 0.1 person × format × view 0.006 0.006 0.0 person × template × format × view 0.002 0.002 0.0 template × format × view -0.000 0.000 0.0 format × view -0.010 0.000 0.0 view -0.009 0.000 0.0 Supplementary Table 11: Decision studies (discovery; both transcript views averaged). For one view, averaging over all models, templates and formats gives Eρ2Eρ^2 = 0.82 and Φ = 0.53; a single configuration with one view gives Φ = 0.11. Models Templates Formats Eρ2Eρ^2 Φ 1 1 1 0.37 0.11 3 1 1 0.58 0.21 11 1 1 0.73 0.32 1 1 3 0.45 0.14 3 1 3 0.67 0.27 11 1 3 0.82 0.40 1 5 1 0.42 0.13 3 5 1 0.62 0.26 11 5 1 0.76 0.41 1 5 3 0.49 0.17 3 5 3 0.71 0.34 11 5 3 0.85 0.55 Supplementary Table 12: Directional predictions specified before the analysis, and their outcomes. Prediction Criterion Discovery Confirmation H1 Configuration contributes more variance to the AUC than participant sampling ratio >> 1 10.6 supported 5.6 supported H2 Participants with discordant decisions among acceptable configurations ≥ 25% 100% supported 100% supported H3 Range of implied prevalence across configurations ≥ 30 p 100 p supported 100 p supported H4 Person share of item-sum variance << 50% 10.5% supported 13.0% supported H5 Rank correlation of Cronbach’s α with the AUC << 0.30 0.02 supported -0.06 supported H6 Median-model AUC from interviewer speech only >> 0.60 0.59 not supported – – H7 Prevalence range after calibration with 40 labels, against before at most half 58 vs 100 p not supported 65 vs 100 p not supported 3 Labels and transcripts All analyses use the registered definition of a positive screen, a PHQ-8 total of 10 or more. It differs from the binary flag distributed with the corpora for one discovery participant and five confirmation participants. Item scores were available for 142 discovery participants and complete for 141. In six discovery transcripts, empty turns had been converted to the literal token ‘nan’ (1–12 tokens per transcript); excluding these six participants changed the median AUC across configurations from 0.753 to 0.757, and the frozen loader was not changed. 4 Deviations from the pre-registered plan Regenerated outputs. Two secondary outputs of the first run, for Qwen2.5 1.5B (the interviewer’s speech for one participant and the automatic transcript for another), contained no finite probabilities, and the median over models used for the interviewer-speech prediction silently omitted that model. Both outputs were regenerated with the frozen code, giving finite outputs with option mass 0.995 and 0.990; the original files are kept. Re-running the frozen analysis left the primary results unchanged and changed the interviewer-speech prediction from supported (median AUC 0.61) to not supported (0.59). Direct-total termination. The frozen code applied the end-of-number factor only to numbers whose tokens begin a longer candidate, not to every number as described (Supplementary Table 3). The code was not changed; in the re-scored sample (four models, 25 participants, participant view), the effect on expected totals was at most 0.008 points, with no change of decision; this measures the defect in that sample rather than bounding it for other models or views. Test attribution and degrees of freedom. The plan names Steiger’s test for comparing a configuration’s correlations with the PHQ-8 and with PTSD severity; the code implements the Meng–Rosenthal–Rubin form43. Steiger’s pooled form61 and a paired participant bootstrap give the same shares of configurations (3.9% and 4.3% in discovery; 1.1% and 0.9% in confirmation). The partial Spearman tests of the regularities used n−2n-2 degrees of freedom; with n−3n-3, every regularity still meets its criterion (PTSD regularity in confirmation, P = 0.049). Decision-study views. In discovery the decision studies average over both transcript views; values for one view are given with Supplementary Table 11. During data collection the analysis code was changed once so that interim runs on an incomplete set of outputs recorded the interviewer-speech prediction as missing instead of stopping. The change was reverted before the registered analyses, which used the code as registered. The local copies of three model weights (Qwen2.5 1.5B, 3B and 7B) were lost during data collection and obtained again from the same repositories. Re-scoring two participants with the new copy of Qwen2.5 1.5B reproduced the earlier outputs exactly. Scoring runs were interrupted and resumed several times, and the order in which models were run changed. Answers were read from next-token probabilities without sampling, and every participant was scored once per model and view; across batch compositions, scores agree to the tolerance in Supplementary Table 3. The two addenda (a label-free rule and nine regularities) were specified after the discovery analysis and before any confirmation output was compared with the criterion, while confirmation outputs were being generated. Analyses added after the confirmation analysis are listed in Methods and are exploratory. References Abdin et al. (2024) M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, Q. Cai, V. Chaudhary, D. Chen, D. Chen, W. Chen, Y. Chen, Y. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V. Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. Del Giorno, A. Goswami, S. Gunasekar, E. Haider, J. Hao, R. J. Hewett, W. Hu, J. Huynh, D. Iter, S. A. Jacobs, M. Javaheripi, X. Jin, N. Karampatziakis, P. Kauffmann, M. Khademi, D. Kim, Y. J. Kim, L. Kurilenko, J. R. Lee, Y. T. Lee, Y. Li, Y. Li, C. Liang, L. Liden, X. Lin, Z. Lin, C. Liu, L. Liu, M. Liu, W. Liu, X. Liu, C. Luo, P. Madan, A. Mahmoudzadeh, D. Majercak, M. Mazzola, C. C. T. Mendes, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, L. Ren, G. de Rosa, C. Rosset, S. Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, Y. Shen, S. Shukla, X. Song, M. Tanaka, A. Tupini, P. Vaddamanu, C. Wang, G. Wang, L. Wang, S. Wang, X. Wang, Y. Wang, R. Ward, W. Wen, P. Witte, H. Wu, X. Wu, M. Wyatt, B. Xiao, C. Xu, J. Xu, W. Xu, J. Xue, S. Yadav, F. Yang, J. Yang, Y. Yang, Z. Yang, D. Yu, L. Yuan, C. Zhang, C. Zhang, J. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, and X. Zhou Phi-3 technical report: a highly capable language model locally on your phone. Note: Preprint at https://arxiv.org/abs/2404.14219 External Links: Document Cited by: Models.. Alkaeed et al. (2026) M. Alkaeed, A. Qayyum, N. A. Kashreef, M. Bilal, and J. Qadir Same patient, different words, different diagnosis? evaluating semantic stability in clinical LLMs. Note: Preprint at https://arxiv.org/abs/2605.30646 External Links: Document Cited by: Introduction. Angelopoulos et al. (2023) A. N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan, and T. Zrnic Prediction-powered inference. Science 382 (6671), p. 669–674. External Links: ISSN 1095-9203, Document, Link Cited by: §1.7, How many versus who, Estimands.. Atil et al. (2024) B. Atil, S. Aykent, A. Chittams, L. Fu, R. J. Passonneau, E. Radcliffe, G. R. Rajagopal, A. Sloan, T. Tudrej, F. Ture, Z. Wu, L. Xu, and B. Baldwin Non-determinism of "deterministic" LLM settings. Note: Preprint at https://arxiv.org/abs/2408.04667 External Links: Document Cited by: Introduction. Batzdorfer and Santagiustina (2026) V. Batzdorfer and C. R. M. A. Santagiustina Reproducibility is not construct validity: LLM measurement of institutionally situated communication. Note: Preprint at https://arxiv.org/abs/2609.19866 External Links: Document Cited by: Introduction. Baumann et al. (2025) J. Baumann, P. Röttger, A. Urman, A. Wendsjö, F. M. Plaza-del-Arco, J. B. Gruber, and D. Hovy Large language model hacking: quantifying the hidden risks of using LLMs for text annotation. Note: Preprint at https://arxiv.org/abs/2509.08825 External Links: Document Cited by: Introduction. Black et al. (2022) E. Black, M. Raghavan, and S. Barocas Model multiplicity: opportunities, concerns, and solutions. In 2022 ACM Conference on Fairness Accountability and Transparency, FAccT ’22, p. 850–863. External Links: Document, Link Cited by: Introduction, Discussion. Botvinik-Nezer et al. (2020) R. Botvinik-Nezer, F. Holzmeister, C. F. Camerer, A. Dreber, J. Huber, M. Johannesson, M. Kirchler, R. Iwanir, J. A. Mumford, R. A. Adcock, P. Avesani, B. M. Baczkowski, A. Bajracharya, L. Bakst, S. Ball, M. Barilari, N. Bault, D. Beaton, J. Beitner, R. G. Benoit, R. M. W. J. Berkers, J. P. Bhanji, B. B. Biswal, S. Bobadilla-Suarez, T. Bortolini, K. L. Bottenhorn, A. Bowring, S. Braem, H. R. Brooks, E. G. Brudner, C. B. Calderon, J. A. Camilleri, J. J. Castrellon, L. Cecchetti, E. C. Cieslik, Z. J. Cole, O. Collignon, R. W. Cox, W. A. Cunningham, S. Czoschke, K. Dadi, C. P. Davis, A. D. Luca, M. R. Delgado, L. Demetriou, J. B. Dennison, X. Di, E. W. Dickie, E. Dobryakova, C. L. Donnat, J. Dukart, N. W. Duncan, J. Durnez, A. Eed, S. B. Eickhoff, A. Erhart, L. Fontanesi, G. M. Fricke, S. Fu, A. Galván, R. Gau, S. Genon, T. Glatard, E. Glerean, J. J. Goeman, S. A. E. Golowin, C. González-García, K. J. Gorgolewski, C. L. Grady, M. A. Green, J. F. Guassi Moreira, O. Guest, S. Hakimi, J. P. Hamilton, R. Hancock, G. Handjaras, B. B. Harry, C. Hawco, P. Herholz, G. Herman, S. Heunis, F. Hoffstaedter, J. Hogeveen, S. Holmes, C. Hu, S. A. Huettel, M. E. Hughes, V. Iacovella, A. D. Iordan, P. M. Isager, A. I. Isik, A. Jahn, M. R. Johnson, T. Johnstone, M. J. E. Joseph, A. C. Juliano, J. W. Kable, M. Kassinopoulos, C. Koba, X. Kong, T. R. Koscik, N. E. Kucukboyaci, B. A. Kuhl, S. Kupek, A. R. Laird, C. Lamm, R. Langner, N. Lauharatanahirun, H. Lee, S. Lee, A. Leemans, A. Leo, E. Lesage, F. Li, M. Y. C. Li, P. C. Lim, E. N. Lintz, S. W. Liphardt, A. B. Losecaat Vermeer, B. C. Love, M. L. Mack, N. Malpica, T. Marins, C. Maumet, K. McDonald, J. T. McGuire, H. Melero, A. S. Méndez Leal, B. Meyer, K. N. Meyer, G. Mihai, G. D. Mitsis, J. Moll, D. M. Nielson, G. Nilsonne, M. P. Notter, E. Olivetti, A. I. Onicas, P. Papale, K. R. Patil, J. E. Peelle, A. Pérez, D. Pischedda, J. Poline, Y. Prystauka, S. Ray, P. A. Reuter-Lorenz, R. C. Reynolds, E. Ricciardi, J. R. Rieck, A. M. Rodriguez-Thompson, A. Romyn, T. Salo, G. R. Samanez-Larkin, E. Sanz-Morales, M. L. Schlichting, D. H. Schultz, Q. Shen, M. A. Sheridan, J. A. Silvers, K. Skagerlund, A. Smith, D. V. Smith, P. Sokol-Hessner, S. R. Steinkamp, S. M. Tashjian, B. Thirion, J. N. Thorp, G. Tinghög, L. Tisdall, S. H. Tompson, C. Toro-Serey, J. J. Torre Tresols, L. Tozzi, V. Truong, L. Turella, A. E. van ’t Veer, T. Verguts, J. M. Vettel, S. Vijayarajah, K. Vo, M. B. Wall, W. D. Weeda, S. Weis, D. J. White, D. Wisniewski, A. Xifra-Porxas, E. A. Yearling, S. Yoon, R. Yuan, K. S. L. Yuen, L. Zhang, X. Zhang, J. E. Zosky, T. E. Nichols, R. A. Poldrack, and T. Schonberg Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582 (7810), p. 84–88. External Links: ISSN 1476-4687, Document, Link Cited by: Discussion. Brennan (2001) R. L. Brennan Generalizability theory. Springer New York. External Links: ISBN 9781475734560, Document, Link Cited by: §1.5, Introduction, Estimands.. Burdisso et al. (2024) S. Burdisso, E. Reyes-Ramírez, E. Villatoro-tello, F. Sánchez-Vega, A. Lopez Monroy, and P. Motlicek DAIC-WOZ: on the validity of using the therapist’s prompts in automatic depression detection from clinical interviews. In Proceedings of the 6th Clinical Natural Language Processing Workshop, p. 82–90. External Links: Document, Link Cited by: What the raters read, Raters and scores.. Chen et al. (2026) A. X. Chen, G. Horga, and S. Escola Using large language models to measure symptom severity scores in patients at-risk for schizophrenia. Schizophrenia Bulletin 52 (5), p. sbag084. External Links: ISSN 1745-1701, Document, Link Cited by: Introduction, Discussion. Cronbach et al. (1972) L. J. Cronbach, G. C. Gleser, H. Nanda, and N. Rajaratnam The dependability of behavioral measurements: theory of generalizability for scores and profiles. Wiley, New York. External Links: Link Cited by: Introduction, Estimands.. Demszky et al. (2023) D. Demszky, D. Yang, D. S. Yeager, C. J. Bryan, M. Clapper, S. Chandhok, J. C. Eichstaedt, C. Hecht, J. Jamieson, M. Johnson, M. Jones, D. Krettek-Cobb, L. Lai, N. JonesMitchell, D. C. Ong, C. S. Dweck, J. J. Gross, and J. W. Pennebaker Using large language models in psychology. Nature Reviews Psychology. External Links: ISSN 2731-0574, Document, Link Cited by: Introduction. DeVault et al. (2014) D. DeVault, R. Artstein, G. Benn, T. Dey, E. Fast, A. Gainer, K. Georgila, J. Gratch, A. Hartholt, M. Lhommet, G. Lucas, S. Marsella, F. Morbini, A. Nazarian, S. Scherer, G. Stratou, A. Suri, D. Traum, R. Wood, Y. Xu, A. Rizzo, and L. Morency SimSensei Kiosk: a virtual human interviewer for healthcare decision support. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), p. 1061–1068. External Links: Link Cited by: Data.. Du et al. (2025) A. Y. Du, D. D. Ngo, and Z. S. Wu Reconciling model multiplicity for downstream decision making. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Introduction, Discussion, Discussion. Feighner (1972) J. P. Feighner Diagnostic criteria for use in psychiatric research. Archives of General Psychiatry 26 (1), p. 57. External Links: ISSN 0003-990X, Document, Link Cited by: Introduction, Discussion. Galatzer-Levy et al. (2023) I. R. Galatzer-Levy, D. McDuff, V. Natarajan, A. Karthikesalingam, and M. Malgaroli The capability of large language models to measure psychiatric functioning. Note: Preprint at https://arxiv.org/abs/2308.01834 External Links: Document Cited by: Introduction. Gallifant et al. (2025) J. Gallifant, M. Afshar, S. Ameen, Y. Aphinyanaphongs, S. Chen, G. Cacciamani, D. Demner-Fushman, D. Dligach, R. Daneshjou, C. Fernandes, L. H. Hansen, A. Landman, L. Lehmann, L. G. McCoy, T. Miller, A. Moreno, N. Munch, D. Restrepo, G. Savova, R. Umeton, J. W. Gichoya, G. S. Collins, K. G. M. Moons, L. A. Celi, and D. S. Bitterman The TRIPOD-LLM reporting guideline for studies using large language models. Nature Medicine 31 (1), p. 60–69. External Links: ISSN 1546-170X, Document, Link Cited by: Discussion. Gelman and Carlin (2014) A. Gelman and J. Carlin Beyond power calculations: assessing Type s (sign) and Type m (magnitude) errors. Perspectives on Psychological Science 9 (6), p. 641–651. External Links: ISSN 1745-6924, Document, Link Cited by: Estimands.. Gratch et al. (2014) J. Gratch, R. Artstein, G. Lucas, G. Stratou, S. Scherer, A. Nazarian, R. Wood, J. Boberg, D. DeVault, S. Marsella, D. Traum, S. Rizzo, and L. Morency The distress analysis interview corpus of human and computer interviews. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), p. 3123–3128. External Links: Link Cited by: Data.. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. De Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma The Llama 3 herd of models. Note: Preprint at https://arxiv.org/abs/2407.21783 External Links: Document Cited by: Models.. Gunal et al. (2024) A. Gunal, B. Lin, and D. Bouneffouf Conversational topic recommendation in counseling and psychotherapy with decision transformer and large language models. In Proceedings of the 6th Clinical Natural Language Processing Workshop, Mexico City, p. 196–201. External Links: Document Cited by: Discussion. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Ding, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Chen, J. Yuan, J. Tu, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. You, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Zhou, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645 (8081), p. 633–638. External Links: ISSN 1476-4687, Document, Link Cited by: Models.. Hanley and McNeil (1982) J. A. Hanley and B. J. McNeil The meaning and use of the area under a receiver operating characteristic (ROC) curve.. Radiology 143 (1), p. 29–36. External Links: ISSN 1527-1315, Document, Link Cited by: Estimands.. Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed Mistral 7b. Note: Preprint at https://arxiv.org/abs/2310.06825 External Links: Document Cited by: Models.. Kebe et al. (2026) G. Y. Kebe, J. M. Girard, E. Liebenthal, J. Baker, F. De la Torre, and L. Morency LLAMADRS: evaluating open-source LLMs on real clinical interviews—to reason or not to reason?. Transactions of the Association for Computational Linguistics 14, p. 1589–1604. External Links: ISSN 2307-387X, Document, Link Cited by: Introduction, Discussion. Kendell (1971) R. E. Kendell Diagnostic criteria of american and british psychiatrists. Archives of General Psychiatry 25 (2), p. 123. External Links: ISSN 0003-990X, Document, Link Cited by: Introduction, Discussion. Kroenke et al. (2009) K. Kroenke, T. W. Strine, R. L. Spitzer, J. B.W. Williams, J. T. Berry, and A. H. Mokdad The PHQ-8 as a measure of current depression in the general population. Journal of Affective Disorders 114 (1-3), p. 163–173. External Links: ISSN 0165-0327, Document, Link Cited by: Introduction, Data.. Lechuga López et al. (2026) L. J. Lechuga López, F. E. Shamout, and T. G. J. Rudner An empirical analysis of calibration and selective prediction in multimodal clinical condition classification. In Proceedings of the 7th Conference on Health, Inference, and Learning, Proceedings of Machine Learning Research, Vol. 333, p. 794–833. External Links: Link Cited by: How many versus who. Levis et al. (2019) B. Levis, A. Benedetti, and B. D. Thombs Accuracy of patient health questionnaire-9 (PHQ-9) for screening to detect major depression: individual participant data meta-analysis. BMJ, p. l1476. External Links: ISSN 1756-1833, Document, Link Cited by: Data.. Lin et al. (2023a) B. Lin, D. Bouneffouf, G. Cecchi, and R. Tejwani Neural topic modeling of psychotherapy sessions. In Artificial Intelligence for Personalized Medicine: Promoting Healthy Living and Longevity, A. Shaban-Nejad, M. Michalowski, and S. Bianco (Eds.), Studies in Computational Intelligence, Vol. 1106, p. 209–219. External Links: Document Cited by: Introduction. Lin et al. (2025) B. Lin, D. Bouneffouf, Y. Landa, R. Jespersen, C. Corcoran, and G. Cecchi COMPASS: computational mapping of patient-therapist alliance strategies with language modeling. Translational Psychiatry 15 (1), p. 166. External Links: ISSN 2158-3188, Document, Link Cited by: Introduction. Lin et al. (2023b) B. Lin, G. Cecchi, and D. Bouneffouf Deep annotation of therapeutic working alliance in psychotherapy. In Artificial Intelligence for Personalized Medicine: Promoting Healthy Living and Longevity, A. Shaban-Nejad, M. Michalowski, and S. Bianco (Eds.), Studies in Computational Intelligence, Vol. 1106, p. 193–207. External Links: Document Cited by: Introduction. Lin et al. (2023c) B. Lin, G. Cecchi, and D. Bouneffouf Psychotherapy AI companion with reinforcement learning recommendations and interpretable policy dynamics. In Companion Proceedings of the ACM Web Conference 2023, New York, p. 932–939. External Links: Document Cited by: Discussion. Lin et al. (2023d) B. Lin, G. Cecchi, and D. Bouneffouf SupervisorBot: NLP-annotated real-time recommendations of psychotherapy treatment strategies with deep reinforcement learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, p. 7149–7153. External Links: Document Cited by: Discussion. Lin et al. (2024) B. Lin, G. Cecchi, and D. Bouneffouf Working alliance transformer for psychotherapy dialogue classification. In Proceedings of the 6th Clinical Natural Language Processing Workshop, Mexico City, p. 64–69. External Links: Document Cited by: Discussion. Lin (2026a) B. Lin COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal. Note: Preprint at https://arxiv.org/abs/2610.06615 External Links: Link Cited by: Discussion. Lin (2026b) Z. Lin From prompts to constructs: a dual-validity framework for large language model research in psychology. Annual Review of Psychology. External Links: ISSN 1545-2085, Document, Link Cited by: Introduction. Lockfisch et al. (2025) S. Lockfisch, K. Schwethelm, M. Menten, R. Braren, D. Rueckert, A. Ziller, and G. Kaissis On arbitrary predictions from equally valid models. Note: Preprint at https://arxiv.org/abs/2507.19408 External Links: Document Cited by: How many versus who. Lorenzoni et al. (2025) G. Lorenzoni, P. E. Velmovitsky, P. Alencar, and D. Cowan GPT-4 on clinic depression assessment: an llm-based pilot study. Note: Preprint at https://arxiv.org/abs/2501.00199 External Links: Document Cited by: Introduction. MacCoun and Perlmutter (2015) R. MacCoun and S. Perlmutter Blind analysis: hide results to seek the truth. Nature 526 (7572), p. 187–189. External Links: ISSN 1476-4687, Document, Link Cited by: Introduction. Marx et al. (2020) C. T. Marx, F. d. P. Calmon, and B. Ustun Predictive multiplicity in classification. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, p. 6765–6774. External Links: Link Cited by: Introduction, Discussion. Meng et al. (1992) X. Meng, R. Rosenthal, and D. B. Rubin Comparing correlated correlation coefficients.. Psychological Bulletin 111 (1), p. 172–175. External Links: ISSN 0033-2909, Document, Link Cited by: §4, Estimands.. Mizrahi et al. (2024) M. Mizrahi, G. Kaplan, D. Malkin, R. Dror, D. Shahaf, and G. Stanovsky State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics 12, p. 933–949. External Links: Document, Link Cited by: Introduction. Norman et al. (2026) J. D. Norman, M. U. Rivera, and D. A. Hughes Reliability without validity: a systematic, large-scale evaluation of LLM-as-a-Judge models across agreement, consistency, and bias. Note: Preprint at https://arxiv.org/abs/2606.19544 External Links: Document Cited by: Introduction. Nosek et al. (2018) B. A. Nosek, C. R. Ebersole, A. C. DeHaven, and D. T. Mellor The preregistration revolution. Proceedings of the National Academy of Sciences 115 (11), p. 2600–2606. External Links: ISSN 1091-6490, Document, Link Cited by: Introduction. Obradovich et al. (2024) N. Obradovich, S. S. Khalsa, W. U. Khan, J. Suh, R. H. Perlis, O. Ajilore, and M. P. Paulus Opportunities and risks of large language models in psychiatry. NPP—Digital Psychiatry and Neuroscience 2 (1), p. 8. External Links: ISSN 2948-1570, Document, Link Cited by: Introduction. OpenAI et al. (2025) OpenAI, S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y. Bai, B. Baker, H. Bao, B. Barak, A. Bennett, T. Bertao, N. Brett, E. Brevdo, G. Brockman, S. Bubeck, C. Chang, K. Chen, M. Chen, E. Cheung, A. Clark, D. Cook, M. Dukhan, C. Dvorak, K. Fives, V. Fomenko, T. Garipov, K. Georgiev, M. Glaese, T. Gogineni, A. Goucher, L. Gross, K. G. Guzman, J. Hallman, J. Hehir, J. Heidecke, A. Helyar, H. Hu, R. Huet, J. Huh, S. Jain, Z. Johnson, C. Koch, I. Kofman, D. Kundel, J. Kwon, V. Kyrylov, E. Y. Le, G. Leclerc, J. P. Lennon, S. Lessans, M. Lezcano-Casado, Y. Li, Z. Li, J. Lin, J. Liss, L. (. Liu, J. Liu, K. Lu, C. Lu, Z. Martinovic, L. McCallum, J. McGrath, S. McKinney, A. McLaughlin, S. Mei, S. Mostovoy, T. Mu, G. Myles, A. Neitz, A. Nichol, J. Pachocki, A. Paino, D. Palmie, A. Pantuliano, G. Parascandolo, J. Park, L. Pathak, C. Paz, L. Peran, D. Pimenov, M. Pokrass, E. Proehl, H. Qiu, G. Raila, F. Raso, H. Ren, K. Richardson, D. Robinson, B. Rotsted, H. Salman, S. Sanjeev, M. Schwarzer, D. Sculley, H. Sikchi, K. Simon, K. Singhal, Y. Song, D. Stuckey, Z. Sun, P. Tillet, S. Toizer, F. Tsimpourlas, N. Vyas, E. Wallace, X. Wang, M. Wang, O. Watkins, K. Weil, A. Wendling, K. Whinnery, C. Whitney, H. Wong, L. Yang, Y. Yang, M. Yasunaga, K. Ying, W. Zaremba, W. Zhan, C. Zhang, B. Zhang, E. Zhang, and S. Zhao Gpt-oss-120b & gpt-oss-20b model card. Note: Preprint at https://arxiv.org/abs/2508.10925 External Links: Document Cited by: Models.. Pugh et al. (2024) S. L. Pugh, C. Chandler, A. S. Cohen, C. Diaz-Asper, B. Elvevåg, and P. W. Foltz Assessing dimensions of thought disorder with large language models: the tradeoff of accuracy and consistency. Psychiatry Research 341, p. 116119. External Links: ISSN 0165-1781, Document, Link Cited by: Introduction. Qwen et al. (2024) Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. Note: Preprint at https://arxiv.org/abs/2412.15115 External Links: Document Cited by: Models.. Ringeval et al. (2019) F. Ringeval, B. Schuller, M. Valstar, N. Cummins, R. Cowie, L. Tavabi, M. Schmitt, S. Alisamir, S. Amiriparian, E. Messner, S. Song, S. Liu, Z. Zhao, A. Mallol-Ragolta, Z. Ren, M. Soleymani, and M. Pantic AVEC 2019 workshop and challenge: state-of-mind, detecting depression with AI, and cross-cultural affect recognition. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, M ’19, p. 3–12. External Links: Document, Link Cited by: Data.. Rosenman et al. (2024) G. Rosenman, T. Hendler, and L. Wolf LLM questionnaire completion for automatic psychiatric assessment. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 403–415. External Links: Document, Link Cited by: Introduction. Sclar et al. (2024) M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr Quantifying language models’ sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Introduction. Shrout and Fleiss (1979) P. E. Shrout and J. L. Fleiss Intraclass correlations: uses in assessing rater reliability.. Psychological Bulletin 86 (2), p. 420–428. External Links: ISSN 0033-2909, Document, Link Cited by: Estimands.. Silberzahn et al. (2018) R. Silberzahn, E. L. Uhlmann, D. P. Martin, P. Anselmi, F. Aust, E. Awtrey, Š. Bahník, F. Bai, C. Bannard, E. Bonnier, R. Carlsson, F. Cheung, G. Christensen, R. Clay, M. A. Craig, A. Dalla Rosa, L. Dam, M. H. Evans, I. Flores Cervantes, N. Fong, M. Gamez-Djokic, A. Glenz, S. Gordon-McKeon, T. J. Heaton, K. Hederos, M. Heene, A. J. Hofelich Mohr, F. Högden, K. Hui, M. Johannesson, J. Kalodimos, E. Kaszubowski, D. M. Kennedy, R. Lei, T. A. Lindsay, S. Liverani, C. R. Madan, D. Molden, E. Molleman, R. D. Morey, L. B. Mulder, B. R. Nijstad, N. G. Pope, B. Pope, J. M. Prenoveau, F. Rink, E. Robusto, H. Roderique, A. Sandberg, E. Schlüter, F. D. Schönbrodt, M. F. Sherman, S. A. Sommer, K. Sotak, S. Spain, C. Spörlein, T. Stafford, L. Stefanutti, S. Tauber, J. Ullrich, M. Vianello, E.-J. Wagenmakers, M. Witkowiak, S. Yoon, and B. A. Nosek Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science 1 (3), p. 337–356. External Links: ISSN 2515-2467, Document, Link Cited by: Discussion. Simonsohn et al. (2020) U. Simonsohn, J. P. Simmons, and L. D. Nelson Specification curve analysis. Nature Human Behaviour 4 (11), p. 1208–1214. External Links: ISSN 2397-3374, Document, Link Cited by: Introduction. Spitzer and Fleiss (1974) R. L. Spitzer and J. L. Fleiss A re-analysis of the reliability of psychiatric diagnosis. British Journal of Psychiatry 125 (587), p. 341–347. External Links: ISSN 1472-1465, Document, Link Cited by: Introduction, Discussion. Spitzer (1978) R. L. Spitzer Research diagnostic criteria: rationale and reliability. Archives of General Psychiatry 35 (6), p. 773. External Links: ISSN 0003-990X, Document, Link Cited by: Introduction, Discussion. Stade et al. (2024) E. C. Stade, S. W. Stirman, L. H. Ungar, C. L. Boland, H. A. Schwartz, D. B. Yaden, J. Sedoc, R. J. DeRubeis, R. Willer, and J. C. Eichstaedt Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation. npj Mental Health Research 3 (1), p. 12. External Links: ISSN 2731-4251, Document, Link Cited by: Introduction. Steegen et al. (2016) S. Steegen, F. Tuerlinckx, A. Gelman, and W. Vanpaemel Increasing transparency through a multiverse analysis. Perspectives on Psychological Science 11 (5), p. 702–712. External Links: ISSN 1745-6924, Document, Link Cited by: Introduction, Discussion. Steiger (1980) J. H. Steiger Tests for comparing elements of a correlation matrix.. Psychological Bulletin 87 (2), p. 245–251. External Links: ISSN 0033-2909, Document, Link Cited by: §4, Estimands.. Teferra et al. (2025) B. G. Teferra, A. Perivolaris, W. Hsiang, C. K. Sidharta, A. Rueda, K. Parkington, Y. Wu, A. Soni, R. Samavi, R. Jetly, Y. Zhang, B. Cao, S. Rambhatla, S. Krishnan, and V. Bhat Leveraging large language models for automated depression screening. PLOS Digital Health 4 (7), p. e0000943. External Links: ISSN 2767-3170, Document, Link Cited by: Introduction. Tierney et al. (2024) A. A. Tierney, G. Gayre, B. Hoberman, B. Mattern, M. Ballesca, P. Kipnis, V. Liu, and K. Lee Ambient artificial intelligence scribes to alleviate the burden of clinical documentation. NEJM Catalyst 5 (3). External Links: ISSN 2642-0007, Document, Link Cited by: Discussion. Tjuatja et al. (2024) L. Tjuatja, V. Chen, T. Wu, A. Talwalkwar, and G. Neubig Do LLMs exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics 12, p. 1011–1026. External Links: ISSN 2307-387X, Document, Link Cited by: Introduction. Valstar et al. (2016) M. Valstar, J. Gratch, B. Schuller, F. Ringeval, D. Lalanne, M. Torres Torres, S. Scherer, G. Stratou, R. Cowie, and M. Pantic AVEC 2016: depression, mood, and emotion recognition workshop and challenge. In Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, M ’16, p. 3–10. External Links: Document, Link Cited by: Data.. Van Calster et al. (2019) B. Van Calster, D. J. McLernon, M. van Smeden, L. Wynants, E. W. Steyerberg, and On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative Calibration: the achilles heel of predictive analytics. BMC Medicine 17 (1), p. 230. External Links: ISSN 1741-7015, Document, Link Cited by: Introduction, Discussion. Wolf et al. (2020) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p. 38–45. External Links: Document, Link Cited by: Models.. Xu and Hu (2026) J. Xu and X. Hu Language shapes mental health evaluations in large language models. Note: Preprint at https://arxiv.org/abs/2603.06910 External Links: Document Cited by: Introduction. Zheng et al. (2024) C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Introduction. Zhu et al. (2026) T. Zhu, A. Tashevski, M. Taquet, M. Azis, T. Jani, M. R. Broome, T. Kabir, A. Minichino, G. K. Murray, M. M. Nour, I. Singh, P. Fusar-Poli, A. Nevado-Holgado, P. McGuire, and D. Oliver Evaluating large language models for assessment of psychosis risk. npj Digital Medicine 9 (1), p. 554. External Links: ISSN 2398-6352, Document, Link Cited by: Introduction, Discussion.