Paper deep dive
Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity
Xinkai Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:A large language model (LLM) can rate depression severity directly from a social media post or mark which clinical criteria the post shows and let code turn the count into a label. The latter is easier to audit because a clinician can check each marked criterion. We compare these approaches on two Reddit corpora using three LLMs (from 9B to frontier scale) and two questionnaires (PHQ-9, BDI-II), and measure agreement with quadratic weighted kappa. For the two frontier models, criteria extraction scores above chain-of-thought on one corpus only when its decision thresholds are fitted on labeled data. Neither model's gain is significant, with or without recalibrating chain-of-thought on the same labels. With thresholds fixed a priori from PHQ-9's criteria, extraction shows no gain on either corpus, even where models mark over two criteria per post. The 9B model behaves differently on a corpus from depression communities. It labels most posts severe, whether prompted directly or with chain-of-thought, while the a priori rule beats both without labels. After chain-of-thought is recalibrated on the same labels, no significant gap remains, consistent with a calibration effect. Yet higher ordinal agreement does not ensure better detection of severe cases. PHQ-9 criteria extraction misses most severe posts, and moving from direct prompting to chain-of-thought and then to extraction increases misses in nearly all comparisons. On the primary corpus, a relabeled stress dataset, a model using that dataset's own features, including word counts from the text, is not significantly different from frontier criteria extraction under the a priori rule.
Tags
Links
- Source: https://arxiv.org/abs/2609.39049v1
- Canonical: https://arxiv.org/abs/2609.39049v1
Trouble viewing inline? Open PDF directly →
Full Text
62,272 characters extracted from source content.
Expand or collapse full text
Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression SeverityThanks: Extended version of a paper accepted at the 2nd Workshop on Mental Health Disorder Detection on Social Media (MHSM 2026), held with IEEE ICDM 2026. It restores material cut from the six-page proceedings version and adds Appendices A–G; numbers that appear in both versions are identical. Xinkai Chen Affiliation: Independent Researcher xinkaichen1997@gmail.com Abstract A large language model (LLM) can rate depression severity directly from a social media post or mark which clinical criteria the post shows and let code turn the count into a label. The latter is easier to audit because a clinician can check each marked criterion. We compare these approaches on two Reddit corpora using three LLMs (from 9B to frontier scale) and two questionnaires (PHQ-9, BDI-I), and measure agreement with quadratic weighted kappa (κw _w). For the two frontier models, criteria extraction scores above chain-of-thought on one corpus only when its decision thresholds are fitted on labeled data. Neither model’s gain is significant, with or without recalibrating chain-of-thought on the same labels. With thresholds fixed a priori from PHQ-9’s criteria, extraction shows no gain on either corpus, even where models mark over two criteria per post. The 9B model behaves differently on a corpus from depression communities. It labels most posts severe, whether prompted directly or with chain-of-thought, while the a priori rule beats both without labels. After chain-of-thought is recalibrated on the same labels, no significant gap remains, consistent with a calibration effect. Yet higher ordinal agreement does not ensure better detection of severe cases. PHQ-9 criteria extraction misses most severe posts, and moving from direct prompting to chain-of-thought and then to extraction increases misses in nearly all comparisons. On the primary corpus, a relabeled stress dataset, a model using that dataset’s own features, including word counts from the text, is not significantly different from frontier criteria extraction under the a priori rule. Index Terms: depression severity, ordinal classification, large language models, chain-of-thought, clinical criteria I Introduction Depression severity in social media posts is usually predicted as an ordinal label, often with large language models (LLMs). One appealing design is to split the task. DSM-5, the standard psychiatric manual, lists nine symptoms of a major depressive episode (its criterion A), and the PHQ-9 questionnaire [1] asks about exactly these nine. An LLM marks which criteria a post shows, and code turns the count into a label that a clinician can check. We test this design on two Reddit corpora, DepSeverity [2] (four severity labels) and DepSign [3, 4] (three labels), and measure agreement with the gold labels using quadratic weighted kappa (κw _w) [5], which is 0 at chance and penalizes an error more the further it is from the true level. We test three models: a small local one (Qwen3.5-9B), an open-weights frontier model (DeepSeek-V4.1-Flash) and a closed one (Claude-Sonnet-5). Whatever its accuracy, criteria extraction has one practical advantage: it is far more stable than chain-of-thought across identical runs, so its outputs are easier to audit. We make four contributions. (1) An unfair comparison, measured. On DepSeverity, criteria extraction with fitted thresholds scores above chain-of-thought by up to +0.149κw+0.149\, _w. With thresholds fixed a priori, all three gains reverse sign (at worst −0.081-0.081). A pipeline that fits thresholds on labels should therefore not be compared directly with a zero-shot prompt that sees no labels: the difference mostly shows the value of the labels, not of the pipeline. (2) A weak model behaves differently. On DepSeverity, fitted thresholds add +0.167κw+0.167\, _w for the 9B model (against +0.103+0.103 and +0.104+0.104 for the frontier models), and +0.109+0.109 remains when chain-of-thought gets the same labels (significant before correction). On DepSign the 9B model labels most posts severe, directly and with chain-of-thought, and the a priori rule beats both without labels (+0.093+0.093 and +0.143+0.143, respectively), and still beats chain-of-thought when it counts BDI-I’s 21 items (+0.108+0.108). This is consistent with a calibration effect: the fixed rule reduces the model’s over-prediction of severe, and recalibrating chain-of-thought with the same 600 labels leaves no significant gap. With BDI-I’s items, the 9B model also beats chain-of-thought without labels on DepSeverity (+0.130+0.130), though not significantly once chain-of-thought is recalibrated (+0.091+0.091). (3) Better ordinal agreement, more missed severe posts. From direct prompting to chain-of-thought, and from chain-of-thought to criteria extraction, the share of gold-severe posts predicted below severe (false negatives) rises in 11 of 12 comparisons, significantly in 8 before correction. severe is rare, 7–8% of posts, so an approach that rarely predicts it scores higher on κw _w and catches fewer severe posts. The approach with the lowest κw _w catches that class best, and criteria extraction’s gain on the 9B model comes with 47 of 50 severe posts missed. (4) Much of one benchmark can be predicted from its source dataset’s own annotations. DepSeverity is a relabeled copy of Dreaddit [6], a stress dataset, and none of its source communities is about depression. As a check on the benchmark itself, a model that uses only Dreaddit’s own features (source community, stress label, and word-category counts computed from the text) reaches κw=0.404 _w=0.404, not significantly different from frontier criteria extraction with a priori thresholds. Dreaddit’s fields that do not read the post reach 0.333, above every zero-shot direct prompt. All comparisons use paired bootstrap tests on the same posts. I Related Work Criteria-level annotation is well established. The Depressive Disorder Annotation scheme [7] covers the DSM-5 criteria, and PRIMATE [8] labels PHQ-9 criteria in Reddit posts and D2S [9] in tweets, although a mental-health professional who re-annotated PRIMATE found many false positives for anhedonia [10]. We instead evaluate extraction end to end, against severity labels, and study the step that turns criteria into a label. The eRisk tasks fill in BDI-I from a user’s whole posting history [11] and rank sentences by relevance to each of its 21 symptoms [12]; we instead mark the 21 items as present or absent in one post and count them (C4, §IV-A). Fitting thresholds is fair when all compared systems use the same labels. Next to a zero-shot prompt, however, fitted thresholds confound the comparison, and this setup is common: supervised classifiers on LLM embeddings beat zero-shot prompting on severity, leading to the view that LLMs work better as interpreters than as classifiers [13], and guidelines learned from labeled examples beat zero-shot prompting on BDI-I items [14]. We know of no work that measures this effect. Chain-of-thought prompting [15] is our control: the model reasons first but still gives the label itself. On DepSeverity, Cognitive-Mental-LLM [16] finds chain-of-thought below direct prompting in accuracy, the opposite of our result; our direct prompts over-predict severity, which chain-of-thought corrects (§V-A), so the direction may depend on how well calibrated the direct prompt is. I Corpora I-A DepSeverity and its origin DepSeverity [2] contains 3,553 Reddit posts labeled minimum, mild, moderate or severe, with no stated license or terms of use. DepSeverity reuses Dreaddit’s posts, as noted by Mental-LLM [17], which evaluates on both, and by a later study [18]. We verify it: after normalizing whitespace, all 3,530 de-duplicated posts appear word for word in Dreaddit, so DepSeverity is a relabeling, not an extension. A model trained or tuned on Dreaddit has therefore already seen every DepSeverity test post, under a different label, and its DepSeverity results may reflect that exposure. Dreaddit also records each post’s subreddit, which DepSeverity drops and, to our knowledge, no earlier work has recovered. The communities are ptsd, relationships, anxiety, domesticviolence, assistance, survivorsofabuse, homeless, almosthomeless, stress and food_pantry. None is a depression community. Consistent with this, only 3 of the 56 gold-severe test posts contain explicit self-harm phrases. Dreaddit’s own split is stratified by stress and leaves only 10 severe and 9 mild test posts, so we make our own stratified 80/20 split with a fixed seed. The annotation documents are not available, so we separate what is known from what we infer. The four label names are the severity bands of the BDI-I questionnaire [19] (PHQ-9 has five), and [2] reports annotating with Beck’s questionnaire and the Depressive Disorder Annotation scheme [7]. We infer that only the band names come from BDI-I, while the symptom-level scheme uses the nine DSM-5 criteria, which a nine-criterion extractor can capture. If that inference is wrong, and the annotators judged severity with BDI-I’s 21 items, PHQ-9 is not the right instrument; condition C4 (§IV-A) therefore repeats the extraction over those 21 items. We drop 4 posts in the two duplicate groups whose copies carry different gold labels (one minimum/severe pair on identical text) and merge 19 other duplicates that would otherwise span train and test, leaving 3,530 posts. I-B DepSign DepSign [3] comes from mental-health subreddits, including r/depression and r/MentalHealth, is labeled not depression, moderate or severe, and has no stated license. Two domain experts labeled the posts against written guidelines, with Cohen’s κ=0.686κ=0.686 between them [3]. We subsample its official splits to DepSeverity’s sizes (706 test, 600 train; Table I), so the corpora differ in class balance (73% minimum against 67% moderate) but not in test size. 1,502 of DepSign’s 16,632 posts (9.0%) contain a [removed] or [deleted] tag and have a median of 11 words, but were still labeled: 57% as not depression, against 25% of the other posts, so a model that learns “little text → not depression” gains accuracy for free. DepSeverity has no such posts. We keep them, so the split stays official, and test their effect in §V-D. Table I: Test splits. Both corpora are evaluated at n=706n=706. DepSeverity DepSign lowest class 513 minimum 184 not dep. middle class(es) 58 mild / 79 mod. 472 mod. highest class 56 severe 50 severe majority acc. 0.727 0.669 majority κw _w 0.000 0.000 median words 80 104 IV Method IV-A Conditions C1 (direct). The model returns only a severity label. C2 (chain-of-thought). The model first reasons about which depressive symptoms the post shows, then gives the label itself. C3 (structured, PHQ-9). The model returns JSON that marks each of the nine PHQ-9 criteria as present, absent or unclear (kept separate from absent), with a word-for-word quote from the post for each present. The model never sees the severity labels and never outputs one; code assigns the label. C4 (structured, BDI-I). Repeats C3 over the 21 items of BDI-I, the questionnaire whose band names DepSeverity uses; these items only partly overlap with the DSM-5 criteria. IV-B From criteria to a label Let s be the number of criteria (C3) or items (C4) marked present. We turn s into one of k ordinal labels with k−1k-1 thresholds, set in two ways. Fitted. We search all threshold settings and keep the one with the highest κw _w on a 600-post fitting split taken only from training data. We then freeze these thresholds for the test set. A priori. The thresholds are label-free: we fix them without looking at any label. For the nine PHQ-9 criteria, DSM-5 requires at least five symptoms for a major depressive episode (one must be depressed mood or loss of interest; our rule uses the count only), so the top threshold is 4.54.5. A post with no symptom gets the lowest label, so the first threshold is 0.50.5. DSM-5 says nothing about counts of 11–44: we split them evenly for DepSeverity’s four labels, [0.5,2.5,4.5][0.5,2.5,4.5], and add no middle threshold for DepSign’s three, [0.5,4.5][0.5,4.5]. The 21 BDI-I items have no exact anchor, so we report C4 under fitting and both imperfect a priori rules: the DSM-5 thresholds above, and BDI-I’s bands rescaled from 0–63 to our 0–21 count. Stricter count thresholds do not help. We also test [1.5,4.5,6.5][1.5,4.5,6.5], requiring at least seven symptoms for severe. This is our operationalization, not an official DSM-5 severity rule; κw _w falls to 0.231/0.223 (DeepSeek-V4.1-Flash/Claude-Sonnet-5) on DepSeverity. On DepSign, where these four bands collapse to three in more than one way, it runs from 0.045/0.039 to 0.276/0.253, the latter above C2 but not significantly. No gold-severe post survives any of them: these thresholds assume an interview, and a post mentions far fewer symptoms. The difference between the two regimes is the key to the paper: fitted thresholds use labeled data, so C3 is no longer zero-shot while C1 and C2 still are, which confounds any direct comparison. absent was used in only 0.06–0.54% of judgments. Of the three statuses, the score counts only present. IV-C Models and protocol We use Qwen3.5-9B (run locally, open weights), DeepSeek-V4.1-Flash (open weights) and Claude-Sonnet-5, with extended thinking turned off, because hidden reasoning in C1 would make the C1/C2 comparison meaningless. We verified this for each vendor and count reasoning leaks on every call. Temperature is 0 where the API allows it, though no hosted model is deterministic even then, and Claude-Sonnet-5 accepts no sampling parameters at all. We therefore report bootstrap intervals over test posts, paired across conditions, and measure run-to-run variation in §V-F. Every model response is cached and logged. V Results Table I: Test-set results, n=706n=706 per corpus. κw _w = quadratic weighted kappa; F1M = macro-F1; sev↓ = fraction of severe posts given a lower label (1−1-recall on that class). C3 rows show fitted / a priori (DSM-5) thresholds, C4 rows fitted / DSM-5 / BDI-I bands; other columns are fitted. Bold: highest κw _w per model and corpus, fewest missed severe posts per corpus. Model Cond. κw _w MAE Acc F1M sev↓ DepSeverity (4 classes, 56 severe) Qwen3.5-9B C1 0.288 0.948 0.445 0.335 29/56 C2 0.340 0.877 0.456 0.327 34/56 C3 0.489 / 0.322 0.465 0.728 0.367 43/56 C4 0.495/0.470/0.057 0.517 0.643 0.420 40/56 DeepSeek C1 0.219 1.280 0.329 0.251 16/56 V4.1-Flash C2 0.504 0.487 0.659 0.408 42/56 C3 0.526 / 0.423 0.452 0.670 0.362 49/56 C4 0.469/0.479/0.079 0.589 0.592 0.386 41/56 Claude C1 0.299 0.963 0.354 0.306 33/56 Sonnet-5 C2 0.462 0.551 0.596 0.390 41/56 C3 0.499 / 0.395 0.467 0.664 0.369 49/56 C4 0.460/0.466/0.110 0.616 0.508 0.347 48/56 DepSign (3 classes, 50 severe) Qwen3.5-9B C1 0.162 0.839 0.271 0.271 13/50 C2 0.112 0.946 0.198 0.201 10/50 C3 0.264 / 0.255 0.333 0.674 0.433 48/50 C4 0.235/0.220/0.135 0.448 0.585 0.433 38/50 DeepSeek C1 0.109 0.956 0.184 0.183 9/50 V4.1-Flash C2 0.228 0.737 0.344 0.341 16/50 C3 0.203 / 0.207 0.347 0.660 0.391 49/50 C4 0.218/0.178/0.119 0.399 0.620 0.429 41/50 Claude C1 0.151 0.796 0.288 0.236 17/50 Sonnet-5 C2 0.220 0.705 0.364 0.339 19/50 C3 0.192 / 0.207 0.499 0.535 0.384 36/50 C4 0.179/0.158/0.152 0.353 0.659 0.400 46/50 V-A Positive control C2 significantly beats C1 in five of six model–corpus pairs (Fig. 1, black), so our setup can detect differences when they exist; the exception is the 9B model on DepSign, where chain-of-thought is significantly worse (−0.050-0.050). Asked directly, all models over-predict severity: they rate 47–77% of posts above their gold label. Chain-of-thought cuts this share sharply for the frontier models (to 18–24% on DepSeverity and 54% on DepSign), hardly changes it for the 9B model on DepSeverity (47% → 45%) and raises it on DepSign (67% → 77%). Chain-of-thought’s gains thus track reduced over-prediction of severity rather than model size: the worst-calibrated model gains most, and the 9B model, whose calibration does not improve, gains least and loses on DepSign. V-B The unfair comparison Table I gives all results. With fitted thresholds, C3 scores above C1 and C2 in four of six model–corpus pairs and significantly beats C2 for the 9B model on both corpora (+0.149+0.149 and +0.152+0.152). With a priori thresholds the results split (Fig. 1). Except for the 9B model on DepSign, no C3 pair shows a significant gain, and DeepSeek-V4.1-Flash on DepSeverity shows a loss of −0.081-0.081 (significant before correction). Across the nine a priori comparisons with C2 for the frontier models and for the 9B model’s C3 on DepSeverity (five for C3, four for C4 under the same DSM-5 rule; Table I), none shows a significant κw _w advantage for criteria extraction; two (−0.081-0.081 for C3, −0.062-0.062 for C4) are significantly worse before correction and a third (−0.050-0.050, C4) is on the boundary, and all three are sensitive to generation variance (§V-F). The 9B model on DepSign keeps nearly all of its gain without labels (+0.143+0.143), and C4 under the same rule also beats C2 there (+0.108+0.108, significant before correction). With C4 the 9B model also gains without labels on DepSeverity (+0.130+0.130; §V-D). On DepSeverity, the three fitted thresholds add +0.167+0.167 (Qwen3.5-9B), +0.103+0.103 (DeepSeek-V4.1-Flash) and +0.104+0.104 (Claude-Sonnet-5) in κw _w over the a priori ones; on DepSign they add almost nothing (+0.009+0.009, −0.004-0.004, −0.015-0.015). For the weakest model, the fitted thresholds become [0.5,1,1.5][0.5,1,1.5], meaning “no symptoms → minimum, any symptom → higher”, for a model that finds on average 0.26 of 9 symptoms. The thresholds do the work. The 9B exception is consistent with calibration. Asked directly or with chain-of-thought, the 9B model labels 444 and 526 of 706 DepSign posts severe (50 in gold). The a priori rule, which needs five symptoms, labels 38 posts severe and also significantly beats direct prediction (+0.093+0.093). The rule reduces over-prediction of severe, but it still misses 47 of 50 severe posts (§V-E). The effect survives Holm correction and removing the [removed] posts (§V-D), but after we recalibrate chain-of-thought with the same 600 labels, no significant gap remains (§V-C). Fig. 1: Paired bootstrap Δκw _w on the same posts (4000 resamples); solid markers exclude zero at 95%, faded hollow ones do not. C2 −- C1 uses no thresholds. C4 is in Table I. V-C Giving C1 and C2 the same supervision The stronger fairness test gives C1 and C2 the same access to labels that C3’s fitted thresholds get: we learn a monotone relabeling of each condition’s predicted label on the same 600-post split and freeze it. Such a map has (2k−1k) 2k-1k forms on k labels (35 for k=4k=4, 10 for k=3k=3), fewer than C3’s threshold options, so the test favors C3. First, the frontier models’ chain-of-thought barely improves: its best map leaves the labels unchanged in three of four cases, and C3 with fitted thresholds still does not significantly beat it in any case. These predictions are already largely calibrated: C2 does not need the labels, but C3 does. Second, the 9B model on DepSeverity is the exception. Its miscalibrated C2 improves (0.340→0.3790.340→ 0.379), which removes a quarter of the +0.149+0.149 gap, but +0.109+0.109 remains (significant before correction). So for this model the unequal labels do not explain the whole effect. If we first reduce C3’s score to four levels without labels and then apply a monotone map, which searches exactly the same 35 forms, the +0.109+0.109 stays: the remaining gain comes from the extraction itself, not from C3’s finer score. Without labels, the result depends on the questionnaire. With a priori thresholds C3 falls to 0.322, below recalibrated C2, so PHQ-9 extraction does not win; C4 under the same rule reaches 0.470, +0.091+0.091 above recalibrated C2, just short of significance ([−0.001,+0.181][-0.001,+0.181]; Appendix C). Third, C1 improves substantially (0.219→0.3300.219→ 0.330 and 0.109→0.2460.109→ 0.246 for DeepSeek-V4.1-Flash), so its weakness is calibration. On DepSeverity, chain-of-thought with no labels (0.504) beats direct prediction with all 600 labels (0.330): reasoning is worth more than the labels, and criteria extraction adds nothing measurable on top. Fourth, on DepSign the 9B model’s gap is calibration. Its chain-of-thought gains more from labels than chain-of-thought does in any other case (0.112→0.2190.112→ 0.219), through a map that never predicts severe, and C3’s lead with fitted thresholds shrinks to a non-significant +0.045+0.045. Recalibrated direct prediction gains less (0.162→0.1850.162→ 0.185) and still trails C3 by +0.078+0.078 (significant before correction). So labels close the gap through chain-of-thought but not through direct prediction. V-D Other explanations DepSeverity posts might show too few symptoms to extract, given their origin. On DepSign, however, the frontier models extract about five times more symptoms (mean 2.31 and 2.29 against 0.45 and 0.46) and the 9B model seven times more (1.86 against 0.26), and the frontier models still show no advantage. Extraction is faithful: 97.7–100% of the frontier models’ quotes appear word for word in the post, and 92.4% and 80.9% of the 9B model’s (90.4% and 81.9% in its C4 runs). Excluding the 53 [removed] test posts lowers every DepSign score, C3 most (C1 by 0.019–0.032, C2 by 0.026–0.045, C3 by 0.053–0.069). C3 fitted −- C2 then only moves to −0.051-0.051 for both frontier models, still not significant, and the 9B model’s label-free gain remains significant (+0.115+0.115). A longer questionnaire helps, but beats C2 only for the 9B model. BDI-I’s 21 items raise the mean count per post, from 0.26–0.46 to 0.72–1.25 on DepSeverity and from 1.86–2.31 to 3.31–4.22 on DepSign, where the maximum rises from 8 to 15. For the frontier models on DepSeverity, C4 under the DSM-5 rule improves on C3 (0.479 and 0.466 against 0.423 and 0.395) but only reaches C2, and on neither corpus does C4 beat C2. For the 9B model it does: under the DSM-5 rule, C4 beats C2 by +0.130+0.130 on DepSeverity and +0.108+0.108 on DepSign, and C1 by +0.182+0.182 and +0.058+0.058, all significant before correction. On DepSeverity this reverses the C3 result, where the same rule stays below C2. One reason is scale: the DSM-5 thresholds were set for nine criteria, and a 21-item count reaches them more easily, so they sit close to what fitting would select. Fitting adds only +0.025+0.025 to the 9B model’s C4, against +0.167+0.167 to its C3 (Appendix B). Under the second a priori rule, BDI-I’s own bands rescaled to our count, C4 is significantly worse than C2 for both frontier models on both corpora and for the 9B model on DepSeverity (down to −0.425-0.425; Table I), because binary extraction finds too few items to reach them; for the 9B model on DepSign the difference is +0.023+0.023, not significant. Prompt wording does not explain the result. We reran C3 on DepSeverity with DeepSeek-V4.1-Flash using two other prompts (2,612 calls; Table X). A permissive prompt (a lower bar for present) finds 60% more symptoms (0.45 → 0.72 per post) but scores slightly worse (fitted κw _w 0.526→0.5070.526→ 0.507). A symmetric prompt (quotes also required for absent) scores 0.5550.555 but still does not beat C2 with a priori thresholds (−0.071-0.071, not significant), and raises the use of absent only from 0.30% to 0.35%. Neither beats chain-of-thought under either rule, so the rare use of absent (§IV-B) comes from the posts, not our wording. A better aggregation rule does not help: an ordinal model learned on the same nine criterion labels, with far more free parameters, reaches only 0.1560.156 and 0.1660.166, significantly below the simple count. Weighting its classes by inverse frequency raises it to 0.421–0.454 on DepSeverity, still below the count with fitted thresholds, but it then misses only 14–26 of 56 severe posts, against 43–49 for the count (Appendix D). V-E Ordinal agreement versus missed severe posts The last column of Table I tells a different story. On DepSign, DeepSeek-V4.1-Flash under C1 has the lowest κw _w in the table but misses only 9 of 50 severe posts; under C3 its κw _w nearly doubles (0.109 → 0.203) but it misses 49 of 50. Across all twelve comparisons, C1 to C2 and C2 to C3, the share of missed severe posts rises in eleven, significantly in eight (95% intervals, not corrected). The one decrease, not significant, is chain-of-thought for Qwen3.5-9B on DepSign, which labels most posts severe. Higher ordinal agreement often means more missed severe posts. BDI-I extraction misses fewer: under the DSM-5 rule on DepSign, C4 misses 30 and 26 of 50 severe posts against 42 for C3, but still more than chain-of-thought (16 and 19), and its κw _w is significantly lower (Appendices A and D). For the 9B model, C4 misses 29 of 50 against 47 for C3 and 10 for chain-of-thought, which labels most posts severe; on DepSeverity it misses 49 of 56, against 56 for C3 and 34 for chain-of-thought. Macro-F1, which ignores ordinal distance, ranks the conditions differently (Table I; paired bootstrap, before correction). On DepSeverity it agrees with κw _w. On DepSign, C3 beats C2 for the frontier models with both kinds of threshold (+0.044+0.044 to +0.071+0.071; the a priori gains remain without the [removed] posts), while κw _w shows no difference. The whole gain comes from the majority moderate class, which chain-of-thought often raises to severe (DeepSeek-V4.1-Flash, C2 vs. C3 a priori: F1 0.39 vs. 0.74), and C3 is worse on both smaller classes. For the 9B model the two metrics agree (+0.219+0.219 with the label-free rule). We report κw _w as the main metric because the labels are ordinal and, unlike accuracy and MAE, κw _w corrects for chance: no LLM condition clearly beats majority-class accuracy (0.727 on DepSeverity). V-F Run-to-run stability Repeating C2 and C3 on DepSeverity changes κw _w by 0.0200.020 and 0.0080.008 (Claude-Sonnet-5) and 0.0220.022 and 0.0200.020 (DeepSeek-V4.1-Flash). Our intervals ignore this run-to-run noise, so we widened each hosted model’s intervals, treating its larger change as single-run noise (the local model is deterministic). All six C2 −- C1 effects, the 9B model’s gains and the fitted margins over Dreaddit’s features (§V-H) stay significant, but the three DSM-5-rule losses of §V-B (−0.081-0.081 for DeepSeek-V4.1-Flash on DepSeverity and both C4 losses on DepSign) do not, so they are not evidence of real harm. Structure is clearly better in one respect: between identical runs, C2 changes 105 and 132 of 706 labels, and C3 only 21 and 29. V-G Can a human reader recover the labels? We do not know how reliable the gold labels are. As a test, the author, with over five years of experience working with clinicians, annotated a stratified sample from each test split (100 and 102 posts) without seeing gold labels, model outputs or any results, using a rubric based on BDI-I’s bands and the nine DSM-5 criteria. On DepSign, the annotator’s κw _w is 0.4160.416 [0.248,0.567][0.248,0.567], against 0.395 and 0.368 for the models’ C2 on the same posts. The paired difference, −0.021-0.021 [−0.174,+0.130][-0.174,+0.130], is too uncertain to show that human and model are equivalent, but the annotator’s own interval is well above zero: a reader following clinical criteria recovers DepSign’s labels, so its low κw _w reflects a hard task with unbalanced classes, not unrecoverable labels, and its results in §V-D are not results on noise. DepSeverity behaves differently. The annotator reaches 0.1990.199 where the best model condition reaches 0.4850.485, a paired gap of +0.285+0.285 [+0.106,+0.450][+0.106,+0.450] in the model’s favor, and read 13 of 25 gold-severe posts as showing no depressive content. Models can reproduce these labels, but a reader who follows clinical criteria cannot, which is hard to explain if the labels follow clinical criteria and fits the idea that models learn patterns of the annotation process. It also fits two other explanations, noise in the gold labels and a mismatch between our rubric and the annotators’ own; with one annotator and no access to the original guidelines, we cannot tell these apart. The difference does not come from the number of labels: merging DepSeverity into three labels raises the annotator to at most 0.2440.244 and leaves the models unchanged, and rank agreement with gold, which does not depend on calibration, is ρ=0.251ρ=0.251 against 0.4350.435 on DepSign. Limits: our rubric rather than the original annotators’, one non-clinician annotator, and stratified samples not comparable to full-split results. V-H What predicts the labels For DepSeverity, the recovered origin (§I-A) shows what the labels follow instead. The same ordinal method [20] fitted to Dreaddit-provided fields, without symptom annotations, recovers much of the label: the source community alone reaches κw=0.196 _w=0.196, Dreaddit’s stress label 0.2590.259, and the fields that do not read the post (community, stress label, karma, votes, comments) 0.3330.333. Adding text-derived features (length, Dreaddit’s sentiment score, LIWC word-category counts) gives 0.4040.404. A TF-IDF → ordinal logistic model trained on the 2,824 labeled training posts reaches 0.3740.374, above zero-shot direct prompting with every model (0.219–0.299), though it sees labels that the prompts do not. The communities anxiety and ptsd, neither about depression, provide 34 of the 56 gold-severe test posts. On the same 706 posts, compared with the full model, C3 with a priori thresholds is not significantly different for either model (+0.019+0.019 for DeepSeek-V4.1-Flash, −0.009-0.009 for Claude-Sonnet-5); against the non-text fields alone it is ahead by +0.090+0.090 (significant) and +0.063+0.063 (not). With fitted thresholds it is significantly ahead of the full model (+0.122+0.122 and +0.095+0.095), which by our own standard (§V-C) is the fair comparison. So Dreaddit’s features are a lower bound, but a high one: without fitted thresholds, criteria extraction is not significantly better than the full set, while chain-of-thought scores above it without any labels. This makes the result of §V-B expected: the posts contain little criterion-based signal, as the human test also showed. It is also a warning about the corpus as a benchmark: a system can score well here without modeling depression severity. VI Discussion Criteria extraction remains useful for auditing: besides the quote check above, all but seven of the 15,672 structured calls on the fit and test splits return the full inventory, and code recovers the statuses from six malformed JSON responses. The seven exceptions are 9B C4 responses that omit one item (sleep_change); we count it as not present, and dropping those responses instead changes no reported κw _w by more than 0.0010.001. The tested counting rules show no significant ordinal-agreement advantage over chain-of-thought for frontier models. The results suggest a mismatch between symptom counts and the gold labels: only 3 of 56 gold-severe DepSeverity posts mention self-harm, and the labels partly follow where posts were collected (§V-H). The a priori thresholds, equal-supervision test, BDI-I items, prompt variants, learned aggregator and human probe support this interpretation (§V-B–§V-G), but do not rule out better extraction or aggregation. In our experiments, structure helps without labels only for the 9B model: on DepSign with either questionnaire, consistent with correcting its over-prediction of severe, and on DepSeverity with BDI-I’s items, where the gain over recalibrated chain-of-thought falls just short of significance. Even then it misses most severe posts. The lesson applies beyond mental health: a pipeline with fitted thresholds and a zero-shot baseline are not comparable systems. Reporting the label-free version next to the fitted one costs nothing, because both use the same extractions. Reporting direct prediction next to chain-of-thought is just as cheap: for the 9B model on DepSign, comparing only against chain-of-thought would show +0.143+0.143, while direct prediction gives +0.093+0.093. Criteria extraction is also far more stable across identical runs than chain-of-thought (§V-F), which makes its outputs easier to audit. VII Code and data The repository11 1 https://github.com/xinkaichen97/depseverity-artifact contains the pipeline, exact prompts, result tables, worked examples and one cleaned record per post per condition, from which every number except the [removed]-post analysis can be recomputed without model calls. It contains no post text, since neither corpus has a stated license. VIII Ethics This work uses public Reddit posts about depression, stress, abuse and self-harm. We did not identify or contact authors or link people across datasets, we report only aggregate results, and we share neither corpus. Nothing here is fit for clinical use: the systems with the best ordinal agreement miss the most severe cases (§V-E), so papers should report the direction of errors, not only their size, for example per-class recall (Appendix D) or expected cost under asymmetric error costs. IX Limitations and Conclusion Our primary corpus contains no depression community, which limits what it says about depression severity; the second is drawn from depression communities but subsampled to 706 test posts, which lowers power for its 50 severe posts. Our confirmatory family is the eighteen comparisons of Fig. 1. With Holm correction at α=0.05α=0.05, the four frontier C2 −- C1 effects, the 9B model’s +0.149+0.149 on DepSeverity and all three of its DepSign effects (−0.050-0.050, +0.152+0.152, +0.143+0.143) remain significant; its DepSeverity C2 −- C1 gain (+0.052+0.052) and the a priori loss of −0.081-0.081 are significant only before correction. For the frontier models we find no significant advantage for criteria extraction, before or after correction. Everything else is exploratory and uncorrected: C4, comparisons of C3 with C1, the equal-supervision test, the missed-severe counts, macro-F1, the Dreaddit-feature comparisons, the prompt variants and the human probe. The conditions also differ in output format and length: C2 writes 363–614 output tokens per post, C3 152–251 and C4 371–579. C3 writes less than half as much as C2 and C4 about as much, yet neither beats C2 for the frontier models, so length does not track the ranking; but we did not run a controlled ablation, such as chain-of-thought forced through the nine criteria. DepSign reports two annotators at κ=0.686κ=0.686 [3], but DepSeverity’s annotation documents are not available, so we cannot measure its label noise; two of its posts have identical text but different gold labels. Our human probe cannot settle this: it uses one non-clinician annotator and our own rubric, so its DepSeverity result is a warning sign, not evidence about the original annotators. We run each condition once, and the generation-variance check in §V-F rests on one repeat per hosted model, on one corpus. There is no exact a priori threshold for a 21-item count (§IV-B): for the frontier models C4 fails under both imperfect options and under fitting, but a better rule might do better. We score only whether each criterion or item is present, while PHQ-9 scores each criterion 0–3 by frequency and BDI-I scores each item 0–3 by severity. Binary scoring loses the intensity a post can express, which chain-of-thought can still use. We did not test graded extraction; it could narrow the gap to chain-of-thought, and because BDI-I’s bands assume graded items, the C4 result with those bands (§V-D) may partly reflect our binary choice. Finally, because of compute limits we ran only one small open-weights model; the calibration exception may not hold for other small models, and a larger local model (for example 27B to 70B parameters) would show whether it disappears with scale. For frontier models, criteria extraction shows no significant κw _w advantage over chain-of-thought across two corpora, two questionnaires and both threshold regimes. Its apparent gains depend on fitted thresholds; its macro-F1 gains on DepSign come from the majority class. For the 9B model on DepSign, the label-free counting rule wins while direct and chain-of-thought predictions over-escalate to severe, consistent with a calibration correction; with BDI-I’s items it also wins on DepSeverity, though not significantly against recalibrated chain-of-thought. In both cases the gains come with more missed severe cases. Studies of this kind should report an a priori (label-free) threshold variant next to the fitted one, and direct prediction next to chain-of-thought. References [1] K. Kroenke, R. L. Spitzer, and J. B. W. Williams, “The PHQ-9: Validity of a brief depression severity measure,” Journal of General Internal Medicine, vol. 16, no. 9, p. 606–613, 2001. [2] U. Naseem, A. G. Dunn, J. Kim, and M. Khushi, “Early identification of depression severity levels on Reddit using ordinal classification,” in Proc. ACM Web Conf. (W ’22), 2022, p. 2563–2572. [3] K. Sampath and T. Durairaj, “Data set creation and empirical analysis for detecting signs of depression from social media postings,” in Computational Intelligence in Data Science, IFIP AICT vol. 654, Cham: Springer, 2022, p. 136–151; arXiv:2202.03047. [4] K. Sampath, T. Durairaj, B. R. Chakravarthi, and J. M. C, “Findings of the shared task on detecting signs of depression from social media,” in Proc. 2nd Workshop on Language Technology for Equality, Diversity and Inclusion (LT-EDI), 2022, p. 331–338. [5] J. Cohen, “Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit,” Psychological Bulletin, vol. 70, no. 4, p. 213–220, 1968. [6] E. Turcan and K. McKeown, “Dreaddit: A Reddit dataset for stress analysis in social media,” in Proc. 10th Int. Workshop Health Text Mining and Information Analysis (LOUHI), 2019, p. 97–107. [7] D. Mowery, C. Bryan, and M. Conway, “Towards developing an annotation scheme for depressive disorder symptoms: A preliminary study using Twitter data,” in Proc. 2nd Workshop on Computational Linguistics and Clinical Psychology (CLPsych), 2015, p. 89–98. [8] S. Gupta, A. Agarwal, M. Gaur, K. Roy, V. Narayanan, P. Kumaraguru, and A. Sheth, “Learning to automate follow-up question generation using process knowledge for depression triage on Reddit posts,” in Proc. 8th Workshop on Computational Linguistics and Clinical Psychology (CLPsych), 2022, p. 137–147. [9] S. Yadav, J. Chauhan, J. P. Sain, K. Thirunarayan, A. Sheth, and J. Schumm, “Identifying depressive symptoms from tweets: Figurative language enabled multitask learning framework,” in Proc. 28th Int. Conf. Computational Linguistics (COLING), 2020, p. 696–709. [10] K. Milintsevich, K. Sirts, and G. Dias, “Your model is not predicting depression well and that is why: A case study of PRIMATE dataset,” in Proc. 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych), 2024, p. 166–171. [11] D. E. Losada, F. Crestani, and J. Parapar, “Overview of eRisk 2019: Early risk prediction on the Internet,” in Proc. CLEF 2019, LNCS, Springer, 2019, p. 340–357. [12] J. Parapar, P. Martín-Rodilla, D. E. Losada, and F. Crestani, “Overview of eRisk 2023: Early risk prediction on the Internet,” in Proc. CLEF 2023, LNCS, Springer, 2023, p. 294–315. [13] S. Kim, O. Imieye, and Y. Yin, “Interpretable depression detection from social media text using LLM-derived embeddings,” in Proc. Int. Conf. Artificial Intelligence (ICAI), 2026. [14] E. Bao, A. Perez, D. Otero, and J. Parapar, “Learning evidence of depression symptoms via prompt induction,” in Proc. ACM SIGIR Conf. Research and Development in Information Retrieval (SIGIR), 2026. [15] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, p. 24824–24837. [16] A. Patil and A. K. Gedhu, “Cognitive-Mental-LLM: Evaluating reasoning in large language models for mental health prediction via online text,” arXiv:2503.10095, 2025. [17] X. Xu, B. Yao, Y. Dong, S. Gabriel, H. Yu, J. Hendler, M. Ghassemi, A. K. Dey, and D. Wang, “Mental-LLM: Leveraging large language models for mental health prediction via online text data,” Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 8, no. 1, 2024. [18] Y. Ibrahimov, T. Anwar, and T. Yuan, “DepressionX: Knowledge infused residual attention for explainable depression severity assessment,” arXiv:2501.14985, 2025. [19] A. T. Beck, R. A. Steer, and G. K. Brown, Manual for the Beck Depression Inventory-I. San Antonio, TX: Psychological Corporation, 1996. [20] E. Frank and M. Hall, “A simple approach to ordinal classification,” in Proc. 12th European Conf. Machine Learning (ECML), 2001, p. 145–156. Appendix A All comparisons Table I lists every paired comparison between conditions. The eighteen marked C form the confirmatory family of the Limitations section; the eight marked H survive Holm correction. All other rows are exploratory, including every C4 comparison and every comparison of C3 with C1. Bold values have 95% intervals that exclude zero before any correction. Table I: All paired comparisons (Δκw _w, 95% paired bootstrap interval, 4000 resamples). Rows marked C are the confirmatory family; H = survives Holm correction at α=0.05α=0.05. C4 uses the DSM-5 rule for a priori. Corpus Model Comparison Δκw _w [95% CI] DepSeverity Qwen3.5-9B C2 −- C1 +0.052 [+0.009, +0.098] C DepSeverity Qwen3.5-9B C3 fitted −- C2 +0.149 [+0.072, +0.223] C,H DepSeverity Qwen3.5-9B C3 a priori −- C2 −-0.018 [−-0.089, +0.054] C DepSeverity Qwen3.5-9B C4 fitted −- C2 +0.155 [+0.082, +0.227] DepSeverity Qwen3.5-9B C4 a priori −- C2 +0.130 [+0.050, +0.208] DepSeverity Qwen3.5-9B C3 fitted −- C1 +0.201 [+0.127, +0.276] DepSeverity Qwen3.5-9B C3 a priori −- C1 +0.034 [−-0.037, +0.105] DepSeverity Qwen3.5-9B C4 fitted −- C1 +0.208 [+0.127, +0.282] DepSeverity Qwen3.5-9B C4 a priori −- C1 +0.182 [+0.101, +0.259] DepSeverity DeepSeek C2 −- C1 +0.285 [+0.218, +0.349] C,H DepSeverity DeepSeek C3 fitted −- C2 +0.022 [−-0.039, +0.086] C DepSeverity DeepSeek C3 a priori −- C2 −-0.081 [−-0.147, −-0.010] C DepSeverity DeepSeek C4 fitted −- C2 −-0.035 [−-0.107, +0.039] DepSeverity DeepSeek C4 a priori −- C2 −-0.026 [−-0.100, +0.049] DepSeverity DeepSeek C3 fitted −- C1 +0.307 [+0.237, +0.376] DepSeverity DeepSeek C3 a priori −- C1 +0.204 [+0.139, +0.270] DepSeverity DeepSeek C4 fitted −- C1 +0.250 [+0.183, +0.312] DepSeverity DeepSeek C4 a priori −- C1 +0.259 [+0.192, +0.324] DepSeverity Claude C2 −- C1 +0.163 [+0.110, +0.216] C,H DepSeverity Claude C3 fitted −- C2 +0.037 [−-0.029, +0.103] C DepSeverity Claude C3 a priori −- C2 −-0.067 [−-0.135, +0.005] C DepSeverity Claude C4 fitted −- C2 −-0.002 [−-0.075, +0.072] DepSeverity Claude C4 a priori −- C2 +0.004 [−-0.068, +0.079] DepSeverity Claude C3 fitted −- C1 +0.200 [+0.131, +0.271] DepSeverity Claude C3 a priori −- C1 +0.097 [+0.028, +0.167] DepSeverity Claude C4 fitted −- C1 +0.161 [+0.096, +0.223] DepSeverity Claude C4 a priori −- C1 +0.167 [+0.102, +0.230] DepSign Qwen3.5-9B C2 −- C1 −-0.050 [−-0.077, −-0.024] C,H DepSign Qwen3.5-9B C3 fitted −- C2 +0.152 [+0.092, +0.212] C,H DepSign Qwen3.5-9B C3 a priori −- C2 +0.143 [+0.086, +0.198] C,H DepSign Qwen3.5-9B C4 fitted −- C2 +0.123 [+0.064, +0.182] DepSign Qwen3.5-9B C4 a priori −- C2 +0.108 [+0.055, +0.158] DepSign Qwen3.5-9B C3 fitted −- C1 +0.102 [+0.044, +0.162] DepSign Qwen3.5-9B C3 a priori −- C1 +0.093 [+0.037, +0.148] DepSign Qwen3.5-9B C4 fitted −- C1 +0.074 [+0.017, +0.130] DepSign Qwen3.5-9B C4 a priori −- C1 +0.058 [+0.008, +0.108] DepSign DeepSeek C2 −- C1 +0.119 [+0.078, +0.160] C,H DepSign DeepSeek C3 fitted −- C2 −-0.026 [−-0.086, +0.033] C DepSign DeepSeek C3 a priori −- C2 −-0.022 [−-0.080, +0.036] C DepSign DeepSeek C4 fitted −- C2 −-0.011 [−-0.074, +0.050] DepSign DeepSeek C4 a priori −- C2 −-0.050 [−-0.100, −-0.001] DepSign DeepSeek C3 fitted −- C1 +0.093 [+0.033, +0.152] DepSign DeepSeek C3 a priori −- C1 +0.097 [+0.038, +0.158] DepSign DeepSeek C4 fitted −- C1 +0.108 [+0.047, +0.168] DepSign DeepSeek C4 a priori −- C1 +0.069 [+0.019, +0.118] DepSign Claude C2 −- C1 +0.069 [+0.036, +0.101] C,H DepSign Claude C3 fitted −- C2 −-0.028 [−-0.083, +0.028] C DepSign Claude C3 a priori −- C2 −-0.013 [−-0.074, +0.052] C DepSign Claude C4 fitted −- C2 −-0.041 [−-0.110, +0.024] DepSign Claude C4 a priori −- C2 −-0.062 [−-0.113, −-0.011] DepSign Claude C3 fitted −- C1 +0.041 [−-0.016, +0.097] DepSign Claude C3 a priori −- C1 +0.056 [−-0.009, +0.121] DepSign Claude C4 fitted −- C1 +0.027 [−-0.036, +0.090] DepSign Claude C4 a priori −- C1 +0.006 [−-0.040, +0.055] Appendix B Thresholds Table IV gives the thresholds behind every C3 and C4 cell. Two patterns stand out. On DepSeverity, C3’s fitted rules move the top threshold down to 1.5–2.5, so a post needs only two or three marked criteria to be labeled severe; the DSM-5 rule needs five, which few posts reach (§V-D). On DepSign, the fitted C4 rules move the top threshold up, to 6.5 (9B), 8.5 and 10.5 of 21 items, so fitted C4 catches fewer severe posts there than the DSM-5 rule does (Appendix D). Table IV: Thresholds on the count of present criteria (C3: 0–9, C4: 0–21). Fitted thresholds maximize κw _w on the 600-post fitting split. Corpus Model Cond. fitted a priori DepSeverity Qwen3.5-9B C3 [0.5, 1, 1.5] [0.5, 2.5, 4.5] DepSeverity Qwen3.5-9B C4 [0.5, 1.5, 3.5] [0.5, 2.5, 4.5] DepSeverity DeepSeek C3 [0.5, 1.5, 2.5] [0.5, 2.5, 4.5] DepSeverity DeepSeek C4 [0.5, 1.5, 3.5] [0.5, 2.5, 4.5] DepSeverity Claude C3 [0.5, 1.5, 2.5] [0.5, 2.5, 4.5] DepSeverity Claude C4 [0.5, 2.5, 5.5] [0.5, 2.5, 4.5] DepSign Qwen3.5-9B C3 [0.5, 5.5] [0.5, 4.5] DepSign Qwen3.5-9B C4 [0.5, 6.5] [0.5, 4.5] DepSign DeepSeek C3 [0.5, 5.5] [0.5, 4.5] DepSign DeepSeek C4 [0.5, 8.5] [0.5, 4.5] DepSign Claude C3 [0.5, 3.5] [0.5, 4.5] DepSign Claude C4 [0.5, 10.5] [0.5, 4.5] Appendix C Giving C1 and C2 the same labels Table V expands §V-C. The map is the monotone relabeling selected on the fitting split: for each predicted label, lowest first, the label it is replaced by. Table VI applies the same test to the a priori rules and to C4. Table V: C1 and C2 given the same labels as C3: a monotone relabeling fitted on the 600-post fitting split. The map lists, for each predicted label from lowest to highest, the label it becomes (0123 leaves labels unchanged). The last column is C3 fitted minus the recalibrated condition. Corpus Model raw +cal map C3 fitted −- +cal DepSeverity Qwen3.5-9B C1 0.288 0.320 0012 +0.169 [+0.090, +0.246] DepSeverity Qwen3.5-9B C2 0.340 0.379 0013 +0.109 [+0.027, +0.189] DepSeverity DeepSeek C1 0.219 0.330 0002 +0.196 [+0.113, +0.277] DepSeverity DeepSeek C2 0.504 0.504 0123 +0.022 [−-0.039, +0.086] DepSeverity Claude C1 0.299 0.352 0013 +0.147 [+0.067, +0.226] DepSeverity Claude C2 0.462 0.462 0123 +0.037 [−-0.029, +0.103] DepSign Qwen3.5-9B C1 0.162 0.185 011 +0.078 [+0.011, +0.148] DepSign Qwen3.5-9B C2 0.112 0.219 001 +0.045 [−-0.024, +0.112] DepSign DeepSeek C1 0.109 0.246 001 −-0.043 [−-0.116, +0.031] DepSign DeepSeek C2 0.228 0.262 011 −-0.059 [−-0.119, +0.003] DepSign Claude C1 0.151 0.234 001 −-0.042 [−-0.112, +0.028] DepSign Claude C2 0.220 0.220 012 −-0.028 [−-0.083, +0.028] Table VI: The a priori rules and C4 against C1 and C2. Every C1 and C2 prediction here is recalibrated on the 600-post fitting split with the monotone relabeling of Table V. Δκw _w with 95% paired bootstrap interval; exploratory; bold intervals exclude zero. Corpus Model Comparison Δκw _w [95% CI] DepSeverity Qwen3.5-9B C3 a priori −- C2 −-0.058 [−-0.136, +0.023] DepSeverity Qwen3.5-9B C4 fitted −- C2 +0.116 [+0.024, +0.202] DepSeverity Qwen3.5-9B C4 a priori −- C2 +0.091 [−-0.001, +0.181] DepSeverity Qwen3.5-9B C3 a priori −- C1 +0.002 [−-0.072, +0.079] DepSeverity Qwen3.5-9B C4 fitted −- C1 +0.176 [+0.085, +0.259] DepSeverity Qwen3.5-9B C4 a priori −- C1 +0.150 [+0.060, +0.235] DepSeverity DeepSeek C3 a priori −- C2 −-0.081 [−-0.147, −-0.010] DepSeverity DeepSeek C4 fitted −- C2 −-0.035 [−-0.107, +0.039] DepSeverity DeepSeek C4 a priori −- C2 −-0.026 [−-0.100, +0.049] DepSeverity DeepSeek C3 a priori −- C1 +0.094 [+0.013, +0.173] DepSeverity DeepSeek C4 fitted −- C1 +0.139 [+0.050, +0.226] DepSeverity DeepSeek C4 a priori −- C1 +0.149 [+0.061, +0.236] DepSeverity Claude C3 a priori −- C2 −-0.067 [−-0.135, +0.005] DepSeverity Claude C4 fitted −- C2 −-0.002 [−-0.075, +0.072] DepSeverity Claude C4 a priori −- C2 +0.004 [−-0.068, +0.079] DepSeverity Claude C3 a priori −- C1 +0.043 [−-0.039, +0.125] DepSeverity Claude C4 fitted −- C1 +0.107 [+0.023, +0.190] DepSeverity Claude C4 a priori −- C1 +0.114 [+0.030, +0.199] DepSign Qwen3.5-9B C3 a priori −- C2 +0.036 [−-0.030, +0.101] DepSign Qwen3.5-9B C4 fitted −- C2 +0.016 [−-0.049, +0.083] DepSign Qwen3.5-9B C4 a priori −- C2 +0.001 [−-0.061, +0.064] DepSign Qwen3.5-9B C3 a priori −- C1 +0.069 [+0.003, +0.138] DepSign Qwen3.5-9B C4 fitted −- C1 +0.050 [−-0.019, +0.119] DepSign Qwen3.5-9B C4 a priori −- C1 +0.034 [−-0.031, +0.102] DepSign DeepSeek C3 a priori −- C2 −-0.055 [−-0.120, +0.010] DepSign DeepSeek C4 fitted −- C2 −-0.044 [−-0.120, +0.027] DepSign DeepSeek C4 a priori −- C2 −-0.084 [−-0.151, −-0.018] DepSign DeepSeek C3 a priori −- C1 −-0.039 [−-0.112, +0.035] DepSign DeepSeek C4 fitted −- C1 −-0.028 [−-0.098, +0.042] DepSign DeepSeek C4 a priori −- C1 −-0.068 [−-0.130, −-0.003] DepSign Claude C3 a priori −- C2 −-0.013 [−-0.074, +0.052] DepSign Claude C4 fitted −- C2 −-0.041 [−-0.110, +0.024] DepSign Claude C4 a priori −- C2 −-0.062 [−-0.113, −-0.011] DepSign Claude C3 a priori −- C1 −-0.027 [−-0.106, +0.052] DepSign Claude C4 fitted −- C1 −-0.056 [−-0.132, +0.021] DepSign Claude C4 a priori −- C1 −-0.077 [−-0.139, −-0.012] Appendix D Per-class recall Tables VII and VIII give recall for every class, not only severe. The conditions with the highest κw _w trade recall on the top class for recall on the majority class: on DepSign, C3 with fitted thresholds raises moderate recall to 0.68–0.88 and cuts severe recall to 0.02–0.28. The learned aggregator of §V-D shows the same trade-off from the other side: with inverse-frequency class weights it reaches κw _w 0.421–0.454 on DepSeverity and misses only 14–26 of 56 severe posts, against 43–49 for the count with fitted thresholds. Table VII: Per-class recall on DepSeverity test (n per class: 513, 58, 79, 56). Model Cond. minimum mild moderate severe Qwen3.5-9B C1 0.46 0.29 0.43 0.48 C2 0.50 0.26 0.38 0.39 C3 fitted 0.92 0.00 0.35 0.23 C3 a priori 0.92 0.36 0.03 0.00 C4 fitted 0.77 0.29 0.35 0.29 C4 a priori 0.77 0.53 0.14 0.12 DeepSeek C1 0.32 0.12 0.29 0.71 C2 0.80 0.36 0.23 0.25 C3 fitted 0.84 0.36 0.16 0.12 C3 a priori 0.84 0.59 0.10 0.02 C4 fitted 0.70 0.34 0.30 0.27 C4 a priori 0.70 0.67 0.16 0.14 Claude C1 0.33 0.36 0.47 0.41 C2 0.72 0.34 0.24 0.27 C3 fitted 0.83 0.36 0.20 0.12 C3 a priori 0.83 0.59 0.09 0.02 C4 fitted 0.58 0.52 0.32 0.14 C4 a priori 0.58 0.52 0.28 0.25 Table VIII: Per-class recall on DepSign test (n per class: 184, 472, 50). Model Cond. not dep. moderate severe Qwen3.5-9B C1 0.22 0.24 0.74 C2 0.13 0.16 0.80 C3 fitted 0.40 0.85 0.04 C3 a priori 0.40 0.78 0.06 C4 fitted 0.38 0.70 0.24 C4 a priori 0.38 0.52 0.42 DeepSeek C1 0.11 0.14 0.82 C2 0.42 0.28 0.68 C3 fitted 0.28 0.88 0.02 C3 a priori 0.28 0.79 0.16 C4 fitted 0.32 0.79 0.18 C4 a priori 0.32 0.52 0.40 Claude C1 0.09 0.33 0.66 C2 0.32 0.36 0.62 C3 fitted 0.24 0.68 0.28 C3 a priori 0.24 0.82 0.16 C4 fitted 0.22 0.89 0.08 C4 a priori 0.22 0.46 0.48 Appendix E Output length Table IX reports mean tokens per test call. C2 writes two to three times as much as C3, and C4 about as much as C2; the ranking of the conditions does not follow length. Table IX: Mean tokens per test call, input / output, as reported by each provider. Corpus Model C1 C2 C3 C4 DepSeverity Qwen3.5-9B 179 / 3 212 / 519 535 / 186 767 / 459 DepSeverity DeepSeek 162 / 2 194 / 363 499 / 152 712 / 371 DepSeverity Claude 238 / 5 282 / 545 737 / 227 1021 / 541 DepSign Qwen3.5-9B 276 / 3 309 / 492 633 / 200 865 / 476 DepSign DeepSeek 262 / 2 294 / 429 600 / 171 813 / 404 DepSign Claude 369 / 4 413 / 614 871 / 251 1155 / 579 Appendix F Prompt variants Table X gives the C3 prompt variants of §V-D in full. Table X: C3 prompt variants, DeepSeek-V4.1-Flash on DepSeverity (same nine criteria and JSON schema; wording only). Last two columns: Δκw _w against C2. Variant pres./post absent fitted a pr. vs C2, fitted vs C2, a pr. original 0.45 0.30% 0.526 0.423 +0.022 [−-0.039, +0.086] −-0.081 [−-0.147, −-0.010] permissive 0.72 0.25% 0.507 0.425 +0.002 [−-0.059, +0.067] −-0.079 [−-0.145, −-0.012] symmetric evidence 0.47 0.35% 0.555 0.433 +0.051 [−-0.011, +0.116] −-0.071 [−-0.138, +0.000] Appendix G Prompts The exact prompts, as sent. text is the post. For DepSign, the label list in C1 and C2 is not depression, moderate, severe. C3P and C3S are the prompt variants of Appendix F; their user prompt is C3’s. C1 (direct), system prompt You are assessing the severity of depression expressed in a single social media post. Reply with exactly one word, the severity level, and nothing else. The permitted levels, from lowest to highest, are: minimum, mild, moderate, severe. C1 (direct), user prompt Post: """ text """ Severity level: C2 (chain-of-thought), system prompt You are assessing the severity of depression expressed in a single social media post. First reason step by step about which depressive symptoms are and are not evident in the post, and how strongly each is expressed. Then state the overall severity level. The permitted levels, from lowest to highest, are: minimum, mild, moderate, severe. End your reply with a final line in exactly this form: FINAL: <level> C2 (chain-of-thought), user prompt Post: """ text """ C3 (PHQ-9 extraction), system prompt You are annotating a single social media post for the presence of nine specific symptoms. You are not rating, scoring, or diagnosing anything - you only report, for each symptom, whether the post gives evidence of it. The nine symptoms are: 1. "anhedonia" - little interest or pleasure in doing things 2. "depressed_mood" - feeling down, depressed, or hopeless 3. "sleep" - trouble falling or staying asleep, or sleeping too much 4. "fatigue" - feeling tired or having little energy 5. "appetite" - poor appetite, overeating, or weight change 6. "worthlessness" - feeling bad about yourself, worthless, or excessively guilty 7. "concentration" - trouble concentrating on things 8. "psychomotor" - moving or speaking noticeably slowly, or being restless and fidgety 9. "self_harm" - thoughts that you would be better off dead, or of hurting yourself For each symptom return one of: "present" - the post gives positive evidence that the writer experiences it "absent" - the post gives positive evidence that the writer does NOT experience it "unclear" - the post does not say either way Use "unclear" when the post is simply silent about a symptom. Do not use "absent" merely because a symptom is unmentioned. When and only when a symptom is "present", also return "evidence": a short span copied verbatim from the post. Copy it exactly; do not paraphrase. Reply with JSON only - no preamble, no code fence, no commentary. Schema: "<symptom>": "status": "present|absent|unclear", "evidence": "<verbatim span or empty>", ... Include all nine symptom keys exactly as written above. C3 (PHQ-9 extraction), user prompt Post: """ text """ C4 (BDI-I extraction), system prompt You are annotating a single social media post for the presence of twenty-one specific symptoms. You are not rating, scoring, or diagnosing anything - you only report, for each symptom, whether the post gives evidence of it. The twenty-one symptoms are: 1. "sadness" - feeling sad or unhappy 2. "pessimism" - feeling discouraged or hopeless about the future 3. "past_failure" - feeling like a failure, or dwelling on past failures 4. "loss_of_pleasure" - getting less pleasure from things previously enjoyed 5. "guilt" - feeling guilty 6. "punishment" - feeling one is being punished, or deserves punishment 7. "self_dislike" - disliking oneself, or having lost confidence in oneself 8. "self_criticism" - blaming or criticising oneself 9. "suicidal" - thoughts of killing oneself, or of being better off dead 10. "crying" - crying, or being unable to cry when one wants to 11. "agitation" - feeling restless, agitated, or keyed up 12. "loss_of_interest" - having lost interest in other people or activities 13. "indecisiveness" - finding it harder than usual to make decisions 14. "worthlessness" - feeling worthless, or of no value 15. "loss_of_energy" - having less energy than usual 16. "sleep_change" - sleeping more or less than usual, or broken sleep 17. "irritability" - being more irritable than usual 18. "appetite_change" - eating more or less than usual, or appetite change 19. "concentration" - finding it harder than usual to concentrate 20. "fatigue" - being too tired to do many of the things one used to do 21. "loss_of_interest_sex" - reduced interest in sex For each symptom return one of: "present" - the post gives positive evidence that the writer experiences it "absent" - the post gives positive evidence that the writer does NOT experience it "unclear" - the post does not say either way Use "unclear" when the post is simply silent about a symptom. Do not use "absent" merely because a symptom is unmentioned. When and only when a symptom is "present", also return "evidence": a short span copied verbatim from the post. Copy it exactly; do not paraphrase. Reply with JSON only - no preamble, no code fence, no commentary. Schema: "<symptom>": "status": "present|absent|unclear", "evidence": "<verbatim span or empty>", ... Include all twenty-one symptom keys exactly as written above. C3P (permissive variant), system prompt You are annotating a single social media post for the presence of nine specific symptoms. You are not rating, scoring, or diagnosing anything - you only report, for each symptom, whether the post indicates it. The nine symptoms are: 1. "anhedonia" - little interest or pleasure in doing things 2. "depressed_mood" - feeling down, depressed, or hopeless 3. "sleep" - trouble falling or staying asleep, or sleeping too much 4. "fatigue" - feeling tired or having little energy 5. "appetite" - poor appetite, overeating, or weight change 6. "worthlessness" - feeling bad about yourself, worthless, or excessively guilty 7. "concentration" - trouble concentrating on things 8. "psychomotor" - moving or speaking noticeably slowly, or being restless and fidgety 9. "self_harm" - thoughts that you would be better off dead, or of hurting yourself For each symptom return one of: "present" - the post indicates the writer experiences it. Count it as present if the writer states it directly, describes it in their own words, or describes circumstances or behaviour from which it reasonably follows. Do not require a clinical phrasing, and do not require the writer to name the symptom. "absent" - the post indicates the writer does NOT experience it "unclear" - the post gives no indication either way Err toward "present" when a reading of the post supports it. Reserve "unclear" for symptoms the post genuinely does not touch on. When a symptom is "present", also return "evidence": a short span copied verbatim from the post that supports the judgement. Copy it exactly; do not paraphrase. Reply with JSON only - no preamble, no code fence, no commentary. Schema: "<symptom>": "status": "present|absent|unclear", "evidence": "<verbatim span or empty>", ... Include all nine symptom keys exactly as written above. C3S (symmetric variant), system prompt You are annotating a single social media post for the presence of nine specific symptoms. You are not rating, scoring, or diagnosing anything - you only report, for each symptom, what the post says about it. The nine symptoms are: 1. "anhedonia" - little interest or pleasure in doing things 2. "depressed_mood" - feeling down, depressed, or hopeless 3. "sleep" - trouble falling or staying asleep, or sleeping too much 4. "fatigue" - feeling tired or having little energy 5. "appetite" - poor appetite, overeating, or weight change 6. "worthlessness" - feeling bad about yourself, worthless, or excessively guilty 7. "concentration" - trouble concentrating on things 8. "psychomotor" - moving or speaking noticeably slowly, or being restless and fidgety 9. "self_harm" - thoughts that you would be better off dead, or of hurting yourself For each symptom return one of: "present" - the post gives evidence that the writer experiences it "absent" - the post gives evidence that the writer does NOT experience it. This includes the writer denying the symptom, describing its opposite, or describing functioning that is incompatible with it. "unclear" - the post does not address the symptom For BOTH "present" and "absent" you must return "evidence": a short span copied verbatim from the post that supports the judgement. If you cannot quote a supporting span, the correct answer is "unclear". Copy spans exactly; do not paraphrase. Reply with JSON only - no preamble, no code fence, no commentary. Schema: "<symptom>": "status": "present|absent|unclear", "evidence": "<verbatim span or empty>", ... Include all nine symptom keys exactly as written above.