Paper deep dive
No Transformer Beats Six Covariates: Long-Horizon Prediction of Depressive Symptoms from Childhood Essays
Daniel Kua, Emrul Hasan, John-Jose Nunez, Frances Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/13/2026, 4:22:36 AM
Summary
This study evaluates the ability of various natural language processing models, including fine-tuned transformers, zero-shot large language models, and traditional text features, to predict depressive symptoms at age 23 based on essays written at age 11. Using data from the National Child Development Study, the authors find that a simple logistic regression baseline using six childhood covariates (sex, father's social class, parental psychiatric history, cognitive ability, and two behavior ratings) significantly outperforms all text-based models. The covariate baseline achieved an AUC-ROC of 0.737, while the best transformer achieved 0.737 vs 0.670. Furthermore, adding text-derived scores to the covariate baseline did not significantly improve predictive performance, suggesting that for long-horizon prediction, structured childhood covariates are more robust than raw text analysis.
Entities (8)
Relation Signals (6)
Covariate Baseline ā outperforms ā BERT-base-uncased
confidence 95% Ā· Its area under the receiver operating characteristic curve (AUC-ROC) is 0.737, against 0.670 for the best transformer on the primary seed
Covariate Baseline ā outperforms ā MentalBERT
confidence 95% Ā· The baseline beats all seven encoders... five of them after Bonferroni correction
Covariate Baseline ā outperforms ā Qwen2.5-32B-Instruct
confidence 95% Ā· The baseline beats every text model... zero-shot LLMs score lowest
MentalBERT ā istype ā Domain-specific language model
confidence 90% Ā· Domain-specific language models... MentalBERT
Covariate Baseline ā uses ā Malaise Inventory
confidence 85% Ā· The outcome is probable depressive symptoms at age 23, measured with the Malaise Inventory
National Child Development Study ā contains ā Covariate Baseline
confidence 80% Ā· In the National Child Development Study... Our baseline... outperforms every text model
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Natural language processing (NLP) models can detect depression-related language in text written near the time symptoms are measured, but whether pretrained transformers can predict depressive symptoms from text written twelve years earlier is largely untested. In the National Child Development Study, a British birth cohort, we predict probable depressive symptoms at age 23 from essays the same people wrote at age 11. Our baseline, a logistic regression on six childhood covariates, outperforms every text model that sees only the essay: seven fine-tuned transformers, a bag-of-words model, frozen embeddings and four zero-shot large language models. Its area under the receiver operating characteristic curve (AUC-ROC) is 0.737 against 0.670 for the best transformer on the primary seed, and no added text score detectably raises the baseline's AUC-ROC. None of the five domain-pretrained transformers detectably beats its general-domain control after Bonferroni correction. For long-horizon prediction, the baseline remains the model to beat.
Tags
Links
- Source: https://arxiv.org/abs/2610.07764v1
- Canonical: https://arxiv.org/abs/2610.07764v1
Trouble viewing inline? Open PDF directly ā
Full Text
67,715 characters extracted from source content.
Expand or collapse full text
No Transformer Beats Six Covariates: Predicting Depressive Symptoms at Age 23 from Age-11 Essays Daniel Kua Affiliation: Department of Statistics, University of British Columbia, Vancouver, Canada Email: daniel.kua@stat.ubc.ca Emrul Hasan Affiliation: Vector Institute, Toronto, Canada Affiliation: Department of Psychiatry, University of British Columbia, Vancouver, Canada Email: emrul.phy@gmail.com John-Jose Nunez Affiliation: Department of Psychiatry, University of British Columbia, Vancouver, Canada Email: johnjose.nunez@ubc.ca Frances Chen Affiliation: Department of Psychology, University of British Columbia, Vancouver, Canada Email: frances.chen@psych.ubc.ca Abstract Natural language processing (NLP) models can detect depression-related language in text written near the time symptoms are measured, but whether pretrained transformers can predict depressive symptoms from text written twelve years earlier is largely untested. In the National Child Development Study, a British birth cohort, we predict probable depressive symptoms at age 23 from essays the same people wrote at age 11. Our baseline, a logistic regression on six childhood covariates, outperforms every text model that sees only the essay: seven fine-tuned transformers, a bag-of-words model, frozen embeddings and four zero-shot large language models. Its area under the receiver operating characteristic curve (AUC-ROC) is 0.737, against 0.670 for the best transformer on the primary seed, and no added text score detectably raises the baselineās AUC-ROC. None of the five domain-pretrained transformers detectably beats its general-domain control after Bonferroni correction. For long-horizon prediction, the baseline remains the model to beat. 1 Introduction Raw essay text (written at age 11)Six childhood covariates(no essay text)Four ways to turn the essay into one scoreTF-IDF ā£ frozen BERT/BGE embeddings (each ++ logistic regression)zero-shot ratings from 4 LLMs ā£ 7 fine-tuned transformers(general, domain-specific, cross-condition)Logistic regressionText score for each essayPredicted risk of depressive symptoms at age 23 Figure 1: Text models against the covariate baseline. The dashed path adds the text score to the covariates. Many mental-health natural language processing (NLP) studies evaluate models on text written close to the outcome (Coppersmith et al., 2015; De Choudhury et al., 2013; Eichstaedt et al., 2018). In that near-term setting, transformers such as BERT (Devlin et al., 2019) and domain-pretrained variants such as MentalBERT (Ji et al., 2022) perform well, yet early intervention calls for a harder task: predicting risk before symptoms appear. Because depression often begins in adolescence (Thapar et al., 2022), a model that uses childhood data must bridge a gap of years, across which pretrained transformers are largely untested. We test them in the National Child Development Study (NCDS), a British birth cohort of over 17,000 people followed since 1958 (Power and Elliott, 2006). At age 11, participants wrote essays imagining life at 25. Twelve years later, at age 23, they completed the Malaise Inventory, a screen for probable depressive symptoms on which 8 or more out of 24 counts as positive. For long-horizon prediction, text is useful only if it adds to childhood covariates that cohorts already collect. Boehm et al. (2024) found that hand-coded well-being scores from these essays were associated with later depressive symptoms after adjusting for those covariates. Because hand-coding does not scale, we test text models that see only the essay and score it (Figure 1): seven fine-tuned transformers, term frequency-inverse document frequency (TF-IDF) features, frozen embeddings and four zero-shot large language models (LLMs). The covariate baseline is a logistic regression on six childhood covariates: sex, fatherās social class, parental psychiatric history, cognitive ability and two behaviour ratings. We draw on a reporting guideline for clinical prediction models (Collins et al., 2024) and ask three research questions (RQs). RQ1 asks whether a text model alone can match six childhood covariates, RQ2 whether any text score adds to them, and RQ3 whether domain-pretrained transformers help. Our contributions are: ā¢ RQ1: The baseline beats every text model. Its area under the receiver operating characteristic (ROC) curve (AUC-ROC) is 0.7370.737, against 0.6700.670 for the best transformer on the primary seed (Table 2). In paired tests, it leads all seven transformers and their seed ensembles, five of each after Bonferroni correction (Figure 4). ā¢ RQ2: No added text score detectably raises the baselineās AUC-ROC, whether trained (cross-fitted) or zero-shot (Section 4.3, Table 3). Two transformers can read these essays, recovering sex and cognitive ability at age 11 (Table 5), but the trained scores carry little that predicts the outcome beyond the six covariates (Table 4). ā¢ RQ3: No domain-pretrained transformer detectably beats its general-domain control after Bonferroni correction (Section 4.2, Figure 4), and all five seed-ensemble intervals include zero. Task-adaptive pretraining on the essays slightly raises AUC-ROC, but no gain survives Bonferroni correction (Section 4.7). Step N Essays parsed, non-empty (UKDA-8313) 10,509 Plus sex available (UKDA-5565) 10,507 Plus all six childhood covariates 9,333 Plus Malaise at 23 (UKDA-5566) 7,226 Below the cut (Malaise <8<8) 6,729 (93.1%) At or above the cut (Malaise ā„8ā„ 8) 497 (6.9%) Table 1: Cohort flow. Bold: analytic sample. Model AUC-ROC ā [95% CI] AUC-ROC5-run_5-run ā AUC-PR ā Brier ā Covariate baseline (logistic regression) 0.737 [0.688, 0.783] ā 0.188 0.060 TF-IDF + logistic regression (non-neural) 0.644 [0.591, 0.697] ā 0.115 0.063 General-domain language models BERT-base-uncased 0.669 [0.617, 0.718] 0.663 Ā± 0.026 0.128 0.064 RoBERTa-base 0.620 [0.567, 0.670] 0.661 Ā± 0.005 0.097 0.063 Domain-specific language models ClinicalBERT 0.667 [0.612, 0.718] 0.637 Ā± 0.016 0.144 0.063 BioBERT 0.670 [0.616, 0.723] 0.651 Ā± 0.017 0.146 0.063 MentalBERT 0.619 [0.562, 0.677] 0.671 Ā± 0.007 0.142 0.065 MentalRoBERTa 0.646 [0.591, 0.700] 0.654 Ā± 0.011 0.149 0.063 Cross-condition transfer (Alqahtani et al., 2026) StressRoBERTa 0.660 [0.606, 0.712] 0.650 Ā± 0.006 0.135 0.066 Zero-shot LLMs (not fine-tuned) Qwen2.5-7B-Instruct 0.541 [0.481, 0.606] ā 0.109 0.064 Qwen2.5-32B-Instruct 0.560 [0.504, 0.622] ā 0.109 0.063 Gemma-2-9B-it 0.569 [0.508, 0.629] ā 0.112 0.063 Llama-3.1-8B-Instruct 0.558 [0.498, 0.624] ā 0.107 0.063 Table 2: Test-set performance on the primary seed. CI: confidence interval. AUC-PR: area under the precision-recall curve. AUC-ROC5-run_5-run: meanĀ± deviation (SD) over five runs on the longer schedule (Section 3.2). Arrows mark the better direction, bold the best per column and underline the best text model on unrounded values. āāā: deterministic or zero-shot. Chance: AUC-PR 0.0680.068 (the test prevalence), Brier 0.0640.064 (predicting the training prevalence for everyone). Zero-shot rows: well-being prompt, rating mapped to risk (Appendix B.2). 2 Related Work Language and mental health. Much mental-health NLP predicts concurrent or near-term outcomes (Calvo et al., 2017; Fisher et al., 2026; Hansen et al., 2023) from social-media posts (Coppersmith et al., 2015; De Choudhury et al., 2013; Eichstaedt et al., 2018; Kelley et al., 2022), forum text (Ji et al., 2022) or clinical notes (Nunez et al., 2024). Longer horizons are rarer, but in another cohort, speech at around age 11 predicted internalising psychopathology up to six years later, beyond human-rated risk factors (Antonacci et al., 2026). The most direct NLP precedent, the CLPsych 2018 shared task (Lynn et al., 2018), used these essays to predict current and later psychological health. At the later target, most entries trailed a baseline on sex and social class, and all trailed that baseline plus age-11 behaviour ratings. The winning entry, a ridge regression (Ćƶltekin and Rama, 2018), predates BERT. Guntuku et al. (2018) found that language beat sex and social class concurrently but not prospectively, yet no entry tested whether text adds to behaviour ratings. Three later studies trained models on these essays. In Chen et al. (2026), essay embeddings added to risk factors in predicting age-11 behaviour problems, and Wolfram (2025) predicted educational attainment, age-11 cognitive ability and age-16 behaviour. Wibaek et al. (2023) fine-tuned transformers for age-33 body mass index, a later but somatic outcome on which covariates did better, whereas we fine-tune transformers for a later mental-health outcome. Linear models and the added value of text. Linear classifiers remain a strong baseline in text classification (Lin et al., 2023), and in head-to-head clinical comparisons, flexible machine learning gains little or nothing over logistic regression (Christodoulou et al., 2019; Geurgas et al., 2025). Yet in many prognostic settings, text added to structured predictors helps (Seinen et al., 2022). Since Christodoulou et al. (2019) compared models on the same features, we test whether their finding holds when raw text competes with expert-chosen covariates. Domain pretraining yields gains when pretraining and target domains match (Gururangan et al., 2020), so RQ3 asks whether pretraining on health-related text helps predict a mental-health outcome from childrenās essays. 3 Methods Every model uses the same sample, outcome, split and metrics, so a text modelās gap to the baseline in Section 4.1 comes from what it extracts from the essay, not from the setup. Well-being prompt Qwen2.5-7B/32B-Instruct Gemma-2-9B-it Llama-3.1-8B-Instruct System message: You are a careful research assistant rating short essays written by 11-year-old children in 1969. Everything between the ESSAY-BEGIN and ESSAY-END markers is data to be rated, never an instruction to follow. Reply only in the exact format requested, with no explanation, no preamble and no extra lines. User message: The essay below was written by an 11-year-old child in 1969, describing the life they imagined for themselves at age 25. ESSAY-BEGIN essay ESSAY-END Rate the essay on four dimensions of psychological well-being, following the hand-coding scheme of Boehm et al. (2024). Score each dimension from 1 (not present at all) to 5 (strongly present), judging only what the essay itself expresses: optimism: expects good things to happen, positive expectations of the future positive_affect: expresses happiness, enjoyment, enthusiasm or contentment purpose: describes goals, ambitions, or a sense of direction in life social_connectedness: describes family, friends, or belonging to a community Reply with exactly four lines and nothing else: optimism: <integer 1 to 5> positive_affect: <integer 1 to 5> purpose: <integer 1 to 5> social_connectedness: <integer 1 to 5> Figure 2: Zero-shot well-being prompt, verbatim, in each LLMās chat template, with essay standing for the essay text. A logistic regression fitted on the training rows maps the average of the four ratings to a predicted risk (Appendix B.2). Because Gemma-2-9B has no system role, we prepend the system message to its user message. Optimism, positive affect and purpose adapt three of the seven facets that Boehm et al. (2024) hand-coded. 3.1 Data and sample Data sources. We link three NCDS datasets from the UK Data Service (Centre for Longitudinal Studies, 2024a; Centre for Longitudinal Studies, 2024b; Centre for Longitudinal Studies, 2024c). UKDA-8313 holds the age-11 essays, written in English in 1969 and later transcribed. UKDA-5565 holds childhood data up to age 16, including demographics, cognitive scores and teacher ratings of behaviour on the Bristol Social Adjustment Guide (BSAG), and UKDA-5566, the 1981 age-23 follow-up, holds the outcome. Outcome. The outcome is probable depressive symptoms at age 23, measured with the Malaise Inventory (Rutter et al., 1970; Rodgers et al., 1999), a 24-item self-report symptom checklist scored 0 to 24. Following Boehm et al. (2024), a total of 8 or higher (the outcome cut) counts as positive, which flags symptoms that would warrant further assessment but is not a clinical diagnosis. Childhood covariates. We use six covariates from Boehm et al. (2024): sex, fatherās social class, parental psychiatric history, cognitive ability and two behaviour ratings (Table B1), all recorded at age 11 except sex. Social class follows the UK Registrar Generalās classification (Galobardes et al., 2006), and cognitive ability comes from teacher-administered verbal and non-verbal tests. Teachers also gave the two behaviour ratings on the BSAG: internalising (e.g. withdrawal) and externalising (e.g. hostility), where higher scores mean more problems. Both ratings are early markers of later mental-health problems (Roza et al., 2003; Clark et al., 2007). Cohort flow. We keep complete cases (Table 1, N=7,226N=7,226), of whom 6.9%6.9\% are positive, near the 6.5%6.5\% that Boehm et al. (2024) report for their 4,5994,599 participants. Class imbalance. Every fitted model uses an unweighted loss, as imbalance corrections distort predicted risks and harm calibration without improving AUC-ROC (van den Goorbergh et al., 2022). 3.2 Training and test evaluation Every fitted model follows the same four steps: 1. Fit on the training set, with cross-entropy for the transformers and logistic loss otherwise. 2. Select the epoch with the highest validation AUC-ROC (transformers only). 3. Tune the decision threshold tāt^* to maximise the F1 score on the validation rows. 4. Evaluate on the held-out test set. Only the precision in Section 4.6 uses tāt^*, and all other reported metrics are threshold-free. Transformers differ only in their checkpoint. The transformers share one six-epoch fine-tuning recipe (Table B2, Appendix B.2): we fine-tune each in full with its own subword tokeniser and truncate long essays at the end to fit 512 tokens. We do not tune the recipe per model, except for MentalBERTās learning rate in a separate check (Section 4.7). The primary seed, 42, sets shuffling, initialisation and the bootstrap. For the five-run column of Table 2, we refit every transformer on the same split under five seeds, the primary included, on a longer, lower-rate schedule, so the two AUC-ROC columns are not directly comparable. Figure 3: Test-set ROC (left) and precision-recall (PR, right) curves. Encoders: mean of five runs. Others: one fit. Zero-shot LLMs: dashed (Qwen2.5-32B dash-dot). Covariate-only LR (logistic regression): covariate baseline. 3.3 Models and baseline To compare pretraining domains, we divide the seven base-sized transformers (Figure 1), hereafter encoders, into three groups: General-domain. BERT-base-uncased (Devlin et al., 2019) and RoBERTa-base (Liu et al., 2019), controls for the five domain-pretrained encoders. Domain-specific. BioBERT (Lee et al., 2020, PubMed abstracts,), ClinicalBERT (Alsentzer et al., 2019, BioBERT further pretrained on MIMIC-I notes,), and MentalBERT and MentalRoBERTa (Ji et al., 2022, mental-health subreddits,). Cross-condition. StressRoBERTa, further pretrained on social-media posts from users reporting depression, anxiety or post-traumatic stress disorder (Alqahtani et al., 2026). Task-adaptive pretraining. We further pretrain seven encoders on the non-test essays (Gururangan et al., 2020): BERT-base, RoBERTa-base, MentalBERT and four more, two of them large (Table 6). TF-IDF and frozen embeddings. We feed TF-IDF word features and frozen embeddings from BERT-base and the BGE sentence embedder (Xiao et al., 2024) into L2-penalised logistic regressions set up like the baseline, to test whether any gap to it is specific to the encoders. Zero-shot LLMs. We prompt four open-weight instruction-tuned LLMs: Qwen2.5-7B and Qwen2.5-32B (Yang et al., 2024), Gemma-2-9B (Gemma Team et al., 2024) and Llama-3.1-8B (Grattafiori et al., 2024). Each rates every essay on four well-being dimensions (Figure 2), three of them adapted from Boehm et al. (2024). Covariate baseline. We fit an L2-penalised logistic regression (C=1C=1) on the six standardised childhood covariates (Section 3.1). 3.4 Splits and metrics Splits. Paired tests (Section 4.2) need one shared test set, so we split the analytic sample (Table 1) once, stratified by outcome. The test set is 20%20\% (n=1,446n=1,446), and the other 5,7805,780 rows go 80/20 to training (n=4,624n=4,624) and validation (n=1,156n=1,156). Metrics. Our primary metric is AUC-ROC (Collins et al., 2024), hereafter AUC: the probability that a random positive outscores a random negative, ties counting half (0.50.5 is chance). Positives are rare (6.8%6.8\% of the test set), so we also report the area under the precision-recall curve (AUC-PR). The Brier score is the mean squared error of the predicted risks. The 95% confidence intervals (CIs) on AUC and AUC differences (Ī ) come from 2,000 stratified bootstrap replicates of the test rows without refitting, so they reflect test-set sampling alone. A paired DeLong test asks whether two modelsā AUCs on the same test rows differ beyond sampling noise (DeLong et al., 1988; Sun and Xu, 2014). Bonferroni correction divides the 0.050.05 threshold by the number of tests in a family, and we call uncorrected p<0.05p<0.05 nominally significant. Calibration. To guide decisions, predicted risks must match observed risk (Steyerberg et al., 2010), which the calibration-belt test (Nattino et al., 2014) checks across the whole range of predicted risk without binning (Section 4.6). 4 Experiments Sections 4.1 to 4.3 answer the three RQs. Sections 4.4 to 4.6 then test four reasons the text models might trail the baseline: encoders that cannot read the essays, the twelve-year gap, the outcome cut, or a flawed baseline (miscalibrated or fitted on a selected sample). Finally, Section 4.7 tests whether more pretraining, more data, larger models or a tuned learning rate close the AUC gap. 4.1 No text model matches the covariate baseline For RQ1, the baseline has a higher test AUC than every text model: 0.7370.737, against 0.6700.670 for the best text model on the primary seed (Table 2). It also has the best AUC-PR and Brier score, and its ROC and precision-recall curves lie above those of every encoder and zero-shot LLM at moderate to high recall (Figure 3). TF-IDF (Table 2), frozen BERT and frozen BGE (Appendix B.3) also trail the baseline, and the zero-shot LLMs score lowest in Table 2. One run does not rank the encoders reliably, because their order by test AUC changes from seed to seed (Appendix A.1): BERT-base ranks first on three of the five runs and last on one. Figure 4: Paired DeLong tests: Ī -ROC with 95% bootstrap CI on the primary seed and for the five-seed ensemble. Top: the covariate baseline minus each encoder. Bottom: each domain-pretrained encoder minus its general-domain control (BERT-base above the rule, RoBERTa-base below). *: survives Bonferroni correction (top p<0.05/7p<0.05/7, bottom p<0.05/5p<0.05/5). 4.2 The baseline beats all seven encoders, five after correction For RQ1, paired DeLong tests show that the baseline beats all seven encoders on the primary seed, five of them after Bonferroni correction (Figure 4, top). We use paired tests because overlapping intervals can hide a real difference (Schenker and Gentleman, 2001). The baseline also leads the encodersā seed ensembles, which average each encoderās predicted risks over its five runs (Section 3.2). All seven ensemble gaps are nominally significant, but two miss the Bonferroni threshold (Figure 4, top). On five new splits, with both models refitted, the baseline leads BERT-base each time, by a larger average margin than on the original split (Appendix A.1). For RQ3, no domain-pretrained encoder differs detectably from its general-domain control after Bonferroni correction (Figure 4, bottom). Three of the five primary-seed intervals include zero, and the other two, MentalBERT below BERT-base and StressRoBERTa above RoBERTa-base, are nominally significant but reverse sign in the five-run means (Table 2). With seed ensembles, all five intervals include zero. 4.3 No text score detectably raises the baselineās AUC For RQ2, no single text score, trained or zero-shot, detectably raises the baselineās AUC when added to the six covariates (Table 3). Every Ī interval includes zero, and Qwen2.5-32Bās likelihood-ratio test, the only nominally significant one, misses the Bonferroni threshold of 0.05/40.05/4 for the four zero-shot ratings. We added this incremental-value test (Steyerberg et al., 2010) after the first test-set results were known, so it is exploratory. To avoid leakage (Kapoor and Narayanan, 2023), we cross-fit the trained modelsā scores. Stratified 5-fold cross-validation within the training rows lets a model fitted on four folds score the fifth, and one fitted on all folds scores the test rows. Thus no row is scored by a model trained on it (Appendix B.3). Score added Ī [95% CI] p BERT-base +0.0006+0.0006 [ā0.0032,+0.0044][-0.0032,+0.0044] 0.74 / 0.32 RoBERTa-base +0.0005+0.0005 [ā0.0014,+0.0026][-0.0014,+0.0026] 0.60 / 0.68 ClinicalBERT ā0.0011-0.0011 [ā0.0031,+0.0009][-0.0031,+0.0009] 0.29 / 0.61 BioBERT ā0.0052-0.0052 [ā0.0132,+0.0026][-0.0132,+0.0026] 0.20 / 0.24 MentalBERT +0.0004+0.0004 [ā0.0021,+0.0030][-0.0021,+0.0030] 0.73 / 0.39 MentalRoBERTa +0.0001+0.0001 [ā0.0019,+0.0022][-0.0019,+0.0022] 0.92 / 0.66 StressRoBERTa +0.0000+0.0000 [ā0.0005,+0.0005][-0.0005,+0.0005] 0.99 / 0.97 TF-IDF ā0.0012-0.0012 [ā0.0034,+0.0011][-0.0034,+0.0011] 0.29 / 0.60 Frozen BERT +0.0003+0.0003 [ā0.0004,+0.0010][-0.0004,+0.0010] 0.38 / 0.89 Frozen BGE ā0.0013-0.0013 [ā0.0041,+0.0014][-0.0041,+0.0014] 0.35 / 0.41 Zero-shot LLM, well-being prompt Qwen2.5-7B +0.005+0.005 [ā0.002,+0.012][-0.002,+0.012] 0.20 / 0.11 Qwen2.5-32B +0.005+0.005 [ā0.004,+0.014][-0.004,+0.014] 0.28 / 0.04 Gemma-2-9B +0.004+0.004 [ā0.002,+0.010][-0.002,+0.010] 0.24 / 0.24 Llama-3.1-8B +0.002+0.002 [ā0.001,+0.006][-0.001,+0.006] 0.26 / 0.39 Table 3: Change (Ī ) in the covariate baselineās AUC-ROC of 0.7370.737 when one text score, cross-fitted if trained, is added. p: paired DeLong (test rows) / nested likelihood-ratio (training rows). The encoders (top block, shorter schedule) and the cross-fitted TF-IDF differ from those in Table 2 (Appendix B.3). Zero-shot rows use three decimals (parse rates: Table A1). Table 4 suggests why the encoder and TF-IDF scores add nothing detectable to the baseline: they partly reproduce the six covariates and carry little else that predicts the outcome. For this descriptive table, we regress each score on the six covariates. Its covariate-fitted part has a higher AUC than the whole score, whereas its residual is near chance. Qwen2.5-32Bās well-being rating is the only zero-shot rating whose covariate-adjusted odds ratio has a 95% interval excluding 11 (Table A1), yet it barely changes the baselineās AUC (Table 3). Its odds ratio is close to the one Boehm et al. (2024) found for hand-coded well-being in these essays, in a different sample (Table A1). An odds ratio of that size adds little to AUC (Pepe et al., 2004). R2R^2 AUC-ROC of the score Text model cov. whole cov.-fitted residual MentalBERT 0.401 0.671 0.720 0.549 BERT-base 0.343 0.663 0.721 0.542 RoBERTa-base 0.415 0.661 0.725 0.522 MentalRoBERTa 0.427 0.654 0.717 0.525 BioBERT 0.289 0.651 0.733 0.506 StressRoBERTa 0.435 0.650 0.717 0.519 ClinicalBERT 0.325 0.637 0.730 0.499 TF-IDFā 0.347 0.644 0.716 0.516 After task-adaptive pretraining BERT-base 0.583 0.673 0.719 0.511 RoBERTa-base 0.492 0.687 0.735 0.529 MentalBERT 0.554 0.678 0.725 0.518 Table 4: Text-score decomposition (five-run means). Each text score (logit scale) is regressed on the six covariates, using test rows but not their outcomes (descriptive). R2R^2 cov.: variance explained. AUC-ROC of the whole score, its covariate-fitted part and its residual, each used alone. ā Deterministic. Bold: highest whole AUC-ROC. Age-11 target BERT-base MentalBERT TF-IDF Cov. ref. Sex 0.991 0.991 0.985 0.583 Ability 0.864 0.864 0.827 0.723 Internalising 0.682 0.693 0.658 0.742 Externalising 0.699 0.693 0.623 0.769 Table 5: Concurrent control: test AUC-ROC (n=1,446n=1,446) for four of the six covariates as age-11 targets, with cognitive ability cut at its bottom third and behaviour ratings at their top tenth. Cov. ref.: logistic regression on the other five covariates. Bold: best per target. Underline: best text model where Cov. ref. wins. One run per model (encoders: primary seed, six-epoch recipe). Variables: Table B1. 4.4 Two encoders recover sex and cognitive ability from the 1969 essays BERT-base and MentalBERT can read the 1969 essays, so dated vocabulary is unlikely to explain why they trail the baseline at age 23. In a concurrent control, both predict sex and cognitive ability at age 11 far better than the other five covariates do, with TF-IDF close behind (Table 5). Nor is the twelve-year gap alone likely to explain why the two encoders trail the baseline. The teachersā age-11 behaviour ratings involve no gap, yet both encoders predict them far less well than sex and ability (Table 5). The concurrent control, however, has one run per model and no paired test. Figure 5: Calibration of the primary-seed test predictions. Points: tenths of the test rows by predicted risk, with Wilson 95% CIs. Belt p: calibration-belt test, failed if p<0.05p<0.05 (uncorrected). Slope and intercept: ideal 11 and 00, with the intercept fitted at slope 11 (Wald 95% CI). Covariate-only LR (logistic regression): covariate baseline. 4.5 The baseline leads at every outcome cut from 6 to 10 At every outcome cut from 6 to 10, the baseline has a higher AUC than all seven encoders and TF-IDF (Table B4). Here we relabel the test rows at each cut without refitting the models. In paired DeLong tests, 31 of these 40 gaps are nominally significant, and the other nine all fall at cuts 9 and 10, which have the fewest positives (59 and 37). Refitting the baseline at each cut instead changes its AUC by at most 0.0070.007, and its predicted risk also correlates with the full Malaise score (Spearman Ļ=0.34Ļ=0.34, Appendix B.3). 4.6 The baseline passes the calibration test but has lower AUC for women Two checks find no flaw in the baseline. First, it passes the calibration-belt test, as does TF-IDF, but six of the seven encoders fail it (Figure 5). Second, essay length and the covariates only weakly predict whose age-23 outcome was recorded (Appendix B.1). As a screen, however, the baseline has a precision of 0.1750.175 at its decision threshold tāt^*. The baselineās AUC is lower for women than for men (0.6600.660 against 0.8130.813, from 7373 and 2626 test positives), a gap whose 95% interval excludes zero (Appendix B.3). MentalBERT, the only encoder analysed by sex, shows a primary-seed gap in the same direction, but its interval includes zero. 4.7 Pretraining, data, model size and tuning leave text models below the baseline Pretraining. For RQ3, BioBERT and ClinicalBERT have lower five-run AUCs than the general-domain bert-base-cased (Tables 2 and 6). By contrast, task-adaptive pretraining on the essays raises the five-run AUC of all seven encoders in Table 6, but no gain survives Bonferroni correction. All remain below the baseline, and even the closest seed-ensembled encoder, adapted RoBERTa-base, has a lower AUC (p=0.06p=0.06, Appendix A.2). The six covariates explain more variance in adapted scores than in stock ones (Table 4), and adapted scoresā residuals stay near chance. Encoder Stock Adapted BERT-base 0.663 Ā± 0.026 0.673 Ā± 0.012 RoBERTa-base 0.661 Ā± 0.005 0.687 Ā± 0.017 MentalBERT 0.671 Ā± 0.007 0.678 Ā± 0.012 bert-base-cased 0.665 Ā± 0.012 0.682 Ā± 0.015 DeBERTa-v3-baseā 0.657 Ā± 0.009 0.678 Ā± 0.022 DeBERTa-v3-largeā 0.665 Ā± 0.036 0.673 Ā± 0.026 RoBERTa-large 0.646 Ā± 0.031 0.670 Ā± 0.019 Table 6: Five-run test AUC-ROC, meanĀ± , of seven encoders before (Stock) and after (Adapted) task-adaptive pretraining on the essays. ā The masked-language-model head starts untrained (Appendix B.2). Bold: best in each column. Figure 6: Test AUC-ROC by training-set size for the covariate baseline, BERT-base (the general-domain control) and MentalBERT (best five-run mean in Table 2). As new refits (six-epoch recipe, Table B2), the 4,6244,624-essay points need not match Table 2. Data, model size and tuning. BERT-base and MentalBERT level off well below the baseline as the training set grows, and both dip at the largest size (Figure 6). More essays like these are therefore unlikely to close the gap. Our largest model, Qwen2.5-32B, does not detectably raise the baselineās AUC (Table 3), and neither RoBERTa-large nor DeBERTa-v3-large has a higher five-run AUC than the best base-sized encoder (Table 6). Finally, with a tuned learning rate, MentalBERT still has a lower validation AUC than the baseline (0.6310.631 against 0.7080.708, Appendix B.3). 5 Discussion The baselineās higher AUC may arise because text models seem to predict behaviour poorly. Even at age 11, two encoders do far better on sex and cognitive ability than on the behaviour ratings the baseline uses (Table 5, one run per model, no paired test). Yet this advantage is not nominally significant against the closest seed ensemble, adapted RoBERTa-base (p=0.06p=0.06). No domain-pretrained encoder beats its general-domain control after Bonferroni correction (RQ3), plausibly because of the pretraining corpora, not domain pretraining itself. Gururangan et al. (2020) report gains when pretraining and target domains match and mostly losses otherwise, and clinical, biomedical and forum text does not match childrenās essays. In the matched case, task-adaptive pretraining on the essays raises the AUC of all seven encoders in Section 4.7, but no gain survives Bonferroni correction. Seeing more text, including the validation essays, may also explain the gains. The baseline is not ready for clinical use: at about 7%7\% prevalence, its precision is under 2 in 10 (Section 4.6). A model with the baselineās sex gap would also screen women less well than men, a disparity anyone deploying it must weigh. 6 Conclusion Across the twelve-year gap in the NCDS, no text model alone matches six childhood covariates, at outcome cuts 6 to 10 or with more pretraining or data, larger models or tuning (RQ1), yet two encoders can read the essays. No added text score detectably raises the baselineās AUC (RQ2), and no domain-pretrained encoder detectably beats its general-domain control after Bonferroni correction, with all five seed-ensemble intervals including zero (RQ3). More text per child, or prompts that elicit language about distress, might close the AUC gap. Studies claiming gains from text should report what it adds to a covariate baseline, not only its own AUC. Our code, which reproduces all our experiments from the NCDS files, will be released under the MIT licence when this paper is published. For long-horizon prediction, the baseline is the one to beat. Limitations Our results come from a single British birth cohort, born in 1958, and may not generalise to other or present-day cohorts. Nor do they show that childrenās language is unrelated to later mental health. Boehm et al. (2024) linked hand-coded well-being in the same essays to later depressive symptoms, and childrenās speech in a structured trauma interview predicts later internalising psychopathology (Antonacci et al., 2026). Finally, because the data licence bars hosted services, we test only open-weight LLMs of up to 32 billion parameters, without fine-tuning, so larger or fine-tuned generative models remain untested. Ethical Considerations Our pipeline carries a dual-use risk even though its result is null. We built it to test whether childhood writing predicts probable depressive symptoms at age 23. Others, however, could run the same code on childrenās writing today and misuse it for surveillance, or for triage with no clinician involved. Deployed models also risk two harms. First, a text model that learns to link gendered, era-specific words to the positive class would stereotype whole groups, a representational harm (Blodgett et al., 2020). Second, the baseline has a lower AUC for women than for men (Section 4.6), so using it to allocate care could be unfair to women, an allocative harm (Obermeyer et al., 2019). If a future model succeeds where ours did not, it should be released only under clinical governance, with a performance audit across subgroups and a clear statement of intended use. Data and code availability. We use three NCDS datasets held by the UK Data Service under its End User Licence (Safeguarded tier): UKDA-8313, UKDA-5566 and UKDA-5565 (Centre for Longitudinal Studies, 2024a; Centre for Longitudinal Studies, 2024b; Centre for Longitudinal Studies, 2024c). We cannot redistribute the data, but qualified researchers may apply to the service directly, provided they register and sign the licence. Appendix B specifies the pipeline, hyperparameters, checkpoints and NCDS item codes, so readers can audit the protocol. Our code will be released under the MIT licence when this paper is published. It contains no cohort data and no per-participant outputs, so users need the three datasets to run it. Model licences and intended use. The End User Licence forbids sending essay text to a third-party service. We sent none: every checkpoint is open-weight, downloaded from the Hugging Face Hub and run locally on institutional hardware. We redistribute no weights and use them only for non-commercial academic research, which all four zero-shot LLM licences permit. Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct are Apache-2.0, whereas Gemma-2-9B-it and Llama-3.1-8B-Instruct are gated and released under the Gemma Terms of Use and the Llama 3.1 Community License, which we accepted before download. Among the eleven encoders and frozen BGE, BERT-base (uncased and cased) and StressRoBERTa are Apache-2.0, and RoBERTa-base and -large, DeBERTa-v3-base and -large, ClinicalBERT and BGE are MIT. MentalBERT and MentalRoBERTa are C BY-NC 4.0, which permits our non-commercial use, but the BioBERT checkpoint we use states no licence on its model card. Cohort and data governance. NCDS enrolled every child born in England, Scotland and Wales during one week of March 1958 and has followed them since. The children wrote the essays in school in 1969, and the depositor transcribed them. The cohort is not representative of a present-day or ethnically diverse population, and we do not analyse ethnicity, which is not among our covariates. We run a secondary analysis under the End User Licence described above: we sought no consent ourselves, had no contact with participants, and make no claim about what the original consent covered. We accessed only de-identified data (Appendix B.1), made no attempt to re-identify participants, and quote no essay text. References Akiba et al. (2019) T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama Optuna: a next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, p. 2623ā2631. External Links: Document Cited by: Ā§B.2. Alqahtani et al. (2026) A. A. Alqahtani, E. Kayi, and M. T. Diab StressRoBERTa: cross-condition transfer learning from depression, anxiety, and PTSD to stress detection. In Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026), Rabat, Morocco, p. 305ā313. External Links: Link, Document Cited by: Table B2, Table 2, Ā§3.3. Alsentzer et al. (2019) E. Alsentzer, J. Murphy, W. Boag, W. Weng, D. Jin, T. Naumann, and M. McDermott Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, p. 72ā78. External Links: Document Cited by: Ā§3.3. Antonacci et al. (2026) C. Antonacci, J. P. Uy, K. Kwan, E. Giampetruzzi, S. Jones, J. W. Pennebaker, and I. H. Gotlib Natural language processing of youth speech predicts psychopathology across adolescence. Nature Mental Health 4 (8), p. 1227ā1238. External Links: Document Cited by: Ā§2, Limitations. Blodgett et al. (2020) S. L. Blodgett, S. Barocas, H. DaumĆ© I, and H. Wallach Language (technology) is power: a critical survey of ābiasā in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 5454ā5476. External Links: Link, Document Cited by: Ethical Considerations. Boehm et al. (2024) J. K. Boehm, F. Qureshi, and L. D. Kubzansky In the words of early adolescents: a novel assessment of positive psychological well-being predicts young adult depressive symptoms. Journal of Adolescent Health 74 (4), p. 713ā719. External Links: Document Cited by: Table A1, Ā§1, Figure 2, Ā§3.1, Ā§3.1, Ā§3.1, Ā§3.3, Ā§4.3, Limitations. Calvo et al. (2017) R. A. Calvo, D. N. Milne, M. S. Hussain, and H. Christensen Natural language processing in mental health applications using non-clinical texts. Natural Language Engineering 23 (5), p. 649ā685. External Links: Document Cited by: Ā§2. Centre for Longitudinal Studies (2024a) Centre for Longitudinal Studies National Child Development Study: Age 11, Sweep 2, āImagine you are 25ā Essays, 1969. Note: [data collection]. 1st Edition. University College London, UCL Institute of Education. UK Data Service. SN: 8313. DOI: 10.5255/UKDA-SN-8313-1 External Links: Document Cited by: Ā§3.1, Data and code availability.. Centre for Longitudinal Studies (2024b) Centre for Longitudinal Studies National Child Development Study: Age 23, Sweep 4, 1981, and Public Examination Results, 1978. Note: [data collection]. 2nd Edition. University of London, Institute of Education. UK Data Service. SN: 5566. DOI: 10.5255/UKDA-SN-5566-1 External Links: Document Cited by: Ā§3.1, Data and code availability.. Centre for Longitudinal Studies (2024c) Centre for Longitudinal Studies National Child Development Study: Childhood Data from Birth to Age 16, Sweeps 0-3, 1958-1974. Note: [data collection]. 3rd Edition. University of London, Institute of Education. UK Data Service. SN: 5565. DOI: 10.5255/UKDA-SN-5565-2 External Links: Document Cited by: Ā§3.1, Data and code availability.. Chen et al. (2026) S. Chen, T. Dang, M. Qian, H. Liang, D. T. Bedada, Q. A. Louw, A. Moore, R. N. Cardinal, T. J. Ford, and F. Jiang Machine learning and natural language processing for the identification of potential mental disorders among school-age children: a prospective birth cohort study. BMC Medicine 24, p. 413. External Links: Document Cited by: Ā§2. Christodoulou et al. (2019) E. Christodoulou, J. Ma, G. S. Collins, E. W. Steyerberg, J. Y. Verbakel, and B. Van Calster A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of Clinical Epidemiology 110, p. 12ā22. External Links: Document Cited by: Ā§2. Clark et al. (2007) C. Clark, B. Rodgers, T. Caldwell, C. Power, and S. Stansfeld Childhood and adulthood psychological ill health as predictors of midlife affective and anxiety disorders: the 1958 British Birth Cohort. Archives of General Psychiatry 64 (6), p. 668ā678. External Links: Document Cited by: Ā§3.1. Collins et al. (2024) G. S. Collins, K. G. M. Moons, P. Dhiman, R. D. Riley, A. L. Beam, B. Van Calster, M. Ghassemi, X. Liu, J. B. Reitsma, M. van Smeden, A. Boulesteix, J. C. Camaradou, L. A. Celi, S. Denaxas, A. K. Denniston, B. Glocker, R. M. Golub, H. Harvey, G. Heinze, M. M. Hoffman, A. P. Kengne, E. Lam, N. Lee, E. W. Loder, L. Maier-Hein, B. A. Mateen, M. D. McCradden, L. Oakden-Rayner, J. Ordish, R. Parnell, S. Rose, K. Singh, L. Wynants, and P. Logullo TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 385, p. e078378. External Links: Document Cited by: Ā§1, Ā§3.4. Ćƶltekin and Rama (2018) Ć. Ćƶltekin and T. Rama TĆ¼bingen-Oslo system: linear regression works the best at predicting current and future psychological health from childhood essays in the CLPsych 2018 shared task. arXiv preprint arXiv:1809.04838. External Links: Document Cited by: Ā§2. Coppersmith et al. (2015) G. Coppersmith, M. Dredze, C. Harman, and K. Hollingshead From ADHD to SAD: analyzing the language of mental health on Twitter through self-reported diagnoses. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, p. 1ā10. External Links: Document Cited by: Ā§1, Ā§2. De Choudhury et al. (2013) M. De Choudhury, M. Gamon, S. Counts, and E. Horvitz Predicting depression via social media. In Proceedings of the 7th International AAAI Conference on Weblogs and Social Media, p. 128ā137. External Links: Document Cited by: Ā§1, Ā§2. DeLong et al. (1988) E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44 (3), p. 837ā845. External Links: Document Cited by: Ā§3.4. Demler et al. (2012) O. V. Demler, M. J. Pencina, and R. B. DāAgostino Misuse of DeLong test to compare AUCs for nested models. Statistics in Medicine 31 (23), p. 2577ā2587. External Links: Document Cited by: Ā§B.3. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 4171ā4186. External Links: Document Cited by: Ā§1, Ā§3.3. Eichstaedt et al. (2018) J. C. Eichstaedt, R. J. Smith, R. M. Merchant, L. H. Ungar, P. Crutchley, D. PreoÅ£iuc-Pietro, D. A. Asch, and H. A. Schwartz Facebook language predicts depression in medical records. Proceedings of the National Academy of Sciences 115 (44), p. 11203ā11208. External Links: Document Cited by: Ā§1, Ā§2. Fisher et al. (2026) H. Fisher, N. M. Jaffe, K. Pidvirny, A. O. Tierney, M. S. Vaidean, P. Dongre, and C. A. Webb Language-based detection of depression with machine learning: systematic review and meta-analysis. npj Digital Medicine 9 (1), p. 273. External Links: Document Cited by: Ā§2. Galobardes et al. (2006) B. Galobardes, M. Shaw, D. A. Lawlor, J. W. Lynch, and G. Davey Smith Indicators of socioeconomic position (part 2). Journal of Epidemiology and Community Health 60 (2), p. 95ā101. External Links: Document Cited by: Ā§3.1. Gemma Team et al. (2024) Gemma Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, et al. Gemma 2: improving open language models at a practical size. arXiv preprint arXiv:2408.00118. External Links: Link, Document Cited by: Ā§3.3. Geurgas et al. (2025) R. Geurgas, S. J. Newman, E. T. Akimova, K. N. Thompson, and R. Wedow What machine learning teaches us about depression prediction across the life course: an exploratory comparison of predictive models. SSM - Population Health 32, p. 101886. External Links: Document Cited by: Ā§2. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. External Links: Link, Document Cited by: Ā§3.3. Guntuku et al. (2018) S. C. Guntuku, S. Giorgi, and L. Ungar Current and future psychological health prediction using language and socio-demographics of children for the CLPysch 2018 shared task. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, New Orleans, LA, p. 98ā106. External Links: Link, Document Cited by: Ā§2. Gururangan et al. (2020) S. Gururangan, A. MarasoviÄ, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith Donāt stop pretraining: adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 8342ā8360. External Links: Document Cited by: Ā§2, Ā§3.3, Ā§5. Hansen et al. (2023) L. Hansen, R. Rocca, A. Simonsen, L. Olsen, A. Parola, V. Bliksted, N. Ladegaard, D. Bang, K. TylĆ©n, E. Weed, S. D. Ćstergaard, and R. Fusaroli Speech- and text-based classification of neuropsychiatric conditions in a multidiagnostic setting. Nature Mental Health 1 (12), p. 971ā981. External Links: Document Cited by: Ā§2. He et al. (2023) P. He, J. Gao, and W. Chen DeBERTaV3: improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In Proceedings of the 11th International Conference on Learning Representations, External Links: Link Cited by: Ā§B.2. Huang (2025) Y. Huang MLstatkit. Note: Python package, version 0.1.91 External Links: Link Cited by: Ā§B.2. Ji et al. (2022) S. Ji, T. Zhang, L. Ansari, J. Fu, P. Tiwari, and E. Cambria MentalBERT: publicly available pretrained language models for mental healthcare. In Proceedings of the 13th Language Resources and Evaluation Conference, p. 7184ā7190. External Links: Link, Document Cited by: Ā§1, Ā§2, Ā§3.3. Kapoor and Narayanan (2023) S. Kapoor and A. Narayanan Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4 (9), p. 100804. External Links: Document Cited by: Ā§4.3. Kelley et al. (2022) S. W. Kelley, C. NĆ Mhaonaigh, L. Burke, R. Whelan, and C. M. Gillan Machine learning of language use on Twitter reveals weak and non-specific predictions. npj Digital Medicine 5 (1), p. 35. External Links: Document Cited by: Ā§2. Lee et al. (2020) J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36 (4), p. 1234ā1240. External Links: Document Cited by: Ā§3.3. Lin et al. (2023) Y. Lin, S. Chen, J. Liu, and C. Lin Linear classifier: an often-forgotten baseline for text classification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Toronto, Canada, p. 1876ā1888. External Links: Link, Document Cited by: Ā§2. Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. External Links: Link, Document Cited by: Ā§3.3. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations, External Links: Link Cited by: Ā§B.2. Lynn et al. (2018) V. Lynn, A. Goodman, K. Niederhoffer, K. Loveys, P. Resnik, and H. A. Schwartz CLPsych 2018 shared task: predicting current and future psychological health from childhood essays. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, New Orleans, LA, p. 37ā46. External Links: Link, Document Cited by: Ā§2. Nattino et al. (2014) G. Nattino, S. Finazzi, and G. Bertolini A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes. Statistics in Medicine 33 (14), p. 2390ā2407. External Links: Document Cited by: Ā§3.4. Nunez et al. (2024) J. Nunez, B. Leung, C. Ho, R. T. Ng, and A. T. Bates Predicting which patients with cancer will see a psychiatrist or counsellor from their initial oncology consultation document using natural language processing. Communications Medicine 4 (1), p. 69. External Links: Document Cited by: Ā§2. Obermeyer et al. (2019) Z. Obermeyer, B. Powers, C. Vogeli, and S. Mullainathan Dissecting racial bias in an algorithm used to manage the health of populations. Science 366 (6464), p. 447ā453. External Links: Document Cited by: Ethical Considerations. Paszke et al. (2019) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kƶpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, p. 8024ā8035. External Links: Link Cited by: Ā§B.2. Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Ć. Duchesnay Scikit-learn: machine learning in Python. Journal of Machine Learning Research 12, p. 2825ā2830. External Links: Link Cited by: Ā§B.2. Pepe et al. (2004) M. S. Pepe, H. Janes, G. Longton, W. Leisenring, and P. Newcomb Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. American Journal of Epidemiology 159 (9), p. 882ā890. External Links: Document Cited by: Ā§4.3. Power and Elliott (2006) C. Power and J. Elliott Cohort profile: 1958 British birth cohort (National Child Development Study). International Journal of Epidemiology 35 (1), p. 34ā41. External Links: Document Cited by: Ā§1. Riley et al. (2020) R. D. Riley, J. Ensor, K. I. E. Snell, F. E. Harrell, G. P. Martin, J. B. Reitsma, K. G. M. Moons, G. Collins, and M. van Smeden Calculating the sample size required for developing a clinical prediction model. BMJ 368, p. m441. External Links: Document Cited by: Ā§B.1. Rodgers et al. (1999) B. Rodgers, A. Pickles, C. Power, S. Collishaw, and B. Maughan Validity of the Malaise Inventory in general population samples. Social Psychiatry and Psychiatric Epidemiology 34 (6), p. 333ā341. External Links: Document Cited by: Ā§3.1. Roza et al. (2003) S. J. Roza, M. B. Hofstra, J. van der Ende, and F. C. Verhulst Stable prediction of mood and anxiety disorders based on behavioral and emotional problems in childhood: a 14-year follow-up during childhood, adolescence, and young adulthood. American Journal of Psychiatry 160 (12), p. 2116ā2121. External Links: Document Cited by: Ā§3.1. M. Rutter, J. Tizard, and K. Whitmore (Eds.) (1970) M. Rutter, J. Tizard, and K. Whitmore (Eds.) Education, health and behaviour. Longman, London. External Links: Link Cited by: Ā§3.1. Schenker and Gentleman (2001) N. Schenker and J. F. Gentleman On judging the significance of differences by examining the overlap between confidence intervals. The American Statistician 55 (3), p. 182ā186. External Links: Document Cited by: Ā§4.2. Seabold and Perktold (2010) S. Seabold and J. Perktold Statsmodels: econometric and statistical modeling with Python. In Proceedings of the 9th Python in Science Conference, p. 92ā96. External Links: Document Cited by: Ā§B.2. Seinen et al. (2022) T. M. Seinen, E. A. Fridgeirsson, S. Ioannou, D. Jeannetot, L. H. John, J. A. Kors, A. F. Markus, V. Pera, A. Rekkas, R. D. Williams, C. Yang, E. M. van Mulligen, and P. R. Rijnbeek Use of unstructured text in prognostic clinical prediction models: a systematic review. Journal of the American Medical Informatics Association 29 (7), p. 1292ā1302. External Links: Document Cited by: Ā§2. Steyerberg et al. (2010) E. W. Steyerberg, A. J. Vickers, N. R. Cook, T. Gerds, M. Gonen, N. Obuchowski, M. J. Pencina, and M. W. Kattan Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21 (1), p. 128ā138. External Links: Document Cited by: Ā§3.4, Ā§4.3. Sun and Xu (2014) X. Sun and W. Xu Fast implementation of DeLongās algorithm for comparing the areas under correlated receiver operating characteristic curves. IEEE Signal Processing Letters 21 (11), p. 1389ā1393. External Links: Document Cited by: Ā§3.4. Thapar et al. (2022) A. Thapar, O. Eyre, V. Patel, and D. Brent Depression in young people. The Lancet 400 (10352), p. 617ā631. External Links: Document Cited by: Ā§1. van den Goorbergh et al. (2022) R. van den Goorbergh, M. van Smeden, D. Timmerman, and B. Van Calster The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression. Journal of the American Medical Informatics Association 29 (9), p. 1525ā1534. External Links: Document Cited by: Ā§3.1. Weigl (2024) M. Weigl pycaleva: a framework for calibration evaluation of binary classification models. Note: Python package, version 0.8.2 External Links: Link Cited by: Ā§B.2. Wibaek et al. (2023) R. Wibaek, G. S. Andersen, C. C. Dahm, D. R. Witte, and A. Hulman Large language models for epidemiological research via automated machine learning: case study using data from the British National Child Development Study. JMIR Medical Informatics 11, p. e43638. External Links: Document Cited by: Ā§2. Wolf et al. (2020) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p. 38ā45. External Links: Document Cited by: Ā§B.2. Wolfram (2025) T. Wolfram Large language models predict cognition and education close to or better than genomics or expert assessment. Communications Psychology 3 (1), p. 95. External Links: Document Cited by: Ā§2. Xiao et al. (2024) S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, and J. Nie C-Pack: packed resources for general Chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 641ā649. External Links: Document Cited by: Ā§3.3. Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. External Links: Link, Document Cited by: Ā§3.3. Appendix A Supporting results A.1 Is the baselineās lead robust? The baselineās lead is robust. Although the encodersā ranking changes over five fine-tuning runs per encoder, each with a different random seed, none of the 35 runs reaches the baselineās AUC of 0.7370.737 (best run: 0.6920.692). Averaging each encoderās five runs into a seed ensemble lifts its AUC above its five-run mean, to between 0.6500.650 and 0.6790.679, still below the baseline. On five new stratified test splits, the refitted baseline leads BERT-base every time, by 0.0850.085 on average (range 0.0640.064 to 0.0960.096). Training on graphics processing units (GPUs) is not fully deterministic, so we repeated all runs on another machine (except the ClinicalBERT and BioBERT runs and the adapted runs of the four encoders that appear only in Table 6). Individual predictions changed, but every five-run mean stayed within a tolerance fixed in advance (1.2651.265 times its reported SD). Every seed-ensemble interval for the baselineās lead over an encoder of Table 2 still excluded zero, and on five new splits the baseline again led BERT-base, by 0.0940.094 on average. Model Parse Odds ratio per SD [95% CI] Qwen2.5-7B 0.9990.999 0.906 [0.803, 1.022] Qwen2.5-32B 1.0001.000 0.881 [0.782, 0.994] Gemma-2-9B 1.0001.000 0.930 [0.825, 1.048] Llama-3.1-8B 1.0001.000 0.948 [0.839, 1.070] Table A1: Zero-shot LLM well-being ratings. Parse: share of the 7,2267,226 essays whose reply parsed. Odds ratio: for the outcome per SD of the rating, adjusted for the six covariates (unpenalised fit on the training rows, Wald 95% CI). Below 11, a higher rating goes with lower odds. Boehm et al. (2024) report 0.870.87 [0.78,0.98][0.78,0.98] per SD of hand-coded well-being, in a differently restricted sample, also adjusted for word count. A.2 Does task-adaptive pretraining help? Task-adaptive pretraining (Appendix B.2) slightly raises the five-run AUC of all seven encoders in Table 6, but no gain survives Bonferroni correction. Paired t-tests over the five seeds, which capture only seed variation on this one test set, give the lowest p values for RoBERTa-base (0.0110.011) and DeBERTa-v3-base (0.0210.021), or 0.0750.075 and 0.150.15 when corrected over all seven encoders. The baseline leads the seed-ensembled adapted encoders by 0.0410.041 (RoBERTa-base, p=0.06p=0.06) to 0.0570.057 in paired DeLong tests, but its lead survives correction (0.05/70.05/7) only against BERT-base and RoBERTa-large (both p=0.006p=0.006). Appendix B Data, implementation and analysis B.1 Data Variable What it measures (NCDS code) Reported by Range Age-11 target Malaise total (outcome) derived total mal of 24 yes/no questions on emotional and bodily symptoms (n6016 to n6039) participant, age 23 0ā20 outcome: positive if ā„8ā„ 8 Sex female or male (n622) cohort records 0 or 1 female Fatherās social class fatherās or male headās occupation: non-manual, manual or no male head (n1685, 1966 General Register Office scheme) parent, age 11 1ā3 not used Parental psychiatric history psychiatric condition coded for the mother (n1406, n1407) or father (n1415, n1416) among chronic or serious illnesses since age 7, other codes 0 parent, age 11 0 or 1 not used Cognitive ability items correct of 80 (40 verbal, 40 non-verbal) on a 30-minute general ability test (n920) test, age 11 0ā79 low: ā¤37ā¤ 37 (bottom third) BSAG internalising sum of depression (n980), withdrawal (n977), unforthcomingness (n974) and writing off adults (n989) teacher, age 11 0ā31 high: ā„10ā„ 10 (top tenth) BSAG externalising sum of hostility towards adults (n986), restlessness (n998) and anxiety for acceptance by adults (n983) teacher, age 11 0ā24 high: ā„5ā„ 5 (top tenth) Essay the life the child imagined at 25, transcribed (UKDA-8313), 201201 words on average child, age 11 free text text modelsā only input Table B1: Variables. Rows 2 to 7 are the covariate baselineās inputs, standardised (sex: 1 for female, social class: one ordinal term). Range: in the analytic sample. A BSAG score counts the behaviour descriptions the teacher underlined. Last column: targets of Table 5, cut at training-row quantiles, not clinical thresholds. Essays and de-identification. The children wrote the essays by hand in school, in 30 minutes. Transcribers kept the childrenās spelling, grammar and punctuation, and those working in 2016 and 2017 marked doubtful words and illegible letters with asterisks. Names and small places became labels such as [name]. We link each transcription to the other two datasets by participant identifier, keep these marks, correct no spelling and strip only the file markup, the identifier header and the word-count line. The depositor de-identified the transcriptions, and we add no identifier scan or offensive-content filter. Selection and sample size. An unpenalised logistic regression predicts which of the 9,3339,333 covariate-complete essay writers (Table 1) have the age-23 outcome recorded. It uses essay length (log of one plus the word count) and the six covariates, with social class as two indicators, and has an in-sample AUC of 0.6000.600. Girls (odds ratio 1.221.22) and children with higher ability (1.181.18 per SD) are more likely to have the outcome recorded, while children with higher behaviour ratings, a parental psychiatric history or no male head are less likely. Essay length makes no detectable difference (odds ratio 1.021.02 per SD, 95% CI [0.96,1.07][0.96,1.07]). Each BSAG sum needs all its items, but our file keeps a partial internalising sum for one writer outside the analytic sample, so this model and row 3 of Table 1 count one participant too many. Finally, the sample exceeds the minimum size (Riley et al., 2020) of 1,5631,563 for the baseline (six predictor parameters, 6.9%6.9\% prevalence, assumed AUC 0.700.70). B.2 Models and training Learning rate 2Ć10ā52Ć 10^-5 Weight decay 0.01 Batch size 16 Warmup 10% Train epochs 6 Rate decay linear Early-stop patience 3 epochs Max gradient norm 1.0 Table B2: Six-epoch recipe: settings from StressRoBERTa (Alqahtani et al., 2026), except our warmup, rate decay and clipping. It covers the primary-seed runs, the five-split refit and the learning curve. Five-run refits use a longer schedule (10 epochs, rate 10ā510^-5) and cross-fitting a shorter one (Appendix B.3). The two large encoders use batch size 8 in training and validation. Fine-tuning and software. We fine-tune all parameters with the AdamW optimiser (Loshchilov and Hutter, 2019) and, over the first token, a two-output classification head whose softmax probability of a positive outcome is the predicted risk, with no recalibration. Of the 7,2267,226 essays, 3.3%3.3\% to 4.3%4.3\% exceed 512 tokens, depending on the tokeniser. BioBERT and ClinicalBERT ship no tokeniser configuration, so we load them cased, as they were pretrained. We use seed 42 as the primary seed and for the test split, 43 for the validation split, 1 to 4 for further runs and 1 to 5 only for the five-split refitās test splits. We pin scikit-learn 1.9.0 (Pedregosa et al., 2011), PyTorch 2.3.1 (Paszke et al., 2019), transformers 4.45.2 (Wolf et al., 2020), MLstatkit 0.1.91 (Huang, 2025, DeLong tests,), Optuna 4.9.0 (Akiba et al., 2019), pycaleva 0.8.2 (Weigl, 2024, calibration belt,) and statsmodels 0.14.6 (Seabold and Perktold, 2010). Each encoder run uses one V100 GPU (16 GB for the seven encoders of Table 2, 32 GB for the other four). Each 7B-to-9B LLM runs on one 32 GB V100, and Qwen2.5-32B on four. Model checkpoints. Table B3 lists every Hugging Face checkpoint. Hugging Face model Fine-tuned encoders (Table 2) bert-base-uncased roberta-base emilyalsentzer/Bio_ClinicalBERT dmis-lab/biobert-base-cased-v1.1 mental/mental-bert-base-uncased mental/mental-roberta-base Amalq/stress-roberta-base Further encoders (Table 6) bert-base-cased microsoft/deberta-v3-base (184M) microsoft/deberta-v3-large (434M) roberta-large (355M) Frozen embeddings (with bert-base-uncased) BAAI/bge-large-en-v1.5 Zero-shot LLMs (Figure 2) Qwen/Qwen2.5-7B-Instruct Qwen/Qwen2.5-32B-Instruct google/gemma-2-9b-it meta-llama/Llama-3.1-8B-Instruct Table B3: Model checkpoints. The seven encoders of Table 2 have 108108 to 125125 million parameters. Task-adaptive pretraining. We further pretrain each adapted encoder once by masked language modelling, with no outcome labels, on 9,0639,063 essays: the training and validation rows plus the 3,2833,283 outside the analytic sample, excluding test essays by participant identifier. We run twenty epochs at learning rate 5Ć10ā55Ć 10^-5, batch size 8 and mask probability 0.150.15, monitor the loss on 500500 held-out essays and keep the last epoch. DeBERTa-v3 (He et al., 2023) was pretrained by replaced-token detection, and transformers 4.45.2 does not load weights for its masked-language-model head, which therefore starts from random weights. DeBERTa-v3ās stock and adapted AUCs are thus less comparable than the other encodersā. The adapted encoders are fine-tuned at the same five seeds and on the same schedule as the stock ones but have seen more text, including the validation essays that select their best epoch. TF-IDF and frozen embeddings. We fit TF-IDF on the training rows only, with word 1ā2-grams, 20,00020,000 features, minimum document frequency 2, sublinear weighting and English stopwords removed. Tied n-gram counts at the vocabulary cut-off make its AUC vary across machines by up to about 0.0050.005. Frozen BERT uses the first-token ([CLS]) vector of each essayās first 512 tokens, whereas frozen BGE averages over the same tokens despite its model card recommending [CLS]. Zero-shot LLMs. Figure 2ās line breaks and columns are layout only. We cap each essay at its first 1,2001,200 words and decode greedily (at most 6464 new tokens), otherwise keeping each checkpointās shipped generation settings (Qwen2.5: repetition penalty 1.051.05). A reply parses only if all four ratings are on their 1-to-5 scale (parse rates: Table A1), and an unparsed essay takes the training-set mean of the averaged rating. A univariate logistic regression fitted on the training rows maps the rating to predicted risk. B.3 Analysis protocols Intervals, tests and calibration. Bootstrap replicates resample positives and negatives separately, 95% CIs are percentile intervals, and each Ī compares both models on the same resampled rows. DeLong tests are two-sided, and AUC-PR is scikit-learnās average precision. For the sex gap of Section 4.6, we resample the test rows of men (695695) and women (751751) independently. The men-minus-women AUC gap is 0.1530.153 [0.056,0.242][0.056,0.242] for the baseline and 0.1360.136 [ā0.004,0.264][-0.004,0.264] for MentalBERT on the primary seed. The Spearman correlation of Section 4.5 relates the baselineās predicted risk to the full Malaise score on the test rows. The slope in Figure 5 comes from a logistic regression of the outcome on the logit of the predicted risk (a slope above 11 means predicted risks spread too little). At tāt^* the baseline flags 211211 test rows, 3737 of them positive (recall 0.3740.374). Incremental value. For Section 4.3, we fit the baseline on the training rows with and without one added text score. A trained modelās score enters as the logit of its predicted risk, whereas a zero-shot LLMās averaged rating enters standardised on the training rows. Ī and DeLong tests compare the two L2-penalised fits (C=1C=1) on the test rows. DeLong tests can be conservative for nested models fitted and tested on the same rows (Demler et al., 2012), so a likelihood-ratio test also compares unpenalised refits on the training rows. Each cross-fitting run, per fold or final, uses the shorter schedule (four epochs at rate 10ā510^-5) and selects its epoch on a stratified 20% of its rows. We cross-fit TF-IDF and the frozen embeddings the same way. The cross-fitted TF-IDF keeps 10,00010,000 features, drops terms in fewer than 5 or over 95% of essays, and uses no sublinear weighting or stopword removal. Alone, the three cross-fitted scores reach AUC 0.6240.624 (TF-IDF), 0.5860.586 (frozen BERT) and 0.6340.634 (frozen BGE). Outcome cut 6 7 8 9 10 Test positives 218 154 99 59 37 Baseline, relabelled 0.690 0.712 0.737 0.733 0.739 Baseline, refitted 0.693 0.714 0.737 0.734 0.746 Baseline minus model AUC-ROC, both relabelled BERT-base 0.043 0.051 0.068 0.069 0.048ā RoBERTa-base 0.068 0.086 0.117 0.104 0.096 ClinicalBERT 0.052 0.064 0.070 0.064ā 0.033ā BioBERT 0.051 0.062 0.066 0.059ā 0.029ā MentalBERT 0.072 0.095 0.118 0.122 0.094ā MentalRoBERTa 0.049 0.065 0.090 0.085 0.086ā StressRoBERTa 0.043 0.063 0.076 0.056ā 0.058ā TF-IDF 0.073 0.080 0.092 0.088 0.094 Table B4: Primary-seed test AUC-ROC by outcome cut for models trained at cut 8 (relabelled) or at the cut (refitted). ā Paired DeLong pā„0.05pā„ 0.05 (uncorrected). Learning curve and learning-rate sweep. The learning curve (Figure 6) refits the baseline, BERT-base and MentalBERT on the same stratified training subsamples of six sizes (578578 to 4,6244,624 rows). Each seed (42 and 1 to 4) draws one subsample for all three models and initialises the encoder runs, which select their epoch on all 1,1561,156 validation rows. For MentalBERT, 12 five-epoch Optuna trials search learning rates in [10ā6,10ā4][10^-6,10^-4] (log scale, tree-structured Parzen estimator, first ten drawn at random). The best rate, 5.4Ć10ā55.4Ć 10^-5, raises validation AUC from 0.5880.588 (six-epoch recipe, Table B2) to 0.6310.631, against the baselineās 0.7080.708.