Paper deep dive
TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment
Merna Bibars, Bolaji Omofojoye, Allan I. Levey, Rachel Hershenberg, Gari D. Clifford, Hyeokhyen Kwon
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Depression and anxiety in older adults with Mild Cognitive Impairment (MCI) are frequently underdiagnosed due to limited access to care. Multimodal analysis of remote clinical interviews is a scalable screening approach, but existing methods have three limitations. First, they do not correct temporal misalignment across multimodal features extracted at different resolutions, inducing spurious cross-modal associations. Second, remote recordings exhibit uneven modality dropout, but missing values are often zero-filled, making them indistinguishable from valid near-zero measurements. Finally, they do not jointly attribute predictions to modalities, questions, and interview moments, limiting fine-grained clinical interpretation. We propose a Temporally-Aligned, Missingness-Aware, Interpretable (TAMI) multimodal fusion framework. TAMI aligns speech, language, facial, and physiological features within question-answer segments on a shared timeline, encodes modality-level missingness over time, and conditions fusion on question context. In interviews with 49 older adults with MCI, TAMI achieved area under the receiver operating characteristic curve (AUROC) scores of 0.68 (depression) and 0.69 (anxiety). Fine-grained temporal alignment of multimodal features produced the largest performance gain ($\Delta{\geq}0.1$). Multi-level interpretability analysis revealed that depression classification relied on eyegaze and open-ended questions, while anxiety classification depended on eyegaze and head pose, with attribution uniformly distributed across questions. Using only responses to the open-ended questions (5.1min), the depression model achieved an AUROC score of 0.67, which was not significantly different from using the full interview (19min) ($p>0.05$). Our findings support designing interview protocols centered on open-ended questions for depression screening in older adults with MCI.
Tags
Links
- Source: https://arxiv.org/abs/2608.30857v1
- Canonical: https://arxiv.org/abs/2608.30857v1
Trouble viewing inline? Open PDF directly →
Full Text
114,009 characters extracted from source content.
Expand or collapse full text
TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment Merna Bibars Bolaji Omofojoye Allan I. Levey Rachel Hershenberg Gari D. Clifford Hyeokhyen Kwon †thanks: Hyeokhyen Kwon, Allan I. Levey, Rachel Hershenberg, Bolaji Omofojoye, and Gari Clifford are partially funded by the National Institute on Deafness and Other Communication Disorders (grant # 1R21DC021029-01A1). The Cognitive Empowerment Program is supported by a generous investment from the James M. Cox Foundation and Cox Enterprises, Inc. The authors thank the participants of the Charlie and Harriet Shaffer Cognitive Empowerment Program at Emory University. †thanks: Merna Bibars is with the Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, USA, and the Department of Systems and Biomedical Engineering, Cairo University, Giza, Egypt.†thanks: Bolaji Omofojoye is with the Department of Biomedical Informatics, Emory University, Atlanta, GA, USA.†thanks: Allan I. Levey is with the Department of Neurology, Emory University, Atlanta, GA, USA.†thanks: Rachel Hershenberg is with the Department of Psychiatry and Behavioral Sciences, Emory University, Atlanta, GA, USA.†thanks: Gari D. Clifford is with the Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, and the Department of Biomedical Informatics, Emory University, Atlanta, GA, USA.†thanks: Hyeokhyen Kwon is with the Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, and the Department of Biomedical Informatics, Emory University, Atlanta, GA, USA. Abstract Depression and anxiety in older adults with Mild Cognitive Impairment (MCI) are frequently underdiagnosed due to limited access to care. Multimodal analysis of remote clinical interviews is a scalable screening approach, but existing methods have three limitations. First, they do not correct temporal misalignment across multimodal features extracted at different resolutions, inducing spurious cross-modal associations. Second, remote recordings exhibit uneven modality dropout, but missing values are often zero-filled, making them indistinguishable from valid near-zero measurements. Finally, they do not jointly attribute predictions to modalities, questions, and interview moments, limiting fine-grained clinical interpretation. We propose a Temporally-Aligned, Missingness-Aware, Interpretable (TAMI) multimodal fusion framework. TAMI aligns speech, language, facial, and physiological features within question-answer segments on a shared timeline, encodes modality-level missingness over time, and conditions fusion on question context. In interviews with 49 older adults with MCI, TAMI achieved area under the receiver operating characteristic curve (AUROC) scores of 0.68±0.040.68± 0.04 (depression) and 0.69±0.090.69± 0.09 (anxiety). Fine-grained temporal alignment of multimodal features produced the largest performance gain (Δ≥0.1 ≥0.1 AUROC). Multi-level interpretability analysis revealed that depression classification relied on eyegaze and open-ended questions, while anxiety classification depended on eyegaze and head pose, with attribution uniformly distributed across questions. Using only responses to the open-ended questions (5.1 min), the depression model achieved an AUROC score of 0.670.67, which was not significantly different from using the full interview (19 min) (p>0.05p>0.05). Our findings support designing interview protocols centered on open-ended questions for depression screening in older adults with MCI. Index Terms: Anxiety, depression, explainable artificial intelligence, mild cognitive impairment, multimodal fusion. I Introduction Fig. 1: Temporal misalignment in multimodal behavioral feature processing in existing work. Conventionally, acoustic features are extracted with overlapping sliding windows (2s window, 1s hop), e.g., W1W_1 (spanning 0–2s), while facial features are frame-level observations sampled at 1 fps. (a) Existing work uses sequence-index pairing to associate feature time series by their position in each series rather than by recording time, linking each acoustic window to the single facial frame at the same index (dotted orange line). This produces coverage misalignment in behavioral analysis. (b) Our proposed multimodal temporal binning approach assigns features according to their temporal support based on the portion of the behavioral recording that each feature represents. Each acoustic window is associated with the facial frames occurring within the same time interval (dotted teal lines). Such temporal alignment of multimodal behavioral features is critical for capturing fine-grained behavioral congruence during remote mental health interviews. Mild Cognitive Impairment (MCI), a clinical condition that precedes Alzheimer’s disease and related dementias, is projected to affect 21 million adults in the United States (U.S.) by 2060 [1]. It is characterized by cognitive decline greater than expected for healthy aging, while preserving daily functioning [2]. Depression affects 32% of adults with MCI [3, 4], while anxiety affects 10–50%, and both are associated with increased risk of progression to dementia [5]. Despite their clinical relevance, they remain underdiagnosed [6, 7], and older adults in the U.S. face multiple barriers to mental health care [8]. Diagnosis is further complicated by atypical late-life presentations: “depression without sadness” [9] and anxiety profiles poorly captured by criteria developed from younger populations [10]. (a) Temporal displacement (b) Visual degradation from occlusion, blur, and overexposure Fig. 2: Challenges in multimodal interview analysis. (a) Existing methods associate misaligned behavioral features with temporal displacements of up to 22 seconds (more than 100 times the 200 ms window over which clinicians perceptually bind cross-modal behavioral cues). (b) Synthetic illustrations of visual degradation in remote interviews. Privacy constraints prevent the display of participant recordings, so we generate illustrative frames using OpenAI ChatGPT Images 2.0, accessed via ChatGPT [11], to reproduce artifacts observed in the real data, including face occlusion, low resolution with overexposure, and blur. Clinical assessment of mental health depends on behaviors that are hard to observe in real time. For example, depression-related changes in speech and eye movements lie in the hundreds-of-milliseconds range [12, 13], too brief to be reliably observed by a clinician during an interview or documented in a clinical report [12]. Assessment also relies on affect, where the clinician judges whether a patient’s nonverbal behavior is congruent with their verbally stated mood [14]. Depressed and anxious individuals may smile and not show sadness while describing sad experiences [15] and anxious events [16]. Reading this congruence requires interpreting modalities jointly and at the moment they co-occur. Human perception binds audiovisual speech into a single event only within a window of 200 ms [17], and behavioral cues separated by intervals longer than 200 ms are unlikely to be perceived as a single event. Therefore, detecting congruence across modalities requires fine-grained and temporally aligned multimodal measurement. Remote video interviews provide a practical setting for measuring these behavioral signals, given that telehealth use among older adults has increased after the COVID-19 pandemic [18] and that an estimated 140 million Americans have limited access to mental health care [19]. Recent work has shown that multimodal machine learning (ML) methods can objectively quantify facial, vocal, language, and physiological cues from video interviews [20, 21], providing measures that may support scalable screening of individuals at risk for psychiatric episodes or suicidal behavior, facilitate earlier triage, and complement clinician assessments [22]. However, prior multimodal ML approaches for remote video interviews remain limited in their ability to analyze multimodal behavioral time series associated with mental health in MCI. Multimodal features are extracted at different temporal resolutions and supports (time intervals over which a feature is computed): window-level features summarize signal content over an interval (e.g., acoustic or physiological features extracted with sliding windows over several seconds), frame-level features are measured at individual timestamps (e.g., facial features extracted at 1 frame per second (fps)), and word-level language features occupy variable temporal spans. Existing approaches temporally associate multimodal features by sequence-index pairing, aligning the features by their sequence position in each extracted feature time series rather than by the timestamps to which they correspond in the original recording [23, 24, 25, 26, 27, 28]. Consequently, this sequence-index pairing creates temporal coverage misalignment (or displacement) across multimodal behavioral features, in which a multi-second window-level feature (e.g., acoustic) is associated with a single frame-level observation (e.g., facial), despite the window covering multiple observations in time, as shown in Fig. 1(a). Fig. 2(a) also shows this issue for an example video: across a segment of consecutive indices, the acoustic, head pose, and eyegaze features assigned to the same index (e.g., 715) were recorded up to 22 seconds apart, more than 100 times the 200 ms window within which humans perceptually bind cross-modal behavioral cues [17]. This causes the model to fuse features from different time segments of the response, thereby learning misaligned behavioral co-occurrences. In contrast, a temporally aligned representation fuses concurrent behavioral cues, supporting clinically meaningful interpretation across frame-, word-, and window-level behavioral features, as shown in Fig. 1(b). Interpretability is necessary for clinicians to trust a model’s prediction [29]. It requires attributing a prediction across modalities, questions, and time, so a clinician can identify which behavioral signals, in response to which questions, and at which moments in the interview drove the prediction. Comprehensive semi-structured interviews that combine open-ended questions with multiple screening instruments may require substantial administration time. In our protocol, interviews lasted from 21 to 67 minutes. Therefore, identifying which portions of the interview contain sufficient predictive evidence may inform shorter protocols that reduce clinician time and participant burden. These question- and time-level attributions are meaningful only when the modalities share a common timeline, making fine-grained cross-modal temporal alignment a prerequisite for interpretation. Remote video recordings collected in real-world settings also exhibit missingness across modalities over time. Participants complete interviews under varying lighting conditions and camera placements so that the video may include blur and occlusion (Fig. 2(b)), and unstable internet connectivity may cause frames to drop [30]. Under these conditions, feature extraction may fail at specific timesteps across the modalities [28, 20]. To handle missingness, previous studies zero-filled missing values [31], which makes true absence indistinguishable from a valid near-zero measurement. A model must therefore explicitly encode feature missingness. Furthermore, participant answers are also question-dependent: a brief yes/no answer can carry different clinical meanings depending on the prompt that elicited it. The answer to a question is the co-occurring cross-modal behavior described above, so the question is more informative when it conditions the multimodal sequence at the moment the behavior occurs. Prior work on mental health prediction in the MCI population does not incorporate the question context [21, 32, 33]. Some of the existing depression prediction approaches incorporate question context, but do not apply this conditioning jointly across modalities at a fine-grained temporal resolution. One approach combines the question with temporally pooled text and audio representations for each answer segment [25]. Other approaches condition only a subset of the modalities on the question. For example, Guo et al. [23] condition only the text modality on the question in their multimodal approach, while Niu et al. [34] condition audio and text in separate models before fusion. As a result, current approaches do not allow the question to influence the joint cross-modal behavior at the moment it occurs during modeling. We propose TAMI, a Temporally Aligned, Missingness-Aware, and Interpretable multimodal fusion framework for predicting depression and anxiety in older adults with MCI from remote interviews. TAMI integrates fine-grained cross-modal temporal alignment of features with heterogeneous sampling rates, modality-level missingness over time, and question conditioning. I Related Work I-A Multimodal Mental Health Interview Analysis Automated prediction of depression and anxiety from recorded interviews has been studied primarily in general adult populations using datasets such as DAIC-WOZ [35, 36, 37, 28, 25]. In contrast, interview-based pipelines in older adults with MCI have largely targeted cognitive assessment [38, 39, 40], whereas mental health prediction in this population remains underexplored [32, 33, 21]. Across both populations, multimodal approaches combine facial, acoustic, language, and physiological remote photoplethysmography (rPPG) features through late, early, or mid-fusion [41, 42, 31, 24, 26, 28]. Some combine facial with acoustic or language features but omit rPPG [43, 44, 45], while others include rPPG but omit eyegaze and head pose [31, 21]. We address these gaps with a multimodal deep learning model that predicts depression and anxiety in older adults with MCI from facial action units and landmarks, head pose, eyegaze, acoustic, language, and rPPG features collectively. I-B Cross-Modal Temporal Alignment Prior multimodal approaches handle temporal misalignment across features in one of two ways. The first approach temporally pools each feature time series into a single vector per feature before multimodal fusion [23, 24, 25]. The second approach keeps per-feature time series but places each feature observation at a single point on the time axis [26, 27, 28], as illustrated in Fig. 1(a). These methods do not provide a shared timeline that aligns frame-level, window-level, and word-level features according to their temporal support. In contrast, our temporal binning approach preserves the time interval covered by each feature by constructing a shared time axis that reflects the original recording times. This allows the model to learn temporally aligned cross-modal interactions, reflecting how clinicians assess patient behaviors in real-world clinical settings. I-C Handling Missingness across Modality and Time Most prior multimodal mental health work handles missingness at the modality level for each sample, treating whether a modality is present for the participant as a whole [31, 46]. Other studies replace missing values at the timestep level with zeros, which cannot be distinguished from valid zero values in observed timesteps [31, 27]. Gimeno-Gomez et al. [28] encode missingness more finely, with per-modality, per-frame presence masks that act as attention masks for depression prediction. We build on this fine-grained masking strategy but place the masks on answer-aligned temporal bins. As a result, at each timestep, the multimodal features, their temporal availability masks, and the corresponding question context share the same timeline. I-D Modeling Question and Answer Interaction Prior work on mental health prediction in older adults with MCI ignores the question context in multimodal modeling and focuses only on the participant answer [21, 32, 33]. In general-population depression classification, some methods condition the multimodal response on the question but in a limited way. Zhang et al. [25] temporally pool the acoustic response time series and text into a single vector for each modality, then fuse them with the question using gated addition. This discards fine-grained temporal information. Other approaches apply question conditioning separately to each modality rather than to a joint multimodal representation. Niu et al. [34] use additive attention to condition the response acoustic time series and text sequence independently on the question before fusion. Guo et al. [23] condition only the text modality on the question context by encoding the concatenated question and response text with RoBERTa [47], while the acoustic response is encoded independently before fusion. As a result, these methods do not condition the fused multimodal sequence on the question at a fine-grained temporal resolution within each answer. In addition, their conditioning incorporates only the acoustic and text modalities. In contrast, we add the question embedding to every timestep of the fused multimodal sequence, which also includes facial and physiological signals, allowing each moment of the answer to be interpreted in the context of the question that elicited it. Fig. 3: Adaptive temporal binning for within-answer multimodal alignment. (a) Each participant’s recording is partitioned along the time axis into question-answer segments, as shown on the upper axis. Each segment is divided into a number of temporal bins depending on its duration, as shown on the lower axis. The bins define a shared timeline that aligns features of different temporal supports: frame-level features (e.g., eyegaze) are assigned to a bin by timestamp, and window-level features (e.g., acoustic) are assigned to every bin they overlap. (b) Bins from all answers are concatenated into a participant-level sequence, with a binary modality-timestep mask indicating observed and missing entries. I-E Interpreting Predictions across Modality, Question, and Time Multimodal mental health interviews produce structured evidence across modalities, questions, and time, with multiple behavioral signals unfolding throughout the interview in response to specific prompts. Prior interpretability methods examine only a subset of these levels. Modality-level methods identify which modalities drive a prediction through performance comparisons [41, 20] or post-hoc attribution scores [31, 21], without localizing importance to specific questions or moments in time. Question-level methods attribute importance to interview questions or to modalities within a question through attention [23, 24] or performance comparisons [48], but do not identify when within an answer the relevant evidence is observed. Gimeno-Gomez et al. apply Integrated Gradients (IG) [49] to visualize modality importance over time within a selected short temporal window from a single participant’s video rather than across the full interview [28]. They do not incorporate question structure or extend the analysis to cohort-level summaries for modality and temporal importance [28]. No prior method derives modality-level, question-level, and interview-wide temporal importance jointly across participants. We address this gap by deriving all three levels of explanation from IG [49] attributions at both cohort and individual levels over the full interview. I Method TAMI processes interviews in three stages: (1) temporally aligned multimodal feature time series with per-modality-feature per-timestep missingness masks; (2) question-conditioned multimodal feature fusion; and (3) post-hoc multi-level attribution that quantifies the importance of each modality, question, and timestep. I-A Problem Formulation Each participant i∈1,…,Ni∈\1,…,N\ provides a remote interview recording represented as a sequence of observed question-answer segments, i=(qij,aij)j=1Ji,X_i=\(q_ij,a_ij)\_j=1^J_i, where qijq_ij is the j-th question answered by participant i and aija_ij is the corresponding answer. Because participants may not answer all interview questions, JiJ_i can vary across participants. Each answer spans a time interval of duration LijL_ij, which we partition into BijB_ij temporal bins. The bins form a shared timeline that aligns features of different temporal supports. For modality-feature m∈ℳm with feature dimension DmD_m, bin bij,kb_ij,k with k∈1,…,Bijk∈\1,…,B_ij\ corresponds to a feature vector ij,k(m)∈ℝDmx_ij,k^(m) ^D_m and a binary availability mask Mij,k(m)∈0,1M_ij,k^(m)∈\0,1\, where 0 indicates that the observation is missing. Given an interview, the task is to predict a binary label, representing the presence or absence of depression or anxiety. Separate models are trained for each task. I-B Temporally Aligned Representation I-B1 Feature Extraction Throughout the paper, we use modality-feature to refer to each of the eight feature time series used as model inputs. The features are extracted from the participant-side audiovisual recordings. From the facial video, we extract frame-level features at 1 fps: combined facial action units and landmarks (AUs & LMs), head pose [50], and eyegaze [51]. Physiological rPPG-derived heart rate is computed as a window-level feature with a sliding window of 6s and 1s hop [52]. From the audio, we extract window-level acoustic features with a sliding window of 2s and 1s hop: eGeMAPS, ComParE [53], and Wav2Vec2 embeddings [54, 55]. The language features are RoBERTa embeddings from WhisperX word-level transcripts [56, 47]. Interview questions are encoded separately using Sentence-BERT [57, 55] and used to condition the participant-side features. These feature sets have been used in previous mental health [21, 58, 59, 20, 24] and behavioral analysis [60, 61] pipelines. Full details are in the Supplementary in Section A-A. I-B2 Answer Segmentation and Temporal Alignment For this feasibility study, the interviewer manually marked the start and end timestamps of each question-answer interval. This step can potentially be automated at deployment by aligning the WhisperX transcript to the known interview questions to locate each question-answer interval. If a participant did not answer a question, that question is excluded for the participant. Each answer segment is divided into temporal bins with a bin width of w=1w=1s, as illustrated in Fig. 3(a). Because answer durations vary widely (1–926s), the number of bins adapts to each answer duration rather than truncating long answers or padding short ones. The number of bins for answer j of participant i is set as Bij=clamp(round(Lij/w)+1,Bmin,Bmax)B_ij=clamp (round(L_ij/w)+1,B_ ,B_ ), where BminB_ and BmaxB_ are lower and upper bin-count bounds. These bounds ensure that every answered question is represented, while limiting the number of bins for long answers. When an answer exceeds the upper bound, the bin width is increased (w>1w>1s) so that the full answer is represented using BmaxB_ bins without truncation. Frame-level features are assigned to the bin containing their frame timestamp. Window-level features are assigned to all bins that overlap their feature windows. If multiple windows overlap the same bin, their feature vectors are averaged to form a single bin-level representation, treating all overlapping windows as an estimate of the signal within that bin. Transcript words are assigned using WhisperX timestamps, and all words overlapping a bin are concatenated and encoded as a single bin-level language representation using RoBERTa embeddings. The bins from all answered questions are concatenated along the temporal axis as participant-level multimodal feature time series, shown in Fig. 3(b), with a length of Si=∑j=1JiBijS_i= _j=1^J_iB_ij, where bin index is s∈1,…,Sis∈\1,…,S_i\ in the concatenated participant-level sequence. I-B3 Modality-Timestep Missingness The availability mask Mij,k(m)M_ij,k^(m) for bin bij,kb_ij,k is set to 00 if no valid observation for feature m is available in that bin. This occurs when no frame-level feature value falls within the bin interval, no window-level feature overlaps the bin interval, or no transcript word is spoken in the bin. In that case, the corresponding feature vector is zero-filled ij,k(m)=x_ij,k^(m)=0. Conversely, Mij,k(m)=1M^(m)_ij,k=1 denotes a valid observation for that feature in the bin, even when the feature vector contains zero-valued measurements, thereby distinguishing missing observations from valid low-signal behavior (Fig. 3(b)). I-C Question-Conditioned Fusion and Prediction Fig. 4: Question-conditioned fusion pipeline (S=14S=14 for illustration). I-C1 Projection Backbones and Fusion We explored a linear and a non-linear per-modality-feature projection backbone (Fig. 4) to test whether the model would benefit from learning a per-modality-feature temporal representation before multimodal fusion. The linear projection backbone projects each modality-feature at each temporal bin s to dimension d independently, with no temporal modeling (no mixing of information across time). Here, temporal modeling is deferred to the Transformer encoder (Enc.) shown in Fig. 4, which applies attention across temporal bins after multimodal fusion. The non-linear projection backbone instead uses a per-modality-feature Transformer that applies attention across time before fusion. Thus, unlike the linear projection backbone, the non-linear projection backbone mixes information for each modality-feature across time before multimodal fusion. Both backbones take the aligned per-modality-feature time series i(m)∈ℝSi×DmX_i^(m) ^S_i× D_m and availability masks Mi(m)∈0,1Si×1M_i^(m)∈\0,1\^S_i× 1 as input. Then they both project each modality-feature onto a common dimension d, producing ^i(m)∈ℝSi×d X_i^(m) ^S_i× d: ^i(m)=Mi(m)⊙(i(m)Wm⊤)(a) linearfm(i(m)Wm⊤,Mi(m))(b) non-linear X_i^(m)= casesM_i^(m) (X_i^(m)\,W_m )&(a) linear\\[4.0pt] f_m\! (X_i^(m)\,W_m ,\;M_i^(m) )&(b) non-linear cases (1) The linear projection gates each feature by its mask. The non-linear projection instead applies a per-modality-feature Transformer fmf_m with temporal self-attention while using the mask as the attention mask. Regardless of which projection backbone is used, the resulting projected per-modality-feature representations ^i,s(m)∈ℝd x_i,s^(m) ^d at each bin s are then concatenated with their availability masks [62]. Before concatenation, a zero-filled missing bin and an observed value at the feature mean are both zero, so without appending the mask as a missingness indicator, the model cannot distinguish a missing observation from a valid zero measurement. The concatenated representation and mask vectors are then projected to a single fused multimodal token i,s∈ℝdz_i,s ^d: i,s=Wf[^i,s(1);⋯;^i,s(Nm);Mi,s(1);⋯;Mi,s(Nm)],z_i,s=W_f [\, x_i,s^(1);·s; x_i,s^(N_m);\;M_i,s^(1);·s;M_i,s^(N_m)\, ], where Nm=8N_m=8 is the total number of modality-features. The tokens are stacked to form the fused multimodal sequence i∈ℝSi×dZ_i ^S_i× d. I-C2 Question Conditioning and Prediction We condition the fused multimodal sequence, iZ_i, on the question context at each temporal bin. Let ijq_ij be the embedding of the j-th question for participant i. For each temporal bin s within the answer to question j, we add the question embedding to the fused multimodal token i,sz_i,s, ~i,s=i,s+Wqij z_i,s=z_i,s+W_q\,q_ij, where WqW_q maps the question embedding to dimension d. The conditioned tokens form a sequence ~i∈ℝSi×d Z_i ^S_i× d, which is then processed by a Transformer encoder (Enc. in Fig. 4) with a prepended [CLS] token, and a linear head maps the [CLS] representation to the prediction logit. Within a batch, participant sequences of unequal length are padded to the longest and excluded from attention through a padding mask. Fig. 5: Multi-level interpretability in modality, question, and time. I-D Post-hoc Multi-Level Attribution After training, we attribute each prediction to the input features using IG, with a zero-vector baseline equal to the mean feature value after standardization [49]. This yields a single scalar attribution |Ac,s(m)||A_c,s^(m)| for each feature c of modality-feature m at each temporal bin s for each participant (Fig. 5(a)). We aggregate these feature-level attributions to build interpretable summaries along clinically interpretable axes: which question and which modality within that question drove the prediction, and when in the interview the relevant evidence occurred (Fig. 5(b)). The arrows in Fig. 5(b) show these aggregations, producing modality importance, question importance, within-question modality importance, and the temporal importance across the interview. Full derivations are provided in the Supplementary in Section A-B. IV Experiments IV-A Dataset Our dataset comprises remote semi-structured interviews from 49 older adults with MCI. Interviews were conducted over Zoom at participants’ preferred time and location, typically at home, using their own devices, including tablets, laptops, and mobile phones. Participants were recruited from the Charlie and Harriet Shaffer Cognitive Empowerment Program (CEP) at Emory University. The study protocol was approved by Emory University Institutional Review Board (protocol #2025P012652). Participants provided written informed consent for research participation. Each interview included six question groups: open-ended narrative prompts including a TAT picture description task [63], an I-CONECT social and functional check [64], the Geriatric Depression Scale (GDS) [65], the Generalized Anxiety Disorder (GAD-7) scale [66], the UCLA Loneliness Scale [67], and the Lubben Social Network Scale (Social Engagement) [68]. Before each interview, participants were told that they could elaborate on any answer. However, answers to yes/no and Likert-scale questions were usually brief. Not every participant answered every question, owing to question skip logic, pilot-phase changes to the administered question groups, and occasional technical issues (details are provided in the Supplementary in Section B-A). Participants had a mean age of 73.4 ± 8.1 years, and 47% were female. Across all questions, answers had a median duration of 5s with a maximum duration of 926s ≈15≈ 15 minutes (interquartile range: 3–12s). Full interview durations (including both participant and interviewer sides) ranged from 21 to 67 minutes, with a median of 31 minutes. Participant-side video resolution varied from 640×480640× 480 to 1920×10801920× 1080 pixels. We define depression risk as GDS ≥ 10, corresponding to at least mild depressive symptoms [65] and anxiety risk as GAD-7 ≥ 5, corresponding to at least mild anxiety symptoms [66]. One participant lacked a GAD-7 score, yielding 49 participants for depression (21 positive, 28 negative) and 48 for anxiety (17 positive, 31 negative). Per-modality-feature average temporal bin missingness was at most 3% except for language (37%). Structured question items usually elicit single-word or yes/no answers (median duration: 3s), so most one-second bins within an answer contain no transcribed word, leading to high language-feature missingness (average ≈44%≈ 44\% across structured question groups), while open-ended questions elicit the longest answers (median duration: 87.5s) and the lowest missingness (16%). The Supplementary reports language-feature missingness and answer duration by question group in Fig. 8. IV-B Implementation and Evaluation Temporal bins are constructed with a minimum of Bmin=1B_ =1 bin and a maximum of Bmax=64B_ =64 bins per answer. The upper bound of 64 bins was chosen so that the majority of answers are represented at 1-second resolution. For the 4% of answers exceeding 64 seconds, bins were widened proportionally so that the full answer duration was still spread across all 64 bins, reducing temporal resolution rather than discarding content (Section I-B2). All Transformer encoders use a hidden dimension d=128d=128. The per-modality-feature encoders in the non-linear projection backbone use a single Transformer layer, while the common encoder (the Enc. in Fig. 4) in both backbones uses two Transformer layers. We train all models for 10 epochs (sufficient for plateauing the model training loss curve) [21] using the Adam optimizer with a learning rate of 0.001, a batch size of 4 participants, and binary cross-entropy loss. A class-balanced batch sampler is used to address class imbalance. Interpretability analysis is applied to the best-performing model variant for each task (Non-linear+T for depression and Linear+T for anxiety). We evaluated all models using participant-independent 5-fold cross-validation across 3 repetitions (15 runs). All feature time series were standardized using training-fold statistics before training to have zero mean and unit variance. We report AUROC scores with 95% confidence intervals (CIs) across all folds. We also report area under the precision-recall curve (AUPRC), macro-F1, precision, and recall scores for all models in the Supplementary in Table V. All performance comparisons use one-sided paired t-tests on fold-level AUROC score differences at p<0.05p<0.05. IV-C Ablations We evaluate temporal alignment (T) and question-conditioned fusion (Q) of the multimodal feature time series as two design options under both linear and non-linear projection backbones. The sequence-index-paired baseline model pairs feature time series by index rather than by the timestamp of the underlying behavior, following prior multimodal models [26, 27]. Models that use Q only without T test two adaptations of prior question-conditioning methods on the temporally unaligned sequence-index-paired baseline. The first is pooled additive conditioning (++ Q, sum), adapted from the question conditioning approach of Zhang et al. [25]. In our variant, all answered-question embeddings are averaged into one interview-level vector and added uniformly across time to the fused multimodal sequence. The second is cross-attention conditioning (++ Q, cross-attn.), which generalizes the question-conditioned additive attention used in HCAG [34]. In our variant, cross-attention is instead applied between the fused multimodal sequence and the sequence of answered-question embeddings. Further details are provided in the Supplementary in Section B-C. We also ablate the modality-timestep missingness mask on the T models with both linear and non-linear projection backbones. We replace the mask with an all-ones mask (no mask variant), which makes zero-filled bins indistinguishable from observed ones, reproducing the no-mask case used by prior work that zero-fills at the timestep level [31, 27]. V Results TABLE I: AUROC performance (mean ± 95% CIs) for depression and anxiety classification. Bold, underline, and italics denote the best, second-best, and third-best results for each task, respectively. Tied results receive the same rank. T denotes temporally aligned features, and Q denotes question conditioning. Linear and Non-linear denote the different projection backbones. Variant Depression AUROC ↑ Anxiety AUROC ↑ Sequence-index-paired baseline [26, 27] Linear 0.51±0.060.51± 0.06 0.58±0.030.58± 0.03 Non-linear 0.58±0.110.58± 0.11 0.56±0.060.56± 0.06 Linear + T 0.64±0.08∗0.64± 0.08 0.69±0.09¶0.69± 0.09 @paragraphsign Non-linear + T 0.68±0.040.68± 0.04 0.58±0.070.58± 0.07 Linear + Q, sum [25] 0.52±0.0030.52± 0.003 0.55±0.060.55± 0.06 Linear + Q, cross-attn. [34] 0.45±0.070.45± 0.07 0.56±0.040.56± 0.04 Non-linear + Q, sum [25] 0.55±0.080.55± 0.08 0.54±0.070.54± 0.07 Non-linear + Q, cross-attn. [34] 0.56±0.110.56± 0.11 0.56±0.090.56± 0.09 Linear + Q + T 0.67±0.10¯† 0.67± 0.10 0.67±0.07¯§ 0.67± 0.07 @sectionsign Non-linear + Q + T 0.67±0.07¯‡ 0.67± 0.07 0.61±0.010.61± 0.01 Selected AUROC comparisons using one-sided Student’s paired t-tests: Depression: ∗Linear+T >> Linear baseline, p≈0.05p≈ 0.05; †Linear+Q+T >> Linear+Q (sum), p≈0.009p≈ 0.009 and >> Linear+Q (cross-attn.), p≈0.001p≈ 0.001, and >> Linear baseline, p≈0.009p≈ 0.009; ‡Non-linear+Q+T >> Non-linear+Q (sum), p≈0.02p≈ 0.02, and >> Non-linear+Q (cross-attn.), p≈0.001p≈ 0.001. Anxiety: ¶Linear+T >> Linear baseline, p≈0.05p≈ 0.05; §Linear+Q+T >> Linear+Q (sum), p≈0.04p≈ 0.04. For all other pairwise comparisons, p>0.05p>0.05. Full pairwise statistical comparisons are provided in the Supplementary in Table VI. Macro-F1, AUPRC, precision, and recall results are provided in the Supplementary in Table V. TABLE I: Mask ablation results for depression and anxiety classification in terms of AUROC (mean ± 95% CIs). Bold denotes the better result within each projection backbone and task. Variant Depression AUROC ↑ Anxiety AUROC ↑ Linear + T (no mask) 0.65±0.090.65± 0.09 0.64±0.050.64± 0.05 Linear + T (mask) 0.64±0.080.64± 0.08 0.69±0.090.69± 0.09 Non-linear + T (no mask) 0.67±0.090.67± 0.09 0.56±0.090.56± 0.09 Non-linear + T (mask) 0.68±0.040.68± 0.04 0.58±0.070.58± 0.07 V-A Overall Performance Table I reports AUROC scores for depression and anxiety under each combination of projection backbone (Linear vs. Non-linear), temporal alignment (T), and question conditioning (Q). Temporally aligning the multimodal features achieves the highest scores for both tasks: Non-linear+T reaches 0.68±0.040.68± 0.04 for depression (Δ=+0.10 =+0.10 over the Non-linear baseline), and Linear+T reaches 0.69±0.090.69± 0.09 for anxiety (Δ=+0.11 =+0.11 over the Linear baseline, p=0.05p=0.05). Adding question conditioning on top of alignment provides no significant gain over alignment alone (all Q+T vs. T p>0.05p>0.05), though Q+T variants remain competitive (Linear +Q+T: 0.670.67 for depression and anxiety). Without temporal alignment, neither summation nor cross-attention for question conditioning shows a consistent advantage across the two tasks (AUROC 0.45–0.56). Models with linear and non-linear projection backbones have overlapping confidence intervals, and their rankings are inconsistent across tasks. Replacing the modality-feature-timestep mask with all-ones, thereby treating all bins as observed (the no-mask variant in Table I), reduces performance for anxiety classification (Linear Δ=−0.05 =-0.05 and Non-linear Δ=−0.02 =-0.02). The effect on depression classification is smaller, where removing the mask increases AUROC by 0.01 for the linear model but decreases it by 0.01 for the non-linear model. V-B Modality Importance The most important features differ across the two tasks (Fig. 6(a)). For depression, eyegaze accounts for 66% of the total attribution, followed by ComParE at 11.2%. For anxiety, the attribution is more evenly distributed between eyegaze at 44% and head pose at 37.2%. Together, the two leading features account for more than 77% of the total attribution in each task. AUs & LMs contribute less than 1% in both tasks, indicating that eyegaze and head pose capture most of the predictive facial signal once they are present. Language features contribute less than 3% in both tasks. Acoustic information is more prominent for depression, with ComParE ranking second overall, whereas no individual acoustic feature exceeds 6.5% for anxiety. This feature dominance holds at the question level for both tasks, as shown for the highest-ranked questions in the Supplementary in Fig. 12. Fig. 6: IG attribution for depression and anxiety across participants. (a) Modality importance across the entire video duration. (b) Temporal importance across interview progress, showing the total attribution together with the eyegaze, head pose, and ComParE time series attributions. Shaded bands indicate participant-level variability, and the colored strip below each plot marks the interview progress of each question group. TABLE I: Question-group IG attribution summaries. Values are scaled by 10310^3. Spread is the difference between the maximum and minimum group means within each task. Bold, underline, and italics denote the highest, second-highest, and third-highest attributions across question groups within each task. Depression Anxiety Question group Range Mean Range Mean Open 1.09–1.65 1.31 1.04–1.39 1.16 I-CONECT 0.66–2.15 1.03 0.91–1.58 1.17 GDS 0.42–1.03 0.68 0.74–1.76 1.17 GAD-7 0.73–1.07 0.90 0.80–1.17 1.03 Loneliness 0.64–1.35 0.88 0.77–1.14 0.93 Social engagement 0.83–1.70 1.17 0.74–1.64 1.10 Spread 0.63 0.24 V-C Question Importance Both tasks distribute attribution across all interview questions but differ in how that attribution spreads across question groups (Table I). Depression shows wider dispersion (spread: 0.63×10−30.63× 10^-3), with the highest mean attribution assigned to the open-ended questions, followed by the social engagement questions. The open-ended questions are Q1 (“Tell me about yourself”), Q2 (Best event in the past two years), Q3 (Worst event in the past two years), and Q4 (TAT picture description). At the individual question level, the four questions with the highest absolute attributions are Q10 (I-CONECT “In the past week, have you had visitors who stayed with you in your home for a night or more? If yes, then how many nights?”), Q89 (Social Engagement: “How often is one of your relatives available for you to talk to when you have an important decision to make?”), Q5 (I-CONECT: “So, would you say your health is Very good, Good, Fair, Poor, Excellent?”), and Q1 (open-ended: “Tell me about yourself”). Anxiety shows a tight group-mean range (spread: 0.24×10−30.24× 10^-3), indicating roughly uniform reliance on the full interview, with the highest-attribution questions coming from multiple groups. The four questions with the highest attribution for anxiety are Q47 (GDS: crying frequency), Q91 (Social Engagement: “How often is one of your friends available for you to talk to when you have an important decision to make?”), Q90 (Social Engagement: “When one of your friends has an important decision to make, how often do they talk to you about it?”), and Q20 (I-CONECT: “Did you spend time communicating with any friends or family members in writing, such as email, text, or letter writing this week?”). The attribution scores for the highest-ranked questions across all question groups are presented for each task in the Supplementary in Fig. 12. Neither GDS nor GAD-7 ranks among the top question groups for its corresponding task, indicating that the model relies on participant behavioral responses across the video rather than on the screening tool itself. This suggests a novel opportunity for developing multimodal behavioral biomarkers associated with depression and anxiety in MCI beyond a brief self-report tool for clinical decision support. V-D Temporal Importance Depression attribution peaks early and decays across the interview from ∼ 2.0 to ∼ 0.75 (×10−3× 10^-3) by mid-interview (Fig. 6(b)). The decline is driven by eyegaze, whose contribution decreases from ∼ 1.25 to ∼ 0.50 (×10−3× 10^-3) over the same period. The early peak aligns with the open-ended questions and is consistent with the question-level finding that these questions receive the highest mean attribution. In contrast, anxiety shows a uniform temporal importance profile. Total attribution decreases from ∼ 1.2 to ∼ 0.90 (×10−3× 10^-3) near the end of the interview before returning to ∼ 1.2. The eye gaze and head pose attributions remain relatively uniform throughout the interview. Representative participant profiles are shown in the Supplementary in Fig. 12. V-E Open-Ended Question Performance To evaluate whether the open-ended portion of the interview alone retains predictive performance, we train and evaluate the best-performing model for each task using only multimodal features extracted from participants’ responses to the four open-ended questions (median duration: 5.1 min). We use the Non-linear+T model for depression and the Linear+T model for anxiety. For depression, the open-ended-only model achieves an AUROC of 0.67±0.130.67± 0.13, which is only 0.01 lower than that of the model trained on the full interview. This difference is not statistically significant (p>0.05)(p>0.05). This result is consistent with the question importance analysis, which shows that depression predictions rely on responses to the open-ended questions. For anxiety, the open-ended-only model achieves an AUROC of 0.59±0.060.59± 0.06. The model trained on the full interview achieves an AUROC that is 0.10 higher, and the difference is statistically significant (p≈0.02p≈ 0.02). This finding is also consistent with the question importance analysis, which suggests that anxiety predictions rely uniformly on the full interview. VI Discussion VI-A Temporal Alignment Temporally aligning the multimodal features was the primary driver of performance across both tasks, with models that used temporal alignment achieving the highest AUROC scores compared to the baseline models. Aligning features by timestamp corrects the temporal misalignment between the multimodal behavioral time series caused by sequence-index pairing used in prior approaches [27, 26] (Fig. 1 and Fig. 2(a)). VI-B Question Conditioning Question-conditioned fusion did not improve the performance of models that used temporally aligned multimodal features, and the temporally unaligned Q-only model variants showed no consistent gains over the baselines on either task. Question conditioning injects semantic context about what was asked, which is directly relevant to interpreting the answer content. However, because language alone contributed less than 3% of total attribution in both tasks, question content may have had limited influence on the prediction. Moreover, the Q-only variants do not use the explicit question-answer mapping provided by temporal alignment, so the model must recover this mapping, which may be difficult to learn in a data-driven manner. VI-C Missingness Awareness Overall, the modality–feature-timestep mask improved AUROC in three of the four model-task combinations, with the largest numerical gain for the Linear+T anxiety model and small, mixed effects for the depression models (Table I). To explore whether label-associated missingness might contribute to this pattern, we followed Che et al. [62] and computed Pearson correlations (r)(r) across participants between modality missingness rates and the binary label for each task (depression or anxiety). We considered missingness informative when its rate correlated with the label and could therefore carry predictive information [62]. Based on the modality importance analysis, we examined this missingness-label association for the highest-ranked modalities in each task: eyegaze and head pose for anxiety prediction, and eyegaze for depression prediction. We also included text as a higher-missingness reference for anxiety prediction. Missingness in eyegaze and head pose had absolute correlations of |r|=0.19|r|=0.19 and 0.180.18, respectively, with the anxiety label, compared with |r|=0.04|r|=0.04 for text (highest-missing modality). This suggests that missingness in the dominant anxiety modalities was informative and that the much higher rate of text missingness was not as informative. For depression, the correlation between eyegaze missingness and the label was small (|r|=0.014|r|=0.014), suggesting that missingness was not informative for this task.11 1 These magnitudes are comparable to those reported in Fig. 1 in [62], where the missingness–label correlations were below 0.3. This task-specific pattern is consistent with the finding reported by Che et al. [62] that modeling missingness can improve prediction when it is associated with the label. In contrast, missingness with little or no label association provides limited benefit. This may explain why the mask numerically improved both anxiety models but had small, mixed effects on the depression models. However, the performance differences were not statistically significant at this sample size (p>0.05p>0.05), and this interpretation requires validation in a larger sample. The mask is also important for the interpretability analysis because, without it, the modality, question, and temporal importance summaries could assign attribution to behavior that was not observed. VI-D Modality Importance Eyegaze was the dominant feature for both depression and anxiety, although anxiety distributed attribution more evenly between eyegaze and head pose. For depression, eyegaze accounted for 66% of the total attribution, consistent with findings of eye movement differences between individuals with and without depression, including longer saccade durations [69]. Among older adults specifically, depression was associated with fewer fixations and saccades [69]. The ComParE acoustic feature set ranked second for depression with 11.2% attribution, supporting previous evidence that depression is associated with distinct vocal characteristics such as monotony, reduced pitch variability, and slower speech [70]. For anxiety, eyegaze and head pose accounted for 44% and 37.2% of the total attribution, respectively. This is consistent with reduced face-directed gaze and shorter fixations in socially anxious individuals during real-time interactions [71, 72], as well as increased head movement and velocity during anxiety and stress states in video-based facial analysis [73]. All remaining acoustic and language features accounted for less than 6.5% of the attribution in both tasks, consistent with previous work on quantifying psychological well-being in older adults with MCI [21]. VI-E Question Importance The most notable result from the interpretability analysis is that the model did not assign its highest attribution to the question group that defined each label. GDS ranked last of six question groups for depression, and GAD-7 ranked second to last for anxiety. Attribution was instead distributed across question groups, suggesting that the model relied on participant behavior throughout the interview rather than on the screening questions. At the individual-question level, the highest-attributed questions for depression concerned social contact and support, and self-rated health. This is consistent with evidence that smaller social networks and lower social participation are associated with depressive symptoms in older adults [74], as well as evidence linking self-rated health to depressive symptoms in this population [75]. For anxiety, crying frequency received the highest attribution, which is consistent with evidence that greater attachment anxiety is associated with longer and more intense crying episodes [76]. Tearfulness is also included under the Tension item of the Hamilton Anxiety Rating Scale [77]. The next highest-attributed questions concerned communication with and support from friends, which is consistent with evidence that low social support and social isolation are among the most reported factors associated with anxiety symptoms in older adults [78]. VI-F Temporal Importance Depression and anxiety differed in where the model’s attributions were concentrated. For depression, attribution was highest during the open-ended question block and declined across the later structured section. This is consistent with prior work that reported higher depression prediction performance from spontaneous speech than from a constrained sentence-reading task [79]. This suggests that spontaneous-speech response formats are more informative for depression screening. Anxiety, instead, showed a uniform spread of attribution across question groups. The most informative features, eyegaze and head pose, were expressed continuously rather than elicited by a particular question. This is consistent with accounts of anxious hypervigilance as sustained monitoring of the surroundings through eye movements that occurs independently of the immediate stimulus [80], indicating that anxiety-related signals reflect a continual behavioral state. VI-G Open-Ended Question Performance Restricting the model input to participants’ responses to the four open-ended questions yielded a depression classification AUROC of 0.67, only 0.01 below the 0.68 achieved using the full interview, with no statistically significant difference between them (p>0.05p>0.05). Although combined open-ended responses had a median duration of 5.1 minutes (range: 2.6–11.5 min) for participants who answered all the open-ended questions (n=25n=25), the short responses retained predictive information. This is consistent with the high attribution assigned to the open-ended question group from the interpretability analysis. For anxiety, however, restricting the input to the open-ended questions significantly reduced AUROC from 0.69 to 0.59 (p≈0.02p≈ 0.02). This result is consistent with the interpretability analysis, which indicates that anxiety-related information is distributed across the full interview. None of the four questions with the highest anxiety attribution belonged to the open-ended question group, suggesting that an abbreviated anxiety protocol may require questions selected from other parts of the interview. Given this cohort’s median participant-side-only interview duration of 19 minutes (maximum 60 min), a shorter protocol centered on open-ended questions (median duration: 5.1 min; maximum: 11.5 min) could reduce clinician time and participant burden for depression assessment. Our findings suggest that using the four open-ended questions alone may be sufficient to screen for depression. However, this finding requires validation in a larger sample. VI-H Limitations and Future Work One major limitation of this study is the small sample size (N=49N=49). In addition, the feature extractors for facial and acoustic features were pretrained on general-population data and may be less reliable on remote recordings from older adults with MCI, as reported in previous studies [21]. In future work, our findings should be validated in larger and more diverse cohorts, including populations with neuropsychiatric conditions who often have symptoms associated with depression and anxiety. VII Conclusion Depression and anxiety are often underdiagnosed in older adults with MCI. Current ML approaches can analyze multimodal behavioral signals for screening. However, these approaches overlook the importance of cross-modal temporal alignment, fail to account for time-varying modality missingness and question context, and provide limited interpretability for clinicians. We presented TAMI, a multimodal framework for detecting depression and anxiety in older adults with MCI from remote interviews, which incorporates cross-modal temporal alignment, modality-timestep missingness handling, question conditioning, and multi-level interpretability. The framework achieved 0.68 (depression) and 0.69 (anxiety) AUROC scores, with temporal alignment providing the largest performance gain over baselines. When restricted to multimodal responses from the four open-ended questions, the model achieved an AUROC score of 0.67 for depression. These responses had a median duration of 5.1 minutes, compared with 19 minutes for the participant-side full interview. This suggests that a shorter open-ended protocol may be sufficient to screen for depression. The proposed temporal alignment approach may benefit other applications involving multimodal time series, including emotion recognition [81] and human activity recognition [82], where features differ in sampling rates and temporal support, and cross-modal temporal correspondence is important for prediction. Clinically, TAMI supports remote mental health screening for adults who face barriers to in-person care or live in underserved regions. Its multi-level interpretability (modality, question, and temporal attributions) also allows clinicians to understand the reasoning underlying the model’s predictions, supporting informed clinical review. References [1] K. B. Rajan, J. Weuve, L. L. Barnes, E. A. McAninch, R. S. Wilson, and D. A. Evans (2021) Population estimate of people with clinical Alzheimer’s disease and mild cognitive impairment in the united states (2020–2060). Alzheimer’s & Dementia 17 (12), p. 1966–1975. External Links: Document Cited by: §I. [2] S. Gauthier, B. Reisberg, M. Zaudig, R. C. Petersen, K. Ritchie, K. Broich, S. Belleville, H. Brodaty, D. Bennett, H. Chertkow, J. L. Cummings, M. de Leon, H. Feldman, M. Ganguli, H. Hampel, P. Scheltens, M. C. Tierney, P. Whitehouse, and B. Winblad (2006) Mild cognitive impairment. Lancet 367 (9518), p. 1262–1270. External Links: Document Cited by: §I. [3] Z. Ismail, H. Elbayoumi, C. E. Fischer, D. B. Hogan, C. P. Millikin, T. Schweizer, M. E. Mortby, E. E. Smith, S. B. Patten, and K. M. Fiest (2017) Prevalence of depression in patients with mild cognitive impairment: a systematic review and meta-analysis. JAMA Psychiatry 74 (1), p. 58–67. External Links: Document Cited by: §I. [4] E. Martin and L. Velayudhan (2020) Neuropsychiatric symptoms in mild cognitive impairment: a literature review. Dementia and Geriatric Cognitive Disorders 49 (2), p. 146–155. External Links: Document Cited by: §I. [5] L. Ma (2020) Depression, anxiety, and apathy in mild cognitive impairment: current perspectives. Frontiers in Aging Neuroscience 12, p. 9. External Links: Document Cited by: §I. [6] S. E. Starkstein and R. Mizrahi (2006) Depression in Alzheimer’s disease. Expert Review of Neurotherapeutics 6 (6), p. 887–895. External Links: Document Cited by: §I. [7] N. Bassil, A. Ghandour, and G. T. Grossberg (2011) How anxiety presents differently in older adults. Current Psychiatry 10 (3), p. 65–71. Cited by: §I. [8] R. Lavingia, K. Jones, and A. A. Asghar-Ali (2020) A systematic review of barriers faced by older adults in seeking and accessing mental health care. Journal of Psychiatric Practice 26 (5), p. 367–382. External Links: Document Cited by: §I. [9] V. Bergua, C. Blanchard, and H. Amieva (2026) Depression in older adults: do current DSM diagnostic criteria really fit?. Clinical Gerontologist 49 (2), p. 224–261. External Links: Document Cited by: §I. [10] C. Bryant, J. Mohlman, A. Gum, M. Stanley, A. T. Beekman, J. L. Wetherell, S. R. Thorp, A. J. Flint, and E. J. Lenze (2013) Anxiety disorders in older adults: looking to DSM-5 and beyond. American Journal of Geriatric Psychiatry 21 (9), p. 872–876. External Links: Document Cited by: §I. [11] OpenAI (2026) ChatGPT Images 2.0 System Card. Note: OpenAI Deployment Safety HubAccessed: July 20, 2026 External Links: Link Cited by: Fig. 2. [12] M. Alpert, E. R. Pouget, and R. R. Silva (2001) Reflections of depression in acoustic measures of the patient’s speech. Journal of Affective Disorders 66 (1), p. 59–69. External Links: Document Cited by: §I. [13] Y. Li, Y. Xu, M. Xia, T. Zhang, J. Wang, X. Liu, Y. He, and J. Wang (2016) Eye movement indices in the study of depressive disorder. Shanghai Archives of Psychiatry 28 (6), p. 326–334. External Links: Document Cited by: §I. [14] R. M. Voss and J. M. Das (2026) Mental status examination. In StatPearls, Note: Last updated April 30, 2024 External Links: Link Cited by: §I. [15] F. Trémeau, D. Malaspina, F. Duval, H. Corrêa, M. Hager-Budny, L. Coin-Bariou, J.-P. Macher, and J. M. Gorman (2005) Facial expressiveness in patients with schizophrenia compared to depressed patients and nonpatient comparison subjects. American Journal of Psychiatry 162 (1), p. 92–101. External Links: Document Cited by: §I. [16] J. A. Harrigan and D. M. O’Connell (1996) How do you look when feeling anxious? Facial displays of anxiety. Personality and Individual Differences 21 (2), p. 205–212. External Links: Document Cited by: §I. [17] V. van Wassenhove, K. W. Grant, and D. Poeppel (2007) Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45 (3), p. 598–607. External Links: Document Cited by: §I, §I. [18] N. G. Choi, D. M. DiNitto, C. N. Marti, and B. Y. Choi (2022) Telehealth use among older adults during COVID-19: associations with sociodemographic and health characteristics, technology device ownership, and technology learning. Journal of Applied Gerontology 41 (3), p. 600–609. External Links: Document Cited by: §I. [19] Health Resources and Services Administration (n.d.) Health workforce shortage areas data dashboard. Note: https://data.hrsa.gov/topics/health-workforce/shortage-areas/dashboardAccessed: May 13, 2026 Cited by: §I. [20] Z. Jiang, S. Seyedi, E. Griner, A. Abbasi, A. B. Rad, H. Kwon, R. O. Cotes, and G. D. Clifford (2024) Multimodal mental health digital biomarker analysis from remote interviews using facial, vocal, linguistic, and cardiovascular patterns. IEEE Journal of Biomedical and Health Informatics 28 (3), p. 1680–1691. External Links: Document Cited by: §I, §I, §I-E, §I-B1. [21] X. Mu, M. Bibars, S. Seyedi, I. Zheng, Z. Jiang, L. Chen, B. Omofojoye, R. Hershenberg, A. I. Levey, G. D. Clifford, H. H. Dodge, and H. Kwon (2026) Feasibility of detecting cognitive impairment and psychological well-being among older adults using facial, acoustic, linguistic, and cardiovascular patterns derived from remote conversations. Machine Learning: Health 2 (1), p. 015005. External Links: Document Cited by: §A-B1, §B-B, §I, §I, §I-A, §I-D, §I-E, §I-B1, §IV-B, §VI-D, §VI-H. [22] D. M. Low, K. H. Bentley, and S. S. Ghosh (2020) Automated assessment of psychiatric disorders using speech: a systematic review. Laryngoscope Investigative Otolaryngology 5 (1), p. 96–116. External Links: Document Cited by: §I. [23] Y. Guo, C. Zhu, S. Hao, and R. Hong (2022) A topic-attentive transformer-based model for multimodal depression detection. arXiv preprint arXiv:2206.13256. Cited by: §I, §I, §I-B, §I-D, §I-E. [24] J. Jung, C. Kang, J. Yoon, S. Kim, and J. Han (2024) HiQuE: hierarchical question embedding network for multimodal depression detection. In Proc. 33rd ACM Int. Conf. Information and Knowledge Management (CIKM), Boise, ID, USA, p. 1049–1059. External Links: Document Cited by: §A-A3, §I, §I-A, §I-B, §I-E, §I-B1. [25] E. Zhang and C. Poellabauer (2025) Mitigating interviewer bias in multimodal depression detection: an approach with adversarial learning and contextual positional encoding. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, p. 12169–12188. External Links: Document Cited by: §B-C2, TABLE V, TABLE V, §I, §I, §I-A, §I-B, §I-D, §IV-C, TABLE I, TABLE I. [26] H. Fan, X. Zhang, Y. Xu, J. Fang, S. Zhang, X. Zhao, and J. Yu (2024) Transformer-based multimodal feature enhancement networks for multimodal depression detection integrating video, audio and remote photoplethysmograph signals. Information Fusion 104, p. 102161. External Links: Document Cited by: §B-C1, TABLE V, §I, §I-A, §I-B, §IV-C, TABLE I, §VI-A. [27] Y. Tao, M. Yang, H. Li, Y. Wu, and B. Hu (2024) DepMSTAT: multimodal spatio-temporal attentional transformer for depression detection. IEEE Transactions on Knowledge and Data Engineering 36 (7), p. 2956–2966. External Links: Document Cited by: §B-C1, TABLE V, §I, §I-B, §I-C, §IV-C, §IV-C, TABLE I, §VI-A. [28] D. Gimeno-Gómez, A.-M. Bucur, A. Cosma, C.-D. Martínez-Hinarejos, and P. Rosso (2024) Reading between the frames: multi-modal depression detection in videos from non-verbal cues. In Advances in Information Retrieval, N. Goharian, N. Tonellotto, Y. He, A. Lipani, G. McDonald, C. Macdonald, and I. Ounis (Eds.), Lecture Notes in Computer Science, Vol. 14608, p. 191–209. External Links: Document Cited by: §I, §I, §I-A, §I-B, §I-C, §I-E. [29] D. W. Joyce, A. Kormilitzin, K. A. Smith, and A. Cipriani (2023) Explainable artificial intelligence for mental health through transparency and interpretability for understandability. npj Digital Medicine 6 (1), p. 6. External Links: Document Cited by: §I. [30] Y. Cheng, Z. Zhang, H. Li, A. Arapin, Y. Zhang, Q. Zhang, Y. Liu, K. Du, X. Zhang, F. Y. Yan, A. Mazumdar, N. Feamster, and J. Jiang (2024) GRACE: loss-resilient real-time video through neural codecs. In Proc. 21st USENIX Symp. Networked Systems Design and Implementation (NSDI), Santa Clara, CA, USA, p. 509–531. External Links: Link Cited by: §I. [31] P. Kumar, S. Misra, Z. Shao, B. Zhu, B. Raman, and X. Li (2025) Multimodal interpretable depression analysis using visual, physiological, audio and textual data. In Proc. IEEE/CVF Winter Conf. Applications of Computer Vision (WACV), p. 5305–5315. External Links: Document Cited by: §I, §I-A, §I-C, §I-E, §IV-C. [32] Y. Zhou, X. Yao, W. Han, Y. Wang, Z. Li, and Y. Li (2022) Distinguishing apathy and depression in older adults with mild cognitive impairment using text, audio, and video based on multiclass classification and shapely additive explanations. International Journal of Geriatric Psychiatry 37 (11). External Links: Document Cited by: §I, §I-A, §I-D. [33] Y. Zhou, W. Han, X. Yao, J. Xue, Z. Li, and Y. Li (2023) Developing a machine learning model for detecting depression, anxiety, and apathy in older adults with mild cognitive impairment using speech and facial expressions: a cross-sectional observational study. International Journal of Nursing Studies 146, p. 104562. External Links: Document Cited by: §I, §I-A, §I-D. [34] M. Niu, K. Chen, Q. Chen, and L. Yang (2021) HCAG: a hierarchical context-aware graph attention model for depression detection. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), p. 4235–4239. External Links: Document Cited by: §B-C2, TABLE V, TABLE V, §I, §I-D, §IV-C, TABLE I, TABLE I. [35] J. Gratch, R. Artstein, G. Lucas, G. Stratou, S. Scherer, A. Nazarian, R. Wood, J. Boberg, D. DeVault, S. Marsella, D. Traum, S. Rizzo, and L.-P. Morency (2014) The distress analysis interview corpus of human and computer interviews. In Proc. Ninth Int. Conf. Language Resources and Evaluation (LREC), Reykjavik, Iceland, p. 3123–3128. Cited by: §I-A. [36] D. DeVault, R. Artstein, G. Benn, T. Dey, E. Fast, A. Gainer, K. Georgila, J. Gratch, A. Hartholt, M. Lhommet, G. Lucas, S. Marsella, F. Morbini, A. Nazarian, S. Scherer, G. Stratou, A. Suri, D. Traum, R. Wood, Y. Xu, A. Rizzo, and L.-P. Morency (2014) SimSensei Kiosk: a virtual human interviewer for healthcare decision support. In Proc. 13th Int. Conf. Autonomous Agents and Multiagent Systems (AAMAS), Paris, France, p. 1061–1068. External Links: Document Cited by: §I-A. [37] F. Ringeval, B. Schuller, M. Valstar, N. Cummins, R. Cowie, L. Tavabi, M. Schmitt, S. Alisamir, S. Amiriparian, E.-M. Messner, S. Song, S. Liu, Z. Zhao, A. Mallol-Ragolta, Z. Ren, M. Soleymani, and M. Pantic (2019) AVEC 2019 workshop and challenge: state-of-mind, detecting depression with AI, and cross-cultural affect recognition. In Proc. 9th Int. Audio/Visual Emotion Challenge and Workshop (AVEC), Nice, France, p. 3–12. External Links: Document Cited by: §I-A. [38] J. Sun, H. H. Dodge, and M. H. Mahoor (2024) MC-ViViT: multi-branch classifier-ViViT to detect mild cognitive impairment in older adults using facial videos. Expert Systems with Applications 238, p. 121929. External Links: Document Cited by: §I-A. [39] M. Alsuhaibani, H. H. Dodge, and M. H. Mahoor (2024) Mild cognitive impairment detection from facial video interviews by applying spatial-to-temporal attention module. Expert Systems with Applications 252, p. 124185. External Links: Document Cited by: §I-A. [40] Z. Jiang, S. Seyedi, K. L. Vickers, C. M. Manzanares, J. J. Lah, A. I. Levey, and G. D. Clifford (2024) Disentangling visual exploration differences in cognitive impairment. IEEE Transactions on Biomedical Engineering 71 (4), p. 1197–1208. External Links: Document Cited by: §I-A. [41] J. R. Williamson, E. Godoy, M. Cha, A. Schwarzentruber, P. Khorrami, Y. Gwon, H.-T. Kung, C. Dagli, and T. F. Quatieri (2016) Detecting depression using vocal, facial and semantic communication cues. In Proc. 6th Int. Workshop Audio/Visual Emotion Challenge (AVEC), Amsterdam, The Netherlands, p. 11–18. External Links: Document Cited by: §I-A, §I-E. [42] G. Drougkas, E. Bakker, and M. Spruit (2024) Multimodal machine learning for language and speech markers identification in mental health. BMC Medical Informatics and Decision Making 24, p. 354. External Links: Document Cited by: §I-A. [43] S. Alghowinem, R. Goecke, M. Wagner, J. Epps, M. Hyett, G. Parker, and M. Breakspear (2018) Multimodal depression detection: fusion analysis of paralinguistic, head pose and eye gaze behaviors. IEEE Transactions on Affective Computing 9 (4), p. 478–490. External Links: Document Cited by: §I-A. [44] J. Zhou, J. Ge, Z. Chen, J. Tan, and Y. Li (2026) MDD-MARF: a multimodal depression detection model based on multi-level attention mechanism and residual fusion. Journal of Biomedical Informatics 173, p. 104965. External Links: Document Cited by: §I-A. [45] M. Sadeghi, R. Richer, B. Egger, L. Schindler-Gmelch, L. H. Rupp, F. Rahimi, M. Berking, and B. M. Eskofier (2024) Harnessing multimodal approaches for depression detection using large language models and facial expressions. npj Mental Health Research 3 (1), p. 66. External Links: Document Cited by: §I-A. [46] J. Chen, M. Tan, H. Lu, Q. Xu, Z. Wang, R. Zeng, and X. Hu (2026) Towards stable cross-domain depression recognition under missing modalities. Pattern Recognition 177, p. 113367. External Links: Document Cited by: §I-C. [47] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §A-A3, §I-D, §I-B1. [48] A. Mandal, D. Atzil-Slonim, T. Solorio, and I. Gurevych (2025) Enhancing depression detection via question-wise modality fusion. In Proc. 10th Workshop Computational Linguistics and Clinical Psychology (CLPsych), Albuquerque, NM, USA, p. 44–61. External Links: Document Cited by: §I-E. [49] M. Sundararajan, A. Taly, and Q. Yan (2017) Axiomatic attribution for deep networks. In Proc. Int. Conf. Machine Learning (ICML), p. 3319–3328. Cited by: §A-B, §I-E, §I-E, §I-D. [50] J. H. Cheong, E. Jolly, T. Xie, S. Byrne, M. Kenney, and L. J. Chang (2023) Py-Feat: python facial expression analysis toolbox. Affective Science 4 (4), p. 781–796. External Links: Document Cited by: §A-A1, §I-B1. [51] A. A. Abdelrahman, T. Hempel, A. Khalifa, A. Al-Hamadi, and L. Dinges (2023) L2CS-Net: fine-grained gaze estimation in unconstrained environments. In Proc. 8th Int. Conf. Frontiers of Signal Processing (ICFSP), p. 98–102. External Links: Document Cited by: §A-A1, §I-B1. [52] G. Boccignone, D. Conte, V. Cuculo, A. D’Amelio, G. Grossi, R. Lanzarotti, and E. Mortara (2022) pyVHR: a Python framework for remote photoplethysmography. PeerJ Computer Science 8, p. e929. External Links: Document Cited by: §A-A4, §I-B1. [53] F. Eyben, M. Wöllmer, and B. Schuller (2010) openSMILE: the Munich versatile and fast open-source audio feature extractor. In Proc. ACM Multimedia, Florence, Italy, p. 1459–1462. External Links: Document Cited by: §A-A2, §I-B1. [54] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli (2020) wav2vec 2.0: a framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, Vol. 33, p. 12449–12460. Cited by: §A-A2, §I-B1. [55] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush (2019) Hugging Face’s Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. Cited by: §A-A2, §A-A3, §I-B1. [56] M. Bain, J. Huh, T. Han, and A. Zisserman (2023) WhisperX: time-accurate speech transcription of long-form audio. In Proc. INTERSPEECH, p. 4489–4493. External Links: Document Cited by: §A-A3, §I-B1. [57] N. Reimers and I. Gurevych (2019) Sentence-BERT: sentence embeddings using siamese BERT-networks. In Proc. Conf. Empirical Methods in Natural Language Processing and 9th Int. Joint Conf. Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, p. 3982–3992. External Links: Document Cited by: §A-A3, §I-B1. [58] D. Emden, M. Richter, A. Chevance, R. Leenings, J. Herpertz, L. Gutfleisch, A. Fleuchaus, R. Blitz, V. L. Holstein, J. Goltermann, N. R. Winter, J. Spanagel, S. Meinert, T. Borgers, K. Flinkenflügel, F. Stein, N. Alexander, H. Jamalabadi, J. Repple, C. Dobel, E. J. Leehr, R. Redlich, U. W. Ebner-Priemer, I. Nenadić, T. Kircher, U. Dannlowski, T. Hahn, and N. Opel (2026) Scalable depression monitoring with smartphone speech using a multimodal benchmark and topic analysis. npj Digital Medicine 9 (1). External Links: Document Cited by: §I-B1. [59] F. A. Wagay and Jahiruddin (2025) Classification of mental illnesses from Reddit posts using Sentence-BERT embeddings and neural networks. Procedia Computer Science 258, p. 1669–1676. Note: International Conference on Machine Learning and Data Engineering External Links: Document Cited by: §I-B1. [60] B. C. dos Santos Melício, K. Karaköse, Á. Fodor, L. Xiang, V. Varga, L. Soorya, E. Dillon, P. Kun, A. Sárkány, M. Chetouani, K. Fenech, and A. Lőrincz (2025) Multimodal framework for automatic behavior analysis of children with autism during ADOS-2. Cognitive Computation 17 (4). External Links: Document Cited by: §I-B1. [61] C. Pabba, V. Bhardwaj, and P. Kumar (2024) A visual intelligent system for students’ behavior classification using body pose and facial features in a smart classroom. Multimedia Tools and Applications 83 (12), p. 36975–37005. External Links: Document Cited by: §I-B1. [62] Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu (2018) Recurrent neural networks for multivariate time series with missing values. Scientific Reports 8 (1), p. 6085. External Links: Document Cited by: §I-C1, §VI-C, footnote 1. [63] C. D. Morgan and H. A. Murray (1935) A method for investigating fantasies: the thematic apperception test. Archives of Neurology & Psychiatry 34 (2), p. 289–306. Cited by: §B-A, §IV-A. [64] K. Yu, K. Wild, K. Potempa, B. M. Hampstead, P. A. Lichtenberg, L. M. Struble, P. Pruitt, E. L. Alfaro, J. Lindsley, M. MacDonald, J. A. Kaye, L. C. Silbert, and H. H. Dodge (2021) The internet-based conversational engagement clinical trial (I-CONECT) in socially isolated adults 75+ years old: randomized controlled trial protocol and COVID-19 related study modifications. Frontiers in Digital Health 3, p. 714813. External Links: Document Cited by: §B-A, §IV-A. [65] J. A. Yesavage and J. I. Sheikh (1986) Geriatric Depression Scale (GDS): recent evidence and development of a shorter version. Clinical Gerontologist 5 (1–2), p. 165–173. External Links: Document Cited by: §B-A, §IV-A, §IV-A. [66] R. L. Spitzer, K. Kroenke, J. B. W. Williams, and B. Löwe (2006) A brief measure for assessing generalized anxiety disorder: the GAD-7. Archives of Internal Medicine 166 (10), p. 1092–1097. External Links: Document Cited by: §B-A, §IV-A, §IV-A. [67] D. Russell, L. A. Peplau, and M. L. Ferguson (1978) Developing a measure of loneliness. Journal of Personality Assessment 42 (3), p. 290–294. External Links: Document Cited by: §B-A, §IV-A. [68] J. E. Lubben (1988) Assessing social networks among elderly populations. Family & Community Health 11 (3), p. 42–52. Cited by: §B-A, §IV-A. [69] J. Takahashi, Y. Hirano, K. Miura, K. Morita, M. Fujimoto, H. Yamamori, Y. Yasuda, N. Kudo, E. Shishido, K. Okazaki, T. Shiino, T. Nakao, K. Kasai, R. Hashimoto, and T. Onitsuka (2021) Eye movement abnormalities in major depressive disorder. Frontiers in Psychiatry 12, p. 673443. External Links: Document Cited by: §VI-D. [70] N. Cummins, S. Scherer, J. Krajewski, S. Schnieder, J. Epps, and T. F. Quatieri (2015) A review of depression and suicide risk assessment using speech analysis. Speech Communication 71, p. 10–49. External Links: Document Cited by: §VI-D. [71] H. Kim, J. E. Shin, Y.-J. Hong, Y.-B. Shin, Y. S. Shin, K. Han, J.-J. Kim, and S.-H. Choi (2018) Aversive eye gaze during a speech in virtual environment in patients with social anxiety disorder. Australian and New Zealand Journal of Psychiatry 52 (3), p. 279–285. External Links: Document Cited by: §VI-D. [72] I. Konovalova, J. V. Antolin, H. Bolderston, and N. J. Gregory (2021) Adults with higher social anxiety show avoidant gaze behaviour in a real-world social setting: a mobile eye tracking study. PLOS ONE 16 (10), p. e0259007. External Links: Document Cited by: §VI-D. [73] G. Giannakakis, M. Pediaditis, D. Manousos, E. Kazantzaki, F. Chiarugi, P. G. Simos, K. Marias, and M. Tsiknakis (2017) Stress and anxiety detection using facial cues from videos. Biomedical Signal Processing and Control 31, p. 89–101. External Links: Document Cited by: §VI-D. [74] F. Wendel, A. Bauer, I. Blotenberg, C. Brettschneider, M. Buchholz, D. Czock, J. Döhring, C. Escales, T. Frese, W. Hoffmann, H. Kaduszkiewicz, H.-H. König, M. Löbner, M. Luppa, R. Schwenker, J. R. Thyrian, M. Weißenborn, B. Wiese, I. Zöllinger, S. G. Riedel-Heller, and J. Gensichen (2022) Social network and participation in elderly primary care patients in Germany and associations with depressive symptoms: a cross-sectional analysis from the AgeWell.de study. Journal of Clinical Medicine 11 (19). External Links: Document Cited by: §VI-E. [75] S. Peleg and G. Nudelman (2021) Associations between self-rated health and depressive symptoms among older adults: does age matter?. Social Science & Medicine 280, p. 114024. External Links: Document Cited by: §VI-E. [76] A. Millings, E. G. Hepper, C. M. Hart, L. Swift, and A. C. Rowe (2016) Holding back the tears: individual differences in adult crying proneness reflect attachment orientation and attitudes to crying. Frontiers in Psychology 7, p. 1003. External Links: Document Cited by: §VI-E. [77] W. Maier, R. Buller, M. Philipp, and I. Heuser (1988) The Hamilton Anxiety Scale: reliability, validity and sensitivity to change in anxiety and depressive disorders. Journal of Affective Disorders 14 (1), p. 61–68. External Links: Document Cited by: §VI-E. [78] A. Shafiee, I. Mohammadi, S. Rajai, K. Jafarabady, and A. Abdollahi (2025) Global prevalence of anxiety symptoms and its associated factors in older adults: a systematic review and meta-analysis. Journal of General and Family Medicine 26 (2), p. 116–127. External Links: Document Cited by: §VI-E. [79] S. Alghowinem, R. Goecke, M. Wagner, J. Epps, M. Breakspear, and G. Parker (2013) Detecting depression: a comparison between spontaneous and read speech. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), p. 7547–7551. External Links: Document Cited by: §VI-F. [80] H. J. Richards, V. Benson, N. Donnelly, and J. A. Hadwin (2014) Exploring the function of selective attention and hypervigilance for threat in anxiety. Clinical Psychology Review 34 (1), p. 1–13. External Links: Document Cited by: §VI-F. [81] Y. Wu, Q. Mi, and T. Gao (2025) A comprehensive review of multimodal emotion recognition: techniques, challenges, and future directions. Biomimetics 10 (7), p. 418. External Links: Document Cited by: §VII. [82] M. Karim, S. Khalid, S. Lee, S. Almutairi, A. Namoun, and M. Abohashrh (2025) Next generation human action recognition: a comprehensive review of state-of-the-art signal processing techniques. IEEE Access 13, p. 135609–135633. External Links: Document Cited by: §VII. [83] G. Boccignone, D. Conte, V. Cuculo, A. D’Amelio, G. Grossi, and R. Lanzarotti (2025) Enhancing rPPG pulse-signal recovery by facial sampling and PSD clustering. Biomedical Signal Processing and Control 101, p. 107158. External Links: Document Cited by: §A-A4. [84] G. Boccignone, D. Conte, V. Cuculo, A. D’Amelio, G. Grossi, and R. Lanzarotti (2020) An open framework for Remote-PPG methods and their assessment. IEEE Access 8, p. 216083–216103. External Links: Document Cited by: §A-A4. [85] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu (2016) WaveNet: a generative model for raw audio. arXiv preprint arXiv:1609.03499. Cited by: §B-C2. [86] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §B-C2. Supplementary A Method A-A Feature Extraction We extract four modalities from the participant-side video recording: facial, acoustic, language, and physiological modalities. The eight modality-feature time series used as model inputs are action units and landmarks, head pose, eyegaze, eGeMAPS, ComParE, Wav2Vec2, rPPG-derived heart rate, and transcript embeddings. Table IV summarizes these inputs. For the extracted features, we define window-level features as features that summarize signal content over an interval (e.g., acoustic or physiological features extracted with sliding windows), and frame-level features as features that are measured at individual timestamps (e.g., facial features extracted at 1 frame per second (fps)). A-A1 Facial video features Frame-level features are extracted at 1 frame per second (fps) from recordings originally captured at 25 fps and include facial action units and landmarks (Da=155D_a=155), head pose (Dh=3D_h=3), and eyegaze (De=2D_e=2), using the Py-Feat library [50] and L2CS-Net [51]. For frames in which face detection or gaze estimation fails because of camera viewpoint, internet connectivity, or lighting, the corresponding temporal bins (defined in Section I-B2) are marked missing. TABLE IV: Model-input features. Feature Modality Type DmD_m AUs & landmarks Facial Frame-level 155 Head pose Facial Frame-level 3 Eyegaze Facial Frame-level 2 eGeMAPS Acoustic Window-level 88 ComParE Acoustic Window-level 6,373 Wav2Vec2 Acoustic Window-level 1,024 RoBERTa transcript Language Bin-level 768 rPPG heart rate Physiological Window-level 1 A-A2 Acoustic features Window-level features are extracted using 2-second windows with 1-second overlap. Features include eGeMAPS (Dg=88D_g=88) and ComParE (Dc=6,373D_c=6,373) extracted using openSMILE [53], and Wav2Vec2 embeddings (Dw=1,024D_w=1,024) [54] extracted using the Hugging Face Transformers library [55] from participant-side audio. A-A3 Language features Participant-side audio is segmented into 30-second windows with 5-second overlap, then transcribed using WhisperX [56] to obtain word-level timestamps. Words whose timestamps fall within a temporal bin (defined in Section I-B2) are concatenated and encoded with roberta-base [47, 24], yielding a Dt=768D_t=768-dimensional feature vector per bin, which we define as a bin-level feature. Interview questions are encoded with a sentence-level encoder (Sentence Transformer [57, 55]), yielding a dq=768d_q=768-dimensional embedding per question to condition the participant answers [57]. A-A4 Physiological features rPPG-derived heart rate (in beats per minute) is estimated using the pyVHR package [83, 84, 52] with a 6-second window and a 1-second stride, yielding one scalar estimate per window. A-B Post-hoc Multi-Level Attribution via Integrated Gradients (IG) Fig. 7: (a) After training, IG assigns a scalar attribution Ai,s,c(m)A_i,s,c^(m) to each feature c of modality-feature m at temporal bin s for participant i. Averaging the absolute attributions over the modality-feature’s DmD_m dimensions and gating by the availability mask (hatched bins are missing and contribute 0) reduces the attributions of all features c to one score per temporal bin ϕi,s(m) _i,s^(m). (b) These per-temporal-bin scores are aggregated along the different axes, and arrows indicate the dimensions along which the attributions are aggregated to produce the modality importance ImI_m, question importance IqI_q, within-question modality importance Hq,mrelH_q,m^rel, and temporal importance RuR_u. This section gives full derivations for the attribution aggregations summarized in Section I-D. We aggregate feature-level IG [49] scores at four levels of the interview structure to produce interpretable summaries across modalities, questions, and time. The aggregation scheme across these axes is shown in Fig. 7. A-B1 Base attribution scores Let Ai,s,c(m)A_i,s,c^(m) denote the IG scalar attribution for feature c of modality-feature m at temporal bin s for participant i. IG is computed over the concatenated participant-level feature time series i(m)∈ℝSi×DmX_i^(m) ^S_i× D_m (defined in Section I-B2), so s indexes temporal bins across the full interview. Attributions are averaged across feature vector dimensionality DmD_m within each modality-feature, so that modality-features of different dimensionalities contribute comparably to downstream aggregations [21]. This averaging produces score ϕi,s(m) _i,s^(m), the per-temporal-bin attribution for modality-feature m and participant i: ϕi,s(m)=Mi,s(m)⋅1Dm∑c=1Dm|Ai,s,c(m)|, _i,s^(m)=M_i,s^(m)· 1D_m _c=1^D_m |A_i,s,c^(m) |, (2) where Mi,s(m)M_i,s^(m) is the mask that ensures that unobserved temporal bins contribute zero attribution. Plots of ϕi,s(m) _i,s^(m) for representative participants across their complete interviews are shown in Fig. 12. The total attribution at temporal bin s from all modality-features is Φi,s=∑m∈ℳϕi,s(m) _i,s= _m _i,s^(m). To enable temporal dataset-level aggregation without bias from variation in interview length or modality availability across participants, attributions are normalized within each participant to sum to one: Φ~i,s=Φi,s∑s′Φi,s′,Φ~i,s(m)=ϕi,s(m)∑s′Φi,s′, _i,s= _i,s _s _i,s , _i,s^(m)= _i,s^(m) _s _i,s , (3) Here Φ~i,s _i,s is the share of participant i’s total attribution that falls at temporal bin s, and Φ~i,s(m) _i,s^(m) is the share attributable to modality-feature m at that bin. Both are expressed as fractions of the same denominator so that ∑mΦ~i,s(m)=Φ~i,s _m _i,s^(m)= _i,s at every temporal bin. A-B2 Modality Importance Let ℳi=m:∑sMi,s(m)>0M_i=\m: _sM_i,s^(m)>0\ be the set of modality-features observed at least once for participant i, and let =i:∑m∑sMi,s(m)>0P=\i: _m _sM_i,s^(m)>0\ be the set of all participants with at least one valid modality-feature observation. Dividing the summed attribution for modality-feature m by its number of observed bins, ∑sMi,s(m) _sM_i,s^(m), gives the mean attribution per observed bin, so features present in fewer bins are not penalized for contributing fewer terms. This yields the per-modality-feature mean attribution μi,m _i,m for each participant: μi,m=∑sϕi,s(m)∑sMi,s(m). _i,m= _s _i,s^(m) _sM_i,s^(m). (4) Within each participant, μi,m _i,m is normalized over observed modality-features so that contributions are comparable across participants. The per-participant normalization produces I~i,m I_i,m: I~i,m=μi,m∑m′∈ℳiμi,m′. I_i,m= _i,m _m _i _i,m . (5) We set I~i,m=0 I_i,m=0 for participants for whom modality-feature m was entirely absent. The dataset-level modality-feature importance (shown in Fig. 6(a)) aggregated over participants is: Im=1||∑i∈I~i,m.I_m= 1|P| _i I_i,m. (6) A-B3 Question Importance Let qi,sq_i,s denote the question identity of temporal bin s for participant i, let Ni,q=∑sqi,s=qN_i,q= _s1\q_i,s=q\ be the number of observed bins for question q, and let q=i:Ni,q>0P_q=\i:N_i,q>0\ be the set of participants who answered question q. We normalize Φ~i,s _i,s by Ni,qN_i,q to reduce bias toward longer answers. We then average that result over participants, producing the dataset-level question importance (shown in Fig. 12): Iq=1|q|∑i∈q1Ni,q∑s:qi,s=qΦ~i,s.I_q= 1|P_q| _i _q 1N_i,q _s:\,q_i,s=q _i,s. (7) A-B4 Within-Question Modality Importance Let Ni,q(m)=∑sqi,s=qMi,s(m)N_i,q^(m)= _s1\q_i,s=q\\,M_i,s^(m) be the number of observed bins for modality-feature m within question q for participant i, and let ℳi,q=m:Ni,q(m)>0M_i,q=\m:N_i,q^(m)>0\ be the modality-features observed at least once during that question. As described in Section A-B2, the per-bin attribution is normalized by observed bin count to avoid penalizing modality-features with more missing temporal bins: μi,q,m=∑s:qi,s=qϕi,s(m)Ni,q(m). _i,q,m= _s:\,q_i,s=q _i,s^(m)N_i,q^(m). (8) Within each participant and question, μi,q,m _i,q,m is normalized over observed modality-features so that contributions are comparable across participants: ri,q,m=μi,q,m∑m′∈ℳi,qμi,q,m′.r_i,q,m= _i,q,m _m _i,q _i,q,m . (9) The dataset-level within-question modality-feature importance (shown in Fig. 12) is: Hq,mrel=1|q|∑i∈qri,q,m.H_q,m^rel= 1|P_q| _i _qr_i,q,m. (10) A-B5 Temporal Importance Bins are within-answer temporal units, so their total count varies with interview length, and a bin does not correspond to a fixed recording position across participants. To place every participant on a common temporal axis for a cohort-level temporal importance summary, we group each participant’s bins into U equal-width buckets by relative position in the interview. Unlike bins, which measure absolute time within a single answer and vary in number per participant, buckets measure normalized position across the whole interview and are fixed at U buckets per participant. Thus, bucket u denotes the same relative point in the interview for all participants. We set U=20U=20, so each bucket spans 5%5\% of the normalized interview duration. Let ℬi,uB_i,u denote the set of bins for participant i in bucket u, and let u=i:|ℬi,u|>0P_u=\i:|B_i,u|>0\ be the set of participants with at least one bin in bucket u. Dividing each participant’s temporal-importance values Φ~i,s _i,s summed over bucket u by |ℬi,u||B_i,u| aggregates them into a single mean importance value. Therefore, a participant with many bins in that bucket does not contribute more than one with fewer bins. Averaging over the |u||P_u| participants combines the means across participants with bins in bucket u. Therefore, RuR_u is the mean per-bin temporal importance at relative position u in the interview: Ru=1|u|∑i∈u1|ℬi,u|∑s∈ℬi,uΦ~i,s.R_u= 1|P_u| _i _u 1|B_i,u| _s _i,u _i,s. (11) Here, Φ~i,s _i,s is the per-bin temporal importance defined in Section A-B1. RuR_u across the full interview is shown in Fig. 6(b). The modality-feature-specific temporal importance Ru(m)R_u^(m) is obtained by substituting Φ~i,s(m) _i,s^(m) for Φ~i,s _i,s. Ru(m)R_u^(m) across the full interview for eyegaze, head pose, and ComParE is likewise shown in Fig. 6(b). Supplementary B Experiments (a) Answer duration (median and IQR) (b) Text feature bin missingness Fig. 8: Answer duration and text-feature missingness by question group. (a) Median answer duration (point) and interquartile range (IQR) (bar) within each question group. Open-ended questions elicit longer answers than structured questions. (b) Percentage of answer-aligned one-second bins that contain no transcribed word, within each question group, with the overall rate (37.0%) marked by the dashed line. Text missingness rises as answer duration shortens, from 16% for open-ended questions to 49% for the UCLA Loneliness questions. B-A Dataset Each interview followed a fixed-order protocol comprising six consecutive question groups: four open-ended narrative prompts (including autobiographical recall and a picture description task using the Thematic Apperception Test (TAT) [63]), an 18-item social and functional health check derived from the I-CONECT protocol [64], the 30-item GDS [65], the GAD-7 [66], the 20-item UCLA Loneliness Scale [67], and the 12-item Lubben Social Network Scale (Social Engagement) [68]. Some questions were not answered by design. The interview used skip logic, so a question could be omitted based on a prior answer. For example, participants who answered “yes” to whether they live alone were not asked the follow-up question of how many people live with them. Other omissions arose from changes to the administered question groups across the study, such as the change from the short to the long version of the UCLA Loneliness Scale. The TAT question was unanswered for several participants due to difficulty showing the picture over Zoom. Because the interviewer annotated answer timestamps by hand, some questions with a spoken answer were occasionally marked as unanswered. Missingness varies across input modality-features. The acoustic features (eGeMAPS, ComParE, and Wav2Vec2) have less than 0.1% of bins missing across all participants. Visual and physiological features have low missingness rates (action units and landmarks: 3.0%; eyegaze: 2.6%; head pose: 2.5%; rPPG: 1.4%). One participant had no valid rPPG signal across the full interview. Language has the highest per-bin missingness (37.0%), because answer duration varies by question type. Open-ended narrative questions elicited the longest answers (median 87.5s) and had the lowest text missingness (16%), whereas Likert-scale and yes/no questions elicited short responses (median 3–8s) and had higher missingness, including 43.1% for the GDS, 46.3% for the GAD-7, 49.3% for the UCLA Loneliness scale, and 46.4% for the Lubben Social Engagement scale (Fig. 8). Furthermore, transcription errors are an additional source of missingness. For example, WhisperX sometimes leaves short fillers such as “Oh” or “Um” untranscribed even when spoken within an answer. In all cases, missing bins are assigned zero feature vectors and flagged by the modality-feature-timestep mask rather than excluded, as described in Section I-B3. The ComParE feature set is standardized and projected to 512 dimensions via Principal Component Analysis fitted on training-fold data, as the raw dimensionality of the feature is 6,3736,373. Participant feature time series are formed by concatenating bins across all answers, and the resulting number of bins ranged from 597 to 2260 across participants (median 935). B-B Training and Evaluation Protocol To mitigate class imbalance, training batches are constructed using a class-balanced sampler that oversamples participants from the minority class. Within each fold, all models are trained for ≈10≈ 10 epochs. After training, each model is evaluated once on that fold’s unseen, held-out test set. We report the AUROC score as the primary metric following previous work for mental health analysis on older adults with MCI [21]. We also report AUPRC, macro-F1, positive class precision, and recall scores in Table V. For significance testing, we also report pairwise statistical comparisons with one-sided Student’s paired t-tests over fold-level AUROC scores; Table VI reports whether the first model in each contrast outperformed the second. For each task, interpretability analysis is performed on the unseen test participants for the fold corresponding to the best-performing temporally aligned (++T) model across all repetition-fold pairs. Fig. 9: Question-conditioning models for the sequence-index-paired baseline. (a) Pooled additive conditioning. (b) Cross-attention conditioning. The stacked blocks denote the H=4H=4 heads, and the faded blocks denote additional timesteps processed by the same attention module with the same question keys and values. B-C Ablations B-C1 Sequence-Index-Paired Baseline To evaluate the impact of temporal alignment in the ablation experiments, we create a temporally unaligned version of the dataset. Each modality-feature is processed as an independent time series (m)∈ℝTm×DmX^(m) ^T_m× D_m, where TmT_m varies across modality-features according to their native sampling rates. Language features are represented as a single transcript-level embedding for the full interview rather than as a sequence of bin-level embeddings. At training time, we apply uniform downsampling to the shortest sequence length T within the batch, with t indexing timesteps within a single sequence. This follows strategies similar to those used to normalize feature time series to a global common temporal length by pooling [26] or cropping and interpolation [27]. Instead, we set the target length per batch rather than globally because interview durations range from 600600 to 3,6003,600s, so a single fixed length would either discard most of the longer interviews or heavily upsample the shorter ones. This sequence-index-paired baseline follows prior approaches that pair features by their index position in the sequence rather than by timestamps during training [26, 27]. B-C2 Question-Conditioning Models Pooled Additive Conditioning (Linear/Non-linear + Q, sum) For the sequence-index-paired baseline, the multimodal feature time series are not segmented by answer timestamps or temporally aligned. We therefore adapt the additive conditioning that Zhang et al. [25] apply to each question and its associated response to operate at the interview level. Their approach is discussed further in the Related Work section of the main text (Section I-D). In our adaptation, all question embeddings are averaged into a single interview-level summary ¯i q_i, and projected to d (defined in Section I-C1) by WqW_q. This projected vector is then added uniformly across timesteps to the fused multimodal sequence, similar to additive global conditioning in WaveNet [85]: ¯i=1Ji∑j=1Jiij,~i,t=i,t+Wq¯i, q_i= 1J_i _j=1^J_iq_ij, z_i,t=z_i,t+W_q q_i, (12) where Wq∈ℝd×dqW_q ^d× d_q, ijq_ij is the embedding of the j-th question answered by participant i, i,tz_i,t is the fused multimodal token at timestep t, and ~i,t z_i,t is the question-conditioned multimodal token. The conditioned sequence ~i Z_i formed by stacking the conditioned tokens is then passed to the Transformer encoder as described in Section I-C2. Cross-Attention Conditioning (Linear/Non-linear + Q, cross-attn.) We generalize the question-conditioned attention used in HCAG [34] to the sequence-index-paired baseline by replacing additive attention with cross-attention. Further discussion of the HCAG approach is provided in the Related Work section of the main text (Section I-D). In our adaptation, each fused multimodal token i,tz_i,t attends to the participant’s set of JiJ_i answered question embeddings through a multi-head cross-attention block [86]. We use H=4H=4 heads, model dimension d=128d=128, per-head dimension dh=d/H=32d_h=d/H=32, and dropout =0.3=0.3. The question embeddings ijj=1Ji\q_ij\_j=1^J_i are projected to d by WqW_q. For head h, the fused multimodal token i,tz_i,t at timestep t forms the query, and each projected question embedding WqijW_qq_ij forms the keys and values: i,th=WQhi,t,ijh=WKh(Wqij),Q_i,t^h=W_Q^hz_i,t, _ij^h=W_K^h(W_qq_ij), (13) ijh=WVh(Wqij),V_ij^h=W_V^h(W_qq_ij), (14) where WQh,WKh,WVh∈ℝdh×dW_Q^h,W_K^h,W_V^h ^d_h× d, and Wq∈ℝd×dqW_q ^d× d_q. The attention weight from timestep t to question j, normalized across the participant’s answered questions, is αi,t,jh _i,t,j^h: αi,t,jh=softmaxj((i,th)⊤ijhdh). _i,t,j^h=softmax_j\! ( (Q_i,t^h) K_ij^h d_h ). (15) The resulting question context i,th∈ℝdhc_i,t^h ^d_h is the attention-weighted combination of the projected question embeddings: i,th=∑j=1Jiαi,t,jhijh.c_i,t^h= _j=1^J_i _i,t,j^hV_ij^h. (16) The head question contexts are concatenated, projected by WO∈ℝd×dW_O ^d× d to form i,tc_i,t, which is added back to the fused multimodal token i,tz_i,t through a residual connection with dropout and layer normalization, giving ~i,t z_i,t, the question-conditioned multimodal token: i,t=WO[i,t1;…;i,tH],c_i,t=W_O [c_i,t^1;…;c_i,t^H ], (17) ~i,t=LayerNorm(i,t+Dropout(i,t)), z_i,t=LayerNorm\! (z_i,t+Dropout(c_i,t) ), (18) where i,t,i,t,c_i,t,z_i,t, and ~i,t∈ℝd z_i,t ^d. Each fused multimodal token therefore receives a learned mixture of all of the participant’s question embeddings. The conditioned sequence is then passed to the Transformer encoder as described in Section I-C2. Supplementary C Results C-A Overall Performance TABLE V: Classification results (mean ± 95% CIs). Precision and recall are reported for the positive class (positive depression- or anxiety-risk class) at a classification threshold of 0.5. Bold, underline, and italics denote the highest, second-highest, and third-highest results within each task and metric, respectively. Tied results receive the same rank. T denotes temporally aligned features, and Q denotes question conditioning. Linear and Non-linear denote the different projection backbones. The no-mask variant replaces the mask with an all-ones mask, effectively disabling masking, whereas the open-only variant uses only multimodal features extracted from participants’ responses to the four open-ended questions. Depression Anxiety Variant AUPRC ↑ Macro-F1 ↑ Precision+ ↑ Recall+ ↑ AUPRC ↑ Macro-F1 ↑ Precision+ ↑ Recall+ ↑ Sequence-index-paired baseline [26, 27] Linear 0.56±0.0040.56± 0.004 0.45±0.080.45± 0.08 0.34±0.120.34± 0.12 0.31±0.160.31± 0.16 0.57±0.020.57± 0.02 0.55±0.070.55± 0.07 0.49±0.150.49± 0.15 0.38±0.150.38± 0.15 Non-linear 0.60±0.090.60± 0.09 0.52±0.080.52± 0.08 0.50±0.180.50± 0.18 0.43±0.140.43± 0.14 0.58±0.050.58± 0.05 0.50±0.070.50± 0.07 0.40±0.140.40± 0.14 0.27±0.090.27± 0.09 Question-conditioned Linear + Q, sum [25] 0.56±0.010.56± 0.01 0.48±0.040.48± 0.04 0.42±0.060.42± 0.06 0.43±0.130.43± 0.13 0.54±0.060.54± 0.06 0.48±0.030.48± 0.03 0.30±0.110.30± 0.11 0.26±0.070.26± 0.07 Linear + Q, cross-attn. [34] 0.50±0.060.50± 0.06 0.47±0.070.47± 0.07 0.43±0.110.43± 0.11 0.39±0.100.39± 0.10 0.54±0.030.54± 0.03 0.53±0.07¯ 0.53± 0.07 0.43±0.190.43± 0.19 0.38±0.130.38± 0.13 Non-linear + Q, sum [25] 0.61±0.030.61± 0.03 0.51±0.080.51± 0.08 0.50±0.070.50± 0.07 0.39±0.120.39± 0.12 0.51±0.080.51± 0.08 0.50±0.070.50± 0.07 0.34±0.080.34± 0.08 0.38±0.070.38± 0.07 Non-linear + Q, cross-attn. [34] 0.61±0.130.61± 0.13 0.52±0.070.52± 0.07 0.46±0.120.46± 0.12 0.41±0.060.41± 0.06 0.55±0.080.55± 0.08 0.49±0.110.49± 0.11 0.33±0.220.33± 0.22 0.38±0.260.38± 0.26 Temporally aligned Linear + T (no mask) 0.68±0.09¯ 0.68± 0.09 0.57±0.03¯ 0.57± 0.03 0.59±0.070.59± 0.07 0.54±0.16¯ 0.54± 0.16 0.55±0.050.55± 0.05 0.49±0.070.49± 0.07 0.31±0.080.31± 0.08 0.34±0.100.34± 0.10 Linear + T 0.67±0.080.67± 0.08 0.54±0.070.54± 0.07 0.58±0.19¯ 0.58± 0.19 0.53±0.120.53± 0.12 0.63±0.120.63± 0.12 0.51±0.090.51± 0.09 0.48±0.29¯ 0.48± 0.29 0.34±0.180.34± 0.18 Non-linear + T (no mask) 0.66±0.040.66± 0.04 0.53±0.050.53± 0.05 0.50±0.130.50± 0.13 0.36±0.100.36± 0.10 0.54±0.090.54± 0.09 0.48±0.020.48± 0.02 0.30±0.030.30± 0.03 0.38±0.050.38± 0.05 Non-linear + T 0.68±0.04¯ 0.68± 0.04 0.56±0.020.56± 0.02 0.54±0.030.54± 0.03 0.53±0.050.53± 0.05 0.57±0.080.57± 0.08 0.47±0.050.47± 0.05 0.40±0.020.40± 0.02 0.37±0.050.37± 0.05 Aligned + question Linear + Q + T 0.69±0.110.69± 0.11 0.52±0.070.52± 0.07 0.55±0.170.55± 0.17 0.48±0.130.48± 0.13 0.60±0.06¯ 0.60± 0.06 0.52±0.020.52± 0.02 0.39±0.190.39± 0.19 0.40±0.04¯ 0.40± 0.04 Non-linear + Q + T 0.68±0.06¯ 0.68± 0.06 0.58±0.080.58± 0.08 0.52±0.130.52± 0.13 0.51±0.160.51± 0.16 0.58±0.010.58± 0.01 0.48±0.050.48± 0.05 0.34±0.140.34± 0.14 0.41±0.270.41± 0.27 Open-ended only Linear + T (open only) – 0.52±0.070.52± 0.07 0.49±0.120.49± 0.12 0.30±0.210.30± 0.21 0.37±0.190.37± 0.19 Non-linear + T (open only) 0.69±0.080.69± 0.08 0.57±0.06¯ 0.57± 0.06 0.50±0.050.50± 0.05 0.55±0.080.55± 0.08 – TABLE VI: Pairwise statistical comparisons for depression and anxiety classification. Δ denotes the mean AUROC difference between the first and second model. Comparisons use one-sided paired t-tests with the alternative Δ>0 >0. Bold p values indicate p<0.05p<0.05. Depression Anxiety Contrast ↑ p ↑ p Linear Q (sum) >> Baseline 0.01 -0.01 0.310.31 −0.03-0.03 0.800.80 Q (cross-attn.) >> Baseline −0.06-0.06 0.960.96 −0.02-0.02 0.770.77 T >> Baseline 0.13 -0.13 0.050.05 0.11 -0.11 0.050.05 T (mask) >> T (no mask) −0.009-0.009 0.560.56 0.04 -0.04 0.120.12 T (all questions) >> T (open only) – 0.09 -0.09 0.020.02 Q+T >> Baseline 0.16 -0.16 0.0090.009 0.09 -0.09 0.090.09 Q+T >> Q (sum) 0.15 -0.15 0.0090.009 0.12 -0.12 0.040.04 Q+T >> Q (cross-attn.) 0.22 -0.22 0.0010.001 0.12 -0.12 0.060.06 Q+T >> T 0.03 -0.03 0.280.28 −0.02-0.02 0.640.64 Non-linear Q (sum) >> Baseline −0.03-0.03 0.730.73 −0.02-0.02 0.660.66 Q (cross-attn.) >> Baseline −0.02-0.02 0.620.62 0.000 -0.000 0.500.50 T >> Baseline 0.10 -0.10 0.060.06 0.02 -0.02 0.390.39 T (mask) >> T (no mask) 0.02 -0.02 0.340.34 0.02 -0.02 0.320.32 T (all questions) >> T (open only) 0.01 -0.01 0.430.43 – Q+T >> Baseline 0.09 -0.09 0.080.08 0.05 -0.05 0.210.21 Q+T >> Q (sum) 0.13 -0.13 0.020.02 0.07 -0.07 0.100.10 Q+T >> Q (cross-attn.) 0.11 -0.11 0.0010.001 0.05 -0.05 0.180.18 Q+T >> T −0.009-0.009 0.600.60 0.03 -0.03 0.200.20 T = temporally aligned features; Q = question conditioning; Q+T = question conditioning with temporal alignment; Baseline = sequence-index-paired model. Mask contrasts compare the real temporal mask against an all-ones mask. Open-only models use only multimodal features from responses to the four open-ended questions. Each comparison uses 15 paired observations (df=14df=14). C-B Question Importance Fig. 12 decomposes the attribution rankings by question groups for both tasks. For depression (Fig. 12(a)), the highest-attribution questions center on social contact and support, self-rated health, and open-ended self-description. The top five questions are Q10 (I-CONECT: overnight visitors), Q89 (Social Engagement: availability of a relative for decision support), Q5 (I-CONECT: self-rated health), Q1 (open-ended: “Tell me about yourself”), and Q90 (Social Engagement: providing decision support to a friend), with attributions of (2.151,1.702,1.681,1.648, and 1.509)×10−3(2.151,1.702,1.681,1.648, and 1.509)× 10^-3, respectively. For anxiety (Fig. 12(b)), the top questions span a narrower range of 0.24×10−30.24× 10^-3 and center on crying and social contact with friends. Q47 (GDS: crying frequency) leads at 1.759×10−31.759× 10^-3, followed by Q91 (Social Engagement: availability of a friend for decision support, 1.636×10−31.636× 10^-3), Q90 (Social Engagement: providing decision support to a friend, 1.578×10−31.578× 10^-3), Q20 (I-CONECT: written communication with friends or family, 1.578×10−31.578× 10^-3), and Q17 (I-CONECT: contact with friends, 1.522×10−31.522× 10^-3). Within-question modality-feature importance Fig. 12 shows the within-question modality-feature share (Hq,mrelH_q,m^rel) for the top-ranked questions in each task. The modality-feature rankings from Fig. 6(a) are preserved at the question level. For depression, eyegaze remains the top contributor across every shown question, followed by ComParE. For anxiety, eyegaze and head pose retain their order of importance regardless of question group. C-C Single Participant Temporal Importance Fig. 12 shows IG attribution over interview time for one representative participant per task, for the three highest-contributing modality-features from Fig. 6(a). Each participant comes from the test fold of the best-performing model for the corresponding task (Non-linear+T for depression and Linear+T for anxiety). For each task, we selected a repetition–fold combination with a test AUROC >0.8>0.8 and chose the most confidently classified positive case from that fold (confidence >0.9>0.9). The attribution scores were smoothed using a 15-s window for plotting. For the depression participant (Fig. 12(a)), eyegaze has the highest attribution across all questions. Total attribution peaks during the open-ended block, driven by a sharp increase in eyegaze. ComParE contributes mainly at the beginning of the interview. Attribution decreases through the middle of the interview. This pattern matches the temporal importance findings (Fig. 6(b)), where eyegaze drives the early-interview concentration for depression. For the anxiety participant (Fig. 12(b)), eyegaze and head pose remain the main contributors across all questions with comparable attribution. rPPG contributes mainly during the open-ended block and decreases afterward. This pattern matches the population-level finding that anxiety attribution is distributed more uniformly across question groups, with comparable contributions from eyegaze and head pose and without the early attribution concentration observed for depression (Fig. 6(b)). (a) Depression (b) Anxiety Fig. 10: IG question-level importance grouped by question group for depression and anxiety prediction. Bars show normalized question-attribution scores for each question group. (a) Depression (b) Anxiety Fig. 11: IG within-question modality-feature importance for depression and anxiety prediction. Each row shows the normalized modality-feature-attribution share for a high-ranking interview question. Across question groups, eyegaze remains dominant for depression, whereas anxiety shows a more balanced attribution between eyegaze and head pose. (a) Depressed participant (b) Anxious participant Fig. 12: Single-participant IG temporal importance for the three highest-attribution modality-features. For depression, these features are eyegaze, ComParE, and head pose. Eyegaze dominates throughout the interview, with peaks during the open-ended question block. For anxiety, the top features are eyegaze, head pose, and rPPG. Eyegaze and head pose contribute comparably across all questions, while rPPG shows a peak in the open-ended question block. The colored strip below each panel marks question-group boundaries.