Paper deep dive
Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network with Domain-Adversarial Training
Ali Tabaraei, Federico Simonetta, Stavros Ntalampiras
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/2/2026, 1:22:29 PM
Summary
The paper proposes a multimodal domain generalization framework for automatic depression detection using acoustic and textual data. The model employs Bidirectional Long Short-Term Memory (BiLSTM) networks with intra- and cross-modal attention mechanisms, enhanced by a gradient reversal layer inspired by Domain-Adversarial Training of Neural Networks (DANN) to learn domain-invariant representations. Experiments on the Androids-Corpus dataset demonstrate that combining MelSpec and ItalianBERT with domain generalization achieves state-of-the-art performance (93.2% accuracy, 94.2% F1-score), outperforming existing benchmarks by reducing patient-specific bias.
Entities (11)
Relation Signals (7)
Multimodal Domain Generalization Framework → evaluatedon → Androids Corpus
confidence 99% · Conducting experiments on the Androids-Corpus dataset
Multimodal Domain Generalization Framework → uses → BiLSTM
confidence 95% · The proposed model integrates bidirectional Long Short-Term Memory (BiLSTM) with intra- and cross-modal attention mechanisms
Multimodal Domain Generalization Framework → uses → DANN
confidence 92% · applying a gradient reversal layer inspired by Domain-Adversarial Training of Neural Networks (DANN)
Multimodal Domain Generalization Framework → achievesbestwith → MelSpec
confidence 90% · determining MelSpec and ItalianBERT as the optimal baseline
Multimodal Domain Generalization Framework → achievesbestwith → ItalianBERT
confidence 90% · determining MelSpec and ItalianBERT as the optimal baseline
Multimodal Domain Generalization Framework → outperforms → existing benchmarks
confidence 90% · surpassing all existing benchmarks
DANN → implementsvia → Gradient Reversal Layer
confidence 88% · applying a gradient reversal layer inspired by Domain-Adversarial Training of Neural Networks (DANN)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Automatic depression detection with deep learning has shown promise but often suffers from limited generalization due to domain shift arising from inter-speaker variability. To address this critical issue, we present the first patient-independent multimodal depression detection framework that incorporates domain generalization (DG), jointly leveraging both acoustic and textual modalities. The proposed model integrates bidirectional Long Short-Term Memory (BiLSTM) with intra- and cross-modal attention mechanisms, accompanied by segment-level fusion for decision-making. Generalization is further enhanced by applying a gradient reversal layer inspired by Domain-Adversarial Training of Neural Networks (DANN), which promotes domain-invariant representations by adversarially limiting the model's ability to identify individual speakers, effectively reducing patient-specific bias. Conducting experiments on the Androids-Corpus dataset with a 5-fold cross-validation (CV) protocol, various pairings of audio and text feature extractors were evaluated over different segment durations, determining MelSpec and ItalianBERT as the optimal baseline at a 30-second segment duration. The addition of DG to this baseline yields a 2.5% increase in accuracy and 3.3% in F1-score, achieving 93.2% accuracy, 93.2% precision, 96.2% recall, and 94.2% F1-score, surpassing all existing benchmarks. Extensive ablation studies assess the impact of multimodal fusion, deep architectural choices, and DG, highlighting their combined contribution to robust and generalizable depression detection.
Tags
Links
- Source: https://arxiv.org/abs/2607.22794v1
- Canonical: https://arxiv.org/abs/2607.22794v1
Trouble viewing inline? Open PDF directly →
Full Text
71,573 characters extracted from source content.
Expand or collapse full text
Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network with Domain-Adversarial Training Ali Tabaraei, Federico Simonetta, and Stavros Ntalampiras Ali Tabaraei and Stavros Ntalampiras are with the Department of Computer Science, University of Milan, Milan, Italy. Emails: ali.tabaraei@unimi.it and stavros.ntalampiras@unimi.itFederico Simonetta is with the Computer Science Department, Gran Sasso Science Institute (GSSI), L’Aquila, Italy. Email: federico.simonetta@gssi.itManuscript received 6 October 2025; revised 5 March 2026 and 13 June 2026; accepted 12 July 2026. (Corresponding author: Ali Tabaraei.) Abstract Automatic depression detection with deep learning has shown promise but often suffers from limited generalization due to domain shift arising from inter-speaker variability. To address this critical issue, we present the first patient-independent multimodal depression detection framework111Source code and documentation for this work can be accessed publicly at: https://github.com/tabaraei/MultimodalDG-depression-detection that incorporates domain generalization (DG), jointly leveraging both acoustic and textual modalities. The proposed model integrates bidirectional Long Short-Term Memory (BiLSTM) with intra- and cross-modal attention mechanisms, accompanied by segment-level fusion for decision-making. Generalization is further enhanced by applying a gradient reversal layer inspired by Domain-Adversarial Training of Neural Networks (DANN), which promotes domain-invariant representations by adversarially limiting the model’s ability to identify individual speakers, effectively reducing patient-specific bias. Conducting experiments on the Androids-Corpus dataset with a 5-fold cross-validation (CV) protocol, various pairings of audio and text feature extractors were evaluated over different segment durations, determining MelSpec and ItalianBERT as the optimal baseline at a 30-second segment duration. The addition of DG to this baseline yields a 2.5% increase in accuracy and 3.3% in F1-score, achieving 93.2% accuracy, 93.2% precision, 96.2% recall, and 94.2% F1-score, surpassing all existing benchmarks. Extensive ablation studies assess the impact of multimodal fusion, deep architectural choices, and DG, highlighting their combined contribution to robust and generalizable depression detection. †publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE I Introduction Figure 1: Overview of the proposed DG framework for multimodal depression detection. Each speaker is represented as a distinct domain to model unique variability in audio-textual patterns. The domain discriminator constrains the representations learned by the feature extractor to suppress speaker-specific traits while retaining depression-related cues. This adversarial setup promotes domain-invariant features and enhances generalization to unseen domains. Depression, one of the most prevalent mental health disorders worldwide [43], can progress to severe outcomes such as self-harm or suicide if left untreated [5]. This risk underscores the urgent need for reliable automated screening methods to complement existing preventive interventions [52]. To enable automatic depression detection, recent advances in Artificial Intelligence (AI) have motivated systematic efforts to collect and analyze behavioral and physiological data, including speech, language, facial expressions, and biological signals, as distinctive indicators of depressive states [51, 34, 55]. Such data are then typically processed via feature extraction and modeling phases to uncover depression-related patterns. Among these modalities, speech and linguistic features have proven to be particularly informative biomarkers of depression. Specifically, depressed patients often exhibit recurring acoustic patterns [26], such as reduced pitch variability, slower speaking rate, and longer pauses [11], as well as linguistic traits like self-focused expressions and negative sentiments [17]. Although standard audio-textual machine learning (ML) models have demonstrated improved performance in detecting depression [22], a key barrier to their clinical deployment remains: the lack of subject-independent generalizability. Each individual exhibits unique speech characteristics and language use, which can cause models to capture personal behavioral markers rather than pathological indicators of depression. Consequently, despite strong performance on seen subjects, these models often struggle to generalize to new patients due to inter-speaker variability [35]. To ensure reliability in real-world screening, diagnostic models must be resilient to such variations. Domain adaptation (DA) [15] and domain generalization (DG) [58] are two approaches addressing performance degradation under domain shift [23], which occurs when training data (source domains) differ from that encountered during inference (target domains). While DA requires access to target data during training, DG relies solely on source domains to simulate domain shifts, making it particularly suited to enhancing robustness when diagnosing unseen patients in depression detection [42]. In order to implement the DG paradigm, as shown in Fig. 1, we treat each patient’s unique audio-textual data as a separate domain, inspired by [40]. Employing this perspective allows us to model the domain shift as inter-speaker variability, with source and target domains corresponding to patients in the training and test sets, respectively. While a variety of DG methods have been explored in the literature, domain-invariant representation learning techniques are extensively studied and proven effective [48]. In particular, these approaches encourage the learning of representations that ignore domain-specific confounding factors while preserving the discriminative patterns of the target task [38]. Within this category, we adopt domain-adversarial training of neural networks (DANN) proposed by Ganin et al. [16] to guide the standard ML pipeline. Our framework incorporates a domain discriminator equipped with a gradient reversal layer (GRL) [16], which penalizes the model when it correctly identifies the speaker. This adversarial mechanism drives the feature extractor to learn representations that are invariant to speaker-specific traits, yet remain pathologically informative. It thus facilitates a privacy-preserving, participant-independent depression detector that generalizes to unseen patients. The proposed framework leverages a multi-source DG paradigm [58], jointly modeling acoustic and textual features via a model-based fusion strategy [25, 28], all encapsulated within a unified architecture. Our key contributions include: • Proposing a novel multimodal audio-textual architecture with BiLSTM encoders, intra- and cross-modal attention modules, and segment-level decision-making. • Conducting a comprehensive analysis of different feature extractor pairs for audio (MelSpec, HuBERT, Wav2Vec2) and text (BERT, ItalianBERT, XLMRoBERTa) alongside varying segment durations. • Evaluating the relative contribution of each modality and architectural component with extensive ablation studies. • Achieving state-of-the-art results of 93.2% accuracy and 94.2% F1-score after incorporating DG, outperforming prior benchmarks despite using 20% less training data. The rest of this paper is organized as follows: Section I reviews prior studies for comparison, summarizes multimodal approaches, and highlights the relevance of DG. Section I formalizes the problem, providing an introductory context for Section IV, which details different modules of the proposed methodology. Section V presents the experimental setup with a standardized protocol, followed by Section VI, which analyzes the results and presents a thorough ablation study assessing the proposed approach from diverse perspectives. Finally, Sections VIII and VII summarize our conclusions and outline directions for future work. I Related Work This section summarizes prior depression detection studies as a reference for comparison, provides a brief but informative overview of multimodal depression detection, and discusses the relevance of domain generalization in this context. TABLE I: Overview of depression classification studies on the Androids-Corpus. Some studies used reading (R) or interview (I) data exclusively, while others reported results for both in fusion (R∧I)(R I) or separately (R∨I)(R I), adopting speech-only (S) or multimodal (M) approaches. Highlighted rows are not comparable to our study as we solely use the interview data. Features (Model) Data Modality OpenSMILE (SVM, LSTM) [44] R∨IR I S x-vector + emoHuBERT + TRILLsson (PTM) [31] R∨IR I S MFCC + TEO + periodicity (HMM + MAP) [30] R∨IR I S OpenSMILE (LSTM-CDMA) [45] R∧IR I S AlexNet (Mixture of Experts) [21] R∧IR I S MFCC (Adaptive Knowledge Fusion) [59] R∧IR I S STFT spectrograms + ResNet-18 (TFCA) [36] R∧IR I S OpenSMILE (CNN-LSTM) / TF-IDF (BERT) [8] R∧IR I M OpenSMILE correlations (SVM, LSTM) [46] R S eGeMAPS, TRILLsson4 (XGBoost) [33] R S OpenSMILE (CNN-BiLSTM + RL) [27] R S PDEM embeddings (KELM) [54] I S HuBERT, Wav2Vec2, Whisper (LR) [9] I S DepAudioNet, ResNet, ECAPA-TDNN (MDFA) [50] I S DepAudioNet, ECAPA-TDNN (MTFS-Block) [49] I S Wav2Vec2 (CNN-GRU) / BERT (GRU) [3] I M AlexNet / ItalianBERT (Cross-Attention) [20] I M OpenSMILE, Wav2Vec2 / LIWC (FFN + MV) [2] I M Wav2Vec2 / XLM-RoBERTa (D-CoPE + QF) [56] I M I-A Depression Detection on Androids-Corpus For comparative purposes in Section VI-C, here we present prior depression detection works on the Androids-Corpus [44] dataset (see Section V-A). This dataset contains recordings of interview (I) and reading (R) tasks, prompting researchers to adopt a variety of strategies, using these data types either independently or in combination, as outlined in Table I. Some studies reported depression detection results on each data type separately (R∨IR I). The initial baselines by Tao et al. [44] utilized OpenSMILE features, with BS1 employing an SVM and BS2 an LSTM with segment-level majority voting. Phukan et al. [31] proposed FuSeR, a deep fusion model with x-vector, TRILLsson, and emoHuBERT embeddings, whereas Ntalampiras [30] achieved competitive performance using an interpretable HMM-based approach with MFCCs, TEO autocorrelation envelopes, and periodicity features. Conversely, some studies integrated both data types in their experiments (R∧IR I). Tao et al. [45] developed Cross-Data Multilevel Attention (CDMA), employing attention-enabled LSTMs over OpenSMILE features. Ilias et al. [21] stacked log-Mel spectrograms with their first- and second-order derivatives into a 3D representation, subsequently extracting features via AlexNet and processing them with a Mixture of Experts (MoE) architecture. Daly and Olukoya [8] processed OpenSMILE features using a hybrid CNN-LSTM while encoding TF-IDF transcripts with BERT, fusing modality-specific outputs at the decision level. Zhou et al. [59] utilized an Adaptive Knowledge Fusion (AKF) framework that jointly models temporal and channel-domain information from MFCC features. A similar study by Rezaee [36] combined a ResNet-18 backbone and a Temporal-Frequency-Channel Attention (TFCA) mechanism to process STFT spectrogram segments. Other investigations focused solely on a specific data type. Using the reading data (R), Tao et al. [46] predicted depression with OpenSMILE-derived correlation representations, whereas Polle et al. [33] fed eGeMAPS and TRILLsson4 features into XGBoost to identify confounding biases. With a brain-inspired reinforcement learning architecture, Li et al. [27] processed OpenSMILE features via 1-D CNN and BiLSTM modules. Using the interview data (I), Yu et al. [54] fine-tuned a Wav2Vec2-based PDEM model and used a Kernel Extreme Learning Machine (KELM) for classification, while de Gennes et al. [9] benchmarked HuBERT, Wav2Vec2, and Whisper embeddings with logistic regression. Alsenani et al. [3] processed Wav2Vec2 audio and BERT text embeddings within a GRU network. Similar to [21], Ilias et al. [20] extracted 3D AlexNet-based log-Mel spectrograms and ItalianBERT text features, applying cross-attention instead. Among recent works, Alsarrani et al. [2] employed majority voting over audio (OpenSMILE or Wav2Vec2) and LIWC text features. Zhang and Poellabauer [56] proposed Dialogue-based CoPE (D-CoPE) on Wav2Vec2 and XLM-RoBERTa, leveraging an adversarial setup with GRL and dialogue-level transformers. Similarly, Wang et al. [50] proposed Multi-Domain Feature Alignment (MDFA), using GRL to learn domain-invariant representations from filter banks via different encoders. In their latest work [49], they used a Multiple Temporal–Frequency Scale Block (MTFS-Block) and DWT to enhance DepAudioNet and ECAPA-TDNN baselines. Most of these studies relied solely on acoustic features, overlooking potential gains from textual information. A further limitation is the lack of a dedicated validation set in their experimental design. While [54] used nested cross-validation and [30, 36, 59] reserved a portion of the training data for validation, others trained for a fixed number of epochs, increasing the risk of overfitting and model selection bias. I-B Multimodal Depression Detection Depression manifests through diverse behavioral signals, making multimodal approaches that integrate audio, text, and visual data well-suited for automated detection. By capturing complementary cues, such methods can typically outperform unimodal systems. Early research by Yang et al. [53] fused audio, video, and text descriptors using a deep CNN, while Haque et al. [18] later applied a causal CNN to model acoustic, linguistic, and facial features. In a more recent study, Li et al. [28] introduced IISFD, achieving strong results by integrating visual feature extractors with acoustic and textual features through contrastive learning. Notably, a major line of research has focused on bimodal audio-text fusion. Early work by Tuka et al. [1] modeled audio and text with separate LSTMs, merging the resulting representations in a feedforward network. Later, Makiuchi et al. [37] estimated depression status by fusing BERT-based text embeddings with speech features extracted via a pretrained VGG-16 network, further processed using CNN, gated CNN, and LSTM layers. Toto et al. [47] proposed AudiBERT, a dual self-attentive BiLSTM framework integrating BERT with pretrained audio encoders such as VGGish, SincNet, or Wav2Vec. Shen et al. [41] used a GRU-based network on audio Mel spectrograms and a BiLSTM on sentence embeddings, combined through modal attention to predict depression. More recently, Jia et al. [24] introduced the bidirectional multimodal block-recurrent transformer (BMBRT), combining BERT and HuBERT for text and audio, respectively. Ding et al. [12] then presented IntervoxNet, integrating a hybrid BERT-CNN text encoder with Audio Mel-Spectrogram Transformer (AMST). Employing BERT for text and Mel spectrograms for audio, Chen et al. [6] achieved promising results with a BiLSTM, cross-attention, and transformer-based fusion network. The demonstrated success of audio-textual approaches [12, 24] and the effectiveness of intra- and cross-modal attention modules [28, 20] motivated us to adopt a similar strategy, yielding an end-to-end framework that serves as a comparative baseline prior to integrating DG. I-C Domain Generalization In general, DG has been extensively studied in computer vision, medical imaging, and natural language processing, but its application to speech or multimodal tasks remains limited [48, 58]. In the context of multimodal DG, Zhang et al. [57] introduced the DeVADG framework, which addresses video-audio domain generalization from a causal perspective by disentangling confounding factors. Planamente et al. [32] proposed a novel audio-visual loss function to balance the contributions of both modalities across domains by aligning their feature norms. Dong et al. [14] developed SimMMDG, a contrastive learning approach on the modality-shared features with distance constraints on modality-specific representations to encourage diversity. In another study, Dong et al. [13] presented MOOSA, which tackles multimodal open-set domain generalization through self-supervised learning. Despite these advances, multimodal speech-based DG for depression detection remains largely unexplored. Similar to our approach, Wang et al.[50] and Liu et al.[29] employed GRL to learn domain-invariant representations, but the former focused on unimodal DA, and the latter conducted unimodal DG using EEG rather than speech. Zhang and Poellabauer [56] is the closest work to ours, applying GRL for multimodal DG. However, they aim to mitigate bias arising from the inclusion of the interviewer’s transcripts, while our adversarial setup targets depression-specific features robust to the inter-speaker variability. Unlike previous works, we investigate multimodal DG specifically for patient-independent depression detection, bridging a critical gap in the literature. I Problem Formulation Consider a dataset with an arbitrary number NiN_i of variable-length audio recordings collected from each speaker i during an interview. These recordings are then preprocessed into KiK_i fixed-size audio segments, and the corresponding transcripts are obtained. Let X denote the joint audio-textual feature space and Y the binary label space for depression. We treat each participant as a distinct domain, for whom a joint distribution PXYP_XY over ×X×Y is observed. Given m source domains defined as i()i=1m\D_i^(S)\_i=1^m, each domain can be represented as: i()=(i,k(a),i,k(t),di,yi)k=1KiD_i^(S)= \(x_i,k^(a),x_i,k^(t),d_i,y_i) \_k=1^K_i (1) where i,k(a)x_i,k^(a) and i,k(t)x_i,k^(t) represent the acoustic and textual features extracted by a modality-specific feature extractor for segment k of the i-th domain; di∈1,…,md_i∈\1,…,m\ is the source domain label; and yi∈0,1y_i∈\0,1\ indicates the depression status. During training, a deep neural network referred to as the multimodal feature extractor (fθf_θ) maps each audio-textual pair to a latent representation i,k=fθ(i,k(a),i,k(t))∈ℝdz_i,k=f_θ(x_i,k^(a),x_i,k^(t)) ^d. This representation is expected to preserve depression-related cues while suppressing speaker-specific characteristics. To this end, the following modules are introduced: • A depression detector (hϕh_φ), predicting depression logits y^i,k=hϕ(i,k)∈ℝ y_i,k=h_φ(z_i,k) . The detection error is measured via binary cross-entropy, such that computing BCE(y^i,k,yi)BCE( y_i,k,y_i) across all KiK_i segments of m source domains yields the overall depression loss ℒdep(θ,ϕ)L_dep(θ,φ). Consequently, this loss is minimized with respect to both θ and ϕφ, enabling more accurate depression detection. • A domain discriminator (gψg_ψ), predicting domain logits d^i,k=gψ(i,k)∈ℝm d_i,k=g_ψ(z_i,k) ^m. The overall discrimination error ℒdom(θ,ψ)L_dom(θ,ψ) is defined by computing the cross-entropy loss CE(d^i,k,di)CE( d_i,k,d_i) across all KiK_i segments of m source domains. While gψg_ψ aims to minimize ℒdomL_dom to better identify domains, fθf_θ maximizes it adversarially, tricking gψg_ψ so that it fails to capture domain-specific features. This adversarial training strategy promotes domain-invariant representations, thereby enabling generalization to n unseen target domains j()j=1n\D_j^(T)\_j=1^n. IV The Proposed Methodology This section presents an overview of the entire pipeline, elaborating on the key components introduced in Section I and their interplay, specifically the a) preprocessing module, b) audio-text feature extractor specifications, c) multimodal feature extractor architecture, d) domain-adversarial training, and e) inference procedure. IV-A Preprocessing and Transcript Extraction As shown in Fig. 2, each domain comprises NiN_i disjoint variable-length audio segments, which are first concatenated in chronological order to reconstruct a continuous waveform representing the participant’s full speech during the interview. This waveform is then split into KiK_i segments of a chosen duration, standardizing the input to a fixed number of samples (segment duration × sample rate). To address the variability in total interview length across domains, if the last split is shorter than 25% of the target duration, it is discarded; otherwise, it is zero-padded to match the intended segment duration. Each resulting fixed-length audio segment is transcribed using the “whisper-large-v3” model, yielding a corresponding textual representation. This process ensures alignment between the acoustic signal and its transcript at the segment level, facilitating effective multimodal analysis. IV-B Modality-Specific Feature Extractors To identify the optimal audio-textual feature extractor pair, different modality-specific feature extractors were evaluated: MelSpec, HuBERT, Wav2Vec2 for audio, alongside BERT, ItalianBERT, XLMRoBERTa for text. Aside from MelSpec, all transformer models were implemented via Hugging Face, leveraging their last hidden state as frame-level embeddings for each segment. These strategies are detailed in the following. IV-B1 Audio Feature Extractor Regardless of the method, the extracted acoustic features are standardized into a tensor i,k(a)∈ℝL(a)×H(a)x_i,k^(a) ^L^(a)× H^(a) for each segment k of the i-th participant, where L(a)L^(a) denotes the number of frames per segment, and H(a)H^(a) represents the frame-level feature embedding dimension. In particular, the following methods are evaluated: • MelSpec: Log-Mel spectrograms with FFT size 4096, hop length 512, and 128 filter banks, with first- and second-order derivatives concatenated along the feature axis (H(a)=128×3=384)(H^(a)=128× 3=384). • HuBERT: “facebook/hubert-large-ls960-ft” as the large variant of HuBERT [19] (H(a)=1024)(H^(a)=1024). • Wav2Vec2: “facebook/wav2vec2-base-960h” as the base variant of Wav2Vec2 [4] (H(a)=768)(H^(a)=768). IV-B2 Text Feature Extractor The extracted textual features are represented as i,k(t)∈ℝLi(t)×H(t)x_i,k^(t) ^L^(t)_i× H^(t), where H(t)H^(t) denotes the token embedding dimension, but Li(t)L^(t)_i is the participant-specific maximum sequence length, to which shorter sequences are zero-padded, since the number of spoken tokens varies across segments of participant i. These embeddings are generated using one of the following pretrained models: • BERT: “google-bert/bert-base-multilingual-cased” as the multilingual variant of BERT [10] (H(t)=768)(H^(t)=768). • ItalianBERT: “dbmdz/bert-base-italian-xxl-cased” as the Italian-specific variant of BERT [39] (H(t)=768)(H^(t)=768). • XLMRoBERTa: “FacebookAI/xlm-roberta-large” as the multilingual variant of RoBERTa [7] (H(t)=1024)(H^(t)=1024). In short, the selected audio-text feature extractor pair converts the preprocessed inputs into (i,k(a),i,k(t))(x_i,k^(a),x_i,k^(t)) for subsequent steps. Figure 2: Preprocessing strategy: Converting NiN_i variable-length audio segments of the i-th domain into KiK_i fixed-length segments of a predefined duration. IV-C Multimodal Feature Extractor (fθf_θ) This module maps each input pair (i,k(a),i,k(t))(x_i,k^(a),x_i,k^(t)) to a latent feature i,k∈ℝdz_i,k ^d through a structured pipeline of specialized sub-modules. As illustrated in Fig. 3, an independent BiLSTM is first applied to each modality to capture both forward and backward temporal dependencies. This bidirectional approach ensures that the evolving context within both the frame-level acoustic sequences and the token-level textual features is fully captured. Following layer normalization to stabilize training, intra-modal attention highlights distinctive temporal segments within each modality, whereas cross-modal attention captures complementary audio-textual interactions. Together, the core components of fθf_θ yield the latent representations i,kz_i,k, with their architectural configurations detailed in the following. IV-C1 BiLSTM module Each includes a single hidden layer of size 256, concatenating forward and backward sequences and passing them through a normalization layer. Given (i,k(a),i,k(t))(x_i,k^(a),x_i,k^(t)), the output sequences are: i,k(a) _i,k^(a) =LayerNorm(BiLSTM(a)(i,k(a)))∈ℝL(a)×512 =LayerNorm (BiLSTM^(a)(x_i,k^(a)) ) ^L^(a)× 512 (2) i,k(t) _i,k^(t) =LayerNorm(BiLSTM(t)(i,k(t)))∈ℝLi(t)×512 =LayerNorm (BiLSTM^(t)(x_i,k^(t)) ) ^L^(t)_i× 512 Figure 3: Feature extraction strategy: For each domain i, the KiK_i fixed-length audio segments and corresponding transcripts are first fed into their modality-specific feature extractors to obtain primary features. Our multimodal feature extractor further processes these features, capturing temporal dependencies via BiLSTM and encoding interactions via attention modules. The resulting latent features are then used in an adversarial setup to learn domain-invariant representations. IV-C2 Cross-modal attention (CMA) Let α=MHA(Q,K,V)α=MHA(Q,K,V) be the output of a multi-head attention layer with 4 heads and hidden size 256 that captures interactions among modalities, with the textual sequence as query attending to the aligned acoustic context (Q=i,k(t)Q=seq_i,k^(t), and K=V=i,k(a)K=V=seq_i,k^(a)). This output is then passed through a dropout layer (p=0.1p=0.1), added to the query, and normalized. Formally: i,k(c)=LayerNorm(Q+Dropout(α))∈ℝLi(t)×256seq_i,k^(c)=LayerNorm (Q+Dropout(α) ) ^L^(t)_i× 256 (3) IV-C3 Intra-modal attention (IMA) A linear attention network with a single hidden layer of size 128 and Tanh activation, followed by Softmax and a dropout layer (p=0.1p=0.1) will first compute the attention weights for a given sequence s∈ℝL×Hs ^L× H. These weights are then applied to the original sequence s via a dot product, yielding a compact representation v∈ℝHv ^H that captures its most salient information. We apply this module to the output sequences from Eqs. (2) and (3) as: IMA(s) (s) =Dropout(Softmax(Linear(s)))⋅s =Dropout (Softmax(Linear(s)) )· s (4) i,k(α) _i,k^(α) =IMA(i,k(α))∈ℝH,α∈a,t,c =IMA (seq_i,k^(α) ) ^H, α∈\a,t,c\ IV-C4 Joint feature representation The final representation is constructed by concatenating the attended features in Eq. (4), with d=512+512+256=1280d=512+512+256=1280 representing the total hidden dimensionality contributed by each component: i,k=fθ(i,k(a),i,k(t))=[i,k(a);i,k(t);i,k(c)]∈ℝdz_i,k=f_θ (x_i,k^(a),x_i,k^(t) )= [v_i,k^(a);v_i,k^(t);v_i,k^(c) ] ^d (5) IV-D Domain-Adversarial Training Figure 4: Adversarial framework between the depression detector (hϕh_φ) and the domain discriminator (gψg_ψ). A gradient reversal layer (GRL) inverts gradients from gψg_ψ during backpropagation, enabling joint optimization of θ, ϕφ, and ψ within a single minimization objective. After extracting the latent features i,kz_i,k using the multimodal feature extractor (fθf_θ) as defined in Eq. (5), the objective is to ensure that they preserve depression-related information while remaining invariant to inter-speaker variability. To this end, an adversarial framework between a depression detector (hϕh_φ) and a domain discriminator (gψg_ψ) is introduced, which is described in detail in the following. IV-D1 Depression detector (hϕh_φ) The depression detector is implemented as a fully connected network with a single hidden layer of 128 neurons, followed by a ReLU activation function and a single output neuron that estimates a depression logit for each segment k, defined as y^i,k=hϕ(i,k)∈ℝ y_i,k=h_φ(z_i,k) . The discrepancy between this predicted logit and the ground-truth label for domain i is measured using the binary cross-entropy (BCE) loss during training, computed as: BCE(y^i,k,yi)=−[yilogσ(y^i,k)+(1−yi)log(1−σ(y^i,k))] BCE( y_i,k,y_i)=- [y_i σ( y_i,k)+(1-y_i) (1-σ( y_i,k)) ] (6) where σ(z)=11+e−zσ(z)= 11+e^-z denotes the sigmoid activation function. Let ℒdepL_dep be the overall depression detection loss, defined as the average loss across all segments k and source domains m: ℒdep(θ,ϕ)=1m∑i=1m(1Ki∑k=1KiBCE(y^i,k,yi))L_dep(θ,φ)= 1m _i=1^m ( 1K_i _k=1^K_iBCE( y_i,k,y_i) ) (7) In this setting, the optimization objective is to learn the model parameters θ and ϕφ that minimize ℒdepL_dep, enabling accurate prediction of depression for individuals: (θ^,ϕ^)←argminθ,ϕℒdep(θ,ϕ)( θ, φ)← *argmin_θ,φL_dep(θ,φ) (8) IV-D2 Domain discriminator (gψg_ψ) This module is designed as another fully connected network with a single hidden layer of size 128, followed by a ReLU activation function and an output layer with m neurons. Taking as input the same latent features i,kz_i,k for each domain i, it produces d^i,k=gψ(i,k)∈ℝm d_i,k=g_ψ(z_i,k) ^m as an output vector over the m possible source domains, such that d^i,k[j] d_i,k[j] denotes the logit corresponding to the j-th domain. During training, the prediction error of gψg_ψ is quantified using the cross-entropy (CE) loss as: CE(d^i,k,di)=−log(exp(d^i,k[di])∑j=1mexp(d^i,k[j]))CE( d_i,k,d_i)=- ( ( d_i,k[d_i]) _j=1^m ( d_i,k[j]) ) (9) where exp(z)=ez (z)=e^z. The overall domain discrimination loss ℒdomL_dom is therefore obtained by averaging over all segments k and source domains m: ℒdom(θ,ψ)=1m∑i=1m(1Ki∑k=1KiCE(d^i,k,di))L_dom(θ,ψ)= 1m _i=1^m ( 1K_i _k=1^K_iCE( d_i,k,d_i) ) (10) Recall that to enable DG, gψg_ψ is expected to perform poorly at domain discrimination, indicating that fθf_θ has successfully removed domain-specific information from i,kz_i,k. In particular, gψg_ψ minimizes ℒdomL_dom to better distinguish source domains, whereas fθf_θ seeks to maximize the same loss, preventing gψg_ψ from achieving its goal. This adversarial update encourages domain-invariant features to emerge, yielding the following minimax optimization objective: (θ^,ψ^)←argmaxθargminψℒdom(θ,ψ)( θ, ψ)← *argmax_θ *argmin_ψL_dom(θ,ψ) (11) To solve this non-trivial optimization via backpropagation, a GRL [16] is integrated into gψg_ψ, as shown in Fig. 4. The domain logits are obtained as d^i,k=gψ(GRL(i,k,λ))∈ℝm d_i,k=g_ψ(GRL(z_i,k,λ)) ^m, where λ>0λ>0 is a constant controlling the strength of domain confusion. During the forward pass, GRL acts as the identity function, while in backpropagation it multiplies gradients by −λ-λ, effectively reversing their direction to −λ∂ℒdom∂θ-λ _dom∂θ. This transformation enables reformulating Eqs. (8) and (11) into a joint minimization objective: minθ,ϕ,ψℒdep(θ,ϕ)+ℒdom(θ,ψ) _θ,φ,ψ\,L_dep(θ,φ)+L_dom(θ,ψ) (12) The full training pseudocode is presented in Algorithm 1, detailing the joint optimization process that ultimately yields the parameters θ and ϕφ used for inference. IV-E Inference To predict the final depression label for an individual target domain, the decision relies solely on the domain-invariant representations learned by fθf_θ and hϕh_φ. Consequently, gψg_ψ can be disabled at inference, as its role is limited to guiding feature learning during adversarial training. Given a target domain i, all its KiK_i depression logits y^i,k y_i,k obtained by the depression detector are averaged to obtain a single score: yi¯=1Ki∑k=1Kiy^i,k=1Ki∑k=1Kihϕ(fθ(i,k(a),i,k(t))⏞i,k)∈ℝ y_i= 1K_i _k=1^K_i y_i,k= 1K_i _k=1^K_ih_φ ( f_θ(x_i,k^(a),x_i,k^(t))^z_i,k ) (13) and the final predicted depression label is then obtained as: yi~=0(non-depressed)if σ(yi¯)≤121(depressed)otherwise y_i= cases0~(non-depressed)&if σ ( y_i )≤ 12\\ 1~(depressed)&otherwise cases (14) V Experimental Setup In this section, we describe the experimental framework used to evaluate the proposed multimodal depression detection approach. Specifically, we detail: a) the dataset, including its structure, distribution, and relevance; b) the standardized cross-validation protocol ensuring reproducibility; and c) the training setup, with selected hyperparameter configurations used in the experiments. V-A Dataset Androids-Corpus [44] is a relatively recent and public Italian dataset, especially noteworthy given the scarcity of depression detection resources in the Italian language. It comprises 228 audio recordings sampled at 16 kHz contributed by 118 native Italian speakers, divided into two groups: 112 recordings of reading speech reciting the same text with a total duration of roughly 1 hour and 34 minutes, and 116 recordings of interview speech answering a fixed set of open-ended questions spanning around a total of 7 hours and 24 minutes. Motivated by its spontaneous nature, our study adopts only the interview data, which provides a more realistic setting for speech-based depression detection. Among the corresponding 116 participants, 64 were clinically diagnosed with depression, while 52 served as non-depressed or control participants, with both groups demographically balanced in age and education. The dataset also provides a segmentation of the audio files to ease the process of eliminating the interviewer’s speech. As shown in Fig. 5, the duration of the concatenated recordings for each individual ranges approximately from 40 seconds to just under 600 seconds. Figure 5: Waveform length distribution of participants in the Androids-Corpus after concatenating their respective interview recordings. TABLE I: Overview of the hyperparameter grid explored in the experimental setup. The configuration adopted for the reported experiments is highlighted in bold. Hyperparameter Candidate Values MelSpec FFT window size 2048, 4096 MelSpec hop length 256, 512, 1024 MelSpec number of Mel filter banks 128, 192 Audio BiLSTM hidden dimension 128, 256, 512 Text BiLSTM hidden dimension 128, 256, 512 Intra-modal attention hidden size 128, 256 Cross-modal attention hidden size 128, 256, 512 Cross-modal attention number of heads 4 Fully-connected layer hidden size 64, 128, 192 Dropout rate 0.1, 0.2 GRL scaling coefficient (λ) 0.3, 0.5, 0.7, 1 Learning rate (η) 1, 2, 5×−×10^-5 AdamW optimizer weight decay 1, 2, 5×−×10^-5 ReduceLROnPlateau scheduler decay factor (γ) 0.1, 0.2, 0.5 ReduceLROnPlateau scheduler patience epochs (PschdP_schd) 1, 2, 3, 5 Early stopping epsilon (ε ) 0.1, 0.2 Early stopping patience epochs (PstopP_stop) 2, 3, 5 Maximum training epochs (N) 100 Algorithm 1 MultimodalDG Training 0: Learning rate η, GRL coefficient λ, max epochs N 0: m source domains i()i=1m\D_i^(S)\_i=1^m, each comprising KiK_i fixed-size segments (i,k(a),i,k(t),di,yi)k=1Ki\(x_i,k^(a),x_i,k^(t),d_i,y_i)\_k=1^K_i 1: Split i()i=1m\D_i^(S)\_i=1^m into training set S and validation set V 2: Initialize randomly the model parameters θ, ϕφ, ψ 3: for epoch =1=1 to N do 4: ℒtrain←0L_train← 0, ℒval←0L_val← 0 5: 6: # Training Phase 7: for each domain i in S do 8: for segment k=1k=1 to KiK_i do 9: Extract the latent features i,k←fθ(i,k(a),i,k(t))z_i,k← f_θ(x_i,k^(a),x_i,k^(t)) 10: Predict depression logits y^i,k←hϕ(i,k) y_i,k← h_φ(z_i,k) 11: Predict domain logits d^i,k←gψ(GRL(i,k,λ)) d_i,k← g_ψ(GRL(z_i,k,λ)) 12: ℓi,k←BCE(y^i,k,yi)+CE(d^i,k,di) _i,k ( y_i,k,y_i)+CE( d_i,k,d_i) 13: end for 14: Average the loss of segments ℒi←1Ki∑k=1Kiℓi,kL_i← 1K_i _k=1^K_i _i,k 15: Update θ,ϕ,ψθ,φ,ψ to minimize ℒiL_i using AdamW(η)AdamW(η) 16: ℒtrain←ℒtrain+ℒiL_train _train+L_i 17: end for 18: 19: # Validation Phase 20: for each domain i in V do 21: for segment k=1k=1 to KiK_i do 22: Extract the latent features i,k←fθ(i,k(a),i,k(t))z_i,k← f_θ(x_i,k^(a),x_i,k^(t)) 23: Predict depression logits y^i,k←hϕ(i,k) y_i,k← h_φ(z_i,k) 24: ℓi,k←BCE(y^i,k,yi) _i,k ( y_i,k,y_i) 25: end for 26: ℒval←ℒval+1Ki∑k=1Kiℓi,kL_val _val+ 1K_i _k=1^K_i _i,k 27: end for 28: 29: if EarlyStopping(ℒtrain,ℒval)EarlyStopping(L_train,L_val) is triggered then 30: break 31: end if 32: Update η using ReduceLROnPlateau(ℒval)ReduceLROnPlateau(L_val) 33: end for 34: return θ,ϕθ,φ V-B k-fold Cross-Validation Replicating the data splits introduced by the authors of the dataset, the experiments in this study adopted a k-fold (k=5k=5) cross-validation protocol. These splits keep all the participant’s recordings in the same fold to ensure no overlap between train and test sets, which prevents memorizing speaker-specific traits. At each iteration k=1,…,5k=1,…,5, the k-th fold was treated as the test set, while the remaining data were further divided using a stratified split to maintain the distribution of labels across splits, with 80% used for training and 20% reserved for validation. As the dataset was almost balanced in terms of depressed vs non-depressed classes, standard evaluation metrics including accuracy, precision, recall, and F1-score were used for performance evaluation. To ensure robustness and account for potential variability, the experimental results reported in Section VI-A are presented as the mean and standard deviation over three independent runs of each experiment. In contrast, the results in Sections VI-B and VI-D are reported as the mean and standard deviation across folds from a single run, in line with prior studies using the Androids-Corpus to ensure fair comparison and reliable ablation analysis. V-C Training Specifications Model training was performed using the AdamW optimizer, with learning rate (η) and weight decay each set to 1×10−51×10^-5. Using PyTorch’s ReduceLROnPlateauReduceLROnPlateau scheduler with decay factor γ=0.5γ=0.5 and patience Pschd=1P_schd=1 epochs, η was adjusted dynamically in response to plateaus in the validation loss. As outlined in Algorithm 1, training and validation losses were monitored using a custom EarlyStopping(ℒtrain,ℒval)EarlyStopping(L_train,L_val) mechanism, which terminated training if: (i) changes in ℒtrainL_train remained below ε=0.1 =0.1 for Pstop=3P_stop=3 consecutive epochs, indicating convergence, or (i) either ℒtrainL_train or ℒvalL_val maintained an increasing trend over the same number of epochs. Although the maximum number of epochs was set a priori to N=100N=100, training typically halted after no more than 35 epochs in most experiments, effectively preventing overfitting due to the rising validation loss. Different hyperparameters in this study were chosen via empirical tuning over the search space summarized in Table I, with the final optimal values highlighted in bold for reference. The proposed approach was developed using Python 3.12.8, built on the PyTorch framework, and executed on a server with two NVIDIA TITAN V GPUs (each with 12 GB of VRAM, running CUDA version 12.8). TABLE I: Performance comparison of audio-text feature extractor pairings across different segment durations. Each cell reports Accuracy / Precision / Recall / F1-Score, presented in the same order as percentages (%). Audio Feature Extractor Text Feature Extractor Segment Duration 20s 30s 45s 60s MelSpec BERT 77.2 / 76.2 / 88.1 / 80.5 85.2 / 85.9 / 89.3 / 86.8 84.3 / 84.9 / 87.3 / 85.4 85.8 / 87.0 / 88.9 / 87.1 ItalianBERT 86.4 / 86.8 / 89.5 / 87.3 90.4 / 90.8 / 92.0 / 90.8 88.1 / 87.1 / 90.9 / 88.7 87.5 / 90.4 / 87.8 / 88.0 XLMRoBERTa 68.5 / 66.1 / 92.7 / 76.1 67.9 / 69.1 / 85.4 / 74.2 70.8 / 69.8 / 85.6 / 75.0 75.5 / 75.0 / 86.2 / 78.9 HuBERT BERT 76.2 / 74.1 / 86.4 / 78.8 77.9 / 76.9 / 88.0 / 80.9 77.3 / 77.5 / 87.4 / 80.4 79.9 / 81.1 / 85.4 / 82.0 ItalianBERT 83.1 / 80.4 / 91.7 / 85.0 82.5 / 80.6 / 91.1 / 84.7 85.4 / 85.7 / 89.5 / 86.6 80.2 / 80.8 / 87.1 / 82.5 XLMRoBERTa 70.4 / 71.4 / 81.6 / 72.8 71.3 / 72.4 / 87.7 / 76.0 70.8 / 71.7 / 83.2 / 74.7 76.4 / 75.5 / 86.5 / 78.6 Wav2Vec2 BERT 78.8 / 77.6 / 86.6 / 80.9 81.0 / 79.6 / 90.4 / 83.8 79.7 / 84.5 / 82.8 / 80.4 80.8 / 83.6 / 83.0 / 82.1 ItalianBERT 86.8 / 85.4 / 93.5 / 88.3 86.2 / 85.9 / 92.2 / 88.1 83.1 / 85.5 / 87.7 / 85.1 84.0 / 85.1 / 87.3 / 85.5 XLMRoBERTa 73.3 / 73.3 / 79.6 / 75.0 73.9 / 75.0 / 82.7 / 76.6 71.0 / 71.5 / 81.8 / 74.3 81.6 / 81.1 / 86.1 / 82.7 VI Results This section presents the experimental results of our study, specifically detailing: a) an analysis of feature extractors across segment durations, b) the impact of domain generalization, c) a comparison with state-of-the-art methods, and d) ablation studies on modalities and architectural variants. VI-A Feature Extractors vs. Segment Durations In order to find a competitive baseline, different selections of modality-specific feature extractor pairs and segment durations for fixed-length audio segments were evaluated in a structured set of experiments. In particular, every possible pairing of three audio extractors (MelSpec, HuBERT, and Wav2Vec2) with three text extractors (BERT, ItalianBERT, and XLMRoBERTa) across segment durations of 20, 30, 45, and 60 seconds were examined for this purpose. As summarized in Table I, the pairing of MelSpec feature extractor for audio and ItalianBERT for the text modality at a 30-second segment duration achieved the highest performance. With an average accuracy and F1-score of 90.4% and 90.8%, respectively, this combination was selected as the baseline for downstream analysis. To assess the impact of segment duration, Fig. 6 presents average performance across all feature extractor combinations for each duration, derived from Table I. Overall, adopting longer speech segments improves accuracy, though at the cost of higher GPU memory usage for processing larger sequences. Given this trade-off, a 30-second segment duration provides a good balance between performance and computational cost. Alternatively, Fig. 7 illustrates the average results of each audio-textual feature extractor. The combination of MelSpec (audio) and ItalianBERT (text) consistently yields the highest performance, likely due to MelSpec’s compact representations and the Italian-specific design of ItalianBERT. Figure 6: Average performance across all experiments for each segment duration, computed across all the audio-text pairings for each duration in Table I. Figure 7: Average performance of modality-specific feature extractor pairings, computed across all segment durations for each pairing from Table I. VI-B Domain Generalization Effect The previously identified baseline model was extended by integrating the domain discriminator gψg_ψ to assess the proposed DG framework. This inclusion led to an improvement of 2.5% in accuracy and 3.3% in F1-score, yielding a final performance of 93.2% accuracy, 93.2% precision, 96.2% recall, and 94.2% F1-score, highlighting the contribution of adversarial training in improving generalization to unseen target domains. Analyzing these performance gains, McNemar’s test with continuity correction was applied to individual predictions. Given roughly 23 testing samples per fold in the standardized experimental protocol, achieving p<0.05p<0.05 requires substantial disagreement between models. Hence, the null hypothesis was not rejected, reflecting that insufficient sample size constrains statistical significance, even with strong observed results. To provide an alternative perspective on model performance, we report the cumulative confusion matrix in Fig. 8, obtained by summing the per-fold confusion matrices across the 5-fold cross-validation. Unlike the averaged metrics reported earlier, this summation conceptually reflects the model’s behavior over the entire dataset. The proposed model correctly identifies 95.3% and 90.4% of depressed and non-depressed individuals, respectively. In addition, it achieves a low false negative rate (FNR=4.7FNR=4.7%), which is crucial in clinical screening to avoid missing individuals at risk while maintaining strong specificity for the non-depressed group. VI-C Comparison with Androids-Corpus Benchmarks Table IV benchmarks the proposed approach against prior studies on the Androids-Corpus dataset (see Section I-A for reference). The proposed model outperforms all these efforts across every evaluation metric, attaining a peak F1-score of 94.2%, which represents a clear improvement over the 93.1% of the closest competitor [2]. This comparison validates the strength and reliability of the proposed multimodal framework, and the effectiveness of the domain generalization strategy in capturing depression-specific feature embeddings. TABLE IV: Performance comparison with prior studies on the Androids-Corpus for depression detection, reported as percentages (%). Study Model Features Accuracy Precision Recall F1-score [44] Rand. Random Classifier MFCC, RMSE, ZCR, etc. (OpenSMILE) 50.5 55.2 55.2 55.2 [44] BS1 SVM MFCC, RMSE, ZCR, etc. (OpenSMILE) 73.3±\,±\,10.6 73.5±\,±\,16.1 74.5±\,±\,13.2 73.6±\,±\,13.6 [44] BS2 LSTM MFCC, RMSE, ZCR, etc. (OpenSMILE) 83.9±\,±\,1.3 85.8±\,±\,3.1 86.1±\,±\,2.7 84.7±\,±\,0.9 [31] FuSeR PTM x-vector, emoHuBERT, TRILLsson 87.9 – – 87.8 [30] UBM-HMM HMM and MAP MFCC, TEO, periodicity 87.0±\,±\,1.2 85.8±\,±\,1.9 92.3±\,±\,1.8 88.9±\,±\,1.7 [54] PDEM KELM PDEM embeddings (based on Wav2Vec2) 88.2±\,±\,8.5 89.6±\,±\,6.5 84.4±\,±\,17.9 86.3±\,±\,11.5 [9] HuBERT L Logistic Regression HuBERT-L embeddings – – – 92.0 [50] MDFA MDFA +GRL ResNet-18 – – – 79.2 [49] MTFS MTFS-Block ECAPA-TDNN – 79.4 78.8 77.4 [3] Multimodal GRU Wav2Vec2 / BERT 86.2±\,±\,7.8 – – – [20] Concatenation Cross-Attention AlexNet (3D log-Mel) / ItalianBERT 91.5±\,±\,6.1 91.5±\,±\,8.7 93.4±\,±\,6.0 92.1±\,±\,5.5 [2] MIL Majority-voting OpenSMILE / LIWC 92.5±\,±\,1.1 – – 93.1±\,±\,1.1 [56] Full model D-CoPE + QF +GRL Wav2Vec2 / XLM-RoBERTa – – – 73.0 This work MultimodalDG MelSpec / ItalianBERT 93.2±\,±\,5.7 93.2±\,±\,9.7 96.2±\,±\,4.7 94.2±\,±\,4.7 TABLE V: Ablation study highlighting the contribution of each modality to model performance. Modality Accuracy Precision Recall F1-Score Multimodal 93.2±\,±\,5.7 93.2±\,±\,9.7 96.2±\,±\,4.7 94.2±\,±\,4.7 Audio-only 73.4±\,±\,9.5 74.9±\,±\,10.1 79.0±\,±\,10.7 76.2±\,±\,8.3 Text-only 85.3±\,±\,8.1 87.7±\,±\,9.7 87.7±\,±\,9.4 87.0±\,±\,6.0 TABLE VI: Ablation study analyzing the impact of alternative design choices on model performance. Abbreviations include: domain generalization (DG), intra-modal attention (IMA), cross-modal attention (CMA), layer normalization (LN). Architecture Accuracy Precision Recall F1-Score Full Model 93.2±\,±\,5.7 93.2±\,±\,9.7 96.2±\,±\,4.7 94.2±\,±\,4.7 No DG 90.7±\,±\,8.9 90.4±\,±\,10.4 91.9±\,±\,7.9 90.9±\,±\,8.5 No IMA 87.3±\,±\,10.4 85.9±\,±\,12.9 92.6±\,±\,6.5 88.7±\,±\,8.9 No CMA 89.8±\,±\,6.8 89.3±\,±\,10.2 92.5±\,±\,6.9 90.4±\,±\,6.4 No LN 88.9±\,±\,7.7 89.5±\,±\,12.1 92.6±\,±\,6.9 90.2±\,±\,6.3 VI-D Ablation Studies A set of ablation experiments was carried out to investigate the contribution of each component in the proposed strategy. First, the role of each modality was evaluated, as reported in Table V. The highest performance was achieved with the full multimodal setup, demonstrating the complementary value of integrating both acoustic and linguistic modalities for accurate depression detection. The large performance drop in unimodal experiments (e.g., decreasing to 73.4% accuracy for audio-only) is directly tied to the forced architectural adaptations within fθf_θ. When restricted to a single modality, the network excludes not only the intra-modally attended BiLSTM stream of the omitted modality (see Fig. 3), but also inherently disables the CMA module and its subsequent IMA block, as there is no complementary modality to drive the attention computation. Together, these alterations justify the substantial performance gap when compared to the full multimodal framework. Next, a series of architectural modifications were applied to assess the contribution of each component, as summarized in Table VI. The results show that disabling any of these modules inevitably degrades performance, thereby confirming the critical role each plays and underscoring the importance of attention mechanisms, normalization layers, and particularly domain generalization in achieving optimal results. Notably, the contribution of IMA to the overall performance outweighs that of CMA. IMA highlights salient audio-textual segments to assist subsequent layers in filtering uninformative temporal noise from the BiLSTM and CMA outputs, playing a critical role in achieving optimal performance. In contrast, the architecture is less sensitive to the removal of the cross-modal sequence i,k(c)seq_i,k^(c) (see Eqs. (3)–(5)) from the final latent feature i,kz_i,k. This suggests that the subsequent feed-forward network hϕh_φ can partially compensate for the absence of CMA, likely by relying on the remaining unimodal branches. VII Limitations and Future Work Despite its methodological contributions and strong results, this study leaves several promising directions for future work. First, due to computational constraints, the modality-specific feature extraction was performed as a stand-alone stage prior to model training. Consequently, the framework is limited by the absence of a fine-tuning stage for the transformer-based encoders, which may reduce performance compared to fully end-to-end approaches. Second, the experiments are limited by the use of a single dataset, due to the scarcity of high-quality, publish-ready datasets in Italian. Future work should explore cross-dataset analysis to confirm the framework’s robustness and generalizability, as more datasets become available. Lastly, interpreting internal embeddings under DG settings remains an open challenge, motivating the integration of explainable AI to strengthen clinical trust and enhance the robustness of mental health monitoring systems. VIII Conclusion This study is the first effort to adopt domain generalization for learning domain-invariant features robust to inter-speaker variability in multimodal depression detection. It implements a novel architecture that combines BiLSTM sequence modeling, attention mechanisms, and segment-level decision-making. To identify the most effective audio-text feature extractor pairing, a series of experiments were conducted across various segment durations, ultimately finding the combination of MelSpec and ItalianBERT with a 30-second duration to be the most effective configuration. Next, this baseline model was integrated with domain-adversarial training, achieving notable improvements in classification performance. The obtained results surpassed all previous benchmarks on the Androids-Corpus, despite the reduced training set size due to validation splitting. Lastly, comprehensive ablation studies confirmed the importance of integrating both modalities and the critical role of each model component. Overall, these findings underscore the applicability of DG as a powerful tool to boost real-world generalizability, enabling patient-independent diagnosis. Figure 8: Cumulative confusion matrix of the proposed model, compiled by summing each fold’s confusion matrix across the 5-fold cross-validation setup. References [1] T. Al Hanai, M. M. Ghassemi, and J. R. Glass (2018) Detecting depression with audio/text sequence modeling of interviews.. In Interspeech, p. 1716–1720. Cited by: §I-B. [2] R. Alsarrani, A. Esposito, and A. Vinciarelli (2025-10) Punctual or Continuous? Analyzing Depression Traces in Language and Paralanguage with Multiple Instance Learning. In Proceedings of the 27th International Conference on Multimodal Interaction, Canberra Australia, p. 614–623. External Links: Document, ISBN 979-8-4007-1499-3 Cited by: §I-A, TABLE I, §VI-C, TABLE IV. [3] B. Alsenani, A. Esposito, A. Vinciarelli, and T. Guha (2024) Assessing privacy risks of attribute inference attacks against speech-based depression detection system. ECAI. Cited by: §I-A, TABLE I, TABLE IV. [4] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli (2020) Wav2vec 2.0: a framework for self-supervised learning of speech representations. Advances in neural information processing systems 33, p. 12449–12460. Cited by: 3rd item. [5] V. Baldini, M. Gnazzo, M. Maragno, R. Biagetti, C. Stefanini, F. Canulli, G. Varallo, C. Donati, G. Neri, A. Fiorillo, et al. (2025) Suicidal risk among adolescent psychiatric inpatients: the role of insomnia, depression, and social-personal factors. European Psychiatry 68 (1), p. e42. Cited by: §I. [6] Z. Chen, D. Wang, L. Lou, S. Zhang, X. Zhao, S. Jiang, J. Yu, and J. Xiao (2025) Text-guided multimodal depression detection via cross-modal feature reconstruction and decomposition. Information Fusion 117, p. 102861. Cited by: §I-B. [7] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2019) Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. Cited by: 3rd item. [8] K. Daly and O. Olukoya (2025) Depression detection in read and spontaneous speech: A Multimodal approach for lesser-resourced languages. Biomedical Signal Processing and Control 108, p. 107959. Cited by: §I-A, TABLE I. [9] M. de Gennes, A. Lesage, M. Denais, X. Cao, S. Chang, P. Van Remoortere, C. Dakhlia, and R. Riad (2024) Probing mental health information in speech foundation models. arXiv preprint arXiv:2409.19042. Cited by: §I-A, TABLE I, TABLE IV. [10] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: 1st item. [11] Y. Di, E. Rahmani, J. Mefford, J. Wang, V. Ravi, A. Gorla, A. Alwan, K. S. Kendler, T. Zhu, and J. Flint (2025) Unraveling the associations between voice pitch and major depressive disorder: a multisite genetic study. Molecular Psychiatry 30 (6), p. 2686–2695. Cited by: §I. [12] H. Ding, Z. Du, Z. Wang, J. Xue, Z. Wei, K. Yang, S. Jin, Z. Zhang, and J. Wang (2024) IntervoxNet: a novel dual-modal audio-text fusion network for automatic and efficient depression detection from interviews. Frontiers in Physics 12, p. 1430035. Cited by: §I-B, §I-B. [13] H. Dong, E. Chatzi, and O. Fink (2024) Towards multimodal open-set domain generalization and adaptation through self-supervision. In European Conference on Computer Vision, p. 270–287. Cited by: §I-C. [14] H. Dong, I. Nejjar, H. Sun, E. Chatzi, and O. Fink (2023) SimMMDG: a simple and effective framework for multi-modal domain generalization. Advances in Neural Information Processing Systems 36, p. 78674–78695. Cited by: §I-C. [15] A. Farahani, S. Voghoei, K. Rasheed, and H. R. Arabnia (2021) A brief review of domain adaptation. Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, p. 877–894. Cited by: §I. [16] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky (2016) Domain-adversarial training of neural networks. Journal of machine learning research 17 (59), p. 1–35. Cited by: §I, §IV-D2. [17] L. Gu, M. Li, and Y. Li (2025) Linguistic markers of depression and emergent self-stigma in online self-disclosures: a mixed-methods study on chinese social media. Journal of Affective Disorders, p. 120765. Cited by: §I. [18] A. Haque, M. Guo, A. S. Miner, and L. Fei-Fei (2018) Measuring depression symptom severity from spoken language and 3d facial expressions. arXiv preprint arXiv:1811.08592. Cited by: §I-B. [19] W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed (2021) Hubert: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing 29, p. 3451–3460. Cited by: 2nd item. [20] L. Ilias and D. Askounis (2024) A cross-attention layer coupled with multimodal fusion methods for recognizing depression from spontaneous speech. In Proc. Interspeech 2024, p. 912–916. Cited by: §I-A, §I-B, TABLE I, TABLE IV. [21] L. Ilias and D. Askounis (2025) Mixture of experts for recognizing depression from interview and reading tasks. arXiv preprint arXiv:2502.20213. Cited by: §I-A, §I-A, TABLE I. [22] S. Jabeen, X. Li, M. S. Amin, O. Bourahla, S. Li, and A. Jabbar (2023) A review on methods and applications in multimodal deep learning. ACM Transactions on Multimedia Computing, Communications and Applications 19 (2s), p. 1–41. Cited by: §I. [23] M. Jahanifar, M. Raza, K. Xu, T. T. L. Vuong, R. Jewsbury, A. Shephard, N. Zamanitajeddin, J. T. Kwak, S. E. A. Raza, F. Minhas, and N. Rajpoot (2025-06) Domain Generalization in Computational Pathology: Survey and Guidelines. ACM Comput. Surv. 57 (11), p. 285:1–285:37. External Links: ISSN 0360-0300, Document Cited by: §I. [24] X. Jia, X. Zhao, B. Tang, and R. Jiang (2024) Bidirectional multimodal block-recurrent transformers for depression detection. In 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), p. 3323–3328. Cited by: §I-B, §I-B. [25] L. S. Khoo, M. K. Lim, C. Y. Chong, and R. McNaney (2024) Machine learning for multimodal mental health detection: a systematic review of passive sensing approaches. Sensors 24 (2), p. 348. Cited by: §I. [26] S. Koops, S. G. Brederoo, J. N. de Boer, F. G. Nadema, A. E. Voppel, and I. E. Sommer (2023) Speech as a biomarker for depression. CNS & Neurological Disorders-Drug Targets-CNS & Neurological Disorders) 22 (2), p. 152–160. Cited by: §I. [27] D. Li, J. Yao, Z. Wang, and Y. Yi (2025) FAD3QN: A Brain-Inspired Deep Reinforcement Learning Model for Speech Depression Detection. IEEE Transactions on Computational Social Systems. Cited by: §I-A, TABLE I. [28] M. Li, Y. Wei, Y. Zhu, S. Wei, and B. Wu (2024) Enhancing multimodal depression detection with intra-and inter-sample contrastive learning. Information Sciences 684, p. 121282. Cited by: §I, §I-B, §I-B. [29] S. Liu, L. An, and Z. Jia (2025) A domain adversarial learning framework for major depression disorder diagnosis. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §I-C. [30] S. Ntalampiras (2025) Interpretable probabilistic identification of depression in speech. Sensors 25 (4), p. 1270. Cited by: §I-A, §I-A, TABLE I, TABLE IV. [31] O. C. Phukan, S. R. Behera, S. Singh, M. Singh, V. Rajan, A. B. Buduru, R. Sharma, and S. Prasanna (2024) Avengers assemble: amalgamation of non-semantic features for depression detection. arXiv preprint arXiv:2409.14312. Cited by: §I-A, TABLE I, TABLE IV. [32] M. Planamente, C. Plizzari, E. Alberti, and B. Caputo (2022) Domain generalization through audio-visual relative norm alignment in first person action recognition. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 1807–1818. Cited by: §I-C. [33] R. Polle, S. Fara, A. Georgescu, S. Goria, and N. Cummins (2024) Revealing confounding biases: a novel benchmarking approach for aggregate-level performance metrics in health assessments. Accepted for Interspeech 2024, Kos Island, Greece. Cited by: §I-A, TABLE I. [34] L. Qu, C. Weber, W. Wang, J. Jin, Y. Gao, T. Li, and S. Wermter (2025-08) Disentanglement of Prosody Representations via Diffusion Models and Scheduled Gradient Reversal. IEEE Transactions on Neural Networks and Learning Systems 36 (8), p. 15043–15054. External Links: ISSN 2162-2388, Document Cited by: §I. [35] J. Quiñonero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence (2022) Dataset shift in machine learning. Mit Press. Cited by: §I. [36] K. Rezaee (2026) Depression detection from speech data using deep learning–based optimized temporal–frequency–channel attention with interpretable acoustic–prosodic mapping. Journal of Affective Disorders, p. 121077. Cited by: §I-A, §I-A, TABLE I. [37] M. Rodrigues Makiuchi, T. Warnita, K. Uto, and K. Shinoda (2019) Multimodal fusion of bert-cnn and gated cnn representations for depression detection. In Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, p. 55–63. Cited by: §I-B. [38] C. Rohlfs (2025-01) Generalization in neural networks: A broad survey. Neurocomputing 611, p. 128701. External Links: ISSN 0925-2312, Document Cited by: §I. [39] S. Schweter (2020) Italian bert and electra models. Note: Available at https://doi.org/10.5281/zenodo.4263142Version 1.0.1, Zenodo External Links: Document Cited by: 2nd item. [40] S. Shankar, V. Piratla, S. Chakrabarti, S. Chaudhuri, P. Jyothi, and S. Sarawagi (2018) Generalizing across domains via cross-gradient training. arXiv preprint arXiv:1804.10745. Cited by: §I. [41] Y. Shen, H. Yang, and L. Lin (2022) Automatic depression detection: an emotional audio-textual corpus and a gru/bilstm-based model. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 6247–6251. Cited by: §I-B. [42] Z. Shen, J. Liu, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui (2021-08) Towards Out-Of-Distribution Generalization: A Survey. ArXiv. Cited by: §I. [43] S. Shorey, E. D. Ng, and C. H. Wong (2022) Global prevalence of depression and elevated depressive symptoms among adolescents: a systematic review and meta-analysis. British journal of clinical psychology 61 (2), p. 287–305. Cited by: §I. [44] F. Tao, A. Esposito, and A. Vinciarelli (2023) The androids corpus: a new publicly available benchmark for speech based depression detection. Depression 47, p. 11–9. Cited by: §I-A, §I-A, TABLE I, §V-A, TABLE IV, TABLE IV, TABLE IV. [45] F. Tao, X. Ge, W. Ma, A. Esposito, and A. Vinciarelli (2024) Cross-data multilevel attention for depression detection: analyzing the interplay between read and spontaneous speech. In 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), p. 1169–1176. Cited by: §I-A, TABLE I. [46] F. Tao, W. Ma, X. Ge, A. Esposito, and A. Vinciarelli (2023) The relationship between speech features changes when you get depressed: feature correlations for improving speed and performance of depression detection. arXiv preprint arXiv:2307.02892. Cited by: §I-A, TABLE I. [47] E. Toto, M. Tlachac, and E. A. Rundensteiner (2021) Audibert: a deep transfer learning multimodal classification framework for depression screening. In Proceedings of the 30th ACM international conference on information & knowledge management, p. 4145–4154. Cited by: §I-B. [48] J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. S. Yu (2023-08) Generalizing to Unseen Domains: A Survey on Domain Generalization. IEEE Transactions on Knowledge and Data Engineering 35 (8), p. 8052–8072. External Links: ISSN 1558-2191, Document Cited by: §I, §I-C. [49] M. Wang, S. Kato, W. Gu, F. Ren, and J. Yan (2026) Depression detection from speech signals using a multiple temporal-frequency scale Channel Attention Mechanism. Biomedical Signal Processing and Control 113, p. 108873. Cited by: §I-A, TABLE I, TABLE IV. [50] M. Wang, S. Kato, W. Gu, J. Yan, and F. Ren (2025) Cross-Language Depression Detection Based on Multi-Domain Feature Alignment. In 2025 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), p. 1–7. Cited by: §I-A, §I-C, TABLE I, TABLE IV. [51] P. Wu, R. Wang, H. Lin, F. Zhang, J. Tu, and M. Sun (2023) Automatic depression recognition by intelligent speech signal processing: a systematic survey. CAAI Transactions on Intelligence Technology 8 (3), p. 701–711. Cited by: §I. [52] L. Xia, Y. Feng, Z. Guo, J. Ding, Y. Li, Y. Li, M. Ma, G. Gan, Y. Xu, J. Luo, Z. Shi, and Y. Guan (2023-12) MuLHiTA: A Novel Multiclass Classification Framework With Multibranch LSTM and Hierarchical Temporal Attention for Early Detection of Mental Stress. IEEE Transactions on Neural Networks and Learning Systems 34 (12), p. 9657–9670. External Links: ISSN 2162-2388, Document Cited by: §I. [53] L. Yang, D. Jiang, X. Xia, E. Pei, M. C. Oveneke, and H. Sahli (2017) Multimodal measurement of depression using deep learning models. In Proceedings of the 7th annual workshop on audio/visual emotion challenge, p. 53–59. Cited by: §I-B. [54] J. Yu and H. Kaya (2025) Using emotionally rich speech segments for depression prediction. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §I-A, §I-A, TABLE I, TABLE IV. [55] W. Yuan, X. Zhang, X. Zhang, S. Wang, T. Wang, T. Zhang, Q. Zhao, and B. Hu (2025) Discovery of shared latent nonlinear effective connectivity for eeg-based depression detection. IEEE Transactions on Neural Networks and Learning Systems 36 (6), p. 10663–10677. External Links: Document Cited by: §I. [56] E. Zhang and C. Poellabauer (2025) Mitigating Interviewer Bias in Multimodal Depression Detection: An Approach with Adversarial Learning and Contextual Positional Encoding. In Findings of the Association for Computational Linguistics: EMNLP 2025, p. 12169–12188. Cited by: §I-A, §I-C, TABLE I, TABLE IV. [57] S. Zhang, X. Feng, W. Fan, W. Fang, F. Feng, W. Ji, S. Li, L. Wang, S. Zhao, Z. Zhao, et al. (2023) Video-audio domain generalization via confounder disentanglement. In Proceedings of the AAAI conference on artificial intelligence, 37, p. 15322–15330. Cited by: §I-C. [58] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy (2022) Domain generalization: a survey. IEEE transactions on pattern analysis and machine intelligence 45 (4), p. 4396–4415. Cited by: §I, §I, §I-C. [59] L. Zhou, X. Zhang, S. Guan, and X. Luo (2025) Adaptive Knowledge Fusion Model for Depression Recognition. IEEE Transactions on Computational Social Systems. Cited by: §I-A, §I-A, TABLE I. Ali Tabaraei is a Ph.D. candidate in Computer Science at the University of Milan, Italy, where he also received his master’s degree in the same field in 2025. His current doctoral research focuses on the generalizability and interpretability aspects of advanced deep neural networks applied to health acoustics. Particularly, his research interests include Domain Generalization, Domain Adaptation, Audio Pattern Recognition, Explainable AI, and Health AI. Federico Simonetta is a post-doctoral researcher in the Laudare ERC AdG project at the Gran Sasso Science Institute (GSSI). He previously worked as a post-doctoral researcher at the Universidad Complutense de Madrid and Instituto Complutense de Ciencias Musicales (ICCMU) in the Didone ERC AdG project. He obtained his Ph.D. in Computer Science from the University of Milan in 2022. He is active in the scientific committees of several international journals and conferences. His main research interests are music information processing, machine learning, audio processing, and handwritten music/text recognition. Stavros Ntalampiras is an Associate Professor at the Department of Computer Science, University of Milan, Italy. He received the engineering and Ph.D. degrees from the Department of Electrical and Computer Engineering, University of Patras, Greece, in 2006 and 2010, respectively. He has carried out research and/or didactic activities at Politecnico di Milano, the Joint Research Center of the European Commission, the National Research Council of Italy, and Bocconi University. Currently, he is an Associate Editor of IEEE TNNLS, PLOS One, IET Signal Processing and CAAI Transactions on Intelligence Technology, as well as member of the IEEE Computational Intelligent Society Task Force on Computational Audio Processing. His research interests include content-based signal processing, machine learning, audio pattern recognition, bioacoustics, health acoustics, and cyber-physical systems.