Paper deep dive
Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Model
Sicheng Mao, Baihan Li, Mathieu Fontaine, Anthony Larcher, Roland Badeau
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/16/2026, 8:08:43 AM
Summary
This paper proposes a neural multichannel distant speaker diarization method that integrates heavy-tailed source separation models into the neural FCASA framework. By replacing the standard Gaussian variance modeling with Leptokurtic Generalized Gaussian and Student's t distributions, the model better captures the heavy-tailed nature of speech signals in difficult acoustic environments. Experiments on AMI, AliMeeting, and CHiME-6 corpora demonstrate significant improvements in Diarization Error Rate (DER) and Jaccard Error Rate (JER) compared to the Gaussian-based baseline.
Entities (17)
Relation Signals (11)
Proposed Method → evaluatedon → CHiME-6 corpus
confidence 98% · Our experiments show consistent large improvements... on various corpora including CHiME-6
Proposed Method → evaluatedon → AMI Corpus
confidence 98% · Our experiments show consistent large improvements... on various corpora including AMI
Proposed Method → evaluatedon → AliMeeting corpus
confidence 98% · Our experiments show consistent large improvements... on various corpora including AliMeeting
Proposed Method → improvesmetric → Diarization Error Rate
confidence 95% · Our experiments show consistent large improvements in Diarization Error Rate (DER)
Proposed Method → improvesmetric → Jaccard Error Rate
confidence 95% · Our experiments show consistent large improvements in ... Jaccard Error Rate (JER)
Neural FCASA → usesdistribution → Gaussian distribution
confidence 95% · The popular Gaussian distribution has been applied for variance modeling in the original source separation model
Proposed Method → usesdistribution → Leptokurtic Generalized Gaussian distribution
confidence 95% · we replace with two families of heavy-tailed models (Leptokurtic Generalized Gaussian distribution...)
Proposed Method → usesdistribution →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Distant speaker diarization remains challenging due to difficult acoustic environments, varying numbers of speakers and overlapping speech. Model-driven methods are proposed to exploit the speech source features in multi-channel recordings that help diarization. This paper generalizes a neural model that jointly learns to perform blind source separation and diarization over speech mixtures (neural FCASA) with heavy-tailed models. The popular Gaussian distribution has been applied for variance modeling in the original source separation model, which we replace with two families of heavy-tailed models (Leptokurtic Generalized Gaussian distribution and Student's t distribution) to better capture the heavy-tailedness in speech signals. Thanks to the Gaussian scale mixture model, we are able to unify the proposed method and the original one under the same form of learning objective. Our experiments show consistent large improvements in Diarization Error Rate (DER) and Jaccard Error Rate (JER) compared to the baseline on various corpora.
Tags
Links
- Source: https://arxiv.org/abs/2609.12154v1
- Canonical: https://arxiv.org/abs/2609.12154v1
Trouble viewing inline? Open PDF directly →
Full Text
40,626 characters extracted from source content.
Expand or collapse full text
Neural Multichannel Distant Speaker Diarization With Heavy-tailed Source Separation Model Sicheng Mao1 Baihan Li12 Mathieu Fontaine1 Anthony Larcher3 Roland Badeau1 Affiliation: 1 LTCI, Telecom Paris, Institut Polytechnique Paris, France 2 SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China 3 LIUM, Le Mans University, France Abstract Distant speaker diarization remains challenging due to difficult acoustic environments, varying numbers of speakers and overlapping speech. Model-driven methods are proposed to exploit the speech source features in multi-channel recordings that help diarization. This paper generalizes a neural model that jointly learns to perform blind source separation and diarization over speech mixtures (neural FCASA) with heavy-tailed models. The popular Gaussian distribution has been applied for variance modeling in the original source separation model, which we replace with two families of heavy-tailed models (Leptokurtic Generalized Gaussian distribution and Student’s t distribution) to better capture the heavy-tailedness in speech signals. Thanks to the Gaussian scale mixture model, we are able to unify the proposed method and the original one under the same form of learning objective. Our experiments show consistent large improvements in Diarization Error Rate (DER) and Jaccard Error Rate (JER) compared to the baseline on various corpora. Index Terms: heavy-tailed distribution, distant speaker diarization, blind source separation, meeting recording I Introduction Speaker diarization (SD) addresses the question of “who spoke when” by logging speaker-specific salient events in audio data. Distant speaker diarization remains challenging due to difficult acoustic environments such as meeting scenarios [24, 34, 8], where speech signals are often recorded by a distant microphone array and contain more noise and reverberation, with a varying number of speakers and overlapping speech. To enhance the robustness of a distant diarization system, data-driven paradigm performs data augmentation by noise, reverberation and synthesized overlapping speech and trains powerful speaker feature extractors [29] within pipeline systems [18] or end-to-end neural diarization (EEND) systems [11, 12]. This paradigm recently drives the application of Large Language Models (LLMs) [32, 35] that demonstrate the scaling laws on the diarization task. On the other hand, model-driven methods still hold certain interest by incorporating traditional signal processing knowledge, especially for multi-channel distant speech recordings. The direction-of-arrival (DOA), the inter-channel time delay (ITD) and other distant speech specific features captured by multi-channel recordings can be used to characterize the speech sources and hence contribute to multi-channel diarization tasks. [1, 23] incorporate beamforming models and extract the steering vector of a signal to enhance the speech features. The multi-channel EEND [13] adapts the Transformer encoder to extract spatio-temporal information and the neural FCASA [4] incorporates a source separation model that performs joint speaker separation and diarization. Fig. 1: The overview of our model, training and inference Dashed: training pipeline; Dotted: inference pipeline; Solid: both; Red: loss terms. The replaced Gaussian model is faded in gray. While the neural FCASA focuses more on enhancing the source separation performance with diarization information, from a multi-task learning perspective, it is also beneficial to the diarization system with an incorporated separation model, because, intuitively, a well-separated speech mixture would reduce the speaker diarization to a simpler voice activity detection task. Therefore, we hypothesize that a better separation model would benefit the diarization performance more. It is known that heavy-tailed separation models [10] are more robust to a wide range of difficult scenarios (e.g. noisy conditions, dynamic range, reverberation etc.). Hence, in this work, we replace the Gaussian model in neural FCASA by heavy-tailed distributions from two typical families: the Leptokurtic Generalized Gaussian distribution and the Student’s t distribution. We then train models and conduct a thorough evaluation of diarization performance over various corpora including AMI [7], AliMeeting [37] and CHiME-6 [34] across different setups and parameter settings. Our results show that this method outperforms the baseline by a large margin on these corpora. The main contribution of this work is to propose a neural speaker diarization method with a heavy-tailed separation model as an improvement and generalization of the neural FCASA. I Blind source separation and diarization in the neural FCASA This section describes the formulation of the neural FCASA, which we further extend with heavy-tailed separation models. I-A Generative model of multichannel mixture signal Let us denote ft∈ℂMx_ft ^M as an M-channel mixture signal, which is a sum of N image signals nftx_nft generated from a source signal snft∈ℂs_nft , represented in the Short Time Fourier Transform (STFT) domain. The neural FCASA has been proposed to jointly train the separation and diarization by introducing a source activity temporal mask unt∈0,1u_nt∈\0,1\ to the typical blind source separation model [9, 3]: ft=∑n=1Nuntnft=∑n=1Nuntnfsnft, _ft= _n=1^Nu_ntx_nft= _n=1^Nu_nta_nfs_nft, (1) where nf∈ℂMa_nf ^M is the steering vector for source n, and t=1,…,Tt=1,…,T and f=1,…,Ff=1,…,F represent the time and frequency indices, respectively. Following the typical variance modeling [31], each snfts_nft is assumed to independently follow a circularly-symmetric complex Gaussian distribution with its power spectrum density (PSD) λnft∈ℝ+λ_nft _+ as the variance of the Gaussian model: snft∼ℂ(0,λnft). s_nft _C (0,λ_nft ). (2) By marginalizing snfts_nft from Eqs. (1) and (2), the mixture signal model is obtained as: ft∼ℂ(,ft), _ft _C ( 0,Y_ft ), (3) where ft≜∑n=1NuntλnftnfY_ft _n=1^Nu_ntλ_nftH_nf and nf=nfnf∈+M×MH_nf=a_nfa_nf H _+^M× M is called the spatial covariance matrix (SCM) [9, 25] of source n at frequency f that characterizes the signal propagation under a certain geometric configuration. To facilitate the inference, SCMs are further assumed to be jointly diagonalizable (JD) by a common projection matrix f∈ℂM×MQ_f ^M× M across all the sources [28, 15]: nf=f−1diag(nf)f−, _nf=Q_f^-1diag(w_nf)Q_f^- H, (4) where nf∈ℝ+Mw_nf _+^M are the diagonal coefficients for source n. Both ≜ff=1FQ \Q_f\_f=1^F and ≜nfn,f=1N,FW \w_nf\_n,f=1^N,F can be efficiently estimated by the iterative source steering (ISS) algorithm [26, 28]. Finally, the deep spectral modeling is applied to the source PSDs λnftλ_nft. It introduces D-dimensional latent source features nt∈ℝDz_nt ^D to generate the PSD λnftλ_nft with a deep neural network (DNN) gθ,f:ℝD→ℝ+g_θ,f:R^D _+ as follows: λnft=gθ,f(nt), λ_nft=g_θ,f(z_nt), (5) where θ represents the model parameters of gθ,fg_θ,f and the latent features ntz_nt are supposed to be spectral characteristics (e.g., pitches and envelopes) of the n-th source. Such a neural generative model can be trained as the decoder of a variational autoencoder (VAE) [19, 5, 20, 17] for isolated signals by assuming nt∼(,)z_nt ( 0, I ). I-B Inference model Given the observed mixture signals ≜ftf,t=1F,TX \x_ft\_f,t=1^F,T as input, an inference model hϕh_φ is designed to estimate its latent source features ≜ntn,t=1N,TZ \z_nt\_n,t=1^N,T, speaker activity masks ≜untn,t=1N,TU \u_nt\_n,t=1^N,T, projection matrices Q and the diagonal coefficients W. To this end, hϕh_φ is constructed in a hybrid manner, which contains a neural encoder modeling the posterior distribution qϕ(|),qϕ(|)q_φ(Z|X),q_φ(U|X) of Z and U and an ISS diagonalizer [27] computing fQ_f and nfw_nf ,,qϕ(|),qϕ(|)←hϕ(). \Q,W,q_φ(Z|X),q_φ(U|X) \← h_φ(X). (6) where the posterior variational distribution qϕ(|)q_φ(Z|X) is assumed to follow Gaussian distribution parametrized by mean μϕ,ntd∈ℝ _φ,ntd and variance σϕ,ntd2∈ℝ+σ^2_φ,ntd _+ and qϕ(|)q_φ(U|X) to Bernoulli distribution parametrized by the logits ηϕ,nt∈[0,1] _φ,nt∈[0,1]. Once all the parameters are inferred from Eq. (6), the separation is done by estimating isolated image signals nftx_nft using a multichannel Wiener filter [3, 28] nft=untnftft−1ftx_nft=u_ntY_nftY_ft^-1x_ft and the diarization is done by thresholding the logits ηϕ,nt _φ,nt to 0 or 1. I Heavy-tailed neural blind source separation and diarization We propose the heavy-tailed neural FCASA as an extension to the neural FCASA in Eq. (2) to better estimate the PSD of the isolated signals by heavy-tailed models in order to improve the diarization performance. I-A Heavy-tailed modeling for power spectrum density A typical variance modeling in source separation assumes the Gaussianity of the signal’s time-frequency representation. However, heavy-tailed models show better robustness against impulsive noise or uncommon scenario [10]. Moreover, the real meeting recordings have a wider dynamic range of the overlap ratio [34] comparing to full-overlap signals in source separation task, hence such signals have a stronger heavy-tailed property. A heavy-tailed extension can be simply formulated by adding a random positive impulse variable rftr_ft to the Gaussian variance, leading to a (circularly-symmetric) Gaussian Scale Mixture (GSM) model: snft|rft∼ℂ(0,rftλnft) s_nft|r_ft _C (0,r_ftλ_nft ) (7) where we can assume different prior distribution of rftr_ft. By marginalizing rftr_ft, we can get a specific heavy-tailed model for the mixture signal ftx_ft. Particularly in this paper, we are interested in two common heavy-tailed model families, the Leptokurtic Generalized Gaussian distribution and the Student’s t distribution [10] (Fig. 2): ft∼ℂβ(0,ft)(Leptokurtic),ft∼ℂν(0,ft)(Student’s t), aligned x_ft& _C^β (0,Y_ft )&(Leptokurtic),\\ x_ft& _C^ν (0,Y_ft )&(Student's t), aligned (8) where β∈(0,2),ν∈ℝ+β∈(0,2),ν ^+ are shape parameters for Leptokurtic Generalized Gaussian and Student’s t distribution respectively. Note that the GSM reduces to a Gaussian model hence to the vanilla neural FCASA when rftr_ft is deterministic. Fig. 2: Standard univariate heavy-tailed models Left: Student’s t distributions with degrees of freedom ν∈ℝ+ν ^+. Right: Leptokurtic Generalized Gaussian distributions with shape parameters β∈(0,2)β∈(0,2). They tend to Gaussian when ν→+∞ν→+∞ and β→2β→ 2, respectively I-B Training and inference We train the model parameters in Eq. (8) by maximizing its log-likelihood (L) logpθ(,) p_θ(X,U) of the mixture observations X and the speaker label U, which is similar to [3, 4]. We use the amortized variational inference to optimize the evidence lower bound (ELBO) of the L [17, 16]: logpθ(,)≥qϕ(∣,)[logpθ(∣,)]⏟≜ℒ1−KL[qϕ(∣,)∣pθ()]⏟≜ℒ2+logp()⏟≜ℒ3, aligned & p_θ(X,U)≥ E_q_φ(Z X,U)[ p_θ(X U,Z)]_ _1\\ & -KL[q_φ(Z X,U) p_θ(Z)]_ _2 + p(U)_ _3, aligned (9) where KLKL stands for the Kullback-Leiber (KL) divergence. For simplicity, we approximate the variational family qϕ(|,)q_φ(Z|X,U) by the inference model qϕ(|)q_φ(Z|X). For the first term, we use the one-point Monte-Carlo estimation with one sample Z sampled from q(|)q(Z|X) and U directly from the ground-truth label as teacher-forcing. According to [10], the conditional likelihood is: ℒ1=∑f,t=1F,Tℒ~1+T∑f=1Flog|ff|, _1= _f,t=1^F,T L_1+T _f=1^F |Q_fQ_f H|, (10) where ℒ~1 L_1 is (up to an additive constant) −(∑m=1M|x~ftm|2y~:ftm)β2−∑m=1Mlogy~:ftm, - ( _m=1^M | x_ftm|^2 y_:ftm ) β2- _m=1^M y_:ftm, (11) for Leptokurtic Generalized Gaussian distribution and −(ν2+M)log(1+2ν∑m=1M|x~ftm|2y~:ftm)−∑m=1Mlogy~:ftm, - ( ν2+M ) (1+ 2ν _m=1^M | x_ftm|^2 y_:ftm )- _m=1^M y_:ftm, (12) for Student’s t distribution, where x~ftm,y~nftm x_ftm, y_nftm are the m-th entry of ~ft≜fft,~nft≜untgθ,f(nt)nf x_ft _fx_ft, y_nft u_ntg_θ,f(z_nt)w_nf and y~:ftm≜∑n=1Ny~nftm y_:ftm _n=1^N y_nftm. The second term is a KL-divergence between two Gaussians and can be computed in closed form [17]. For the third term, as we don’t assume a prior model p()p(U) for the speaker activity mask U [21, 4], instead, we directly maximize the log-posterior of surrogate model, logqϕ(|) q_φ(U|X), which corresponds to minimize the KL-divergence between the empirical posterior distribution, pdata(|)p_data(U|X) and the surrogate posterior, qϕ(|)q_φ(U|X), which is equivalent to minimizing the binary cross entropy (BCE) between the ground truth label, untu_nt, and the logit ηϕ,nt _φ,nt, i.e. ℒ3≈logqϕ(|)=−∑n,t=1N,TBCE[unt|ηϕ,nt]. _3≈ q_φ(U|X)=- _n,t=1^N,TBCE[u_nt| _φ,nt]. (13) Note that this shares the same insights as the extended objective function in the semi-supervised VAE [16], where we want to train the inference model for U directly from the labelled data as well. Meanwhile, the permutation invariant training (PIT) is applied to solve the speaker permutation ambiguity [21]. Combining them altogether, we deduce the final form of our maximization objective: ℒ=1TF(ℒ1+δℒ2)+γ1TNℒ3, = 1TF (L_1+ _2 )+γ 1TNL_3, (14) where δ and γ are scaling hyperparameters. It turns out that the heavy-tailed neural FCASA can fully inherit the neural FCASA training and inference procedure by just replacing the original objective to Eq. (14), leading to a straightforward extension. IV Experiments To fairly compare with the neural FCASA as our baseline model, we follow the same experimental settings as in [4] 11 1 Codes available at https://github.com/alephpi/neural-fcasa.. IV-A Dataset description We evaluate the proposed method on three public corpora that provide a multi-channel far-field subset: 1. AMI corpus [7]. This dataset contains 100 hours of 16 kHz English meeting recordings with 3 to 5 participants. Audio was captured using an 8-mic circular array with a radius of 10 cm placed on the table. Our experiments adopt the official split: training (80.7h) for training, development (9.7h) for validation and evaluation (9.1h) for evaluation. 2. AliMeeting corpus [37]. This dataset contains 118.75 hours of 16 kHz Mandarin meeting recordings with 2 to 4 participants. Audio was captured using an 8-mic circular array with a radius of 5 cm placed on the table. Our experiments adopt the official split: training (104.75h) for training, evaluation (4h) for validation and test (10h) for evaluation.22 2 Note that the dataset split naming convention of AliMeeting is different from the one of AMI or CHiME-6. Besides, we use the aligned RTTM transcription from [14] as the ground truth labels instead of the original one. 3. CHiME-6 corpus [33] contains 50 hours of 16 kHz English dinner party conversation recordings with 4 participants. Audio was captured using 6 distant Microsoft Kinect devices. Each device has a linear array of 4 sample-synchronized microphones. The participants are moving around different rooms (kitchen, dining room and living room) in the house. Our experiments adopt the official split: training (40.5h) for training, development (4.5h) for validation and evaluation (5h) for evaluation. For a fair comparison with AMI and AliMeeting, we only use the recordings from device 1 and 2 and merge them as 8 channel recordings. IV-B Configurations The network architecture of the heavy-tailed neural FCASA follows that of the vanilla neural FCASA and we refer the reader to [4] for more details. All the signals are dereverberated in advance using weighted prediction error (WPE) [36]. The spectrograms are generated via STFT with a window size of 512 and a hop size of 160. The source number N=6N=6 containing 5 speaker channels and a noise channel. The latent dimension D=10D=10 for the noise channel and D=64D=64 for the speaker channels to avoid channel modeling ambiguity [2]. Speaker activations untu_nt are derived from the oracle diarization labels, while the noise channel is always set to active to prevent confusion with silence. δ,γδ,γ in Eq. (14) are both set to 1.0. During training, the audio recordings are segmented into 20s clips, from which 10s continuous crops are randomly sampled and fed into the training pipeline with a batch size of 128. The model is trained for 200 epochs using the AdamW optimizer with a learning rate of 1e-4 and a weight decay of 1e-5. For evaluation, the audio recordings are segmented into 10s clips and fed into the inference pipeline. The speaker activity predictions ηϕ,nt _φ,nt are smoothed by a median filter of 11 frames for stability and then cast to 0 or 1 using a threshold of 0.5. We use Pyannote [6] to report the diarization error rate (DER) and its three components: miss, false alarm (FA) and speaker confusion (Conf.), as well as the Jaccard Error Rate (JER). 33 3 Note that we do not evaluate the objective source separation metrics since all of these corpora consist of real conversations without isolated ground truth. To evaluate the diarization performance under different standards, we adopt four different evaluation setups. Three of them follow the convention proposed in [18], where Forgiving applies a 0.25s collar without overlap, Fair applies a 0.25s collar within overlap and Full applies no collar within overlap. In addition, we adopt the Overlap setup, which applies no collar but evaluates only on the overlap, since the overlapping part of speech is a major challenge for diarization systems. We first train all our models on the AMI corpus training set. We have trained five models in the Leptokurtic Generalized Gaussian family and four in the Student’s t family, with shape parameters β∈0.4,0.8,1.0,1.2,1.6β∈\0.4,0.8,1.0,1.2,1.6\ and ν∈0.1,1,10,100ν∈\0.1,1,10,100\ respectively. We also retrained a baseline Gaussian model proposed in the neural FCASA. Note that the Gaussian model can be regarded as Leptokurtic Generalized Gaussian with β=2β=2 or Student’s t with ν→∞ν→∞. For each model setting, we track the diarization loss ℒ3L_3 on the validation set and find the best checkpoints and evaluate all the metrics on the evaluation set. Once found the optimal parameter settings on the AMI corpus, we launch the model training and evaluate their performances on AliMeeting corpus and CHiME-6 corpus to verify the generalization ability of our method. We also perform cross-tests for models trained on one corpus and test them on another corpus. TABLE I: Diarization metrics of the trained models on AMI evaluation set. All metrics are reported in percentage with the best score in bold and the second-best score underlined. We also list the single channel diarization baseline Pyannote 3.1 [6] as a reference. Note that the evaluation of our method is performed chunk-wisely (10 seconds), which is not directly comparable to the Pyannote baselines. Model Shape param. Setup Miss (↓ ) FA (↓ ) Conf. (↓ ) DER (↓ ) JER (↓ ) Gaussian (baseline) - Forgiving 5.59 8.16 0.73 14.48 13.50 Fair 8.61 6.33 0.79 15.73 23.16 Full 10.77 6.81 1.16 18.73 27.65 Overlap 19.40 3.80 0.90 24.11 26.53 Leptokurtic 0.4 Forgiving 7.98 6.23 0.86 15.07 15.81 Fair 11.52 4.97 0.92 17.41 27.33 Full 13.85 5.42 1.28 20.54 31.52 Overlap 24.07 3.05 0.90 28.02 31.35 0.8 Forgiving 6.14 7.84 0.83 14.81 13.98 Fair 9.35 6.29 0.96 16.61 24.36 Full 11.63 6.82 1.30 19.75 28.70 Overlap 20.79 4.20 0.96 25.95 28.10 1.0 Forgiving 5.45 7.36 0.79 13.60 13.01 Fair 8.84 5.76 0.80 15.41 22.66 Full 11.08 6.26 1.13 18.47 27.38 Overlap 20.13 3.52 0.80 24.45 27.02 1.2 Forgiving 5.13 9.12 0.86 15.11 13.90 Fair 8.30 6.80 0.95 16.06 23.59 Full 10.38 7.58 1.34 19.30 28.17 Overlap 18.98 4.11 1.03 24.12 26.78 1.6 Forgiving 6.04 6.65 0.71 13.39 13.32 Fair 9.55 5.08 0.75 15.39 23.33 Full 11.82 5.65 1.06 18.52 27.92 Overlap 21.09 3.10 0.74 24.93 27.82 Student’s t 0.1 Forgiving 5.04 5.59 0.61 11.24 10.78 Fair 7.49 4.42 0.62 12.53 18.38 Full 9.23 5.86 0.92 16.01 23.45 Overlap 16.18 3.18 0.67 20.03 21.99 1 Forgiving 4.30 6.44 0.67 11.41 11.04 Fair 6.92 5.02 0.69 12.62 18.80 Full 8.55 6.52 0.98 16.05 23.82 Overlap 15.57 3.44 0.70 19.71 21.92 10 Forgiving 5.45 7.58 0.78 13.80 12.17 Fair 8.55 6.17 0.80 15.52 20.98 Full 10.93 5.99 1.07 17.99 25.95 Overlap 19.76 3.71 0.74 24.21 26.01 100 Forgiving 5.57 6.78 0.72 13.08 12.48 Fair 8.93 5.39 0.80 15.12 22.09 Full 11.25 5.96 1.09 18.30 26.82 Overlap 20.12 3.55 0.78 24.44 26.72 Pyannote 3.1 IDM Full 9.5 3.6 5.7 18.8 - Pyannote 3.1 SDM Full 11.2 3.8 7.5 22.4 - IV-C Results and discussions The evaluation results on AMI corpus are illustrated in Table I. We first observe that for each model, the error rates become larger when the evaluation setup becomes stricter on overlaps. We then observe some heavy-tailed models have consistent improvements against the baseline in error rates across different setups. For example, we have better results from Leptokurtic Generalized Gaussian models with β=1.0,1.6β=1.0,1.6, whereas Student’s t models outperform the baseline by a large margin (2-3% in absolute and 10-15% in relative) with ν=0.1,1ν=\0.1,1\. This validates the effectiveness of the proposed heavy-tailed neural FCASA method and also demonstrates the choice of the shape parameters substantially affects model performance. Note that for both heavy-tailed families, a smaller parameter implies a heavier tail. In particular, when ν=1ν=1, the Student’s t corresponds to the Cauchy distribution, which preserves the law under linear combination, making it naturally compatible with the linear model assumption in Eq. (1). Larger shape values (ν=100,β=1.6ν=100,β=1.6) converge towards Gaussian model leading to worse performance. Also, there are high potentials that we could find a better performing shape parameter when β∈[1.0,1.6]β∈[1.0,1.6] or ν∈[0.1,1]ν∈[0.1,1]. TABLE I: Diarization metrics of the trained models on AliMeeting test set. All metrics are reported in percentage with the best score in bold and the second-best score underlined. We also list the single channel diarization baseline Pyannote 3.1 [6] as a reference. Note that the evaluation of our method is performed chunk-wisely (10 seconds), which is not directly comparable to the Pyannote baseline. Model Shape param. Setup Miss (↓ ) FA (↓ ) Conf. (↓ ) DER (↓ ) JER (↓ ) Gaussian (baseline) - Forgiving 2.52 2.53 0.37 5.42 7.04 Fair 3.46 1.90 0.40 5.75 10.29 Full 7.25 4.52 0.70 12.47 19.89 Overlap 14.98 2.00 0.47 17.46 20.25 Student’s t 0.1 Forgiving 2.28 2.35 0.22 4.85 5.97 Fair 2.93 1.76 0.22 4.91 8.46 Full 6.38 4.60 0.49 11.47 18.11 Overlap 13.42 2.01 0.35 15.78 18.28 1 Forgiving 1.87 2.16 0.28 4.32 5.88 Fair 2.62 1.59 0.28 4.49 8.47 Full 6.41 4.09 0.52 11.03 18.18 Overlap 14.17 1.76 0.33 16.26 18.88 Pyannote 3.1 channel 1 Full 10.0 4.4 10.0 24.4 - TABLE I: Diarization metrics of the trained models on CHiME-6 evaluation set. All metrics are reported in percentage with the best score in bold and the second-best score underlined. We also list the single channel diarization official baseline from CHiME-6 [34] as a reference. Note that the evaluation of our method is performed chunk-wisely (10 seconds), which is not directly comparable to the CHiME-6 baseline. Model Shape param. Setup Miss (↓ ) FA (↓ ) Conf. (↓ ) DER (↓ ) JER (↓ ) Gaussian (baseline) - Forgiving 10.73 61.51 6.96 79.20 42.08 Fair 14.72 38.46 7.46 60.65 49.27 Full 17.48 35.61 8.64 61.74 52.89 Overlap 29.84 6.74 6.56 43.14 51.68 Student’s t 0.1 Forgiving 9.16 30.47 5.75 45.37 32.40 Fair 14.83 19.30 6.22 40.35 41.67 Full 17.85 29.01 7.23 54.10 46.61 Overlap 27.15 10.95 6.14 44.25 46.60 1 Forgiving 15.89 19.20 6.58 41.67 38.41 Fair 20.52 12.54 6.80 39.86 46.97 Full 23.72 15.54 7.57 46.84 51.44 Overlap 32.61 3.66 6.08 42.35 51.89 CHiME-6 official baseline Single channel Full - - - 62.0 71.4 The performances of models trained respectively on AliMeeting and CHiME-6 training sets with optimal parameter settings found on AMI corpus (i.e. Student’s t distribution with ν∈0.1,1ν∈\0.1,1\) are reported in Table I and Table I. We observe consistent improvements against the baseline. For AliMeeting corpus, we observe 1%-2% in absolute (10%-20% in relative) improvements compared to the baseline. For CHiME-6 corpus, we even observe up to 20% in absolute (35% in relative) improvements compared to the baseline, due to the difficulty of CHiME-6 corpus and the relatively poor performance of the baseline. Therefore, this verifies the generalization ability of our proposed method and especially justifies the necessity of the heavy-tailed models for difficult acoustic environments. The cross-tests results are reported in Table IV. We observe that for train-test pairs such as AMI-AliMeeting and AliMeeting-AMI, there are minor but consistent improvements over the baseline. However, the performances are still far worse than the in-domain results. This can be justified by the similarity of the meeting scenario and the microphone array configuration between AMI and Alimeeting and the major mismatch is the working language (English vs. Mandarin). On the other hand, cross-tests involving CHiME-6 show less consistency, we think this is due to the large gap between the scenario (meeting vs. party, steady vs. moving), the microphone array configuration (circular vs. linear) and language. TABLE IV: Diarization metrics of the cross-tests. All metrics are reported percentage. We only select the DER of the Full setup for simplicity. Model Shape param. test AMI AliMeeting CHiME-6 train Gaussian (baseline) - AMI 18.73 41.99 80.53 AliMeeting 44.11 12.47 70.71 CHiME-6 47.06 38.17 61.74 Student’s t 0.1 AMI 16.01 39.35 64.25 AliMeeting 43.28 11.47 74.41 CHiME-6 75.45 43.15 54.10 1 AMI 16.05 38.48 80.96 AliMeeting 43.33 11.03 65.52 CHiME-6 40.21 35.94 46.84 IV-D Limitations We notice that for some parameter configurations of Leptokurtic Generalized Gaussian distributions, there is a diarization performance degradation compared to the baseline, e.g. for β∈0.4,0.8,1.2β∈\0.4,0.8,1.2\. This suggests the potential sensitivity of our method to hyperparameter tuning. We will leave the investigation and improvements as future work such as direct estimation of those parameters. V Conclusion and future work We propose a heavy-tailed extension to the neural FCASA system which jointly performs speaker separation and diarization. The proposed method can be formulated as a Gaussian scale mixture model with a random impulse variable to the variance, resulting in a fairly simple objective modification of the original baseline. Through extensive experiments, we find various parameter settings that outperform the baseline by a large margin. In future work, we would like to explore the extension to more heavy-tailed model families and their performance on speaker separation. Alternative directions include finding a better shape parameter tuning strategy, incorporating the speaker activity model into the heavy-tailed modeling [22] and incorporating the geometrical knowledge of microphone array [30]. VI Acknowledgement We acknowledge the use of Claude (https://claude.ai/) to polish this paper. We pasted the draft of the abstract, the first three paragraphs of Section I, Section IV-A and Section IV-B with the instruction “Please rephrase the text in an academic manner”. The outputs were then modified further to better represent our own writing. We thank Dr. Yoshiaki Bando for insightful discussions. This project is funded by ANR Project SAROUMANE (ANR-22-CE23-0011) and granted access to the HPC resources of IDRIS under the allocation 20191014876 attribution made by GENCI. VII Reference References [1] X. Anguera, C. Wooters, and J. Hernando (2007) Acoustic beamforming for speaker diarization of meetings. ieee-taslp 15 (7), p. 2011–2022. External Links: Document Cited by: §I. [2] Y. Bando, T. Aizawa, K. Itoyama, and K. Nakadai (2022) Weakly-Supervised Neural Full-Rank Spatial Covariance Analysis for a Front-End System of Distant Speech Recognition. In interspeech, p. 3824–3828. External Links: Document, ISSN 2958-1796 Cited by: §IV-B. [3] Y. Bando, Y. Masuyama, A. A. Nugraha, and K. Yoshii (2023) Neural Fast Full-Rank Spatial Covariance Analysis for Blind Source Separation. In eusipco, p. 51–55. External Links: Document, ISBN 978-94-645936-0-0 Cited by: §I-A, §I-B, §I-B. [4] Y. Bando, T. Nakamura, and S. Watanabe (2024) Neural Blind Source Separation and Diarization for Distant Speech Recognition. In interspeech, p. 722–726. External Links: Document, ISSN 2958-1796 Cited by: §I, §I-B, §I-B, §IV-B, §IV. [5] Y. Bando, K. Sekiguchi, Y. Masuyama, A. A. Nugraha, M. Fontaine, and K. Yoshii (2021) Neural Full-Rank Spatial Covariance Analysis for Blind Source Separation. ieee-spl 28, p. 1670–1674. External Links: ISSN 1558-2361, Document Cited by: §I-A. [6] H. Bredin (2023) pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In interspeech, Cited by: §IV-B, TABLE I, TABLE I. [7] J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, I. McCowan, W. Post, D. Reidsma, and P. Wellner (2005) The AMI meeting corpus: a pre-announcement. In Proceedings of the Second International Conference on Machine Learning for Multimodal Interaction, MLMI’05, Berlin, Heidelberg, p. 28–39. External Links: Document, ISBN 3-540-32549-2 Cited by: §I, item 1. [8] S. Cornell, T. J. Park, H. Huang, C. Boeddeker, X. Chang, M. Maciejewski, M. S. Wiesner, P. Garcia, and S. Watanabe (2024) The CHiME-8 DASR challenge for generalizable and array agnostic distant automatic speech recognition and diarization. In 8th International Workshop on Speech Processing in Everyday Environments, p. 1–6. External Links: Document Cited by: §I. [9] N. Q. K. Duong, E. Vincent, and R. Gribonval (2010) Under-Determined Reverberant Audio Source Separation Using a Full-Rank Spatial Covariance Model. ieee-taslp 18 (7), p. 1830–1840. External Links: ISSN 1558-7916, 1558-7924, Document Cited by: §I-A, §I-A. [10] M. Fontaine, K. Sekiguchi, A. Nugraha, Y. Bando, and K. Yoshii (2022) Generalized Fast Multichannel Nonnegative Matrix Factorization Based on Gaussian Scale Mixtures for Blind Source Separation. ieee-acm-taslp 30, p. 1734–1748. External Links: ISSN 2329-9290, 2329-9304, Document Cited by: §I, §I-A, §I-A, §I-B. [11] Y. Fujita, N. Kanda, S. Horiguchi, K. Nagamatsu, and S. Watanabe (2019) End-to-End Neural Speaker Diarization with Permutation-free Objectives. In Interspeech, p. 4300–4304. Cited by: §I. [12] S. Horiguchi, Y. Fujita, S. Watanabe, Y. Xue, and P. Garcia (2022) Encoder-Decoder Based Attractors for End-to-End Neural Diarization. ieee-acm-taslp 30, p. 1493–1507. External Links: ISSN 2329-9290, 2329-9304, Document Cited by: §I. [13] S. Horiguchi, Y. Takashima, P. Garcia, S. Watanabe, and Y. Kawaguchi (2022) Multi-Channel End-to-End Neural Diarization with Distributed Microphones. In icassp, p. 7332–7336. Cited by: §I. [14] S. Horiguchi, N. Tawara, T. Ashihara, A. Ando, and M. Delcroix (2025) Can we really repurpose multi-speaker asr corpus for speaker diarization?. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Cited by: item 2. [15] N. Ito and T. Nakatani (2019) FastMNMF: Joint Diagonalization Based Accelerated Algorithms for Multichannel Nonnegative Matrix Factorization. icassp, p. 371–375. External Links: Document, ISBN 9781479981311 Cited by: §I-A. [16] D. P. Kingma, D. J. Rezende, S. Mohamed, and M. Welling (2014) Semi-supervised learning with deep generative models. In NIPS, Vol. 27. Cited by: §I-B, §I-B. [17] D. P. Kingma and M. Welling (2014) Auto-Encoding Variational Bayes. In iclr, Cited by: §I-A, §I-B, §I-B. [18] F. Landini, J. Profant, M. Diez, and L. Burget (2022) Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization: Theory, implementation and analysis on standard tasks. csl 71, p. 101254. External Links: ISSN 08852308, Document Cited by: §I, §IV-B. [19] S. Leglaive, X. Alameda-Pineda, L. Girin, and R. Horaud (2020) A Recurrent Variational Autoencoder for Speech Enhancement. icassp, p. 371–375. External Links: Document, ISBN 9781509066315 Cited by: §I-A. [20] L. Li, H. Kameoka, and S. Makino (2022) FastMVAE2: On improving and accelerating the fast variational autoencoder-based source separation algorithm for determined mixtures. ieee-acm-taslp 31, p. 96–110. External Links: ISSN 2329-9290, Document Cited by: §I-A. [21] S. Maiti, Y. Ueda, S. Watanabe, C. Zhang, M. Yu, . X. Zhang, and Y. Xu (2023) EEND-S: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers. In slt, p. 480–487. Cited by: §I-B, §I-B. [22] S. Mao, M. Fontaine, A. Larcher, and R. Badeau (2026) Neural multichannel distant speaker diarization and source separation with beta speaker activity prior. External Links: 2608.28661, Link Cited by: §V. [23] T. Mariotte, A. Larcher, S. Montrésor, and J. Thomas (2024) ASoBO: Attentive beamformer selection for distant speaker diarization in meetings. In interspeech, p. 1620–1624. External Links: ISSN 2958-1796, Document Cited by: §I. [24] J. M. Pardo, X. Anguera, and C. Wooters (2006) Speaker diarization for multiple distant microphone meetings: mixing acoustic features and inter-channel time differences. In interspeech, p. paper 1337–Thu1A1O.5–0. External Links: Document Cited by: §I. [25] H. Sawada, H. Kameoka, S. Araki, and N. Ueda (2013) Multichannel Extensions of Non-Negative Matrix Factorization With Complex-Valued Data. ieee-taslp 21 (5), p. 971–982. External Links: ISSN 1558-7924, Document Cited by: §I-A. [26] R. Scheibler and N. Ono (2020) Fast and Stable Blind Source Separation with Rank-1 Updates. In icassp, p. 236–240. External Links: ISSN 2379-190X, Document Cited by: §I-A. [27] R. Scheibler and M. Togami (2021) Surrogate Source Model Learning for Determined Source Separation. In icassp, p. 176–180. External Links: ISSN 2379-190X, Document Cited by: §I-B. [28] K. Sekiguchi, Y. Bando, A. A. Nugraha, K. Yoshii, and T. Kawahara (2020) Fast Multichannel Nonnegative Matrix Factorization With Directivity-Aware Jointly-Diagonalizable Spatial Covariance Matrices for Blind Source Separation. ieee-acm-taslp 28, p. 2610–2625. External Links: ISSN 2329-9304, Document Cited by: §I-A, §I-A, §I-B. [29] D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur (2017) Deep Neural Network Embeddings for Text-Independent Speaker Verification. In interspeech, p. 999–1003. External Links: Document Cited by: §I. [30] Y. Sumura, D. D. Carlo, A. Arie Nugraha, Y. Bando, and K. Yoshii (2024) Joint audio source localization and separation with distributed microphone arrays based on spatially-regularized multichannel NMF. In 2024 18th International Workshop on Acoustic Signal Enhancement (IWAENC), p. 145–149. External Links: Document Cited by: §V. [31] E. Vincent (2011) Probabilistic modeling paradigms for audio source separation. In Machine Audition: Principles, Algorithms and Systems, External Links: Document Cited by: §I-A. [32] Q. Wang, Y. Huang, G. Zhao, E. Clark, W. Xia, and H. Liao (2024) DiarizationLM: Speaker diarization post-processing with large language models. In interspeech, p. 3754–3758. External Links: ISSN 2958-1796, Document Cited by: §I. [33] S. Watanabe, M. Mandel, J. Barker, E. Vincent, A. Arora, X. Chang, S. Khudanpur, V. Manohar, D. Povey, D. Raj, D. Snyder, A. S. Subramanian, J. Trmal, B. B. Yair, C. Boeddeker, Z. Ni, Y. Fujita, S. Horiguchi, N. Kanda, T. Yoshioka, and N. Ryant (2020) CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings. In 6th International Workshop on Speech Processing in Everyday Environments (CHiME 2020), p. 1–7. External Links: Document Cited by: item 3. [34] S. Watanabe, M. Mandel, J. Barker, E. Vincent, A. Arora, X. Chang, S. Khudanpur, V. Manohar, D. Povey, D. Raj, D. Snyder, A. S. Subramanian, J. Trmal, B. B. Yair, C. Boeddeker, Z. Ni, Y. Fujita, S. Horiguchi, N. Kanda, T. Yoshioka, and N. Ryant (2020) CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings. In Proc. CHiME, p. 1–7. External Links: Document Cited by: §I, §I, §I-A, TABLE I. [35] H. Yin, Y. Chen, C. Deng, L. Cheng, H. Wang, C. Tan, Q. Chen, W. Wang, and X. Li (2025) SpeakerLM: End-to-End Versatile Speaker Diarization and Recognition with Multimodal Large Language Models. External Links: 2508.06372, Link Cited by: §I. [36] T. Yoshioka and T. Nakatani (2012) Generalization of multi-channel linear prediction methods for blind MIMO impulse response shortening. ieee-taslp 20 (10), p. 2707–2720. External Links: Document Cited by: §IV-B. [37] F. Yu, S. Zhang, Y. Fu, L. Xie, S. Zheng, Z. Du, W. Huang, P. Guo, Z. Yan, B. Ma, X. Xu, and H. Bu (2022) M2MeT: The ICASSP 2022 multi-channel multi-party meeting transcription challenge. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, p. 6167–6171. External Links: 2110.07393 Cited by: §I, item 2.