Paper deep dive
A New Transformer-Based Approach for Audio-Based Kinship Verification and a New Uncontrolled Mandarin Kinship Speech Dataset
Qiyang Sun, Langqing Zhang, Yupei Li, Björn Schuller
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/20/2026, 4:23:19 AM
Summary
This paper introduces CONVTRAP-TN, a novel transformer-based architecture for audio-based kinship verification, and the ARKIN dataset, a new uncontrolled Mandarin speech dataset designed to reflect real-world recording conditions. The authors demonstrate that existing methods lack cross-dataset robustness and propose their model to address signal sparsity via attention mechanisms.
Entities (12)
Relation Signals (7)
KAN-AV → containslanguage → English
confidence 95% · The majority of speakers included in the KAN-AV dataset are English-speaking public figures.
ARKIN → containslanguage → Mandarin
confidence 95% · The dataset contains 2554 utterances, out of which 95% are in Mandarin
CONVTRAP-TN → uses → Transformer
confidence 95% · CONVTRAP-TN uses the CONVTRAP embedding function (CNN + transformer + attention pooling)
ARKIN → affiliatedwith → Imperial College London
confidence 90% · Authors are affiliated with Imperial College London and they created the dataset
ARKIN → createdby → Qiyang Sun
confidence 90% · Qiyang Sun is the first author and the paper introduces ARKIN
CONVTRAP-TN → outperforms → TripletNet
confidence 90% · Out of the methods we experimented with, our new approach, CONVTRAP-TN, performed the best.
CONVTRAP-TN → usesfeature → MFCC
confidence 90% · CONVTRAP first takes in an MFCC sequence
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Kinship verification is a task involving determining whether two individuals share a first-order kin relation. To tackle this task, we propose CONVTRAP-TN, a new architecture for audio-based kinship verification, and conduct an ablation study on the proposed model. To the best of our knowledge, we are the first to apply the successful transformer architecture to the task of audio-based kinship verification. Furthermore, we also collect a custom speech dataset, ARKIN, which accurately reflects everyday recording conditions. We do this because only a few speech datasets with kinship labels currently exist, all of which either source extremely noisy in-the-wild data from the internet, or instruct speakers to record in specific environments. These settings fail to reflect real-world scenarios where users record on personal devices under unrestrained conditions. Additionally, we perform a series of preliminary baseline experiments on the collected dataset, including speaker verification and recognition, speech recognition, age estimation, and kinship verification, as well as cross-dataset kinship verification experiments to show that existing methods are not robust across datasets.
Tags
Links
- Source: https://arxiv.org/abs/2609.14145v1
- Canonical: https://arxiv.org/abs/2609.14145v1
Trouble viewing inline? Open PDF directly →
Full Text
38,961 characters extracted from source content.
Expand or collapse full text
A New Transformer-Based Approach for Audio-Based Kinship Verification and a New Uncontrolled Mandarin Kinship Speech Dataset 1st Qiyang Sun Affiliation: Department of Computing Imperial College London London, United Kingdom q.sun23@imperial.ac.uk 1st Langqing Zhang* Affiliation: Department of Computing Imperial College London London, United Kingdom langqing.zhang24@imperial.ac.uk 2nd Yupei Li Affiliation: Department of Computing Imperial College London London, United Kingdom yl7622@ic.ac.uk 3rd Björn W. Schuller Affiliation: Chair of Health Informatics TUM University Hospital Munich, Germany bjoern.schuller@imperial.ac.uk Abstract Kinship verification is a task involving determining whether two individuals share a first-order kin relation. To tackle this task, we propose CONVTRAP-TN, a new architecture for audio-based kinship verification, and conduct an ablation study. We additionally analyze the proposed model from the perspective of explainable AI (XAI), which is rarely, if at all, done in existing literature on audio-based kinship verification. Our architecture targets the issue of signal sparsity in kinship verification through attention mechanisms. To the best of our knowledge, we are the first to apply a transformer-based architecture to the task of audio-based kinship verification. Furthermore, we also collect a custom speech dataset, ARKIN, which accurately reflects everyday recording conditions. We do this because only a few speech datasets with kinship labels currently exist, all of which either source extremely noisy in-the-wild data from the internet, or instruct speakers to record in specific environments. These settings fail to reflect real-world scenarios where users record on personal devices under unrestrained conditions. Additionally, we perform a series of preliminary baseline experiments on the collected dataset, including speaker verification and recognition, speech recognition, age estimation, and kinship verification, as well as cross-dataset kinship verification experiments to show that existing methods are not robust across datasets. Index Terms: Audio Kinship Verification, Transformer, Deep Metric Learning, Kinship Dataset, Cross-Corpus Evaluation I Introduction Kinship verification involves deciding whether a kinship relation (first-order unless specified) exists between two individuals by analyzing signals collected from them. It is an important field of research with a broad range of applications, including searching for lost family members, forensics, home security, and authentication [1, 2]. The kinship verification task is considerably more challenging than speaker-level tasks such as speaker verification [3]. This can be remedied by incorporating data of multiple modalities, as seen in the large performance improvement of models trained on multiple modalities compared to models trained on a single modality [3, 2, 4]. Nevertheless, this also means that if the system does not have access to data of multiple modalities, performance severely degrades, which can easily be the case in real-world applications. Therefore, it is important to develop robust methods for various modalities individually [5]. While visual kinship datasets are abundant [6, 7, 8, 9, 10], audio-based counterparts remain scarce [3, 11, 4]. Furthermore, any finite dataset inevitably contains biases [12], which negatively impact the ability of a model to stay invariant with respect to demographic and environmental features. This is a problem even in visual datasets [13, 14], which are much more abundant than audio. Researchers have attempted to mitigate this problem by applying explicit techniques to address dataset bias. For example, on a visual face dataset, we can use age progression to perform data augmentation [15], or on a speech dataset, convert audio samples to an age-standardized domain by performing age domain conversion [16]. Such efforts to mitigate algorithmic bias align with the core principles of Friendly AI (FAI) that remain beneficial to humanity in the future [17]. However, many features, such as the environment in which data is collected, cannot be explicitly modeled. For the task of audio-based kinship verification specifically, considering the limited number of publicly available audio kinship datasets, it is evident that robust audio-based kinship verification systems that can maintain performance across datasets are much harder to develop compared to systems with access to visual or multimodal data. To our best knowledge, almost all existing literature on audio-based kinship verification do not perform cross-dataset evaluation, which may be simply due to the limited number of datasets available. It is also acknowledged in literature that more robust voice models are needed [18, 19, 20], as they can in turn help improve fusion models that make use of audio data [11]. On top of that, all publicly available audio kinship datasets either consist of purely in-the-wild data from public figures appearing in sources such as movies and YouTube videos [3, 2, 4], or of data collected under specific environments with limited noise, such as a studio or a quiet room [11]. Neither of these settings reflects average end-users interacting with audio recording systems under realistic conditions. Moreover, the existing body of audio data available for kinship verification consists primarily of English speech, which hinders progress in developing language-invariant models. Therefore, we construct a new, everyday-condition-oriented, purely audio dataset containing speech in Mandarin, with complete kinship, age, and gender labels, called Audio under Realistic conditions for KINship verification (ARKIN). We try to preserve natural recording conditions of users, aiming to provide a balance between noisy conditions and quiet conditions, as well as a bridge between research and real-world deployment. We also perform cross-dataset benchmarking experiments on existing methods and show that cross-dataset generalization remains challenging. Having obtained the dataset, we develop a new method that takes into consideration the sparse nature of kinship signals and makes use of the attention mechanism [21, 22] to encode audio. While attention has been previously applied to feature fusion across multiple modalities for kinship verification [2], to the best of our knowledge, we are the first to propose such an approach for audio-based kinship verification. Existing literature on audio-based kinship verification [3, 11] rely either on traditional approaches such as GMM-UBM [23] or purely convolutional architectures. To tackle the issue of training instability, we use a two-stage training scheme where we first start with the easier task of speaker recognition. In addition, existing literature on audio-based kinship verification rarely perform explainability analysis, which we will explore in this paper. Hence, the main contributions of our paper are the following: 1. We introduce ARKIN, a new uncontrolled speech dataset focusing on everyday recording conditions for audio-based kinship verification, and perform a series of baseline experiments on it. 2. We propose CONVTRAP-TN, a novel transformer-based architecture for audio-based kinship verification. We further investigate CONVTRAP by analyzing model attention weights and conducting an ablation study. 3. We investigate the cross-dataset robustness of existing methods for audio-based kinship verification. I Related work TABLE I: Comparison of existing audio kinship datasets Dataset #Families #Speakers #Clips Recording Condition Language Age Range Availability KAN-AV 255 970 28 003 In-the-wild English 3–100 On request TALKIN-Family 246 1012 4048 Semi-controlled Mandarin 5–81 On request/Restricted FIW-M 150 500 937 In-the-wild English N/A On request/Restricted ARKIN (ours) 86 256 2554 Uncontrolled Mandarin 5–84 On request Few datasets contain both audio data as well as kinship labels. Current public datasets include the Kinship Age geNder - Audio Visual (KAN-AV) dataset [3], the TALking KINship - Family (TALKIN-Family) dataset [11], and the Families In Wild Multimedia (FIW-M) database [4]. A comparison of the datasets (including ours) is presented in Table I. KAN-AV [3] is a manually annotated dataset consisting of in-the-wild audio and video data from the internet. The dataset contains around 28 000 utterances, with roughly 66% of the speakers having at least one kinship relation with another in the dataset. The majority of speakers included in the KAN-AV dataset are English-speaking public figures. KAN-AV is the largest publicly available audio dataset with kinship labels. TALKIN-Family [11] is a manually annotated dataset collected from offline participants in a semi-controlled setting, where participants were instructed to record in a quiet location. The dataset contains 1012 subjects, totaling 4048 utterances. All speakers are from China and speak Mandarin, each having at least one kinship relation with another. FIW-M extends the Families-In-Wild [8] image dataset to include audio data. Similar to KAN-AV, speakers consist of public figures, the majority of which is English speaking and data is collected from the internet. The dataset contains 937 utterances (given by “# clips” in the paper). We stress that our dataset is the first to provide Mandarin recordings in uncontrolled conditions using consumer-grade devices. We also differentiate our approach from sourcing recordings in-the-wild in that we do not source public recordings from the internet, which often have very low signal-to-noise ratio and are recorded on specialized equipment not available to the average population. I ARKIN dataset I-A Overview The ARKIN dataset is a benchmark dataset that captures real-world recording conditions and serves as a bridge between datasets collected under controlled conditions and datasets with in-the-wild data. The dataset contains 256 speakers, each labeled with age, gender, and nationality. There are 2554 utterances, out of which 95% are in Mandarin and only 5% in English. We decided to primarily focus on Mandarin speech due to the current imbalance between the amount of English and non-English data available. Each utterance is provided with a manual text transcription and a language label. There are 230 kinship pairs, and every speaker has at least one kinship relation with another speaker. The dataset is organized by speaker IDs. Audio files are provided in 16-bit PCM format sampled at 48 kHz. I-B Data collection and processing In order to obtain audio with realistic environmental conditions that reflect average use-cases, we asked participants to record on any device they normally use (usually, this is their mobile phone) with no restriction on background conditions. Participants can record wherever they want, however they want, and whenever they want. As a result, we obtain a dataset with a large variety of background conditions. For the content of the recordings, participants were asked to read out first a fixed phrase, followed by an improvisation of anything they would like. Some participants did not improvise, whereas others had trouble improvising and asked for a reference instead, in which case we provided them with additional passages as inspiration. We manually divide audio clips into utterances of varying lengths (roughly 0.3 s - 13.3 s) and cut out sections that do not contain transcribable audio from the subject. We were able to capture a variety of speech patterns such as rapid pausing, false starts, and filler sounds. Because of this, our utterances can vary greatly in length and content. The data collection process is illustrated in Figure 1. Fig. 1: The data collection process IV Baseline experiments IV-A Speech representation To verify the performance of the ARKIN dataset, we need to extract speech features from utterances. We first resample our audio to 16 kHz and extract MFCCs [24] with 24 coefficients using the Kaldi toolkit [25]. We choose a frame length of 25 ms and a frame shift of 15 ms. Then We apply Cepstral Mean and Variance Normalization (CMVN) and Voice Activity Detection (VAD) with an energy threshold of 5 to prepare the input features for subsequent modeling. We evaluate a series of baseline representations derived from these features. First, we use Kaldi to train a 512-component GMM-UBM [23], projecting the UBM statistics into a low-dimensional total variability space to extract 400-dimensional i-vectors[26]. Second, we implement TDNN architectures [27, 28] trained from scratch on our dataset for speaker classification. From this model, we extract standard 512-dimensional x-vectors, reduced 300-dimensional embeddings obtained by modifying the final fully-connected layers to suit our data scale, as well as 256-dimensional embeddings from the ECAPA-TDNN architecture. Finally, to leverage transfer learning, we utilize a ResNet model [29] from SpeechBrain [30]. Pretrained on the massive VoxCeleb corpus [31, 32], this model yields compact 256-dimensional embeddings. We perform a 7:1:2 split to divide our data into disjoint training, validation, and test sets, ensuring no speakers in different sets share a kinship relation. For each utterance, we extract MFCC sequences, as well as i-vectors, x-vectors, and ResNet feature vectors, and perform, on the extracted features, various experiments described below. We average our results over five independent experiments. Details of the partitions are presented in Table I. TABLE I: Detailed statistics of the ARKIN dataset partitions. M: Male, F: Female. Kinship types: F-S (Father-Son), F-D (Father-Daughter), M-S (Mother-Son), M-D (Mother-Daughter), B-B (Brother-Brother), S-S (Sister-Sister), B-S (Brother-Sister). Partition Speakers Audio Age (Years) Kinship Pairs Distribution Total Pairs M F Total # Utts Mean ± SD Range F-S F-D M-S M-D B-B S-S B-S Train 98 98 196 1796 34.3 ± 18.9 5–84 27 22 44 33 12 12 18 168 Val 9 10 19 286 29.3 ± 17.5 8–61 2 3 6 6 1 3 7 28 Test 19 22 41 472 29.5 ± 15.5 6–57 5 3 10 9 3 1 3 34 Total 126 130 256 2554 33.2 ± 18.3 5–84 34 28 60 48 16 16 28 230 IV-B Kinship verification For (audio-based) kinship verification, we use the standard DNN-based triplet network (TripletNet) [33], as well as our own method, which we refer to as the CONVolution-TRansformer-Attention-Pool Triplet Network (CONVTRAP-TN), to perform deep metric learning. We use a simple random triplet mining strategy. We construct N positive ordered pairs of form (v→a,v→p)( v_a, v_p) of utterances from speakers with a kinship relation, in which the first speaker is the anchor and the second is the positive sample. During training time, we select a random speaker from the training set that does not share a kinship relation with the anchor and use that speaker as a negative sample to form the triplet (v→a,v→p,v→n)( v_a, v_p, v_n). IV-B1 TripletNet The triplets are fed into the TripletNet, which uses a shared embedding function fDNN:ℝd→ℝ⌊d/2⌋f_DNN:R^d d/2 , parameterized by a 4-layer DNN with layer norm, leaky ReLU, and dropout to map each sample onto an embedding metric space, obtaining triplets of form (u→a,u→p,u→n)=(fDNN(v→a),fDNN(v→p),fDNN(v→n))( u_a, u_p, u_n)=(f_DNN( v_a),f_DNN( v_p),f_DNN( v_n)). The first layers consist of d neurons, whereas the output layer consists of ⌊d/2⌋ d/2 neurons, followed by a normalization operation so that all embeddings sit on a unit hypersphere. The model then calculates the Euclidean distance between the embeddings and outputs ordered pairs of form (dp,dn)=(∥u→a−u→p∥,∥u→a−u→n∥)(d_p,d_n)=( u_a- u_p , u_a- u_n ). The goal then is to minimize the triplet loss function given by (1), where M∈ℝ+M ^+ is the margin. We set M=10M=10. L(dp,dn)=∑i=1NReLU(dp2−dn2+M)L(d_p,d_n)= _i=1^NReLU(d_p^2-d_n^2+M) (1) IV-B2 CONVTRAP-TN Fig. 2: The CONVTRAP architecture Similarly, CONVTRAP-TN uses a shared embedding function to map triplets before computing the triplet loss. However, unlike the standard TripletNet, our CONVTRAP-TN uses the CONVTRAP embedding function (CNN + transformer + attention pooling). Our design takes into account the sparse nature of kinship signals. The CNN provides local context to the transformer, which then aggregates features that capture important clues spaced far apart, while discarding noise. The attention pooling module is specifically chosen to tackle the issue of kinship signal sparsity, as it can selectively aggregate features instead of diluting them, as other approaches such as mean pooling would do. CONVTRAP first takes in an MFCC sequence C∈ℝL×24C ^L× 24 and projects it onto a high-dimensional space via a 3-layer DNN. We pick a hidden dimension of 512, following x-vectors. We then use a convolutional block (1D-Conv, GELU, Linear) as a form of positional encoding before feeding the sequence into a transformer encoder [21] (3-layers and 8 heads empirically found to work best). The output is then using simple attention pooling [34] defined as: attnPoolq→(C)=Csoftmax[mlp(C)q→512],attnPool_ q(C)=C Tsoftmax [ mlp(C) q 512 ], (2) where q→ q is a learnable query vector. Finally, the pooled vector is fed into a 2-layer DNN with layer norm, leaky ReLU, and dropout of 0.3. The complete CONVTRAP embedding function fCONVTRAPf_CONVTRAP is visualized in Figure 2. Additionally, we employ a two-stage training approach to help with training stability. We first train the model on speaker recognition via a separate classification head (2-layer DNN with layer norm, leaky ReLU, and a dropout of 0.5) attached after the attention pool. We then freeze all parameters prior to the attention pool and train the partially frozen model on kinship verification. We are essentially extracting intermediate features, similar to x-vectors [27], with the difference being that we have a sequence instead of a single vector. TABLE I: Kinship verification balanced accuracy (%, higher is better) at FAR = FRR, EER (%, lower is better), and MinDCF with Ptarget=0.5,Cfn=Cfp=1P_target=0.5,C_fn=C_fp=1 (lower is better) Method Features Acc ↑ EER ↓ MinDCF ↓ TripletNet i-vector 56.1 43.9 0.90 x-vector 63.4 36.6 0.73 x-vector 300-dim 64.7 35.3 0.69 x-vector ECAPA 55.2 44.8 0.87 ResNet 61.2 38.8 0.73 COVTRAP-TN MFCCs 69.9 30.1 0.53 TABLE IV: Sensitivity (%, higher is better) at FAR = FRR to different kinship types: F-S (Father-Son), F-D (Father-Daughter), M-S (Mother-Son), M-D (Mother-Daughter), B-B (Brother-Brother), S-S (Sister-Sister), B-S (Brother-Sister). Bold: most sensitive, underline: below average sensitive. Method Features F-S F-D M-S M-D B-B S-S B-S Average ↑ TripletNet i-vector 42.8 43.6 81.1 67.1 71.8 38.2 87.2 61.7 x-vector 27.8 15.1 75.5 73.3 21.9 59.8 88.8 51.7 x-vector 300-dim 28.6 57.8 79.5 63.7 25.0 50.1 85.3 55.7 x-vector ECAPA 31.0 55.9 76.1 70.4 64.9 48.2 72.5 59.9 ResNet 32.8 55.7 65.7 61.7 47.8 43.5 49.2 50.9 COVTRAP-TN MFCCs 87.8 93.0 54.4 60.2 64.6 94.7 63.7 74.1 TABLE V: Cross-dataset (train on KAN-AV, evaluate on ARKIN) kinship verification balanced accuracy scores (%) at FAR = FRR. ft: fine-tuned, TN: TripleNet, vec.: vector Method Features KAN-AV ARKIN ARKIN-ft TN i-vec 51.7 49.7 53.5 x-vec 53.4 48.9 52.6 x-vec 300-dim 52.8 50.2 49.8 x-vec ECAPA 51.9 49.8 49.2 ResNet 51.2 46.0 47.6 CONVTRAP-TN MFCCs 56.9 50.0 44.1 We implement both models in PyTorch, with random Gaussian initialization, and train with a batch size of 16 using SGD with a learning rate of 0.001 and momentum 0.9. We compute the verification threshold using the validation set (at FAR = FRR), fixing it for the test set. Results are shown in Table I. Out of the methods we experimented with, our new approach, CONVTRAP-TN, performed the best. This is likely because our frozen transformer backbone outputs a sequence of features with no reduction in dimensionality, meaning the downstream attention pool and DNN have access to relatively uncompressed features extracted directly from raw MFCCs. The attention mechanism in the transformer also helped to capture any nuanced long-range patterns, which would be much harder for a CNN-only architecture. For this task, we also experimented with both 512-dimensional and 300-dimensional x-vectors and found that the lower 300-dimensional features performed better, likely due to the size of the dataset. Furthermore, to investigate biases in verifying different types of kinship relations, we present sensitivity scores by kin type for each model in Table IV. We observe that the sensitivity variation between kinship types is large. The standard triplet network is least sensitive to father-son (F-S), father-daughter (F-D), brother-brother (B-B), and sister-sister (S-S) relations, and usually most sensitive to mother-son (M-S), mother-daughter (M-D), and brother-sister (B-S) relations. On the other hand, CONVTRAP-TN is highly sensitive to F-S and F-D relations, and less sensitive to M-S and M-D relations. While sensitivity scores by kin type are somewhat proportional to the kin type frequency in training data, the model itself is also shown to be an influential factor for kin type bias. Certain kin types may naturally be easier to verify, but we do not have enough evidence to support this claim. IV-C Cross-dataset kinship verification We additionally perform cross-dataset baseline kinship verification. We choose to experiment with the KAN-AV dataset, as it is the largest currently available dataset for our needs. We train baseline models on the KAN-AV dataset and compare evaluation performance on KAN-AV with performance on our own dataset. We also evaluate the effectiveness of fine-tuning the model using the first 20% of the training speakers from our dataset (detailed in our repository). We use the same methods explained in Section IV-B. Results are shown in Table V, from which it is clear that baseline models trained on KAN-AV suffer drastic performance degradation when tested on our dataset. Although CONVTRAP-TN performed the best, like other methods, its performance worsened by a large factor on KAN-AV compared to our ARKIN. We suspect one reason is that KAN-AV is mostly in English, while our dataset is mostly in Mandarin. Another reason is that recording environments of speeches, interviews, or YouTube videos differ greatly from our everyday style environment. As seen in Table V, after fine-tuning on a portion of ARKIN, the performance of CONVTRAP-TN, like most other approaches, worsened (worse than a random model). We suspect this is because training on the noisy KANAV has effectively served as a poor initialization for the model to start learning ARKIN, as the datasets are so differently distributed, making transfer learning difficult. To clearly visualize the domain gap between our dataset and KAN-AV, we reduce high-dimensional kin embeddings (generated from 512-dimensional x-vectors) to 2-dimensional points using the t-distributed stochastic neighbor embedding (t-SNE) method [35]. The points are plotted in Figure 3, from which it is evident that there is a large domain gap between KAN-AV and our dataset. Fig. 3: Embeddings (after dimensionality reduction) of utterances from KAN-AV (red circles) and ARKIN (blue triangles) obtained from a TripletNet trained on x-vectors IV-D Auxiliary Tasks Given the rich meta-labels provided in ARKIN, we also perform baseline experiments on auxiliary tasks to validate the dataset’s utility across different speech dimensions. We summarize the performance of these tasks in Table VI. First, we evaluated identity-related tasks: speaker verification and speaker recognition. Despite the uncontrolled recording environments, the ResNet-based model achieved a recognition accuracy of 96.7% and a verification accuracy of 82.5%, outperforming traditional i-vector baselines. This confirms that the dataset preserves high-fidelity speaker identity features necessary for robust biometric modeling. Second, for paralinguistic and linguistic analysis, we benchmarked age estimation and Automatic Speech Recognition (ASR). In age estimation, ResNet yielded the lowest Mean Absolute Error (MAE) of 5.6 years. For ASR, the Whisper model demonstrated superior robustness against environmental noise with a Character Error Rate (CER) of 18.6%, compared to 50.8% for Wav2Vec 2.0. These results highlight the challenging nature of the dataset for standard acoustic models while showing that robust transcription is achievable. For conciseness, we only present a summary of results here, verifying the reliability of dataset labels. Detailed experimental setups, hyperparameters, and full reproduction scripts for all auxiliary tasks will be available in our public repository. TABLE VI: Baseline performance on all auxiliary tasks. Spk Ver./Rec. are accuracy (%, higher is better), Age is MAE (years, lower is better), and ASR is CER (%, lower is better). Method Spk Ver. Spk Rec. Age Est. ASR (Acc %) ↑ (Acc %) ↑ (MAE) ↓ (CER %) ↓ i-vector 69.0 34.7 10.3 - x-vector 74.4 73.2 8.2 - ResNet 82.5 96.7 5.6 - Wav2Vec 2.0 - - - 50.8 Whisper - - - 18.6 V Further results on CONVTRAP-TN V-A Attention visualization We visualize attention scores from the attention pool component of the CONVTRAP model trained on ARKIN and visualize them alongside spectrograms of input audio clips. We pick 4 samples, which include a true positive pair, a true negative pair, and a false positive pair. We present the visualizations in Figure 4. Results show that the output of CONVTRAP consistently aggregates information from salient regions corresponding to active speech segments. In the failure (FP) case, the model still attended to active speech regions, meaning the error likely comes from ambiguity within learned representations rather than confounding factors such as background noise. Fig. 4: Visualization of input spectrograms alongside attention pool scores: (a) and (b) are a true positive pair, (b) and (c) are a true negative pair, (c) and (d) are a false positive pair. V-B Ablation study To investigate the effectiveness of different components of CONVTRAP, we remove or decrease the capacity of different components and evaluate each ablated model. “Two-stage” notes the full model trained using the two-stage approach described previously. “Single layer” denotes the full model except with 1 transformer layer instead of 3. “W/o convBlock” denotes the full model without the convolution block before the transformer encoder. “W/o attnPool” denotes the full model with mean pooling instead of attention pooling. We also attempt to train the model directly on kinship verification without first training on speaker recognition, which we denote by “Single-stage”. We present results in Table VII. It is clear that the most vital aspects of our approach are attention pooling and two-stage training, as the model fails catastrophically without either of them. We suspect the reason why attention pooling performs much better than mean pooling is that our model does not have any other downsampling module, so the pooling mechanism has to aggregate all frames at once. Since useful kinship clues are usually concentrated in just a few regions, mean pooling struggles to provide a useful aggregation, whereas attention pooling can learn to select the most relevant frames, as evidenced by Figure 4. When it comes to single-stage training, kinship verification has proven to be too difficult a task. Without training the model first on speaker recognition, the model easily gets stuck in a bad local minimum due to the instability of the triplet network, unable to learn a good general representation of kinship. The convolution block also proved critical, which is not surprising considering the permutation-invariant nature of attention mechanisms. Lastly, having just a single transformer layer decreased performance by only 2.8%, which is small compared to other components, though we suspect this may change if much larger datasets were used for training. TABLE VII: Kinship verification performance of ablated models trained on ARKIN. Balanced accuracy scores (%, higher is better) at FAR = FRR, EER (%, lower is better), and MinDCF with Ptarget=0.5,Cfn=Cfp=1P_target=0.5,C_fn=C_fp=1 (lower is better). Model Acc ↑ EER ↓ MinDCF ↓ w/o attnPool 47.2 52.8 0.76 w/o convBlock 53.4 46.6 0.72 Single layer 67.1 32.9 0.57 Single-stage 50.0 50.0 0.98 Two-stage 69.9 30.1 0.53 VI Conclusion We proposed CONVTRAP-TN, a new transformer-based architecture for kinship verification. Ablation study showed that attention pooling and two-stage training are the most influential components of the effectiveness of the approach. Through splitting kinship verification sensitivity scores by kinship type, we found that models are generally the most sensitive to brother-sister type relations and that the sensitivity variation between kinship types is large, raising concerns of fairness and bias, which is a direction for future improvements. We also introduced ARKIN, a kinship dataset enabling cross-dataset evaluation of audio models. Our evaluations reveal that models trained in one environment generalize poorly to others, regardless of dataset size, confirming a remarkable domain gap. One limitation is that we are only experimenting with supervised features. Future work will look at SSL features and focus on further developing robust audio models. To mitigate demographic homogeneity, we will expand data collection and evaluation across diverse populations. Ethics declaration Approved by the institutional Research Ethics Committee of Anonymous Institution, the data collection protocol was fully compliant with GDPR and PIPL frameworks, involving compensated, voluntary participants who provided informed written consent. Furthermore, all released audio data have been strictly anonymised to eliminate personally identifiable metadata and are made available exclusively for non-commercial research purposes. Disclosure of AI tools No generative AI was used to produce any part of this paper. References [1] N. Nader, F. E. El-Gamal, S. El-Sappagh, K. S. Kwak, and M. Elmogy (2021) Kinship verification and recognition based on handcrafted and deep learning feature-based techniques. PeerJ Comput. Sci. 7, p. e735. Note: PMID: 34977344; PMCID: PMC8670373 External Links: Document Cited by: §I. [2] X. Wu, E. Granger, T. H. Kinnunen, X. Feng, and A. Hadid (2019) Audio-visual kinship verification in the wild. In 2019 International Conference on Biometrics (ICB), Crete, Greece, p. 1–8. External Links: Document Cited by: §I, §I, §I. [3] T. Kefalas, E. Fotiadou, M. Georgopoulos, Y. Panagakis, P. Ma, S. Petridis, T. Stafylakis, and M. Pantic (2023) KAN-av dataset for audio-visual face and speech analysis in the wild. Image and Vision Computing 140, p. 104839. External Links: ISSN 0262-8856, Document, Link Cited by: §I, §I, §I, §I, §I. [4] J. P. Robinson, Z. Khan, Y. Yin, M. Shao, and Y. Fu (2022) Families in wild multimedia: a multimodal database for recognizing kinship. IEEE Trans. Multimedia 24, p. 3582–3594. External Links: Document Cited by: §I, §I, §I. [5] Q. Sun, A. Akman, and B. W. Schuller (2025) Explainable artificial intelligence for medical applications: a review. ACM Transactions on Computing for Healthcare 6 (2), p. 1–31. Cited by: §I. [6] R. Fang, K. D. Tang, N. Snavely, and T. Chen (2010) Towards computational models of kinship verification. In 2010 IEEE International Conference on Image Processing (ICIP), Hong Kong, China, p. 1577–1580. External Links: Document Cited by: §I. [7] J. Lu, X. Zhou, Y.-P. Tan, Y. Shang, and J. Zhou (2014) Neighborhood repulsed metric learning for kinship verification. IEEE Trans. Pattern Anal. Mach. Intell. 36 (2), p. 331–345. External Links: Document Cited by: §I. [8] J. P. Robinson, M. Shao, Y. Wu, H. Liu, T. Gillis, and Y. Fu (2018) Visual kinship recognition of families in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 40 (11), p. 2624–2637. External Links: Document Cited by: §I, §I. [9] H. Dibeklioğlu, A. A. Salah, and T. Gevers (2012) Are you really smiling at me? spontaneous versus posed enjoyment smiles. In Computer Vision – ECCV 2012, Lecture Notes in Computer Science, vol. 7574, A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid (Eds.), Note: ECCV 2012 External Links: Document Cited by: §I. [10] H. Yan and J. Hu (2018) Video-based kinship verification using distance metric learning. Pattern Recognition 75, p. 15–24. External Links: ISSN 0031-3203, Document Cited by: §I. [11] X. Wu, X. Zhang, X. Feng, M. B. López, and L. Liu (2024) Audio-visual kinship verification: a new dataset and a unified adaptive adversarial multimodal learning approach. IEEE Trans. Cybern. 54 (3), p. 1523–1536. External Links: Document Cited by: §I, §I, §I, §I, §I, §I. [12] A. Torralba and A. A. Efros (2011) Unbiased look at dataset bias. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Colorado Springs, CO, USA, p. 1521–1528. External Links: Document Cited by: §I. [13] M. Georgopoulos, Y. Panagakis, and M. Pantic (2020) Investigating bias in deep face analysis: the kanface dataset and empirical study. Image and Vision Computing 102, p. 103954. External Links: ISSN 0262-8856, Document Cited by: §I. [14] Z. Liu and K. He (2025) A decade’s battle on dataset bias: are we there yet?. Note: arXiv:2403.08632preprint External Links: Link Cited by: §I. [15] M. Georgopoulos, J. Oldfield, M. A. Nicolaou, and et al. (2021) Mitigating demographic bias in facial datasets with style-based multi-attribute transfer. Int. J. Comput. Vis. 129, p. 2288–2307. External Links: Document Cited by: §I. [16] Q. Sun, A. Akman, X. Jing, M. Milling, and B. W. Schuller (2025) Audio-based kinship verification using age domain conversion. IEEE Signal Process. Lett. 32, p. 301–305. External Links: Document Cited by: §I. [17] Q. Sun, Y. Li, E. Alturki, S. M. K. Murthy, and B. W. Schuller (2026) Towards friendly ai: a comprehensive review and new perspectives on human-ai alignment. AI and Ethics 6 (2), p. 193. Cited by: §I. [18] R. Li, J.-Y. Jiang, J. Liu, C.-C. Hsieh, and W. Wang (2020) Automatic speaker recognition with limited data. In Proceedings of the 13th International Conference on Web Search and Data Mining (WSDM ’20), New York, NY, USA, p. 340–348. External Links: Document Cited by: §I. [19] S. Kataria, P. S. Nidadavolu, J. Villalba, N. Chen, P. García-Perera, and N. Dehak (2020) Feature enhancement with deep feature losses for speaker verification. In ICASSP 2020 - IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, p. 7584–7588. External Links: Document Cited by: §I. [20] Y. Zhao and et al. (2025) Investigation of zero-shot text-to-speech models for enhancing short-utterance speaker verification. Note: arXiv preprint External Links: Link Cited by: §I. [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30, p. . Cited by: §I, §IV-B2. [22] S. Islam, H. Elmekki, A. Elsebai, J. Bentahar, N. Drawel, G. Rjoub, and W. Pedrycz (2023) A comprehensive survey on applications of transformers for deep learning tasks. External Links: 2306.07303, Link Cited by: §I. [23] F. Bimbot, J.-F. Bonastre, C. Fredouille, and et al. (2004) A tutorial on text-independent speaker verification. EURASIP J. Adv. Signal Process., p. 101962. External Links: Document Cited by: §I, §IV-A. [24] S. Davis and P. Mermelstein (1980) Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Trans. Acoust., Speech, Signal Process. 28 (4), p. 357–366. External Links: Document Cited by: §IV-A. [25] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, and J. Silovsky (2011) The kaldi speech recognition toolkit. In IEEE Workshop on Automatic Speech Recognition and Understanding, Note: Vol. 1, p. 5–1 Cited by: §IV-A. [26] N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet (2011) Front-end factor analysis for speaker verification. IEEE Trans. Audio, Speech, Lang. Process. 19 (4), p. 788–798. External Links: Document Cited by: §IV-A. [27] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur (2018) X-vectors: robust dnn embeddings for speaker recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, p. 5329–5333. External Links: Document Cited by: §IV-A, §IV-B2. [28] B. Desplanques, J. Thienpondt, and K. Demuynck (2020) ECAPA-tdnn: emphasized channel attention, propagation and aggregation in tdnn based speaker verification. In Interspeech 2020, p. 3830–3834. External Links: Document Cited by: §IV-A. [29] J. Villalba and et al. (2020) State-of-the-art speaker recognition with neural network embeddings in nist sre18 and speakers in the wild evaluations. Computer Speech & Language 60, p. 101026. Cited by: §IV-A. [30] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio (2021) SpeechBrain: a general-purpose speech toolkit. External Links: Link Cited by: §IV-A. [31] A. Nagrani, J. S. Chung, and A. Zisserman (2017) VoxCeleb: a large-scale speaker identification dataset. Note: Interspeech 2017 External Links: Document Cited by: §IV-A. [32] J. S. Chung, A. Nagrani, and A. Zisserman (2018) VoxCeleb2: deep speaker recognition. In Proc. Interspeech, p. 1086–1090. External Links: Document Cited by: §IV-A. [33] E. Hoffer and N. Ailon (2015) Deep metric learning using triplet network. In Similarity-Based Pattern Recognition, Lecture Notes in Computer Science, vol. 9370, A. Feragen, M. Pelillo, and M. Loog (Eds.), External Links: Document Cited by: §IV-B. [34] K. Okabe, T. Koshinaka, and K. Shinoda (2018) Attentive statistics pooling for deep speaker embedding. In Interspeech 2018, p. 2252–2256. External Links: Document Cited by: §IV-B2. [35] L. van der Maaten and G. Hinton (2008) Visualizing data using t-sne. Journal of Machine Learning Research 9 (86), p. 2579–2605. Cited by: §IV-C.