Paper deep dive
A Holistic Framework for Robust Bangla ASR and Speaker Diarization with Optimized VAD and CTC Alignment
Zarif Ishmam, Zarif Mahir, Shafnan Wasif, Md. Ishtiak Moin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 9:41:14 AM
Summary
This paper presents a robust framework for long-form Bangla Automatic Speech Recognition (ASR) and Speaker Diarization, addressing the challenges of low-resource language processing. The methodology involves optimizing Voice Activity Detection (VAD), using Connectionist Temporal Classification (CTC) for forced word alignment to segment audio under 30 seconds, and fine-tuning the Tugstugi Whisper model for transcription. For diarization, a three-stage curriculum learning pipeline is employed, including base adaptation, source separation using Demucs, and dynamic data augmentation. The approach significantly reduces Word Error Rate (WER) and Diarization Error Rate (DER) compared to baseline models.
Entities (11)
Relation Signals (7)
Tugstugi → isfinetunedfor → Bangla
confidence 95% · fine-tune the bengaliAI/tugstugi_bengaliai-asr_whisper-medium model... focusing on Bengali token distributions
CTC → isusedfor → forced alignment
confidence 93% · Utilizing Connectionist Temporal Classification (CTC) via forced word-alignment to maintain precise temporal boundaries
Demucs → isusedfor → Source Separation
confidence 92% · We systematically process the entire training directory using the Demucs hybrid Transformer-based architecture... strips away background noise
Whisper → haslimitation → Long-form Audio
confidence 90% · Whisper is limited to 30-second inputs, leading to hallucinations, forgetting, and drift in long-form audio.
VAD → isusedfor → Noise Filtering
confidence 90% · Implementing a refined Voice Activity Detection (VAD) layer to accurately identify and filter non-speech segments
Tugstugi → achieves → Low WER
confidence 88% · Table IV shows Tugstugi Fine-tuned Public WER 0.21988
DL Sprint 4.0 → isusedfor → Training
confidence 85% · utilizing the provided dataset for DL Sprint 4.0... we address these limitations by proposing a specialized pipeline
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Despite being one of the most widely spoken languages globally, Bangla remains a low-resource language in the field of Natural Language Processing (NLP). Mainstream Automatic Speech Recognition (ASR) and Speaker Diarization systems for Bangla struggles when processing longform audio exceeding 3060 seconds. This paper presents a robust framework specifically engineered for extended Bangla content by leveraging preexisting models enhanced with novel optimization pipelines for the DL Sprint 4.0 contest. Our approach utilizes Voice Activity Detection (VAD) optimization and Connectionist Temporal Classification (CTC) segmentation via forced word alignment to maintain temporal accuracy and transcription integrity over long durations. Additionally, we employed several finetuning techniques and preprocessed the data using augmentation techniques and noise removal. By bridging the performance gap in complex, multi-speaker environments, this work provides a scalable solution for real-world, longform Bangla speech applications.
Tags
Links
- Source: https://arxiv.org/abs/2602.22935v1
- Canonical: https://arxiv.org/abs/2602.22935v1
Trouble viewing inline? Open PDF directly →
Full Text
18,592 characters extracted from source content.
Expand or collapse full text
A Holistic Framework for Robust Bangla ASR and Speaker Diarization with Optimized VAD and CTC Alignment Zarif Mahir Computer Science and Engineering Bangladesh University of Engineering and Technology Dhaka, Bangladesh zarifmahir21@gmail.com Zarif Ishmam Computer Science and Engineering Bangladesh University of Engineering and Technology Dhaka, Bangladesh zarif.ishmam9@gmail.com Shafnan Wasif Computer Science and Engineering Bangladesh University of Engineering and Technology Dhaka, Bangladesh shafnanwasif@gmail.com Md. Ishtiak Moin Computer Science and Engineering Bangladesh University of Engineering and Technology Dhaka, Bangladesh zarifishtiak@gmail.com Abstract—Despite being one of the most widely spoken lan guages globally, Bangla remains a lowresource language in the field of Natural Language Processing (NLP). Mainstream Automatic Speech Recognition (ASR) and Speaker Diarization systems for Bangla struggles when processing longform audio exceeding 3060 seconds. This paper presents a robust frame work specifically engineered for extended Bangla content by leveraging preexisting models enhanced with novel optimization pipelines for the DL Sprint 4.0 contest. Our approach utilizes Voice Activity Detection (VAD) optimization and Connectionist Temporal Classification (CTC) segmentation via forced word alignment to maintain temporal accuracy and transcription integrity over long durations. Additionally, we employed several finetuning techniques and preprocessed the data using augmen tation techniques and noise removal. By bridging the perfor mance gap in complex, multispeaker environments, this work provides a scalable solution for realworld, longform Bangla speech applications. Index Terms—Scientific Writing, NLP, ASR, Speaker Diariza tion, Longform ASR, Finetuning, Bangla, Bengali I.Introduction Bangla is the seventh most spoken native language in the world, with over 270 million speakers across the Indian sub continent and a vast global diaspora[1]. Despite its linguistic prominence, Bangla remains categorized as a lowresource language in the context of Natural Language Processing (NLP) and Automatic Speech Recognition (ASR)[2]. While recent advancements in Deep Learning and Transformerbased archi tectures have improved shortsnippet transcription, a critical performance bottleneck persists: longform audio processing. Most stateoftheart ASR models for Bangla are trained and evaluated on segmented, highquality clips typically under 30 seconds[3]. However, realworld applications—such as podcast transcription, judicial proceedings, and broadcast media—require systems capable of handling continuous audio streams spanning several minutes or hours. In these extended contexts, standard models often suffer from “hallucinations”, catastrophic forgetting, or loss of temporal alignment, leading to a significant spike in Word Error Rate (WER)[4]. The challenge is twofold. First, longform ASR requires precise segmentation to prevent the encoder from being over whelmed by sequence length. Second, Speaker Diarization (SD)—the task of “who spoke when”—becomes increasingly complex as the number of speakers and the duration of the audio increases. Existing Bangla datasets rarely provide the necessary metadata for robust diarization in longform, multi speaker environments. In this paper, utilizing the provided dataset for DL Sprint 4.0[5], we address these limitations by proposing a specialized pipeline engineered specifically for longform Bangla content. Our framework enhances both transcription and speaker di arization through five core contributions: 1) Audio Preprocessing and Noise Removal: Employing advanced source separation techniques [6] 2) Dynamic Data Augmentation: Applying variable gain adjustments and dynamic audio augmentation during training to improve model robustness across diverse, realworld recording environments. 3) VAD Optimization: Implementing a refined Voice Activity Detection (VAD) layer to accurately identify and filter nonspeech segments prior to transcription. 4) CTC Alignment and Segmentation: Utilizing Con nectionist Temporal Classification (CTC) via forced word alignment[7] to maintain precise temporal bound aries, allowing for intelligent audio chunking that preserves continuous speech context. 5) Integrated Diarization: Developing a unified frame work that combines finetuned segmentation and embeddingbased clustering to execute precise multi speaker labeling in tandem with ASR. I.Literature Review The development of robust Automatic Speech Recognition (ASR) and Speaker Diarization (SD) systems for Bengali has evolved from traditional statistical methods to deep learning architectures. However, longform, multispeaker audio pro cessing remains challenging in lowresource settings. A.Advancements in Bengali Automatic Speech Recognition Early Bengali ASR relied on Hidden Markov Models (HMM) and Gaussian Mixture Models (GMM) using read speech or broadcast news corpora[8]. These models struggled with Bengali’s morphological complexity and dialectal vari ance. Crowdsourced datasets like Bengali Common Voice enabled deep learning shifts, such as Wav2Vec 2.0[9]. Re cently, Transformerbased models like OpenAI’s Whisper have shown strong zeroshot and finetuning performance. Fine tuned Whisper models support tasks like clinical translation and speechtosign language generation[10]. However, Whis per is limited to 30second inputs, leading to hallucinations, forgetting, and drift in longform audio. B.Long-Form Processing and Temporal Alignment Extended audio requires advanced segmentation and align ment. Traditional slidingwindow methods often cut words, disrupting context for Transformer decoders. Recent work integrates Connectionist Temporal Classification (CTC) for forced wordlevel alignment, enabling precise timestamp extraction and boundaryrespecting chunking to preserve lin guistic integrity. C.Speaker Diarization in Low-Resource Contexts Speaker Diarization involves Voice Activity Detection (VAD), segmentation, embedding extraction, and clustering. Toolkits like pyannote.audio standardize these for high resource languages[11]. Bengali diarization resources are scarce, but initiatives like BengaliLoop and DL Sprint 4.0 provide annotated longform datasets[5]. Out ofthebox mod els show high Diarization Error Rates (DER) due to noise, overlaps, and environments. Research now focuses on hybrid approaches with vocal source separation and finetuning for scalable pipelines. I.Methodology A.Long-form ASR Our proposed approach is structured into two primary phases: robust dataset preprocessing via CTCbased forced alignment, and acoustic model finetuning utilizing a pre trained Whisper architecture. The complete pipeline is implemented using Python, leveraging the Hugging Face transformers and datasets libraries, alongside ctc- forced-aligner for precise temporal bounding. a) Dataset Preprocessing and Forced Alignment: To address the challenges of longform Bangla speech, we utilize datasets such as the DL Sprint 4.0 (Bengali Longform Speech Recognition) corpus. Raw audio sequences must be precisely segmented to accommodate the fixed input constraints of transformerbased ASR models without truncating contextual speech. Data Normalization: Audio files (.wav) are paired with their corresponding transcripts (.txt). To ensure textual consistency, all transcripts undergo Unicode Normalization Form Canonical Composition (NFC) and rigorous whitespace stripping to remove orthographic anomalies. CTC Forced Alignment: We utilize the multilingual MMS300M forced aligner, specifically configured to its Ben gali language head. The model is loaded in float16 precision on GPU hardware to accelerate emission generation. For each audiotranscript pair, the process is as follows: 1)Emission Generation: The alignment model’s encoder generates framelevel CTC emissions. 2)Textual Preprocessing: Transcripts are romanized to map the Bengali script to the acoustic phonetic repre sentations expected by the aligner. 3)Word Level Alignment: The system computes Viterbi alignments, extracting precise start and end timestamps for each localized word. 4)Error Handling: A fallback mechanism is implemented to filter or flag instances of mismatched or empty alignments caused by extreme background noise or untranscribed speech. Chunking and Segmentation: Utilizing the extracted word level timestamps, the continuous audio stream is intel ligently segmented into discrete chunks of strictly under 30 seconds. This step ensures that word boundaries are preserved —preventing cuts in the middle of a spoken word—while fine-tuning loop CTC forced alignment of each word Raw Dataset Merge them to create segments Processed Dataset tugstugi bengaliAI whisper medium model Inference Fig. 1. Long-form ASR fine-tuning pipeline strictly adhering to the maximum input length of the Whisper architecture. b) Acoustic Model Fine-Tuning: For the tran scription engine, we finetune the bengaliAI/ tugstugi_bengaliai-asr_whisper-medium [12] model using our newly aligned and segmented dataset. All training procedures were executed on an NVIDIA A100 GPU. Data Preparation: Prior to feature extraction, all audio chunks are strictly resampled to 16 kHz to match the Whisper encoder’s expected temporal resolution. Transcripts are tok enized using the native Whisper processor to map Bengali text to the corresponding bytelevel BPE tokens.[13] TABLE I Summary Statistics of the Bengali ASR Chunked Dataset AttributeDescription Dataset Name zarifmahir21/ bengali-asr-chunked Primary TaskAutomatic Speech Recognition (ASR) Total Utterances13,547 Total Duration≈ 158 Hours Audio FormatWAV (Mono) Sampling Rate16 kHz Chunking StrategySegmented to < 30s (Whisper limit) Training Protocol: The model is trained focusing on Ben gali token distributions using a batch size of 16. To manage distributed training and optimize GPU memory utilization, we employ the Hugging Face Accelerate library. Model performance is continuously evaluated across epochs by cal culating the Word Error Rate (WER) using the jiwer metric, as shown in Table I. TABLE I Training and Validation Metrics for Whisper FineTuning StepTraining LossValidation LossWER 1900.9391290.19589526.391037 3800.5516140.16525622.931987 5700.2440600.16789722.711062 B.Bangla Long-form Speaker Diarization To address the highly complex acoustic environments of realworld Bengali audio—which frequently feature back ground noise, overlapping speech, and varying recording volumes—we engineered a sophisticated threestage Curricu lum Learning pipeline. Rather than training a model natively on a static dataset, our approach progressively guides the model from broad acoustic adaptation to highly specific vocal refinement, culminating in a robustness stresstest using dynamic data augmentation. a) Stage 1: Base Adaptation (Real-World Acoustics): The initial phase focuses on adapting a multilingual baseline (pyannote/segmentation-3.0) to the specific pho netic cadence and overlapping conversational patterns of the Bengali language. Data Preparation: Raw audio files (.wav) from the training corpus are aligned with their respective CSV annotations, converted into PyAnnote’s native Rich Transcription Time Mark (RTTM) format. Noisy FineTuning: The model is trained on these unaltered, raw audio chunks. In this phase, the network learns to identify fundamental Bengali speech characteristics while simultane Base PyAnnote Segmentation 3.0 model Demucs Clean and Finetuned PyAnnote model Augmentation Final PyAnnote Model on Diarization Inference Phase 1: Base Adaptation Phase 2: Clean Refinement Phase 3: Robustness and Augmentation Raw Dataset Fig. 2. Three-phase Speaker Diarization fine-tuning pipeline ously navigating the natural background noise, static, and music inherent to the original recordings. The bestperforming checkpoint is versioned and exported for subsequent refine ment. b) Stage 2: Clean Refinement (Source Separation): While Phase 1 builds a generalized understanding of the acoustic environment, Phase 2 isolates the distinct mathematical signa tures—or speaker embeddings—of individual voices. Vocal Isolation: We systematically process the entire training directory using the Demucs hybrid Transformer based architecture (htdemucs --two-stems=vocals). This effectively strips away background noise and nonspeech artifacts, outputting pure vocal tracks. Audio Standardization: The isolated vocal tracks are converted to a single monaural channel and strictly resampled to 16 kHz to align with the PyAnnote ecosystem’s input requirements. Clean FineTuning: Utilizing the specialized model weights derived from Stage 1, we resume training exclusively on the Demucscleaned audio. By removing background dis tractions, the network focuses entirely on perfecting boundary detection and speaker differentiation. c) Stage 3: Robustness and Dynamic Augmentation: Models trained exclusively on sourceseparated audio risk overfitting to the artificial cleanliness and specific volume levels of the Demucs output. To mitigate this brittleness, the final phase introduces controlled stochastic distortions. Dynamic Dataloader Augmentation: We integrate torch_audiomentations to apply onthefly volume transformations. Specifically, a Gain augmentation (ranging from −6.0 dB to +6.0 dB) is applied with a 40% probability (푝=0.4) to any given 5.0second chunk during the training loop. Custom Wrapper Implementation: Due to the strict data formatting constraints of PyAnnote’s internal architecture, we implemented a custom ManualExternal PyTorch module. This wrapper safely catches the augmented audio tensors and seamlessly bundles them with their corresponding target labels, ensuring stability during the forward pass. Training Protocol and Hyperparameters: All finetuning stages were executed using PyTorch Lightning on GPU hardware (particularly the Tesla T4 GPUs on Kaggle). The SpeakerDiarization task was configured to process fixed audio chunks, optimizing for crossentropy loss. The complete hyperparameter configuration utilized across the advanced training phases is detailed in Table I. TABLE I Curriculum Learning Hyperparameters for Speaker Diarization ParameterPhase 1Phase 2Phase 3 (Augmenta tion) Chunk Duration5.0 s5.0 s10.0 s Batch Size323232 Learning Rate 5×10 −5 5×10 −5 5×10 −5 EpochsVariableVariable20 Audio SourceRaw AudioDemucs Vo cals Demucs Vocals Augmentation Protocol NoneNone Dynamic Gain (±6.0 dB, 푝=0.4) Loss FunctionCrossEn tropy CrossEn tropy CrossEntropy IV.Conclusion Modern Transformerbased ASR models typically utilize restricted context windows—often between 30 and 60, as computational complexity scales quadratically with sequence length. Consequently, specialized strategies are required to maintain accuracy in longform ASR and speaker diarization. The following summary outlines our findings across these various approaches: TABLE IV ASR Performance Benchmark on Bengali Public and Private Test Sets Model Architec ture Configura tion Public WER ↓ Private WER ↓ Time TugstugiFine tuned 0.219880.23585 4h TugstugiZeroshot0.361420.378714h BanglaASRFine tuned0.500470.543292.53h Mozilla LargeBase0.631710.6972612h Whisper Large Turbo v3 Zeroshot0.865940.8863045h TABLE V Speaker Diarization Performance (DER) across different Training Strategies Training StrategyPublic DER ↓Private DER ↓ Normal Finetuning (Base)0.231470.31129 Fine tuning + Demucs Refinement0.216210.33454 Fine tuning + Data Augmentation 0.214600.32663 External Dataset + Augmentation0.225230.33982 External Dataset (No Augmenta tion)[14] 0.231630.33543 We optimized inference performance on Kaggle by utilizing dual T4 GPUs in parallel, reducing the total inference time for the finetuned bengaliAI/tugstugi_bengaliai- asr_whisper-medium model to approximately two hours. References [1]D. M. Eberhard, G. F. Simons, and C. D. Fennig, Ethnologue: Lan- guages of the World, 28th ed. Dallas, Texas: SIL International, 2025. [Online]. Available: http://w.ethnologue.com/ [2]P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The State and Fate of Linguistic Diversity in the NLP World,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, Association for Computational Linguistics, 2020, p. 6282–6293. [Online]. Available: https://aclanthology.org/2020.aclmain.560 [3]A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust Speech Recognition via LargeScale Weak Supervi sion,” in Proceedings of the 40th International Conference on Machine Learning, PMLR, 2023, p. 28448–28466. [Online]. Available: https:// proceedings.mlr.press/v202/radford23a.html [4]A. Koenecke, A. S. G. Choi, K. X. Mei, M. Sloane, and H. Schell mann, “Careless Whisper: SpeechtoText Hallucination Harms,” in Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT '24), Association for Computing Machinery, 2024, p. 1671–1681. [Online]. Available: https://doi.org/10.1145/ 3630106.3658992 [5]H. M. S. Tabib et al., “BengaliLoop: Community Benchmarks for LongForm Bangla ASR and Speaker Diarization,” arXiv preprint arXiv:2602.14291, 2026, doi: 10.48550/arXiv.2602.14291. [6]M. Ashraf, “Demucs: Music Source Separation with AI.” GitHub, 2024. [7]M. Ashraf, “CTCForcedAligner: A fast and accurate forced alignment tool using CTC models.” GitHub, 2023. [8]S. Ahmed, N. Sadeq, S. S. Shubha, M. N. Islam, M. A. Adnan, and M. Z. Islam, “Preparation of Bangla Speech Corpus from Publicly Avail able Audio & Text,” in Proceedings of the 12th Language Resources and Evaluation Conference, 2020, p. 6586–6592. [9]S. Alam et al., “Bengali Common Voice Speech Dataset for Automatic Speech Recognition,” arXiv preprint arXiv:2206.14053, 2022. [10]A. H. Nabil et al., “An EndtoEnd Bengali SpeechtoSign Language Generation Framework Using FineTuned Whisper ASR and Grapheme Level Visual Mapping,” Research Square, 2025, doi: 10.21203/ rs.3.rs7652580/v1. [11]H. Bredin et al., “pyannote.audio: neural building blocks for speaker diarization,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, p. 7124– 7128. [12]Bengali.AI, “Tugstugi: Whisper Medium for Bengali ASR.” Hugging Face, 2023. [13]Z. Mahir, “Bengali ASR Chunked Dataset.” Hugging Face, 2026. [14]I. Zarif, “Bangla Diarization Dataset 2.” Hugging Face, 2026.