Paper deep dive
Relative Time Intervals Representation for Word-level Timestamping with Masked Training
Quanwei Tang, Zhiyu Tang, Xu Li, Dong Zhang, Shoushan, Guodong Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 8/26/2026, 5:25:18 AM
Summary
The paper introduces a novel framework for word-level timestamping in Speech Large Language Models (SpeechLLMs) by replacing absolute timestamps with relative time intervals. This approach mitigates vocabulary explosion and cumulative error propagation associated with absolute timestamps. The authors propose a hybrid fine-tuning strategy combining full-parameter fine-tuning of timestamp-augmented embeddings and the LM head with LoRA fine-tuning of decoder layers. Additionally, a masked timestamp training objective is designed to enhance robustness against noisy annotations. Experiments on datasets like AISHELL-2, LibriSpeech, and Wenet-Meeting demonstrate significant improvements in timestamp prediction accuracy and speech transcription performance compared to baselines like Qwen2-Audio and WhisperTimestamped.
Entities (11)
Relation Signals (10)
Relative Timestamp → replaces → Absolute Timestamp
confidence 98% · Specifically, we replace traditional absolute timestamps with relative timestamps
Hybrid Fine-Tuning → uses → LoRA
confidence 97% · combined with LoRA fine-tuning of the decoder layers
Hybrid Fine-Tuning → uses → Full-parameter fine-tuning
confidence 97% · full-parameter fine-tuning of the timestamp-augmented embedding layer and language model head
Relative Timestamp → mitigates → vocabulary explosion
confidence 96% · This approach mitigates vocabulary explosion and temporal drift
Relative Timestamp → achieves → compact vocabulary
confidence 95% · achieving a more compact vocabulary and stronger generalization capabilities
Masked Timestamp Training → enhances → Robustness
confidence 95% · preventing the model from over-relying on ground-truth timestamps, and thereby enhancing robustness against noisy real-world annotations
Relative Timestamp (Ours) → outperforms → Qwen2-Audio
confidence 94% · our model reached an impressive 91.13% Precision and 86.88% Recall on Wenet-Meeting, substantially outperforming all baselines.
Relative Timestamp (Ours) → outperforms → WhisperTimestamped
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Although Speech Large Language Models (SpeechLLMs) excel at speech understanding and generation, their capacity for fine-grained, temporally aligned outputs remains underexplored. Our work addresses this gap by enabling SpeechLLMs to jointly model speech content and temporal structure, effectively transforming them from ``content understanding machines" into ``temporal-aware content understanding machines". Specifically, we replace traditional absolute timestamps with relative timestamps, achieving a more compact vocabulary and stronger generalization capabilities. To efficiently infuse timestamp prediction ability into pre-trained large language models, we introduce a hybrid fine-tuning strategy: full-parameter fine-tuning of the timestamp-augmented embedding layer and language model head, combined with LoRA fine-tuning of the decoder layers. Moreover, we design a masked timestamp training objective, preventing the model from over-relying on ground-truth timestamps, and thereby enhancing robustness against noisy real-world annotations. Extensive experiments demonstrate that our approach achieves significant improvements in timestamp prediction accuracy while maintaining strong speech transcription performance.
Tags
Links
- Source: https://arxiv.org/abs/2608.24041v1
- Canonical: https://arxiv.org/abs/2608.24041v1
Trouble viewing inline? Open PDF directly →
Full Text
27,518 characters extracted from source content.
Expand or collapse full text
RELATIVE TIME INTERVALS REPRESENTATION FOR WORD-LEVEL TIMESTAMPING WITH MASKED TRAINING Quanwei Tang 1 , Zhiyu Tang 2 , Xu Li 3 , Dong Zhang 1,4† , Shoushan Li 1 , Guodong Zhou 1 1 Soochow University 2 University of Queenland 3 AISpeech Ltd 4 Jiangsu Key Lab of Language Computing ABSTRACT Although Speech Large Language Models (SpeechLLMs) excel at speech understanding and generation, their capacity for fine-grained, temporally aligned outputs remains underexplored. Our work ad- dresses this gap by enabling SpeechLLMs to jointly model speech content and temporal structure, effectively transforming them from “content understanding machines” into “temporal-aware content un- derstanding machines”. Specifically, we replace traditional abso- lute timestamps with relative timestamps, achieving a more compact vocabulary and stronger generalization capabilities. To efficiently infuse timestamp prediction ability into pre-trained large language models, we introduce a hybrid fine-tuning strategy: full-parameter fine-tuning of the timestamp-augmented embedding layer and lan- guage model head, combined with LoRA fine-tuning of the decoder layers. Moreover, we design a masked timestamp training objec- tive, preventing the model from over-relying on ground-truth times- tamps, and thereby enhancing robustness against noisy real-world annotations. Extensive experiments demonstrate that our approach achieves significant improvements in timestamp prediction accuracy while maintaining strong speech transcription performance. 1 Index Terms— Timestamping, Speech LLM, Absolute Times- tamp, Relative Timestamp, Masked Training 1. INTRODUCTION Recent research has explored the integration of large language models (LLMs) with speech processing [1, 2]. However, gener- ating fine-grained, word-level timestamped transcription has not been fully explored yet [3, 4]. This task demands not only ac- curate speech understanding but also precise temporal alignment between linguistic units and their acoustic realizations [5, 6]. A core challenge lies in how to represent and predict temporal structure efficiently within the constrained vocabulary and sequential decod- ing paradigm of LLMs [7, 8, 9, 10]. Existing methods like Qwen2 Audio [11] often employ absolute timestamps for modeling. e.g., <|0.45|>Okay<|1.10|>let’s<|2.50|>go<|5.70| >... indicates that the word ”Okay” starts at 0.45 seconds and ends at 1.10 seconds. While this representation is intuitive and facilitates evaluation, its dependence on an absolute temporal reference limits scalability and generalization over long-duration audio segments. As shown in Figure 1, predicting absolute timestamps is prone to cumulative error propagation. Absolute timestamps are monoton- ically increasing, causing LLMs to learn a cumulative pattern: gen- erating larger values by adding estimated segment durations to prior timestamps. Furthermore, this approach exhibits poor generalization to unseen temporal ranges. For instance, it fails to predict times- tamps beyond the training duration (e.g., at 60 seconds when trained † Corresponding author. dzhang@suda.edu.cn 1 https://github.com/tangquanwei/Timestamp-Aware-Speech-LLM Fig. 1. Traditional methods rely on fixed-reference absolute times- tamps, often struggling with cumulative errors or precise synchro- nization.Our approach utilizes dynamic interval-based relative timestamps, which inherently model temporal relationships and of- fer improved robustness and accuracy. only on utterances up to 30 seconds). A major practical limitation lies in the prohibitively large vocabulary required for high temporal precision: achieving 0.01-second resolution over a 300-second au- dio clip requires 30,000 distinct timestamp tokens. This explosion in vocabulary significantly increases training complexity and com- putational overhead, making the approach unscalable for long-form speech processing. To address these limitations, we propose a relative timestamp representation, defined as the time interval between consecutive words. (e.g., <|0.65|> for words aligned at <|0.45|> and <|1.10|>). This method requires solely estimating the current segment’s duration, thereby minimizing error susceptibility. Only a limited time interval (such as 0.01 seconds to 5 seconds) needs to be learned, and by accumulating these intervals, any length of audio can be represented. Vocabulary only needs to contain limited time interval tokens, compared to absolute timestamps. Human speech perception naturally focuses on relative timing: we intuitively pro- cess how long after the previous word the next one occurs, rather than its absolute position in time. This relative interval modeling aligns naturally with the autoregressive generation paradigm of language models. Absolute timestamps force the model to learn an entirely new absolute value that is unrelated to the previously arXiv:2608.24041v1 [cs.AI] 25 Aug 2026 ( , ) ( , ) Hello <|1.80|> World <|2.12|> <|1.20|> Audio Fetures Text Prompt Text Embedding Decoder Layers LM Head Audio Encoder Proj <|sil|><|0.00|><|30.00|> ... Hello <|0.60|> World <|0.32|> <|1.20|> Audio Fetures Text Prompt Text Embedding Decoder Layers LM Head Audio Encoder Proj <|sil|><|0.00|><|5.00|> ... <|0.00|> <|sil|> Hello <mask> World <|0.32|> <|1.20|> Predicate Label mask token Absulote Timestamp Our Relative Timestamp FreezeFull paramLoRA<mask> Fig. 2. Architectural Comparison of an Absolute Timestamp-based Model (Left) and Our Proposed Relative Timestamp-based Model (Right). The traditional method models timestamps as independent absolute values and struggles with representing timestamps. In contrast, our approach utilizes a compact vocabulary of relative timestamps. By leveraging LoRA fine-tuning on the decoder, our model efficiently learns to jointly generate both text and temporal markers. To ensure accuracy and robustness, our training incorporates a combined loss function and timestamp regularization (token masking), which forces the model to learn a more resilient representation of temporal information. generated content, which is contrary to its natural generation mode. We make the following three key contributions: 1. A Novel Relative Time Interval Representation for Word- Level Timestamping: We introduce a relative timestamp represen- tation that models inter-word intervals instead of absolute positions. This approach mitigates vocabulary explosion and temporal drift, en- abling accurate timestamp prediction for arbitrarily long audio with- out length-dependent degradation. 2. Advanced Regularization and Alignment Strategies: We propose a regularization technique, Timestamp Masking, which mit- igates overfitting to perfect alignments, along with a dynamically weighted Joint Temporal Alignment Loss that adaptively balances transcription accuracy and timestamp fidelity during training. 3. Hybrid Parameter-Efficient Fine-Tuning: We fully fine- tune only timestamp-specific modules (embedding layer and LM head), while adapting pre-trained decoder layers via LoRA. This role-aware strategy preserves linguistic knowledge with minimal pa- rameter updates, aligning adaptation granularity with each module’s functional role. 2. METHODOLOGY 2.1. Relative Timestamp Representation Unlike traditional methods that rely on an extensive vocabulary of absolute timestamps, our approach employs relative timestamps to represent temporal information. Each timestamp token signifies the time interval from the end of the preceding word to the current word. The core advantage of this method is its superior generalization and compact vocabulary. By learning a constrained set of time intervals, the model can cumulatively represent temporal information for audio of any length, eliminating the need to define a unique token for every possible absolute time point. 2.2. LLMs-based Architecture and Fine-Tuning Our model architecture is built upon a pre-trained large speech model, which consists of an encoder for extracting speech features and a decoder for sequence generation. As shown in Figure 2 (right), to effectively adapt our model for the new task of temporal alignment, we employ a hybrid fine-tuning strategy that combines full parameter updates for new components with parameter-efficient fine-tuning for backbone modules. We apply full-parameter fine-tuning to the newly introduced timestamp embedding and the Language Model (LM) head. These modules are directly responsible for mapping the new timestamp tokens to a continuous vector space and generating them in the output sequence, respectively. Since their initial weights are ran- dom, a comprehensive update of all parameters is necessary for them to effectively learn and represent the new temporal-augmented vocabulary from scratch. For the architecture of the backbone model, specifically the de- coder layers, we utilize LoRA (Low-Rank Adaptation) [12]. This strategic choice is driven by both computational efficiency and the need for targeted adaptation. Instead of updating all parameters, LoRA freezes the original weights and injects small, trainable low- rank matrices. This approach drastically reduces the number of train- able parameters, leading to significant savings in computational re- sources. Furthermore, by fine-tuning the decoder with LoRA, the model can efficiently learn to insert corresponding temporal markers when generating specific text content, without interfering with the powerful, pre-trained understanding of speech features. This hybrid approach ensures computational efficiency while achieving precise control over the model’s generative behavior for robust and accurate temporal alignment. 2.3. Training Strategies for Temporal Alignment To ensure that our model generates accurate and robust timestamps, we designed two key training strategies: 2.3.1. Joint Temporal Alignment Loss Our training objective is to minimize a joint loss function composed of two components, which balances the weights between text tran- scription and timestamp generation: L total = L text + λL timestamp where L text is a standard cross-entropy loss measuring the discrep- ancy between the model’s generated text tokens and the ground-truth tokens. Similarly, L timestamp is a cross-entropy loss that evaluates the prediction accuracy of the discrete timestamp tokens. The hyperparameter λ serves as a weighting factor between the two loss components. We employ a dynamic weighting strategy: in the early training stages, we set a smaller weight (λ = 1) to allow the model to prioritize learning the speech-to-text mapping. As training progresses, we gradually increase the value of λ to encourage more precise temporal alignment. Specifically, for each training epoch, we increment the value of λ by 1. 2.3.2. Timestamp Masking for Regularization To mitigate overfitting to ground-truth during training, we propose a timestamp masking technique. The core idea involves randomly re- placing a subset of timestamp tokens with a [MASK] token. During Table 1. Performance comparison of different models on timestamp prediction. Precision and Recall are in percentages, and the average timestamp difference is in milliseconds. ’-’ denotes the model does not support Chinese speech. ModelTolerance (ms) LibrispeechWenet-Meeting P (%) R (%) Avg. Diff (ms) P(%) R (%) Avg. Diff (ms) Qwen2-Audio [11] 800.000.001031.2360.8156.0659.03 1600.010.001031.1876.0770.1349.11 2400.160.051030.4282.3075.8743.07 WhisperTimestamed [13] 802.702.36131.923.541.51164.09 16010.098.83120.5224.3010.38134.03 24019.9017.42101.1443.9718.79102.48 SenseVoiceSmall [14] 801.101.10176.3828.5227.9680.80 1605.205.19166.7171.4069.9945.26 24012.8812.86148.5982.8281.1932.47 Canary [15] 8035.5135.27444.30--- 16071.2370.74268.18--- 24084.5783.99200.47--- Absolute Timestamp 8012.305.51159.0545.1442.52186.73 16017.037.63156.5862.9959.34168.29 24021.449.61152.5172.4168.22152.17 Relative Timestamp (Ours) 8040.4938.44145.7261.0760.6255.80 16077.7475.24139.2178.0475.4642.45 24083.6582.64127.9191.1386.8830.34 training, the previous timestamp token is masked, the model cannot reference its ground-truth value, and must instead infer the current timestamp from speech content and historical tokens, thereby en- hancing robustness to inaccurate annotations. Similar to BERT’s masked language modeling, this random masking encourages the model to leverage varying input components across iterations, pre- venting over-reliance on specific timestamps. 3. EXPERIMENTS 3.1. Experimental Setup Datasets. We conduct experiments on five benchmark datasets: AISHELL-1: A Mandarin corpus with∼178 hours of speech from 400 speakers [16]. AISHELL-2: A larger Mandarin dataset con- taining∼1,000 hours from 1,991 speakers in diverse acoustic envi- ronments [17]. Wenet Meeting: A multi-domain Mandarin corpus consisting of 10000+ hours of high-quality labeled speech [18]. LibriSpeech: An English read-speech corpus with ∼1,000 hours of audiobooks, widely adopted for ASR evaluation [19]. Com- mon Voice (English subset): A crowd-sourced corpus with diverse accents and recording conditions [20]. Baseline Models. We compare our proposed method against several state-of-the-art models. Qwen2-Audio: A large language model with integrated audio understanding capabilities, trained us- ing a large amount of diverse data and multiple training method- ologies [11]. WhisperTimestamped: This model utilizes Whisper- large-v3, a robust, multilingual ASR model trained on approximately 680,000 hours of weakly supervised audio data [13]. SenseVoice- Small: A lightweight and efficient ASR system, which is trained on over 400,000 hours of data and supports more than 50 languages [14]. Canary: Employs a data-driven approach to enable word-level timestamp prediction and supports multiple languages other than Chinese [15]. Metric. For timestamp prediction, we introduce the concept of time tolerance. Due to the inherent ambiguity of speech bound- aries, a predicted timestamp is considered correct if its difference from the ground-truth timestamp falls within a predefined tolerance [21, 22, 23]. We use Precision and Recall to evaluate the accuracy of these timestamps: Precision measures the proportion of correctly predicted timestamps among all predicted timestamps. Recall mea- sures the proportion of correctly predicted timestamps among all ground-truth timestamps. Additionally, we use the Average Time Difference to quantify the temporal accuracy, defined as: Avg.Diff = P N i=1 |timestamp i gt − timestamp i pred | N For ASR analysis, we use Word Error Rate (WER) as the pri- mary metric [24]. Implementation Details. Our training methodology extends the FireRedASR-LLM’s [25] architecture to support timestamp outputs. This model is built upon a conformer audio encoder and the Qwen2- 7B-Instruct backbone LLM. Crucially, the native model cannot gen- erate timestamps. The model is trained on a cluster of 24 x Ascend 910B (64G) NPUs for 7k steps, with each NPU handling a batch duration of 500 seconds. Setting λ as 1 increases by 1 for each epoch, and empiri- cally setting masking probability as 10% from the second epoch. For training, we use the AdamW optimizer with a low learning rate of 5× 10 −6 and a WarmupCosineLR scheduler to ensure stable con- vergence. The entire process utilizes bf16 mixed-precision training for enhanced efficiency. To enable timestamp generation, we incor- porate new tokens into the LLM’s vocabulary: Absolute Timestamp Pattern: We add tokens from <|0.00|> to <|30.00|>, repre- senting a time range of 0 to 30 seconds. Relative Timestamp Pattern: We add tokens from <|0.00|> to <|5.00|>, representing time differences up to 5 seconds. Unlike traditional methods that require a special instruction token <|timestamp|> Our model is naturally prompted for this task by a simple, intuitive command like ”Speech to text with timestamp.” Our training regimen was confined to a pair of widely-used datasets: the AISHELL-2 corpus [17] for Mandarin and the English subset of the Common Voice dataset [20]. Table 2. Ablation study on timestamp prediction performance at a tolerance of 240 ms (WER % and Precision/Recall) across different model configurations. Configuration AISHELL-2 iOS (Chinese)Common Voice (English) WER (%)PrecisionRecallWER (%)PrecisionRecall Absolute Timestamp2.870.95440.954616.410.78610.7838 - TS Loss2.960.95060.944718.630.74280.7407 Relative Timestamp2.150.97630.963411.630.87700.8546 - TS Loss2.310.97150.954812.660.87120.8463 - Timestamp Masking2.560.96580.957814.470.80670.7853 3.2. Main Results Table 1 compares the timestamp prediction performance of various models across the LibriSpeech and Wenet-Meeting datasets. Our Relative Timestamp method achieved the highest Precision and Recall scores across all tolerance levels on both datasets. For instance, at a 240 ms tolerance, our model reached an impressive 91.13% Precision and 86.88% Recall on Wenet-Meeting, substan- tially outperforming all baselines. Crucially, our model also exhib- ited the best temporal accuracy. With an average timestamp differ- ence of just 30.34 ms at a 240 ms tolerance on Wenet-Meeting, our predictions were the closest to the ground truth. In contrast, Canary performs worse than our method at low tol- erance levels but slightly better at high tolerance levels. This dis- crepancy can be attributed to the fact that Canary was trained on the Librispeech dataset, whereas our model was not.Qwen2-Audio failed on LibriSpeech, with Precision and Recall scores near zero, indicating its inability to handle timestamps on this English cor- pus.WhisperTimestamped and SenseVoiceSmall generally delivered lower performance, especially at lower tolerances, with their results falling far behind ours. We also conducted an ASR experiment, and the WER results 3 showed that our model outperforms all baselines on all datasets. In conclusion, this table definitively proves that our Relative Timestamp method sets a new benchmark for timestamp predic- tion. By focusing on relative temporal relationships, our approach achieves superior accuracy and precision, validating its potential for building high-performance speech language models. 3.3. Ablation study Table 2 presents a detailed ablation study on the performance of dif- ferent model configurations. Our Relative Timestamp model consis- tently achieves the best performance across all metrics. The consis- tently higher Precision and Recall scores further confirm our model’s superior ability to not only transcribe words accurately but also to predict their timestamps with greater precision. To dissect our method’s key contributions, we conducted further ablations: Impact of Timestamp Loss (TS Loss): Removing the times- tamp loss term (- TS Loss) results in a performance drop for both baseline and our models, as evidenced by an increase in WER and a decrease in Precision/Recall. This highlights the critical role of TS Loss in guiding the model to learn accurate timestamp prediction, preventing it from solely focusing on the text sequence generation task. Impact of Timestamp Mask: The ablation study on our model reveals a significant performance degradation when the timestamp mask is removed (- Timestamp Mask). The WER increases from 2.15% to 2.56% on AISHELL-2 iOS and from 11.63% to 14.47% on Common Voice. This compellingly demonstrates that the Timestamp Mask is crucial for the training process. It helps the model focus on valid timestamp predictions and filters out noise, ensuring the robustness of our relative timestamp. 3.4. Analysis of ASR As depicted in Table 3, a comprehensive performance compar- ison based on Word Error Rate (WER) is presented across five diverse datasets. Our proposed method, Relative Timestamp, con- sistently demonstrates superior performance across all evaluated datasets. This is visually confirmed by its polygon having the largest area, signifying its robust generalization and leading performance across multilingual and domain-specific tasks.Specifically, our model achieves WERs of 1.26% and 2.15% on the Chinese datasets AISHELL-1 (AS-1) and AISHELL-2 (AS-2), respectively, signifi- cantly outperforming all baselines. On the more challenging Wenet Meeting (Wenet) dataset, our model’s WER of 5.56% establishes a substantial lead. Furthermore, our approach also secures the best results on the English datasets LibriSpeech (Libri) (2.78%) and Common Voice (CV) (11.63%). Table 3. WER (%) comparison. Lower is better (↓). MethodAS-1AS-2WenetLibriCV (CN)(CN)(CN)(EN)(EN) Qwen2 Audio1.623.3821.9943.3274.37 SenseVoiceSmall3.103.877.543.8015.97 WhisperTimestamped12.468.0856.6620.5038.09 Absolute Timestamp1.382.8711.309.1116.41 Ours (Relative Timestamp)1.262.155.562.7811.63 4. CONCLUSION We propose a novel framework aimed at addressing the core chal- lenges of existing speech models in precise temporal sequence understanding. We have gone beyond the simple method of adding timestamp tags to LLMs, instead focusing on endowing the model with fine-grained, aligned multi-modal spatiotemporal understand- ing capabilities, thereby upgrading LLMs from “content under- standing machines” to “temporal-aware content understanding ma- chines”. 5. ACKNOWLEDGEMENTS This was supported by Jiangsu Province Frontier Program Project (BF2025036), Hong Kong RGC grant GRF #15611021, NSFC grant (No. 62376178), and Jiangsu Key Lab of Language Computing. 6. REFERENCES [1] Rotem Rousso, Eyal Cohen, Joseph Keshet, and Eleanor Chodroff, “Tradition or innovation: A comparison of mod- ern asr methods for forced alignment,” arXiv preprint arXiv:2406.19363, 2024. [2] Jing Peng, Yucheng Wang, Yu Xi, Xu Li, Xizhuo Zhang, and Kai Yu, “A survey on speech large language models,” arXiv e-prints, p. arXiv–2410, 2024. [3] Sara Papi, Peidong Wang, Junkun Chen, Jian Xue, Naoyuki Kanda, Jinyu Li, and Yashesh Gaur, “Leveraging timestamp in- formation for serialized joint streaming recognition and trans- lation,” in ICASSP 2024-2024. IEEE, 2024, p. 10381–10385. [4] Sunit Sivasankaran, Eric Sun, Jinyu Li, Yan Huang, and Jing Pan, “Target word activity detector: An approach to obtain asr word boundaries without lexicon,” in ICASSP 2025. IEEE, 2025, p. 1–5. [5] Naoki Makishima, Keita Suzuki, Satoshi Suzuki, Atsushi Ando, and Ryo Masumura,“Joint autoregressive model- ing of end-to-end multi-talker overlapped speech recognition and utterance-level timestamp prediction,” in Proc. INTER- SPEECH, 2023, 2023, p. 2913–2917. [6] Ke Hu, Krishna Puvvada, Elena Rastorgueva, Zhehuai Chen, He Huang, Shuoyang Ding, Kunal Dhawan, Hainan Xu, Ja- gadeesh Balam, and Boris Ginsburg, “Word level timestamp generation for automatic speech recognition and translation,” arXiv preprint arXiv:2505.15646, 2025. [7] Hiroyoshi Yamasaki, J ́ er ˆ ome Louradour, Julie Hunter, and Laurent Pr ́ evot,“Transcribing and aligning conversational speech: A hybrid pipeline applied to french conversations,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, p. 1–6. [8] Tianshu Yu, Zihan Gong, Minghuan Tan, Guhong Chen, and Min Yang, “Unsupervised speech-text word-level alignment with dynamic programming,” in Findings of the Association for Computational Linguistics: NAACL 2025, 2025, p. 2323– 2334. [9] Quanwei Tang, Sophia Yat Mei Lee, Junshuang Wu, Dong Zhang, Shoushan Li, Erik Cambria, and Guodong Zhou, “A comprehensive graph framework for question answering with mode-seeking preference alignment,” in Findings of ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar, Eds., Vienna, Austria, July 2025, p. 21504–21523, Association for Computational Linguistics. [10] Dan Wu, Xincheng Ju, Dong Zhang, Shoushan Li, Erik Cam- bria, and Guodong Zhou, “Emotion across modalities and cul- tures: Multilingual multimodal emotion-cause analysis with memory-inspired framework,” in Proceedings of the 33rd ACM International Conference on Multimedia, 2025, p. 5775– 5783. [11] Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhi- fang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou, “Qwen2-audio technical report,” 2024. [12] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al., “Lora: Low-rank adaptation of large language models.,” ICLR, vol. 1, no. 2, p. 3, 2022. [13] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever,“Robust speech recognition via large-scale weak supervision,” 2022. [14] Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, Xian Shi, Hao Wang, Wen Wang, Yuxuan Wang, Zhangyu Xiao, Zhijie Yan, Yexin Yang, Bin Zhang, Qinglin Zhang, Shiliang Zhang, Nan Zhao, and Siqi Zheng, “Funaudi- ollm: Voice understanding and generation foundation models for natural interaction between humans and llms,” CoRR, vol. abs/2407.04051, 2024. [15] Ke Hu, Krishna Puvvada, Elena Rastorgueva, Zhehuai Chen, He Huang, Shuoyang Ding, Kunal Dhawan, Hainan Xu, Ja- gadeesh Balam, and Boris Ginsburg, “Word Level Timestamp Generation for Automatic Speech Recognition and Transla- tion,” in Interspeech 2025, 2025, p. 2565–2569. [16] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng, “Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline.,” in O-COCOSDA. 2017, p. 1–5, IEEE. [17] Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu, “Aishell- 2: Transforming mandarin asr research into industrial scale,” 2018. [18] Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et al., “Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition,” in ICASSP 2022. IEEE, 2022, p. 6182–6186. [19] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, “Librispeech: An asr corpus based on public do- main audio books,” in ICASSP 2015, April 2015, p. 5206– 5210. [20] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saun- ders, Francis Tyers, and Gregor Weber, “Common voice: A massively-multilingual speech corpus,” in Proceedings of the Twelfth Language Resources and Evaluation Conference, Mar- seille, France, May 2020, p. 4218–4222, European Language Resources Association. [21] Elena Rastorgueva, Vitaly Lavrukhin, and Boris Ginsburg, “Nemo forced aligner and its application to word alignment for subtitle generation,” in Proc. Interspeech, 2023. [22] Seung-Eun Kim, Bronya R Chernyak, Olga Seleznova, Joseph Keshet, Matthew Goldrick, and Ann R Bradlow, “Automatic recognition of second language speech-in-noise,” JASA Ex- press Letters, vol. 4, no. 2, 2024. [23] Yiming Ji, Suyang Zhu, Dong Zhang, and Shoushan Li, “Pathological section staining transferring with tailored metric- based model selection,” in ICASSP 2025. IEEE, 2025, p. 1–5. [24] Hainan Xu, Fei Jia, Somshubra Majumdar, He Huang, Shinji Watanabe, and Boris Ginsburg, “Efficient sequence transduc- tion by jointly predicting tokens and durations,” in Interna- tional Conference on Machine Learning. PMLR, 2023, p. 38462–38484. [25] Kai-Tuo Xu, Feng-Long Xie, Xu Tang, and Yao Hu, “Fir- eredasr: Open-source industrial-grade mandarin speech recog- nition models from encoder-decoder to llm integration,” arXiv preprint arXiv:2501.14350, 2025.