Paper deep dive
Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer
Ivan Mikheev, Viacheslav Vasilev, Anna Dmitrienko, Alexey Letunovskiy, Ivan Kirillov, Kirill Chernyshev, Denis Dimitrov
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/22/2026, 2:48:45 AM
Summary
This paper introduces a method to add voice cloning capabilities to Text-to-Audio-Video (T2AV) diffusion models by adding a single zero-initialized linear layer to the audio backbone. The approach uses reference audio latents prepended to the audio stream and a global speaker embedding modulated via FiLM. The enhanced model, k6a_5b, significantly outperforms five strong TTS baselines in speaker similarity (SECS) on a 674-sample benchmark. Additionally, an audio-only inference mode (k6av_lite) provides a ~30x speed-up while preserving voice-cloning behavior.
Entities (14)
Relation Signals (12)
k6a_5b → evaluatedon → VCTK
confidence 95% · evaluate on a voice-cloning benchmark built from the VCTK corpus
k6a_5b → isbasedon → AV-DiT
confidence 95% · Our backbone is an asymmetric audio–video diffusion transformer (AV-DiT)
k6a_5b → measuredby → SECS
confidence 95% · attains the highest speaker-encoder cosine similarity (SECS)
k6a_5b → outperforms → Qwen3-TTS
confidence 95% · k6a_5b attains the highest speaker-encoder cosine similarity... statistically significantly outperforming every baseline.
k6a_5b → outperforms → NAVA
confidence 95% · statistically significantly outperforming every baseline.
k6a_5b → outperforms → XTTS-V2
confidence 95% · statistically significantly outperforming every baseline.
k6a_5b → outperforms → IndexTTS2
confidence 95% · statistically significantly outperforming every baseline.
k6av_lite → providesspeedup → 30x
confidence 95% · yielding a ~30x speed-up over the full audio-video diffusion loop
k6a_5b → uses →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.
Tags
Links
- Source: https://arxiv.org/abs/2608.15690v1
- Canonical: https://arxiv.org/abs/2608.15690v1
Trouble viewing inline? Open PDF directly →
Full Text
30,058 characters extracted from source content.
Expand or collapse full text
Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer Ivan Mikheev Affiliation: Kandinsky Lab, Moscow, Russia Correspondence to: ivan.mikheev@kandinskylab.ai Viacheslav Vasilev Affiliation: Kandinsky Lab, Moscow, Russia Correspondence to: viacheslav.vasilev@kandinskylab.ai Anna Dmitrienko Affiliation: Kandinsky Lab, Moscow, Russia Alexey Letunovskiy Affiliation: Kandinsky Lab, Moscow, Russia Ivan Kirillov Affiliation: Kandinsky Lab, Moscow, Russia Kirill Chernyshev Affiliation: Kandinsky Lab, Moscow, Russia Denis Dimitrov Affiliation: Kandinsky Lab, Moscow, Russia Abstract Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674674 speaker–text pairs spanning 3030 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ∼ 30× speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour. 1 Introduction Text-to-audio-video (T2AV) diffusion models (13; 12; 7) jointly synthesize a video clip and its soundtrack from a textual prompt. Existing systems can faithfully describe a scene, including the kind of sound that should accompany it, but they offer no control over the identity of the speaker that appears in the synthesized audio. A voice instruction in the prompt is realised by an arbitrary sample from the distribution trained on the data, which limits the use of T2AV models for personalized content creation, dubbing of avatar characters, or audio-visual rendering of a specific person. A parallel line of work in text-to-speech (TTS) has produced voice-cloning systems that copy a target speaker’s timbre from a short reference clip (17; 3). These models, however, are speech-only: they cannot generate the corresponding video, and they typically rely on specialized architectures with a dedicated speaker branch. Equipping an existing T2AV foundation model with the same ability requires either a costly retraining from scratch or a careful architectural surgery that does not destroy the already-learned audio-visual prior. In this paper we present a minimal recipe for adding reference-conditioned voice cloning to pretrained T2AV models. Our central observation is that the two modifications – prepending reference latents to the audio stream and modulating audio signals with a global speaker embedding through FiLM (14) – fit naturally into the asymmetric audio–video DiT architecture. They require only a single new linear layer, which is zero-initialized so that the augmented model starts as a functional copy of the base model. We apply the approach on two our models – k5 (5 B parameters) and k5-lite (0.6 B parameters). We summarize our contributions as follows: • A simple drop-in extension of a T2AV diffusion models that adds voice-cloning conditioning with one zero-initialized linear layer on top of the audio backbone. • A short fine-tuning recipe consisting of a single voice-aware stage on top of the base T2AV backbone, which allows the model to perform voice cloning while preserving the original audio-visual generation quality. • A three-encoder evaluation protocol for speaker similarity on a 674674-sample benchmark of unique-text speaker pairs, on which our model significantly outperforms strong text-to-speech baselines in reference speaker fidelity. • A practical inference variant in which the audio path of the asymmetric AV-DiT is run in isolation, giving a ∼ 30× speed-up while preserving the voice-cloning behaviour of the full AV inference loop. 2 Method 2.1 Base model Figure 1: Decoder transformer block for our AV-DiT architecture. Audio and video streams are fused using cross-attention. Each stream is based on the CrossDiT architecture (1). Our backbone is an asymmetric audio–video diffusion transformer (AV-DiT) with separate video and audio model dimensions (dvd_v = 1792 and dad_a = 896) connected through cross-modal attention inside each fused block (Figure 1). We based the DiT architecture for both video and audio streams on the CrossDiT architecture from the open-source model Kandinsky 5.0 (1). The text condition is shared between the two streams; audio latents are produced by an off-the-shelf neural audio VAE (5) at 44.1 kHz, and video latents by a HunyuanVideo VAE (11). Other details about this architecture can be found in Appendix A.1. We experiment with two checkpoints of this architecture: k5, a 5 B-parameter model trained on a large internal corpus of audio–video scenes, and k5-lite, a 3 B-parameter variant trained on the same data whose audio stream accounts for only ≈0.6≈ 0.6 B parameters. For brevity we call the augmented models, after fine-tuning, k6a_5b (the 5 B checkpoint, full audio–video inference) and k6av_lite (the 3 B checkpoint run in the audio-only inference mode of Section 3.5, which evaluates only its ≈0.6≈ 0.6 B audio sub-network). 2.2 Reference-conditioned voice cloning We expose two complementary reference signals to the audio backbone of the AV-DiT. Prepended reference latents. Given a target audio sequence xa=VAE(atarget)∈ℝTa×dax_a=VAE(a_target) ^T_a× d_a and a reference recording arefa_ref of 1.2–41.2--4 seconds taken from the same scene during training, we encode the reference with the same audio VAE and prepend the resulting latents to the audio stream: x~a=[VAE(aref)∥xa]∈ℝ(Tref+Ta)×dz. x_a\;=\;[\,VAE(a_ref)\,\|\,x_a\,]\;∈\;R^(T_ref+T_a)× d_z. (1) Throughout the 32 fused decoder blocks, the target tokens attend to the reference tokens via the same self-attention mechanism that already exists in the base AV-DiT (Figure 1), so the model acquires reference conditioning without any new attention layers. At inference, the diffusion update is restricted to the target portion of the composite sequence; the reference portion is held fixed so that the model is forced to keep its timbre consistent across all denoising steps. Speaker FiLM. Long-range timbre information that is not easily captured by local self-attention is provided through a global speaker embedding e∈ℝ1024e ^1024 obtained from a frozen Qwen3-TTS speaker encoder (9). A single new linear layer Wfilm:ℝ1024→ℝ2daW_film:R^1024\!→\!R^2d_a maps this embedding to scale and shift coefficients (γ,β)(γ,β) that modulate each audio token of the target portion via FiLM (14): ha←ha⊙(1+γ)+β,(γ,β)=Wfilm(e).h_a\;←\;h_a (1+γ)+β, (γ,β)=W_film(e). (2) Reference tokens are not modulated. The weights and bias of WfilmW_film are initialized to zero, so (γ,β)=(0,0)(γ,β)=(0,0) at the start of fine-tuning and the augmented model produces exactly the same outputs as the base T2AV model on its first forward pass. 2.3 Classifier-free guidance with two directions During training, with independent probabilities of 0.10.1 each we drop the text condition and the reference signal (latents and speaker embedding jointly), giving the model the four configurations required for a three-way split classifier-free guidance at inference (8): ϵ^=ϵ∅+wt(ϵtext−ϵ∅)+wr(ϵfull−ϵtext), ε\;=\; _ +w_t\,( _text- _ )+w_r\,( _full- _text), (3) where ϵ∅ _ refers to unconditional generation, ϵtext _text – text-conditioned generation and ϵfull _full – generation with condition to text and reference audio. The second term controls the prompt adherence strength, and the third controls the reference voice imposition strength. This split lets us trade text fidelity and speaker fidelity at inference time without retraining. 2.4 Training schedule We fine-tune the entire model – WfilmW_film together with the existing audio backbone weights – on top of the base T2AV checkpoint using a single voice-aware stage. We use AdamW optimizer with ×10−51\!×\!10^-5 for the base parameters and ×10−55\!×\!10^-5 for WfilmW_film, and sample the reference window from speech-rich segments of each clip after the target portion. In 10% of training steps, we independently feed unnoised audio and video latents to retain the original AV joint distribution. That is, the model remains a T2AV generator first and a voice-cloner second. Implementation details can be found in Appendix A. 3 Experiments 3.1 Benchmark and metrics We evaluate on a voice-cloning benchmark built from the VCTK corpus (18) at its native 4848 kHz studio quality: 674674 samples spanning 3030 native-English speakers, where each speaker is assigned its own sentences (100%100\% unique texts, 55–1616 words) rather than a shared passage, so that intelligibility and speaker similarity are not inflated by text overlap. Each sample provides one reference clip, three additional enrollment clips of the same speaker, and a held-out target text. For a fair comparison across architectures, we feed each reference to every model in its own native sample rate, apply an identical pre-processing pipeline before all metrics (silence trimming, peak-normalization to −3-3 dBFS, downmixing to 1616 kHz mono), and evaluate all systems at the same 1616 kHz rate so that audio bandwidth cannot bias any metric. As k6a_5b and k6av_lite are diffusion models with no explicit target-length conditioning, we set the generated duration adaptively from the prompt word count (seconds=words/2.6+1.3seconds=words/2.6+1.3, clamped to [4,12][4,12] s); the text-to-speech baselines determine their own output length. Baselines. We compare against five external voice-cloning text-to-speech systems: a 1.71.7 B-parameter Qwen3-TTS 1.7B and its 0.60.6 B counterpart Qwen3-TTS 0.6B (9), the autoregressive XTTS-v2 (2), the auto-regressive IndexTTS2 (19), and the public NAVA checkpoint (10), a 6.36.3 B joint audio–video diffusion model based on an Align-then-Fuse MMDiT architecture with reference timbre control provided by Timbre-in-Context Conditioning. Metrics. We report speaker-encoder cosine similarity (SECS) using three complementary verification networks: ECAPA-TDNN (6), WavLM-base-plus-sv (4), and Resemblyzer / GE2E (16). For each network we compute two similarities that differ only in what the generated utterance is compared against. The vs. reference score is the cosine similarity between the generated embedding and the single reference clip given to the model. The vs. centroid score instead compares the generated embedding against the centroid of four clips of the same speaker (the reference plus three enrollment clips), which reduces the variance of a single-utterance reference and gives a more stable estimate of the target speaker identity. Intelligibility is measured by word and character error rates WER%/CER% of Whisper transcriptions (15), and by WER0%, the fraction of samples transcribed with zero word errors. 3.2 Main results Table 1 reports the comparison on the full 674674-sample benchmark. k6a_5b attains the highest speaker similarity on every one of the six SECS columns, outperforming all six competing systems on both the reference-based and the centroid-based scores across all three verification networks. The lead holds on the two most reliable encoders, WavLM-SV (0.9440.944 vs. reference) and Resemblyzer (0.8660.866), as well as on ECAPA-TDNN (0.7660.766); averaged over the three networks the margin over the strongest external baseline (Qwen3-TTS 0.6B) is 0.0410.041 and is confirmed by a paired Wilcoxon signed-rank test (p<10−89p<10^-89 against each competitor). The Qwen3-TTS and IndexTTS2 baselines achieve very low WER/CER, but do so by re-rendering each prompt with a clean, studio-like voice that is more distant from the actual reference speaker (lower SECS). k6a_5b and k6av_lite make a different trade-off: they faithfully imitate the reference, including its acoustic quirks, at the cost of a higher WER inherited from the underlying T2AV prior. We attribute the stronger speaker fidelity to the generative audio prior of the base T2AV model: rather than mapping text to a normalized voice as autoregressive TTS systems tend to do, the diffusion backbone reconstructs fine-grained acoustic detail of the prepended reference, at the price of occasional hallucinated words that inflate WER on short or difficult prompts. Table 1: English VCTK benchmark (674674 samples per model, 3030 speakers, unique texts). The SECS columns are speaker-encoder cosine similarities (higher is better) for three verification networks (E: ECAPA-TDNN, W: WavLM-SV, R: Resemblyzer), reported both against the single reference clip (vs. reference) and against the centroid of the reference and three enrollment clips (vs. centroid). WER%/CER% are word/character error rates (lower is better) and WER0% is the fraction of samples transcribed with zero word errors (higher is better). Best value per column in bold, second best underlined. SECS vs. reference SECS vs. centroid Model E W R E W R WER% CER% WER0% k6a_5b (ours) 0.766 0.944 0.866 0.770 0.951 0.878 5.76 5.21 86.6 Qwen3-TTS 0.6B 0.678 0.936 0.840 0.711 0.946 0.864 00.81 00.36 95.7 Qwen3-TTS 1.7B 0.674 0.938 0.839 0.709 0.951 0.864 00.74 00.17 95.3 IndexTTS2 0.695 0.885 0.854 0.693 0.878 0.867 00.96 00.22 93.2 XTTS-v2 0.575 0.923 0.816 0.611 0.940 0.841 01.14 00.32 93.2 k6av_lite (ours) 0.640 0.847 0.826 0.630 0.846 0.843 5.08 3.38 69.4 NAVA 0.624 0.852 0.759 0.625 0.850 0.780 5.82 3.21 69.1 3.3 No regression of the base audio model We verify that adding voice-cloning conditioning does not degrade the reference-free generation ability of the pretrained audio backbone. We compare the base T2AV checkpoint against itself after our reference-aware fine-tuning, running both models in the reference-free regime (text→ only, no reference clip) on 400400 prompts covering 3030 speakers, with identical seeds and texts. A regression would show up as the fine-tuned model moving away from real speech or losing intelligibility. We measure Fréchet Audio Distance (FAD) against real speech, word/character error rates (WER/CER), the fraction of perfectly transcribed samples (WER0), CLAP text–audio alignment, and UTMOS naturalness. Table 2 reports the relative change of the fine-tuned model with respect to the base model. Rather than regressing, the fine-tuned checkpoint improves on every objective axis: it is substantially closer to the distribution of real speech (FAD), roughly halves the transcription error and more than doubles the fraction of perfectly transcribed samples, aligns better with the prompt (CLAP), and matches perceptual naturalness (UTMOS). We attribute this to the zero-init design of WfilmW_film (Sec. 2.2): when the reference is absent the FiLM path stays close to identity, so the augmented model remains a strict functional superset of the base T2AV generator while benefiting from extra in-domain updates. Table 2: No-regression check: relative change of the base T2AV model after reference-aware fine-tuning, evaluated in the reference-free regime on 400400 prompts (3030 speakers, identical seeds/texts). A positive change is an improvement for ↑ -metrics and a negative change is an improvement for ↓ -metrics; every metric moves in the improving direction. Metric Relative change FAD vs. real speech ↓ −30.6%-30.6\% WER ↓ −46.0%-46.0\% CER ↓ −47.0%-47.0\% WER0 (perfect samples) ↑ +113%+113\% CLAP text–audio ↑ +29.4%+29.4\% UTMOS naturalness ↑ +0.6%+0.6\% 3.4 Human side-by-side study In addition to the objective no-regression check (Sec. 3.3, Table 2), we run a human side-by-side (SBS) study to confirm that reference-aware fine-tuning does not degrade the perceived quality of reference-free generation. We compare the base audio model against itself after our reference-aware fine-tuning on 100100 held-out text-to-audio prompts, with both models generating without a reference recording, so that any regression would show up as a lower win rate for the fine-tuned model. Annotators rate each pair on prompt following, technical quality, and speech/aesthetic quality as either a win for one of the two models or a draw. Table 3 shows that the fine-tuned checkpoint scores marginally higher win rates (5252–57%57\%) on all three axes, consistent with the objective results. A smaller SBS study on the full T2AV pipeline (reference-free) shows the same pattern, so the “strict superset” property carries over from audio-only to full multimodal. Table 3: SBS human evaluation on 100100 held-out text-to-audio prompts, base audio model vs. the same model after reference-aware fine-tuning (both generate without a reference). Numbers are percentages of pair-wise judgements per axis. Prompt Technical Speech/Aesth. follow. quality quality Base wins 09 08 11 Fine-tuned wins 14 23 17 Both good 78 75 78 Both bad 09 04 04 Base win rate 48 43 47 Fine-tuned win rate 52 57 53 3.5 Fast Audio-only Inference from an AV Checkpoint The asymmetry of the AV-DiT lets us run the audio path in isolation at inference without changing any weights, by short-circuiting the video sub-block of each fused decoder block and skipping the video VAE decoder (details in Appendix B). This drops the cost of one diffusion step from a 3.183.18 B-parameter joint pass to an effective ∼ 0.580.58 B audio-only pass, a ∼ 30×30× speed-up. The speed-up is not bought at the cost of voice cloning: in Table 1, k6av_lite preserves the reference timbre well, at a modest SECS drop of ≈0.09≈ 0.09 vs. the full k6a_5b, which we read as the video signal acting as a mild regularizer on the audio path. In practice, k6av_lite enables quick auditioning of a reference across many prompts before committing to full k6a_5b rendering. 4 Conclusion We have shown that T2AV diffusion models can be turned into voice-cloning systems with a remarkably small architectural change: prepending the reference latents to the audio stream and adding a single zero-initialized linear layer that drives speaker FiLM on the audio backbone, trained in a short voice-aware stage. On a 674674-sample benchmark our k6a_5b obtains the highest speaker similarity across three verification networks, significantly outperforming every baseline, while the audio-only variant k6av_lite preserves this behavior at a ∼ 30× lower inference cost and the original text-to-audio-video functionality stays intact. Two extensions are natural: the prepend+FiLM recipe should transfer to any asymmetric AV-DiT paired with a frozen speaker encoder, and the WER gap to dedicated TTS suggests adding a small text-fidelity loss (e.g. CTC against the prompt) without disturbing the speaker conditioning. References Arkhipkin et al. (2026) V. Arkhipkin, V. Korviakov, N. Gerasimenko, D. Parkhomenko, V. Vasilev, A. Letunovskiy, N. Vaulin, M. Kovaleva, I. Kirillov, L. Novitskiy, D. Koposov, N. Kiselev, A. Varlamov, D. Mikhailov, V. Polovnikov, A. Shutkin, J. Agafonova, I. Vasiliev, A. Kargapoltseva, A. Dmitrienko, A. Maltseva, A. Averchenkova, O. Kim, T. Nikulina, and D. Dimitrov Kandinsky 5.0: a family of foundation models for image and video generation. External Links: 2511.14993, Link Cited by: Figure 1, Figure 1, §2.1. Casanova et al. (2024) E. Casanova, K. Davis, E. Gölge, et al. XTTS: a massively multilingual zero-shot text-to-speech model. In Interspeech, Cited by: §3.1. Casanova et al. (2022) E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Gölge, and M. A. Ponti YourTTS: towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In Proceedings of the 39th International Conference on Machine Learning, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, p. 2709–2720. External Links: Link Cited by: §1. Chen et al. (2022) S. Chen, C. Wang, Z. Chen, et al. WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Signal Process.. Cited by: §3.1. Cheng et al. (2025) H. K. Cheng, M. Ishii, A. Hayakawa, T. Shibuya, A. Schwing, and Y. Mitsufuji MMAudio: taming multimodal joint training for high-quality video-to-audio synthesis. External Links: 2412.15322, Link Cited by: §2.1. Desplanques et al. (2020) B. Desplanques, J. Thienpondt, and K. Demuynck ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In Interspeech, Cited by: §3.1. HaCohen et al. (2026) Y. HaCohen, B. Brazowski, N. Chiprut, Y. Bitterman, A. Kvochko, A. Berkowitz, D. Shalem, D. Lifschitz, D. Moshe, E. Porat, E. Richardson, G. Shiran, I. Chachy, J. Chetboun, M. Finkelson, M. Kupchick, N. Zabari, N. Guetta, N. Kotler, O. Bibi, O. Gordon, P. Panet, R. Benita, S. Armon, V. Kulikov, Y. Inger, Y. Shiftan, Z. Melumian, and Z. Farbman LTX-2: efficient joint audio-visual foundation model. External Links: 2601.03233, Link Cited by: §1. Ho and Salimans (2021) J. Ho and T. Salimans Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, External Links: Link Cited by: §2.3. Hu et al. (2026) H. Hu, X. Zhu, T. He, D. Guo, B. Zhang, X. Wang, Z. Guo, Z. Jiang, H. Hao, Z. Guo, X. Zhang, P. Zhang, B. Yang, J. Xu, J. Zhou, and J. Lin Qwen3-tts technical report. External Links: 2601.15621, Link Cited by: §A.4, §2.2, §3.1. Ji et al. (2026) L. Ji, G. Wang, X. Wei, C. Yang, X. Liu, Z. Zhang, S. Wang, Y. Sun, and J. He Native audio-visual alignment for generation. External Links: 2605.30073, Link Cited by: §3.1. Kong et al. (2025) W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, K. Wu, Q. Lin, J. Yuan, Y. Long, A. Wang, A. Wang, C. Li, D. Huang, F. Yang, H. Tan, H. Wang, J. Song, J. Bai, J. Wu, J. Xue, J. Wang, K. Wang, M. Liu, P. Li, S. Li, W. Wang, W. Yu, X. Deng, Y. Li, Y. Chen, Y. Cui, Y. Peng, Z. Yu, Z. He, Z. Xu, Z. Zhou, Z. Xu, Y. Tao, Q. Lu, S. Liu, D. Zhou, H. Wang, Y. Yang, D. Wang, Y. Liu, J. Jiang, and C. Zhong HunyuanVideo: a systematic framework for large video generative models. External Links: 2412.03603, Link Cited by: §2.1. Li et al. (2025) Y. Li, H. Si, F. Landi, P. O. Gallegos, I. Koutsoumpas, O. R. C. Vazquez, R. Fu, Q. Guo, X. Jin, S. Liu, and M. Song 3MDiT: unified tri-modal diffusion transformer for text-driven synchronized audio-video generation. External Links: 2511.21780, Link Cited by: §1. Liu et al. (2024) H. Liu, G. L. Lan, X. Mei, Z. Ni, A. Kumar, V. Nagaraja, W. Wang, M. D. Plumbley, Y. Shi, and V. Chandra SyncFlow: toward temporally aligned joint audio-video generation from text. External Links: 2412.15220, Link Cited by: §1. Perez et al. (2018) E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville FiLM: visual reasoning with a general conditioning layer. In AAAI, AAAI’18/IAAI’18/EAAI’18. External Links: ISBN 978-1-57735-800-8 Cited by: §A.4, §1, §2.2. Radford et al. (2023) A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever Robust speech recognition via large-scale weak supervision. In ICML, Cited by: §3.1. Wan et al. (2018) L. Wan, Q. Wang, A. Papir, and I. L. Moreno Generalized end-to-end loss for speaker verification. In ICASSP, Cited by: §3.1. Wang et al. (2023) C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei Neural codec language models are zero-shot text to speech synthesizers. External Links: 2301.02111, Link Cited by: §1. Yamagishi et al. (2019) J. Yamagishi, C. Veaux, and K. MacDonald CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92). University of Edinburgh. The Centre for Speech Technology Research (CSTR). External Links: Document, Link Cited by: §3.1. Zhou et al. (2025) S. Zhou, Y. Zhou, Y. He, X. Zhou, J. Wang, W. Deng, and J. Shu IndexTTS2: a breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech. External Links: 2506.21619, Link Cited by: §3.1. Appendix A Implementation details A.1 Base architectures k5 and k5-lite share the same asymmetric AV-DiT backbone described in Section 2.1 (Figure 1). Both consist of 32 fused decoder blocks, with a video stream of width dv=1792d_v=1792 (heads of dimension 64) and an audio stream of width da=896d_a=896 (heads of dimension 64). A 2-block text encoder consumes the prompt and feeds it to both streams. k5 uses the full 32 fused blocks, k5-lite a reduced configuration with the same block topology but smaller text and output layers, totaling ≈3≈ 3 B parameters, of which the audio stream (the part evaluated in the audio-only inference mode) accounts for only ≈0.6≈ 0.6 B. A.2 Reference window sampling At training time, given a training clip, we sample the target audio from the first slice of the parquet record (typically 5 seconds of audio aligned with the video latents). The reference window is drawn from the remaining audio of the same clip, with a buffer of 0.50.5 s between target and reference to avoid trivial copy. Speech references are sampled with duration in [1.2, 2.5][1.2,\,2.5] s from a 12 s search window, non-speech references with duration in [1.5, 3.5][1.5,\,3.5] s from a 3.5 s search window. A minimum-RMS and a maximum-clipping-ratio quality filter discards silent or saturated references. A.3 Hyperparameters • Optimizer: AdamW, (β1,β2)=(0.9,0.95)( _1, _2)=(0.9,0.95), weight decay 10−310^-3, max grad norm 1.01.0. • Base LR: 10−510^-5; LR multiplier for the new linear layer WfilmW_film and its biases: 5×5×. • Warmup: 80008000 steps, constant afterwards. • Reference-conditioning drop probability: 0.10.1; text-conditioning drop probability: 0.10.1. • Modality-clean schedule: 10%10\% of steps run with the video latent clamped to zero (audio-only loss) and 10%10\% with the audio latent clamped to zero (video-only loss), to retain joint AV behavior throughout fine-tuning. • Default inference: 5050 diffusion steps, split CFG with wt=5w_t=5 and wr=4w_r=4, fixed seed. A.4 Speaker encoder The frozen speaker encoder used to compute the FiLM (14) embedding is a publicly released Qwen3-TTS speaker network (9) operating on 2424 kHz audio. We run it once per training sample on CPU in a background worker and only transmit the resulting 1024-dim vector to the training step. Appendix B Audio-only inference details Because the AV-DiT is asymmetric, the same fine-tuned checkpoint can be evaluated with the video stream disabled, without any weight changes (Sec. 3.5). Concretely, inside each of the 3232 fused decoder blocks the video sub-block is short-circuited: its 17921792-dim video self-attention and FiLM modulation are never computed, while the audio sub-block, the speaker FiLM layer, and the audio classifier-free-guidance loop keep operating exactly as in training. The video VAE decoder is likewise skipped. This reduces the per-step forward from a 3.183.18 B-parameter joint-modality pass to an effective ∼ 0.580.58 B audio-only pass, giving the ∼ 30×30× speed-up reported in the main text. The audio-only variant k6av_lite trades a small amount of speaker similarity (≈0.09≈ 0.09 in SECS averaged over the three encoders) for this speed-up; we attribute the gap to the loss of the video stream, which otherwise acts as a mild regularizer on the audio path. Appendix C Additional analysis We carried out a more extensive factor study to identify which inference-time and data-time choices matter most for voice cloning quality. The following findings are based on a 82-run sweep across a held-out subset of the benchmark. Reference length. Speaker similarity grows monotonically with reference length from ∼1 1 s up to a plateau at 44 s, beyond which returns are negligible. Below 22 s the Resemblyzer SECS drops sharply, suggesting that the speaker encoder rather than the diffusion model is the bottleneck on very short references. Reference-guidance weight. The optimum of the second CFG weight wrw_r (Eq. 3) is in the range 44–66. Below 33 the reference is under-imposed; above 77 we observe a small over-smoothing of the audio that reduces UTMOS. Number of diffusion steps. The mean SECS is essentially saturated by 3030 steps; moving from 3030 to 6060 adds <0.02<0.02 in mean SECS and is not worth the doubled wall-clock cost for routine use. Language matching. Generating from a prompt whose language differs from the reference language is possible but lowers the mean SECS by about 0.050.05 on average. We therefore recommend matching prompt and reference languages whenever possible. Reference pre-processing. A simple de-noise / VAD pre-processing of the reference recording is helpful on noisy or amateur captures but is harmful on already-clean studio references, where it can slightly hurt SECS. A safe default is to skip pre-processing when the reference already has UTMOS ≳3 3.