Paper deep dive
LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs
Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate this by reducing the effective frame rate using a compression step to merge multiple frames together. However, most prior methods either operate on single-codebook codecs or apply a single compression step before multi-layer quantization. This forces all quantization layers to share the same segmentation boundaries, despite the residual embeddings at different quantization layers exhibiting different rates of change over time. We propose LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries. To use LACE tokens in downstream text-to-speech (TTS), we further introduce union alignment and boundary anchor mechanisms to make durations consistent across layers while preserving compression benefits. Experiments on LibriTTS show that LACE offers a better rate-quality tradeoff than prior dynamic frame rate methods on the reconstruction task and improves TTS inference efficiency while maintaining competitive synthesis quality. Our code is released as part of the ESPnet3 codec recipe.
Tags
Links
- Source: https://arxiv.org/abs/2609.17509v1
- Canonical: https://arxiv.org/abs/2609.17509v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
71,435 characters extracted from source content.
Expand or collapse full text
LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe Affiliation: Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA ttrachu, scornell, wc4, swatanab@andrew.cmu.edu Abstract Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate this by reducing the effective frame rate using a compression step to merge multiple frames together. However, most prior methods either operate on single-codebook codecs or apply a single compression step before multi-layer quantization. This forces all quantization layers to share the same segmentation boundaries, despite the residual embeddings at different quantization layers exhibiting different rates of change over time. We propose LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries. To use LACE tokens in downstream text-to-speech (TTS), we further introduce union alignment and boundary anchor mechanisms to make durations consistent across layers while preserving compression benefits. Experiments on LibriTTS show that LACE offers a better rate-quality tradeoff than prior dynamic frame rate methods on the reconstruction task and improves TTS inference efficiency while maintaining competitive synthesis quality. Our code is released as part of the ESPnet3 codec recipe. Index Terms: Audio Codec, Text-to-Speech, Dynamic Frame Rate Codec, Compression I Introduction Neural audio codecs have emerged as an important component for speech language modeling. When used as speech tokenizers, they convert continuous speech into discrete codes, allowing speech generation [1, 2, 3] and speech language models [4, 5, 6] to be formulated as next-token prediction problems similar to those used in large language models. However, general neural audio codecs [7, 8, 9] typically operate at high frame rates, producing long sequences of discrete codes. Long sequences of discrete codes introduce two limitations. First, long sequences increase the computational cost, as the complexity of transformer-based models tends to scale with sequence length [10]. Second, the length mismatch between speech and text tokens can degrade the performance of speech language models, since substantially longer speech sequences make syntactic and semantic modeling more difficult, as shown in [11]. These limitations motivate dynamic frame rate codecs [12, 13, 14, 15], which represent speech as discrete codes paired with durations. A compression model produces segmentation boundaries, which are frame positions where the frame-level embeddings are partitioned. These boundaries divide the embeddings into variable-length segments, and each segment is represented by one code and its duration. Since each code now corresponds to a segment rather than a single frame, the frame rate is measured as the number of segments per second instead of the number of frames per second. We refer to this as the effective frame rate. For example, if one second of audio with 75 frame-level embeddings is compressed into 25 segments, the effective frame rate is 25 Hz, instead of 75 Hz. By assigning longer durations to regions with lower information density, dynamic frame rate codecs use fewer codes to represent speech than general codecs. Most existing dynamic frame rate codecs either operate on single-codebook codec models [15, 13], or apply a single compression step before multi-layer quantization [14, 16, 17]. In both cases, all quantization layers are constrained to share the same segmentation boundaries. This constraint may be suboptimal for multi-codebook codecs due to the hierarchical nature of their quantization process. In residual vector quantization (RVQ) [18], the first layer quantizes the input directly, while each subsequent layer quantizes the residual embeddings from the previous layer. Consequently, earlier layers capture the main signal structure, whereas deeper layers encode finer details. As a result, representations at different layers may change at different rates over time, suggesting that each layer requires its own segmentation boundaries. This intuition is supported by our analysis in Section V-C1, which shows that deeper-layer residual embeddings change more rapidly than earlier-layer residual embeddings. This is further supported by the findings of SNAC [19], which uses different frame rates across quantization layers, although it is not a dynamic frame rate codec because its frame rates are fixed. Motivated by these findings, we propose LACE (Layer-Adaptive Codec Encoding), a dynamic frame rate codec that independently applies a compression step at each quantization layer. This allows each layer to choose its own segmentation boundaries. To adapt LACE to the common speech language model framework, we propose union alignment, which constructs shared segmentation boundaries by taking the union across layers. Since union alignment can increase the effective frame rate, we further propose boundary anchor, which constrains deeper layers to reuse boundaries introduced by earlier layers. Our contributions are summarized as follows: • We propose LACE, a dynamic frame rate codec that applies an independent compression step at each quantization layer. • We propose union alignment and boundary anchor to resolve duration inconsistency across layers, enabling LACE tokens to be used in downstream TTS training. • We provide a theoretical analysis showing that LACE achieves a lower upper bound on the expected quantization error than single compression. • Experiments on LibriTTS [20] show that LACE offers a better rate-quality tradeoff than prior dynamic frame rate methods on the reconstruction task and improves TTS inference efficiency while maintaining competitive synthesis quality. I Background I-A Neural Audio Codec Preliminaries A neural audio codec consists of an encoder, a quantization module, and a decoder. The encoder maps the raw waveform into frame-level embeddings ∈ℝT×DH ^T× D, where T is the number of frames and D is the embedding dimension. The quantization module discretizes H into frame-level codes ∈[1,V]T×CZ∈[1,V]^T× C, where each frame is represented by C codes from a codebook of size V. The decoder then reconstructs the waveform from these codes. A commonly used quantization method is RVQ. RVQ discretizes H using C quantization layers, where each layer l contains a codebook l∈ℝV×DQ^l ^V× D. Here, kl∈ℝDq_k^l ^D denotes the k-th codeword of lQ^l. RVQ performs quantization sequentially across quantization layers. At layer l, the nearest codeword is selected from the corresponding codebook lQ^l as zil=argmink∈[1,V]‖il−kl‖22z_i^l= _k∈[1,V]\|r_i^l-q_k^l\|_2^2, where zil∈[1,V]z_i^l∈[1,V] and il∈ℝDr_i^l ^D denote a code and a residual embedding at frame i and quantization layer l, respectively. The residual embedding is updated recursively as il+1=il−zillr_i^l+1=r_i^l-q_z_i^l^l, with i1=ir_i^1=h_i, where i∈ℝDh_i ^D is the i-th frame-level embedding in H. We define the stacked frame-level residual embeddings of layer l as l∈ℝT×DR^l ^T× D. I-B Compression in Dynamic Frame-Rate Codecs The encoder of a neural audio codec produces frame-level embeddings H at a fixed codec frame rate, yielding T frames. A compression model groups these T frames into M variable-length segments. Each segment is represented by C codes and an integer duration. Formally, the compression model determines segmentation boundaries ℬ=b0,b1,…,bMB=\b_0,b_1,…,b_M\, a sorted set with b0=0b_0=0, bM=Tb_M=T, and bm∈[1,T−1]b_m∈[1,T-1]. The m-th segment spans the frame interval (bm−1,bm](b_m-1,b_m]. Some methods control M explicitly using a target compression rate γ∈(0,1]γ∈(0,1], yielding M=⌊γT⌋M= γ T , while others determine M implicitly through a threshold τ. The frame-level embeddings within each segment are averaged to produce segment-level embeddings ¯∈ℝM×D H ^M× D. The segment-level embeddings are paired with the durations ∈ℕMd ^M, where the duration of the m-th segment dmd_m is bm−bm−1b_m-b_m-1. The segment-level embeddings ¯ H are quantized using RVQ, as described in Section I-A, producing segment-level codes ¯∈[1,V]M×C Z∈[1,V]^M× C. This (code, duration) format resembles the classical run-length encoding scheme [21]. During reconstruction, ¯ Z is repeated according to d to produce frame-level codes ^∈[1,V]T×C Z∈[1,V]^T× C. This restores the frame rate expected by the decoder. The three compression models below share this pipeline and differ only in how they derive ℬB. Dynamic Programming (DP)-Based Compression. CodecSlime [15] determines ℬB by minimizing the total reconstruction error between the original frame-level embeddings H and the segment-level embeddings ¯ H. For a segment of s consecutive frames ending at frame j, the segment-level embedding is defined as the mean of the frames within the segment: ¯j,s=1s∑t=j−s+1jt h_j,s= 1s _t=j-s+1^jh_t. The reconstruction loss for this segment is ℓ(j,s)=∑t=j−s+1j‖t−¯j,s‖22 (j,s)= _t=j-s+1^j\|h_t- h_j,s\|_2^2. Each segment is constrained to contain at most U frames. Let f[j,n]f[j,n] denote the minimum total reconstruction loss when the first j frames are compressed into n segments. The DP objective is defined recursively as: f[j,n]=min1≤s≤Uf[j−s,n−1]+ℓ(j,s), f[j,n]= _1≤ s≤ U \f[j-s,\,n-1]+ (j,s) \, (1) with f[0,0]=0f[0,0]=0. At each state (j,n)(j,n), we record the segment length s∗[j,n]s^*[j,n] that achieves the minimum in Equation (1). Given the number of segments M, we start from the last boundary bM=Tb_M=T and recover each preceding boundary as bn−1=bn−s∗[bn,n]b_n-1=b_n-s^*[b_n,n] for n=M,M−1,…,1n=M,M-1,…,1, which terminates at b0=0b_0=0 and yields the segmentation boundaries ℬB. Cosine Similarity-Based Compression. FlexiCodec [14] determines ℬB by thresholding the similarity between adjacent frame-level embeddings. Let σt=cos(t,t+1) _t= (h_t,h_t+1) denote the similarity between frames t and t+1t+1. A boundary is placed wherever the similarity drops below a threshold τ, yielding ℬ=0,T∪t:σt<τB=\0,T\∪\\,t: _t<τ\,\. The number of segments M is therefore not fixed in advance but controlled implicitly by the threshold τ, unlike the DP-based compression model. Density Peak Clustering-Based Compression. VARSTok [13] derives ℬB using density peak clustering. For each frame t, a local density ρt _t is computed from its κ nearest neighbors, and a peak distance δt _t is the temporal distance to the nearest frame with higher density: ρt=1κ∑j∈KNN(t)ϕ(t,j),δt=minj:ρj>ρt|j−t|, _t= 1κ _j (t)φ(h_t,h_j), _t= _j: _j> _t|j-t|, (2) where ϕ(i,j)=(1+⟨i,j⟩)/2φ(h_i,h_j)=(1+ _i,h_j )/2 and KNN(t)KNN(t) denotes the set of κ nearest neighbors of frame t. Frames with a high peak score pt=ρtδtp_t= _t _t are selected as segment centers. Each center is greedily expanded to adjacent unassigned frames that satisfy ϕ(i∗,t)−βpt>τφ(h_i^*,h_t)-β p_t>τ, up to a maximum of U frames. This process repeats until all frames are assigned to segments. The segment endpoints are then sorted to form ℬB. The number of segments M is controlled implicitly by τ. We use κ=5κ=5, β=0.2β=0.2, and U=4U=4, following the original paper. All of these prior works force all quantization layers to share the same segmentation boundaries. In contrast, LACE applies an independent compression step at each quantization layer, enabling layer-specific segmentation boundaries. Because LACE is agnostic to the choice of compression model, any of these methods can be used as the compression step within the proposed framework. I Method We propose LACE, a dynamic frame rate codec that applies an independent compression step at each quantization layer, allowing each layer to choose its own segmentation boundaries. We then introduce two mechanisms, union alignment and boundary anchor, to make the duration consistent for downstream TTS training. I-A Layer-Wise Compression Fig. 1: Overview of the LACE codec. At each quantization layer, the residual embeddings are independently compressed before being passed to the quantization layer. Frames with the same color belong to the same segment. Unlike prior dynamic frame rate codecs that apply a single compression step and share segmentation boundaries across quantization layers, LACE performs layer-wise compression on the residual embeddings. At layer l, we apply the compression step described in Section I-B to lR^l. This produces segment-level residual embeddings ¯l∈ℝMl×D R^l ^M^l× D, durations l∈ℕMld^l ^M^l, and layer-specific segmentation boundaries ℬlB^l, where MlM^l is the number of segments at layer l. The segment-level residual embeddings ¯l R^l are then passed to quantization layer l, yielding segment-level codes ¯l∈[1,V]Ml Z^l∈[1,V]^M^l. To compute the residual embeddings for the next layer, we expand ¯l Z^l to frame-level codes ^l∈[1,V]T Z^l∈[1,V]^T by repeating each code according to ld^l. The next residual embedding is then computed as il+1=il−z^illr_i^l+1=r_i^l-q^l_ z^l_i. Unlike the standard RVQ update in Section I-A, z^illq^l_ z^l_i is obtained by quantizing a segment-level residual embedding, whereas zillq^l_z^l_i in the original update is obtained from a frame-level residual embedding. We use the frame-level residual embedding ilr_i^l on the right-hand side so that il+1r_i^l+1 captures both the compression error and the quantization error from layer l. Figure 1 illustrates layer-wise compression. I-B Union Alignment Fig. 2: An overview of union alignment. Before alignment (top), quantization layers 1 and 2 have different segmentation boundaries, resulting in inconsistent durations across layers. After alignment (bottom), the union of segmentation boundaries from both layers is applied to all layers. The layer-wise compression produces different segmentation boundaries ℬlB^l for each quantization layer l. As a result, codes from different layers that cover the same time span can have different durations. This is problematic for TTS because the model would need to predict durations for each layer while ensuring that the total duration after upsampling is identical across layers. We address this with union alignment, which constructs shared segmentation boundaries by taking the union across layers: ℬunion=⋃l=1CℬlB^union= _l=1^CB^l. We then re-segment the segment-level codes of every layer using ℬunionB^union. As shown in Figure 2, when a boundary in ℬunionB^union falls inside an existing segment, that segment is split into sub-segments. For example, the boundary b1=2b_1=2 from the first layer splits the first segment in the second layer. The resulting sub-segments keep the same code and receive new corresponding durations. After alignment, all layers share the same segmentation boundaries and durations, at the cost of increasing the effective frame rate. I-C Boundary Anchor Union alignment can increase the effective frame rate because |ℬunion|≥|ℬl||B^union|≥|B^l| for all l. In the worst case, each layer introduces distinct boundaries and |ℬunion||B^union| approaches T, negating the compression benefit. To mitigate this, we introduce a boundary anchor. We choose an anchor layer l∗l^*, where layers l≤l∗l≤ l^* are allowed to introduce new boundaries, while deeper layers l>l∗l>l^* must reuse boundaries from earlier layers: ℬanchor=⋃l=1l∗ℬlB^anchor= _l=1^l^*B^l. By restricting the compression step at layers l>l∗l>l^* to place boundaries only at positions in ℬanchorB^anchor, we limit the growth of |ℬunion||B^union|. Here, l∗l^* is a tunable hyperparameter that trades off between quality and the effective frame rate. For DP-based compression, we implement this restriction with a constrained cost ℓ~(j,s) (j,s), which equals the segment loss ℓ(j,s) (j,s) of Section I-B if both j and j−sj-s lie in ℬanchorB^anchor and +∞+∞ otherwise. We solve the same recursion as Equation (1) with ℓ replaced by ℓ~ , and use the unconstrained DP for layers l≤l∗l≤ l^*. While this formulation is specific to DP-based compression, the boundary anchor itself is model-agnostic. It can be applied to other compression models in Section I-B by restricting its candidate boundaries to ℬanchorB^anchor. I-D Theoretical Analysis Section I-C introduced the anchor layer l∗l^* as a hyperparameter that trades off quality against the effective frame rate. We now analyze the quality side of this tradeoff: how the choice of l∗l^* affects the expected quantization error. In Sections I-B and I-A, the compression step merges the frames before the quantization layer, and the codes are repeated back to the frame level after it. Since all repeated frames within a segment are identical, repeating before or after quantization yields the same result. We therefore combine merging and repetition into a single matrix multiplication llA^lR^l, where lR^l is the frame-level residual embedding at layer l. Concretely, l∈ℝT×TA^l ^T× T is block-diagonal with one block per segment, and the block for a segment of length dmd_m is the dm×dmd_m× d_m matrix with every entry equal to 1/dm1/d_m. Since lA^l is symmetric and idempotent, the compression error l−llR^l-A^lR^l is orthogonal to any frame-level embeddings that use the same segmentation boundaries as layer l. Let ^l∈ℝT×D Q^l ^T× D denote the frame-level quantized embeddings, which stack the selected codewords z^illq^l_ z^l_i, so the residual update of Section I-A becomes l+1=l−^lR^l+1=R^l- Q^l. The quantized embeddings ^l Q^l repeat one codeword across each segment. Subtracting them in the residual update can therefore only change the part of lR^l that is constant within each segment. The compression error, which varies within segments, passes to the next layer untouched. When all layers share segmentation boundaries, the quantized embeddings of every layer are constant within the same segments, so the compression error survives all C quantization layers. It becomes an error that quantization cannot remove. We formalize this intuition under three assumptions: (i) every layer merges at least two frames (Ml<TM^l<T), so the compression error is strictly positive, (i) each quantization layer is well-trained, approximately satisfying the centroid condition [22], so that [‖ll−^l‖F2]≤ϵl[‖ll‖F2]E[\|A^lR^l- Q^l\|_F^2]≤ _l\,E[\|A^lR^l\|_F^2] with ϵmax=suplϵl<1 _ = _l _l<1, and (i) the fraction of expected residual energy captured by the compression step, αl=[‖ll‖F2]/[‖l‖F2] _l=E[\|A^lR^l\|_F^2]/E[\|R^l\|_F^2], is bounded below by αmin=inflαl>0 _ = _l _l>0. Under these assumptions, the expected residual error after all C quantization layers with anchor layer l∗l^* satisfies11 1 The analysis assumes that layers l>l∗l>l^* reuse the segmentation boundaries of layer l∗l^* exactly, a special case of the boundary anchor in Section I-C. [‖C+1‖F2]≤[1−αl∗(1−ϵmaxC−l∗+1)]∏l=1l∗−1λl[‖F2], [\|R^C+1\|_F^2]≤ [1- _l^* (1- _ ^\,C-l^*+1 ) ] _l=1^l^*-1 _l\,E[\|H\|_F^2], (3) where C+1R^C+1 is the residual after the last layer and λl=1−αl(1−ϵl)<1 _l=1- _l(1- _l)<1. As C→∞C→∞, Equation (3) reveals a hierarchy. A single compression, which shares the same segmentation boundaries across layers (l∗=1l^*=1), converges to a constant error floor (1−α1)[‖F2](1- _1)\,E[\|H\|_F^2], which is exactly the compression error of the first layer. The boundary anchor reduces this floor by the factor ∏l=1l∗−1λl<1 _l=1^l^*-1 _l<1. Full layer-wise compression (l∗=Cl^*=C) drives the bound to zero. Therefore, recomputing boundaries at each quantization layer removes the error floor inherent to shared segmentation boundaries. The full derivation is provided in the supplementary material. This bound concerns the codec quantization error only, not downstream TTS, where union alignment (Section I-B) also changes the token sequence. I-E TTS Training with LACE Tokens Unlike standard TTS models [2, 23], a TTS model trained on LACE tokens must also predict the durations d. After union alignment, all C quantization layers share the segmentation boundaries ℬunionB^union and durations d. Therefore, the model predicts durations only for the first quantization layer and applies the predicted durations to all subsequent layers. We use an autoregressive decoder-only Transformer that predicts the segment-level codes ¯ Z in a delay-pattern format [23]. At each generation step for the first-layer code, the model additionally predicts a duration for that code. Duration prediction is formulated as a classification task. We add a duration head alongside the code prediction head and include a learned duration embedding in the input representation, which is added to the code embedding. The model is trained with cross-entropy loss for code prediction and focal loss for duration prediction to address class imbalance, with loss weights of 1 and 3, respectively. At inference, we use nucleus sampling with a top-p value of 0.8 and a temperature of 1.0. TABLE I: Reconstruction performance across different codecs and compression models on LibriTTS test-clean. All baseline results use a single compression step without LACE. “+ LACE” denotes our proposed layer-wise compression. The frame rate column reports the effective frame rate and codec frame rate for codecs with and without compression models, respectively. For threshold-based compression models, we report the average effective frame rate across layers. Baseline rows are re-implemented using the same backbones. The results reported by the cited systems are not directly comparable. Codec Compression Frame Rate Bitrate WER ↓ UTMOS ↑ PESQ ↑ STOI ↑ SpkSim ↑ (Hz) (kbps) (%) Ground Truth – - - 2.02 4.06 - - - DAC [7] – 75 24.00 2.16 3.95 3.59 0.97 0.99 Encodec [9] – 75 24.00 2.09 3.93 3.25 0.96 0.98 SoundStream [8] – 75 24.00 2.41 3.61 2.56 0.93 0.97 Finetuned DAC DP [15] 27 8.69 4.92 2.49 1.61 0.86 0.94 DP + LACE 22.5 8.64 2.22 3.81 3.08 0.95 0.98 Cosine [14] 27 8.69 6.90 1.84 1.29 0.81 0.90 Cosine + LACE 22.5 8.64 2.57 3.55 2.73 0.94 0.97 Density Clustering [13] 27 8.69 4.25 2.19 1.45 0.85 0.94 Density Clustering + LACE 22.5 8.64 2.62 2.89 2.08 0.95 0.96 Finetuned SoundStream DP 27 8.69 7.37 2.54 1.52 0.83 0.92 DP + LACE 22.5 8.64 2.78 3.60 2.45 0.92 0.97 Finetuned Encodec DP 27 8.69 3.10 3.08 1.99 0.89 0.96 DP + LACE 22.5 8.64 2.18 3.83 2.86 0.95 0.98 IV Experimental Setting IV-A Dataset We use the LibriTTS dataset [20] at 24 kHz sampling rate. The training set consists of train-clean-100, train-clean-360, and train-other-500. Evaluation is performed on test-clean. The same dataset is used for both reconstruction and TTS experiments. For TTS, a reference utterance is randomly selected from a different utterance of the same speaker to condition the model on speaker identity. IV-B Model Architecture IV-B1 Codec Models We build our method on three neural audio codecs: SoundStream [8], EnCodec [9], and the Descript Audio Codec (DAC) [7], all pretrained on LibriTTS using ESPnet-Codec [24] recipes22 2 https://huggingface.co/espnet/libritts_soundstream24k, https://huggingface.co/espnet/libritts_encodec_24k, https://huggingface.co/espnet/libritts_dac_24k. We integrate the proposed layer-wise compression into the quantization layers (Section I-A) and fine-tune each model end-to-end from the pretrained weights for 80k iterations using the original codec objective. We use Adam [25] with learning rate 10−410^-4, an exponential decay schedule (rate 0.99980.9998 per step, minimum 10−510^-5), and shared optimization settings for the generator and discriminator. All experiments use 4 V100 32GB GPUs. IV-B2 TTS Model The TTS model is a 24-layer Transformer with 16 attention heads, model dimension 1024, feedforward dimension 4096, and dropout 0.1. Input transcripts are converted to phoneme sequences using the espeak-ng phonemizer [26]. A randomly truncated 3-second reference utterance is prepended to condition the model on speaker identity. The model is trained for 40k iterations on 4 V100 32GB GPUs using Adam with learning rate 10−410^-4 and 4000 warmup steps. IV-C Baselines We compare LACE with CodecSlime [15], FlexiCodec [14], and VARSTok [13]. Because these methods use different codec backbones and training setups, we re-implement them on LibriTTS with the same codec backbones and fine-tuning configuration, changing only the compression model. These baselines use single compression, where the compression step is applied once before the quantization layers so all layers share the same segmentation boundaries. In contrast, LACE applies compression independently at each quantization layer. IV-D Rate Computation For reconstruction, we report the effective frame rate before union alignment, since alignment is not required. For threshold-based compression models that yield different effective frame rates across layers, we report the mean effective frame rate across layers. The bitrate accounts for both code and duration bits, where each duration costs log2U _2U bits per segment. For a fair comparison, we compare at matched bitrate (∼8.6 8.6 kbps). Note that because LACE adds one duration sequence per layer, it reaches this bitrate at a lower effective frame rate than single compression (22.5 vs. 27 Hz). IV-E Evaluation Metrics Reconstruction. We compute word error rate (WER) with Whisper-Large [27] by transcribing the reconstructed audio. UTMOS [28] assesses perceptual naturalness, while PESQ [29] and STOI [30] measure signal quality and intelligibility. For speaker similarity (SpkSim), we compute the cosine similarity between X-vectors from the generated and reference audio, extracted with WavLM-base [31] fine-tuned for speaker verification33 3 https://huggingface.co/microsoft/wavlm-base-plus-sv. All metrics use VERSA [32] to compute. TTS. Since TTS generation is non-deterministic, we generate 10 utterances per input and select the one with the lowest WER under Whisper-Small, following prior work [33]. The selected utterance is evaluated with Whisper-Large for WER, along with UTMOS and SpkSim. We also report the real-time factor (RTF), the ratio of generation time to audio duration, to assess inference efficiency. To complement these objective metrics, we conduct a human evaluation using mean opinion score (MOS) and speaker similarity mean opinion score (SMOS) to evaluate naturalness and speaker similarity, respectively. Specifically, we randomly sample 25 test utterances and synthesize them with each TTS system. Then, we collect ratings from 35 evaluators. V Results V-A Reconstruction Task We evaluate reconstruction quality across three codec backbones (DAC [7], SoundStream [8], EnCodec [9]) and three compression models (DP [15], Cosine Similarity [14], and Density Clustering [13]). All systems are matched at a similar bitrate (∼8.6 8.6 kbps), resulting in different effective frame rates. Table I shows that LACE surpasses the single-compression baselines in every configuration, indicating that its benefit is agnostic to both the compression model and the codec backbone. Figure 3 provides a finer-grained view through the rate-quality curve. LACE outperforms single compression at every bitrates, even on the pretrained model without fine-tuning. This indicates that the improvement comes from LACE itself, while fine-tuning provides further gains. V-B Text-to-Speech Task TABLE I: TTS evaluation on LibriTTS test-clean. Rate is the effective frame rate after union alignment. We report MOS and SMOS, with 95% confidence intervals. RTF is measured on a single V100 GPU for one candidate. Method γ l∗l^* Rate WER ↓ UTMOS ↑ SpkSim ↑ MOS ↑ SMOS ↑ RTF ↓ (Hz) (%) No compression – – 75.0 2.31 4.18 0.91 3.94 ± 0.05 3.94 ± 0.06 1.95 Single 0.5 – 37.5 30.73 1.79 0.84 3.80 ± 0.06 3.80 ± 0.07 1.01 LACE 0.5 2 52.5 4.93 2.69 0.86 3.88 ± 0.06 3.86 ± 0.07 1.58 LACE 0.5 3 58.0 8.52 2.70 0.88 3.89 ± 0.06 3.84 ± 0.06 1.68 Single 0.7 – 52.5 6.77 2.59 0.85 3.85 ± 0.06 3.82 ± 0.07 1.45 LACE 0.7 2 61.8 5.11 3.45 0.90 3.96 ± 0.05 3.90 ± 0.06 1.73 We train and evaluate the TTS model using DAC as the codec backbone with DP compression. We compare against two baselines: DAC without compression and DAC with single DP compression. Table I shows that, at the same target compression rate γ, LACE outperforms single compression on all quality metrics. This improvement comes at the cost of a higher RTF because union alignment increases the effective frame rate. Although the MOS scores of single compression with γ=0.5γ=0.5 and no compression are close, the paired t-test shows that raters can still distinguish between them (p<0.01)(p<0.01). Compared to the no-compression baseline, LACE achieves a lower RTF by reducing the effective frame rate. This comes at a quality cost, partly because the compressed codec has lower reconstruction quality (Table I). The target compression rate γ directly controls the quality–efficiency tradeoff. Increasing γ improves synthesis quality while increasing RTF. In contrast, increasing the anchor layer l∗l^* beyond 2 does not improve quality and degrades WER. We hypothesize that this behavior arises from the union alignment step. A larger l∗l^* causes long segments to be split into shorter ones, introducing repeated discrete code IDs after alignment. These repeated code IDs bias the TTS model toward repeatedly predicting the same code IDs during generation. Incorporating repetition-aware sampling [2] to prevent repetitive code prediction is a promising direction for future work. To ensure that the observed WER trends are robust to the reranking, we computed the mean WER over 10 generated candidates using Whisper-Small. The ranking remains consistent. LACE achieves 14% WER compared with 61% for single compression at γ=0.5γ=0.5, and 14% compared with 21% at γ=0.7γ=0.7. Fig. 3: Rate-distortion comparison between LACE and Single compression on LibriTTS test-clean using DAC as the codec backbone and DP compression. Fig. 4: Distribution of cosine similarity between consecutive frame-level residual embeddings across quantization layers. Fig. 5: Empirical validation of the theoretical analysis using the pretrained DAC model with DP compression. Left: quantization error at various compression rates γ using all 32 quantization layers. Right: quantization error as a function of the number of quantization layers, at γ=0.5γ=0.5. V-C Analysis We perform two analyses to better understand LACE. The first examines how fast the residual embeddings change at each quantization layer. The second empirically validates the quantization error bound from Section I-D. V-C1 Histogram of Cosine Similarity On the pretrained codecs, we compute the cosine similarity between consecutive frame-level residual embeddings, σtl=cos(tl,t+1l)σ^l_t= (r^l_t,r^l_t+1). Figure 4 shows that its distribution concentrates around 11 at earlier layers and gradually shifts toward 00 at deeper layers. This indicates that residual embeddings change more rapidly at deeper layers and confirms that different layers change at different rates, motivating layer-wise compression. Moreover, the slower variation of earlier layers suggests that segmentation decisions have a larger impact on these layers than on deeper layers. We therefore let earlier layers choose its own segmentation boundaries before deeper layers, justifying the boundary anchor (Section I-C). V-C2 Empirical Result of VQ Error While Figure 4 only motivates layer-wise compression, this analysis directly measures the error caused by shared segmentation boundaries. We measure the quantization error [‖C+1‖F2]E[\|R^C+1\|_F^2] on the pretrained DAC model with DP compression while varying the anchor layer l∗l^*. Figure 5 shows that LACE yields consistently lower quantization error than single compression, matching Equation (3). The right panel shows the gap widening as more layers are used. Single compression converges to a constant error floor, while LACE keeps reducing the error. The left panel shows the role of γ. As γ→1γ→ 1, compression merges almost no frames, so α1→1 _1→ 1 and the floor (1−α1)[‖F2](1- _1)\,E[\|H\|_F^2] vanishes, explaining the narrow gap at high compression rates. At lower rates the floor grows and the benefit of LACE becomes more pronounced. VI Conclusion We presented LACE, a dynamic frame rate codec that applies an independent compression step at each quantization layer. Unlike prior methods that force all layers to share the same segmentation boundaries, LACE allows each layer to choose its own segmentation boundaries. To enable downstream TTS training, we introduced union alignment, which constructs shared segmentation boundaries across layers, and boundary anchor, which limits the growth of the effective frame rate. Experiments on LibriTTS show that LACE consistently outperforms single-compression baselines across multiple codec backbones and compression models on the reconstruction task. Applied to TTS, LACE improves inference efficiency while maintaining competitive quality. Acknowledgment Experiments of this work used the Bridges2 system at PSC and Delta and DeltaAI system at NCSA through allocations CIS210014 and IRI120008P from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. Generative AI Use Disclosure Generative AI tools were used only to edit and polish the manuscript language and to assist with code writing. All research ideas, experimental design, implementation, analysis, and reported results are the work of the authors. The conceptual framing, methodology, and scientific contributions are entirely human-generated. References [1] S. Chen, C. Wang, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei (2025) Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing 33, p. 705–718. External Links: Document Cited by: §I. [2] S. Chen, S. Liu, L. Zhou, E. Liu, X. Tan, J. Li, S. Zhao, Y. Qian, and F. Wei (2025) VALL-E 2: neural codec language models are human parity zero-shot text to speech synthesizers. External Links: Link Cited by: §I, §I-E, §V-B. [3] P. Peng, P. Huang, S. Li, A. Mohamed, and D. Harwath (2024) VoiceCraft: zero-shot speech editing and text-to-speech in the wild. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Bangkok, Thailand, p. 12442–12462. External Links: Link, Document Cited by: §I. [4] S. Guo, S. Zhang, Q. Fang, Z. Ma, M. Zhang, and Y. Feng (2026) FastLongSpeech: enhancing large speech-language models for efficient long-speech processing. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §I. [5] J. Tian, J. Shi, W. Chen, S. Arora, Y. Masuyama, T. Maekaku, Y. Wu, J. Peng, S. Bharadwaj, Y. Zhao, et al. (2025) ESPnet-SpeechLM: an open speech language model toolkit. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): System Demonstrations, p. 116–124. Cited by: §I. [6] D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, H. Guo, X. Chang, J. Shi, S. Zhao, J. Bian, Z. Zhao, X. Wu, and H. M. Meng (2024) UniAudio: towards universal audio generation with large language models. In International Conference on Machine Learning, External Links: Link Cited by: §I. [7] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar (2023) High-fidelity audio compression with improved RVQGAN. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 27980–27993. Cited by: §I, TABLE I, §IV-B1, §V-A. [8] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi (2022) SoundStream: an end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30, p. 495–507. External Links: Document Cited by: §I, TABLE I, §IV-B1, §V-A. [9] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi (2023) High fidelity neural audio compression. Transactions on Machine Learning Research. Note: Featured Certification, Reproducibility Certification External Links: ISSN 2835-8856, Link Cited by: §I, TABLE I, §IV-B1, §V-A. [10] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour (2023) AudioLM: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, p. 2523–2533. External Links: Document Cited by: §I. [11] H. Wang, H. Wang, Y. Guo, Z. Li, C. Du, and K. Yu (2026) Why do speech language models fail to generate semantically coherent outputs? a modality evolving perspective. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 16117–16121. External Links: Document Cited by: §I. [12] H. Zhang, Y. Guo, Z. Li, X. Hao, X. Chen, and K. Yu (2025) Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate. In Interspeech 2025, p. 5003–5007. External Links: Document, ISSN 2958-1796 Cited by: §I. [13] R. Zheng, W. Liu, H. Du, Q. Zhang, C. Deng, Q. Chen, W. Wang, Y. Ai, and Z. Ling (2026) Say more with less: variable-frame-rate speech tokenization via adaptive clustering and implicit duration coding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 35021–35029. Cited by: §I, §I, §I-B, TABLE I, §IV-C, §V-A. [14] J. Li, Y. Qian, Y. Hu, L. Zhang, X. Wang, H. Lu, M. Thakker, J. Li, S. Zhao, and Z. Wu (2026) FlexiCodec: a dynamic neural audio codec for low frame rates. In International Conference on Learning Representations, External Links: Link Cited by: §I, §I, §I-B, TABLE I, §IV-C, §V-A. [15] H. Wang, Y. Guo, C. Shao, B. Li, and K. Yu (2026) CodecSlime: temporal redundancy compression of neural speech codec via dynamic frame rate. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 17017–17021. External Links: Document Cited by: §I, §I, §I-B, TABLE I, §IV-C, §V-A. [16] W. Zhang, Y. Qian, Y. Cao, C. He, S. Xu, and M. Wang (2026) CARVE: content-adaptive rate-variable encoding for neural speech codecs. IEEE Signal Processing Letters 33, p. 2036–2040. External Links: Document Cited by: §I. [17] Y. Qian, W. Zhang, X. Zhuang, S. Xu, L. Zhou, and M. Wang (2026) Arbitrarily settable frame rate neural speech codec with content adaptive variable length segmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 14452–14456. External Links: Document Cited by: §I. [18] D. Lee, C. Kim, S. Kim, M. Cho, and W. Han (2022) Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11523–11532. Cited by: §I. [19] H. Siuzdak, F. Grötschla, and L. A. Lanzendörfer (2024) SNAC: multi-scale neural audio codec. In Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, External Links: Link Cited by: §I. [20] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu (2019) LibriTTS: a corpus derived from LibriSpeech for text-to-speech. In Interspeech 2019, p. 1526–1530. External Links: Document, ISSN 2958-1796 Cited by: 4th item, §IV-A. [21] S. Golomb (1966) Run-length encodings. IEEE Transactions on Information Theory 12 (3), p. 399–401. External Links: Document Cited by: §I-B. [22] A. Gersho and R. M. Gray (1992) Vector quantization and signal compression. Kluwer Academic Publishers. Cited by: §I-D, item 2. [23] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Defossez (2023) Simple and controllable music generation. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 47704–47720. External Links: Link Cited by: §I-E, §I-E. [24] J. Shi, J. Tian, Y. Wu, J. Jung, J. Q. Yip, Y. Masuyama, W. Chen, Y. Wu, Y. Tang, M. Baali, D. Alharthi, D. Zhang, R. Deng, T. Srivastava, H. Wu, A. Liu, B. Raj, Q. Jin, R. Song, and S. Watanabe (2024) ESPnet-Codec: comprehensive training and evaluation of neural codecs for audio, music, and speech. In IEEE Spoken Language Technology Workshop (SLT), p. 562–569. External Links: Document Cited by: §IV-B1. [25] D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations, External Links: Link Cited by: §IV-B1. [26] M. Bernard and H. Titeux (2021) Phonemizer: text to phones transcription for multiple languages in Python. Journal of Open Source Software 6 (68), p. 3958. External Links: Document, Link Cited by: §IV-B2. [27] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 28492–28518. External Links: Link Cited by: §IV-E. [28] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari (2022) UTMOS: UTokyo-SaruLab system for VoiceMOS challenge 2022. In Interspeech 2022, p. 4521–4525. External Links: Document, ISSN 2958-1796 Cited by: §IV-E. [29] A.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra (2001) Perceptual evaluation of speech quality (PESQ) — a new method for speech quality assessment of telephone networks and codecs. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. 2, p. 749–752. External Links: Document Cited by: §IV-E. [30] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen (2010) A short-time objective intelligibility measure for time-frequency weighted noisy speech. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 4214–4217. Cited by: §IV-E. [31] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, et al. (2022) WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing 16 (6), p. 1505–1518. Cited by: §IV-E. [32] J. Shi, H. Shim, J. Tian, S. Arora, H. Wu, D. Petermann, J. Q. Yip, Y. Zhang, Y. Tang, W. Zhang, D. S. Alharthi, Y. Huang, K. Saito, J. Han, Y. Zhao, C. Donahue, and S. Watanabe (2025) VERSA: a versatile evaluation toolkit for speech, audio, and music. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): System Demonstrations, External Links: Link Cited by: §IV-E. [33] P. Mousavi, G. Maimon, A. Moumen, D. Petermann, J. Shi, H. Wu, H. Yang, A. Kuznetsova, A. Ploujnikov, R. Marxer, B. Ramabhadran, B. Elizalde, L. Lugosch, J. Li, C. Subakan, P. Woodland, M. Kim, H. Lee, S. Watanabe, Y. Adi, and M. Ravanelli (2025) Discrete audio tokens: more than a survey!. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: §IV-E. Supplementary Material I Supplementary Material All sections, equation, and figure references below refer to the main paper unless stated otherwise. I-A Upper Bound of the Expected Quantization Error Setup. Let ∈ℝT×DH ^T× D be the frame-level embeddings drawn from an in-domain speech distribution. At layer l, the compression step performs two operations: (1) averaging the frame-level embeddings within each segment, and (2) repeating each segment-level embedding according to its duration. In Sections I-B and I-A, the repetition happens after the quantization layer rather than before, since all repeated frames within a segment are identical, the two orders yield the same result. We therefore combine both operations into a single matrix multiplication llA^lR^l, where l∈ℝT×TA^l ^T× T is the compression matrix at layer l and lR^l is the frame-level residual embedding from Section I-A, with 1=R^1=H. The matrix lA^l is block-diagonal: the block of the m-th segment has every entry equal to 1/dm1/d_m, where dmd_m is the segment duration. From this structure, lA^l is symmetric ((l)⊤=l(A^l) =A^l) and idempotent ((l)2=l(A^l)^2=A^l). Let VQl(⋅)VQ_l(·) denote quantization at layer l followed by upsampling, so that the frame-level quantized embedding is ^l=VQl(ll) Q^l=VQ_l(A^lR^l) and the residual update of Section I-A becomes l+1=l−^lR^l+1=R^l- Q^l. Since ^l Q^l is constant within each segment of layer l, applying the compression matrix leaves it unchanged: l^l=^l. ^l Q^l= Q^l. (1) We analyze the boundary anchor setting of Section I-C: layers l≤l∗l≤ l^* compute new boundaries, while layers l>l∗l>l^* reuse the compression matrix of layer l∗l^*, denoted ⋆=l∗A =A^l^*. Assumptions. 1. Strict compression: every layer merges at least two frames (Ml<TM^l<T), so the compression error is strictly positive: [‖l−ll‖F2]>0E[\|R^l-A^lR^l\|_F^2]>0. 2. Centroid condition: each quantization layer is well-trained. Its codewords satisfy the centroid condition [22], i.e., each codeword equals the conditional mean of the inputs assigned to it. We further assume that each layer captures positive energy, [‖^l‖F2]>0E[\| Q^l\|_F^2]>0, and the resulting distortion ratios (Lemma 2) are uniformly bounded by ϵmax=suplϵl<1 _ = _l _l<1. 3. Positive energy capture: we define αl=[‖ll‖F2][‖l‖F2] _l= E[\|A^lR^l\|_F^2]E[\|R^l\|_F^2] (2) as the fraction of expected residual energy captured by the compression step at layer l, and assume it is uniformly bounded below by αmin=inflαl>0 _ = _l _l>0. Lemma 1 shows αl≤1 _l≤ 1, together with Assumption 1 and Equation (11), αl<1 _l<1. Lemma 1 (Compression never increases energy). For any ∈ℝT×DR ^T× D and any compression matrix A, ‖F2≤‖F2. \|AR\|_F^2≤\|R\|_F^2. (3) Proof. The compression matrix acts independently on each segment and each feature dimension, so it suffices to prove the claim for a single segment and a single dimension. Let x1,…,xdmx_1,…,x_d_m denote the scalar values within a segment of duration dmd_m. Compression replaces every value with the segment mean μ=1dm∑i=1dmxiμ= 1d_m _i=1^d_mx_i, so the energy of the segment changes from ∑i=1dmxi2 _i=1^d_mx_i^2 to dmμ2=1dm(∑i=1dmxi)2d_mμ^2= 1d_m ( _i=1^d_mx_i )^2. By the Cauchy–Schwarz inequality with the all-ones vector, (∑i=1dmxi⋅1)2≤(∑i=1dmxi2)(∑i=1dm12)=dm∑i=1dmxi2, ( _i=1^d_mx_i· 1 )^2≤ ( _i=1^d_mx_i^2 ) ( _i=1^d_m1^2 )=d_m _i=1^d_mx_i^2, (4) which gives dmμ2≤∑i=1dmxi2d_mμ^2≤ _i=1^d_mx_i^2. Summing over all segments and all feature dimensions yields the claim. ■ Lemma 2 (Expected rate-distortion bound). Under Assumption 2, the expected quantization error at layer l satisfies [‖ll−^l‖F2]≤ϵl[‖ll‖F2], [\|A^lR^l- Q^l\|_F^2 ]≤ _l\,E [\|A^lR^l\|_F^2 ], (5) where ϵl=1−[‖^l‖F2]/[‖ll‖F2]<1 _l=1-E[\| Q^l\|_F^2]/E[\|A^lR^l\|_F^2]<1. Proof. Write =llX=A^lR^l for the quantization input and ^=^l X= Q^l for its output. The quantization partitions the input space into regions k\V_k\ and maps every input in region kV_k to the output kc_k. By the centroid condition, k=[∣∈k]c_k=E[X _k], so the expected quantization error within each region is zero: [−k∣∈k]=[∣∈k]−k=. [X-c_k _k]=E[X _k]-c_k=0. (6) By the law of total expectation, the quantization error is therefore orthogonal to the quantization output in expectation: [⟨−^,^⟩F] [ - X,\, X _F ] =∑kℙ(∈k)⟨[−k∣∈k],k⟩F = _kP(X _k)\, [X-c_k _k],\,c_k _F =0, =0, (7) where kc_k is pulled out of the conditional expectation because it is constant within its region. The cross-term thus vanishes in the expected Pythagorean decomposition: [‖F2]=[‖^‖F2]+[‖−^‖F2]. [\|X\|_F^2]=E[\| X\|_F^2]+E[\|X- X\|_F^2]. (8) Rearranging gives the exact equality [‖−^‖F2]=(1−[‖^‖F2][‖F2])[‖F2]=ϵl[‖F2], [\|X- X\|_F^2]= (1- E[\| X\|_F^2]E[\|X\|_F^2] )E[\|X\|_F^2]= _l\,E[\|X\|_F^2], (9) and Assumption 2 ([‖^‖F2]>0E[\| X\|_F^2]>0) gives ϵl<1 _l<1. ■ Lemma 3 (Orthogonality). For any ,∈ℝT×DR,S ^T× D and any compression matrix A, the compression error −R-AR is orthogonal to any compressed embedding AS under the Frobenius inner product: ⟨−,⟩F=Tr(⊤(−2))=0, -AR,\,AS _F=Tr (R (A-A^2)S )=0, (10) using symmetry and idempotence of A. In particular, choosing =S=R yields the Pythagorean identity: ‖F2=‖F2+‖−‖F2. \|R\|_F^2=\|AR\|_F^2+\|R-AR\|_F^2. (11) Lemma 4 (Per-layer energy contraction). For every layer l that computes new boundaries, [‖l+1‖F2]≤λl[‖l‖F2], [\|R^l+1\|_F^2]≤ _l\,E[\|R^l\|_F^2], (12) where λl=1−αl(1−ϵl)<1 _l=1- _l(1- _l)<1. Proof. Decompose the residual update as l+1=(l−ll)+(ll−^l)R^l+1=(R^l-A^lR^l)+(A^lR^l- Q^l). By Equation (1), the second term can be rewritten as a compressed embedding: ll−^l=ll−l^l=l(l−^l). ^lR^l- Q^l=A^lR^l-A^l Q^l=A^l(R^l- Q^l). (13) Therefore, the two terms are orthogonal by Lemma 3 with =l−^lS=R^l- Q^l, and their squared norms add: [‖l+1‖F2] [\|R^l+1\|_F^2] =[‖l−ll‖F2]+[‖ll−^l‖F2] =E[\|R^l-A^lR^l\|_F^2]+E[\|A^lR^l- Q^l\|_F^2] ≤(1−αl)[‖l‖F2]+ϵlαl[‖l‖F2] ≤(1- _l)\,E[\|R^l\|_F^2]+ _l\, _l\,E[\|R^l\|_F^2] =λl[‖l‖F2]. = _l\,E[\|R^l\|_F^2]. (14) The first term follows from Equation (11) and the definition of αl _l in Equation (2), which together give [‖l−ll‖F2]=(1−αl)[‖l‖F2]E[\|R^l-A^lR^l\|_F^2]=(1- _l)\,E[\|R^l\|_F^2]. The second term applies Equation (5) and then Equation (2): [‖ll−^l‖F2]≤ϵl[‖ll‖F2]=ϵlαl[‖l‖F2]E[\|A^lR^l- Q^l\|_F^2]≤ _l\,E[\|A^lR^l\|_F^2]= _l\, _l\,E[\|R^l\|_F^2]. ■ Lemma 5 (Invariance of the compression error). If l=⋆A^l=A for all l≥l∗l≥ l^*, then for all such layers: l+1−⋆l+1=l−⋆l. ^l+1-A R^l+1=R^l-A R^l. (15) Proof. Applying ⋆A to the residual update l+1=l−^lR^l+1=R^l- Q^l and using Equation (1): ⋆l+1=⋆l−^l. R^l+1=A R^l- Q^l. (16) Subtracting Equation (16) from the residual update cancels ^l Q^l and yields the claim. Hence, the compression error introduced at layer l∗l^* passes through all subsequent layers unchanged: it cannot be reduced by any quantization that reuses ⋆A . ■ Theorem 1 (Hierarchy of error bounds). After all C quantization layers, the expected residual error satisfies: [‖C+1‖F2]≤[1−αl∗(1−ϵmaxC−l∗+1)]∏l=1l∗−1λl[‖F2]. [\|R^C+1\|_F^2]≤ [1- _l^* (1- _ ^\,C-l^*+1 ) ] _l=1^l^*-1 _l\,E[\|H\|_F^2]. (17) Proof. Decompose the final residual as C+1=(C+1−⋆C+1)+⋆C+1R^C+1=(R^C+1-A R^C+1)+A R^C+1. The two parts are orthogonal by Lemma 3, and applying Lemma 5 repeatedly over layers l∗,…,Cl^*,…,C replaces the first part with the compression error at layer l∗l^*: [‖C+1‖F2] [\|R^C+1\|_F^2] =[‖l∗−⋆l∗‖F2] =E [\|R^l^*-A R^l^*\|_F^2 ] +[‖⋆C+1‖F2]. +E [\|A R^C+1\|_F^2 ]. (18) The first term equals (1−αl∗)[‖l∗‖F2](1- _l^*)\,E[\|R^l^*\|_F^2] by Equation (11) and the definition of αl∗ _l^* in Equation (2). For the second term, consider any layer l≥l∗l≥ l^*. These layers share the same compression matrix (l=⋆A^l=A ), so Equation (16) shows that ⋆l+1A R^l+1 is exactly the quantization error of layer l. Since l=⋆A^l=A , the quantizer input llA^lR^l equals ⋆lA R^l, so Lemma 2 (Equation (5)) applies with input ⋆lA R^l and bounds it: [‖⋆l+1‖F2]=[‖⋆l−^l‖F2]≤ϵmax[‖⋆l‖F2]. [\|A R^l+1\|_F^2]=E[\|A R^l- Q^l\|_F^2]≤ _ \,E[\|A R^l\|_F^2]. (19) Applying this bound recursively over the C−l∗+1C-l^*+1 layers from l∗l^* to C, and then converting the compressed energy at layer l∗l^* to residual energy via Equation (2): [‖⋆C+1‖F2] [\|A R^C+1\|_F^2] ≤ϵmaxC−l∗+1[‖⋆l∗‖F2] ≤ _ ^\,C-l^*+1\,E[\|A R^l^*\|_F^2] =ϵmaxC−l∗+1αl∗[‖l∗‖F2]. = _ ^\,C-l^*+1\, _l^*\,E[\|R^l^*\|_F^2]. (20) Summing the two terms gives [1−αl∗(1−ϵmaxC−l∗+1)][‖l∗‖F2] [1- _l^* (1- _ ^\,C-l^*+1 ) ]\,E[\|R^l^*\|_F^2]. Bounding [‖l∗‖F2]E[\|R^l^*\|_F^2] by unrolling Lemma 4 over layers 1,…,l∗−11,…,l^*-1 yields Equation (17). ■ Equation (17) reveals a hierarchy of bounds as C→∞C→∞: • Shared segmentation boundary (l∗=1l^*=1). All layers reuse the boundaries of the first layer, corresponding to single compression. The second term of Equation (18) vanishes and the error converges to a constant floor: limC→∞[‖C+1‖F2]=(1−α1)[‖F2], _C→∞E[\|R^C+1\|_F^2]=(1- _1)\,E[\|H\|_F^2], (21) which is exactly the compression error of the first layer and is strictly positive by Assumption 1. • Boundary anchor (1<l∗<C1<l^*<C). The error floor is reduced by the contraction of the layer-wise phase: limC→∞[‖C+1‖F2]≤(1−αl∗)∏l=1l∗−1λl[‖F2]. _C→∞E[\|R^C+1\|_F^2]≤(1- _l^*) _l=1^l^*-1 _l\,E[\|H\|_F^2]. (22) • Layer-wise compression without anchor (LACE). Every layer computes new boundaries, so Lemma 4 applies at every layer: [‖C+1‖F2]≤∏l=1Cλl[‖F2]≤λmaxC[‖F2], [\|R^C+1\|_F^2]≤ _l=1^C _l\,E[\|H\|_F^2]≤ _ ^\,C\,E[\|H\|_F^2], (23) where λmax=1−αmin(1−ϵmax)<1 _ =1- _ (1- _ )<1, so the bound converges to zero as C→∞C→∞. Therefore, recomputing boundaries inside the residual loop removes the constant error floor inherent to a shared segmentation boundary, while the boundary anchor interpolates between the two regimes. Remark. This analysis assumes that layers l>l∗l>l^* reuse the segmentation boundary of layer l∗l^* exactly. In our implementation (Section I-C), these layers may instead select any subset of ℬanchorB^anchor. The theorem corresponds to the special case where the full segmentation boundary of layer l∗l^* is reused. References [1] S. Chen, C. Wang, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei (2025) Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing 33, p. 705–718. External Links: Document Cited by: §I. [2] S. Chen, S. Liu, L. Zhou, E. Liu, X. Tan, J. Li, S. Zhao, Y. Qian, and F. Wei (2025) VALL-E 2: neural codec language models are human parity zero-shot text to speech synthesizers. External Links: Link Cited by: §I, §I-E, §V-B. [3] P. Peng, P. Huang, S. Li, A. Mohamed, and D. Harwath (2024) VoiceCraft: zero-shot speech editing and text-to-speech in the wild. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Bangkok, Thailand, p. 12442–12462. External Links: Link, Document Cited by: §I. [4] S. Guo, S. Zhang, Q. Fang, Z. Ma, M. Zhang, and Y. Feng (2026) FastLongSpeech: enhancing large speech-language models for efficient long-speech processing. In Advances in Neural Information Processing Systems, External Links: Link Cited by: §I. [5] J. Tian, J. Shi, W. Chen, S. Arora, Y. Masuyama, T. Maekaku, Y. Wu, J. Peng, S. Bharadwaj, Y. Zhao, et al. (2025) ESPnet-SpeechLM: an open speech language model toolkit. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): System Demonstrations, p. 116–124. Cited by: §I. [6] D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, H. Guo, X. Chang, J. Shi, S. Zhao, J. Bian, Z. Zhao, X. Wu, and H. M. Meng (2024) UniAudio: towards universal audio generation with large language models. In International Conference on Machine Learning, External Links: Link Cited by: §I. [7] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar (2023) High-fidelity audio compression with improved RVQGAN. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 27980–27993. Cited by: §I, TABLE I, §IV-B1, §V-A. [8] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi (2022) SoundStream: an end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30, p. 495–507. External Links: Document Cited by: §I, TABLE I, §IV-B1, §V-A. [9] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi (2023) High fidelity neural audio compression. Transactions on Machine Learning Research. Note: Featured Certification, Reproducibility Certification External Links: ISSN 2835-8856, Link Cited by: §I, TABLE I, §IV-B1, §V-A. [10] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour (2023) AudioLM: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, p. 2523–2533. External Links: Document Cited by: §I. [11] H. Wang, H. Wang, Y. Guo, Z. Li, C. Du, and K. Yu (2026) Why do speech language models fail to generate semantically coherent outputs? a modality evolving perspective. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 16117–16121. External Links: Document Cited by: §I. [12] H. Zhang, Y. Guo, Z. Li, X. Hao, X. Chen, and K. Yu (2025) Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate. In Interspeech 2025, p. 5003–5007. External Links: Document, ISSN 2958-1796 Cited by: §I. [13] R. Zheng, W. Liu, H. Du, Q. Zhang, C. Deng, Q. Chen, W. Wang, Y. Ai, and Z. Ling (2026) Say more with less: variable-frame-rate speech tokenization via adaptive clustering and implicit duration coding. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 35021–35029. Cited by: §I, §I, §I-B, TABLE I, §IV-C, §V-A. [14] J. Li, Y. Qian, Y. Hu, L. Zhang, X. Wang, H. Lu, M. Thakker, J. Li, S. Zhao, and Z. Wu (2026) FlexiCodec: a dynamic neural audio codec for low frame rates. In International Conference on Learning Representations, External Links: Link Cited by: §I, §I, §I-B, TABLE I, §IV-C, §V-A. [15] H. Wang, Y. Guo, C. Shao, B. Li, and K. Yu (2026) CodecSlime: temporal redundancy compression of neural speech codec via dynamic frame rate. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 17017–17021. External Links: Document Cited by: §I, §I, §I-B, TABLE I, §IV-C, §V-A. [16] W. Zhang, Y. Qian, Y. Cao, C. He, S. Xu, and M. Wang (2026) CARVE: content-adaptive rate-variable encoding for neural speech codecs. IEEE Signal Processing Letters 33, p. 2036–2040. External Links: Document Cited by: §I. [17] Y. Qian, W. Zhang, X. Zhuang, S. Xu, L. Zhou, and M. Wang (2026) Arbitrarily settable frame rate neural speech codec with content adaptive variable length segmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 14452–14456. External Links: Document Cited by: §I. [18] D. Lee, C. Kim, S. Kim, M. Cho, and W. Han (2022) Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11523–11532. Cited by: §I. [19] H. Siuzdak, F. Grötschla, and L. A. Lanzendörfer (2024) SNAC: multi-scale neural audio codec. In Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, External Links: Link Cited by: §I. [20] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu (2019) LibriTTS: a corpus derived from LibriSpeech for text-to-speech. In Interspeech 2019, p. 1526–1530. External Links: Document, ISSN 2958-1796 Cited by: 4th item, §IV-A. [21] S. Golomb (1966) Run-length encodings. IEEE Transactions on Information Theory 12 (3), p. 399–401. External Links: Document Cited by: §I-B. [22] A. Gersho and R. M. Gray (1992) Vector quantization and signal compression. Kluwer Academic Publishers. Cited by: §I-D, item 2. [23] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Defossez (2023) Simple and controllable music generation. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 47704–47720. External Links: Link Cited by: §I-E, §I-E. [24] J. Shi, J. Tian, Y. Wu, J. Jung, J. Q. Yip, Y. Masuyama, W. Chen, Y. Wu, Y. Tang, M. Baali, D. Alharthi, D. Zhang, R. Deng, T. Srivastava, H. Wu, A. Liu, B. Raj, Q. Jin, R. Song, and S. Watanabe (2024) ESPnet-Codec: comprehensive training and evaluation of neural codecs for audio, music, and speech. In IEEE Spoken Language Technology Workshop (SLT), p. 562–569. External Links: Document Cited by: §IV-B1. [25] D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations, External Links: Link Cited by: §IV-B1. [26] M. Bernard and H. Titeux (2021) Phonemizer: text to phones transcription for multiple languages in Python. Journal of Open Source Software 6 (68), p. 3958. External Links: Document, Link Cited by: §IV-B2. [27] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever (2023) Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 28492–28518. External Links: Link Cited by: §IV-E. [28] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari (2022) UTMOS: UTokyo-SaruLab system for VoiceMOS challenge 2022. In Interspeech 2022, p. 4521–4525. External Links: Document, ISSN 2958-1796 Cited by: §IV-E. [29] A.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra (2001) Perceptual evaluation of speech quality (PESQ) — a new method for speech quality assessment of telephone networks and codecs. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. 2, p. 749–752. External Links: Document Cited by: §IV-E. [30] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen (2010) A short-time objective intelligibility measure for time-frequency weighted noisy speech. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 4214–4217. Cited by: §IV-E. [31] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, et al. (2022) WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing 16 (6), p. 1505–1518. Cited by: §IV-E. [32] J. Shi, H. Shim, J. Tian, S. Arora, H. Wu, D. Petermann, J. Q. Yip, Y. Zhang, Y. Tang, W. Zhang, D. S. Alharthi, Y. Huang, K. Saito, J. Han, Y. Zhao, C. Donahue, and S. Watanabe (2025) VERSA: a versatile evaluation toolkit for speech, audio, and music. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): System Demonstrations, External Links: Link Cited by: §IV-E. [33] P. Mousavi, G. Maimon, A. Moumen, D. Petermann, J. Shi, H. Wu, H. Yang, A. Kuznetsova, A. Ploujnikov, R. Marxer, B. Ramabhadran, B. Elizalde, L. Lugosch, J. Li, C. Subakan, P. Woodland, M. Kim, H. Lee, S. Watanabe, Y. Adi, and M. Ravanelli (2025) Discrete audio tokens: more than a survey!. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: §IV-E.