Paper deep dive
Music Restoration via Latent Operator Optimization and Diffusion Model Priors
Michal Švento, Eloi Moliner, Valtteri Kallinen, Lauri Juvela, Vesa Välimäki, Pavel Rajmic
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Music restoration seeks to recover a clean signal from an observed recording degraded by an unknown effect, distortion, or corruption. Existing systems often rely on paired training data and distortion-specific supervision, which limits their use when the forward process is not known in advance. We propose LOUDAR (Latent-space Optimization of Unknown Distortion for Audio Restoration) a general-purpose restoration method that operates in the latent space of a pretrained audio autoencoder and models the unknown distortion as a learnable latent operator. At inference time, LOUDAR alternates between estimating the clean latent variable and updating the latent operator parameters. An unconditional latent diffusion model provides a prior over clean audio and regularizes this inference by steering the latent estimate toward the manifold of clean recordings. Because the degradation model is adapted per input, the approach is broadly applicable across diverse restoration problems. We evaluate LOUDAR on singing voice effect removal and restoration, as well as guitar distortion removal, and show that it consistently improves over degraded inputs and is competitive with supervised and unsupervised baselines in waveform and latent domains.
Tags
Links
- Source: https://arxiv.org/abs/2608.01972v1
- Canonical: https://arxiv.org/abs/2608.01972v1
Trouble viewing inline? Open PDF directly →
Full Text
48,088 characters extracted from source content.
Expand or collapse full text
LDM Latent Diffusion Model MMSE Minimum Mean Squared Error DiT Diffusion Transformer KAD Kernel Audio Distance AE Autoencoder CD cosine distance SRS Smule Renaissance Small M2L Music2Latent EM Expectation Maximization Music Restoration via Latent Operator Optimization and Diffusion Model Priors Abstract Music restoration seeks to recover a clean signal from an observed recording degraded by an unknown effect, distortion, or corruption. Existing systems often rely on paired training data and distortion-specific supervision, which limits their use when the forward process is not known in advance. We propose LOUDAR (Latent-space Optimization of Unknown Distortion for Audio Restoration) a general-purpose restoration method that operates in the latent space of a pretrained audio autoencoder and models the unknown distortion as a learnable latent operator. At inference time, LOUDAR alternates between estimating the clean latent variable and updating the latent operator parameters. An unconditional latent diffusion model provides a prior over clean audio and regularizes this inference by steering the latent estimate toward the manifold of clean recordings. Because the degradation model is adapted per input, the approach is broadly applicable across diverse restoration problems. We evaluate LOUDAR on singing voice effect removal and restoration, as well as guitar distortion removal, and show that it consistently improves over degraded inputs and is competitive with supervised and unsupervised baselines in waveform and latent domains. 1 Introduction Music recordings are rarely observed in a dry or studio-ready form. Vocals, guitars, and full mixes are typically shaped by chains of effects such as reverberation, compression, and saturation. Recordings may also be affected by distortions such as compression artifacts, band-limitation or clipping. In many cases, it is desirable to recover the unprocessed signal from such observations, a process we refer to as music “restoration.” Such recovery should ideally remain possible even when the degradations or effect chains, which we broadly refer to as “distortion,” are unknown and overly complex. This capability is useful for applications such as music production, remixing or editing, as well as for preparing training data for music technology systems such as automatic mixing [25], large language model-guided post-production [8], or source separation [48]. Most restoration approaches rely on supervised training with paired clean/distorted data and a predefined family of distortions assumed during model development [14, 33, 40, 19]. This assumption is limiting in practice. Audio effects applied to music can be highly complex in serial and parallel processing chains [17], so designing a comprehensive pipeline is nearly impossible. Moreover, certain corruption processes are inherently difficult to simulate and lack a clean reference entirely, such as historical recordings [29]. As a result, supervised systems are tailored to the distortions represented in training and can struggle with more complex or mismatched effect chains at test time [18]. There is, nonetheless, an unsupervised alternative: framing restoration as a blind inverse problem, where a diffusion model trained on clean audio serves as a prior while jointly estimating a parametric distortion model for approximate posterior sampling. Existing methods in this paradigm, however, typically remain restricted to particular forward-model families [18, 29, 28, 45]. A general restoration framework that can adapt at test time to an unknown processing chain while exploiting strong priors over clean music is still lacking. Learned audio representations provide an appealing foundation for such a framework [20, 9]. Autoencoders map waveforms to compact and semantically-organized latent spaces. Prior work suggests latent-space restoration as a technically promising direction [43, 1, 7, 32]. We hypothesize that such representations enable tractable unsupervised restoration in latent space by compressing both the representation and the distortion model, enabling a compact neural network to approximate a broad range of distortions while remaining easy to optimize. This paper introduces LOUDAR (Latent-space Optimization of Unknown Distortion for Audio Restoration), an unsupervised framework for music restoration. LOUDAR follows a similar inference strategy as recent unsupervised restoration methods [18, 28], but transfers it to the latent space of an AE. The proposed method parameterizes the unknown distortion directly in latent space using a learnable operator, while an unconditional Latent Diffusion Model (LDM) trained on clean music provides a prior over unprocessed signals. At test time, LOUDAR alternates between estimating a clean latent vector and updating the latent distortion operator and residual perturbation, yielding an Expectation Maximization (EM) inference procedure specialized to the input recording. Our main contributions are threefold: (i) a formulation of music restoration as a blind inverse problem in the latent space; (i) a compact, general-purpose latent operator architecture designed to stabilize blind optimization; and (i) a comprehensive evaluation across singing voice and guitar domains, using objective and subjective metrics. Code and listening examples are available on the project website111https://michalsvento.github.io/loudar/. 2 Methods 2.1 Problem Definition We consider restoration problems where an observed waveform is generated from a clean one through an unknown distortion process. Let ∈ℝLx ^L and ∈ℝLy ^L denote the clean and distorted waveforms, respectively, both of length L. We assume the forward process =f()+,y=f(x)+n, (1) where f:ℝL→ℝLf ^L ^L is an unknown, possibly nonlinear, distortion operator and ∈ℝLn ^L is an additive residual term. This process involving waveforms is shown in the top part of Figure 1. Our goal is to recover the clean (unprocessed) waveform x given an observation y. Inspired by [32], rather than solving this inverse problem in the waveform domain, we work in the latent space of a pretrained (frozen) audio AE, with encoder E:ℝL→ℝC×NE ^L ^C× N and decoder D:ℝC×N→ℝLD ^C× N ^L. The clean and distorted latent vectors are denoted by 0:=E()∈ℝC×Nz_0:=E(x) ^C× N and y:=E()∈ℝC×Nz_y:=E(y) ^C× N, respectively, where C is the channel size and N is the number of latent frames. We denote pxp_x the clean waveform data distribution, and pzp_z the corresponding clean latent distribution induced by the encoder E. Empirical access to these distributions is provided by a collection of clean training examples ii∼px\x^i\_i p_x and corresponding latent vectors 0ii=E(i)i∼pz\z_0^i\_i=\E(x^i)\_i p_z. The AE is also assumed to approximately reconstruct clean signals on the support of pxp_x, i.e., D(E())≈D(E(x)) for ∼pxx p_x. The distorted latent vector is then modeled as y=E(f()+)≈gϕ(0)+,z_y=E (f(x)+n )≈ g_φ(z_0)+r, (2) where gϕ:ℝC×N→ℝC×Ng_ φ ^C× N ^C× N is a parametric latent distortion operator with unknown parameters ϕ φ. The residual ∈ℝC×Nr ^C× N relaxes the time-invariance of gϕg_φ, absorbing small localized artifacts or noise it cannot represent. Figure 1 illustrates this signal model. While [32] assume knowledge of gϕg_ φ by pre-training the latent operator in a dedicated stage using paired data, we treat this setup as a blind inverse problem in the latent space: given only yz_y, estimate the clean latent code 0∼pzz_0 p_z together with the operator parameters ϕ φ and the residual matrix r. WaveformLatentxf++yn0z_0gϕg_ φ++yz_yrE(⋅)E(·)E(⋅)E(·) Figure 1: Parallel view of degradation in waveform and latent space. Clean signals are shown in blue and distorted signals in red. 2.2 Latent Diffusion Model Prior We use an unconditional LDM as a prior over the clean latent distribution p()p(z). Diffusion models define a simple forward process that gradually corrupts samples from the target distribution p()p(z) with Gaussian noise, and then learn to reverse this process by denoising [12]. Given a clean latent vector 0∼pzz_0 p_z and noise ∼(,) (0,I), the forward perturbation at any noise level τ∈[0,σmax]τ∈[0, _max] is defined as τ=0+τ.z_τ=z_0+τ . (3) Sampling from p()p(z) is then performed by approximately reversing this process: starting from σmax∼(,σmax2I)z_ _max (0,σ^2_maxI), the model progressively denoises toward τ=0τ=0. In the framework proposed by Karras et al. [15], the continuous reverse-time dynamics are characterized by the probability flow ordinary differential equation (PF-ODE): dτ=−τ∇τlogp(τ)dτ. dz_τ=-τ _z_τ p(z_τ)\, dτ. (4) The key quantity is the score function ∇τlogp(τ) _z_τ p(z_τ). At each noise level, it points toward regions of higher pτp_z_τ probability density. The score function is generally intractable, but it can be approximated with a neural network sθ(τ,τ)≈∇τlogp(τ)s_θ(z_τ,τ)≈ _z_τ p(z_τ) with parameters θ. A useful property of Gaussian perturbations is that the score function is directly related to the Minimum Mean Square Error denoiser [0∣τ]E[z_0 _τ]. We denote its learned approximation by ^θ(τ,τ)≈[0∣τ] z_θ(z_τ,τ) [z_0 _τ], a deep neural network with parameters θ. The score follows directly as ∇τlogp(τ)≈sθ(τ,τ)=^θ(τ,τ)−τ2. _z_τ p(z_τ)≈ s_θ(z_τ,τ)= z_θ(z_τ,τ)-z_τ^2. (5) In our setup, the denoiser is parameterized as ^θ(τ,τ)=cskip(τ)τ+cout(τ)Fθ(cin(τ)τ,τ), z_θ(z_τ,τ)=c_skip(τ)z_τ+c_out(τ)F_θ(c_in(τ)z_τ,τ), (6) where the scaling coefficients are defined as cin(τ)=cskip(τ)=1/(τ+1)c_in(τ)=c_skip(τ)=1/(τ+1) and cout(τ)=τ/(τ+1)c_out(τ)=τ/(τ+1), and FθF_θ a Diffusion Transformer (DiT) backbone [31]. We train FθF_θ with the Rectified Flow objective [22]: ℒ(θ)=0,ϵ,τ[‖Fθ(cin(τ)τ,τ)−(−0)‖F2].L(θ)=E_z_0,ε,τ [\|F_θ(c_in(τ)z_τ,τ)-( -z_0)\|_F^2 ]. (7) 2.3 Diffusion Posterior Sampling Diffusion models can also be used to solve inverse problems at test time, without additional training, and several strategies have been proposed [6]. In this work, we adopt Diffusion Posterior Sampling (DPS) [4], which targets the posterior distribution p(0∣y)p(z_0 _y). By Bayes’ rule, the posterior score decomposes into a prior term and a likelihood term: ∇τlogp(τ∣y)=∇τlogp(τ)+∇τlogp(y∣τ). _z_τ p(z_τ _y)= _z_τ\! p(z_τ)+ _z_τ\! p(z_y _τ). (8) The first term is the unconditional score, modeled by the diffusion prior s(τ,τ)s_ θ(z_τ,τ), which steers samples toward the clean latent manifold. The second term is a likelihood score that guides the reverse process toward posterior samples that remain consistent with the observation yz_y [4]. Although the likelihood score is generally intractable, it can be approximated using the forward model as ∇τlogp(y∣τ)≈l(y,τ;ϕ,) _z_τ p(z_y _τ)≈ l(z_y,z_τ; φ,r). We use the following approximation, similar to [4]: l(y,τ;ϕ,)≈−ζτ∇τ‖y−(gϕ(^(τ,τ))+)‖F2, -2.0ptl(z_y,z_τ; φ,r) -2.0pt≈ -2.0pt- _τ _z_τ\|z_y -1.0pt- -1.0pt (g_ φ( z_ θ(z_τ,τ)) -2.0pt+ -2.0ptr )\|_F^2, (9) where ζτ _τ is a noise-level dependent scaling factor. However, in our setting the latent operator parameters ϕ φ and the residual term r are unknown. We therefore infer them jointly with the clean latent through an alternating blind inference procedure, described next. 2.4 Distortion-Blind Inference We adopt an EM-style alternating inference algorithm for blind inverse problems along the reverse diffusion trajectory, following prior work on inverse problems [16, 18, 28]. At a high level, the goal is to estimate the unknown operator parameters by maximizing the expected log-likelihood of the observation under the posterior of the clean latent: (ϕ,)=argmaxϕ,p(0∣y)[logp(y∣0;ϕ,)].( φ,r)= _ φ,r\ E_p(z_0 _y)\! [ p(z_y _0;\, φ,r) ]. (10) In practice, the inference alternates between three steps, which are also shown in Figure 2. At each reverse-time step, we estimate the clean latent under the current operator (E-step), update the operator to better explain the observation (M-step), and then take one guided ODE step (update). τMz_ _Mτiz_ _iτi−1z_ _i-1τ0z_ _0DecoderD(⋅)D(·) x⋮ ⋮ (⋅)E(·)yz_y E-step ^θ(τi,τi) z_θ(z_ _i, _i) ^0(i) z_0^(i) M-step fit gϕ(^x(i))+g_ φ( z_x^(i))+r to yz_y Update τi−Δτi[θ(τi,τi)+l(y,τi;ϕ,)]z_ _i- _i\,[s_θ(z_ _i, _i)+l(z_y,z_ _i; φ,r)] ϕ, φ,r Figure 2: Iterative inference procedure. The observed latent yz_y is refined over M E-M update cycles to produce the final estimate τ0z_ _0, which is decoded into waveform x. E-step: At iteration i (running from M down to 0), we compute the clean-latent estimate via (6): ^0(i):=^(τi,τi) z_0^(i):= z_ θ(z_ _i, _i). Treating ^0(i) z_0^(i) as a Dirac approximation to the intractable posterior p(0∣y)p(z_0 _y) yields the surrogate log-likelihood: p(x∣y)[logp(y∣0;ϕi,i)]≈logp(y∣^0(i);ϕi,i),E_p(z_x _y)\! [ p(z_y -2.0pt -2.0ptz_0; φ_i,r_i) ]≈ p\! (z_y -2.0pt -2.0pt z_0^(i); φ_i,r_i ), (11) reducing the E-step to a single forward-pass evaluation rather than an explicit posterior sampling. M-step: With fixed estimate ^0(i) z_0^(i), we update the operator parameters and residual perturbation by optimizing: (ϕi+1,i+1)=argminϕ, ( φ_i+1,r_i+1)= _φ,r ‖y−(gϕ(^0(i))+)‖1 \|z_y-(g_ φ( z_0^(i))+r)\|_1 (12) +γϕ‖ϕ‖22+γ‖F2. + _ φ\| φ\|_2^2+ _r\|r\|_F^2. In practice, we optimize this objective with AdamW [23] and apply decoupled weight decay to both ϕ φ and r; the quadratic penalties above summarize this regularization. Update: The update towards the clean latent variable is an Euler step along the reverse dynamics: τi−1=τi−Δτi[sθ(zτi,τi)+l(y,τi;ϕi+1,i+1)],z_ _i-1=z_ _i- _ _i[s_θ(z_ _i, _i)+l(z_y,z_ _i; φ_i+1,r_i+1)], (13) where Δτi=(τi−τi−1) _ _i=( _i- _i-1) denotes the step size. Alternating these steps adapts the latent operator and residual while steering the trajectory toward the clean latent manifold. The restored signal is obtained via the decoder, ^=D(^θ(τ0,τ0)) x=D( z_θ(z_ _0, _0)), back to the waveform domain, as shown at the bottom of Figure 2. 2.5 Latent Operator Model ^0(i) z_0^(i)gϕ(^0(i))g_ φ( z_0^(i))time NNCCUUCausalConv1DVVReLU++UUCausalConv1DVV++repeated ×2× 2time NNCC Figure 3: Diagram of the proposed latent operator gϕg_ φ. A crucial design choice in LOUDAR is how to parameterize the unknown latent operator gϕg_ φ. The model should be flexible enough to represent a broad range of distortions, while remaining sufficiently constrained to avoid overfitting during each M-step optimization. This trade-off is especially important because blind optimization is performed along the reverse diffusion trajectory, where the intermediate clean-latent estimate ^0 z_0 can be inaccurate at early steps. Without explicit capacity control, the operator can overfit these transient estimates and destabilize the subsequent denoising updates. We implement gϕg_ φ as a causal convolutional network acting on latent matrices. Figure 3 summarizes the architecture. Convolutions along the temporal axis preserve translation equivariance over time, which matches many audio effects and distortions, while the temporal compression of the latent representation makes even a small receptive field correspond to a longer waveform context. In the default configuration used throughout this work, gϕg_ φ contains three bias-enabled low-rank causal Conv1D layers with kernel size 33 and channel size 6464. The first two layers are followed by ReLU nonlinearities, and residual connections are applied at every layer. To limit capacity, each layer is implemented through a rank-1212 intermediate channel space: instead of applying the temporal convolution directly at the full 64-channel dimensionality, the layer projects to 12 channels, performs the causal convolution in that reduced space, and projects back to 64 channels. The projection layers are denoted as U and V in Figure 3. The additive residual term r complements gϕg_ φ by capturing sample-specific mismatch that is not well explained by the shared convolutional operator. Unlike gϕg_ φ, however, r has no temporal structure, so it is more prone to absorbing noise or other interferences. Consequently, we treat r as a free optimizable matrix and apply a weight penalty to it, as well as to the weights ϕ φ, as described in (12). 3 Experiments and Results We evaluate LOUDAR on singing voice and guitar to test whether the same method transfers across substantially different musical material and remains competitive with representative baselines. For singing voice, we study removal of common vocal production effects (objective and subjective evaluation) and restoration from nonlinear distortions (subjective only). For guitar, we study recovery of the direct injection (DI) signal from amplifier-processed recordings. 3.1 Experimental Details Autoencoder. We adopt Music2Latent (M2L) [30], an AE based on a consistency model [38], as the latent representation employed in our experiments. This AE encodes 48 kHz audio waveforms to a temporal rate of 12 Hz with a channel size of 64, corresponding to a dimensionality reduction of approximately 64×64×. In preliminary experiments, M2L proved qualitatively more effective for our specific task than alternative variational AEs [9]. We use the M2L weights released by [42], obtained with a larger training dataset. In preliminary experiments, M2L proved qualitatively more effective than alternative Variational AEs [9], which we attribute to its lower dimensionality, favorable diffusability properties [36], and decoder robustness. Diffusion Model. The diffusion backbone, FθF_θ, is implemented as an unconditional Diffusion Transformer (DiT) [31]. The implementation is based on Stable Audio Open [10] and contains approximately 6868 M trainable parameters. All models operate on monaural audio segments of 526,080526,080 samples (approximately 1111 s at 4848 kHz). In the M2L representation, these segments correspond to 128×64128× 64 matrices. Models are trained by minimizing the objective in (7) using the AdamW optimizer [23] with a learning rate of 10−410^-4, weight decay of 0.010.01, (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999), and a batch size of 16. The training data varies by experimental condition, as detailed in the sections below. Inference Parameters. To ensure robustness across all experiments, we maintain a consistent hyperparameter configuration for our inference algorithm. We utilize M=300M=300 steps following the reverse-time schedule proposed in EDM [15], with boundary noise levels σmax=30 _max=30, σmin=10−4 _min=10^-4, and a curvature parameter ρ=7ρ=7. Furthermore, we incorporate stochasticity into the updates as described in [15] to mitigate generation artifacts in the clean estimates, setting Schurn=20S_churn=20. Following [26], the DPS scaling factor ζτ _τ is parameterized as: ζτ=ζ~CNτ‖G‖F _τ= ζ CNτ\|G\|_F, where G denotes the gradient of the measurement consistency term (9). We set ζ~=0.3 ζ=0.3 empirically to balance fidelity against generative expressiveness: lower values lead to hallucinations, higher values over-rely on the observation and degrade quality. The latent operator, as specified in Sec. 2.5, contains approximately 9.4k9.4\,k parameters. For its optimization, we use AdamW [23] with a learning rate of 10−410^-4 and a purposedly high weight decay of 11. Within each of the M inference steps, we perform 10 operator optimization iterations. 3.2 Evaluation Objective Metrics. We evaluate our framework using specialized embedding spaces rather than waveform-level comparisons, as the latter are often ill-suited for assessing generative models [11]. Furthermore, the AE can introduce phase differences; while these are often perceptually negligible, they may unfairly penalize some methods in sample-accurate evaluations. We utilize the embeddings AFx-Rep mid [39] and FxEncoder++ [46], both of which are explicitly trained to capture audio effect characteristics, alongside CLAP [44], which has demonstrated robust sensitivity to audio effects [3]. For singing voice tasks, we additionally incorporate a BYOL-based singer identity embedding [41] to monitor potential drift in vocal characteristics and ensure the preservation of the original performer’s identity. We report both pairwise and distributional metrics to capture different facets of performance. Pairwise performance is measured via the cosine distance (CD) between the embeddings of the clean reference and the model estimates, quantifying how accurately the model restores specific features for each individual input. To complement this, we evaluate the overall statistical consistency of the results using Kernel Audio Distance (KAD) [5]; while pairwise metrics focus on individual reconstruction accuracy, KAD assesses the alignment between the generated output manifold and the target ground truth distribution, ensuring the stylistic realism of the results. Lower values indicate better performance for all reported measures, with zero optimal. Subjective Listening Test. To validate the perceptual quality of the proposed method against several baselines, we conducted a formal listening test using the webMUSHRA framework [34]. The evaluation involved 13 volunteers recruited from the authors’ institutions, 11 of whom were experienced in subjective audio evaluation. Participants performed the test remotely using headphones in quiet environments. Following the MUSHRA protocol, each trial included a “dry” reference and a distorted anchor alongside the test conditions, which listeners rated on a scale from 0 to 100. The test was divided into two parts: a four-trial session for singing voice effect removal and a 12-trial session for severe nonlinear distortions. The evaluated material, along with a comprehensive analysis of the results, are detailed in the following section. Statistical significance was assessed using the Wilcoxon signed-rank test with a significance level of p<0.05p<0.05. 3.3 Singing Voice Effect Removal To evaluate the performance of LOUDAR in singing voice restoration, we train a diffusion model on the OpenSinger dataset [13], a 50-hour, high-quality multi-singer dataset of singers performing in Chinese. We partitioned the data by singer, reserving six singers for a held-out test set, two for validation, and utilizing the remaining singers for training. The model was trained for 375 k iterations. Benchmark. Our primary benchmark assesses the ability of the model to remove vocal effects common in modern music production. To simulate the effects, we utilize Diffvox [47], a differentiable audio effects pipeline which includes a parametric equalizer, a dynamic range compressor, a feedback delay network reverberator, and a ping-pong delay. DiffVox is particularly well-suited for this evaluation as it encompasses both nonlinear processing and challenging long-term temporal dependencies. We use 365 presets provided by the DiffVox authors, which represent reverse-engineered effect chains from professionally produced vocals. These presets were applied to the OpenSinger test set to create 365 unique 11-seconds-long examples, with a balanced representation of test singers. From this set, four random examples featuring different singers were included in the listening test. Baselines. We compare LOUDAR against a set of representative baselines, beginning with supervised models retrained on the OpenSinger dataset. To facilitate supervised training, we applied a distortion pipeline of randomized equalization, compression, and reverb, similar to [27]. The supervised baselines include: (1) a Latent Regressor, which utilizes the same transformer architecture as LOUDAR but minimizes a Euclidean objective to predict the clean embedding 0z_0 directly from the distorted yz_y; (2) an LDM Conditional, a latent diffusion transformer conditioned on yz_y via input concatenation; and (3) Apollo [19], a waveform-domain restoration model designed for MP3 restoration and retrained for this task. We further evaluate models pretrained on other datasets that are expected to solve the task. This includes RemFX [33], which employs a compositional restoration approach; SRS [49], a recent speech and singing voice restoration model trained on large curated datasets; and VoiceFixer [21], a general restoration model specialized for speech. Finally, we compare LOUDAR against BUDDy [18], an unsupervised waveform-domain model. While BUDDy uses an algorithmic framework similar to LOUDAR, its operator is primarily designed for linear room impulse responses, which may limit its effectiveness across the broader distortions present in this benchmark. Results. Objective metrics are reported in the upper part of Table 1, while listening test results are shown in Figure 4. A first observation is the apparent discrepancy between objective and perceptual evaluation. Apollo achieves the best overall scores across objective metrics (see Table 1), yet it is outperformed by LOUDAR in the listening test (see Figure 4), with a statistically significant difference (p=0.002p=0.002). A plausible explanation is that Apollo is not affected by the AE, whereas the objective metrics are sensitive to the distortions introduced by this component. This is supported by the fact that reconstructions obtained directly from the AE—which can be interpreted as an upper bound for LOUDAR—yield substantially higher metric values than the evaluated methods, suggesting that part of the gap is due to metric bias rather than perceptual quality. Looking more closely at representation-specific metrics, LOUDAR obtains the lowest scores on Singer-ID embeddings. This indicates that the method preserves speaker identity characteristics more faithfully. More broadly, when compared to the Latent Regressor and LDM conditional baselines, LOUDAR achieves overall comparable performance. It consistently outperforms both methods on AFxRep, CLAP, and Singer-ID metrics, considering both pairwise and distributional distances, although it does not match their performance on FxEnc++. Importantly, these improvements translate to perceptual gains, as LOUDAR significantly surpasses the LDM conditional model in the listening test (p<0.001p<0.001). A somewhat counterintuitive result is observed for SRS. While it scores surprisingly low on effect-based embeddings such as AFxRep and FxEnc++, it achieves the lowest CLAP distance and one of the best listening test scores (see Figure 4). Its perceptual performance is statistically comparable to LOUDAR (p=0.433p=0.433). RemFX and VoiceFixer exhibit the weakest performance overall, likely due to a mismatch between their training data and our task conditions. Finally, BUDDy, which operates in the waveform domain, is likewise outperformed by LOUDAR, consistent with the guitar experiment (Sec. 3.5) in supporting the benefit of restoring in latent space. Table 1: Objective metrics for both singing voice and guitar restoration experiments. Pairwise cosine distance ↓ Distributional distance (KAD) ↓ Experiment Method AFxRep FxEnc++ CLAP Singer-ID AFxRep FxEnc++ CLAP Singer-ID Singing Distorted 0.369±0.1650.369± 0.165 0.097±0.0660.097± 0.066 0.147±0.0680.147± 0.068 0.465±0.2290.465± 0.229 28.83 10.29 18.59 46.24 voice SRS 0.497±0.1220.497± 0.122 0.080±0.0320.080± 0.032 0.067±0.0370.067± 0.037 0.059±0.0250.059± 0.025 72.02 16.22 6.49 9.76 VoiceFixer 0.373±0.1050.373± 0.105 0.107±0.0480.107± 0.048 0.121±0.0390.121± 0.039 0.185±0.0920.185± 0.092 37.77 16.58 14.60 21.91 BUDDy 0.334±0.1490.334± 0.149 0.083±0.0450.083± 0.045 0.091±0.0480.091± 0.048 0.113±0.0770.113± 0.077 23.90 11.68 7.59 7.79 RemFx 0.322±0.1540.322± 0.154 0.083±0.0560.083± 0.056 0.110±0.0520.110± 0.052 0.268±0.1750.268± 0.175 23.33 7.19 11.44 27.28 LDM conditional 0.183±0.0860.183± 0.086 0.050±0.0220.050± 0.022 0.080±0.0340.080± 0.034 0.056±0.0300.056± 0.030 15.80 5.24 9.06 5.18 Latent regressor 0.179±0.0860.179± 0.086 0.053±0.0260.053± 0.026 0.090±0.0440.090± 0.044 0.065±0.0540.065± 0.054 18.21 7.42 11.18 7.20 Apollo 0.146±0.0860.146± 0.086 0.045±0.0380.045± 0.038 0.056±0.0310.056± 0.031 0.067±0.0330.067± 0.033 5.10 1.56 3.18 9.54 LOUDAR (ours) 0.184±0.1020.184± 0.102 0.063±0.0260.063± 0.026 0.076±0.0350.076± 0.035 0.048±0.0250.048± 0.025 14.36 6.04 6.81 3.45 AE reconstruction 0.100±0.0680.100± 0.068 0.026±0.0120.026± 0.012 0.047±0.0210.047± 0.021 0.013±0.0060.013± 0.006 14.54 4.96 4.39 1.75 Guitar Distorted 0.713±0.1300.713± 0.130 0.381±0.1810.381± 0.181 0.242±0.0950.242± 0.095 56.49 33.43 43.06 Apollo 0.780±0.1160.780± 0.116 0.433±0.1890.433± 0.189 0.244±0.0920.244± 0.092 62.21 41.13 34.30 UnAFx 0.619±0.1610.619± 0.161 0.374±0.1850.374± 0.185 0.245±0.0670.245± 0.067 50.87 33.53 48.39 Latent regressor 0.549±0.1140.549± 0.114 0.323±0.1420.323± 0.142 0.242±0.0660.242± 0.066 39.62 22.21 35.99 LDM conditional 0.547±0.1250.547± 0.125 0.330±0.1420.330± 0.142 0.237±0.0690.237± 0.069 35.26 20.69 33.40 LOUDAR (ours) 0.547±0.1200.547± 0.120 0.308±0.1420.308± 0.142 0.219±0.0670.219± 0.067 36.00 19.96 33.28 AE reconstruction 0.203±0.0590.203± 0.059 0.105±0.0630.105± 0.063 0.152±0.0560.152± 0.056 12.70 6.95 21.78 Figure 4: Results of the MUSHRA listening test on singing voice effect removal. 3.4 Singing Voice Distortion Restoration Figure 5: MUSHRA scores for experiment of Sec. 3.4. Additionally, we evaluate LOUDAR on restoring singing voices subjected to three aggressive distortions processed via Pedalboard [37] (see Figure 5). To assess generalization across datasets, we evaluate the diffusion prior trained on OpenSinger (Chinese) using English samples from the NHSS dataset [35]. The second session of the listening test included 12 such examples (four per distortion). We compare LOUDAR against two restoration baselines: SRS and VoiceFixer. As shown in Figure 5, LOUDAR exhibits the most stable performance across all conditions. This experiment suggests that SRS and VoiceFixer struggle with distortions potentially underrepresented in their training data. For instance, SRS performs poorly on clipping and bitcrushing and is outperformed by LOUDAR in GSM compression (p=0.045p=0.045). Similarly, while VoiceFixer matches LOUDAR’s scores on clipping (p=0.191p=0.191), it is significantly outperformed by LOUDAR in bitcrushing (p<0.001p<0.001) and GSM compression (p=0.008p=0.008). 3.5 Guitar Distortion Removal We further evaluate LOUDAR on dry guitar recovery from amplifier-distorted recordings. We train on the GOAT dataset [24], which contains paired DI and effected guitar recordings spanning a wide range of guitar amplifier transformations. For LOUDAR, only the DI recordings are used to train the diffusion prior, whereas the supervised baselines are trained with paired distorted/dry guitar data. For evaluation we use 238 examples from EGDB [2], balanced across five amplifier conditions. Following the protocol used for singing voice, we compare LOUDAR against the same supervised baselines, retrained using the paired data in GOAT. Additionally, we evaluate UnAFx [28], an unsupervised baseline similar to LOUDAR that fits a convolutional neural network as a prior, but operates in the waveform domain. All models are trained for a total of 175 k iterations. Objective results are reported in Table 1. Across the FxEnc++ and CLAP embeddings, LOUDAR achieves the strongest results. On AFxRep, LOUDAR remains close to LDM Conditional, which performs slightly better in KAD and comparably in CD. Apollo transfers poorly in this setting. LOUDAR clearly outperforms UnAFx, suggesting that dry-guitar recovery is easier to optimize in latent space than with an unsupervised waveform-domain baseline. 4 Conclusions This paper presents LOUDAR, an unsupervised framework for music restoration under arbitrary distortions. Rather than learning a fixed inverse model for a predefined distortion family, LOUDAR formulates restoration as joint inference of a clean latent vector and a latent operator, regularized by an unconditional LDM prior. Operating in the latent space of an AE makes this inference tractable while preserving broad applicability to complex distortions. Across our experiments on singing voice and guitar, LOUDAR consistently improves over distorted inputs and compares favorably with strong baselines, showing the clearest gains where supervised models struggle to generalize. Despite the promising results, LOUDAR inherits important limitations from the latent representation on which it relies. Since restoration is performed entirely in the latent space of an AE, the best attainable output quality is bounded by that reconstruction fidelity of the autoencoder. Moreover, LOUDAR also depends on the encoding of the distorted observation itself: if this is mapped unreliably into latent space, the subsequent restoration can become unpredictable. This limitation becomes more pronounced for signals that are weakly represented by the encoder. A second limitation arises from prior mismatch. The diffusion prior may favor samples that are plausible under its training distribution rather than faithfully recovering the original signal, potentially introducing corpus-specific timbral or phonetic artifacts. For instance, heavily distorted English singing can acquire characteristics of the Chinese singing data used to train the prior. Furthermore, LOUDAR is sensitive to strong measurement noise. While it can handle mild interference, performance degrades when noise dominates the observation, as the method is designed to project inputs toward a clean-audio manifold rather than explicitly separate large additive noise components. A natural direction for future work is to combine LOUDAR with discriminative denoising approaches to improve robustness in such conditions. 5 Acknowledgments This study was supported by the Czech Science Foundation (Project No. 23-07294S), the Research Council of Finland (Grant no. 371845, REMUS—Restoring Music Recordings Using Generative Models), and the HUCE infrastructure of the Aalto School of Electrical Engineering. We acknowledge the computational resources from the Aalto University “Science-IT” project. We thank all anonymous participants of the listening tests. 6 Ethics Statement All responses from the listening test participants were collected anonymously — no names or other identifying information were recorded. References [1] D. Bralios, P. Smaragdis, and J. Casebeer (2025) Learning to upsample and upmix audio in the latent domain. In Proc. IEEE WASPAA, p. 1–5. Cited by: §1. [2] Y. Chen, W. Hsiao, T. Hsieh, J. R. Jang, and Y. Yang (2022) Towards automatic transcription of polyphonic electric guitar music: a new dataset and a multi-loss transformer model. In Proc. IEEE ICASSP, Vol. , p. 786–790. External Links: Document Cited by: §3.5. [3] A. Chu, P. O’Reilly, J. Barnett, and B. Pardo (2025) Text2FX: harnessing CLAP embeddings for text-guided audio effects. In Proc. IEEE ICASSP, p. 1–5. Cited by: §3.2. [4] H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye (2023) Diffusion posterior sampling for general noisy inverse problems. In Proc. ICLR, Cited by: §2.3, §2.3. [5] Y. Chung, P. Eu, J. Lee, K. Choi, J. Nam, and B. S. Chon (2025) KAD: No more FAD! An effective and efficient evaluation metric for audio generation. AI Heard That! ICML 2025 Workshop on Machine Learning for Audio. Cited by: §3.2. [6] G. Daras, H. Chung, C. Lai, Y. Mitsufuji, J. C. Ye, P. Milanfar, A. G. Dimakis, and M. Delbracio (2024) A survey on diffusion models for inverse problems. arXiv preprint arXiv:2410.00083. Cited by: §2.3. [7] T. Dhyani, F. Lux, M. Mancusi, G. Fabbro, F. Hohl, and N. T. Vu (2025) High-resolution speech restoration with latent diffusion model. In Proc. IEEE ICASSP, p. 1–5. Cited by: §1. [8] S. Doh, J. Koo, M. A. Martínez-Ramírez, W. Choi, W. Liao, Q. Wu, J. Nam, and Y. Mitsufuji (2026) LLM2Fx-Tools: tool calling for music post-production. In Proc. ICLR, Cited by: §1. [9] Z. Evans, C. Carr, J. Taylor, S. H. Hawley, and J. Pons (2024) Fast timing-conditioned latent audio diffusion. In Proc. ICML, Cited by: §1, §3.1. [10] Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons (2025) Stable audio open. In Proc. IEEE ICASSP, Vol. , p. 1–5. External Links: Document Cited by: §3.1. [11] A. Gui, H. Gamper, S. Braun, and D. Emmanouilidou (2024) Adapting Frechet audio distance for generative music evaluation. In Proc. IEEE ICASSP, Vol. , p. 1331–1335. Cited by: §3.2. [12] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Proc. NeurIPS 33, p. 6840–6851. Cited by: §2.2. [13] R. Huang, F. Chen, Y. Ren, J. Liu, C. Cui, and Z. Zhao (2021) Multi-singer: fast multi-singer singing voice vocoder with a large-scale corpus. In Proc. 29th ACM International Conference on Multimedia, p. 3945–3954. Cited by: §3.3. [14] J. Imort, G. Fabbro, M. A. M. Ramírez, S. Uhlich, Y. Koyama, and Y. Mitsufuji (2022) Distortion audio effects: learning how to recover the clean signal. In Proc. 23rd ISMIR, p. 218–225. External Links: Document Cited by: §1. [15] T. Karras, M. Aittala, T. Aila, and S. Laine (2022) Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, Cited by: §2.2, §3.1. [16] C. Laroche, A. Almansa, and E. Coupete (2024) Fast diffusion EM: A diffusion model for blind inverse problems with application to deconvolution. In Proc. IEEE/CVF Winter Conf. Applications of Computer Vision, p. 5271–5281. Cited by: §2.4. [17] S. Lee, J. Park, S. Paik, and K. Lee (2023) Blind estimation of audio processing graph. In Proc. IEEE ICASSP, p. 1–5. Cited by: §1. [18] J. Lemercier, E. Moliner, S. Welker, V. Välimäki, and T. Gerkmann (2025) Unsupervised blind joint dereverberation and room acoustics estimation with diffusion models. IEEE Transactions on Audio, Speech and Language Processing 33, p. 2244–2258. Cited by: §1, §1, §1, §2.4, §3.3. [19] K. Li and Y. Luo (2025) Apollo: band-sequence modeling for high-quality audio restoration. In Proc. IEEE ICASSP, p. 1–5. Cited by: §1, §3.3. [20] H. Liu, Z. Chen, Y. Yuan, et al. (2023-Jul.) AudioLDM: text-to-audio generation with latent diffusion models. In Proc. ICML, Vol. 202, p. 21450–21474. Cited by: §1. [21] H. Liu, Q. Kong, Q. Tian, Y. Zhao, D. Wang, C. Huang, and Y. Wang (2021) VoiceFixer: toward general speech restoration with neural vocoder. arXiv:2109.13731 [cs.SD]. Cited by: §3.3. [22] X. Liu, C. Gong, and Q. Liu (2022) Flow straight and fast: learning to generate and transfer data with rectified flow. In Proc. NeurIPS 2022 Workshop on Score-Based Methods, Cited by: §2.2. [23] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In Proc. ICLR, Cited by: §2.4, §3.1, §3.1. [24] J. Loth, P. Sarmento, S. Sarkar, Z. Guo, M. Barthet, and M. Sandler (2025) GOAT: a large dataset of paired guitar audio recordings and tablatures. In Proc. ISMIR, p. 669–676. Cited by: §3.5. [25] M. A. Martínez-Ramírez, W. Liao, G. Fabbro, S. Uhlich, C. Nagashima, and Y. Mitsufuji (2022) Automatic music mixing with deep learning and out-of-domain data. In Proc. ISMIR, p. 411–418. External Links: Document Cited by: §1. [26] E. Moliner, J. Lehtinen, and V. Välimäki (2023) Solving audio inverse problems with a diffusion model. In Proc. IEEE ICASSP, p. 1–5. Cited by: §3.1. [27] E. Moliner, M. A. Martínez-Ramírez, J. Koo, W. Liao, K. W. Cheuk, J. Serrà, V. Välimäki, and Y. Mitsufuji (2026) Automatic music mixing using a generative model of effect embeddings. In Proc. IEEE ICASSP, p. 14582–14586. Cited by: §3.3. [28] E. Moliner, M. Švento, A. Wright, L. Juvela, P. Rajmic, and V. Välimäki (2025) Unsupervised estimation of nonlinear audio effects: comparing diffusion-based and adversarial approaches. In Proc. DAFx, Cited by: §1, §1, §2.4, §3.5. [29] E. Moliner, M. Turunen, F. Elvander, and V. Välimäki (2024) A diffusion-based generative equalizer for music restoration. In Proc. DAFx, p. 1–8. Cited by: §1, §1. [30] M. Pasini, S. Lattner, and G. Fazekas (2024) Music2Latent: consistency autoencoders for latent audio compression. In Proc. ISMIR, p. 111–119. Cited by: §3.1. [31] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proc. IEEE/CVF Int. Conf. Computer Vision, p. 4195–4205. Cited by: §2.2, §3.1. [32] R. Raphaeli, S. Man, and M. Elad (2025) SILO: solving inverse problems with latent operators. In Proc. IEEE/CVF ICCV, p. 10570–10580. Cited by: §1, §2.1, §2.1. [33] M. Rice, C. J. Steinmetz, G. Fazekas, and J. D. Reiss (2023) General purpose audio effect removal. In Proc. IEEE WASPAA, p. 1–5. Cited by: §1, §3.3. [34] M. Schoeffler, S. Bartoschek, F. Stöter, M. Roess, S. Westphal, B. Edler, and J. Herre (2018-Feb.) webMUSHRA—A comprehensive framework for web-based listening tests. J. Open Research Software 6, p. . External Links: Document Cited by: §3.2. [35] B. Sharma, X. Gao, K. Vijayan, X. Tian, and H. Li (2021) NHSS: a speech and singing parallel database. Speech Communication 133, p. 9–22. External Links: ISSN 0167-6393 Cited by: §3.4. [36] I. Skorokhodov, S. Girish, B. Hu, W. Menapace, Y. Li, R. Abdal, S. Tulyakov, and A. Siarohin (2025) Improving the diffusability of autoencoders. In Proc. ICML, p. 55876–55905. Cited by: §3.1. [37] P. Sobot (2021-07) Pedalboard. Zenodo. External Links: Document, Link Cited by: §3.4. [38] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever (2023) Consistency models. In Proc. ICML, p. 32211–32252. Cited by: §3.1. [39] C. J. Steinmetz, S. Singh, M. Comunità, I. Ibnyahya, S. Yuan, E. Benetos, and J. D. Reiss (2024) ST-ITO: controlling audio effects for style transfer with inference-time optimization. In Proc. ISMIR, p. 661–668. Cited by: §3.2. [40] O. Take, K. Watanabe, T. Nakatsuka, T. Cheng, T. Nakano, M. Goto, S. Takamichi, and H. Saruwatari (2024) Audio effect chain estimation and dry signal recovery from multi-effect-processed musical signals. In Proc. DAFx, p. 1–8. Cited by: §1. [41] B. Torres, S. Lattner, and G. Richard (2023) Singer identity representation learning using self-supervised techniques. In Proc. ISMIR, p. 448–456. Cited by: §3.2. [42] B. Torres, M. Moussallam, and G. Meseguer-Brocal (2026) Learning linearity in audio consistency autoencoders via implicit regularization. In Proc. IEEE ICASSP, p. 15927–15931. Cited by: §3.1. [43] Y. Wang, Z. Ju, X. Tan, L. He, Z. Wu, J. Bian, et al. (2023) Audit: audio editing by following instructions with latent diffusion models. Proc. NeurIPS, p. 71340–71357. Cited by: §1. [44] Y. Wu, K. Chen, T. Zhang, et al. (2023) Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In Proc. IEEE ICASSP, Cited by: §3.2. [45] Z. Xu, X. Fan, Z. Wang, X. Jiang, and R. R. Choudhury (2025) ArrayDPS: unsupervised blind speech separation with a diffusion prior. In Proc. ICML, Cited by: §1. [46] Y. Yeh, J. Koo, M. A. Martínez-Ramírez, W. Liao, Y. Yang, and Y. Mitsufuji (2025) Fx-Encoder++: extracting instrument-wise audio effects representations from mixtures. In Proc. ISMIR, p. 612–620. Cited by: §3.2. [47] C. Yu, M. A. Martínez-Ramírez, J. Koo, B. Hayes, W. Liao, G. Fazekas, and Y. Mitsufuji (2025) DiffVox: a differentiable model for capturing and analysing vocal effects distributions. In Proc. DAFx, Cited by: §3.3. [48] Y. Zang, Z. Dai, M. D. Plumbley, and Q. Kong (2025) Music source restoration. In Proc. IEEE MMSP, p. 138–143. Cited by: §1. [49] Y. Zang, C. Manchester, D. Young, I. Ivanov, J. Lufkin, M. Vladimirov, P. Solomon, S. Kepchelev, F. Y. Chen, D. Cai, et al. (2025) Smule renaissance small: efficient general-purpose vocal restoration. Technical report Cited by: §3.3.