Paper deep dive
Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution
Xue Wu, Kang Zhao, Kafeng Wang, Jianfei Chen, Jingwei Xin, Nannan Wang, Xinbo Gao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 7/20/2026, 3:33:25 AM
Summary
The paper proposes Difficulty-aware Dynamic Routing (DDR-SR), a framework for real-world image super-resolution that adapts processing capacity based on input difficulty. It uses a high-frequency energy attenuation metric to estimate restoration difficulty and routes images to one of two expert networks: Expert-D4 (high fidelity, low compression VAE) for hard cases and Expert-D8 (high efficiency, high compression VAE) for easy cases. This approach balances reconstruction quality and computational efficiency, outperforming static one-size-fits-all diffusion-based methods.
Entities (20)
Relation Signals (19)
Difficulty Estimator → calculatesmetric → High-Frequency Energy Attenuation
confidence 95% · we quantify restoration difficulty using high-frequency energy attenuation
DDR-SR → uses → High-Frequency Energy Attenuation
confidence 95% · we adopt high-frequency energy attenuation as a quantitative metric to assess its restoration difficulty
DDR-SR → uses → Difficulty Estimator
confidence 95% · we first design a difficulty estimator to predict the restoration cost of each input image
DDR-SR → routesto → Expert-D4
confidence 92% · challenging samples are routed to a model equipped with a VAE featuring a low spatial compression ratio
DDR-SR → routesto → Expert-D8
confidence 92% · easier samples are assigned to a model using a VAE with a higher compression ratio
Expert-D8 → uses → VAE-D8
confidence 92% · use the VAE from SD 2.1-base as the VAE in Expert-D8
Expert-D4 → uses → VAE-D4
confidence 92% · we adopt the VAE-D4 from (Yi et al., 2025) as the VAE in Expert-D4
DDR-SR → improves → Real-ISR
confidence 90% · Extensive experiments have demonstrated the superior efficiency and effectiveness of the proposed model compared to recent state-of-the-art methods.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Diffusion-based methods have achieved impressive performance in real-world image super-resolution (Real-ISR) by leveraging large pre-trained stable diffusion (SD) models as powerful generative priors. However, these methods still face two key limitations. First, existing SD-based one-step and multi-step Real-ISR approaches adopt a unified processing paradigm for all input samples, ignoring the varying restoration difficulty across images. Second, the aggressive resolution reduction of the VAE in SD models (e.g., 8x downsampling) leads to irreversible loss of fine-scale details, which cannot be recovered by the subsequent diffusion process. To address these limitations, we propose a Difficulty-aware Dynamic Routing (DDR) strategy that overcomes the rigid, one-size-fits-all processing paradigm. Specifically, we first design a difficulty estimator to predict the restoration cost of each input image, enabling automatic assignment to a network of appropriate capacity. Then, we construct a set of Real-ISR networks with varying model capacities by modulating the spatial downsampling ratio of the VAE in the SD backbone, thereby preserving more high-frequency information for challenging cases while maintaining efficiency for simpler inputs. Extensive experiments have demonstrated the superior efficiency and effectiveness of the proposed model compared to recent state-of-the-art methods.
Tags
Links
- Source: https://arxiv.org/abs/2607.15711v1
- Canonical: https://arxiv.org/abs/2607.15711v1
Trouble viewing inline? Open PDF directly →
Full Text
49,905 characters extracted from source content.
Expand or collapse full text
Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution Xue Wu 1 Kang Zhao 2 Kafeng Wang 2 Jianfei Chen 2 Jingwei Xin 1 Nannan Wang 1 Xinbo Gao 1 Abstract Diffusion-based methods have achieved impres- sive performance in real-world image super- resolution (Real-ISR) by leveraging large pre- trained stable diffusion (SD) models as power- ful generative priors. However, these methods still face two key limitations. First, existing SD-based one-step and multi-step Real-ISR ap- proaches adopt a unified processing paradigm for all input samples, ignoring the varying restoration difficulty across images. Second, the aggressive resolution reduction of the VAE in SD models (e.g., 8× downsampling) leads to irreversible loss of fine-scale details, which cannot be recovered by the subsequent diffusion process. To address these limitations, we propose a Difficulty-aware Dynamic Routing (DDR) strategy that overcomes the rigid, one-size-fits-all processing paradigm. Specifically, we first design a difficulty estimator to predict the restoration cost of each input image, enabling automatic assignment to a network of appropriate capacity. Then, we construct a set of Real-ISR networks with varying model capacities by modulating the spatial downsampling ratio of the VAE in the SD backbone, thereby preserving more high-frequency information for challenging cases while maintaining efficiency for simpler in- puts. Extensive experiments have demonstrated the superior efficiency and effectiveness of the proposed model compared to recent state-of-the- art methods. 1. Introduction Single image super-resolution (SR) (Wang et al., 2020) is a classical yet still active low-level vision problem, aiming to reconstruct a high-resolution (HR) image from its low- resolution (LR) counterpart, which suffers from unknown 1 Xidian University, Xi’an 710071, Shaanxi, China. 2 Tsinghua University, Beijing, China. Correspondence to: Nannan Wang <nnwang@xidian.edu.cn.>. Preprint. July 20, 2026. Image Latent Diffusion Network Gaussian Noise ControlNet Decoder 64 × 64 512 × 512 Latent Encoder 512 × 512 64 × 64 (a) Image Latent Diffusion Network Encoder 512 × 512 64 × 64 Decoder 64 × 64 512 × 512 Latent (b) Decoder Latent Decoder Image Latent Diffusion Network Difficulty Estimator ? Latent Diffusion Network 128 × 128512 × 512 64 × 64 512 × 512 Latent Encoder 512 × 512 64 × 64 Encoder 128 × 128512 × 512 (c) Figure 1. Architecture comparison of diffusion-based SR ap- proaches. (a) Controller-based methods employ conditional mech- anisms to guide the diffusion process; (b) Standard diffusion pro- cesses latent representations directly; (c) Our proposed DDR em- ploys difficulty-aware dynamic routing to adaptively select expert networks with varying VAE compression ratios. degradations. SR is also a challenging ill-posed problem, as multiple plausible HR images may correspond to the same LR input. This problem has attracted long-standing and widespread attention in the computer vision community (Yang et al., 2010; Dong et al., 2015). Over the past decade, numerous seminal studies have been conducted to tackle this challenge, utilizing convolutional Neural Networks (CNNs) (Dong et al., 2016; Kim et al., 2016; Lim et al., 2017), vision transformers (ViTs) (Liang et al., 2021; Chen et al., 2023a), and their combinations (Chen et al., 2023b). Despite the remarkable progress achieved, these methods mostly fail in real-world scenarios. The failure stems from the fact that these traditional SR approaches assume a known degradation process, thereby failing to account for the unknown and complex degrada- tions present in real-world low-quality images (Wang et al., 2021a; Zhang et al., 2021). Consequently, real-world im- 1 arXiv:2607.15711v1 [cs.CV] 17 Jul 2026 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution age super-resolution (Real-ISR) is designed to reconstruct perceptually realistic HR images in real-world scenarios, thereby offering greater practical utility (Cai et al., 2019a). In recent years, Real-ISR has attracted increasing attention from researchers. Many researchers have focused on the use of generative models, particularly generative adversarial networks (GANs), for Real-ISR, owing to their ability to recover sharp and photorealistic textures from low-quality inputs via adversarial training and efficient one-step infer- ence (Ledig et al., 2017; Wang et al., 2021b). However, GAN-based methods suffer from inherent limitations, in- cluding training instability, mode collapse, and a limited generative prior, which often result in hallucinated details and perceptually unpleasant artifacts under complex real- world degradations (Goodfellow et al., 2020). Recently, diffusion models (DMs) have emerged as a pow- erful class of generative models and have shown impressive performance in image generation as well as many down- stream tasks such as Real-ISR. Their robust priors empower them to produce more realistic images with richer details than GAN-based methods (Ho et al., 2020; Rombach et al., 2022; Saharia et al., 2022). Based on pre-trained stable dif- fusion (SD) models (Rombach et al., 2022), many Real-ISR methods (Wang et al., 2023b; Yang et al., 2024; Yu et al., 2024; Wu et al., 2024b; Lin et al., 2024) have been proposed to improve the realistic details of SR outputs. Representative works such as StableSR (Wang et al., 2023b), PASD (Yang et al., 2024), SUPIR (Yu et al., 2024), and SeeSR (Wu et al., 2024b) take the LR input as the control signal and utilize ControlNet (Zhang et al., 2023) to guide the SR process. However, by typically initializing the diffusion process with random noise, these methods not only introduce unwanted randomness into the output images but also require multiple denoising steps to reconstruct the final high-quality result, rendering the Real-ISR process computationally expensive (Song et al., 2021). Very recently, to accelerate the gen- eration process of Real-ISR, one-step SD-based methods (Wang et al., 2024a; Xie et al., 2024b; Li et al., 2024; Yin et al., 2024b) have been developed. These methods directly take LR images as input and use distillation techniques to fine-tune the denoising process, presenting a promising approach for efficient and realistic super-resolution under real-world degradations. Despite these advancements, our investigation has revealed two critical limitations associated with existing DM-based Real-ISR methods. Firstly, existing DM-based Real-ISR methods predominantly focus on architectural design and training objective optimization. As shown in Figure 1(a) and (b), once trained, these models are typically applied in a uniform manner to all input samples, regardless of their intrinsic degradation characteristics or restoration dif- ficulty. However, we observe that real-world low-quality images exhibit significant variability in degradation com- plexity, and little attention has been paid to dynamically allocating different SR networks or inference strategies ac- cording to the difficulty of the input samples. This limitation motivates the exploration of adaptive Real-ISR frameworks that can more efficiently and effectively handle diverse real- world degradations (Kong et al., 2021; Liu et al., 2022). Secondly, the aforementioned multi-step and one-step SD- based approaches encode LR inputs into a latent space using a pretrained variational autoencoder (VAE) (Rombach et al., 2022) and either directly feed these latents into the denois- ing UNet or use them to condition the denoising process through a controller mechanism. However, the VAE’s ag- gressive downsampling (e.g., 8× in SD 2.1-base (Rombach et al., 2022)) inevitably discards high-frequency details (Zhu et al., 2023). Moreover, since VAEs are trained on high- quality images, encoding low-quality LR inputs often de- grades structural fidelity (Yi et al., 2025). Consequently, these methods struggle to reconstruct complex textures and fine patterns. To address these limitations, we propose an efficient SD- based Difficulty-aware Dynamic Routing (DDR) approach, coined as DDR-SR, which can adaptively assign an appropri- ate inference model to each sample based on its restoration difficulty, thereby achieving fine-structure-preserving Real- ISR while balancing reconstruction performance and effi- ciency. Figure 1(c) illustrates the overall framework. In im- age restoration tasks, the recovery difficulty of an input im- age is primarily determined by the degree of high-frequency structural information loss. Accordingly, we adopt high- frequency energy attenuation as a quantitative metric to assess its restoration difficulty (Ma et al., 2020). To en- able targeted processing of inputs with varying complexity, we design a set of specialized expert networks, each tai- lored to a specific range of image difficulty. Specifically, challenging samples are routed to a model equipped with a VAE featuring a low spatial compression ratio, which preserves richer fine-grained details in the latent space (Yi et al., 2025). In contrast, easier samples are assigned to a model using a VAE with a higher compression ratio, priori- tizing inference efficiency without compromising perceptual quality. This dynamic allocation strategy not only avoids over-processing simple inputs but also ensures that complex images receive sufficient modeling capacity, leading to a more optimal trade-off between visual fidelity and compu- tational cost. Moreover, the flexible design of DDR allows users to explicitly control the trade-off between visual fi- delity and inference efficiency by adjusting the routing ratio between high- and low-capacity expert networks, thereby enabling application-specific customization. Extensive ex- periments demonstrate that our DDR method outperforms state-of-the-art SD-based Real-ISR methods in achieving a better trade-off between reconstruction performance and inference efficiency. In summary, our contributions are as 2 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution follows: •We propose DDR-SR, the first difficulty-aware dy- namic routing framework for SD-based Real-ISR, which moves beyond the conventional one-size-fits- all paradigm by adaptively selecting inference mod- els based on restoration difficulty to enable sample- adaptive processing. •We design a set of VAE-modulated expert networks with varying spatial compression ratios, which jointly preserve fine structural details for challenging images and ensure efficient restoration for simpler inputs. •Extensive experiments demonstrate that DDR achieves a superior balance between reconstruction performance and inference efficiency, outperforming state-of-the-art SD-based Real-ISR approaches while offering flexible customization for diverse application scenarios. 2. Related Work 2.1. Multi-step Diffusion-based Real-ISR Diffusion models have revolutionized generative image mod- eling with their robust priors and high-quality outputs (Ho et al., 2020; Rombach et al., 2022). A significant line of research leverages pre-trained Text-to-Image (T2I) diffu- sion models, such as Stable Diffusion (SD) (Rombach et al., 2022), as powerful generative priors for the Real-ISR task. These methods typically use the Low-Resolution (LR) im- age as a conditioning signal to guide a multi-step iterative denoising process, achieving impressive perceptual quality. StableSR (Wang et al., 2023b) fine-tunes a time-aware en- coder and employs controllable feature wrapping modules to balance fidelity and realism. DiffBIR (Lin et al., 2024) adopts a two-stage strategy, first using a reconstruction net- work for preliminary restoration and then employing SD for detail enhancement. To better harness the generative power of diffusion models, several works introduce more sophisti- cated conditioning mechanisms. SeeSR (Wu et al., 2024b) extracts semantic information from the image to provide high-level guidance. PASD (Yang et al., 2024) introduces a pixel-aware cross-attention module for local structure per- ception and a degradation removal module to extract robust features. SUPIR (Yu et al., 2024) scales up model and data size, utilizing negative prompts and restoration-guided sampling to achieve state-of-the-art generative and fidelity capabilities. Recent works continue to explore new fron- tiers: DiffSteISR (Xie et al., 2024a) extends diffusion prior to stereo image super-resolution, addressing view consis- tency, while EPDiff (Zhang et al., 2025) and ConsisSR (Sun et al., 2025) focus on integrating stronger semantic priors and improving pixel-level consistency, respectively. How- ever, a fundamental limitation of all these methods is their reliance on iterative denoising (often 20-50 steps), result- ing in high computational cost and slow inference, which hinders practical deployment (Li et al., 2024). 2.2. One-step Diffusion-based Real-ISR To overcome the speed bottleneck of multi-step diffusion, there is a growing interest in distilling these models into effi- cient one-step generators (Yin et al., 2024a; Liu et al., 2023; Salimans & Ho, 2022). This line of work aims to achieve real-time or near-real-time Real-ISR while preserving gener- ative quality. SinSR (Wang et al., 2024a) applies consistency distillation to compress the multi-step ResShift (Yue et al., 2024) model into a single step, though its generalization is limited by training data scale. AddSR (Xie et al., 2024b) introduces Adversarial Diffusion Distillation (ADD) (Sauer et al., 2025) to Real-ISR, producing a fast four-step model that can sometimes generate excessive details. OSEDiff (Li et al., 2024) proposes a more direct approach by using the LR image as the starting point of the diffusion process (instead of random noise) and employs Variational Score Distillation (VSD) loss (Wang et al., 2024b) for regular- ization. To address the common issue of spurious details in one-step models, StructSR (Li et al., 2025) proposes a plug-and-play, inference-time mechanism that suppresses artifacts and enhances structural fidelity. Beyond single images, the one-step paradigm is also being explored for video; DLoRAL (Zhang et al., 2024b) employs a dual-LoRA design to decouple temporal consistency and spatial detail learning for efficient video super-resolution. While these methods significantly improve speed, they still operate un- der a static, “one-size-fits-all” inference paradigm, applying the same network capacity to all inputs regardless of their restoration difficulty (Wu et al., 2024b). 3. Methodology 3.1. Motivation To better understand the impact of input difficulty on exist- ing methods, we conduct a systematic evaluation of state- of-the-art one-step SD-based Real-ISR approaches across samples of varying restoration complexity. Using our high- frequency energy attenuation metric, we estimate the dif- ficulty of all images in the RealSR (Cai et al., 2019b) and DRealSR (Wei et al., 2020a) benchmarks and split each dataset into “easy” and “hard” subsets based on the me- dian difficulty score. As shown in Figure 2, we compare the PSNR and SSIM performance of three representative SOTA methods (OSEDiff (Wu et al., 2024a), S3Diff(Zhang et al., 2024a), and TVTSR(Yi et al., 2025)) on both sub- sets. Strikingly, we find that while performance gaps among competing methods are relatively small on the easy subset, indicating that most models can adequately handle mild degradations, these gaps become significantly pronounced 3 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution Image Latent Diffusion Network Gaussian Noise ControlNet Decoder 64 × 64 512 × 512 Latent Encoder 512 × 512 64 × 64 (a) Image Latent Diffusion Network Encoder 512 × 512 64 × 64 Decoder 64 × 64 512 × 512 Latent (b) Decoder Latent Decoder Image Latent Diffusion Network Difficulty Estimator ? Latent Diffusion Network 128 × 128512 × 512 64 × 64 512 × 512 Latent Encoder 512 × 512 64 × 64 Encoder 128 × 128512 × 512 (c) 29.63 26.59 29.70 25.84 29.85 27.02 22 24 26 28 30 32 easyhard PSNR(dB) DrealSR OSEDiffS3DiffTVTSR Δ=0.22 Δ=1.18 0.7984 0.6778 0.7926 0.6710 0.8134 0.7138 0.6 0.65 0.7 0.75 0.8 0.85 easyhard SSIM RealSR OSEDiffS3DiffTVTSR Δ=0.0208 Δ=0.0428 26.13 24.32 26.71 23.89 26.70 25.04 22 23 24 25 26 27 easyhard PSNR(dB) RealSR OSEDiffS3DiffTVTSR Δ=0.58 Δ=1.15 0.8627 0.7182 0.8320 0.6837 0.8593 0.7352 0.65 0.7 0.75 0.8 0.85 0.9 easyhard SSIM DrealSR OSEDiffS3DiffTVTSR Δ=0.0307 Δ=0.0515 Figure 2. The PSNR and SSIM performance of three representa- tive SOTA methods (OSEDiff (Wu et al., 2024a), S3Diff(Zhang et al., 2024a), and TVTSR(Yi et al., 2025)) on both easy and hard subsets. Notably, the performance gaps among these methods are consistently smaller on the easy subset than on the hard subset across both metrics. on the hard subset, where fine structural details are severely degraded. This observation reveals a critical shortcoming: current one-size-fits-all frameworks lack the capacity to adapt their inference strategy to the intrinsic complexity of each input, leading to suboptimal reconstruction of chal- lenging samples. Motivated by this insight, we argue that an effective Real-ISR system should not treat all inputs equally, but rather allocate computational resources and modeling capacity in a difficulty-aware manner. This motivates our design of a dynamic routing mechanism that tailors the inference path to the restoration difficulty of each sample. 3.2. Difficulty Estimator In image restoration tasks, the recovery difficulty is largely determined by the amount of high-frequency structural in- formation lost during degradation, as high-frequency com- ponents correspond to edges and fine details that are critical yet difficult to reconstruct. Motivated by this observation, we quantify restoration difficulty using high-frequency en- ergy attenuation by comparing the high-frequency spectral energy of the low-quality image with that of its ground-truth counterpart. This metric directly reflects the severity of detail loss caused by degradation and aligns well with the physical characteristics of real-world imaging processes, which predominantly act as low-pass filters. Moreover, it is model-agnostic, computationally efficient, and thus suitable for large-scale dataset analysis. Formally, we define the restoration difficultydof a degraded imageI deg relative to its ground-truth counterpartI gt as the normalized loss of high-frequency spectral energy induced by the degradation process. Specifically, letF (·)denote the 2D Fourier transform, and letM high be a binary mask that selects high-frequency regions in the frequency domain (e.g., coefficients beyond a radial threshold). The high-frequency energy of an image I is then computed as: E high (I) = X (u,v)∈M high F (I)(u,v) 2 .(1) The restoration difficulty of a degraded image is defined as: d = 1− E high (I deg ) E high (I gt ) .(2) This formulation yields a value in the range[0, 1], where d = 0indicates no high-frequency loss (trivial to restore), andd = 1implies complete suppression of high-frequency content (maximally challenging). By design,dcaptures the intrinsic complexity of restoring fine structures from the degraded input, providing a reliable and interpretable signal for dynamic routing in our framework. 3.3. Training of DDR-SR Existing SD-based one-step and multi-step Real-ISR meth- ods adopt a one-size-fits-all paradigm, which lacks the abil- ity to adapt their inference strategy to the intrinsic complex- ity of each input. This limitation leads to suboptimal recon- struction of challenging samples with severe degradation, while simultaneously wasting computational resources on easy inputs that require less modeling capacity. To overcome this fundamental drawback, our goal is to move beyond the uniform processing paradigm and build an effective Real- ISR system that does not treat all inputs equally. Instead, DDR-SR allocates computational resources and modeling capacity in a difficulty-aware manner, ensuring that each sample receives an appropriate level of processing tailored to its restoration difficulty. To realize this vision, we first estimate the restoration dif- ficulty of each training sample using the high-frequency energy attenuation metric introduced in Section 3.2. Based on the estimated difficulty scores, the entire training set is partitioned into “easy” and “hard” subsets. We then con- struct two specialized expert networks, denoted as Expert- D4 and Expert-D8, which share the same diffusion UNet backbone but employ VAEs with different spatial compres- sion ratios (4×and 8×, respectively). Specifically, we adopt the VAE-D4 from (Yi et al., 2025) as the VAE in Expert- D4 due to its superior detail preservation capability, and use the VAE from SD 2.1-base as the VAE in Expert-D8. Crucially, rather than training each expert exclusively on its designated subset, we adopt a mixed-training strategy to enhance robustness and generalization. Specifically, Expert- D4, designed for high-fidelity reconstruction, is trained on a mixture of 80% hard samples and 20% easy samples, encouraging it to focus on complex cases while retaining baseline capability on simpler inputs. Conversely, Expert- D8, optimized for inference efficiency, is trained on 80% 4 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution Diffusion Network Difficulty Estimator Diffusion Network VAE - D8 Encoder 512 × 512 64 × 64 VAE - D4 Encoder 128 × 128512 × 512 d “hard” subset “easy” subset 20% 80% 20% 80% Text Embedding DAPE CLIP Text Encoder VAE - D4 Decoder 128 × 128512 × 512 LR Images HR Images VAE - D8 Decoder 64 × 64 512 × 512 (a) Training process of DDR-SR Diffusion Network Difficulty Estimator Diffusion Network VAE - D4 Encoder 128 × 128512 × 512 d > Text Embedding DAPE CLIP Text Encoder VAE - D4 Decoder 128 × 128512 × 512 LR Images HR Images VAE - D8 Decoder 64 × 64 512 × 512 VAE - D8 Encoder 512 × 512 64 × 64 d < d (b) Inference process of DDR-SR TrainableFrozen is determined by the user based on specific applications 12SR MSELPIPSVSD Figure 3. Overview of the proposed Difficulty-aware Dynamic Routing (DDR) framework. During training, the dataset is split into “easy” and “hard” subsets using a high-frequency energy–based difficulty estimator. We train two expert models via LoRA: VAE-D4 + diffusion network on 80% hard + 20% easy samples for detail preservation, and VAE-D8 + diffusion network on 80% easy + 20% hard samples for efficiency. The two networks are trained with a combination of a reconstruction loss, an LPIPS loss, and a VSD loss. At inference, users can explicitly control the trade-off between visual fidelity and speed by adjusting the routing ratio between the high-capacity (D4) and low-capacity (D8) experts. easy samples and 20% hard samples, allowing it to handle common degradations efficiently without completely failing on challenging instances. Both experts are fine-tuned from a pre-trained SD using Low-Rank Adaptation (LoRA) (Hu et al., 2021), applied to the VAE encoder and the weights of all convolution and MLP layers of the diffusion UNet. This parameter-efficient adaptation enables each expert to develop distinct sensitivity to degradation patterns corre- sponding to their target difficulty range, while preserving the rich prior knowledge of the base model. The resulting experts thus exhibit complementary strengths: Expert-D4 excels at recovering fine structures in severely degraded images, whereas Expert-D8 achieves fast and perceptually satisfactory results on mildly degraded inputs. Figure 3(a) illustrates the training process of our approach. Taking the training of Expert-D8 as an example, after di- viding the dataset into easy and hard subsets based on their restoration difficulty, a degraded imageI deg is fed into the Expert-D8. We encode it into a latent feature z deg = E 8 (I deg ), and extract the text embeddingc t from it by a prompt extractor, which includes the DAPE (Wu et al., 2024b) and CLIP text encoder (Rombach et al., 2022). The restored latent featurez SR is obtained by passingz deg and c t through the denoising UNet ε 8 : z SR = z− √ 1− ̄α t ε 8 (z,c t ,t) √ ̄α t ,(3) where the time steptis set to 1 and ̄α t is the corresponding diffusion coefficient (Ho et al., 2020). Finally, feedingz SR into the decoder D 8 , we obtain the SR image I SR : I SR = D 8 (z SR ).(4) To achieve Real-ISR with one-step diffusion, following OSEDiff (Wu et al., 2024a), we useL MSE loss, LPIPS loss and VSD loss to train our Expert-D4 and Expert-D8: L SR =L MSE + λ 1 L LP IP S + λ 2 L V SD .(5) whereλ 1 = 2,λ 2 = 1are weighting hyper-parameters. In summary, the overall training procedure of DDR-SR is presented in Algorithm 1. 5 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution Algorithm 1 DDR-SR Training Procedure Require:Training datasetD = (I (i) deg ,I (i) gt ) N i=1 ; Diffi- culty estimatorE diff (·); VAE compression ratiosr 4 = 4, r 8 = 8; Mixing ratios α = 0.8, β = 0.2. Ensure: Trained expert models: Expert-D4, Expert-D8. 1: // Step 1: Estimate difficulty and split dataset 2: for each (I deg ,I gt )∈D do 3:Compute difficulty score: d←E diff (I deg ,I gt ) 4: end for 5: SplitDinto easy subsetD easy and hard subsetD hard using median d. 6: // Step 2: Construct mixed training sets 7: D D4 ← α·D hard + β·D easy // for high-fidelity expert 8: D D8 ← α·D easy + β·D hard // for efficient expert 9: // Step 3: Initialize experts from pre-trained SD 10: Initialize Expert-D4 with VAE (r 4 ) + UNet 11: Initialize Expert-D8 with VAE (r 8 ) + UNet 12: Apply LoRA to the VAE encoder and UNet in both experts. 13: // Step 4: Fine-tune experts 14: for epoch = 1 to T do 15:for batchB D4 ∼D D4 do 16:Update Expert-D4 via diffusion lossL SR 17:end for 18:for batchB D8 ∼D D8 do 19:Update Expert-D8 via diffusion lossL SR 20:end for 21: end for 22: Return Expert-D4, Expert-D8 3.4. The Inference Process of DDR-SR During inference, DDR-SR operates in a fully automatic and sample-adaptive manner. Given a low-quality input image I deg , the system first estimates its restoration difficulty using the difficulty estimator (as defined in Section 3.2). This yields a scalar difficulty scored ∈ [0, 1], where higher values indicate more severe high-frequency loss and greater reconstruction challenge. Based ond, a dynamic router assigns the input to one of two pre-trained expert networks: • Ifd ≥ τ, the sample is routed to Expert-D4, which employs a VAE with a low spatial compression ratio (4×) to preserve fine-grained details in the latent space, prioritizing visual fidelity. •Otherwise, it is processed by Expert-D8, which uses a higher compression ratio (8×) VAE for faster infer- ence while maintaining perceptual quality on mildly degraded inputs. The thresholdτcontrols the routing balance between the two experts. By default, we setτto the median difficulty of the training set, yielding an approximately 50/50 split. How- ever, a key advantage of DDR-SR is its user-controllable trade-off : practitioners can explicitly adjustτ(or directly specify the desired routing ratio) to favor either reconstruc- tion quality or computational efficiency according to appli- cation needs. The inference process of DDR-SR is presented in Algorithm 2. Algorithm 2 Inference Process of DDR-SR Require:Degraded input imageI deg ; Difficulty estimator E diff (·); Trained experts: Expert-D4, Expert-D8; Rout- ing thresholdτ ∈ [0, 1](default: median of training difficulties). Ensure: High-resolution output I hr . 1: // Step 1: Estimate restoration difficulty 2: Compute difficulty score: d←E diff (I deg ,I gt ) 3: // Step 2: Dynamic routing based on difficulty 4: if d≥ τ then 5:// Route to high-fidelity expert 6: I SR ← Expert-D4(I deg ) 7: else 8:// Route to efficient expert 9: I SR ← Expert-D8(I deg ) 10: end if 11: Return I SR 4. Experiments 4.1. Experimental Settings Training settings. Following SeeSR (Wu et al., 2024b) and OSEDiff (Li et al., 2024), we employ LSDIR (Li et al., 2023) and the first 10K images from FFHQ (Karras et al., 2019) as the training data. We use the same degradation pipeline as RealESRGAN (Wang et al., 2021b) to synthesize LR-HR pairs. During training, the synthesized LR images are upscaled to match the HR resolution of512× 512before feeding into our SR model. We train our model using the AdamW optimizer (Loshchilov & Hutter, 2017) with a learn- ing rate of5× 10 −5 on 4 A100 GPUs. The training process takes over 30k iterations, with a batch size of 16. We utilize the SD 2.1-base as the pretrained diffusion model for the×4 SR task. The trainable LoRA modules (Hu et al., 2021) are applied to the weights of all convolution and MLP layers and are initialized using a Gaussian distribution. The rank of the LoRA utilized in our Expert-D4 training is set to 4. The rank in LoRA is set as 16 for the VAE encoder and 4 for the diffusion UNet in our Expert-D8 training, respectively. Testing Details. Following previous methods (Li et al., 2024; Wu et al., 2024b), we evaluated our model on the synthetic DIV2K-Val (Agustsson & Timofte, 2017) dataset and two real-world datasets, RealSR (Cai et al., 2019a), and 6 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution Table 1. Quantitative comparison with state-of-the-art multi-step and one-step methods across both a synthetic and two real-world benchmarks. “-N” behind the method name represents the number of inference steps. Horizontal line is used to separate multi-step and one-step methods, which improves the readability of the table. The best and second best results of each metric are highlighted in red and blue, respectively. DatasetsMethods Metrics PSNR↑ SSIM↑ LPIPS↓ DISTS↓ FID↓ CLIPIQA↑ MUSIQ↑ MANIQA↑ NIQE↓ DIV2K-val StableSR-20023.310.5728 0.31290.213824.670.668265.630.61884.76 DiffBIR-5023.670.5653 0.35410.212930.930.665265.660.62044.71 SeeSR-5023.710.6045 0.32070.196725.830.685768.490.62394.82 PASD-2023.140.5489 0.36070.221929.320.671168.830.64844.40 ResShift-1524.690.6175 0.33740.221536.010.608960.920.54506.82 SinSR-124.430.6012 0.32620.206635.450.649962.800.53956.02 OSEDiff-123.720.6108 0.29410.197626.320.668367.970.61484.71 DDR-SR-124.090.62570.28040.191125.130.690868.660.62305.02 RealSR StableSR-20024.700.7085 0.30180.2167 127.200.617865.780.62215.9122 DiffBIR-5024.750.6567 0.36360.2290 124.560.646364.980.62465.5346 SeeSR-5025.180.7216 0.30090.2213 125.660.661269.770.64425.4081 PASD-2025.210.6798 0.33800.2259 123.080.662068.750.64875.4137 ResShift-1526.310.7421 0.34600.2498 142.810.544458.430.52857.2635 SinSR-126.280.7347 0.31880.2346 137.050.612260.800.53856.2872 OSEDiff-125.150.7341 0.29210.2128 123.500.669369.090.63265.6476 DDR-SR-125.920.75550.26340.2044114.860.672569.320.64295.8048 DRealSR StableSR-20028.030.7536 0.32840.2287 147.030.635658.510.56016.5239 DiffBIR-5026.710.6571 0.45570.2706 167.380.639561.070.59306.3124 SeeSR-5028.170.7691 0.31890.2306 149.860.680464.930.60426.3967 PASD-2027.360.7073 0.37600.2535 157.360.680864.870.61695.5474 ResShift-1528.460.7673 0.40060.2700 175.920.534250.600.45868.1249 SinSR-128.360.7515 0.36650.2488 177.050.638355.330.48846.9907 OSEDiff-127.920.7835 0.29680.2165 135.290.696364.650.58996.4902 DDR-SR-128.430.79450.29020.2191137.820.708465.980.60126.7650 DiffBIR-50StableSR-200SeeSR-50PASD-20ResShift-15SinSR-1OSEDiff-1DDR-SR-1 Figure 4. Visual comparisons of different Real-ISR methods. Please zoom in for a better view. DRealSR (Wei et al., 2020b). The synthetic dataset consists of 3,000 image pairs, which are created by downgrading the 512×512HQ images cropped from DIV2K-Val to128×128 LQ images using the Real-ESRGAN (Wang et al., 2021b) degradation pipeline. The real-world datasets comprised 128× 128 and 512× 512 HQ image pairs. 7 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution Compared Methods. We categorize the test models into two groups: multi-step and single-step inference. We com- pare DDR-SR with leading multi-step DM-based methods StableSR (Wang et al., 2023b), DiffBIR (Lin et al., 2024), SeeSR (Wu et al., 2024b), PASD (Yang et al., 2024), and ResShift (Yue et al., 2024); one-step DM-based methods SinSR (Wang et al., 2024a) and OSEDiff (Li et al., 2024). Evaluation Metrics. For the Real-ISR task, we utilize a set of reference-based and no-reference metrics to evaluate the competing methods. The reference-based metrics include PSNR, SSIM (Wang et al., 2004), LPIPS (Zhang et al., 2018), DISTS (Ding et al., 2020), and FID (Heusel et al., 2017). Note that PSNR and SSIM, computed on the Y chan- nel in the YCbCr space, are used to measure the fidelity of SR results; LPIPS and DISTS, computed in the RGB space, are used to assess the perceptual quality of SR results; FID evaluates the distance of distributions between GT and re- stored images. The no-reference metrics include CLIPIQA (Wang et al., 2023a), MUSIQ (Ke et al., 2021), MANIQA (Yang et al., 2022) and NIQE (Zhang et al., 2015). 4.2. Comparison with State-of-the-Art Methods Quantitative Comparisons. Table 1 shows the quantitative comparison of our method with SD-based multi-step and single-step methods on three datasets for the Real-ISR task. Our DDR-SR achieves the best results in SSIM, LPIPS, and CLIPIQA metrics. Moreover, our method demonstrates highly competitive performance on both full-reference met- rics (DIST and FID) and the no-reference metric MUSIQ, consistently ranking among the top two across all bench- marks. In terms of PSNR, DDR-SR outperforms all existing SD-based multi-step approaches, yet remains marginally be- hind ResShift and SinSR. DiffBIR, SeeSR, PASD, and Diff- BIR exhibit better performance on the MANIQA and NIQE metrics, which may be attributed to the fact that multi-step models have more denoising iterations to produce rich de- tails. However, our DDR-SR yields slightly inferior results on the MANIQA and NIQE, suggesting space for further improvement. Qualitative Comparisons. Figure 4 presents visual com- parisons of different Real-ISR methods. It is evident that DRR-SR significantly outperforms other methods on struc- ture preservation. Although the competing methods can improve image quality over the LR input, they struggle with preserving fine structures due to the high spatial compres- sion ratio of the VAE, which discards high-frequency details during encoding. For instance, none of the compared meth- ods were able to reconstruct the text in the first image or the pupil and eyelid textures in the second image. In contrast, our proposed DDR-SR method produces clearer and more legible text, as well as more realistic and detailed pupil and eyelid textures. These results underscore DDR-SR’s supe- Table 2. Comparison of computational complexity across different SD-based Real-ISR methods. MethodsStableSRDiffBIRSeeSRPASDOSEDiffExpert-D4 Steps20050502011 Param. (B)1.561.682.512.311.771.63 FLOPs (T)79.9424.3165.8629.132.271.81 Table 3. Ablation study on LoRA rank configurations in Expert-D4 on the RealSR dataset. Rank of VAE-D4Rank of UNetPSNR↑SSIM↑CLIPIQA↑MUSIQ↑ 4426.040.75600.699670.12 8426.040.75800.694470.19 16425.910.75670.687369.81 8826.190.75840.674870.18 81626.160.76040.688070.03 Table 4. Ablation study on LoRA rank configurations in Expert-D8 on the RealSR dataset. Rank of VAE-D8Rank of UNetPSNR↑SSIM↑CLIPIQA↑MUSIQ↑ 4425.860.75870.641468.16 16425.780.75500.645468.51 8826.260.76180.623767.54 rior detail fidelity. More visualization comparisons can be found in the supplementary materials. Complexity Comparisons. Table 2 compares the SD-based Real-ISR methods in terms of parameter count (Para.) and floating-point operations per second (FLOPs). Specifically, Expert-D4 achieves the lowest FLOPs at 1.81T, and it has 1.65B parameters, fewer than OSEDiff. 4.3. Ablation Study To better understanding of the DDR strategy, and the roles of the LoRA modules. We conduct ablation studies to in- vestigate the impact of the LoRA rank configuration in both the VAE and the diffusion UNet of our expert networks. Tables 3 and 4 show the impact of varying the LoRA rank on Expert-D4 and Expert-D8, respectively. We adopt LoRA ranks4, 4 for Expert-D4 and16, 4 for Expert-D8. 5. Conclusion We present DDR-SR, a difficulty-aware dynamic rout- ing framework for real-world image super-resolution. It adaptively routes inputs between two specialized experts based on a high-frequency energy-based difficulty estimator: Expert-D4 for high-fidelity restoration of hard samples, and Expert-D8 for efficient processing of easy ones. Through mixed training and LoRA-based tuning, DDR-SR achieves complementary expertise and efficient inference. Experi- ments show DDR-SR outperforms state-of-the-art methods across synthetic and real-world benchmarks, offering a tun- able trade-off between quality and speed. By moving be- yond one-size-fits-all processing, DDR-SR provides a more adaptive and practical solution for Real-ISR. 8 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. References Agustsson, E. and Timofte, R. Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPRW, 2017. Cai, J., Zeng, H., Yong, H., Cao, Z., and Zhang, L. To- ward real-world single image super-resolution: A new benchmark and a new model, 2019a. Cai, J., Zeng, H., Yong, H., Cao, Z., and Zhang, L. Toward real-world single image super-resolution: A new bench- mark and a new model. In Proceedings of the IEEE/CVF international conference on computer vision, p. 3086– 3095, 2019b. Chen, X., Wang, X., Zhang, W., Kong, X., Qiao, Y., Zhou, J., and Dong, C. Hat: Hybrid attention transformer for image restoration. arXiv preprint arXiv:2309.05239, 2023a. Chen, Z., Zhang, Y., Gu, J., Kong, L., Yang, X., and Yu, F. Dual aggregation transformer for image super-resolution. In ICCV, 2023b. Ding, K., Ma, K., Wang, S., and Simoncelli, E. P. Im- age quality assessment: Unifying structure and texture similarity. TPAMI, 2020. Dong, C., Loy, C. C., He, K., and Tang, X. Image super- resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2015. Dong, C., Loy, C. C., and Tang, X. Accelerating the super- resolution convolutional neural network. In ECCV, 2016. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. COMMUN ACM, 2020. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurIPS, 2020. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 4401–4410, 2019. Ke, J., Wang, Q., Wang, Y., Milanfar, P., and Yang, F. Musiq: Multi-scale image quality transformer. In ICCV, 2021. Kim, J., Lee, J. K., and Lee, K. M. Accurate image super- resolution using very deep convolutional networks. In CVPR, 2016. Kong, X., Zhao, H., Qiao, Y., and Dong, C. Classsr: A gen- eral framework to accelerate super-resolution networks by data characteristic. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12016–12025, 2021. Ledig, C., Theis, L., Husz ́ ar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 4681–4690, 2017. Li, J., Cao, J., Zou, Z., Su, X., Yuan, X., Zhang, Y., Guo, Y., and Yang, X. Distillation-free one-step diffusion for real-world image super-resolution. arXiv preprint arXiv:2410.04224, 2024. Li, Y., Zhang, K., Liang, J., Cao, J., Liu, C., Gong, R., Zhang, Y., Tang, H., Liu, Y., Demandolx, D., et al. Lsdir: A large scale dataset for image restoration. In CVPRW, 2023. Li, Y., Liang, D., Ding, T., and Huang, S.-J. Structsr: Refuse spurious details in real-world image super-resolution. arXiv preprint arXiv:2501.05777, 2025. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., and Timofte, R. Swinir: Image restoration using swin trans- former. In Proceedings of the IEEE/CVF international conference on computer vision, p. 1833–1844, 2021. Lim, B., Son, S., Kim, H., Nah, S., and Mu Lee, K. En- hanced deep residual networks for single image super- resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, p. 136–144, 2017. Lin, X., He, J., Chen, Z., Lyu, Z., Fei, B., Dai, B., Ouyang, W., Qiao, Y., and Dong, C. Diffbir: Towards blind image restoration with generative diffusion prior. In ECCV, 2024. Liu, M., Pan, J., Yan, Z., Zuo, W., and Zhang, L. Adaptive network combination for single-image reflection removal: A domain generalization perspective. arXiv preprint arXiv:2204.01505, 2022. 9 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution Liu, X., Zhang, X., Ma, J., Peng, J., et al. Instaflow: One step is enough for high-quality diffusion-based text-to- image generation. In ICLR, 2023. Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv:1711.05101, 2017. Ma, C., Rao, Y., Cheng, Y., Chen, C., Lu, J., and Zhou, J. Structure-preserving super resolution with gradient guidance. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 7769– 7778, 2020. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in Neural Information Processing Systems, 35: 36479–36494, 2022. Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022. Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffusion distillation. ECCV, 2025. Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In ICLR, 2021. Sun, L., Wu, R., Yang, T., and Zhang, L.Consissr: Consistency-model-driven single-step diffusion network for real-world image super-resolution. arXiv preprint arXiv:2501.08857, 2025. Wang, J., Chan, K. C., and Loy, C. C. Exploring clip for assessing the look and feel of images. In AAAI, 2023a. Wang, J., Yue, Z., and Loy, C. C. Stablesr: Boosting real- world image super-resolution via time-aware diffusion. In International Conference on Computer Vision (ICCV), 2023b. Wang, X., Xie, L., Dong, C., and Shan, Y. Real-esrgan: Training real-world blind super-resolution with pure syn- thetic data. In ICCV, 2021a. Wang, X., Xie, L., Dong, C., and Shan, Y. Real-esrgan: Training real-world blind super-resolution with pure syn- thetic data. In ICCV, 2021b. Wang, Y., Yang, W., Chen, X., Wang, Y., Guo, L., Chau, L.-P., Liu, Z., Qiao, Y., Kot, A. C., and Wen, B. Sinsr: Diffusion-based image super-resolution in a single step. In CVPR, 2024a. Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. Wang, Z., Chen, J., and Hoi, S. C. Deep learning for image super-resolution: A survey. TPAMI, 2020. Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, J. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS, 2024b. Wei, P., Xie, Z., Lu, H., Zhan, Z., Ye, Q., Zuo, W., and Lin, L. Component divide-and-conquer for real-world image super-resolution. In European conference on computer vision, p. 101–117, 2020a. Wei, P., Xie, Z., Lu, H., Zhan, Z., Ye, Q., Zuo, W., and Lin, L. Component divide-and-conquer for real-world image super-resolution. In ECCV, 2020b. Wu, R., Sun, L., Ma, Z., and Zhang, L. One-step effective diffusion network for real-world image super-resolution. arXiv preprint arXiv:2406.08177, 2024a. Wu, R., Yang, T., Sun, L., Zhang, Z., Li, S., and Zhang, L. Seesr: Towards semantics-aware real-world image super-resolution. In CVPR, 2024b. Xie, C., Wu, Y., Chen, L., and Zhang, L. Diffsteisr: Diffu- sion model for real-world stereo image super-resolution with texture-semantic aware cross-attention. arXiv preprint arXiv:2412.06322, 2024a. Xie, R., Tai, Y., Zhang, K., Zhang, Z., Zhou, J., and Yang, J. Addsr: Accelerating diffusion-based blind super- resolution with adversarial diffusion distillation. arXiv preprint arXiv:2404.01717, 2024b. Yang, J., Wright, J., Huang, T. S., and Ma, Y. Image super- resolution via sparse representation. IEEE transactions on image processing, 19(11):2861–2873, 2010. Yang, S., Wu, T., Shi, S., Lao, S., Gong, Y., Cao, M., Wang, J., and Yang, Y. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In CVPR, 2022. Yang, T., Wu, R., Ren, P., Xie, X., and Zhang, L. Pixel- aware stable diffusion for realistic image super-resolution and personalized stylization. In ECCV, 2024. Yi, Q., Li, S., Wu, R., Sun, L., Wu, Y., and Zhang, L. Fine- structure preserved real-world image super-resolution via transfer vae training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 12415– 12426, 2025. 10 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In CVPR, 2024a. Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In CVPR, 2024b. Yu, F., Gu, J., Li, Z., Hu, J., Kong, X., Wang, X., He, J., Qiao, Y., and Dong, C. Scaling up to excellence: Practic- ing model scaling for photo-realistic image restoration in the wild. In CVPR, 2024. Yue, Z., Wang, J., and Loy, C. C. Resshift: Efficient diffu- sion model for image super-resolution by residual shifting. In NeurIPS, 2024. Zhang, A., Yue, Z., Pei, R., Ren, W., and Cao, X. Degradation-guided one-step image super-resolution with diffusion priors. arXiv preprint arXiv:2409.17058, 2024a. Zhang, K., Liang, J., Van Gool, L., and Timofte, R. Design- ing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, p. 4791–4800, 2021. Zhang, L., Zhang, L., and Bovik, A. C. A feature-enriched completely blind image quality evaluator. TIP, 2015. Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In ICCV, 2023. Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. Zhang, X., Zhang, Z., Wu, R., and Zhang, L. Epdiff: En- hancing semantic prior for diffusion model based real- world image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19456–19466, 2025. Zhang, Z., Yi, Q., Chen, D., and Zhang, L. Dloral: Dual- lora learning for real-world video super-resolution with temporal consistency and spatial detail enhancement. In Proceedings of the European Conference on Computer Vision, p. 355–372, 2024b. Zhu, Z., Feng, X., Chen, D., Bao, J., Wang, L., Chen, Y., Yuan, L., and Hua, G. Designing a better asymmetric vq- gan for stablediffusion. arXiv preprint arXiv:2306.04632, 2023. 11 Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution A. Visualization Comparisons Figure 5 provides a visual comparison of various diffusion-based methods applied to real-world datasets, covering both multi- step and one-step approaches. As shown, none of the competing methods could effectively reconstruct the intricate ruler texture in the first image, the letters and graphics in the second image, or the facial details in the third image. Specifically, these methods either produced blurred or incomplete reconstructions, failing to capture the fine structural details. In contrast, our DDR-SR method excels in these challenging scenarios. It successfully reconstructs the ruler’s texture with sharp edges and minimal artifacts, renders the letters and graphics clearly and accurately, and captures detailed facial features, including subtle details like beard stubble. These results highlight DDR-SR’s superior capability in faithfully restoring high-fidelity image details, demonstrating its effectiveness in handling diverse and complex real-world degradations. DiffBIR-50StableSR-200SeeSR-50PASD-20ResShift-15SinSR-1OSEDiff-1DDR-SR-1 DiffBIR-50StableSR-200SeeSR-50PASD-20ResShift-15SinSR-1OSEDiff-1DDR-SR-1 Figure 5. Visual comparisons of different Real-ISR methods. Please zoom in for a better view. 12