Paper deep dive
Restoring Without Forgetting: Continual Learning Across Image Degradations
Alif Ashrafee, Bartosz Krawczyk
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/26/2026, 4:35:24 AM
Summary
The paper introduces Restoring without Forgetting (RwF), a framework for continual multi-degradation image restoration. It addresses the challenge of sequentially learning new image degradations (e.g., noise, blur, haze) without forgetting previous ones or retaining historical data. RwF uses a frozen pretrained backbone (Restormer or NAFNet) and attaches lightweight, degradation-specific low-rank adapters. An unsupervised routing mechanism identifies the correct restoration path at test time. The method is evaluated on a synthetic benchmark with shared content across five degradation domains and transfers well to real-world benchmarks.
Entities (11)
Relation Signals (10)
RwF → solvesproblem → Continual Learning
confidence 95% · We formulate multi-degradation image restoration as a continual domain-incremental learning problem
RwF → usesbackbone → Restormer
confidence 95% · RwF improves final average PSNR ... on the Restormer ... backbones
RwF → usesbackbone → NAFNet
confidence 95% · RwF improves final average PSNR ... on the NAFNet backbones
RwF → usescomponent → Low-rank Adapter
confidence 95% · RwF framework learns a lightweight adapter for each new degradation
RwF → handlesdegradation → Haze
confidence 90% · five sequentially learned degradations (..., +Haze, ...)
RwF → handlesdegradation → Low-light
confidence 90% · five sequentially learned degradations (..., +Lowlight)
RwF → handlesdegradation → Noise
confidence 90% · five sequentially learned degradations (Noise, ...)
RwF → handlesdegradation → Motion Blur
confidence 90% · five sequentially learned degradations (..., +Blur, ...)
RwF → handlesdegradation →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent progress in image restoration has converged on all-in-one architectures that jointly handle multiple degradations within a single network. These methods are effective on static benchmarks but target a closed-world setting that assumes simultaneous access to every target degradation at training time. In practice, degradations are encountered sequentially as field-deployed systems progressively face new environmental conditions, and historical training data is often unavailable due to privacy or storage constraints. Accommodating a new degradation then requires either retraining on the union of all prior data, which is often costly or infeasible, or fine-tuning, which causes catastrophic forgetting. We formulate multi-degradation image restoration as a continual domain-incremental learning problem, in which degradations arrive incrementally and prior data is unavailable. Our proposed Restoring without Forgetting (RwF) framework learns a lightweight adapter for each new degradation, eliminating forgetting by construction at a fraction of the cost of dedicated per-domain networks. To isolate degradation learning from dataset variation, we construct a benchmark spanning five degradation domains under shared image content. At test time, an unsupervised routing mechanism identifies the appropriate restoration path for unknown inputs without requiring domain labels. Across the five-domain sequence, RwF improves final average PSNR over naive sequential fine-tuning by 15.25 dB and 11.83 dB on the Restormer and NAFNet backbones, respectively. The framework transfers to eleven canonical real-degradation benchmarks (3,465 images) at 89.5% routing accuracy with only a +0.94 dB oracle PSNR gap, establishing, to our knowledge, the first systematic baseline for continual multi-degradation image restoration.
Tags
Links
- Source: https://arxiv.org/abs/2608.23799v1
- Canonical: https://arxiv.org/abs/2608.23799v1
Trouble viewing inline? Open PDF directly →
Full Text
62,607 characters extracted from source content.
Expand or collapse full text
Restoring Without Forgetting Restoring Without Forgetting: Continual Learning Across Image Degradations Alif Ashrafee Bartosz Krawczyk Abstract Recent progress in image restoration has converged on all-in-one architectures that jointly handle multiple degradations within a single network. These methods are effective on static benchmarks but target a closed-world setting that assumes simultaneous access to every target degradation at training time. In practice, degradations are encountered sequentially as field-deployed systems progressively face new environmental conditions, and historical training data is often unavailable due to privacy or storage constraints. Accommodating a new degradation then requires either retraining on the union of all prior data, which is often costly or infeasible, or fine-tuning, which causes catastrophic forgetting. We formulate multi-degradation image restoration as a continual domain-incremental learning problem, in which degradations arrive incrementally and prior data is unavailable. Our proposed Restoring without Forgetting (RwF) framework learns a lightweight adapter for each new degradation, eliminating forgetting by construction at a fraction of the cost of dedicated per-domain networks. To isolate degradation learning from dataset variation, we construct a benchmark spanning five degradation domains under shared image content. At test time, an unsupervised routing mechanism identifies the appropriate restoration path for unknown inputs without requiring domain labels. Across the five-domain sequence, RwF improves final average PSNR over naive sequential fine-tuning by 15.25 dB and 11.83 dB on the Restormer and NAFNet backbones, respectively. The framework transfers to eleven canonical real-degradation benchmarks (3,465 images) at 89.5% routing accuracy with only a +0.94+0.94 dB oracle PSNR gap, establishing, to our knowledge, the first systematic baseline for continual multi-degradation image restoration. Code, benchmark, and weights are available at: https://github.com/AlifAshrafee/Restoring-Without-Forgetting. †email: a5264@rit.edu†email: bartosz.krawczyk@rit.edu†affiliation: Rochester Institute of Technology New York, USA 1 Introduction Image restoration has advanced rapidly in recent years, propelled by attention-based and transformer architectures that exploit long-range pixel dependencies for high-fidelity reconstruction [29, 54, 47]. Networks such as Restormer [59], NAFNet [7], SFHformer [22], and FFTformer [24] have established increasingly strong baselines across denoising, deblurring, dehazing, deraining, and low-light enhancement. Yet the dominant practice remains task-specialized: each network is trained on a single degradation type, with dedicated data, hyperparameter tuning, and model weights for every problem [30, 13, 3, 37, 46]. This single-degradation pipeline is poorly suited to real deployment. Field-deployed sensors progressively encounter new environmental conditions over their operational lifetime, often without access to data from previously seen degradations due to storage, privacy, or licensing constraints. Naively fine-tuning an existing network on a new degradation overwrites the representations responsible for prior modes, leading to catastrophic forgetting [34, 12]. In contrast, training a fresh network from scratch for every encountered degradation is expensive, data-hungry, and increasingly unscalable as the catalog of degradations grows. Equally absent from this paradigm is a mechanism to decide which restoration network to apply to a given input at test time. Existing pipelines implicitly assume an oracle that pairs each test image with the correct network, an assumption that breaks down for intelligent systems operating in the open world. [] [] Figure 1: (a) Three paradigms for multi-degradation image restoration: One-By-One (one network per degradation), All-In-One (a single jointly trained network), and RwF (Ours), which freezes a pretrained backbone and attaches lightweight degradation-specific adapters routed without supervision at inference. (b) Average incremental PSNR across five sequentially learned degradations (Noise, +Blur, +Rain, +Haze, +Lowlight). Sequential fine-tuning forgets as degradations accumulate, while RwF holds performance across all domains. To reduce this parameter redundancy, recent work consolidates multiple degradations into “all-in-one” networks trained jointly on the union of target tasks [26, 39, 62, 40, 15]. Effective as they are on closed-set benchmarks, these models presuppose simultaneous access to every target degradation in a single joint training run, with the degradation set fixed at design time. Extending one to a newly observed degradation then requires reassembling the full union of past and present data and retraining, which is costly at scale and infeasible when prior data cannot be retained. Weather-specialized restoration [48] narrows the problem to atmospheric phenomena and offers no clear extension to the sensor-side artifacts (noise, low-light, blur) that deployed perception stacks routinely encounter. The challenge of accumulating capability over a sequential data stream while preserving previously learned competence is the central problem of continual learning (CL) [38, 52]. Domain-incremental learning (DIL) [49] in particular targets the setting where new domains arrive sequentially and domain identity is unavailable at test time. Yet CL research remains predominantly evaluated in discriminative, class-incremental settings on classification benchmarks, with limited engagement with dense, pixel-level generative tasks. Image restoration, acutely sensitive to distributional shifts across degradation types and demanding faithful per-pixel reconstruction, is conspicuously underexplored under the CL framework. The central challenge is therefore not only to restore images from diverse degradations, but to acquire these capabilities sequentially, without retaining past data and without knowing the degradation label at test time. Motivated by this, we propose RwF (Restoring without Forgetting), a parameter-efficient framework for continual multi-degradation restoration. RwF freezes a pretrained restoration backbone that provides a strong natural-image prior and incrementally attaches a lightweight low-rank adapter for each new degradation, isolating degradation-specific parameters so that previously learned degradations are preserved exactly. To resolve degradation identity at inference, a prototype-matching mechanism routes each input to the appropriate adapter without domain labels. As shown in Fig. 1(b), naive sequential fine-tuning (FT) of strong backbones suffers severe forgetting as degradations accumulate, whereas RwF preserves performance across the full sequence, improving final average PSNR by up to 15.25 dB. Our main contributions are as follows: • A continual learning formulation of multi-degradation image restoration. We frame multi-degradation restoration as a domain-incremental learning problem in which qualitatively distinct degradations arrive sequentially and degradation identity is unknown at test time, a setting that has remained largely unexplored within the broader continual learning literature. • A parameter-efficient framework with unsupervised inference-time routing. We introduce a backbone-frozen architecture that eliminates forgetting by construction through degradation-specific low-rank adapters at a fraction of the cost of dedicated per-domain networks, and routes test inputs to the appropriate restoration path without domain labels via a lightweight prototype-matching mechanism. • A content-controlled benchmark and sim-to-real evaluation. We construct a five-domain benchmark spanning blur, noise, haze, rain, and low-light over a shared image corpus, isolating degradation learning from dataset variation, and complement it with evaluations on canonical real-degradation benchmarks to demonstrate sim-to-real transferability of the learned adapters. 2 Related Work 2.1 Image Restoration Modern image restoration is dominated by transformer-based architectures that capture long-range dependencies for high-fidelity reconstruction. SwinIR [29] adapted shifted-window self-attention to restoration, Restormer [59] reorganized attention along the channel dimension for efficient high-resolution processing, NAFNet [7] demonstrated that careful simplification of nonlinear activations yields a competitive convolutional baseline, FFTformer [24] exploited frequency-domain representations, and SFHformer [22] fused spatial and frequency cues within a unified architecture. Despite their strong per-task performance, these networks remain single-degradation specialists, requiring a distinct trained instance for every problem. To consolidate this redundancy, the all-in-one paradigm trains a single network jointly on the union of target degradations. AirNet [26] and TransWeather [48] pair a shared backbone with degradation-aware modules, PromptIR [39] learns visual prompts that adapt the network to the input degradation, IDR [62] decomposes degradations into shared ingredients, AdaIR [10] modulates frequency-domain features adaptively, and RAM [40] brings masked image modeling to blind multi-degradation restoration. These methods assume joint access to every target degradation at training time, with the full degradation set specified in advance. Subsequent work scales this paradigm without departing from the assumption. FoundIR [27] trains on a million-image corpus through an incremental schedule that serves optimization stability rather than continual acquisition. DegAE [31] synthesizes degradations to pretrain transferable low-level features for downstream fine-tuning. DCPT [19] pretrains a degradation classifier over a predetermined set to initialize a joint restoration model. A nascent line of work has begun to examine continual learning in restoration more directly. CauSiam [9] performs continual test-time adaptation for defocus deblurring but not offline domain-incremental training. MINI [15] adds new restoration capabilities through a meta-convolution module, though its capacity is bounded by an embedding pool that must be fixed at design time and cannot be expanded as degradations accumulate. None of these works report a forgetting protocol across heterogeneous degradation types under unsupervised inference. We position RwF in this gap, as a domain-incremental framework that retains no past data, assigns each degradation its own restoration path under strict parameter isolation, and selects among paths by label-free routing. 2.2 Domain-Incremental Learning Domain-incremental learning addresses sequential adaptation across new data domains without access to domain identity at inference [49, 52]. The dominant strategies fall into three families: regularization of important parameters [23, 60], replay of stored exemplars [4, 5, 21], and parameter isolation [28, 32]. Replay carries storage and privacy costs that are impractical at high resolution [57, 51], motivating a recent shift toward exemplar-free parameter-isolation approaches over frozen pretrained backbones [35, 67, 65, 45]. AdaptFormer [8] established adapters as a parameter-efficient mechanism for transformer fine-tuning, and methods such as SOYO [53] and DUCT [66] have refined exemplar-free domain selection and embedding-space calibration. Beyond classification, DIL has been extended to dense semantic segmentation [43], but the literature remains overwhelmingly discriminative. Pixel-level generative tasks such as image restoration are largely absent, which is the gap our work addresses. 3 Methodology 3.1 Problem Formulation We cast multi-degradation image restoration as a domain-incremental learning (DIL) problem [49]. Throughout, degradation refers to a physical corruption operator (motion blur, sensor noise, atmospheric haze, and so on), and domain refers to the distribution of images it produces. Thus, each degradation type constitutes a distinct domain, and the domains are revealed to the model one at a time. Let x∈ℝH×W×3x ^H× W× 3 denote a clean image drawn from a content distribution p(x)p(x), and let a degradation be described by a (possibly stochastic) operator :ℝH×W×3→ℝH×W×3T:R^H× W× 3 ^H× W× 3 that maps a clean image to a degraded observation y=(x)y=T(x). A restoration model seeks an approximate inverse that recovers x from y. We consider a sequence of T degradation domains 1,2,…,TD_1,D_2,…,D_T, each induced by a distinct operator tT_t and instantiated as a set of paired samples t=(xi,yi(t))i=1NtD_t=\(x_i,y_i^(t))\_i=1^N_t with yi(t)=t(xi)y_i^(t)=T_t(x_i). The operators span qualitatively different physical processes. In this work we instantiate five canonical degradations through the following forward models: Noise: n(x)=x+,∼(0,σ2); _n(x)=x+n, (0,σ^2I); (1) Motion blur: b(x)=L,θ,C⊗x+b; _b(x)=k_L,θ,C x+n_b; (2) Haze: h(x)=x⊙+A(1−),=e−β; _h(x)=x +A\,(1-t), =e^- ; (3) Rain: r(x)=(1−v)(x+α∑l=1Ll)+vAr; _r(x)=(1-v) (x+α _l=1^Ls_l )+v\,A_r; (4) Low-light: l(x)=1kPois(kxγ)+r, _l(x)= 1k\,Pois (k\,x^γ )+n_r, (5) where ⊗ denotes convolution with a motion blur kernel L,θ,Ck_L,θ,C rasterized from a parametric trajectory of length L, direction θ, and curvature C; ⊙ is the Hadamard product; t is the transmission map governed by per-pixel scene depth d and scattering coefficient β; A is the global atmospheric light; ls_l are per-layer rain streak maps at increasing depth planes (with α controlling streak intensity, v the atmospheric veiling, and ArA_r the gray atmospheric light from droplet scattering); γ>1γ>1 controls illumination attenuation; Pois(⋅)Pois(·) denotes a pixelwise Poisson resampling with photon gain k; and bn_b, rn_r are Gaussian noise terms modeling post-blur sensor readout and read noise, respectively. These models follow established degradation formulations in the restoration literature (Sec. 3.3). In the DIL setting, domains are encountered strictly sequentially: at stage t only tD_t is accessible, while all prior domains <tD_<t are unavailable and no exemplars are retained. Let fθf_θ denote a restoration network with parameters θ. At the initial stage the backbone is trained on 1D_1 under an ℓ1 _1 reconstruction loss, as is standard for restoration [59, 7]. For any later domain, unconstrained fine-tuning from the previous parameters minimizes the current-domain loss but overwrites the representations supporting earlier domains, the hallmark of catastrophic forgetting [34, 12]. After traversing all T domains, the model parameters ΘT _T should minimize the average reconstruction loss over every domain seen so far, ℒCL=1T∑t=1T1Nt∑i=1Nt‖fΘT(yi(t))−xi‖1,L_CL= 1T _t=1^T 1N_t _i=1^N_t \|f_ _T(y_i^(t))-x_i \|_1, (6) subject to the constraint that domain data is never jointly available, which separates our setting from all-in-one joint training [26, 39, 62], and that no past samples are stored, which separates it from replay-based continual learning [4, 5]. Crucially, the domain index t is also withheld at test time, so the model must additionally infer which restoration behavior to apply for each input. We resolve the training-time tension in Sec. 3.4 and the test-time routing in Sec. 3.5. 3.2 Pretrained Backbone and Degradation-Isolated Learning Modern continual learning increasingly builds on a strong pretrained backbone used as a frozen, general-purpose feature extractor, with adaptation confined to lightweight modules [35, 67, 45]. Where discriminative CL takes an ImageNet [44]-pretrained encoder as this prior, we seek its restoration analogue, and image denoising is a natural candidate: a denoiser must learn the statistics of clean natural images to separate signal from corruption, which is precisely the prior that plug-and-play and regularization-by-denoising methods reuse as a universal proximal operator across inverse problems [50, 63]. A denoising-optimized backbone therefore captures degradation-agnostic restoration primitives that transfer to other degradations. We accordingly take the publicly released Gaussian color denoising weights of Restormer [59] off the shelf as our backbone fθ1∗f_ _1^* and freeze them throughout, so they serve as the shared, fixed substrate over which all subsequent degradations are learned. The same scheme transfers without modification to NAFNet [7] using its released denoising weights. The DIL formulation idealizes a domain as differing from another only in its degradation operator, but real benchmarks violate this: each is collected for a single degradation and carries its own clean-content distribution (street scenes, indoor scenes, hazy landscapes). Training one restoration path per real dataset therefore conflates the degradation operator with the dataset content, and an adapter cannot tell which it is correcting for. To make this precise, let the clean content be a latent c∼p(c)c p(c) and write y=t(c)y=T_t(c), with the frozen backbone producing a routing embedding e=gθ1∗(y)e=g_ _1^*(y). Under a multi-dataset construction, domain t draws content from its own distribution pt(c)p_t(c), so the degraded marginal qt(y)=∫p(y∣c;t)pt(c)cq_t(y)= p(y c;T_t)\,p_t(c)\,dc varies with t through both tT_t and pt(c)p_t(c). In the counterfactual where all operators are identical, t≡T_t , any remaining dependence of e on t is purely content-driven, and the mutual information I(e,t)I(e;t) decomposes as I(e,t)=Ideg(e,t)⏟from t+Icontent(e,t)⏟from pt(c),Icontent(e,t)>0wheneverpt(c)≢pt′(c).I(e;t)\;=\; I_deg(e;t)_from \T_t\\;+\; I_content(e;t)_from \p_t(c)\, I_content(e;t)>0\;\;whenever\;\;p_t(c) ≡ p_t (c). (7) The router can then reach high domain-identification accuracy by recognizing content rather than degradation, and each path absorbs the dataset-specific prior pt(c)p_t(c) into its learned correction, entangling it with the degradation inverse t−1T_t^-1. The remedy is to draw all domains from one shared corpus, pt(c)=p(c)p_t(c)=p(c) for all t, defining each domain solely by its operator, t=(xi,yi(t)):xi∼p(x),yi(t)=t(xi).D_t= \(x_i,\,y_i^(t)):x_i p(x),\;y_i^(t)=T_t(x_i) \. (8) Now the counterfactual t≡T_t renders e identically distributed across domains, forcing Icontent(e,t)=0I_content(e;t)=0 and hence I(e,t)=Ideg(e,t).I(e;t)\;=\;I_deg(e;t). (9) All domain-distinguishing information, and therefore all routing accuracy, is then attributable to the degradation operators alone. Symmetrically, since the clean targets share a common manifold, each path is optimized to invert its operator tT_t without absorbing a content prior. This identifiability is why our benchmark is synthesized over a shared corpus rather than assembled from heterogeneous real datasets. 3.3 Degradation Synthesis To instantiate Eq. (8), we apply the forward models of Eqs. (1)–(5) to a single clean corpus (DIV2K [1]), producing one paired dataset per degradation in which every domain shares identical clean source images. Each operator is parameterized to match the degradation characteristics reported in the corresponding restoration literature, so that synthetic samples approximate their real-world counterparts in appearance and difficulty. Additive Gaussian noise follows the AWGN model used throughout learned denoising [64]. Motion blur kernels are rasterized from parametric trajectories of explicit length, direction, and curvature following [2], with mild post-blur Gaussian sensor noise to mimic long-exposure handheld capture. Haze is rendered via the Koschmieder atmospheric scattering model with per-pixel transmission derived from monocular depth estimated by MiDaS [41], matching the synthesis convention of standard dehazing benchmarks [25]. Rain is synthesized as a multi-layer composite in which each layer represents a depth plane with progressively shorter, thinner, and more strongly out-of-focus streaks, followed by atmospheric veiling from suspended droplet scattering, combining [14] with the multi-layer rendering of [58]. Low-light is generated by gamma-based illumination attenuation followed by signal-dependent Poisson shot noise and signal-independent Gaussian read noise, following the See-in-the-Dark sensor model [6] and consistent with low-light enhancement datasets [56]. 3.4 Restoration Paths via Low-Rank Adaptation Given the frozen backbone fθ1∗f_ _1^*, we learn each new degradation by attaching compact, trainable modules while leaving the backbone untouched. The atomic module is a low-rank adapter that applies a bottleneck residual correction to a block’s features, ϕ()=+supσ(downLN()),A_φ(z)=z+s\,W_up\,σ\! (W_down\,LN(z) ), (10) where ∈ℝdz ^d is a feature vector, down∈ℝr×dW_down ^r× d and up∈ℝd×rW_up ^d× r form a rank-r bottleneck with r≪dr d, σ is a ReLU, LNLN is layer normalization, and s is a fixed scale. Both projections are linear and act pointwise across the channel dimension. The trainable parameters are ϕ=down,upφ=\W_down,W_up\, totaling 2dr+d2dr+d per module. Following LoRA-style initialization [18, 16], downW_down is Kaiming-initialized and upW_up is set to zero, so each adapter is the identity at insertion and leaves the pretrained features unperturbed until training begins. A single adapter only corrects one block, whereas inverting a degradation requires coordinated corrections along the entire encode–decode trajectory. We therefore instantiate an adapter at every block of the backbone and define a restoration path as the complete, network-spanning set of these per-block adapters for a given degradation, Φt=ϕt,1,ϕt,2,…,ϕt,L, _t=\ _t,1, _t,2,…, _t,L\, (11) where L is the number of blocks. For a Transformer block, the path-augmented forward pass wraps the post-FFN residual, l′ _l =l+Attn(LN(l)), =z_l+Attn(LN(z_l)), l+1 _l+1 =ϕt,l(l′+FFN(LN(l′))), =A_ _t,l\! (z_l +FFN(LN(z_l )) ), (12) with AttnAttn and FFNFFN frozen. For convolutional blocks, the same injection follows the channel-mixing stage. Because a path is defined purely at the block level through the channel dimension dld_l, the construction is backbone-agnostic, and we validate it on both a Transformer (Restormer [59]) and a fully convolutional network (NAFNet [7]) under identical path definitions. The full restoration network for domain t is fθ1∗,Φtf_ _1^*, _t: a shared frozen backbone routed through one degradation-specific path. Sequential learning enforces strict parameter isolation. At domain t, the backbone and all committed paths Φ<t _<t are frozen and only a freshly initialized Φt _t is optimized, Φt∗=argminΦt1Nt∑i=1Nt‖fθ1∗,Φt(yi(t))−xi‖1. _t^*= _ _t 1N_t _i=1^N_t \|f_ _1^*, _t(y_i^(t))-x_i \|_1. (13) Upon convergence, Φt∗ _t^* is committed to a frozen bank and the procedure repeats. The final model state ΘT=θ1∗∪Φ1∗,…,ΦT∗ _T=\ _1^*\∪\ _1^*,…, _T^*\ thus occupies mutually disjoint parameter subspaces for distinct domains, so the cross-domain gradient vanishes and forgetting is exactly zero by construction, ∂Φt∗∂Φj∗=(j≠t)⟹ℱj,t=0(j<t), ∂ _t^*∂ _j^*=0 (j≠ t)\;\; \;\;F_j,t=0 (j<t), (14) achieved without regularization penalties or replay, at a per-domain overhead of only |Φt|≈0.03|θ1∗|| _t|≈ 0.03\,| _1^*| that scales linearly in the number of domains rather than duplicating the backbone. Because each path Φt _t and its prototype are estimated independently against the frozen backbone, with no gradient coupling across domains, the model state and the prototype bank are identical under any permutation of the domain order. Figure 2: RwF Framework: a denoising-pretrained encoder–decoder backbone is frozen, and each new degradation trains a network-spanning path of low-rank adapters under strict parameter isolation, leaving prior paths and the backbone unchanged. A domain prototype is formed by pooling first-encoder-stage features. At inference, an unknown input is routed by cosine similarity to the matching path, which restores the image. 3.5 Prototype-Based Degradation Routing Because the domain index is withheld at test time, the model must decide which restoration path to engage for each input. We route without any auxiliary classifier or extra trainable parameters by exploiting the geometry of the frozen backbone’s feature space. A central design question is where along the backbone to read out features for prototyping. In classification CL, the deepest pre-classifier features are canonical, on the intuition that representational abstraction sharpens with depth. We find this prescription inverts for cross-degradation DIL: a restoration backbone is optimized to suppress the degradation signal en route to a clean reconstruction, so each successive encoder block, by design, attenuates precisely the cues a router needs. By the bottleneck stage, distinct degradations have largely converged toward a shared clean-image attractor and become difficult to discriminate, whereas early-encoder features still carry visible imprints of the operator. We empirically corroborate this depth-discrimination trade-off in Sec. 4 (Tab. 4) and accordingly read out at the output of the last block of the first encoder stage, denoted gθ1∗(1)(⋅)g^(1)_ _1^*(·). A second design choice is how to aggregate this spatial feature map into a fixed-length embedding. Different degradations leave signatures in different orders of spatial statistics. Noise inflates per-channel standard deviation without altering channel means, haze flattens both, rain streaks introduce structured high spatial variance over an otherwise clean background, and low-light shifts mean intensities. A first-order summary (Global Average Pooling, GAP) alone discards the variance signal, while a joint normalization of mean and standard deviation lets the larger-magnitude component dominate the cosine similarity. We therefore compute both moments and normalize them independently. Let =gθ1∗(1)(y)∈ℝC×H×WF=g^(1)_ _1^*(y) ^C× H× W, and define F=1HW∑h,w:,h,w,F=1HW∑h,w(:,h,w−F)2. μ_F= 1HW _h,wF_:,h,w, σ_F= 1HW _h,w (F_:,h,w- μ_F )^2. (15) We ℓ2 _2-normalize each moment independently and concatenate them into a single embedding, =[F/∥F∥2∥F/∥F∥2]∈ℝ2C.e= [\, μ_F/\| μ_F\|_2\; \|\; σ_F/\| σ_F\|_2\, ] ^2C. (16) The domain prototype is the mean embedding over a subset tS_t of the domain’s training samples, t=1|t|∑i∈ti μ_t= 1|S_t| _i _te_i. All embeddings are extracted through the frozen backbone with no path engaged, so prototypes inhabit a single shared space and cross-domain cosine comparisons are well-posed. At inference, an unknown input y∗y^* is embedded via Eqs. (15)–(16) and routed to the path whose prototype it most resembles, t∗=argmaxt∈1,…,T⟨∗,t⟩‖∗‖2‖t‖2,t^*= _t∈\1,…,T\ ^*, μ_t \|e^*\|_2\,\| μ_t\|_2, (17) after which the image is restored by a forward pass through the selected path, x^∗=fθ1∗,Φt∗(y∗) x^*=f_ _1^*, _t^*(y^*). In practice we compute each prototype from ρ=50%ρ=50\% of the domain’s training samples, halving the prototype-extraction cost with negligible impact on routing accuracy. Fig. 2 summarizes the complete training and inference pipeline. 3.6 Continual Learning Evaluation Protocol Standard restoration benchmarks score a model on one degradation, or under joint access to all degradations, and thus cannot reveal how performance on earlier degradations evolves as later ones are learned. We adopt the continual learning evaluation framework of [4], organized around an evaluation matrix ∈ℝT×TR ^T× T whose entry Rb,jR_b,j is the restoration quality (PSNR or SSIM) on domain jD_j after the model has been trained through domain bD_b, defined for j≤bj≤ b. From this lower-triangular matrix we report three summaries, AB=1T∑j=1TRT,j,A¯=1T∑b=1T1b∑j=1bRb,j,ℱ=1T−1∑j=1T−1(Rj,j−RT,j),A_B= 1T _j=1^TR_T,j, A= 1T _b=1^T 1b _j=1^bR_b,j, = 1T-1 _j=1^T-1 (R_j,j-R_T,j ), (18) where ABA_B is the average final quality across all domains after the last stage, A¯ A is the average incremental quality over all stages and thus reflects stability throughout the sequence, and ℱF is the average forgetting, the mean drop on each domain relative to its peak, with ℱ=0F=0 denoting perfect retention. 4 Experiments 4.1 Experimental Setup Backbones and adapter paths. We instantiate RwF on two restoration backbones spanning the dominant architectural families: a Transformer, Restormer [59], and a fully convolutional network, NAFNet [7]. For each, we initialize from publicly released Gaussian-denoising weights (Restormer’s blind Gaussian color denoising checkpoint; NAFNet’s SIDD width-32 checkpoint) and freeze the backbone throughout. These choices instantiate 1D_1 directly: the pretrained backbone serves as the domain-1 restorer (Sec. 3.2), and only adapter paths are trained for 2D_2–5D_5. All adapter slots use a bottleneck dimension of r=64r=64, input LayerNorm, dropout 0.10.1, and a fixed scale s=1.0s=1.0, applied within the FFN of each Restormer block and the channel-mixing branch of each NAFNet block. Per-domain adapter parameters total 0.980.98M for Restormer (3.63%3.63\% of the 27.1127.11M backbone) and 1.291.29M for NAFNet (4.25%4.25\% of the 30.4530.45M backbone), scaling linearly with T rather than duplicating the backbone. Aggregated over the full five-domain sequence, RwF-Restormer totals one shared 27.1127.11M backbone plus four adapters (2D_2–5D_5), roughly 3131M parameters, versus 5×27.11=135.65×27.11=135.6M for five separate per-degradation specialists (over 4×4× fewer), while routing adds only a single stage-1 forward pass at inference and no trainable parameters. Restormer’s bias-free denoising LayerNorm [36] caused exploding gradients under the larger degradation shift, so we reintroduced the bias term. NAFNet required no analogous adjustment. Training and evaluation protocol. Each domain stage is trained for 100100k iterations with ℓ1 _1 pixel loss, AdamW (lr 2×10−42×10^-4, weight decay 10−410^-4, β=(0.9,0.999)β=(0.9,0.999)), and a single-cycle cosine schedule decaying to 10−610^-6, on 224×224224×224 patches with batch size 88 and standard geometric augmentations. The starting learning rate is reduced below the from-scratch defaults of Restormer (3×10−43×10^-4) and NAFNet (10−310^-3) because we fine-tune over a pretrained backbone. Identical hyperparameters are used for the sequential FT baseline and the RwF variant of each backbone; the only difference is which parameters are unfrozen. We deliberately omit progressive patch-size training, larger architectural variants, and extended schedules common in single-task restoration recipes, since our aim is to establish a continual learning baseline rather than a state-of-the-art per-task result. Each run uses a single NVIDIA A100 80GB GPU. We evaluate under two settings. The oracle setting supplies the ground-truth domain identity at test time, and the domain-agnostic setting reflects the true DIL scenario, where the domain label is withheld and the prototype router (Sec. 3.5) selects an adapter from the input alone. We report PSNR [17] and SSIM [55], summarized by the metrics ABA_B, A¯ A, and ℱF (Sec. 3.6). Following denoising-evaluation convention, 1D_1 is evaluated on CBSD68 [33] (the canonical test corpus for Restormer’s released denoising weights), with 1D_1 prototypes extracted on DIV2K-synthesized noise images to preserve a common content distribution across prototypes (Eq. (9)). 2D_2–5D_5 are evaluated on held-out DIV2K-synthesized test sets. Baselines. We compare RwF against four references. Sequential FT of the same two backbones provides the natural reference for catastrophic forgetting under unconstrained parameter updates. We add two regularization-based continual learning baselines, EWC [23] and LwF [28], applied to both backbones under conditions identical to sequential FT, with their regularization strengths tuned once at the 1→2D_1→D_2 transition and held fixed thereafter. As these methods share a single model, we evaluate them against RwF in the oracle setting. We further report a Joint all-in-one model trained on the union of all five domains as a non-continual reference. We do not retrain specialized all-in-one networks (PromptIR [39], AirNet [26], IDR [62], AdaIR [10], TransWeather [48], DCPT [19]) under our sequential protocol, treating them as complementary closed-world methods rather than direct competitors. Deployment on canonical real-degradation benchmarks is evaluated in Sec. 4.3. 4.2 Comparison with Continual Learning Baselines Table 1: Continual learning comparison across the five-domain sequence (1→5D_1→D_5: noise, blur, rain, haze, low-light) under the oracle setting. PSNR (dB) SSIM Method Backbone AB↑A_B\! A¯↑ A\! ℱ↓F\! AB↑A_B\! A¯↑ A\! ℱ↓F\! Sequential FT NAFNet 12.20 20.79 15.83 0.5697 0.6262 0.3287 Restormer 13.62 24.16 22.05 0.6148 0.7102 0.3705 EWC NAFNet 11.63 20.78 16.00 0.5376 0.6256 0.3366 Restormer 12.00 21.26 14.43 0.4521 0.6133 0.3193 LwF NAFNet 18.10 22.14 0.22 0.5559 0.6591 0.0305 Restormer 17.59 23.25 3.04 0.5361 0.6922 0.1351 Joint (All-in-one) NAFNet 25.49 – – 0.7809 – – Restormer 26.72 – – 0.8230 – – RwF (Ours) NAFNet 24.03 24.80 0.00 0.8003 0.7722 0.0000 Restormer 28.87 29.59 0.00 0.8744 0.8658 0.0000 Tab. 1 summarizes the five-domain continual trajectory under the oracle setting. Sequential FT suffers severe forgetting on both backbones. ℱF reaches 22.0522.05 dB for Restormer and 15.8315.83 dB for NAFNet, and the final-stage average ABA_B collapses to 13.6213.62 dB and 12.2012.20 dB respectively, far below the per-domain peaks captured in A¯ A. This gap between ABA_B and A¯ A, exceeding 1010 dB on Restormer and 88 dB on NAFNet, tracks the progressive overwriting of earlier-domain representations as each new degradation is absorbed. The standard continual-learning remedies fare little better under this degree of inter-degradation shift. EWC does not improve on naive fine-tuning, its ABA_B of 12.0012.00 dB on Restormer and 11.6311.63 dB on NAFNet sitting at or below sequential FT, with forgetting reduced only on Restormer. LwF holds forgetting to ℱF of 3.043.04 and 0.220.22 dB, but restricting a single shared model to every domain caps its final quality at ABA_B of 17.5917.59 and 18.1018.10 dB. Regularization of this kind, whether over gradients or outputs, preserves competence in class-incremental classification but not when successive domains demand qualitatively different restoration behaviors. RwF removes this collapse by construction. With zero forgetting, final ABA_B rises to 28.8728.87 dB for Restormer, a +15.25+15.25 dB gain over sequential FT, and to 24.0324.03 dB for NAFNet (+11.83+11.83 dB), narrowing the gap between ABA_B and A¯ A to under 11 dB on both backbones, with SSIM following the same trend. These results are competitive with joint all-in-one training over the five domains, which reaches 26.7226.72 and 25.4925.49 dB while optimizing on all degradations at once. RwF-Restormer exceeds its joint counterpart and RwF-NAFNet trails it only slightly, so assigning each degradation an isolated path recovers the quality of joint optimization without revisiting past data, and on the stronger backbone surpasses it by sidestepping cross-degradation interference. The behavior is consistent across architectures, holding for both a Transformer and a convolutional backbone under identical hyperparameters, with Restormer leading NAFNet by close to 55 dB under RwF. 4.3 Domain-Agnostic Inference on Real Degradations Under the domain-agnostic setting, the router must infer adapter identity (Sec. 3.5) from the degraded input alone. To test whether this transfers beyond the synthetic training corpus, we deploy RwF-Restormer on eleven canonical real-degradation benchmarks across all five domains without any retraining or recalibration: CBSD68 [33], Kodak24 [11], and Urban100 [20] with synthetic additive Gaussian noise at σ=25σ=25 for 1D_1; RealBlur-J and RealBlur-R [42] for 2D_2; Rain100H, Rain100L [58], and Test100 [61] for 3D_3; SOTS-Indoor and SOTS-Outdoor [25] for 4D_4; and LOL-v1 [56] for 5D_5, totaling 3,4653,465 images. Tab. 2 reports per-benchmark routing accuracy and reconstruction quality. Table 2: Domain-agnostic evaluation of RwF-Restormer on eleven real-degradation benchmarks. Routing accuracy and reconstruction quality are reported under both domain-agnostic (Pred.) and oracle settings. PSNR (dB) ↑ SSIM ↑ Domain Test set N ID Acc. (%) Pred. Oracle Pred. Oracle 1D_1 (noise) CBSD68 68 92.6 28.76 30.18 0.8448 0.8713 Kodak24 24 100.0 31.66 31.66 0.8754 0.8754 Urban100 100 90.0 28.74 30.75 0.8622 0.8906 2D_2 (blur) RealBlur-J 980 93.6 26.55 26.57 0.8349 0.8347 RealBlur-R 980 94.9 32.78 33.88 0.8962 0.9379 3D_3 (rain) Rain100H 100 100.0 16.92 16.92 0.4696 0.4696 Rain100L 100 83.0 30.18 32.13 0.8942 0.9291 Test100 98 70.4 21.20 21.64 0.6399 0.6762 4D_4 (haze) SOTS-Indoor 500 91.2 19.94 20.58 0.8728 0.8946 SOTS-Outdoor 500 72.4 25.50 27.93 0.9305 0.9625 5D_5 (low-light) LOL-v1 15 46.7 12.13 16.65 0.4495 0.7356 Overall 3,465 89.5 26.96 27.90 0.8568 0.8808 Figure 3: A 4D_4 hazy input (left) misrouted to the 3D_3 derain path is still largely restored (right). Overall routing accuracy reaches 89.5%89.5\% (3,101 / 3,465 images correctly routed), with a residual +0.94+0.94 dB PSNR gap to the oracle ceiling. Seven of the eleven benchmarks exceed 90%90\% routing accuracy, and two (Kodak24 noise, Rain100H) achieve 100%100\% with predicted and oracle reconstruction quality matching exactly. The framework therefore transfers to real degradations without adaptation, supporting the content-controlled design rationale (Sec. 3.2): isolating adapter learning from dataset-specific content priors lets the restoration paths generalize to unseen content of the same physical family. Crucially, misrouting is also graceful rather than catastrophic. As adjacent operators share restoration primitives (rain synthesis includes atmospheric veiling, low-light includes sensor noise; Sec. 3.3), a wrong route still applies a related correction rather than an arbitrary one, as in Fig. 3. Two cases, however, warrant attention. LOL-v1 is the weakest at 46.7%46.7\% routing accuracy, the regime our depth-discrimination argument predicts will be hard, since low-light’s signature is a global intensity statistic that early encoder features partially equalize. Oracle routing on the same data recovers 16.6516.65 dB, so the failure is localized to routing, not the adapter. Rain100H shows the opposite imbalance, routing perfectly (100%100\%) but reaching only 16.9216.92 dB because the dataset’s heavy-rain severity exceeds our 3D_3 synthesis distribution, an adapter generalization gap rather than a routing failure. 4.4 Ablation: Feature Readout Depth and Aggregation Two design choices govern the quality of the routing embedding: where along the backbone to read out the feature map, and how to summarize the resulting spatial tensor into a fixed-length vector. We ablate each independently on RwF-Restormer using the synthetic DIV2K-derived test set, which provides controlled comparison across configurations on identical content. The real-benchmark transfer of the selected configuration is reported in Sec. 4.3. Table 3: Routing quality vs. feature readout depth on RwF-Restormer. Readout location ID Acc. (%) Oracle gap (dB) Encoder stage 1 (Ours) 79.7 +2.01 Encoder stage 2 70.7 +2.54 Encoder stage 3 69.2 +2.81 Latent (bottleneck) 58.8 +4.14 Table 4: Routing quality vs. embedding aggregation. Aggregation ID Acc. (%) GAP only 73.9 GAP + std (joint ℓ2 _2) 78.4 GAP + std (sep. ℓ2 _2, Ours) 79.7 Tab. 4 reports routing accuracy and oracle PSNR gap as a function of readout depth, holding aggregation fixed at GAP + std with separate ℓ2 _2 normalization. Routing quality degrades monotonically with depth: encoder stage 1 achieves 79.7%79.7\% accuracy (+2.01+2.01 dB gap), while the bottleneck collapses to 58.8%58.8\% and +4.14+4.14 dB. This reverses the classification-CL convention that deeper pre-classifier features yield more discriminative prototypes. A restoration backbone, by design, suppresses the very signal that distinguishes the domains, so late features grow informative about the clean image but less so about what was removed. Aggregating across scales, as adopted for degradation classification in DCPT [19], does not overcome this. Concatenating stages 1–3 yields 77.1%77.1\% accuracy and adding the latent stage drops it to 72.0%72.0\%, both below the single shallow readout. This proves that combining scales only reintroduces the deeper features that dilute the degradation cue. Tab. 4 reports the effect of aggregation at the fixed encoder-stage-1 readout. GAP alone reaches 73.9%73.9\% accuracy. Adding per-channel spatial standard deviation (GAP + std, jointly ℓ2 _2-normalized) lifts it to 78.4%78.4\%, confirming that second-order spatial statistics carry meaningful degradation signal beyond the mean, and normalizing the two moments separately before concatenation lifts it further to 79.7%79.7\%. We adopt the separately-normalized variant on the basis of its higher domain-identification accuracy. 4.5 Qualitative Analysis Figure 4: Qualitative comparison across all five degradation domains for a shared scene. Top: degraded inputs. Middle: sequentially fine-tuned Restormer after the full 1→5D_1→D_5 schedule. Bottom: RwF-Restormer under domain-agnostic inference, with the router’s prediction (→t _t ✓). Per-image PSNR is shown in the lower-left of each restored panel. Fig. 4 visualizes the practical consequence of the quantitative gap in Tab. 1 on a single shared scene degraded under each of the five operators. Sequential fine-tuning recovers only 5D_5, the most recently trained domain (25.05 dB). On every earlier domain, the final model outputs heavily artifacted reconstructions, collapsing to 3.78 dB on 4D_4 where subsequent training stages have overwritten the model’s haze-removal capability. RwF, by contrast, produces visually consistent reconstructions across all five domains (23.84 – 35.15 dB), with the prototype router correctly selecting the appropriate restoration path for every input despite the substantial appearance differences between degraded variants. 5 Conclusion We formulated multi-degradation image restoration as a continual domain-incremental learning problem and introduced Restoring without Forgetting (RwF), a parameter-efficient framework that freezes a denoising-pretrained backbone, attaches network-spanning low-rank adapter paths under strict parameter isolation, and routes unknown inputs to the matching path via unsupervised prototype matching. On a content-controlled five-domain benchmark spanning noise, blur, rain, haze, and low-light, RwF eliminates catastrophic forgetting by construction and improves final-stage average PSNR by up to +15.25+15.25 dB over naive sequential fine-tuning, on both a Transformer and a convolutional backbone. The framework transfers without modification to eleven canonical real-degradation benchmarks (3,465 images), reaching 89.5%89.5\% routing accuracy and a +0.94+0.94 dB oracle PSNR gap, establishing, to our knowledge, the first systematic baseline of its kind. Two directions remain open. First, routing accuracy degrades on degradations whose signature is primarily global rather than spatially localized, low-light enhancement in particular. A degradation-aware embedding head or a differentiable router could absorb the remaining accuracy gap on such regimes. Second, designing adapter pathway primitives that explicitly capture degradation-specific structure through frequency-domain projections, kernel-aware re-parameterizations, or richer cross-block interactions, is a promising direction for further per-domain quality gains, particularly on real benchmarks whose distribution exceeds the synthesis range of the training corpus. References [1] E. Agustsson and R. Timofte (2017) Ntire 2017 challenge on single image super-resolution: dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, p. 126–135. Cited by: §3.3. [2] G. Boracchi and A. Foi (2012) Modeling the performance of image restoration from motion blur. IEEE Transactions on Image Processing 21 (8), p. 3502–3517. Cited by: §3.3. [3] B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao (2016) Dehazenet: an end-to-end system for single image haze removal. IEEE transactions on image processing 25 (11), p. 5187–5198. Cited by: §1. [4] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny (2018) Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420. Cited by: §2.2, §3.1, §3.6. [5] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. Torr, and M. Ranzato (2019) On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486. Cited by: §2.2, §3.1. [6] C. Chen, Q. Chen, J. Xu, and V. Koltun (2018) Learning to see in the dark. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3291–3300. Cited by: §3.3. [7] L. Chen, X. Chu, X. Zhang, and J. Sun (2022) Simple baselines for image restoration. In European conference on computer vision, p. 17–33. Cited by: §1, §2.1, §3.1, §3.2, §3.4, §4.1. [8] S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo (2022) Adaptformer: adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems 35, p. 16664–16678. Cited by: §2.2. [9] S. Cui, Y. Li, J. Li, X. Tang, B. Su, F. Xu, and H. Xiong (2025) Continual test-time adaptation for single image defocus deblurring via causal siamese networks. International Journal of Computer Vision 133 (7), p. 4134–4157. Cited by: §2.1. [10] Y. Cui, S. W. Zamir, S. Khan, A. Knoll, M. Shah, and F. S. Khan (2025) Adair: adaptive all-in-one image restoration via frequency mining and modulation. In 13th international conference on learning representations, ICLR 2025, p. 57335–57356. Cited by: §2.1, §4.1. [11] R. Franzen (1999) Kodak lossless true color image suite. Rochester, NY, USA. Cited by: §4.3. [12] R. M. French (1999) Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3 (4), p. 128–135. Cited by: §1, §3.1. [13] X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley (2017) Removing rain from single images via a deep detail network. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3855–3863. Cited by: §1. [14] K. Garg and S. K. Nayar (2007) Vision and rain. International Journal of Computer Vision 75 (1), p. 3–27. Cited by: §3.3. [15] X. Gong and J. Ma (2025) A minimalistic unified framework for incremental learning across image restoration tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1, §2.1. [16] K. He, X. Zhang, S. Ren, and J. Sun (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, p. 1026–1034. Cited by: §3.4. [17] A. Hore and D. Ziou (2010) Image quality metrics: psnr vs. ssim. In 2010 20th international conference on pattern recognition, p. 2366–2369. Cited by: §4.1. [18] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. Iclr 1 (2), p. 3. Cited by: §3.4. [19] J. Hu, L. Jin, Z. Yao, and Y. Lu (2025) Universal image restoration pre-training via degradation classification. In International Conference on Learning Representations, Vol. 2025, p. 64647–64668. Cited by: §2.1, §4.1, §4.4. [20] J. Huang, A. Singh, and N. Ahuja (2015) Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 5197–5206. Cited by: §4.3. [21] K. Jeeveswaran, E. Arani, and B. Zonooz (2024) Gradual divergence for seamless adaptation: a novel domain incremental learning method. arXiv preprint arXiv:2406.16231. Cited by: §2.2. [22] X. Jiang, X. Zhang, N. Gao, and Y. Deng (2024) When fast fourier transform meets transformer for image restoration. In European conference on computer vision, p. 381–402. Cited by: §1, §2.1. [23] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. (2017) Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114 (13), p. 3521–3526. Cited by: §2.2, §4.1. [24] L. Kong, J. Dong, J. Ge, M. Li, and J. Pan (2023) Efficient frequency domain-based transformers for high-quality image deblurring. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5886–5895. Cited by: §1, §2.1. [25] B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang (2018) Benchmarking single-image dehazing and beyond. IEEE transactions on image processing 28 (1), p. 492–505. Cited by: §3.3, §4.3. [26] B. Li, X. Liu, P. Hu, Z. Wu, J. Lv, and X. Peng (2022) All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 17452–17462. Cited by: §1, §2.1, §3.1, §4.1. [27] H. Li, X. Chen, J. Dong, J. Tang, and J. Pan (2025) Foundir: unleashing million-scale training data to advance foundation models for image restoration. In Proceedings of the IEEE/CVF international conference on computer vision, p. 12626–12636. Cited by: §2.1. [28] Z. Li and D. Hoiem (2017) Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence 40 (12), p. 2935–2947. Cited by: §2.2, §4.1. [29] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte (2021) Swinir: image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, p. 1833–1844. Cited by: §1, §2.1. [30] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee (2017) Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, p. 136–144. Cited by: §1. [31] Y. Liu, J. He, J. Gu, X. Kong, Y. Qiao, and C. Dong (2023) Degae: a new pretraining paradigm for low-level vision. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 23292–23303. Cited by: §2.1. [32] A. Mallya and S. Lazebnik (2018) Packnet: adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, p. 7765–7773. Cited by: §2.2. [33] D. Martin, C. Fowlkes, D. Tal, and J. Malik (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001, Vol. 2, p. 416–423. Cited by: §4.1, §4.3. [34] M. McCloskey and N. J. Cohen (1989) Catastrophic interference in connectionist networks: the sequential learning problem. In Psychology of learning and motivation, Vol. 24, p. 109–165. Cited by: §1, §3.1. [35] M. D. McDonnell, D. Gong, A. Parvaneh, E. Abbasnejad, and A. Van den Hengel (2023) Ranpac: random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems 36, p. 12022–12053. Cited by: §2.2, §3.2. [36] S. Mohan, Z. Kadkhodaie, E. P. Simoncelli, and C. Fernandez-Granda (2019) Robust and interpretable blind image denoising via bias-free convolutional neural networks. arXiv preprint arXiv:1906.05478. Cited by: §4.1. [37] S. Nah, T. Hyun Kim, and K. Mu Lee (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 3883–3891. Cited by: §1. [38] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter (2019) Continual lifelong learning with neural networks: a review. Neural networks 113, p. 54–71. Cited by: §1. [39] V. Potlapalli, S. W. Zamir, S. Khan, and F. S. Khan (2023) PromptIR: prompting for all-in-one blind image restoration. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Cited by: §1, §2.1, §3.1, §4.1. [40] C. Qin, R. Wu, Z. Liu, X. Lin, C. Guo, H. H. Park, and C. Li (2024) Restore anything with masks: leveraging mask image modeling for blind all-in-one image restoration. In European Conference on Computer Vision, p. 364–380. Cited by: §1, §2.1. [41] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun (2020) Towards robust monocular depth estimation: mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence 44 (3), p. 1623–1637. Cited by: §3.3. [42] J. Rim, H. Lee, J. Won, and S. Cho (2020) Real-world blur dataset for learning and benchmarking deblurring algorithms. In European conference on computer vision, p. 184–201. Cited by: §4.3. [43] X. Rui, Z. Li, Y. Cao, Z. Li, and W. Song (2023) DILRS: domain-incremental learning for semantic segmentation in multi-source remote sensing data. Remote Sensing 15 (10), p. 2541. Cited by: §2.2. [44] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. (2015) Imagenet large scale visual recognition challenge. International journal of computer vision 115 (3), p. 211–252. Cited by: §3.2. [45] H. Sun, D. Zhou, H. Zhao, L. Gan, D. Zhan, and H. Ye (2025) Mos: model surgery for pre-trained model-based class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 20699–20707. Cited by: §2.2, §3.2. [46] C. Tian, Y. Xu, and W. Zuo (2020) Image denoising using deep cnn with batch renormalization. Neural Networks 121, p. 461–473. Cited by: §1. [47] Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li (2022) Maxim: multi-axis mlp for image processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5769–5780. Cited by: §1. [48] J. M. J. Valanarasu, R. Yasarla, and V. M. Patel (2022) Transweather: transformer-based restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 2353–2363. Cited by: §1, §2.1, §4.1. [49] G. M. Van de Ven and A. S. Tolias (2019) Three scenarios for continual learning. arXiv preprint arXiv:1904.07734. Cited by: §1, §2.2, §3.1. [50] S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg (2013) Plug-and-play priors for model based reconstruction. In 2013 IEEE global conference on signal and information processing, p. 945–948. Cited by: §3.2. [51] G. Wang, L. Bai, Y. Wu, T. Chen, and H. Ren (2023) Rethinking exemplars for continual semantic segmentation in endoscopy scenes: entropy-based mini-batch pseudo-replay. Computers in Biology and Medicine 165, p. 107412. Cited by: §2.2. [52] L. Wang, X. Zhang, H. Su, and J. Zhu (2024) A comprehensive survey of continual learning: theory, method and application. IEEE transactions on pattern analysis and machine intelligence 46 (8), p. 5362–5383. Cited by: §1, §2.2. [53] Q. Wang, X. Song, Y. He, J. Han, C. Ding, X. Gao, and Y. Gong (2025) Boosting domain incremental learning: selecting the optimal parameters is all you need. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 4839–4849. Cited by: §2.2. [54] Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li (2022) Uformer: a general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 17683–17693. Cited by: §1. [55] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13 (4), p. 600–612. Cited by: §4.1. [56] C. Wei, W. Wang, W. Yang, and J. Liu (2018) Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560. Cited by: §3.3, §4.3. [57] C. Xu, W. Zhang, H. Zhang, X. Xu, H. Zhang, J. Zou, and J. Qin (2025) Fr2seg: continual segmentation across multiple sites via fourier style replay and adaptive consistency regularization. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence, p. 8815–8823. Cited by: §2.2. [58] W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, and S. Yan (2017) Deep joint rain detection and removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 1357–1366. Cited by: §3.3, §4.3. [59] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M. Yang (2022) Restormer: efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5728–5739. Cited by: §1, §2.1, §3.1, §3.2, §3.4, §4.1. [60] F. Zenke, B. Poole, and S. Ganguli (2017) Continual learning through synaptic intelligence. In International conference on machine learning, p. 3987–3995. Cited by: §2.2. [61] H. Zhang, V. Sindagi, and V. M. Patel (2019) Image de-raining using a conditional generative adversarial network. IEEE transactions on circuits and systems for video technology 30 (11), p. 3943–3956. Cited by: §4.3. [62] J. Zhang, J. Huang, M. Yao, Z. Yang, H. Yu, M. Zhou, and F. Zhao (2023) Ingredient-oriented multi-degradation learning for image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5825–5835. Cited by: §1, §2.1, §3.1, §4.1. [63] K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte (2021) Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (10), p. 6360–6376. Cited by: §3.2. [64] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang (2017) Beyond a gaussian denoiser: residual learning of deep cnn for image denoising. IEEE transactions on image processing 26 (7), p. 3142–3155. Cited by: §3.3. [65] D. Zhou, Z. Cai, H. Ye, D. Zhan, and Z. Liu (2025) Revisiting class-incremental learning with pre-trained models: generalizability and adaptivity are all you need. International Journal of Computer Vision 133 (3), p. 1012–1032. Cited by: §2.2. [66] D. Zhou, Z. Cai, H. Ye, L. Zhang, and D. Zhan (2025) Dual consolidation for pre-trained model-based domain-incremental learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 20547–20557. Cited by: §2.2. [67] D. Zhou, H. Sun, H. Ye, and D. Zhan (2024) Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 23554–23564. Cited by: §2.2, §3.2.