Paper deep dive
Spectral Super-Resolution using Spatial-Spectral Residual Operator Networks
Seokhyun Chin
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Spectral super-resolution of multispectral satellite images can enable high temporal- and spatial-resolution hyperspectral satellite imagery at a modest cost, significantly increasing the applicability of hyperspectral remote sensing. This task is inherently ill-posed, making it well-suited for deep learning-based methods. In this study, the spectral super-resolution task is framed as an operator learning problem, and SSRON is proposed as a Deep Operator Network that effectively learns function-to-function mappings from downsampled spectra to continuous spectra. The model is trained to super-resolve Sentinel-2A-like multispectral imagery to EMIT images. Compared to baseline models, SSRON achieves superior performance across all metrics. The model also demonstrates zero-shot spectral super-resolution capability by predicting bands unseen during training. Furthermore, its continuous-output formulation suggests the potential to estimate spectra at finer wavelength intervals than the native sensor. These results suggest the potential of SSRON and establishes operator learning as a promising direction for spectral super-resolution.
Tags
Links
- Source: https://arxiv.org/abs/2609.35410v1
- Canonical: https://arxiv.org/abs/2609.35410v1
Trouble viewing inline? Open PDF directly →
Full Text
24,220 characters extracted from source content.
Expand or collapse full text
SPECTRAL SUPER-RESOLUTION USING SPATIAL-SPECTRAL RESIDUAL OPERATOR NETWORKS Seokhyun Chin Affiliation: California Institute of Technology 1200 E California Boulevard, Pasadena, CA, USA schin@caltech.edu Abstract Spectral super-resolution of multispectral satellite images can enable high temporal- and spatial-resolution hyperspectral satellite imagery at a modest cost, significantly increasing the applicability of hyperspectral remote sensing. This task is inherently ill-posed, making it well-suited for deep learning-based methods. In this study, the spectral super-resolution task is framed as an operator learning problem, and SSRON is proposed as a Deep Operator Network that effectively learns function-to-function mappings from downsampled spectra to continuous spectra. The model is trained to super-resolve Sentinel-2A-like multispectral imagery to EMIT images. Compared to baseline models, SSRON achieves superior performance across all metrics. The model also demonstrates zero-shot spectral super-resolution capability by predicting bands unseen during training. Furthermore, its continuous-output formulation suggests the potential to estimate spectra at finer wavelength intervals than the native sensor. These results suggest the potential of SSRON and establishes operator learning as a promising direction for spectral super-resolution. Index Terms: Spectral Super Resolution, Deep Operator Networks, Neural Operator, Hyperspectral Satellite Imagery I Introduction Hyperspectral satellite images (HSIs) have become important tools in earth observation. By capturing hundreds of spectral bands, they provide detailed spectral information that is unavailable in conventional satellite imagery. As a result, HSIs have been widely used in applications such as weather forecasting [23], precision agriculture [22], and water quality monitoring [4]. However, their broader use is limited by low spatio-temporal resolution and high data acquisition costs [12]. To address these limitations, three main approaches have been developed. Spatial Super Resolution seeks to directly enhance the spatial resolution of an HSI; HSI sharpening fuses HSIs with high-resolution multispectral satellite images (MSIs) to improve spatial detail; Spectral Super Resolution (SSR) seeks to enhance the spectral resolution of an MSI to obtain an HSI. HSI sharpening suffers from the lack of high-spatial-resolution HSI reference data, and Spatial Super Resolution cannot overcome the low temporal resolution of HSI or the high cost of HSI acquisition [27]. Consequently, SSR provides a more practical and flexible alternative. However, SSR is an ill-posed inverse problem in which hundreds of HSI bands are recovered from only a limited number of MSI bands. Traditional SSR is based on shallow encoding methods such as dictionary-based methods [2, 11, 6]. With the emergence of deep learning methods, research has increasingly shifted toward their use. Specifically, variations of convolutional neural networks, generative adversarial networks, and transformer-based architectures have shown strong performance [14, 19, 31, 5, 7, 24]. Neural operators have emerged as a deep learning framework that learns mappings between infinite-dimensional function spaces [17, 15]. This enables continuous function evaluation, which makes it particularly well-suited for super-resolution. Indeed, various neural operator architectures have shown promise in spatial [26, 10] and temporal [29] super-resolution tasks, but their application to SSR remains limited. While prior work, such as Zhang et al. [30], has explored the integration of neural operator architectures for SSR, the direct application of neural operators for SSR remains largely unexplored, with no prior work explicitly formulating the problem as an operator learning problem. In this paper, SSR is cast as an operator learning problem and addressed using a neural operator framework. The key contributions are as follows. • The SSR problem is formulated as an operator learning problem, laying the mathematical groundwork for applying neural operators to SSR. • Spatial-Spectral Residual deep Operator Network (SSRON) is proposed, which applies the Deep Operator Network (DeepONet) architecture to SSR while using a spatial–spectral residual CNN as the branch network–adapting architectural principles proven effective in SSR to the operator learning framework. While CNNs have been used in the branch networks of DeepONets [18, 9], this paper, to the best of the author’s knowledge, is the first to adapt a spatial–spectral residual CNN architecture to the operator-learning framework for spectral super-resolution. • The strength of SSRON in SSR is demonstrated, achieving improved performance compared to state-of-the-art methods. The application of this architecture to zero-shot SSR for unseen bands during training and its potential to leverage infinite-dimensional mappings is also demonstrated. I Problem Formulation The sensor calibrated radiance value L(λ)L(λ) can be written as [5] L(λ,x→)=I(λ,x→)R(λ,x→),L(λ, x)=I(λ, x)R(λ, x), where I denotes solar radiation and R is the reflectance of the region that the sensor takes its value from. Both the MSI and HSI are downsampled from this continuous function, with Lm(i,x→)=∫L(λ,x→)gm(i,λ)λ L_m(i, x)= L(λ, x)g_m(i,λ)dλ Lh(i,x→)=∫L(λ,x→)gh(i,λ)λ, L_h(i, x)= L(λ, x)g_h(i,λ)dλ, where gm(i,λ),gh(i,λ)g_m(i,λ),g_h(i,λ) are the Spectral Response Function (SRF) of iith band of the multispectral and hyperspectral instruments, respectively. Because the hyperspectral instruments’ SRF are centered about a central wavelength λi _i, we can approximate, Lh(λi,x→)≈L(λi,x→)∫gh(i,λ)λ=L(λi,x→). L_h( _i, x)≈ L( _i, x) g_h(i,λ)dλ=L( _i, x). Now consider the functional L′(g→,x→)≔∫L(λ,x→)g→(λ)λ=Lg→(x→).L ( g, x) L(λ, x) g(λ)dλ= L_g( x). Effectively, the MSI can be written as L′(gm→,x→)L ( g_m, x). The SSR problem, therefore, can be framed as an operator learning problem that learns G:L′(g→,x→)↦L(λ,x→)G:L ( g, x) L(λ, x). In this study, we solve a specific case of the problem where g is fixed to some gmg_m though extension of this problem to multiple ggs are possible. I Methodologies I-A Architecture The proposed architecture is based on the DeepONet [17], which creates a branch network that encodes the input function at discrete sensor points and a trunk network that encodes locations where the output is evaluated. The overall architecture is outlined in Figure 1. (a) (b) Fig. 1: Proposed architecture of SSRON. The framework consists of a branch network and trunk network, as shown in (a). The branch network encodes the MSI into a latent representation, and the trunk network encodes the spatial-spectral query positions. (b) shows the specific architecture of the branch network. The standard multi-layer perceptron branch network is replaced with a spatial–spectral residual CNN, motivated by the strong performance of such architectures in remote sensing SSR [31, 21, 3]. The embedding layer in Figure 1a maps each coordinate value to a vector according to the following: Embedding:c↦[csin(f0⋅c)cos(f0⋅c)sin(fj⋅c)cos(fj⋅c)]Embedding:c bmatrixc\\ (f_0· c)\\ (f_0· c)\\ \\ (f_j· c)\\ (f_j· c) bmatrix where fi=2if_i=2^i. The coordinates (x,y,λ)(x,y,λ) are each embedded independently, and the resulting vectors are concatenated. The trunk network is a simple two-layer fully-connected neural network with 256 hidden dimensions. I-B Dataset The EMIT Imaging Spectrometer’s L1B at sensor calibrated radiance [8] was used as the HSI source. EMIT is an advanced imaging spectrometer instrument on the International Space Station that outputs 285 discrete bands from the visible to the short-infrared. Bands 74-79, 100-107, 131-143, and 191-217 were dropped due to absorption via water vapor and ozone, leaving a total of 229 bands. 10 scenes were randomly selected from December 2024 with zero cloud coverage, downloaded from NASA’s Earthdata Search. Details of the selected scenes are presented in Table I. Spatio-temporally aligned MSIs are generated by downsampling the HSI using the spectral response function of the Sentinel 2A satellite [1]. The initial SRF was downsampled to the 229 bands and normalized, following the approach shown in [14]. Band 9 was dropped because it samples corrupted bands in the HSI. The downsampled SRF is shown in Figure 2. TABLE I: Dataset specifications Scene no. Granule ID (excluding common prefix) 1 20241214T150837_2434910_009 2 20241214T151321_2434910_033 3 20241215T094506_2435006_010 4 20241215T203042_2435013_020 5 20241216T071956_2435105_007 6 20241216T193914_2435113_006 7 20241217T093419_2435206_006 8 20241223T050524_2435803_051 9 20241230T150153_2436510_038 10 20241231T141323_2436609_033 Fig. 2: Normalized and downsampled SRF of the Sentinel 2A satellite I-C Training Non-overlapping 16x16 patches were selected from each scene, resulting in a total of 61,600 patches. An 8:1:1 train-validation-test split was used. The models were trained with an Adam optimizer at a learning rate of 5e-5, with cosine annealing and early stopping with patience of 5 epochs for 300 epochs, using Mean Absolute Error (MAE) as the loss function. All training was conducted on a single RTX 5060 Ti GPU. I-C1 Training SSRON Unlike conventional image-to-image models, the SSRON is trained to predict the value of the learned function at a specified spatial and spectral coordinates. To enable this, 2048 possible positions (spatial and spectral) were selected in the HSI label to be predicted by the SSRON for each epoch. I-C2 Zero-shot super resolution As neural operators learn maps between infinite-dimensional function spaces, they can be evaluated at previously unseen coordinates. To test this capacity, a fixed percentage of bands was intentionally excluded from training, and the model’s performance was evaluated across all bands. The excluded bands were selected to be relatively equidistant and to span the same wavelength range as the data. I-D Baselines AWAN, Restormer, and SSRAN architectures were selected as state-of-the-art baselines because they have demonstrated strong performance for SSR tasks [31, 28, 13]. The U-shaped Neural Operator (UNO) [20] and the Fourier Neural Operator (FNO) [15] were included as comparable baselines for operator learning. The RSNO proposed by Zhang et al. [30] was not included as a baseline because it relies on additional physics-based information as input to the model, which was neither available nor assumed in this experimental setting. I-E Metrics In addition to MAE, Root Mean Squared Error (RMSE), Peak Signal to Noise Ratio (PSNR), and Structural Similarity Score (SSIM) were used as evaluation metrics. SSIM is calculated like so [25]: SSIM(x,y)=(2μxμy+ϵx)(2σxy+ϵy)(μx2+μy2+ϵx)(σx2+σy2+ϵy),SSIM(x,y)= (2 _x _y+ _x)(2 _xy+ _y)( _x^2+ _y^2+ _x)( _x^2+ _y^2+ _y), where μx _x and μy _y are the mean values of images x and y, σx2 _x^2 and σy2 _y^2 are the variances, ϵx=(0.01⋅L)2,ϵy=(0.03⋅L)2 _x=(0.01· L)^2, _y=(0.03· L)^2 are small constants to stabilize division with L being the dynamic range of data, and σxy _xy is the covariance of x and y. IV Results Table I reports the results evaluated over the testing dataset. SSRON outperforms all baseline models across all tested metrics. UNO also performs strongly, outperforming all non-operator baseline models. In contrast, the FNO shows substantially higher errors, which may be attributed to the sharpness in spectra that needs to be reconstructed; standard FNOs struggle with such high-dimensional features [16]. TABLE I: Error metrics over testing dataset Model type MAE ↓ RMSE ↓ PSNR ↑ SSIM ↑ AWAN 0.01529 0.03476 50.92 0.9861 FNO 0.1690 0.2745 31.23 0.2516 Restormer 0.01606 0.03670 50.24 0.9845 SSRAN 0.01871 0.03999 49.29 0.9820 UNO 0.01498 0.03255 51.01 0.9865 SSRON 0.01183 0.02599 52.67 0.9889 Figure 3 shows the per-band errors of each model. SSRON achieves a lower error across all bands, with the exception of band 115, where the UNO shows a lower error. A sharp increase in error is observed in the bands below 5 and above 200. This behavior is likely due to the limited spectral information of the Sentinel-2A satellite in those regions, as shown in Figure 2. The relationship between SRF coverage and the errors is further supported by Pearson’s correlation test. The correlation between the MAE (↓ ), RMSE(↓ ), and PSNR(↑ ) with the SRF value at each wavelength is -0.222, -0.175, and 0.408 (p << 0.01), respectively, demonstrating a clear inverse relationship between error and SRF values. Table I reports the zero-shot spectral resolution performance of the SSRON. As the percentage of bands used decreases, MAE increases and SSIM decreases, indicating that zero-shot inference is difficult. RMSE and PSNR do not strictly follow the monotonic trend, with the 75% model Fig. 3: Per-band (a) MAE, (b) PSNR, (c) RMSE, and (d) 1-SSIM value comparison. FNO is excluded from the analysis due to extremely high errors. SSIM is reported as 1-SSIM and on a log scale for enhanced visibility. demonstrating a lower RMSE than the 80% model and the 60% model demonstrating a lower RMSE and higher PSNR than the 65% model. Such behavior likely relates to which bands were removed for training. If bands with higher errors in figure 3 were removed, the model would not be penalized for poor performance for those bands, increasing the error of the final trained model. TABLE I: Error metrics over testing dataset with reduced number of bands seen during training % bands used MAE ↓ RMSE ↓ PSNR ↑ SSIM ↑ 100 0.01183 0.02599 52.67 0.9889 95 0.01422 0.04756 48.02 0.9764 90 0.01509 0.04836 47.50 0.9748 85 0.01701 0.05290 46.64 0.9730 80 0.01844 0.05638 46.10 0.9716 75 0.01946 0.05599 45.64 0.9707 70 0.02077 0.07779 43.27 0.9690 65 0.02615 0.08810 41.92 0.9644 60 0.02837 0.08463 42.32 0.9620 55 0.03239 0.1046 40.62 0.9557 50 0.03929 0.1075 39.84 0.9502 Nevertheless, SSRON remains competitive with the state-of-the-art performance at 90% of bands used and attains a lower MAE than other methods. Because SSRON takes wavelengths as a continuous input, the model can in-principle be evaluated between the native sensor bands. The zero-shot results suggest the potential for such spectral interpolation, although validation against denser spectral measurements would be needed to confirm the accuracy of HSI beyond the native discretization. V Conclusion and future directions This work presents SSRON, a deep operator network that uses spatial-spectral convolutions with residual connections as its branch network. Empirically, the SSRON achieves state-of-the-art accuracy on simple SSR and demonstrates its potential for zero-shot prediction of additional bands. The paper offers a novel perspective on the SSR task and opens a new direction for applications of operator learning. Future work should focus on optimizing neural operator architectures to further improve model performance. In addition, the continuous spectral representation enabled by this framework motivates further study of the densely sampled spectral interpolation for downstream applications, such as materials classification of imaged regions. Finally, varying the input SRF to support the general application of SSRON to various satellites is an important direction for future work. VI Acknowledgements The author is grateful to receive funding from the George W. Housner Student Discovery Fund for support in the research and attendance at the conference. The author would like to thank Enes Banushi for productive discussion on the problem formulation. References [1] E. S. A. (ESA) (4) Sentinel-2 spectral response functions. External Links: Link Cited by: §I-B. [2] B. Arad and O. Ben-Shahar (2016) Sparse recovery of hyperspectral signal from natural rgb images. In Computer Vision – ECCV 2016, p. 19–34. External Links: ISBN 9783319464787, ISSN 1611-3349, Document Cited by: §I. [3] S. O. Atik (2024) Dual-stream spectral-spatial convolutional neural network for hyperspectral image classification and optimal band selection. Advances in Space Research 74 (5), p. 2025–2041. External Links: ISSN 0273-1177, Document Cited by: §I-A. [4] S. Chander, A. Gujrati, A. V. Krishna, A. Sahay, and R.P. Singh (2020) Remote sensing of inland water quality: a hyperspectral perspective. In Hyperspectral Remote Sensing, p. 197–219. External Links: ISBN 9780081028940, Document Cited by: §I. [5] D. Du, Y. Gu, T. Liu, and X. Li (2023) Spectral reconstruction from satellite multispectral imagery using convolution and transformer joint network. IEEE Transactions on Geoscience and Remote Sensing 61, p. 1–15. External Links: ISSN 1558-0644, Document Cited by: §I, §I. [6] K. Fotiadou, G. Tsagkatakis, and P. Tsakalides (2019) Spectral super resolution of hyperspectral images via coupled dictionary learning. IEEE Transactions on Geoscience and Remote Sensing 57 (5), p. 2777–2797. External Links: Document Cited by: §I. [7] R. Gonzalez, C. M. Albrecht, N. A. Ali Braham, D. Lambhate, J. L. De Sousa Almeida, P. Fraccaro, B. Blumenstiel, T. Brunschwiler, and R. Bangalore (2025) Multispectral to hyperspectral using pretrained foundational model. In IGARSS 2025 - 2025 IEEE International Geoscience and Remote Sensing Symposium, Vol. , p. 785–789. External Links: Document Cited by: §I. [8] R. Green (2022) EMIT l1b at-sensor calibrated radiance and geolocation data 60 m v001. NASA Land Processes Distributed Active Archive Center. External Links: Document, Link Cited by: §I-B. [9] Z. Guo, L. Chai, S. Huang, and Y. Li (2024) Inversion-deeponet: a novel deeponet-based network with encoder-decoder for full waveform inversion. arXiv preprint arXiv:2408.08005. Cited by: 2nd item. [10] L. Han and X. Zhang (2024) Scalable super-resolution neural operator. In Proceedings of the 32nd ACM International Conference on Multimedia, M ’24, New York, NY, USA, p. 10036–10045. External Links: ISBN 9798400706868, Link, Document Cited by: §I. [11] X. Han, J. Yu, J. Luo, and W. Sun (2019) Reconstruction from multispectral to hyperspectral image using spectral library-based dictionary learning. IEEE Transactions on Geoscience and Remote Sensing 57 (3), p. 1325–1335. External Links: Document Cited by: §I. [12] S. Imran, S. Mandal, A. Goswami, M. Thakur, and A. Raju (2024) A novel deep learning-based landsat 7 etm+ multi-spectral to hyperspectral reconstruction model: application for water bodies in an indian region. In IGARSS 2024 - 2024 IEEE International Geoscience and Remote Sensing Symposium, Vol. , p. 3121–3124. External Links: Document Cited by: §I. [13] J. Li, C. Wu, R. Song, Y. Li, and F. Liu (2020) Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from rgb images. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vol. , p. 1894–1903. External Links: Document Cited by: §I-D. [14] T. Li and Y. Gu (2022) Progressive spatial–spectral joint network for hyperspectral image reconstruction. IEEE Transactions on Geoscience and Remote Sensing 60 (), p. 1–14. External Links: Document Cited by: §I, §I-B. [15] Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandkumar (2021) Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, External Links: Link Cited by: §I, §I-D. [16] M. Liu-Schiaffini, J. Berner, B. Bonev, T. Kurth, K. Azizzadenesheli, and A. Anandkumar (2024) Neural operators with localized integral and differential kernels. arXiv preprint arXiv:2402.16845. Cited by: §IV. [17] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis (2021) Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nat. Mach. Intell. 3 (3), p. 218–229 (en). Cited by: §I, §I-A. [18] Y. Mei, Y. Zhang, X. Zhu, R. Gou, and J. Gao (2024) Fully convolutional network-enhanced deeponet-based surrogate of predicting the travel-time fields. IEEE Transactions on Geoscience and Remote Sensing 62 (), p. 1–12. External Links: Document Cited by: 2nd item. [19] K. Mu, Z. Zhang, Y. Qian, S. Liu, M. Sun, and R. Qi (2022) SRT: a spectral reconstruction network for gf-1 pms data based on transformer and resnet. Remote Sensing 14 (13), p. 3163. External Links: ISSN 2072-4292, Document Cited by: §I. [20] M. A. Rahman, Z. E. Ross, and K. Azizzadenesheli (2023) U-NO: u-shaped neural operators. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §I-D. [21] Z. Shu, Z. Liu, J. Zhou, S. Tang, Z. Yu, and X. Wu (2023) Spatial–spectral split attention residual network for hyperspectral image classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 16 (), p. 419–430. External Links: Document Cited by: §I-A. [22] P. Singh, P. C. Pandey, G. P. Petropoulos, A. Pavlides, P. K. Srivastava, N. Koutsias, K. A. K. Deng, and Y. Bao (2020) Hyperspectral remote sensing in precision agriculture: present status, challenges, and future trends. In Hyperspectral Remote Sensing, p. 121–146. External Links: ISBN 9780081028940, Document Cited by: §I. [23] W. L. Smith, Q. Zhang, M. Shao, and E. Weisz (2020) Improved severe weather forecasts using leo and geo satellite soundings. Journal of Atmospheric and Oceanic Technology 37 (7), p. 1203–1218. External Links: ISSN 1520-0426, Document Cited by: §I. [24] T. J. Vandal, D. McDuff, W. Wang, K. Duffy, A. Michaelis, and R. R. Nemani (2022) Spectral synthesis for geostationary satellite-to-satellite translation. IEEE Transactions on Geoscience and Remote Sensing 60, p. 1–11. External Links: ISSN 1558-0644, Document Cited by: §I. [25] Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), p. 600–612. External Links: ISSN 1941-0042, Document Cited by: §I-E. [26] M. Wei and X. Zhang (2023) Super-resolution neural operator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 18247–18256. Cited by: §I. [27] J. Xie, L. Fang, C. Wu, F. Xie, and J. Chanussot (2024) Blind spectral super-resolution by estimating spectral degradation between unpaired images. IEEE Transactions on Geoscience and Remote Sensing 62 (), p. 1–14. External Links: Document Cited by: §I. [28] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M. Yang (2022) Restormer: efficient transformer for high-resolution image restoration. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 5718–5729. External Links: Document Cited by: §I-D. [29] Y. Zhang, H. Zheng, D. Yang, Z. Chen, H. Ma, and W. Ding (2025) Space-time video super-resolution with neural operator. IEEE Transactions on Image Processing 34, p. 6742–6754. External Links: ISSN 1941-0042, Document Cited by: §I. [30] Z. Zhang, B. Pan, and Z. Shi (2025) Radiative-structured neural operator for continuous spectral super-resolution. arXiv. External Links: Document Cited by: §I, §I-D. [31] X. Zheng, W. Chen, and X. Lu (2022) Spectral super-resolution of multispectral images using spatial–spectral residual attention network. IEEE Transactions on Geoscience and Remote Sensing 60 (), p. 1–14. External Links: Document Cited by: §I, §I-A, §I-D.