Paper deep dive

Implicit Neural Representation for Hyperspectral Video Compression

Alfredo Scalera, Paul Murray, Jaime Zabalza

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 22

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:With the advent of snapshot cameras, hyperspectral video is becoming more readily available. In recent years, new applications have emerged which have led to increasingly larger datasets. However, hyperspectral video compression remains in the early stages. In this study, we explore the use of implicit neural representation as a candidate solution. We propose a novel extension of an existing RGB video compression model, achieving Bjøntegaard Delta PSNR gains of +4.99 dB and Bjøntegaard Delta rate of -88.88% compared to traditional hyperspectral image compression methods applied frame-by-frame. In addition to reconstruction quality, the effects on downstream task performance are measured in the form of object tracking success. Compared to video compressed with methods based on principal component analysis and JPEG2000 in low data regimes, our proposed method improves tracking area under the curve by up to 23.42% and distance precision by up to 35.56% on examples from the HOT2026 dataset.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

19,636 characters extracted from source content.

Expand or collapse full text

Implicit Neural Representation for Hyperspectral Video Compression Abstract With the advent of snapshot cameras, hyperspectral video is becoming more readily available. In recent years, new applications have emerged which have led to increasingly larger datasets. However, hyperspectral video compression remains in the early stages. In this study, we explore the use of implicit neural representation as a candidate solution. We propose a novel extension of an existing RGB video compression model, achieving Bjøntegaard Delta PSNR gains of +4.99+4.99 dB and Bjøntegaard Delta rate of −88.88%-88.88\% compared to traditional hyperspectral image compression methods applied frame-by-frame. In addition to reconstruction quality, the effects on downstream task performance are measured in the form of object tracking success. Compared to video compressed with methods based on principal component analysis and JPEG2000 in low data regimes, our proposed method improves tracking area under the curve by up to 23.42%23.42\% and distance precision by up to 35.56%35.56\% on examples from the HOT2026 dataset. Name of author Address - Line 1 Address - Line 2 Address - Line 3 Index Terms— Hyperspectral video, Video compression, Implicit neural representation, Computer vision, Dimensionality reduction 1 Introduction Hyperspectral (HS) imagery is commonly used in: quality assessment of food; agriculture; remote sensing; and medical analysis, among others [2]. Through technological innovations in camera sensors, HS videos have become more prevalent in recent years. The advent of snapshot hyperspectral sensors has made a number of varied applications emerge, from HS object tracking [19] to scene segmentation [1]. Yet, the field dedicated to HS video compression is still in its infancy and, so far, image compression methods for HS data have been mainly applied frame-by-frame [6]. In this paper a more wholistic approach is taken towards compressing HS video. Consequently, we propose the use of implicit neural representation (INR) for HS video compression [4]. INR-based methods leverage the strong data representations enabled by neural networks while avoiding the need for large datasets. The main contributions in this paper are: Novel extension of INR-based compression method to HS; Comparison against common PCA-based approach for HS imagery; Quantifying gains from the INR-based method compared to current practices. 2 Related Work Video compression in the domain of RGB video is a mature field in both academic and industrial settings. Most video systems today are built on top of standardised classical hybrid codecs such as AVC [18], HEVC [14], and VVC [3] which have been developed in the last 25 years. Over the past decade, with the advent of deep learning, research and development in learned video codecs has emerged. These can broadly be split into two main categories: general and overfit video codecs. General learned video codecs search compressing functions that are learned from end-to-end training on large amounts of data [10, 8]. Overfit video codecs, on the other hand, leverage the overfitting abilities of neural networks thus obtaining an approximation of the functions that generate each pixel or frame of a video. Examples include HNeRV [4], NVRC [11], and COOL-CHIC [12]. In recent years, snapshot HS video cameras have emerged due to innovations in mosaic-array sensors [5]. This has led to the collection and release of various HS video datasets such as HS object tracking (HOT) challenge (HOT2020 [19], HOT2026 [9], IMEC25 [5]) and automated driving systems (HSI-Drive [1]). HS video compression is often achieved by compressing each datacube frame individually, making use of the spectral correlation between channels [17, 15]. The work in [6] uses sparse tucker tensor decompositions per-frame, outperforming other HS image compression methods applied in the same manner. Authors in [16] propose an HS video coder by building on top of an HS image coder to compensate both spectrally and temporally, achieving higher compression rates than applying only spectral prediction. To the best of our knowledge, overfit neural compression methods are yet to be applied to the field of HS video compression. In this work we explore their application to the problem and specifically target INR to show the potential of overfit compression. 3 Methods In the following sections, we detail the methods employed in this study, as well as the experimental setup. Fig. 1: HS video compression with PCA+JPEG2000. The projection matrix P is quantized to P before being written into the bitstream. Q represents min-max quantization and =(min,scale) s_P=( P_min, P_scale) and analogously for m. 3.1 PCA-based Video Compression PCA is commonly used for spectral dimensionality reduction, and its combination with RGB compression standards has been proposed for HS image and video compression. Here we explore PCA and JPEG2000 as first proposed in [7]. The technique is chosen as a strong baseline for overfit HS video compression (since the principal components must be fit to each frame or video). Furthermore, both PCA and JPEG2000 are familiar to the hyperspectral community and readily deployable. PCA is applied individually to each frame and linearly projects the input datacube ∈ℝCin×H×W D ^C_in× H× W from CinC_in channels to Cout=3C_out=3 channels to fit JPEG2000. The reduced datacube is then compressed with JPEG2000 to the desired target bits-per-pixel (bpp). The PCA projection matrix ∈ℝCin×Cout P ^C_in× C_out and mean vector ∈ℝCin m ^C_in are min-max quantized to improve memory footprint (Eq. 1). q=⌊Qmax−minscale⌉q= Q_max x- x_min x_scale (1) Where x represents the full-precision value, q is the corresponding quantized value, QmaxQ_max the maximum of the quantized range, and ⌊⋅⌉ · is the rounding operation. Symmetric signed min-max quantization requires storing the minimum values min=min⁡() x_min= ( x) the scale factors scale=max⁡()−min⁡() x_scale= ( x)- ( x) for dequantization. These are included in the JPEG2000 bitstream as side information =(min,scale,) s_P=( P_min, P_scale,) and =(min,scale,) s_m=( m_min, m_scale,). When decoding the inverse operations are applied: first the JPEG2000 bitstream is decoded, the projection matrix and mean vector are de-quantized, and finally inverse PCA is applied to the reduced datacube to restore all CinC_in channels, obtaining the decompressed datacube ~ D. 3.2 INR-based Video Compression Implicit neural compression can be leveraged in various ways for video compression. Forms include compressed models that learn to map pixel or frame coordinates to pixel values or frame tensors; hybrid schemes where the latent variables and decoders of autoencoder-like networks are compressed [4]; and fully end-to-end overfit video codecs [12, 11]. In this paper, we adapt HNeRV [4] for HS videos. HNeRV is a hybrid RGB approach, treating video compression as a model and latent compression problem (as shown in Fig. 2). We apply three extensions to HNeRV: the encoder and decoder networks are adapted to allow multiple channel inputs, L2L2 loss is replaced with bespoke loss HSFusionHSFusion, and a hyperparameter search was conducted to find a more optimal combination for the HOT2026 dataset. We refer to the hyperspectral version as HS-HNeRV The HSFusionHSFusion loss (Eq. 2) combines L2 and cosine-similarity losses. The former targets general reconstruction, while the latter ensures higher spectral fidelity in each pixel. HSFusion=λ⋅L2⁡(,~)+(1−λ)⋅Sim⁡(,~)HSFusion=λ·L2 ( D, D )+(1-λ)·Sim ( D, D ) (2) Here D and ~ D are the original and reconstructed frame datacubes, respectively. SimSim is cosine-similarity, while λ∈[0,1]λ∈ [0,1 ] weighs the two losses. Fig. 2: HS video compression with HNeRV-based method. (a) High-level frame reconstruction for overfitting. (b) Compression - decoder parameters θdec _dec are shared across all video frames while each frame has a unique set of latent variables L. Huffman coding losslessly compresses quantized latents L and decoder parameters θ^dec θ_dec. 3.3 Experiments We measure the effectiveness of INR compared to PCA+JPEG2000 for HS video compression in two ways: (1) reconstruction quality metrics and (2) downstream task performance. PSNR and spectral angle mapper (SAM) are applied to measure the fidelity of the decompressed representations. The first measures general reconstruction, while the latter evaluates the spectral divergence of decoded pixels. Hyperspectral object tracking (HOT) is chosen as a representative downstream task to measure the impact of compression on computer vision algorithms. HOT consists of HS videos with moving targets, with the bounding box of a single target provided in the first frame. Algorithms track the object by temporally propagating the bounding box through each frame. We evaluate CSSTrack [13], which represents the current state-of-the-art for HOT tasks. AUC and DP are measured on both raw and decompressed sequences. AUC is the area under the curve of the success plot, which measures the percentage of frames where the intersection-over-union exceeds a particular threshold. DP is distance precision measured as the proportion of frames in which the bounding box center is below a distance error of 20 pixels. Four test video sequences were obtained from the HOT2026 [9] VIS training set 11 1 heartsurgery5, high_playground1, high_truck, snowcat. The sequences have a native resolution of 512×272512× 272 pixels and 16 spectral bands and were center cropped to 512×256512× 256 in all experiments (achieving a 2:1 aspect ratio to accommodate the model). HS-HNeRV model sizes and PCA+JPEG2000 target bpp are swept through and PSNR and SAM metrics are computed between the decompressed and original frames for each test sequence. To obtain per-video values, both PSNR and SAM are averaged across all frames. For performance on HOT tasks, AUC and DP are obtained over three HS-HNeRV model sizes and on PCA+JPEG2000 sequences with matching bpp. We use a CSSTrack checkpoint trained on HOT202022 2 https://github.com/hscv/CSSTrack. PCA projection matrix values are quantized to 8-bit unsigned integers after fitting, with minimum and scale factors (side information) embedded as 16-bit floats in the bitstream, as illustrated in Fig. 1. As for HS-HNeRV, the same hyperparameters are set for each video: 64 encoder channels, 16 latent channels, and a learning rate of 1.1e−31.1e-3 and trained for 300 epochs with λ=0.7λ=0.7. These were obtained via a random hyperparameter search with 15 trials on four videos from the HOT2026 validation set 33 3 athlete1, car8, cat2, pingpong4, maximizing peak signal-to-noise ratio (PSNR). To obtain different bpp values, the model size was varied (3.0, 1.5, 0.75, 0.375 million parameters, respectively). 4 Results Rate-distortion curves are provided in Fig. 3. Four HS-HNeRV sizes and six PCA+JPEG2000 bpp targets were evaluated. Values are averaged across all four test sequences. Over equivalent bpp ranges, HS-HNeRV achieves both higher PSNR and lower SAM scores. Specifically, HS-HNeRV achieves a Bjøntegaard Delta (BD)-rate of −88.88%-88.88\% and a BD-PSNR of 4.994.99 dB with respect to PCA+JPEG2000. Fig. 3: INR-based compression affords higher fidelity. HS-HNeRV surpasses PCA+JPEG2000 in both spatial (left) and spectral (right) reconstruction at similar compression levels. In low bitrate regimes, HS-HNeRV also outperforms PCA+JPEG2000 in maintaining tracking performance, as reported in Table 1. For bpp of approx. 0.130.13, HS-HNeRV improves over the former by 23.42%23.42\% and 35.56%35.56\% in tracking AUC and DP, respectively. On the other hand, PCA+JPEG2000 surpasses HS-HNeRV at lower compression rates. Interestingly, the DP of CSSTrack improves on sequences compressed with PCA+JPEG2000 at 0.60.6 bpp compared to the raw data. Table 1: CSSTrack performance on compressed HOT2026. Compression Method bpp AUC DP Raw Sequence – 43.5943.59 67.8467.84 PCA+JPEG2000 0.135 24.5524.55 42.8042.80 0.275 27.8727.87 46.4446.44 0.604 37.9237.92 69.9269.92 HS-HNeRV 0.133 30.3030.30 58.0258.02 0.276 33.3733.37 61.2461.24 0.568 35.0235.02 62.4862.48 5 Discussion INR-based methods show clear potential for HS video compression compared to PCA-based overfit approaches. Stronger representation is seen across all bitrates with HS-HNeRV (example illustrated in Fig. 5), achieving the same PSNR as PCA+JPEG2000 while requiring only 11.12%11.12\% of the data (BD-rate of −88.88%-88.88\%). The potential for low-data environments is also demonstrated by the increased HOT performance. Overall, an INR-based overfit approach removes the need for large amounts of data for a general model while leveraging the function fitting capabilities of deep learning. The proposed HSFusionHSFusion loss is also beneficial for algorithms that operate at the pixel-level as it forces spectral consistency compared to the standard L2L2 loss, as seen in Fig. 4. Fig. 4: Generally, for equivalent PSNR values, HSFusion forces better spectral fidelity within each pixel (SAM metric). Each datum represents a hyperparameter search trial. Higher compression ratios could be further achieved by employing more advanced quantization and encoding schemes. For example, quantization-aware training and compact entropy models. The performance of HNeRV and similar methods is highly dependent on the right choice of hyperparameters. General hyperparameters can be found via search strategies on multiple videos, but this defeats the purpose of dataset-free compression when input distributions differ. This can be mitigated by adding hyperparameter optimization strategies, such as Bayesian or gradient-based optimization. Fig. 5: Qualitative comparison of reconstruction capabilities. Channel 16 of first frame from high_truck sequence. HS-HNeRV achieves a higher spatial and spectral quality. 6 Conclusion In this work, we demonstrate the way in which INR can be leveraged for HS video compression. The methods shown demonstrate strong performance compared to PCA combined with JPEG2000 for video encoding, improving reconstruction quality by +4.99+4.99 dB. Future work will include the adoption of bespoke architectures for HS video, the use of hyperparameter optimization for ideal overfitting, and adjusting quantization and encoding procedures to match the latest overfitting RGB codecs. 7 REFERENCES References [1] K. Basterretxea, V. Martínez, J. Echanobe, J. Gutiérrez–Zaballa, and I. Del Campo (2021) HSI-Drive: A Dataset for the Research of Hyperspectral Image Processing Applied to Autonomous Driving Systems. In 2021 IEEE Intelligent Vehicles Symposium (IV), p. 866–873. External Links: Document, Link Cited by: §1, §2. [2] A. Bhargava, A. Sachdeva, K. Sharma, M. H. Alsharif, P. Uthansakul, and M. Uthansakul (2024) Hyperspectral imaging and its applications: A review. 10 (12), p. e33208. External Links: ISSN 2405-8440, Document, Link Cited by: §1. [3] B. Bross, Y. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J. Ohm (2021) Overview of the Versatile Video Coding (VVC) Standard and its Applications. 31 (10), p. 3736–3764. External Links: ISSN 1558-2205, Document, Link Cited by: §2. [4] H. Chen, M. Gwilliam, S. Lim, and A. Shrivastava (2023)HNeRV: A Hybrid Neural Representation for Videos(Website) External Links: Link Cited by: §1, §2, §3.2, §3.2. [5] L. Chen, Y. Zhao, J. C. Chan, and S. G. Kong (2022) Histograms of oriented mosaic gradients for snapshot spectral image description. 183, p. 79–93. External Links: ISSN 0924-2716, Document, Link Cited by: §2. [6] S. Das (2021) Hyperspectral image, video compression using sparse tucker tensor decomposition. 15 (4), p. 964–973. External Links: ISSN 1751-9667, Document, Link Cited by: §1, §2. [7] Q. Du and J. E. Fowler (2007) Hyperspectral Image Compression Using JPEG2000 and Principal Component Analysis. 4 (2), p. 201–205. External Links: ISSN 1558-0571, Document, Link Cited by: §3.1. [8] H. He, Y. Shi, J. Wang, J. Liang, N. Ling, and B. Yin (2026) End-to-end learned video compression: A comprehensive review. 694, p. 133839. External Links: ISSN 0925-2312, Document, Link Cited by: §2. [9] HotTracking2026 (2026) Hyperspectral object tracking challenge 2026. Note: https://kaggle.com/competitions/hyperspectral-object-tracking-challenge-2026Kaggle Cited by: §2, §3.3. [10] Z. Jia, B. Li, J. Li, W. Xie, L. Qi, H. Li, and Y. Lu (2025)Towards Practical Real-Time Neural Video Compression(Website) External Links: Link Cited by: §2. [11] H. M. Kwan, G. Gao, F. Zhang, A. Gower, and D. Bull (2024) NVRC: Neural Video Representation Compression. In Advances in Neural Information Processing Systems, Vol. 37, p. 132440–132462. External Links: Document, Link Cited by: §2, §3.2. [12] T. Leguay, T. Ladune, P. Philippe, and O. Déforges (2024) Cool-chic video: Learned video coding with 800 parameters. In 2024 Data Compression Conference (DCC), p. 23–32. External Links: ISSN 2375-0359, Document, Link Cited by: §2, §3.2. [13] Z. Li, J. Wang, J. Zhang, D. Zhao, G. Fu, J. Wang, and J. Lu (2026) A unified spatial-spectral-temporal network for hyperspectral object tracking. 174, p. 113005. External Links: ISSN 0031-3203, Document, Link Cited by: §3.3. [14] N. Minallah, S. Gul, and M.M. Bokhari (2015) Performance Analysis of H.265/HEVC (High-Efficiency Video Coding) with Reference to Other Codecs. In 2015 13th International Conference on Frontiers of Information Technology (FIT), p. 216–221. External Links: Document, Link Cited by: §2. [15] O. J. Pellicer-Valero, C. Aybar, and G. C. Valls (2025)Video Compression for Spatiotemporal Earth System Data(Website) External Links: 2506.19656, Document, Link Cited by: §2. [16] F. Sippel, J. Seiler, and A. Kaup (2023) Synthetic hyperspectral array video database with applications to cross-spectral reconstruction and hyperspectral video coding. 40 (3), p. 479–491. External Links: ISSN 1520-8532, Document, Link Cited by: §2. [17] S. Takamura and Y. Yashima (2004) Multiband video coding using H.264/AVC, MPEG-4 studio profile and JPEG 2000. In Data Compression Conference, 2004. Proceedings. DCC 2004, p. 567–. External Links: ISSN 1068-0314, Document, Link Cited by: §2. [18] T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra (2003) Overview of the H.264/AVC video coding standard. 13 (7), p. 560–576. External Links: ISSN 1558-2205, Document, Link Cited by: §2. [19] F. Xiong, J. Zhou, and Y. Qian (2020) Material Based Object Tracking in Hyperspectral Videos. 29, p. 3719–3733. External Links: ISSN 1941-0042, Document, Link Cited by: §1, §2.

Instant research discovery