Paper deep dive
Challenges in Hyperspectral Imaging for Autonomous Driving: The HSI-Drive Case
Koldo Basterretxea, Jon GutiĂŠrrez-Zaballa, Javier Echanobe
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/27/2026, 1:37:36 AM
Summary
This paper analyzes the challenges of integrating hyperspectral imaging (HSI) into autonomous driving (AD) systems, focusing on real-time processing constraints and environmental variability. The authors introduce the HSI-Drive v2.1 dataset, featuring improved annotations and a new pseudo-reflectance correction algorithm. They also propose an enhanced U-Net architecture incorporating Efficient Channel Attention (ECA) modules to improve semantic segmentation accuracy while maintaining computational efficiency for embedded platforms.
Entities (5)
Relation Signals (3)
HSI-Drive â appliedto â Autonomous Driving
confidence 100% ¡ In this article, we analyse several techniques explored in the research of HSI-based vision systems with application to AD, using as an example results obtained from experiments using data from the most recent version of the HSI-Drive dataset.
Efficient Channel Attention â integratedinto â U-Net
confidence 98% ¡ To effectively integrate ECA into the previously developed U-Net architecture, two attention blocks were incorporated at each encoder and decoder stage
HSI-Drive â usedby â U-Net
confidence 95% ¡ The U-Net trained with previous versions of HSI-Drive has been enhanced by incorporating attention modules.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The use of hyperspectral imaging (HSI) in autonomous driving (AD), while promising, faces many challenges related to the specifics and requirements of this application domain. On the one hand, non-controlled and variable lighting conditions, the wide depth-of-field ranges, and dynamic scenes with fast-moving objects. On the other hand, the requirements for real-time operation and the limited computational resources of embedded platforms. The combination of these factors determines both the criteria for selecting appropriate HSI technologies and the development of custom vision algorithms that leverage the spectral and spatial information obtained from the sensors. In this article, we analyse several techniques explored in the research of HSI-based vision systems with application to AD, using as an example results obtained from experiments using data from the most recent version of the HSI-Drive dataset.
Tags
Links
- Source: https://arxiv.org/abs/2603.25510v1
- Canonical: https://arxiv.org/abs/2603.25510v1
Trouble viewing inline? Open PDF directly â
Full Text
24,655 characters extracted from source content.
Expand or collapse full text
CHALLENGES IN HYPERSPECTRAL IMAGING FOR AUTONOMOUS DRIVING: THE HSI-DRIVE CASE Koldo Basterretxea 1 , Jon Guti Ě errez-Zaballa 2 , and Javier Echanobe 2 1 University of the Basque Country, Dep. of Electronics Technology 2 University of the Basque Country, Dep. of Electricity and Electronics ABSTRACT The use of hyperspectral imaging (HSI) in autonomous driv- ing (AD), while promising, faces many challenges related to the specifics and requirements of this application domain. On the one hand, non-controlled and variable lighting con- ditions, the wide depth-of-field ranges, and dynamic scenes with fast-moving objects. On the other hand, the require- ments for real-time operation and the limited computational resources of embedded platforms. The combination of these factors determines both the criteria for selecting appropriate HSI technologies and the development of custom vision al- gorithms that leverage the spectral and spatial information obtained from the sensors. In this article, we analyse sev- eral techniques explored in the research of HSI-based vision systems with application to AD, using as an example results obtained from experiments using data from the most recent version of the HSI-Drive dataset. Index Termsâ Hyperspectral imaging, autonomous driving, image segmentation, spectral attention 1. INTRODUCTION Recent advances in hyperspectral imaging (HSI) sensing technologies have enabled the commercialization of snapshot cameras capable of capturing spectral information across dozens or even hundreds of bands at video rates [1]. These developments have sparked significant interest in exploring the potential of HSI in emerging application domains that demand affordable, compact, and portable spectral imaging systems with moderate to low power consumption such as precision agriculture, remote sensing in the new space, med- ical imaging for surgery, food processing, and autonomous navigation [2]. In particular, some researchers have focused their attention on the potential of HSI to overcome some of the limitations of current intelligent vision systems for Au- tonomous Driving (AD), as well as to enhance their capability for comprehensive scene understanding [3], [4]. However, the use of HSI snapshot cameras in AD requires a careful anal- ysis of involved technologies and the customization of both This work was partially supported by the University of the Basque Coun- try (UPV/EHU) under grant GIU21/007. algorithms and processing architectures to meet the specific requirements of this field of application. The challenges associated with applying HSI to AD stem from the combination of factors that affect the quality and accuracy of collected data -constraints imposed by snapshot camera technologies, uncontrolled illumination, the presence of fast moving elements, the need for variable exposure times, etc.- together with the requirements of low-latency operation and the limitations in the computational power of available embedded processing platforms. In this article, we share several lessons learned during the development of HSI-based image segmentation systems for AD using the HSI-Drive dataset. We also describe recent improvements introduced in both the latest public release of this dataset and the deep neu- ral network (DNN) image segmentation models developed through experiments with these data. 2. HSI SNAPSHOT CAMERA TECHNOLOGIES The selection of an HSI camera to experiment with the devel- opment of advanced machine vision systems for AD must be grounded on a set of well established criteria. Ideally, ⢠it must be capable of operating at video-rates (snapshot cameras) under different illumination conditions, ⢠it must be small, compact, and mechanically robust, ⢠it must provide enough spatial resolution for the depth of field required in the application, ⢠it must provide enough spectral resolution to achieve good spectral separability of the classes/materials to be identified, ⢠and it must be based on a scalable sensor technology that makes it competitive for future mass deployment. Regarding the ability of HSI systems to operate at video rates, an additional factor that is often overlooked, but of con- siderable importance, is the processing required to reconstruct spatial and spectral information from the raw data. A thor- ough analysis of commercially available HSI snapshot cam- era technologies is beyond the scope of this work. However, c Š2025 IEEE. Final published version of the article can be found at . arXiv:2603.25510v1 [cs.CV] 26 Mar 2026 it is worth noting that the main players in the small-form fac- tor, high-throughput (>10-20 fps) HSI snapshot camera seg- ment employ either on-chip deposition technology of mosaics of narrow-band interferometric filters (e.g., Imec sensors in- tegrated in Ximea and Photonfocus cameras), or light-field technology based on microlens arrays for multiple projection of spectrally filtered sub-images onto the sensor (as in the case of Cubert). In both cases, the incident light is projected onto a standard CMOS sensor. Alternative technologies, such as coded aperture snapshot spectral imaging (CASSI) combined with scattered spectral sampling, require complex and time- consuming data processing. Consequently, these solutions are not yet capable of meeting the real-time performance require- ments of autonomous driving applications. The HSI-Drive dataset was generated from recordings with a Photonfocus MV2 camera featuring an Imec 25-band Red-NIR sensor with on-chip mosaic filter technology. The selection of this system was motivated by several factors: its relatively low cost, its high throughput combined with accept- able spatial and spectral resolution, and the accessibility to the cube-generation processing code, which could be adapted and optimised for efficient execution on an embedded processing platform. Moreover, spectral filter-on-chip technology offers far greater scalability than competing approaches, enabling mass production at costs comparable to those of traditional CMOS sensors. Nonetheless, there are also some drawbacks to be considered when compared to other technological al- ternatives. These include the presence of spectral leakage and second-order response peaks, the constraints imposed by mosaic patterns on the number of bands, and the need for pixel-level spatial realignment of the mosaic filters. All in all, the design of the HSI-Drive dataset was guided primarily by principles of simplicity and feasibility, rather than by the pur- suit of maximum spectral quality. In other words, our guiding question was: Given currently available and potentially scal- able technologies for vehicle integration at reasonable cost, what level of useful information can be extracted from an HSI snapshot camera for the development of machine vision systems for autonomous driving? 3. SPECTRAL INFORMATION: DATA QUALITY, CONSISTENCY, AND PROCESSING CONSTRAINTS Considering that the approach for the development of the ma- chine vision system under study, i.e. an image segmenta- tion processing pipeline that could be executed on an em- bedded processing platform at video rates, was to be based on a machine learning (ML) model, to what extent the corre- spondence of the acquired spectral data with the real physical spectral reflectance signatures of the materials in a scene is of real importance? If that is the case, in principle, as far as there is data consistency, accuracy does not seem to be of any concern. However, this is not entirely true, as it must be un- derstood that the spectral separability between materials may depend on subtle differences in the spectral reflectance sig- natures. However, here again the approach in the HSI-Drive dataset leans towards computational simplicity as far as the model quality is not compromised. The latest version (v2.1) of the HSI-Drive dataset presents two major differences with respect to the previous v2.0 ver- sion: first, the annotation, while still favouring the preserva- tion of the spectral information, has been carefully reviewed and the total number of labelled pixels has been increased with more than 1,100,000 new pixels (+2,5 %). Secondly, a new function to enhance data consistency has been added to the cube processing pipeline, which performs a pseudo- reflectance correction algorithm using data contained in the recorded frame itself. (a) False RGB (b) HSI-Drive 2.0 labelling (c) HSI-Drive 2.1 labelling Fig. 1: Example of manually labelled ground true images in HSI-Drive versions v2.0 and v2.1 3.1. New data labeling The v2.1 version of the HSI-Drive dataset does not provide more annotated images, but a new, more careful annotation of the images already in version v2.0. The aim of this new la- belling effort has been twofold. Firstly, to increase the amount of labelled pixels for training, especially in the most underrep- resented categories. Secondly, to provide higher quality test images for the evaluation of segmentation models. However, the primary approach to the image labelling of the dataset is not changed, i.e. keeping unlabelled the pixels that a hu- man labeller cannot clearly decide which category they be- long to This usually includes many background pixels and all the edges that delimit different items or surfaces in a scene. 3.2. Reflectance correction and data normalization With the aim of preserving the feasibility of real-time deploy- ments, the cube processing pipeline applied to generate the hyperspectral cubes in the HSI-Drive 2.1 was kept simple as in previous versions. The target application (AD) requires real-time processing of the acquired images and, consider- ing the recording circumstances -outdoor recording, different camera setups, no additional lighting and spectra measure- ments etc., trying more accurate yet more complex cube gen- eration pipelines would become the principal processing bot- tleneck with no noticeable improvements in the final results (accuracy). The applied spectral cube generation process is a re- flectance processing pipeline that comprehends the following steps 1. Image cropping and framing. 2. Bias removal and reflectance correction. 3. Partial demosaicing (with 1/5 spatial resolution loss). 4. Spatial filtering (optional). 5. Translation to centre (band alignment by bilinear inter- polation). Since data normalization techniques (per band normaliza- tion, pixel normalization etc.) have different objectives de- pending on the algorithms to be used to subsequently process the hyperspectral images, and since it is the final step of the processing pipeline, unlike in the v1.x versions of the dataset, we do not provide normalized cubes in v2.x versions. This process, if necessary, is left to the dataset users. Fig. 2: figure Spectra of the maximum values of the averaged reference white tile images for the four different camera configurations used in the dataset. Images were taken by exposing the calibrated tile to direct sunlight in a clear day with the sun at its zenith Reflectance correction is aimed at cancelling the irradi- ance spectrum of the illuminant. In a laboratory, this can be performed by using a calibrated reference white tile to divide the data in the image to be processed by the data in a refer- ence white image acquired under the same illumination con- ditions and with the same camera configuration. In dynamic outdoor conditions, as is the case for the recording of the HSI- Drive dataset scenes, an accurate reflectance correction is not possible at all. However, when generating data for the pre- vious versions of the HSI-Drive dataset, we chose to keep a âpseudo reflectance correctionâ stage by using a common white reference image obtained for each of the four different f-number/AG configurations of the camera. These white ref- erence images were generated by averaging various shots over a spectrally calibrated Spectralon white reflectance tile un- der expected natural maximum illumination conditions, i.e. at midday on a sunny day (Fig. 2). Since no point spectrometer or other light measuring devices were present in the recording setup to compensate for illumination variations, obviously the applied correction does not fully normalize the spectral signa- tures of the different images in the dataset, and thus the data are not entirely consistent. However, this processing still pro- vides some benefits. Firstly, it reduces sensor non-uniformity issues and image vignetting, and secondly, it cancels to some extent the irradiance spectrum of natural light. Applying a posteriori data normalization techniques can reduce this issue to some extent. The data used to train the CNN models for the HSI-Drive 2.0 experiments (see results published in [3]) were obtained after applying a per-pixel nor- malization of the spectral signatures. This technique removes the irradiance offset produced by variations in incident light intensity. The beneficial consequences are that shadow effects Table 1: Frequency of each class in the HSI Drive v2.0 dataset. TotalRoadR.Marks a Veg. b Pain.Met. c SkyConcretePed. d WaterUnpain.Met. e Glass Pixels43,947,50326,690,6191,325,3439,339,224948,8522,511,4962,315,153209,53112,330348,341246,614 %10060.733.0221.252.165.715.270.480.030.790.56 a Road Marks. b Vegetation. c Painted Metal. d Pedestrian. e Unpainted Metal. Table 2: Frequency of each class in the HSI Drive v2.1 dataset. TotalRoadR.Marks a Veg. b Pain.Met. c SkyConcretePed. d WaterUnpain.Met. e Glass Pixels45,055,51226,753,8111,364,9089,799,4751,113,5732,549,5272,485,658231,01910,592467,688279,261 %10059.383.0321.752.475.665.520.510.021.040.62 a Road Marks. b Vegetation. c Painted Metal. d Pedestrian. e Unpainted Metal. are mitigated -since reflectances on the same material surfaces are equalized- and that spectral information is favoured over the general reflectance of surfaces. The negative effects are that the differences in the overall reflectance levels of differ- ent materials are removed, which results in the loss of valu- able information for training AI models. As an improvement to data quality, in the v2.1 version of the dataset we incorporate an additional processing func- tion that estimates the relative level of illumination of the recorded scene by searching for the pixels with the highest albedo at each image. These pixels usually correspond to high-reflectance white surfaces such as road marks, white ve- hicle bodies, etc., although in some cases the algorithm se- lects pixels corresponding to the sky. By comparing the ir- radiance of these pixels with the reference white images, a scaling factor is calculated to correct the reference white im- ages stored in memory. Ideally, if the procedure was perfect, all images in the dataset would be scaled in the [0,1] range. The search for reference pixels for scaling is not straightfor- ward, since it involves rejecting pixels from artificial light sources such as illuminated signs, traffic lights, and front and rear lights of vehicles. The programmed algorithm automat- ically segregates these âsuspiciousâ pixels on the foundation of their spectral signatures, so no human intervention is re- quired and thus, this function can be embedded in the image processing pipeline of the image segmentation processor (see Fig. 3). Artificial light pixels are thus treated as outliers and clipped to 1 at the end of the cube preprocessing sequence. 4. ENHANCED SEGMENTATION MODELS AND EXPERIMENTAL RESULTS 4.1. U-Net with spectral attention modules The U-Net trained with previous versions of HSI-Drive has been enhanced by incorporating attention modules. These mechanisms are inspired by the human visual system, where the brain selectively prioritizes certain regions of the visual field while suppressing less relevant information. In CNNs, a similar principle is implemented by weighting feature maps through an attention function. This enables the network to emphasize discriminative spatial features, spectral features, or a combination of both, ultimately improving its represen- tational capacity. To leverage the spectral richness provided by hyper- spectral images, several attention mechanisms have been investigated, namely Convolutional Block Attention Mod- ule (CBAM) [5], Squeeze-and-Excitation (SE) [6], Efficient Channel Attention (ECA) [7], and Coordinate Attention (CA) [8]. Among these, the best segmentation accuracy in our experiments was achieved with ECA which, additionally, introduces the least computational complexity and memory overhead to the model. Efficient Channel Attention (ECA) operates by adap- tively re-weighting channel-wise features without relying on dimensionality reduction. Unlike SE, which compresses and then expands channel dimensions, ECA applies a local cross- channel interaction through a fast 1D convolution with a kernel size adaptively determined by the channel dimension. This design ensures efficient information exchange between channels while avoiding additional fully connected layers, thus maintaining both accuracy and efficiency. To effectively integrate ECA into the previously devel- oped U-Net architecture, two attention blocks were incorpo- rated at each encoder and decoder stage: one before the first convolutional block and another one before the second convo- lutional block (a detailed diagram of the original U-net model can be consulted in [9]). This placement allows the network to refine its feature representations at multiple depths, ensuring that both low-level and high-level spectral-spatial information are adaptively emphasized during the segmentation process. 4.2. Experimental results Performed testing experiments on the HSI-Drive 2.0 and 2.1 datasets demonstrates the superiority of both the new mod- ified U-Net model with spectral attention modules, and the new scaled reflectance correction processing over the use of non-scaled cubes both with and without pixel normalization. Table 3 summarizes the results obtained in the HSI-Drive 2.0 dataset for a 5-class experiment with the previous U-Net using the pixel normalization technique. All figures correspond to (a) Sensed irradiance values (int12) (b) False RGB image of the 25-band HSI cube and coordinates of the pixel with maximum albedo Fig. 3: Example of the identification of a maximum albedo pixel for the white balance scaling. This image corresponds to a cloudy Autumn morning recording with low lightning. Although the maximum irradiance values are generated by the rear and front lights of the cars (a), the algorithm successfully rejects those pixels and selects a pixel corresponding to the road mark as the highest reflectance pixel in the image (b) mean IoU values over a 5-fold cross-validation experimental setup. Attention modules show over 2% accuracy improve- ment in the weighted IoU index. In Table 4 figures show a comparative study of the use of the new reflectance correc- tion scaling algorithm with respect to the pixel normalization technique on the v2.1 dataset using the attention U-Net as predictor. Here again, the improvement exceeds 2% in ac- curacy. Tables 5 and 5 show the results obtained for six class experiments by combining the attention modules with the new scaling reflectance correction technique. The additional sixth classes, painted metal (vehicle bodywork, road signals etc.) and pedestrians/cyclists respectively, are specially challeng- ing due to high intraclass spectral variability and low spectral interclass separability, as well as to small number of train- ing data (see 2). The results obtained on these two classes are specially noteworthy, with accuracy improvements of 10.22% and 5.09% respectively. Examples of segmented videos using these models can be found at https://ipaccess.ehu.eus/HSI-Drive/ . Table 3: Segmentation results (%) for the 5 class experiment on the HSI-Drive 2.0 dataset. ModelVersionroadroad m.veg.skyâothersâglobalweighted U-NetNo scaling+PN97.5385.9495.0493.0278.5994.6487.52 Att.U-NetNo scaling+PN98.0587.7495.5495.2582.9495.6489.71 Table 4: Segmentation results (%) for the 5 class experiment on the HSI-Drive 2.1 dataset. ModelVersionroadroad m.veg.skyâothersâglobalweighted Att.U-NetNo scaling+PN97.6485.3394.5592.8981.7994.7187.75 Att.U-NetScaling+PN97.8387.2794.6094.1482.5095.0489.16 Att.U-NetScaling98.0489.9794.4692.0583.2695.1790.03 Table 5: Segmentation results (%) for the 6 class experiment (painted metal) ModelVersionroadroad m.veg.p.metalskyâothersâglobalweighted U-Net v2.0No scaling+PN97.3485.2093.8458.6192.3068.6593.0774.45 Att.U-Net v2.1Scaling98.0890.3493.6368.8391.4474.6193.9781.09 Table 6: Segmentation results (%) for the 6 class experiment (pedestrians) ModelVersionroadroad m.veg.ped.skyâothersâglobalweighted U-Net v2.0No scaling+PN97.0481.8593.5661.9489.2374.2093.2667.13 Att.U-Net v2.1Scaling97.6087.9393.6467.0389.3480.2894.1572.23 5. CONCLUDING REMARKS The successful adoption of HSI technology in autonomous driving (AD) will depend on several key factors. Firstly, it will depend on advances in HSI sensor technologies that en- able the production of affordable yet technically precise snap- shot cameras, capable of combining high image throughput with sufficient spectral and spatial resolution to support the development of high-performance machine vision systems for autonomous driving. Secondly, it will depend on research into more capable and robust, yet computationally efficient algorithms that can make the most of the information pro- vided by HSI data. Finally, the combination of improvements achieved in both areas should lead to machine vision sys- tems that demonstrate either their superiority or, at least, their complementarity to increasingly capable and precise systems based on more mature technologies. In this paper, we share some research results obtained using the latest published version of the HSI-Drive dataset. HSI-Drive is a dataset developed by recording real driving scenes with a single snapshot hyperspectral camera featur- ing a 25-band Red-NIR on-chip filter mosaic sensor.In this version, we refined the labelling of ground-truth im- ages, improved data consistency by introducing a customized illuminant intensity estimation algorithm for reflectance cor- rection, and developed enhanced image segmentation models by incorporating spectral attention modules. These additional lightweight attention blocks have been placed at key points in the encoder and decoder branches of the previous backbone U-net architecture to ensure efficient spectral information exchange between channels during inference. The result is a consistent improvement in segmentation accuracy and robust- ness, while preserving processing simplicity for deployment on embedded devices. 6. REFERENCES [1] Michael West, John Grossman, and Chris Galvan, âCom- mercial snapshot spectral imaging: the art of the possi- ble,â 2018. [2] Motoki Yako,âHyperspectral imaging: history and prospects,â Optical Review, Sep 2025. [3] Jon Guti Ě errez-Zaballa,Koldo Basterretxea,Javier Echanobe, M Victoria Mart Ě Äąnez, and Unai Martinez- Corral, âHSI-Drive v2.0: More data for new challenges in scene understanding for autonomous driving,â in 2023 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2023, p. 207â214. [4] Imad Ali Shah, Jiarong Li, Martin Glavin, Edward Jones, Enda Ward, and Brian Deegan, âHyperspectral imaging-based perception in autonomous driving scenar- ios: Benchmarking baseline semantic segmentation mod- els,â in 2024 14th Workshop on Hyperspectral Imag- ing and Signal Processing: Evolution in Remote Sensing (WHISPERS), 2024, p. 1â5. [5] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon, âCBAM: Convolutional Block Attention Module,â in Computer Vision â ECCV. 2018, p. 3â19, Springer. [6] Jie Hu, Li Shen, and Gang Sun, âSqueeze-and-Excitation Networks,â in IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, 2018, p. 7132â7141. [7] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu,âECA-Net: Effi- cient Channel Attention for Deep Convolutional Neural Networks,â in IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), 2020, p. 11531â 11539. [8] Qibin Hou, Daquan Zhou, and Jiashi Feng, âCoordi- nate Attention for Efficient Mobile Network Design,â in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, p. 13708â13717. [9] Jon Guti Ě errez-Zaballa, Koldo Basterretxea, and Javier Echanobe, âEvaluating single event upsets in deep neural networks for semantic segmentation: An embedded sys- tem perspective,â Journal of Systems Architecture, vol. 154, p. 103242, 2024.