In the image fusion field,fusing infrared images(IRIs)and visible images(VIs)excelled is a key area.The differences between IRIs and VIs make it challenging to fuse both types into a high-quality image.Accordingly,eff...In the image fusion field,fusing infrared images(IRIs)and visible images(VIs)excelled is a key area.The differences between IRIs and VIs make it challenging to fuse both types into a high-quality image.Accordingly,efficiently combining the advantages of both images while overcoming their shortcomings is necessary.To handle this challenge,we developed an end-to-end IRI andVI fusionmethod based on frequency decomposition and enhancement.By applying concepts from frequency domain analysis,we used the layering mechanism to better capture the salient thermal targets from the IRIs and the rich textural information from the VIs,respectively,significantly boosting the image fusion quality and effectiveness.In addition,the backbone network combined Restormer Blocks and Dense Blocks;Restormer blocks utilize global attention to extract shallow features.Meanwhile,Dense Blocks ensure the integration between shallow and deep features,thereby avoiding the loss of shallow attributes.Extensive experiments on TNO and MSRS datasets demonstrated that the suggested method achieved state-of-the-art(SOTA)performance in various metrics:Entropy(EN),Mutual Information(MI),Standard Deviation(SD),The Structural Similarity Index Measure(SSIM),Fusion quality(Qabf),MI of the pixel(FMI_(pixel)),and modified Visual Information Fidelity(VIF_(m)).展开更多
Fine spatial and temporal resolution land surface temperature(LST)data are of great importance for various researches and applications.Spatio-temporal fusion provides an important solution to obtain fine spatio-tempor...Fine spatial and temporal resolution land surface temperature(LST)data are of great importance for various researches and applications.Spatio-temporal fusion provides an important solution to obtain fine spatio-temporal resolution LST.For example,100-m,daily LST data can be created by fusing 1-km,daily Moderate Resolution Imaging Spectroradiometer(MODIS)LST with 100-m,16-day Landsat LST data.However,the quality of MODIS LST products has been decreasing noticeably in recent years,which has a great impact on fusion accuracy.To address this issue,this paper proposes to use Visible Infrared Imaging Radiometer Suite(VIIRS)LST to replace MODIS LST in spatio-temporal fusion.Meanwhile,to cope with the data discrepancy caused by the large difference in overpass time between VIIRS LST and Landsat LST,a spatio-temporal fusion method based on the Restormer(RES-STF)is proposed.Specifically,to effectively model the differences between the 2 types of data,RES-STF uses Transformer modules in Restormer,which combines the advantages of convolutional neural networks(CNN)and Transformer to effectively capture both local and global context in images.In addition,the calculation of self-attention is re-designed by concatenating CNN to increase the efficiency of feature extraction.Experimental results on 3 areas validated the effectiveness of RES-STF,which outperforms one non-deep learning-and 3 deep learning-based spatio-temporal fusion methods.Moreover,compared to MODIS LST,VIIRS LST data contain richer spatial texture information,leading to more accurate fusion results,with both RMSE and MAE reduced by about 0.5 K.展开更多
基金funded by Anhui Province University Key Science and Technology Project(2024AH053415)Anhui Province University Major Science and Technology Project(2024AH040229)+3 种基金Talent Research Initiation Fund Project of Tongling University(2024tlxyrc019)Tongling University School-Level Scientific Research Project(2024tlxyptZD07)TheUniversity Synergy Innovation Programof Anhui Province(GXXT-2023-050)Tongling City Science and Technology Major Special Project(Unveiling and Commanding Model)(200401JB004).
文摘In the image fusion field,fusing infrared images(IRIs)and visible images(VIs)excelled is a key area.The differences between IRIs and VIs make it challenging to fuse both types into a high-quality image.Accordingly,efficiently combining the advantages of both images while overcoming their shortcomings is necessary.To handle this challenge,we developed an end-to-end IRI andVI fusionmethod based on frequency decomposition and enhancement.By applying concepts from frequency domain analysis,we used the layering mechanism to better capture the salient thermal targets from the IRIs and the rich textural information from the VIs,respectively,significantly boosting the image fusion quality and effectiveness.In addition,the backbone network combined Restormer Blocks and Dense Blocks;Restormer blocks utilize global attention to extract shallow features.Meanwhile,Dense Blocks ensure the integration between shallow and deep features,thereby avoiding the loss of shallow attributes.Extensive experiments on TNO and MSRS datasets demonstrated that the suggested method achieved state-of-the-art(SOTA)performance in various metrics:Entropy(EN),Mutual Information(MI),Standard Deviation(SD),The Structural Similarity Index Measure(SSIM),Fusion quality(Qabf),MI of the pixel(FMI_(pixel)),and modified Visual Information Fidelity(VIF_(m)).
基金supported by the National Natural Science Foundation of China under grants 42171345 and 42222108.
文摘Fine spatial and temporal resolution land surface temperature(LST)data are of great importance for various researches and applications.Spatio-temporal fusion provides an important solution to obtain fine spatio-temporal resolution LST.For example,100-m,daily LST data can be created by fusing 1-km,daily Moderate Resolution Imaging Spectroradiometer(MODIS)LST with 100-m,16-day Landsat LST data.However,the quality of MODIS LST products has been decreasing noticeably in recent years,which has a great impact on fusion accuracy.To address this issue,this paper proposes to use Visible Infrared Imaging Radiometer Suite(VIIRS)LST to replace MODIS LST in spatio-temporal fusion.Meanwhile,to cope with the data discrepancy caused by the large difference in overpass time between VIIRS LST and Landsat LST,a spatio-temporal fusion method based on the Restormer(RES-STF)is proposed.Specifically,to effectively model the differences between the 2 types of data,RES-STF uses Transformer modules in Restormer,which combines the advantages of convolutional neural networks(CNN)and Transformer to effectively capture both local and global context in images.In addition,the calculation of self-attention is re-designed by concatenating CNN to increase the efficiency of feature extraction.Experimental results on 3 areas validated the effectiveness of RES-STF,which outperforms one non-deep learning-and 3 deep learning-based spatio-temporal fusion methods.Moreover,compared to MODIS LST,VIIRS LST data contain richer spatial texture information,leading to more accurate fusion results,with both RMSE and MAE reduced by about 0.5 K.
