The fusion of infrared and visible images should emphasize the salient targets in the infrared image while preserving the textural details of the visible images.To meet these requirements,an autoencoder-based method f...The fusion of infrared and visible images should emphasize the salient targets in the infrared image while preserving the textural details of the visible images.To meet these requirements,an autoencoder-based method for infrared and visible image fusion is proposed.The encoder designed according to the optimization objective consists of a base encoder and a detail encoder,which is used to extract low-frequency and high-frequency information from the image.This extraction may lead to some information not being captured,so a compensation encoder is proposed to supplement the missing information.Multi-scale decomposition is also employed to extract image features more comprehensively.The decoder combines low-frequency,high-frequency and supplementary information to obtain multi-scale features.Subsequently,the attention strategy and fusion module are introduced to perform multi-scale fusion for image reconstruction.Experimental results on three datasets show that the fused images generated by this network effectively retain salient targets while being more consistent with human visual perception.展开更多
Fourier ptychographic microscopy(FPM)is an innovative computational microscopy approach that enables high-throughput imaging with high resolution,wide field of view,and quantitative phase imaging(QPI)by simultaneously...Fourier ptychographic microscopy(FPM)is an innovative computational microscopy approach that enables high-throughput imaging with high resolution,wide field of view,and quantitative phase imaging(QPI)by simultaneously capturing bright-field and dark-field images.However,effectively utilizing dark-field intensity images,including both normally exposed and overexposed data,which contain valuable high-angle illumination information,remains a complex challenge.Successfully extracting and applying this information could significantly enhance phase reconstruction,benefiting processes such as virtual staining and QPI imaging.To address this,we introduce a multi-exposure image fusion(MEIF)framework that optimizes dark-field information by incorporating it into the FPM preprocessing workflow.MEIF increases the data available for reconstruction without requiring changes to the optical setup.We evaluate the framework using both feature-domain and traditional FPM,demonstrating that it achieves substantial improvements in intensity resolution and phase information for biological samples that exceed the performance of conventional high dynamic range(HDR)methods.This image preprocessing-based information-maximization strategy fully leverages existing datasets and offers promising potential to drive advancements in fields such as microscopy,remote sensing,and crystallography.展开更多
Osteosarcoma is the most common primary bone tumor with high malignancy.It is particularly necessary to achieve rapid and accurate diagnosis in its intraoperative examination and early diagnosis.Accordingly,the multim...Osteosarcoma is the most common primary bone tumor with high malignancy.It is particularly necessary to achieve rapid and accurate diagnosis in its intraoperative examination and early diagnosis.Accordingly,the multimodal microscopic imaging diagnosis system constructed by bright field,spontaneous fluorescence and polarized light microscopic imaging was used to study the pathological mechanism of osteosarcoma from the tissue microenvironment level and achieve rapid and accurate diagnosis.First,the multimodal microscopic images of normal and osteosarcoma tissue slices were collected to characterize the overall morphology of the tissue microenvironment of the samples,the arrangement structure of collagen fibers and the content and distribution of endogenous fluorescent substances.Second,based on the correlation and complementarity of the feature information contained in the three single-mode images,combined with convolutional neural network(CNN)and image fusion methods,a multimodal intelligent diagnosis model was constructed to effectively improve the information utilization and diagnosis accuracy.The accuracy and true positivity of the multimodal diagnostic model were significantly improved to 0.8495 and 0.9412,respectively,compared to those of the single-modal models.Besides,the difference of tissue microenvironments before and after cancerization can be used as a basis for cancer diagnosis,and the information extraction and intelligent diagnosis of osteosarcoma tissue can be achieved by using multimodal microscopic imaging technology combined with deep learning,which significantly promoted the application of tissue microenvironment in pathological examination.This diagnostic system relies on its advantages of simple operation,high efficiency and accuracy and high cost-effectiveness,and has enormous clinical application potential and research significance.展开更多
Images with complementary spectral information can be recorded using image sensors that can identify visible and near-infrared spectrum.The fusion of visible and nearinfrared(NIR)aims to enhance the quality of images ...Images with complementary spectral information can be recorded using image sensors that can identify visible and near-infrared spectrum.The fusion of visible and nearinfrared(NIR)aims to enhance the quality of images acquired by video monitoring systems for the ease of user observation and data processing.Unfortunately,current fusion algorithms produce artefacts and colour distortion since they cannot make use of spectrum properties and are lacking in information complementarity.Therefore,an information complementarity fusion(ICF)model is designed based on physical signals.In order to separate high-frequency noise from important information in distinct frequency layers,the authors first extracted texture-scale and edge-scale layers using a two-scale filter.Second,the difference map between visible and near-infrared was filtered using the extended-DoG filter to produce the initial visible-NIR complementary weight map.Then,to generate a guide map,the near-infrared image with night adjustment was processed as well.The final complementarity weight map was subsequently derived via an arctanI function mapping using the guide map and the initial weight maps.Finally,fusion images were generated with the complementarity weight maps.The experimental results demonstrate that the proposed approach outperforms the state-of-the-art in both avoiding artificial colours as well as effectively utilising information complementarity.展开更多
Infrared and visible light image fusion technology integrates feature information from two different modalities into a fused image to obtain more comprehensive information.However,in low-light scenarios,the illuminati...Infrared and visible light image fusion technology integrates feature information from two different modalities into a fused image to obtain more comprehensive information.However,in low-light scenarios,the illumination degradation of visible light images makes it difficult for existing fusion methods to extract texture detail information from the scene.At this time,relying solely on the target saliency information provided by infrared images is far from sufficient.To address this challenge,this paper proposes a lightweight infrared and visible light image fusion method based on low-light enhancement,named LLE-Fuse.The method is based on the improvement of the MobileOne Block,using the Edge-MobileOne Block embedded with the Sobel operator to perform feature extraction and downsampling on the source images.The intermediate features at different scales obtained are then fused by a cross-modal attention fusion module.In addition,the Contrast Limited Adaptive Histogram Equalization(CLAHE)algorithm is used for image enhancement of both infrared and visible light images,guiding the network model to learn low-light enhancement capabilities through enhancement loss.Upon completion of network training,the Edge-MobileOne Block is optimized into a direct connection structure similar to MobileNetV1 through structural reparameterization,effectively reducing computational resource consumption.Finally,after extensive experimental comparisons,our method achieved improvements of 4.6%,40.5%,156.9%,9.2%,and 98.6%in the evaluation metrics Standard Deviation(SD),Visual Information Fidelity(VIF),Entropy(EN),and Spatial Frequency(SF),respectively,compared to the best results of the compared algorithms,while only being 1.5 ms/it slower in computation speed than the fastest method.展开更多
Medical image fusion technology is crucial for improving the detection accuracy and treatment efficiency of diseases,but existing fusion methods have problems such as blurred texture details,low contrast,and inability...Medical image fusion technology is crucial for improving the detection accuracy and treatment efficiency of diseases,but existing fusion methods have problems such as blurred texture details,low contrast,and inability to fully extract fused image information.Therefore,a multimodal medical image fusion method based on mask optimization and parallel attention mechanism was proposed to address the aforementioned issues.Firstly,it converted the entire image into a binary mask,and constructed a contour feature map to maximize the contour feature information of the image and a triple path network for image texture detail feature extraction and optimization.Secondly,a contrast enhancement module and a detail preservation module were proposed to enhance the overall brightness and texture details of the image.Afterwards,a parallel attention mechanism was constructed using channel features and spatial feature changes to fuse images and enhance the salient information of the fused images.Finally,a decoupling network composed of residual networks was set up to optimize the information between the fused image and the source image so as to reduce information loss in the fused image.Compared with nine high-level methods proposed in recent years,the seven objective evaluation indicators of our method have improved by 6%−31%,indicating that this method can obtain fusion results with clearer texture details,higher contrast,and smaller pixel differences between the fused image and the source image.It is superior to other comparison algorithms in both subjective and objective indicators.展开更多
This paper presents an enhanced version of the correlation-driven dual-branch feature decomposition framework(CDDFuse)for fusing low-and high-exposure images captured by the G400BSI sensor.We introduce a novel neural ...This paper presents an enhanced version of the correlation-driven dual-branch feature decomposition framework(CDDFuse)for fusing low-and high-exposure images captured by the G400BSI sensor.We introduce a novel neural long-term memory(NLM)module into the CDDFuse architecture to improve feature extraction by leveraging persistent global feature representations across image sequences.The proposed method effectively preserves dynamic range and structural details,and is evaluated using a new metric,the ATEF dynamic range preservation index(ATEF-DRPI).Experimental results on a G400BSI dataset demonstrate superior fusion quality,with ATEF-DRPI scores of 0.90,a 12.5%improvement over that of the baseline CDDFuse(0.80),indicating better detail retention in bright and dark regions.This work advances image fusion techniques for extreme lighting conditions,offering improved performance for downstream vision tasks.展开更多
Multimodal image fusion plays an important role in image analysis and applications.Multimodal medical image fusion helps to combine contrast features from two or more input imaging modalities to represent fused inform...Multimodal image fusion plays an important role in image analysis and applications.Multimodal medical image fusion helps to combine contrast features from two or more input imaging modalities to represent fused information in a single image.One of the critical clinical applications of medical image fusion is to fuse anatomical and functional modalities for rapid diagnosis of malignant tissues.This paper proposes a multimodal medical image fusion network(MMIF-Net)based on multiscale hybrid attention.The method first decomposes the original image to obtain the low-rank and significant parts.Then,to utilize the features at different scales,we add amultiscalemechanism that uses three filters of different sizes to extract the features in the encoded network.Also,a hybrid attention module is introduced to obtain more image details.Finally,the fused images are reconstructed by decoding the network.We conducted experiments with clinical images from brain computed tomography/magnetic resonance.The experimental results show that the multimodal medical image fusion network method based on multiscale hybrid attention works better than other advanced fusion methods.展开更多
Visible-infrared object detection leverages the day-night stable object perception capability of infrared images to enhance detection robustness in low-light environments by fusing the complementary information of vis...Visible-infrared object detection leverages the day-night stable object perception capability of infrared images to enhance detection robustness in low-light environments by fusing the complementary information of visible and infrared images.However,the inherent differences in the imaging mechanisms of visible and infrared modalities make effective cross-modal fusion challenging.Furthermore,constrained by the physical characteristics of sensors and thermal diffusion effects,infrared images generally suffer from blurred object contours and missing details,making it difficult to extract object features effectively.To address these issues,we propose an infrared-visible image fusion network that realizesmultimodal information fusion of infrared and visible images through a carefully designedmultiscale fusion strategy.First,we design an adaptive gray-radiance enhancement(AGRE)module to strengthen the detail representation in infrared images,improving their usability in complex lighting scenarios.Next,we introduce a channelspatial feature interaction(CSFI)module,which achieves efficient complementarity between the RGB and infrared(IR)modalities via dynamic channel switching and a spatial attention mechanism.Finally,we propose a multi-scale enhanced cross-attention fusion(MSECA)module,which optimizes the fusion ofmulti-level features through dynamic convolution and gating mechanisms and captures long-range complementary relationships of cross-modal features on a global scale,thereby enhancing the expressiveness of the fused features.Experiments on the KAIST,M3FD,and FLIR datasets demonstrate that our method delivers outstanding performance in daytime and nighttime scenarios.On the KAIST dataset,the miss rate drops to 5.99%,and further to 4.26% in night scenes.On the FLIR and M3FD datasets,it achieves AP50 scores of 79.4% and 88.9%,respectively.展开更多
The goal of infrared and visible image fusion(IVIF)is to integrate the unique advantages of both modalities to achieve a more comprehensive understanding of a scene.However,existing methods struggle to effectively han...The goal of infrared and visible image fusion(IVIF)is to integrate the unique advantages of both modalities to achieve a more comprehensive understanding of a scene.However,existing methods struggle to effectively handle modal disparities,resulting in visual degradation of the details and prominent targets of the fused images.To address these challenges,we introduce Prompt Fusion,a prompt-based approach that harmoniously combines multi-modality images under the guidance of semantic prompts.Firstly,to better characterize the features of different modalities,a contourlet autoencoder is designed to separate and extract the high-/low-frequency components of different modalities,thereby improving the extraction of fine details and textures.We also introduce a prompt learning mechanism using positive and negative prompts,leveraging Vision-Language Models to improve the fusion model's understanding and identification of targets in multi-modality images,leading to improved performance in downstream tasks.Furthermore,we employ bi-level asymptotic convergence optimization.This approach simplifies the intricate non-singleton non-convex bi-level problem into a series of convergent and differentiable single optimization problems that can be effectively resolved through gradient descent.Our approach advances the state-of-the-art,delivering superior fusion quality and boosting the performance of related downstream tasks.Project page:https://github.com/hey-it-s-me/PromptFusion.展开更多
Infrared and visible image fusion technology integrates the thermal radiation information of infrared images with the texture details of visible images to generate more informative fused images.However,existing method...Infrared and visible image fusion technology integrates the thermal radiation information of infrared images with the texture details of visible images to generate more informative fused images.However,existing methods often fail to distinguish salient objects from background regions,leading to detail suppression in salient regions due to global fusion strategies.This study presents a mask-guided latent low-rank representation fusion method to address this issue.First,the GrabCut algorithm is employed to extract a saliency mask,distinguishing salient regions from background regions.Then,latent low-rank representation(LatLRR)is applied to extract deep image features,enhancing key information extraction.In the fusion stage,a weighted fusion strategy strengthens infrared thermal information and visible texture details in salient regions,while an average fusion strategy improves background smoothness and stability.Experimental results on the TNO dataset demonstrate that the proposed method achieves superior performance in SPI,MI,Qabf,PSNR,and EN metrics,effectively preserving salient target details while maintaining balanced background information.Compared to state-of-the-art fusion methods,our approach achieves more stable and visually consistent fusion results.The fusion code is available on GitHub at:https://github.com/joyzhen1/Image(accessed on 15 January 2025).展开更多
Aim To fuse the fluorescence image and transmission image of a cell into a single image containing more information than any of the individual image. Methods Image fusion technology was applied to biological cell imag...Aim To fuse the fluorescence image and transmission image of a cell into a single image containing more information than any of the individual image. Methods Image fusion technology was applied to biological cell imaging processing. It could match the images and improve the confidence and spatial resolution of the images. Using two algorithms, double thresholds algorithm and denoising algorithm based on wavelet transform,the fluorescence image and transmission image of a Cell were merged into a composite image. Results and Conclusion The position of fluorescence and the structure of cell can be displyed in the composite image. The signal-to-noise ratio of the exultant image is improved to a large extent. The algorithms are not only useful to investigate the fluorescence and transmission images, but also suitable to observing two or more fluoascent label proes in a single cell.展开更多
Infrared and visible light images can be obtained simultaneously by building fluorescence imaging system,which includes fluorescence excitation,images acquisition,mechanical part,image transmission and processing sect...Infrared and visible light images can be obtained simultaneously by building fluorescence imaging system,which includes fluorescence excitation,images acquisition,mechanical part,image transmission and processing section.This system studied the 2charge-coupled device(CCD)camera(AD-080CL)of the JAI company.Fusion algorithm of visible light and near infrared images was designed for the fluorescence imaging system with wavelet transform image fusion algorithm.In order to enhance the fluorescent moiety of the fusion image,the luminance value of the green component of the color image was changed.And using microsoft foundation classes(MFC)application architecture,the supporting software system was bulit in VS2010 environment.展开更多
Objective： The arrival of precision medicine plan brings new opportunities and challenges for patients undergoing precision diagnosis and treatment of malignant tumors. With the development of medical imaging, inform...Objective： The arrival of precision medicine plan brings new opportunities and challenges for patients undergoing precision diagnosis and treatment of malignant tumors. With the development of medical imaging, information on different modality imaging can be integrated and comprehensively analyzed by imaging fusion system. This review aimed to update the application of multimodality imaging fusion technology in the precise diagnosis and treatment of malignant tumors under the precision medicine plan. We introduced several multimodality imaging fusion technologies and their application to the diagnosis and treatment of malignant tumors in clinical practice. Date Sources： The data cited in this review were obtained mainly from the PubMed database from 1996 to 2016, using the keywords of ＂precision medicine＂, ＂fusion imaging＂, ＂multimodality＂, and ＂tumor diagnosis and treatment＂. Study Selection： Original articles, clinical practice, reviews, and other relevant literatures published in English were reviewed. Papers focusing on precision medicine, fusion imaging, multimodality, and tumor diagnosis and treatment were selected. Duplicated papers were excluded. Results： Multimodality imaging fusion technology plays an important role in tumor diagnosis and treatment under the precision medicine plan, such as accurate location, qualitative diagnosis, tumor staging, treatment plan design, and real-time intraoperative monitoring. Multimodality imaging fusion systems could provide more imaging information of tumors from different dimensions and angles, thereby offing strong technical support for the implementation of precision oncology. Conclusion： Under the precision medicine plan, personalized treatment of tumors is a distinct possibility. We believe that multimodality imaging fusion technology will find an increasingly wide application in clinical practice.展开更多
Image fusion is a key technology in the field of digital image processing.In the present study,an effect-based pseudo color fusion model of infrared and visible images based on the rattlesnake vision imaging system(th...Image fusion is a key technology in the field of digital image processing.In the present study,an effect-based pseudo color fusion model of infrared and visible images based on the rattlesnake vision imaging system(the rattlesnake bimodal cell fusion mechanism and the visual receptive field model)is proposed.The innovation point of the proposed model lies in the following three features:first,the introduction of a simple mathematical model of the visual receptive field reduce computational complexity;second,the enhanced image is obtained by extracting the common information and unique information of source images,which improves fusion image quality;and third,the Waxman typical fusion structure is improved for the pseudo color image fusion model.The performance of the image fusion model is verified through comparative experiments.In the subjective visual evaluation,we find that the color of the fusion image obtained through the proposed model is natural and can highlight the target and scene details.In the objective quantitative evaluation,we observe that the best values on the four indicators,namely standard deviation,average gradient,entropy,and spatial frequency,accounts for 90%,100%,90%,and 100%,respectively,indicating that the fusion image exhibits superior contrast,image clarity,information content,and overall activity.Experimental results reveal that the performance of the proposed model is superior to that of other models and thus verified the validity and reliability of the model.展开更多
Objective： The aim of our study was to compare the value of computed tomography （CT） and 99mTc-methylene- diphosphonate （MDP） SPECT （single photon emission computed tomography）/CT fusion imaging in determining ...Objective： The aim of our study was to compare the value of computed tomography （CT） and 99mTc-methylene- diphosphonate （MDP） SPECT （single photon emission computed tomography）/CT fusion imaging in determining the extent of mandibular invasion by malignant tumor of the oral cavity. Methods： This study had local ethical committee approval, and all patients gave written informed consent. Fifty-three patients were revealed mandibular invasion by malignant tumor of the oral cavity underwent CT and SPECT/CT. The patients were divided into two groups： group A （invasion-periphery-type） and group B （invasion-center- type）. Two radiologists assessed the CT images and two nuclear medicine physicians separately assessed the $PECT/CT images in consensus and without knowledge of the results of other imaging tests. The extent of bone involvement suggested with an imaging modality was compared with pathological findings in the surgical specimen. Results： With pathological findings as the standard of reference, Group A： The extent of mandibular invasion by malignant tumor under- went SPECT/CT was 1.02 _＋ 0.20 cm larger than that underwent pathological findings. And the extent of mandibular invasion underwent CT was 1.42 ＋ 0.35 cm smaller than that underwent pathological examination. There were significant difference among the three methods （P 〈 0.01）. Group B： The extent of mandibular invasion by malignant tumor underwent SPECT/CT was 1.3 ＋ 0.39 cm larger than that underwent pathological examination. The extent of mandibular invasion underwent CT was 2.55 ＋ 1.44 cm smaller than that underwent pathological findings. There were significant difference among the three methods （P 〈 0.01）. The extent of mandibular invasion underwent SPECT/CT was the extent which surgeon must excise to get clear margins. Conclusion： SPECT/CT fusion imaging has significant clinical value in determining the extent of mandibular inva- sion by malignant tumor of oral cavity.展开更多
A novel image fusion network framework with an autonomous encoder and decoder is suggested to increase thevisual impression of fused images by improving the quality of infrared and visible light picture fusion. The ne...A novel image fusion network framework with an autonomous encoder and decoder is suggested to increase thevisual impression of fused images by improving the quality of infrared and visible light picture fusion. The networkcomprises an encoder module, fusion layer, decoder module, and edge improvementmodule. The encoder moduleutilizes an enhanced Inception module for shallow feature extraction, then combines Res2Net and Transformerto achieve deep-level co-extraction of local and global features from the original picture. An edge enhancementmodule (EEM) is created to extract significant edge features. A modal maximum difference fusion strategy isintroduced to enhance the adaptive representation of information in various regions of the source image, therebyenhancing the contrast of the fused image. The encoder and the EEM module extract features, which are thencombined in the fusion layer to create a fused picture using the decoder. Three datasets were chosen to test thealgorithmproposed in this paper. The results of the experiments demonstrate that the network effectively preservesbackground and detail information in both infrared and visible images, yielding superior outcomes in subjectiveand objective evaluations.展开更多
To address the issues of incomplete information,blurred details,loss of details,and insufficient contrast in infrared and visible image fusion,an image fusion algorithm based on a convolutional autoencoder is proposed...To address the issues of incomplete information,blurred details,loss of details,and insufficient contrast in infrared and visible image fusion,an image fusion algorithm based on a convolutional autoencoder is proposed.The region attention module is meant to extract the background feature map based on the distinct properties of the background feature map and the detail feature map.A multi-scale convolution attention module is suggested to enhance the communication of feature information.At the same time,the feature transformation module is introduced to learn more robust feature representations,aiming to preserve the integrity of image information.This study uses three available datasets from TNO,FLIR,and NIR to perform thorough quantitative and qualitative trials with five additional algorithms.The methods are assessed based on four indicators:information entropy(EN),standard deviation(SD),spatial frequency(SF),and average gradient(AG).Object detection experiments were done on the M3FD dataset to further verify the algorithm’s performance in comparison with five other algorithms.The algorithm’s accuracy was evaluated using the mean average precision at a threshold of 0.5(mAP@0.5)index.Comprehensive experimental findings show that CAEFusion performs well in subjective visual and objective evaluation criteria and has promising potential in downstream object detection tasks.展开更多
Objective： The aim of the study was to evaluate the clinical value of ^99mTc-methylene diphosphonic acid （MDP） SPECT/CT fusion imaging and CT scanning in diagnosis of infiltrated mandible by gingival carcinoma. Met...Objective： The aim of the study was to evaluate the clinical value of ^99mTc-methylene diphosphonic acid （MDP） SPECT/CT fusion imaging and CT scanning in diagnosis of infiltrated mandible by gingival carcinoma. Methods： 18 cases of gingival carcinoma were processed infiltrated mandible by ^99mTc-MDP SPECT/CT fusion image and CT, and their scanning results compared with pathology findings. Results： Eleven of 13 cases with well-differentiated squamous cell carcinoma showed positive images, one of 11 cases was false positive images by pathology findings, and 10 cases were exhibited infiltrated mandibles; 5 cases with moderately differentiated and poorly differentiated squamous call carcinoma showed positive images, pathology showed carcinoma call had infiltrated cavum ossis of mandible. Five of 18 cases were positive images by CT. Conclusion： ^99mTc-MDP SPECT/CT fusion imaging is a useful method in diagnosis of infiltrated mandible by gingival carcinoma.展开更多
Multimodal medical image fusion can help physicians provide more accurate treatment plans for patients, as unimodal images provide limited valid information. To address the insufficient ability of traditional medical ...Multimodal medical image fusion can help physicians provide more accurate treatment plans for patients, as unimodal images provide limited valid information. To address the insufficient ability of traditional medical image fusion solutions to protect image details and significant information, a new multimodality medical image fusion method(NSST-PAPCNNLatLRR) is proposed in this paper. Firstly, the high and low-frequency sub-band coefficients are obtained by decomposing the source image using NSST. Then, the latent low-rank representation algorithm is used to process the low-frequency sub-band coefficients;An improved PAPCNN algorithm is also proposed for the fusion of high-frequency sub-band coefficients. The improved PAPCNN model was based on the automatic setting of the parameters, and the optimal method was configured for the time decay factor αe. The experimental results show that, in comparison with the five mainstream fusion algorithms, the new algorithm has significantly improved the visual effect over the comparison algorithm,enhanced the ability to characterize important information in images, and further improved the ability to protect the detailed information;the new algorithm has achieved at least four firsts in six objective indexes.展开更多
基金Supported by the Henan Province Key Research and Development Project(231111211300)the Central Government of Henan Province Guides Local Science and Technology Development Funds(Z20231811005)+2 种基金Henan Province Key Research and Development Project(231111110100)Henan Provincial Outstanding Foreign Scientist Studio(GZS2024006)Henan Provincial Joint Fund for Scientific and Technological Research and Development Plan(Application and Overcoming Technical Barriers)(242103810028)。
文摘The fusion of infrared and visible images should emphasize the salient targets in the infrared image while preserving the textural details of the visible images.To meet these requirements,an autoencoder-based method for infrared and visible image fusion is proposed.The encoder designed according to the optimization objective consists of a base encoder and a detail encoder,which is used to extract low-frequency and high-frequency information from the image.This extraction may lead to some information not being captured,so a compensation encoder is proposed to supplement the missing information.Multi-scale decomposition is also employed to extract image features more comprehensively.The decoder combines low-frequency,high-frequency and supplementary information to obtain multi-scale features.Subsequently,the attention strategy and fusion module are introduced to perform multi-scale fusion for image reconstruction.Experimental results on three datasets show that the fused images generated by this network effectively retain salient targets while being more consistent with human visual perception.
基金supported by the National Natural Science Foundation of China(Grant No.12104500)the Key Research and Development Projects of Shaanxi Province of China(Grant No.2023-YBSF-263).
文摘Fourier ptychographic microscopy(FPM)is an innovative computational microscopy approach that enables high-throughput imaging with high resolution,wide field of view,and quantitative phase imaging(QPI)by simultaneously capturing bright-field and dark-field images.However,effectively utilizing dark-field intensity images,including both normally exposed and overexposed data,which contain valuable high-angle illumination information,remains a complex challenge.Successfully extracting and applying this information could significantly enhance phase reconstruction,benefiting processes such as virtual staining and QPI imaging.To address this,we introduce a multi-exposure image fusion(MEIF)framework that optimizes dark-field information by incorporating it into the FPM preprocessing workflow.MEIF increases the data available for reconstruction without requiring changes to the optical setup.We evaluate the framework using both feature-domain and traditional FPM,demonstrating that it achieves substantial improvements in intensity resolution and phase information for biological samples that exceed the performance of conventional high dynamic range(HDR)methods.This image preprocessing-based information-maximization strategy fully leverages existing datasets and offers promising potential to drive advancements in fields such as microscopy,remote sensing,and crystallography.
基金the National Natural Science Foundation of China(62375127,82272664)Hunan Provincial Natural Science Foundation of China(2022JJ30843)+5 种基金the Science and Technology Development Fund Guided by Central Govern-ment(2021Szvup169)the Scientic Research Program of Hunan Provincial Health Commission(B202304077077)the Fundamental Research Funds for the Central Universities(NS2022035)Prospective Layout Special Fund of Nanjing University of Aero-nautics and Astronautics(ILA-22022)Graduate Research and Innovation Program of Nanjing University of Aeronautics and Astronautics(xcxjh20220328)Experimental Technology Research and Development Project of NUAA(No.SYJS202303Z)for the grant。
文摘Osteosarcoma is the most common primary bone tumor with high malignancy.It is particularly necessary to achieve rapid and accurate diagnosis in its intraoperative examination and early diagnosis.Accordingly,the multimodal microscopic imaging diagnosis system constructed by bright field,spontaneous fluorescence and polarized light microscopic imaging was used to study the pathological mechanism of osteosarcoma from the tissue microenvironment level and achieve rapid and accurate diagnosis.First,the multimodal microscopic images of normal and osteosarcoma tissue slices were collected to characterize the overall morphology of the tissue microenvironment of the samples,the arrangement structure of collagen fibers and the content and distribution of endogenous fluorescent substances.Second,based on the correlation and complementarity of the feature information contained in the three single-mode images,combined with convolutional neural network(CNN)and image fusion methods,a multimodal intelligent diagnosis model was constructed to effectively improve the information utilization and diagnosis accuracy.The accuracy and true positivity of the multimodal diagnostic model were significantly improved to 0.8495 and 0.9412,respectively,compared to those of the single-modal models.Besides,the difference of tissue microenvironments before and after cancerization can be used as a basis for cancer diagnosis,and the information extraction and intelligent diagnosis of osteosarcoma tissue can be achieved by using multimodal microscopic imaging technology combined with deep learning,which significantly promoted the application of tissue microenvironment in pathological examination.This diagnostic system relies on its advantages of simple operation,high efficiency and accuracy and high cost-effectiveness,and has enormous clinical application potential and research significance.
基金supports in part by the Natural Science Foundation of China(NSFC)under contract No.62171253the Young Elite Scientists Sponsorship Program by CAST under program No.2022QNRC001,as well as the Fundamental Research Funds for the Central Universities.
文摘Images with complementary spectral information can be recorded using image sensors that can identify visible and near-infrared spectrum.The fusion of visible and nearinfrared(NIR)aims to enhance the quality of images acquired by video monitoring systems for the ease of user observation and data processing.Unfortunately,current fusion algorithms produce artefacts and colour distortion since they cannot make use of spectrum properties and are lacking in information complementarity.Therefore,an information complementarity fusion(ICF)model is designed based on physical signals.In order to separate high-frequency noise from important information in distinct frequency layers,the authors first extracted texture-scale and edge-scale layers using a two-scale filter.Second,the difference map between visible and near-infrared was filtered using the extended-DoG filter to produce the initial visible-NIR complementary weight map.Then,to generate a guide map,the near-infrared image with night adjustment was processed as well.The final complementarity weight map was subsequently derived via an arctanI function mapping using the guide map and the initial weight maps.Finally,fusion images were generated with the complementarity weight maps.The experimental results demonstrate that the proposed approach outperforms the state-of-the-art in both avoiding artificial colours as well as effectively utilising information complementarity.
基金This researchwas Sponsored by Xinjiang Uygur Autonomous Region Tianshan Talent Programme Project(2023TCLJ02)Natural Science Foundation of Xinjiang Uygur Autonomous Region(2022D01C349).
文摘Infrared and visible light image fusion technology integrates feature information from two different modalities into a fused image to obtain more comprehensive information.However,in low-light scenarios,the illumination degradation of visible light images makes it difficult for existing fusion methods to extract texture detail information from the scene.At this time,relying solely on the target saliency information provided by infrared images is far from sufficient.To address this challenge,this paper proposes a lightweight infrared and visible light image fusion method based on low-light enhancement,named LLE-Fuse.The method is based on the improvement of the MobileOne Block,using the Edge-MobileOne Block embedded with the Sobel operator to perform feature extraction and downsampling on the source images.The intermediate features at different scales obtained are then fused by a cross-modal attention fusion module.In addition,the Contrast Limited Adaptive Histogram Equalization(CLAHE)algorithm is used for image enhancement of both infrared and visible light images,guiding the network model to learn low-light enhancement capabilities through enhancement loss.Upon completion of network training,the Edge-MobileOne Block is optimized into a direct connection structure similar to MobileNetV1 through structural reparameterization,effectively reducing computational resource consumption.Finally,after extensive experimental comparisons,our method achieved improvements of 4.6%,40.5%,156.9%,9.2%,and 98.6%in the evaluation metrics Standard Deviation(SD),Visual Information Fidelity(VIF),Entropy(EN),and Spatial Frequency(SF),respectively,compared to the best results of the compared algorithms,while only being 1.5 ms/it slower in computation speed than the fastest method.
基金supported by Gansu Natural Science Foundation Programme(No.24JRRA231)National Natural Science Foundation of China(No.62061023)Gansu Provincial Education,Science and Technology Innovation and Industry(No.2021CYZC-04)。
文摘Medical image fusion technology is crucial for improving the detection accuracy and treatment efficiency of diseases,but existing fusion methods have problems such as blurred texture details,low contrast,and inability to fully extract fused image information.Therefore,a multimodal medical image fusion method based on mask optimization and parallel attention mechanism was proposed to address the aforementioned issues.Firstly,it converted the entire image into a binary mask,and constructed a contour feature map to maximize the contour feature information of the image and a triple path network for image texture detail feature extraction and optimization.Secondly,a contrast enhancement module and a detail preservation module were proposed to enhance the overall brightness and texture details of the image.Afterwards,a parallel attention mechanism was constructed using channel features and spatial feature changes to fuse images and enhance the salient information of the fused images.Finally,a decoupling network composed of residual networks was set up to optimize the information between the fused image and the source image so as to reduce information loss in the fused image.Compared with nine high-level methods proposed in recent years,the seven objective evaluation indicators of our method have improved by 6%−31%,indicating that this method can obtain fusion results with clearer texture details,higher contrast,and smaller pixel differences between the fused image and the source image.It is superior to other comparison algorithms in both subjective and objective indicators.
文摘This paper presents an enhanced version of the correlation-driven dual-branch feature decomposition framework(CDDFuse)for fusing low-and high-exposure images captured by the G400BSI sensor.We introduce a novel neural long-term memory(NLM)module into the CDDFuse architecture to improve feature extraction by leveraging persistent global feature representations across image sequences.The proposed method effectively preserves dynamic range and structural details,and is evaluated using a new metric,the ATEF dynamic range preservation index(ATEF-DRPI).Experimental results on a G400BSI dataset demonstrate superior fusion quality,with ATEF-DRPI scores of 0.90,a 12.5%improvement over that of the baseline CDDFuse(0.80),indicating better detail retention in bright and dark regions.This work advances image fusion techniques for extreme lighting conditions,offering improved performance for downstream vision tasks.
基金supported by Qingdao Huanghai University School-Level ScientificResearch Project(2023KJ14)Undergraduate Teaching Reform Research Project of Shandong Provincial Department of Education(M2022328)+1 种基金National Natural Science Foundation of China under Grant(42472324)Qingdao Postdoctoral Foundation under Grant(QDBSH202402049).
文摘Multimodal image fusion plays an important role in image analysis and applications.Multimodal medical image fusion helps to combine contrast features from two or more input imaging modalities to represent fused information in a single image.One of the critical clinical applications of medical image fusion is to fuse anatomical and functional modalities for rapid diagnosis of malignant tissues.This paper proposes a multimodal medical image fusion network(MMIF-Net)based on multiscale hybrid attention.The method first decomposes the original image to obtain the low-rank and significant parts.Then,to utilize the features at different scales,we add amultiscalemechanism that uses three filters of different sizes to extract the features in the encoded network.Also,a hybrid attention module is introduced to obtain more image details.Finally,the fused images are reconstructed by decoding the network.We conducted experiments with clinical images from brain computed tomography/magnetic resonance.The experimental results show that the multimodal medical image fusion network method based on multiscale hybrid attention works better than other advanced fusion methods.
基金supported by the National Natural Science Foundation of China(Grant No.62302086)the Natural Science Foundation of Liaoning Province(Grant No.2023-MSBA-070)the Fundamental Research Funds for the Central Universities(Grant No.N2317005).
文摘Visible-infrared object detection leverages the day-night stable object perception capability of infrared images to enhance detection robustness in low-light environments by fusing the complementary information of visible and infrared images.However,the inherent differences in the imaging mechanisms of visible and infrared modalities make effective cross-modal fusion challenging.Furthermore,constrained by the physical characteristics of sensors and thermal diffusion effects,infrared images generally suffer from blurred object contours and missing details,making it difficult to extract object features effectively.To address these issues,we propose an infrared-visible image fusion network that realizesmultimodal information fusion of infrared and visible images through a carefully designedmultiscale fusion strategy.First,we design an adaptive gray-radiance enhancement(AGRE)module to strengthen the detail representation in infrared images,improving their usability in complex lighting scenarios.Next,we introduce a channelspatial feature interaction(CSFI)module,which achieves efficient complementarity between the RGB and infrared(IR)modalities via dynamic channel switching and a spatial attention mechanism.Finally,we propose a multi-scale enhanced cross-attention fusion(MSECA)module,which optimizes the fusion ofmulti-level features through dynamic convolution and gating mechanisms and captures long-range complementary relationships of cross-modal features on a global scale,thereby enhancing the expressiveness of the fused features.Experiments on the KAIST,M3FD,and FLIR datasets demonstrate that our method delivers outstanding performance in daytime and nighttime scenarios.On the KAIST dataset,the miss rate drops to 5.99%,and further to 4.26% in night scenes.On the FLIR and M3FD datasets,it achieves AP50 scores of 79.4% and 88.9%,respectively.
基金partially supported by China Postdoctoral Science Foundation(2023M730741)the National Natural Science Foundation of China(U22B2052,52102432,52202452,62372080,62302078)
文摘The goal of infrared and visible image fusion(IVIF)is to integrate the unique advantages of both modalities to achieve a more comprehensive understanding of a scene.However,existing methods struggle to effectively handle modal disparities,resulting in visual degradation of the details and prominent targets of the fused images.To address these challenges,we introduce Prompt Fusion,a prompt-based approach that harmoniously combines multi-modality images under the guidance of semantic prompts.Firstly,to better characterize the features of different modalities,a contourlet autoencoder is designed to separate and extract the high-/low-frequency components of different modalities,thereby improving the extraction of fine details and textures.We also introduce a prompt learning mechanism using positive and negative prompts,leveraging Vision-Language Models to improve the fusion model's understanding and identification of targets in multi-modality images,leading to improved performance in downstream tasks.Furthermore,we employ bi-level asymptotic convergence optimization.This approach simplifies the intricate non-singleton non-convex bi-level problem into a series of convergent and differentiable single optimization problems that can be effectively resolved through gradient descent.Our approach advances the state-of-the-art,delivering superior fusion quality and boosting the performance of related downstream tasks.Project page:https://github.com/hey-it-s-me/PromptFusion.
基金supported by Universiti Teknologi MARA through UiTM MyRA Research Grant,600-RMC 5/3/GPM(053/2022).
文摘Infrared and visible image fusion technology integrates the thermal radiation information of infrared images with the texture details of visible images to generate more informative fused images.However,existing methods often fail to distinguish salient objects from background regions,leading to detail suppression in salient regions due to global fusion strategies.This study presents a mask-guided latent low-rank representation fusion method to address this issue.First,the GrabCut algorithm is employed to extract a saliency mask,distinguishing salient regions from background regions.Then,latent low-rank representation(LatLRR)is applied to extract deep image features,enhancing key information extraction.In the fusion stage,a weighted fusion strategy strengthens infrared thermal information and visible texture details in salient regions,while an average fusion strategy improves background smoothness and stability.Experimental results on the TNO dataset demonstrate that the proposed method achieves superior performance in SPI,MI,Qabf,PSNR,and EN metrics,effectively preserving salient target details while maintaining balanced background information.Compared to state-of-the-art fusion methods,our approach achieves more stable and visually consistent fusion results.The fusion code is available on GitHub at:https://github.com/joyzhen1/Image(accessed on 15 January 2025).
文摘Aim To fuse the fluorescence image and transmission image of a cell into a single image containing more information than any of the individual image. Methods Image fusion technology was applied to biological cell imaging processing. It could match the images and improve the confidence and spatial resolution of the images. Using two algorithms, double thresholds algorithm and denoising algorithm based on wavelet transform,the fluorescence image and transmission image of a Cell were merged into a composite image. Results and Conclusion The position of fluorescence and the structure of cell can be displyed in the composite image. The signal-to-noise ratio of the exultant image is improved to a large extent. The algorithms are not only useful to investigate the fluorescence and transmission images, but also suitable to observing two or more fluoascent label proes in a single cell.
基金National Natural Science Foundation of China(No.61171177)National Major Scientific Equipment Development Projects of China(No.2013YQ240803)+1 种基金Natural Science Foundation for Young Scientists of Shanxi Province(No.2012021011-1)Scientific and Technological Project in Shanxi Province(No.20140321010-02)
文摘Infrared and visible light images can be obtained simultaneously by building fluorescence imaging system,which includes fluorescence excitation,images acquisition,mechanical part,image transmission and processing section.This system studied the 2charge-coupled device(CCD)camera(AD-080CL)of the JAI company.Fusion algorithm of visible light and near infrared images was designed for the fluorescence imaging system with wavelet transform image fusion algorithm.In order to enhance the fluorescent moiety of the fusion image,the luminance value of the green component of the color image was changed.And using microsoft foundation classes(MFC)application architecture,the supporting software system was bulit in VS2010 environment.
文摘Objective： The arrival of precision medicine plan brings new opportunities and challenges for patients undergoing precision diagnosis and treatment of malignant tumors. With the development of medical imaging, information on different modality imaging can be integrated and comprehensively analyzed by imaging fusion system. This review aimed to update the application of multimodality imaging fusion technology in the precise diagnosis and treatment of malignant tumors under the precision medicine plan. We introduced several multimodality imaging fusion technologies and their application to the diagnosis and treatment of malignant tumors in clinical practice. Date Sources： The data cited in this review were obtained mainly from the PubMed database from 1996 to 2016, using the keywords of ＂precision medicine＂, ＂fusion imaging＂, ＂multimodality＂, and ＂tumor diagnosis and treatment＂. Study Selection： Original articles, clinical practice, reviews, and other relevant literatures published in English were reviewed. Papers focusing on precision medicine, fusion imaging, multimodality, and tumor diagnosis and treatment were selected. Duplicated papers were excluded. Results： Multimodality imaging fusion technology plays an important role in tumor diagnosis and treatment under the precision medicine plan, such as accurate location, qualitative diagnosis, tumor staging, treatment plan design, and real-time intraoperative monitoring. Multimodality imaging fusion systems could provide more imaging information of tumors from different dimensions and angles, thereby offing strong technical support for the implementation of precision oncology. Conclusion： Under the precision medicine plan, personalized treatment of tumors is a distinct possibility. We believe that multimodality imaging fusion technology will find an increasingly wide application in clinical practice.
基金supported by the National Natural Science Foundation of China(NSFC)under grant numbers 61201368.
文摘Image fusion is a key technology in the field of digital image processing.In the present study,an effect-based pseudo color fusion model of infrared and visible images based on the rattlesnake vision imaging system(the rattlesnake bimodal cell fusion mechanism and the visual receptive field model)is proposed.The innovation point of the proposed model lies in the following three features:first,the introduction of a simple mathematical model of the visual receptive field reduce computational complexity;second,the enhanced image is obtained by extracting the common information and unique information of source images,which improves fusion image quality;and third,the Waxman typical fusion structure is improved for the pseudo color image fusion model.The performance of the image fusion model is verified through comparative experiments.In the subjective visual evaluation,we find that the color of the fusion image obtained through the proposed model is natural and can highlight the target and scene details.In the objective quantitative evaluation,we observe that the best values on the four indicators,namely standard deviation,average gradient,entropy,and spatial frequency,accounts for 90%,100%,90%,and 100%,respectively,indicating that the fusion image exhibits superior contrast,image clarity,information content,and overall activity.Experimental results reveal that the performance of the proposed model is superior to that of other models and thus verified the validity and reliability of the model.
文摘Objective： The aim of our study was to compare the value of computed tomography （CT） and 99mTc-methylene- diphosphonate （MDP） SPECT （single photon emission computed tomography）/CT fusion imaging in determining the extent of mandibular invasion by malignant tumor of the oral cavity. Methods： This study had local ethical committee approval, and all patients gave written informed consent. Fifty-three patients were revealed mandibular invasion by malignant tumor of the oral cavity underwent CT and SPECT/CT. The patients were divided into two groups： group A （invasion-periphery-type） and group B （invasion-center- type）. Two radiologists assessed the CT images and two nuclear medicine physicians separately assessed the $PECT/CT images in consensus and without knowledge of the results of other imaging tests. The extent of bone involvement suggested with an imaging modality was compared with pathological findings in the surgical specimen. Results： With pathological findings as the standard of reference, Group A： The extent of mandibular invasion by malignant tumor under- went SPECT/CT was 1.02 _＋ 0.20 cm larger than that underwent pathological findings. And the extent of mandibular invasion underwent CT was 1.42 ＋ 0.35 cm smaller than that underwent pathological examination. There were significant difference among the three methods （P 〈 0.01）. Group B： The extent of mandibular invasion by malignant tumor underwent SPECT/CT was 1.3 ＋ 0.39 cm larger than that underwent pathological examination. The extent of mandibular invasion underwent CT was 2.55 ＋ 1.44 cm smaller than that underwent pathological findings. There were significant difference among the three methods （P 〈 0.01）. The extent of mandibular invasion underwent SPECT/CT was the extent which surgeon must excise to get clear margins. Conclusion： SPECT/CT fusion imaging has significant clinical value in determining the extent of mandibular inva- sion by malignant tumor of oral cavity.
文摘A novel image fusion network framework with an autonomous encoder and decoder is suggested to increase thevisual impression of fused images by improving the quality of infrared and visible light picture fusion. The networkcomprises an encoder module, fusion layer, decoder module, and edge improvementmodule. The encoder moduleutilizes an enhanced Inception module for shallow feature extraction, then combines Res2Net and Transformerto achieve deep-level co-extraction of local and global features from the original picture. An edge enhancementmodule (EEM) is created to extract significant edge features. A modal maximum difference fusion strategy isintroduced to enhance the adaptive representation of information in various regions of the source image, therebyenhancing the contrast of the fused image. The encoder and the EEM module extract features, which are thencombined in the fusion layer to create a fused picture using the decoder. Three datasets were chosen to test thealgorithmproposed in this paper. The results of the experiments demonstrate that the network effectively preservesbackground and detail information in both infrared and visible images, yielding superior outcomes in subjectiveand objective evaluations.
文摘To address the issues of incomplete information,blurred details,loss of details,and insufficient contrast in infrared and visible image fusion,an image fusion algorithm based on a convolutional autoencoder is proposed.The region attention module is meant to extract the background feature map based on the distinct properties of the background feature map and the detail feature map.A multi-scale convolution attention module is suggested to enhance the communication of feature information.At the same time,the feature transformation module is introduced to learn more robust feature representations,aiming to preserve the integrity of image information.This study uses three available datasets from TNO,FLIR,and NIR to perform thorough quantitative and qualitative trials with five additional algorithms.The methods are assessed based on four indicators:information entropy(EN),standard deviation(SD),spatial frequency(SF),and average gradient(AG).Object detection experiments were done on the M3FD dataset to further verify the algorithm’s performance in comparison with five other algorithms.The algorithm’s accuracy was evaluated using the mean average precision at a threshold of 0.5(mAP@0.5)index.Comprehensive experimental findings show that CAEFusion performs well in subjective visual and objective evaluation criteria and has promising potential in downstream object detection tasks.
文摘Objective： The aim of the study was to evaluate the clinical value of ^99mTc-methylene diphosphonic acid （MDP） SPECT/CT fusion imaging and CT scanning in diagnosis of infiltrated mandible by gingival carcinoma. Methods： 18 cases of gingival carcinoma were processed infiltrated mandible by ^99mTc-MDP SPECT/CT fusion image and CT, and their scanning results compared with pathology findings. Results： Eleven of 13 cases with well-differentiated squamous cell carcinoma showed positive images, one of 11 cases was false positive images by pathology findings, and 10 cases were exhibited infiltrated mandibles; 5 cases with moderately differentiated and poorly differentiated squamous call carcinoma showed positive images, pathology showed carcinoma call had infiltrated cavum ossis of mandible. Five of 18 cases were positive images by CT. Conclusion： ^99mTc-MDP SPECT/CT fusion imaging is a useful method in diagnosis of infiltrated mandible by gingival carcinoma.
基金funded by the National Natural Science Foundation of China,grant number 61302188.
文摘Multimodal medical image fusion can help physicians provide more accurate treatment plans for patients, as unimodal images provide limited valid information. To address the insufficient ability of traditional medical image fusion solutions to protect image details and significant information, a new multimodality medical image fusion method(NSST-PAPCNNLatLRR) is proposed in this paper. Firstly, the high and low-frequency sub-band coefficients are obtained by decomposing the source image using NSST. Then, the latent low-rank representation algorithm is used to process the low-frequency sub-band coefficients;An improved PAPCNN algorithm is also proposed for the fusion of high-frequency sub-band coefficients. The improved PAPCNN model was based on the automatic setting of the parameters, and the optimal method was configured for the time decay factor αe. The experimental results show that, in comparison with the five mainstream fusion algorithms, the new algorithm has significantly improved the visual effect over the comparison algorithm,enhanced the ability to characterize important information in images, and further improved the ability to protect the detailed information;the new algorithm has achieved at least four firsts in six objective indexes.
