Precise polyp segmentation is vital for the early diagnosis and prevention of colorectal cancer(CRC)in clinical practice.However,due to scale variation and blurry polyp boundaries,it is still a challenging task to ach...Precise polyp segmentation is vital for the early diagnosis and prevention of colorectal cancer(CRC)in clinical practice.However,due to scale variation and blurry polyp boundaries,it is still a challenging task to achieve satisfactory segmentation performance with different scales and shapes.In this study,we present a novel edge-aware feature aggregation network(EFA-Net)for polyp segmentation,which can fully make use of cross-level and multi-scale features to enhance the performance of polyp segmentation.Specifically,we first present an edge-aware guidance module(EGM)to combine the low-level features with the high-level features to learn an edge-enhanced feature,which is incorporated into each decoder unit using a layer-by-layer strategy.Besides,a scale-aware convolution module(SCM)is proposed to learn scale-aware features by using dilated convolutions with different ratios,in order to effectively deal with scale variation.Further,a cross-level fusion module(CFM)is proposed to effectively integrate the cross-level features,which can exploit the local and global contextual information.Finally,the outputs of CFMs are adaptively weighted by using the learned edge-aware feature,which are then used to produce multiple side-out segmentation maps.Experimental results on five widely adopted colonoscopy datasets show that our EFA-Net outperforms state-of-the-art polyp segmentation methods in terms of generalization and effectiveness.Our implementation code and segmentation maps will be publicly at https://github.com/taozh2017/EFANet.展开更多
We present the first comprehensive video polyp segmentation(VPS)study in the deep learning era.Over the years,developments in VPS are not moving forward with ease due to the lack of a large-scale dataset with fine-gra...We present the first comprehensive video polyp segmentation(VPS)study in the deep learning era.Over the years,developments in VPS are not moving forward with ease due to the lack of a large-scale dataset with fine-grained segmentation annotations.To address this issue,we first introduce a high-quality frame-by-frame annotated VPS dataset,named SUN-SEG,which contains 158690colonoscopy video frames from the well-known SUN-database.We provide additional annotation covering diverse types,i.e.,attribute,object mask,boundary,scribble,and polygon.Second,we design a simple but efficient baseline,named PNS+,which consists of a global encoder,a local encoder,and normalized self-attention(NS)blocks.The global and local encoders receive an anchor frame and multiple successive frames to extract long-term and short-term spatial-temporal representations,which are then progressively refined by two NS blocks.Extensive experiments show that PNS+achieves the best performance and real-time inference speed(170 fps),making it a promising solution for the VPS task.Third,we extensively evaluate 13 representative polyp/object segmentation models on our SUN-SEG dataset and provide attribute-based comparisons.Finally,we discuss several open issues and suggest possible research directions for the VPS community.Our project and dataset are publicly available at https://github.com/GewelsJI/VPS.展开更多
Most polyp segmentation methods use convolutional neural networks(CNNs)as their backbone,leading to two key issues when exchanging information between the encoder and decoder:(1)taking into account the differences in ...Most polyp segmentation methods use convolutional neural networks(CNNs)as their backbone,leading to two key issues when exchanging information between the encoder and decoder:(1)taking into account the differences in contribution between different-level features,and(2)designing an effective mechanism for fusing these features.Unlike existing CNN-based methods,we adopt a transformer encoder,which learns more powerful and robust representations.In addition,considering the image acquisition influence and elusive properties of polyps,we introduce three standard modules,including a cascaded fusion module(CFM),a camouflage identification module(CIM),and a similarity aggregation module(SAM).Among these,the CFM is used to collect the semantic and location information of polyps from high-level features;the CIM is applied to capture polyp information disguised in low-level features,and the SAM extends the pixel features of the polyp area with high-level semantic position information to the entire polyp area,thereby effectively fusing cross-level features.The proposed model,named Polyp-PVT,effectively suppresses noises in the features and significantly improves their expressive capabilities.Extensive experiments on five widely adopted datasets show that the proposed model is more robust to various challenging situations(e.g.,appearance changes,small objects,and rotation)than existing representative methods.The proposed model is available at https://github.com/DengPingFan/Polyp-PVT.展开更多
Unlike existing fully-supervised approaches,we rethink colorectal polyp segmentation from an out-of-distribution perspective with a simple but effective self-supervised learning approach.We leverage the ability of mas...Unlike existing fully-supervised approaches,we rethink colorectal polyp segmentation from an out-of-distribution perspective with a simple but effective self-supervised learning approach.We leverage the ability of masked autoencoders-self-supervised vision transformers trained on a reconstruction task-to learn in-distribution representations,here,the distribution of healthy colon images.We then perform out-of-distribution reconstruction and inference,with feature space standardisation to align the latent distribution of the diverse abnormal samples with the statistics of the healthy samples.We generate per-pixel anomaly scores for each image by calculating the difference between the input and reconstructed images and use this signal for out-of-distribution(i.e.,polyp)segmentation.Experimental results on six benchmarks show that our model has excellent segmentation performance and generalises across datasets.Our code is publicly available at https://github.com/GewelsJI/Polyp-OOD.展开更多
Objective:Deep learning is employed increasingly in Gastroenterology(GI)endoscopy computer-aided diagnostics for polyp segmentation and multi-class disease detection.In the real world,implementation requires high accu...Objective:Deep learning is employed increasingly in Gastroenterology(GI)endoscopy computer-aided diagnostics for polyp segmentation and multi-class disease detection.In the real world,implementation requires high accuracy,therapeutically relevant explanations,strong calibration,domain generalization,and efficiency.Current Convolutional Neural Network(CNN)and transformer models compromise border precision and global context,generate attention maps that fail to align with expert reasoning,deteriorate during cross-center changes,and exhibit inadequate calibration,hence diminishing clinical trust.Methods:HMA-DER is a hierarchical multi-attention architecture that uses dilation-enhanced residual blocks and an explainability-aware Cognitive Alignment Score(CAS)regularizer to directly align attribution maps with reasoning signals from experts.The framework has additions that make it more resilient and a way to test for accuracy,macro-averaged F1 score,Area Under the Receiver Operating Characteristic Curve(AUROC),calibration(Expected Calibration Error(ECE),Brier Score),explainability(CAS,insertion/deletion AUC),cross-dataset transfer,and throughput.Results:HMA-DER gets Dice Similarity Coefficient scores of 89.5%and 86.0%on Kvasir-SEG and CVC-ClinicDB,beating the strongest baseline by+1.9 and+1.7 points.It gets 86.4%and 85.3%macro-F1 and 94.0%and 93.4%AUROC on HyperKvasir and GastroVision,which is better than the baseline by+1.4/+1.6macro-F1 and+1.2/+1.1AUROC.Ablation study shows that hierarchical attention gives the highest(+3.0),followed by CAS regularization(+2–3),dilatation(+1.5–2.0),and residual connections(+2–3).Cross-dataset validation demonstrates competitive zero-shot transfer(e.g.,KS→CVC Dice 82.7%),whereas multi-dataset training diminishes the domain gap,yielding an 88.1%primary-metric average.HMA-DER’s mixed-precision inference can handle 155 pictures per second,which helps with calibration.Conclusion:HMA-DER strikes a compromise between accuracy,explainability,robustness,and efficiency for the use of reliable GI computer-aided diagnosis in real-world clinical settings.展开更多
Colonoscopy is currently one of the most sensitive screening methods for colorectal cancer.This study investigates the frontiers of intelligent colonoscopy techniques and their prospective implications for multimodal ...Colonoscopy is currently one of the most sensitive screening methods for colorectal cancer.This study investigates the frontiers of intelligent colonoscopy techniques and their prospective implications for multimodal medical applications.With this goal,we begin by assessing the current data-centric and model-centric landscapes through four tasks for colonoscopic scene perception,including classification,detection,segmentation,and vision-language understanding.Our assessment reveals domain-specific challenges and underscores the need for further multimodal research in colonoscopy.To address these gaps,we establish three foundational initiatives:a large-scale multimodal instruction tuning dataset ColonINST,a colonoscopy-designed multimodal language model ColonGPT,and a multimodal benchmark.To facilitate continuous advancements in this rapidly evolving field,we provide a public website for the latest updates:https://github.com/ai4colonoscopy/IntelliScope.展开更多
基金supported in part by National Natural Science Foundation of China(Nos.62172228,62201263,62106043 and 62201265).
文摘Precise polyp segmentation is vital for the early diagnosis and prevention of colorectal cancer(CRC)in clinical practice.However,due to scale variation and blurry polyp boundaries,it is still a challenging task to achieve satisfactory segmentation performance with different scales and shapes.In this study,we present a novel edge-aware feature aggregation network(EFA-Net)for polyp segmentation,which can fully make use of cross-level and multi-scale features to enhance the performance of polyp segmentation.Specifically,we first present an edge-aware guidance module(EGM)to combine the low-level features with the high-level features to learn an edge-enhanced feature,which is incorporated into each decoder unit using a layer-by-layer strategy.Besides,a scale-aware convolution module(SCM)is proposed to learn scale-aware features by using dilated convolutions with different ratios,in order to effectively deal with scale variation.Further,a cross-level fusion module(CFM)is proposed to effectively integrate the cross-level features,which can exploit the local and global contextual information.Finally,the outputs of CFMs are adaptively weighted by using the learned edge-aware feature,which are then used to produce multiple side-out segmentation maps.Experimental results on five widely adopted colonoscopy datasets show that our EFA-Net outperforms state-of-the-art polyp segmentation methods in terms of generalization and effectiveness.Our implementation code and segmentation maps will be publicly at https://github.com/taozh2017/EFANet.
基金supported by the National Natural Science Foundation of China(No.62072223)supported by the Natural Science Foundation of Fujian Province,China(No.2020J01131199)。
文摘We present the first comprehensive video polyp segmentation(VPS)study in the deep learning era.Over the years,developments in VPS are not moving forward with ease due to the lack of a large-scale dataset with fine-grained segmentation annotations.To address this issue,we first introduce a high-quality frame-by-frame annotated VPS dataset,named SUN-SEG,which contains 158690colonoscopy video frames from the well-known SUN-database.We provide additional annotation covering diverse types,i.e.,attribute,object mask,boundary,scribble,and polygon.Second,we design a simple but efficient baseline,named PNS+,which consists of a global encoder,a local encoder,and normalized self-attention(NS)blocks.The global and local encoders receive an anchor frame and multiple successive frames to extract long-term and short-term spatial-temporal representations,which are then progressively refined by two NS blocks.Extensive experiments show that PNS+achieves the best performance and real-time inference speed(170 fps),making it a promising solution for the VPS task.Third,we extensively evaluate 13 representative polyp/object segmentation models on our SUN-SEG dataset and provide attribute-based comparisons.Finally,we discuss several open issues and suggest possible research directions for the VPS community.Our project and dataset are publicly available at https://github.com/GewelsJI/VPS.
文摘Most polyp segmentation methods use convolutional neural networks(CNNs)as their backbone,leading to two key issues when exchanging information between the encoder and decoder:(1)taking into account the differences in contribution between different-level features,and(2)designing an effective mechanism for fusing these features.Unlike existing CNN-based methods,we adopt a transformer encoder,which learns more powerful and robust representations.In addition,considering the image acquisition influence and elusive properties of polyps,we introduce three standard modules,including a cascaded fusion module(CFM),a camouflage identification module(CIM),and a similarity aggregation module(SAM).Among these,the CFM is used to collect the semantic and location information of polyps from high-level features;the CIM is applied to capture polyp information disguised in low-level features,and the SAM extends the pixel features of the polyp area with high-level semantic position information to the entire polyp area,thereby effectively fusing cross-level features.The proposed model,named Polyp-PVT,effectively suppresses noises in the features and significantly improves their expressive capabilities.Extensive experiments on five widely adopted datasets show that the proposed model is more robust to various challenging situations(e.g.,appearance changes,small objects,and rotation)than existing representative methods.The proposed model is available at https://github.com/DengPingFan/Polyp-PVT.
文摘Unlike existing fully-supervised approaches,we rethink colorectal polyp segmentation from an out-of-distribution perspective with a simple but effective self-supervised learning approach.We leverage the ability of masked autoencoders-self-supervised vision transformers trained on a reconstruction task-to learn in-distribution representations,here,the distribution of healthy colon images.We then perform out-of-distribution reconstruction and inference,with feature space standardisation to align the latent distribution of the diverse abnormal samples with the statistics of the healthy samples.We generate per-pixel anomaly scores for each image by calculating the difference between the input and reconstructed images and use this signal for out-of-distribution(i.e.,polyp)segmentation.Experimental results on six benchmarks show that our model has excellent segmentation performance and generalises across datasets.Our code is publicly available at https://github.com/GewelsJI/Polyp-OOD.
文摘Objective:Deep learning is employed increasingly in Gastroenterology(GI)endoscopy computer-aided diagnostics for polyp segmentation and multi-class disease detection.In the real world,implementation requires high accuracy,therapeutically relevant explanations,strong calibration,domain generalization,and efficiency.Current Convolutional Neural Network(CNN)and transformer models compromise border precision and global context,generate attention maps that fail to align with expert reasoning,deteriorate during cross-center changes,and exhibit inadequate calibration,hence diminishing clinical trust.Methods:HMA-DER is a hierarchical multi-attention architecture that uses dilation-enhanced residual blocks and an explainability-aware Cognitive Alignment Score(CAS)regularizer to directly align attribution maps with reasoning signals from experts.The framework has additions that make it more resilient and a way to test for accuracy,macro-averaged F1 score,Area Under the Receiver Operating Characteristic Curve(AUROC),calibration(Expected Calibration Error(ECE),Brier Score),explainability(CAS,insertion/deletion AUC),cross-dataset transfer,and throughput.Results:HMA-DER gets Dice Similarity Coefficient scores of 89.5%and 86.0%on Kvasir-SEG and CVC-ClinicDB,beating the strongest baseline by+1.9 and+1.7 points.It gets 86.4%and 85.3%macro-F1 and 94.0%and 93.4%AUROC on HyperKvasir and GastroVision,which is better than the baseline by+1.4/+1.6macro-F1 and+1.2/+1.1AUROC.Ablation study shows that hierarchical attention gives the highest(+3.0),followed by CAS regularization(+2–3),dilatation(+1.5–2.0),and residual connections(+2–3).Cross-dataset validation demonstrates competitive zero-shot transfer(e.g.,KS→CVC Dice 82.7%),whereas multi-dataset training diminishes the domain gap,yielding an 88.1%primary-metric average.HMA-DER’s mixed-precision inference can handle 155 pictures per second,which helps with calibration.Conclusion:HMA-DER strikes a compromise between accuracy,explainability,robustness,and efficiency for the use of reliable GI computer-aided diagnosis in real-world clinical settings.
基金supported by NSFC,China(No.62476143)the Fundamental Research Funds for the Central Universities,China(Nankai University,No.63253218)+1 种基金ANU-Optus Bushfire Research Centre of Excellence(BRCoE)(scholarship awarded to Ge-Peng Ji)supported by Natural Science Foundation of China(NSFC)(No.62306162).
文摘Colonoscopy is currently one of the most sensitive screening methods for colorectal cancer.This study investigates the frontiers of intelligent colonoscopy techniques and their prospective implications for multimodal medical applications.With this goal,we begin by assessing the current data-centric and model-centric landscapes through four tasks for colonoscopic scene perception,including classification,detection,segmentation,and vision-language understanding.Our assessment reveals domain-specific challenges and underscores the need for further multimodal research in colonoscopy.To address these gaps,we establish three foundational initiatives:a large-scale multimodal instruction tuning dataset ColonINST,a colonoscopy-designed multimodal language model ColonGPT,and a multimodal benchmark.To facilitate continuous advancements in this rapidly evolving field,we provide a public website for the latest updates:https://github.com/ai4colonoscopy/IntelliScope.
