To avoid the laborious annotation process for dense prediction tasks like semantic segmentation,unsupervised domain adaptation(UDA)methods have been proposed to leverage the abundant annotations from a source domain,s...To avoid the laborious annotation process for dense prediction tasks like semantic segmentation,unsupervised domain adaptation(UDA)methods have been proposed to leverage the abundant annotations from a source domain,such as virtual world(e.g.,3D games),and adapt models to the target domain(the real world)by narrowing the domain discrepancies.However,because of the large domain gap,directly aligning two distinct domains without considering the intermediates leads to inefficient alignment and inferior adaptation.To address this issue,we propose a novel learnable evolutionary Category Intermediates(CIs)guided UDA model named Leci,which enables the information transfer between the two domains via two processes,i.e.,Distilling and Blending.Starting from a random initialization,the CIs learn shared category-wise semantics automatically from two domains in the Distilling process.Then,the learned semantics in the CIs are sent back to blend the domain features through a residual attentive fusion(RAF)module,such that the categorywise features of both domains shift towards each other.As the CIs progressively and consistently learn from the varying feature distributions during training,they are evolutionary to guide the model to achieve category-wise feature alignment.Experiments on both GTA5 and SYNTHIA datasets demonstrate Leci's superiority over prior representative methods.展开更多
Traditional models for semantic segmentation in point clouds primarily focus on smaller scales.However,in real-world applications,point clouds often exhibit larger scales,leading to heavy computational and memory requ...Traditional models for semantic segmentation in point clouds primarily focus on smaller scales.However,in real-world applications,point clouds often exhibit larger scales,leading to heavy computational and memory requirements.The key to handling large-scale point clouds lies in leveraging random sampling,which offers higher computational efficiency and lower memory consumption compared to other sampling methods.Nevertheless,the use of random sampling can potentially result in the loss of crucial points during the encoding stage.To address these issues,this paper proposes cross-fusion self-attention network(CFSA-Net),a lightweight and efficient network architecture specifically designed for directly processing large-scale point clouds.At the core of this network is the incorporation of random sampling alongside a local feature extraction module based on cross-fusion self-attention(CFSA).This module effectively integrates long-range contextual dependencies between points by employing hierarchical position encoding(HPC).Furthermore,it enhances the interaction between each point’s coordinates and feature information through cross-fusion self-attention pooling,enabling the acquisition of more comprehensive geometric information.Finally,a residual optimization(RO)structure is introduced to extend the receptive field of individual points by stacking hierarchical position encoding and cross-fusion self-attention pooling,thereby reducing the impact of information loss caused by random sampling.Experimental results on the Stanford Large-Scale 3D Indoor Spaces(S3DIS),Semantic3D,and SemanticKITTI datasets demonstrate the superiority of this algorithm over advanced approaches such as RandLA-Net and KPConv.These findings underscore the excellent performance of CFSA-Net in large-scale 3D semantic segmentation.展开更多
Large-scale unsupervised semantic segmentation(LUSS)is a sophisticated process that aims to segment similar areas within an image without relying on labeled training data.While existing methodologies have made substan...Large-scale unsupervised semantic segmentation(LUSS)is a sophisticated process that aims to segment similar areas within an image without relying on labeled training data.While existing methodologies have made substantial progress in this area,there is ample scope for enhancement.We thus introduce the PASS-SAM model,a comprehensive solution that amalgamates the benefits of various models to improve segmentation performance.展开更多
Learning-based multi-task models have been widely used in various scene understanding tasks,and complement each other,i.e.,they allow us to consider prior semantic information to better infer depth.We boost the unsupe...Learning-based multi-task models have been widely used in various scene understanding tasks,and complement each other,i.e.,they allow us to consider prior semantic information to better infer depth.We boost the unsupervised monocular depth estimation using semantic segmentation as an auxiliary task.To address the lack of cross-domain datasets and catastrophic forgetting problems encountered in multi-task training,we utilize existing methodology to obtain redundant segmentation maps to build our cross-domain dataset,which not only provides a new way to conduct multi-task training,but also helps us to evaluate results compared with those of other algorithms.In addition,in order to comprehensively use the extracted features of the two tasks in the early perception stage,we use a strategy of sharing weights in the network to fuse cross-domain features,and introduce a novel multi-task loss function to further smooth the depth values.Extensive experiments on KITTI and Cityscapes datasets show that our method has achieved state-of-the-art performance in the depth estimation task,as well improved semantic segmentation.展开更多
Scene segmentation is widely used in autonomous driving for environmental perception.Semantic scene segmentation has gained considerable attention owing to its rich semantic information.It assigns labels to the pixels...Scene segmentation is widely used in autonomous driving for environmental perception.Semantic scene segmentation has gained considerable attention owing to its rich semantic information.It assigns labels to the pixels in an image,thereby enabling automatic image labeling.Current approaches are based mainly on convolutional neural networks(CNN),however,they rely on numerous labels.Therefore,the use of a small amount of labeled data to achieve semantic segmentation has become increasingly important.In this study,we developed a domain adaptation framework based on optimal transport(OT)and an attention mechanism to address this issue.Specifically,we first generated the output space via a CNN owing to its superior of feature representation.Second,we utilized OT to achieve a more robust alignment of the source and target domains in the output space,where the OT plan defined a well attention mechanism to improve the adaptation of the model.In particular,the OT reduced the number of network parameters and made the network more interpretable.Third,to better describe the multiscale properties of the features,we constructed a multiscale segmentation network to perform domain adaptation.Finally,to verify the performance of the proposed method,we conducted an experiment to compare the proposed method with three benchmark and four SOTA methods using three scene datasets.The mean intersection-over-union(mIOU)was significantly improved,and visualization results under multiple domain adaptation scenarios also show that the proposed method performed better than semantic segmentation methods.展开更多
语义分割技术能够对复杂、多元的场景实现细粒度理解,是促进无人系统高效、智能工作的关键技术之一.大规模无监督语义分割旨在从大规模未标记图像中学习语义分割能力.然而,现有方法由于自学习伪标签存在类别混淆和形状表示欠佳的问题,...语义分割技术能够对复杂、多元的场景实现细粒度理解,是促进无人系统高效、智能工作的关键技术之一.大规模无监督语义分割旨在从大规模未标记图像中学习语义分割能力.然而,现有方法由于自学习伪标签存在类别混淆和形状表示欠佳的问题,导致最终分割精度较低.为此,本文提出一种伪标签去噪和SAM优化(Pseudo-label Denoising and SAM Optimization,PDSO)方法以解决大规模无监督语义分割问题.本文设计了一种基于去噪的特征微调模块,在基于小损失准则从大规模数据集中筛选出具有干净图像级伪标签的潜在样本后,利用这些干净样本对预训练的主干网络进行微调,使网络获得更稳健的类别表示.为了进一步减少伪标签中的类别噪声,设计了一种基于聚类的样本去噪模块,根据类别占比和样本与聚类中心之间的距离来去除干扰聚类任务的噪声样本,从而提升聚类性能.本文还设计了一种SAM提示优化模块,根据聚类距离识别出图像中的活跃类别,以过滤噪声目标,并将点和框作为SAM的目标提示信息,生成预期的目标掩膜以细化伪标签中目标的边缘.实验结果表明,在大规模语义分割数据集ImageNet-S_(50)、ImageNet-S_(300)和ImageNet-S_(919)的测试集上,本文方法在平均交并比指标上分别达到了45.0%、26.6%和14.5%,显著提高了分割目标的类别准确率和边缘精度.展开更多
在图像语义分割领域,无监督领域自适应技术的发展有效降低了模型对标注数据的依赖,提升了自动驾驶等智能系统的效率和广泛适用性。针对无监督领域自适应技术在新场景泛化能力有限及在稀有类别中分割效果差的问题,文章提出了一种基于强...在图像语义分割领域,无监督领域自适应技术的发展有效降低了模型对标注数据的依赖,提升了自动驾驶等智能系统的效率和广泛适用性。针对无监督领域自适应技术在新场景泛化能力有限及在稀有类别中分割效果差的问题,文章提出了一种基于强弱一致性的无监督领域自适应语义分割算法。算法首先通过增加特征级别的增强,拓展图像增强空间的维度,改善了只利用图像级增强局限性。其次,采用基于能量分数的伪标签筛选方法,筛选出足够接近当前训练数据的样本赋予伪标签,避免了使用Softmax置信度方法在稀有类别中存在局限性,使模型更新更加稳健。最后,构建结合图像级别增强和特征级别增强的双重一致性框架,更充分的利用一致性训练,进一步提高模型的泛化能力。实验结果证明,提出的方法在GTA5-to-Cityscapes公开数据集中平均交并比指标(mean Intersection over Union,mIoU)可提升至52.6%,较PixMatch算法,性能提升了4.3%。展开更多
不同成像模式设备采集的医学图像存在不同程度的分布差异,无监督域自适应方法为了将源域训练的模型泛化到无标注的目标域,通常是将差异分布最小化,使用源域和目标域的共有特征进行结果预测,但会忽略目标域的私有特征.为了解决该问题,文...不同成像模式设备采集的医学图像存在不同程度的分布差异,无监督域自适应方法为了将源域训练的模型泛化到无标注的目标域,通常是将差异分布最小化,使用源域和目标域的共有特征进行结果预测,但会忽略目标域的私有特征.为了解决该问题,文中提出基于目标域增强表示的医学图像无监督跨域分割方法(Enhanced Target Domain Representation Based Unsupervised Cross-Domain Medical Image Segmentation,TreUCMIS).首先,通过共有特征学习获取源域和目标域的共有特征,通过图像重构训练目标域特征编码器,提取目标域完整特征.然后,通过目标域的无监督自学习方式,加强深层特征和浅层特征的共有性.最后,对齐使用共有特征和完整特征得到的预测结果,利用目标域的完整特征分割目标,提高模型在目标域的泛化性.在两个具有CT和MRI双向域自适应任务的医学图像分割数据集(腹部、心脏)上的实验表明TreUCMIS的有效性与优越性.展开更多
基金Australian Research Council Project(FL-170100117).
文摘To avoid the laborious annotation process for dense prediction tasks like semantic segmentation,unsupervised domain adaptation(UDA)methods have been proposed to leverage the abundant annotations from a source domain,such as virtual world(e.g.,3D games),and adapt models to the target domain(the real world)by narrowing the domain discrepancies.However,because of the large domain gap,directly aligning two distinct domains without considering the intermediates leads to inefficient alignment and inferior adaptation.To address this issue,we propose a novel learnable evolutionary Category Intermediates(CIs)guided UDA model named Leci,which enables the information transfer between the two domains via two processes,i.e.,Distilling and Blending.Starting from a random initialization,the CIs learn shared category-wise semantics automatically from two domains in the Distilling process.Then,the learned semantics in the CIs are sent back to blend the domain features through a residual attentive fusion(RAF)module,such that the categorywise features of both domains shift towards each other.As the CIs progressively and consistently learn from the varying feature distributions during training,they are evolutionary to guide the model to achieve category-wise feature alignment.Experiments on both GTA5 and SYNTHIA datasets demonstrate Leci's superiority over prior representative methods.
基金funded by the National Natural Science Foundation of China Youth Project(61603127).
文摘Traditional models for semantic segmentation in point clouds primarily focus on smaller scales.However,in real-world applications,point clouds often exhibit larger scales,leading to heavy computational and memory requirements.The key to handling large-scale point clouds lies in leveraging random sampling,which offers higher computational efficiency and lower memory consumption compared to other sampling methods.Nevertheless,the use of random sampling can potentially result in the loss of crucial points during the encoding stage.To address these issues,this paper proposes cross-fusion self-attention network(CFSA-Net),a lightweight and efficient network architecture specifically designed for directly processing large-scale point clouds.At the core of this network is the incorporation of random sampling alongside a local feature extraction module based on cross-fusion self-attention(CFSA).This module effectively integrates long-range contextual dependencies between points by employing hierarchical position encoding(HPC).Furthermore,it enhances the interaction between each point’s coordinates and feature information through cross-fusion self-attention pooling,enabling the acquisition of more comprehensive geometric information.Finally,a residual optimization(RO)structure is introduced to extend the receptive field of individual points by stacking hierarchical position encoding and cross-fusion self-attention pooling,thereby reducing the impact of information loss caused by random sampling.Experimental results on the Stanford Large-Scale 3D Indoor Spaces(S3DIS),Semantic3D,and SemanticKITTI datasets demonstrate the superiority of this algorithm over advanced approaches such as RandLA-Net and KPConv.These findings underscore the excellent performance of CFSA-Net in large-scale 3D semantic segmentation.
文摘Large-scale unsupervised semantic segmentation(LUSS)is a sophisticated process that aims to segment similar areas within an image without relying on labeled training data.While existing methodologies have made substantial progress in this area,there is ample scope for enhancement.We thus introduce the PASS-SAM model,a comprehensive solution that amalgamates the benefits of various models to improve segmentation performance.
基金This work was supported by the national key research development plan(Project No.YS2018YFB1403703)research project of the communication university of china(Project No.CUC200D058).
文摘Learning-based multi-task models have been widely used in various scene understanding tasks,and complement each other,i.e.,they allow us to consider prior semantic information to better infer depth.We boost the unsupervised monocular depth estimation using semantic segmentation as an auxiliary task.To address the lack of cross-domain datasets and catastrophic forgetting problems encountered in multi-task training,we utilize existing methodology to obtain redundant segmentation maps to build our cross-domain dataset,which not only provides a new way to conduct multi-task training,but also helps us to evaluate results compared with those of other algorithms.In addition,in order to comprehensively use the extracted features of the two tasks in the early perception stage,we use a strategy of sharing weights in the network to fuse cross-domain features,and introduce a novel multi-task loss function to further smooth the depth values.Extensive experiments on KITTI and Cityscapes datasets show that our method has achieved state-of-the-art performance in the depth estimation task,as well improved semantic segmentation.
基金supported by the National Natural Science Foundation of China(11971296)National Key R&D Program of China(2021YFA1003004).
文摘Scene segmentation is widely used in autonomous driving for environmental perception.Semantic scene segmentation has gained considerable attention owing to its rich semantic information.It assigns labels to the pixels in an image,thereby enabling automatic image labeling.Current approaches are based mainly on convolutional neural networks(CNN),however,they rely on numerous labels.Therefore,the use of a small amount of labeled data to achieve semantic segmentation has become increasingly important.In this study,we developed a domain adaptation framework based on optimal transport(OT)and an attention mechanism to address this issue.Specifically,we first generated the output space via a CNN owing to its superior of feature representation.Second,we utilized OT to achieve a more robust alignment of the source and target domains in the output space,where the OT plan defined a well attention mechanism to improve the adaptation of the model.In particular,the OT reduced the number of network parameters and made the network more interpretable.Third,to better describe the multiscale properties of the features,we constructed a multiscale segmentation network to perform domain adaptation.Finally,to verify the performance of the proposed method,we conducted an experiment to compare the proposed method with three benchmark and four SOTA methods using three scene datasets.The mean intersection-over-union(mIOU)was significantly improved,and visualization results under multiple domain adaptation scenarios also show that the proposed method performed better than semantic segmentation methods.
文摘语义分割技术能够对复杂、多元的场景实现细粒度理解,是促进无人系统高效、智能工作的关键技术之一.大规模无监督语义分割旨在从大规模未标记图像中学习语义分割能力.然而,现有方法由于自学习伪标签存在类别混淆和形状表示欠佳的问题,导致最终分割精度较低.为此,本文提出一种伪标签去噪和SAM优化(Pseudo-label Denoising and SAM Optimization,PDSO)方法以解决大规模无监督语义分割问题.本文设计了一种基于去噪的特征微调模块,在基于小损失准则从大规模数据集中筛选出具有干净图像级伪标签的潜在样本后,利用这些干净样本对预训练的主干网络进行微调,使网络获得更稳健的类别表示.为了进一步减少伪标签中的类别噪声,设计了一种基于聚类的样本去噪模块,根据类别占比和样本与聚类中心之间的距离来去除干扰聚类任务的噪声样本,从而提升聚类性能.本文还设计了一种SAM提示优化模块,根据聚类距离识别出图像中的活跃类别,以过滤噪声目标,并将点和框作为SAM的目标提示信息,生成预期的目标掩膜以细化伪标签中目标的边缘.实验结果表明,在大规模语义分割数据集ImageNet-S_(50)、ImageNet-S_(300)和ImageNet-S_(919)的测试集上,本文方法在平均交并比指标上分别达到了45.0%、26.6%和14.5%,显著提高了分割目标的类别准确率和边缘精度.
文摘在图像语义分割领域,无监督领域自适应技术的发展有效降低了模型对标注数据的依赖,提升了自动驾驶等智能系统的效率和广泛适用性。针对无监督领域自适应技术在新场景泛化能力有限及在稀有类别中分割效果差的问题,文章提出了一种基于强弱一致性的无监督领域自适应语义分割算法。算法首先通过增加特征级别的增强,拓展图像增强空间的维度,改善了只利用图像级增强局限性。其次,采用基于能量分数的伪标签筛选方法,筛选出足够接近当前训练数据的样本赋予伪标签,避免了使用Softmax置信度方法在稀有类别中存在局限性,使模型更新更加稳健。最后,构建结合图像级别增强和特征级别增强的双重一致性框架,更充分的利用一致性训练,进一步提高模型的泛化能力。实验结果证明,提出的方法在GTA5-to-Cityscapes公开数据集中平均交并比指标(mean Intersection over Union,mIoU)可提升至52.6%,较PixMatch算法,性能提升了4.3%。
文摘不同成像模式设备采集的医学图像存在不同程度的分布差异,无监督域自适应方法为了将源域训练的模型泛化到无标注的目标域,通常是将差异分布最小化,使用源域和目标域的共有特征进行结果预测,但会忽略目标域的私有特征.为了解决该问题,文中提出基于目标域增强表示的医学图像无监督跨域分割方法(Enhanced Target Domain Representation Based Unsupervised Cross-Domain Medical Image Segmentation,TreUCMIS).首先,通过共有特征学习获取源域和目标域的共有特征,通过图像重构训练目标域特征编码器,提取目标域完整特征.然后,通过目标域的无监督自学习方式,加强深层特征和浅层特征的共有性.最后,对齐使用共有特征和完整特征得到的预测结果,利用目标域的完整特征分割目标,提高模型在目标域的泛化性.在两个具有CT和MRI双向域自适应任务的医学图像分割数据集(腹部、心脏)上的实验表明TreUCMIS的有效性与优越性.
