Previous multi-view 3D human pose estimation methods neither correlate different human joints in each view nor model learnable correlations between the same joints in different views explicitly,meaning that skeleton s...Previous multi-view 3D human pose estimation methods neither correlate different human joints in each view nor model learnable correlations between the same joints in different views explicitly,meaning that skeleton structure information is not utilized and multi-view pose information is not completely fused.Moreover,existing graph convolutional operations do not consider the specificity of different joints and different views of pose information when processing skeleton graphs,making the correlation weights between nodes in the graph and their neighborhood nodes shared.Existing Graph Convolutional Networks(GCNs)cannot extract global and deeplevel skeleton structure information and view correlations efficiently.To solve these problems,pre-estimated multiview 2D poses are designed as a multi-view skeleton graph to fuse skeleton priors and view correlations explicitly to process occlusion problem,with the skeleton-edge and symmetry-edge representing the structure correlations between adjacent joints in each viewof skeleton graph and the view-edge representing the view correlations between the same joints in different views.To make graph convolution operation mine elaborate and sufficient skeleton structure information and view correlations,different correlation weights are assigned to different categories of neighborhood nodes and further assigned to each node in the graph.Based on the graph convolution operation proposed above,a Residual Graph Convolution(RGC)module is designed as the basic module to be combined with the simplified Hourglass architecture to construct the Hourglass-GCN as our 3D pose estimation network.Hourglass-GCNwith a symmetrical and concise architecture processes three scales ofmulti-viewskeleton graphs to extract local-to-global scale and shallow-to-deep level skeleton features efficiently.Experimental results on common large 3D pose dataset Human3.6M and MPI-INF-3DHP show that Hourglass-GCN outperforms some excellent methods in 3D pose estimation accuracy.展开更多
3D human pose estimation is a major focus area in the field of computer vision,which plays an important role in practical applications.This article summarizes the framework and research progress related to the estimat...3D human pose estimation is a major focus area in the field of computer vision,which plays an important role in practical applications.This article summarizes the framework and research progress related to the estimation of monocular RGB images and videos.An overall perspective ofmethods integrated with deep learning is introduced.Novel image-based and video-based inputs are proposed as the analysis framework.From this viewpoint,common problems are discussed.The diversity of human postures usually leads to problems such as occlusion and ambiguity,and the lack of training datasets often results in poor generalization ability of the model.Regression methods are crucial for solving such problems.Considering image-based input,the multi-view method is commonly used to solve occlusion problems.Here,the multi-view method is analyzed comprehensively.By referring to video-based input,the human prior knowledge of restricted motion is used to predict human postures.In addition,structural constraints are widely used as prior knowledge.Furthermore,weakly supervised learningmethods are studied and discussed for these two types of inputs to improve the model generalization ability.The problem of insufficient training datasets must also be considered,especially because 3D datasets are usually biased and limited.Finally,emerging and popular datasets and evaluation indicators are discussed.The characteristics of the datasets and the relationships of the indicators are explained and highlighted.Thus,this article can be useful and instructive for researchers who are lacking in experience and find this field confusing.In addition,by providing an overview of 3D human pose estimation,this article sorts and refines recent studies on 3D human pose estimation.It describes kernel problems and common useful methods,and discusses the scope for further research.展开更多
The field of vision-based human hand three-dimensional(3D)shape and pose estimation has attracted significant attention recently owing to its key role in various applications,such as natural human computer interaction...The field of vision-based human hand three-dimensional(3D)shape and pose estimation has attracted significant attention recently owing to its key role in various applications,such as natural human computer interactions.With the availability of large-scale annotated hand datasets and the rapid developments of deep neural networks(DNNs),numerous DNN-based data-driven methods have been proposed for accurate and rapid hand shape and pose estimation.Nonetheless,the existence of complicated hand articulation,depth and scale ambiguities,occlusions,and finger similarity remain challenging.In this study,we present a comprehensive survey of state-of-the-art 3D hand shape and pose estimation approaches using RGB-D cameras.Related RGB-D cameras,hand datasets,and a performance analysis are also discussed to provide a holistic view of recent achievements.We also discuss the research potential of this rapidly growing field.展开更多
As an essential tool for quantitative analysis of lower limb coordination,optical motion capture systems with marker-based encoding still suffer from inefficiency,high costs,spatial constraints,and the requirement for...As an essential tool for quantitative analysis of lower limb coordination,optical motion capture systems with marker-based encoding still suffer from inefficiency,high costs,spatial constraints,and the requirement for multiple markers.While 3D pose estimation algorithms combined with ordinary cameras offer an alternative,their accuracy often deteriorates under significant body occlusion.To address the challenge of insufficient 3D pose estimation precision in occluded scenarios—which hinders the quantitative analysis of athletes’lower-limb coordination—this paper proposes a multimodal training framework integrating spatiotemporal dependency networks with text-semantic guidance.Compared to traditional optical motion capture systems,this work achieves low-cost,high-precision motion parameter acquisition through the following innovations:(1)spatiotemporal dependency attention module is designed to establish dynamic spatiotemporal correlation graphs via cross-frame joint semantic matching,effectively resolving the feature fragmentation issue in existing methods.(2)noise-suppressed multi-scale temporal module is proposed,leveraging KL divergence-based information gain analysis for progressive feature filtering in long-range dependencies,reducing errors by 1.91 mm compared to conventional temporal convolutions.(3)text-pose contrastive learning paradigm is introduced for the first time,where BERT-generated action descriptions align semantic-geometric features via the BERT encoder,significantly enhancing robustness under severe occlusion(50%joint invisibility).On the Human3.6M dataset,the proposed method achieves an MPJPE of 56.21 mm under Protocol 1,outperforming the state-of-the-art baseline MHFormer by 3.3%.Extensive ablation studies on Human3.6M demonstrate the individual contributions of the core modules:the spatiotemporal dependency module and noise-suppressed multi-scale temporal module reduce MPJPE by 0.30 and 0.34 mm,respectively,while the multimodal training strategy further decreases MPJPE by 0.6 mm through text-skeleton contrastive learning.Comparative experiments involving 16 athletes show that the sagittal plane coupling angle measurements of hip-ankle joints differ by less than 1.2°from those obtained via traditional optical systems(two one-sided t-tests,p<0.05),validating real-world reliability.This study provides an AI-powered analytical solution for competitive sports training,serving as a viable alternative to specialized equipment.展开更多
We address the problem of 3D human pose estimation in a single real scene image. Normally, 3D pose estimation from real image needs background subtraction to extract the appropriate features. We do not make such assum...We address the problem of 3D human pose estimation in a single real scene image. Normally, 3D pose estimation from real image needs background subtraction to extract the appropriate features. We do not make such assumption, In this paper, a two-step approach is proposed, first, instead of applying background subtraction to get the segmentation of human, we combine the segmentation with human detection using an ISM-based detector. Then, silhouette feature can be extracted and 3D pose estimation is solved as a regression problem. RVMs and ridge regression method are applied to solve this problem. The results show the robustness and accuracy of our method.展开更多
In this paper,we present a robot vision based system for coordinate measurement of feature points on large scale automobile parts.Our system consists of an industrial 6-DOF robot mounted with a CCD camera and a PC.The...In this paper,we present a robot vision based system for coordinate measurement of feature points on large scale automobile parts.Our system consists of an industrial 6-DOF robot mounted with a CCD camera and a PC.The system controls the robot into the area of feature points.The images of measuring feature points are acquired by the camera mounted on the robot.3D positions of the feature points are obtained from a model based pose estimation that applies to the images.The measured positions of all feature points are then transformed to the reference coordinate of feature points whose positions are obtained from the coordinate measuring machine（CMM）.Finally,the point-to-point distances between the measured feature points and the reference feature points are calculated and reported.The results show that the root mean square error（RMSE） of measure values obtained by our system is less than 0.5 mm.Our system is adequate for automobile assembly and can perform faster than conventional methods.展开更多
基金supported in part by the National Natural Science Foundation of China under Grants 61973065,U20A20197,61973063.
文摘Previous multi-view 3D human pose estimation methods neither correlate different human joints in each view nor model learnable correlations between the same joints in different views explicitly,meaning that skeleton structure information is not utilized and multi-view pose information is not completely fused.Moreover,existing graph convolutional operations do not consider the specificity of different joints and different views of pose information when processing skeleton graphs,making the correlation weights between nodes in the graph and their neighborhood nodes shared.Existing Graph Convolutional Networks(GCNs)cannot extract global and deeplevel skeleton structure information and view correlations efficiently.To solve these problems,pre-estimated multiview 2D poses are designed as a multi-view skeleton graph to fuse skeleton priors and view correlations explicitly to process occlusion problem,with the skeleton-edge and symmetry-edge representing the structure correlations between adjacent joints in each viewof skeleton graph and the view-edge representing the view correlations between the same joints in different views.To make graph convolution operation mine elaborate and sufficient skeleton structure information and view correlations,different correlation weights are assigned to different categories of neighborhood nodes and further assigned to each node in the graph.Based on the graph convolution operation proposed above,a Residual Graph Convolution(RGC)module is designed as the basic module to be combined with the simplified Hourglass architecture to construct the Hourglass-GCN as our 3D pose estimation network.Hourglass-GCNwith a symmetrical and concise architecture processes three scales ofmulti-viewskeleton graphs to extract local-to-global scale and shallow-to-deep level skeleton features efficiently.Experimental results on common large 3D pose dataset Human3.6M and MPI-INF-3DHP show that Hourglass-GCN outperforms some excellent methods in 3D pose estimation accuracy.
基金supported by the Program of Entrepreneurship and Innovation Ph.D.in Jiangsu Province(JSSCBS20211175)the School Ph.D.Talent Funding(Z301B2055)the Natural Science Foundation of the Jiangsu Higher Education Institutions of China(21KJB520002).
文摘3D human pose estimation is a major focus area in the field of computer vision,which plays an important role in practical applications.This article summarizes the framework and research progress related to the estimation of monocular RGB images and videos.An overall perspective ofmethods integrated with deep learning is introduced.Novel image-based and video-based inputs are proposed as the analysis framework.From this viewpoint,common problems are discussed.The diversity of human postures usually leads to problems such as occlusion and ambiguity,and the lack of training datasets often results in poor generalization ability of the model.Regression methods are crucial for solving such problems.Considering image-based input,the multi-view method is commonly used to solve occlusion problems.Here,the multi-view method is analyzed comprehensively.By referring to video-based input,the human prior knowledge of restricted motion is used to predict human postures.In addition,structural constraints are widely used as prior knowledge.Furthermore,weakly supervised learningmethods are studied and discussed for these two types of inputs to improve the model generalization ability.The problem of insufficient training datasets must also be considered,especially because 3D datasets are usually biased and limited.Finally,emerging and popular datasets and evaluation indicators are discussed.The characteristics of the datasets and the relationships of the indicators are explained and highlighted.Thus,this article can be useful and instructive for researchers who are lacking in experience and find this field confusing.In addition,by providing an overview of 3D human pose estimation,this article sorts and refines recent studies on 3D human pose estimation.It describes kernel problems and common useful methods,and discusses the scope for further research.
基金the National Key R&D Program of China(2018YFB1004600)the National Natural Science Foundation of China(61502187,61876211)the National Science Foundation Grant CNS(1951952).
文摘The field of vision-based human hand three-dimensional(3D)shape and pose estimation has attracted significant attention recently owing to its key role in various applications,such as natural human computer interactions.With the availability of large-scale annotated hand datasets and the rapid developments of deep neural networks(DNNs),numerous DNN-based data-driven methods have been proposed for accurate and rapid hand shape and pose estimation.Nonetheless,the existence of complicated hand articulation,depth and scale ambiguities,occlusions,and finger similarity remain challenging.In this study,we present a comprehensive survey of state-of-the-art 3D hand shape and pose estimation approaches using RGB-D cameras.Related RGB-D cameras,hand datasets,and a performance analysis are also discussed to provide a holistic view of recent achievements.We also discuss the research potential of this rapidly growing field.
基金supported by the Major Sports Research Projects of Jiangsu Provincial Sports Bureau in 2022(No.ST221101).
文摘As an essential tool for quantitative analysis of lower limb coordination,optical motion capture systems with marker-based encoding still suffer from inefficiency,high costs,spatial constraints,and the requirement for multiple markers.While 3D pose estimation algorithms combined with ordinary cameras offer an alternative,their accuracy often deteriorates under significant body occlusion.To address the challenge of insufficient 3D pose estimation precision in occluded scenarios—which hinders the quantitative analysis of athletes’lower-limb coordination—this paper proposes a multimodal training framework integrating spatiotemporal dependency networks with text-semantic guidance.Compared to traditional optical motion capture systems,this work achieves low-cost,high-precision motion parameter acquisition through the following innovations:(1)spatiotemporal dependency attention module is designed to establish dynamic spatiotemporal correlation graphs via cross-frame joint semantic matching,effectively resolving the feature fragmentation issue in existing methods.(2)noise-suppressed multi-scale temporal module is proposed,leveraging KL divergence-based information gain analysis for progressive feature filtering in long-range dependencies,reducing errors by 1.91 mm compared to conventional temporal convolutions.(3)text-pose contrastive learning paradigm is introduced for the first time,where BERT-generated action descriptions align semantic-geometric features via the BERT encoder,significantly enhancing robustness under severe occlusion(50%joint invisibility).On the Human3.6M dataset,the proposed method achieves an MPJPE of 56.21 mm under Protocol 1,outperforming the state-of-the-art baseline MHFormer by 3.3%.Extensive ablation studies on Human3.6M demonstrate the individual contributions of the core modules:the spatiotemporal dependency module and noise-suppressed multi-scale temporal module reduce MPJPE by 0.30 and 0.34 mm,respectively,while the multimodal training strategy further decreases MPJPE by 0.6 mm through text-skeleton contrastive learning.Comparative experiments involving 16 athletes show that the sagittal plane coupling angle measurements of hip-ankle joints differ by less than 1.2°from those obtained via traditional optical systems(two one-sided t-tests,p<0.05),validating real-world reliability.This study provides an AI-powered analytical solution for competitive sports training,serving as a viable alternative to specialized equipment.
基金Supported by the National Basic Research Program of China (Grant No.2006CB303103)Key Program of the National Natural Science Foundation of China (Grant No.60833009)
文摘We address the problem of 3D human pose estimation in a single real scene image. Normally, 3D pose estimation from real image needs background subtraction to extract the appropriate features. We do not make such assumption, In this paper, a two-step approach is proposed, first, instead of applying background subtraction to get the segmentation of human, we combine the segmentation with human detection using an ISM-based detector. Then, silhouette feature can be extracted and 3D pose estimation is solved as a regression problem. RVMs and ridge regression method are applied to solve this problem. The results show the robustness and accuracy of our method.
基金wsupported by the Thailand Research Fund and Solimac Automation Co.,Ltd.under the Research and Researchers for Industry Program(RRI)under Grant No.MSD56I0098Office of the Higher Education Commission under the National Research University Project of Thailand
文摘In this paper,we present a robot vision based system for coordinate measurement of feature points on large scale automobile parts.Our system consists of an industrial 6-DOF robot mounted with a CCD camera and a PC.The system controls the robot into the area of feature points.The images of measuring feature points are acquired by the camera mounted on the robot.3D positions of the feature points are obtained from a model based pose estimation that applies to the images.The measured positions of all feature points are then transformed to the reference coordinate of feature points whose positions are obtained from the coordinate measuring machine（CMM）.Finally,the point-to-point distances between the measured feature points and the reference feature points are calculated and reported.The results show that the root mean square error（RMSE） of measure values obtained by our system is less than 0.5 mm.Our system is adequate for automobile assembly and can perform faster than conventional methods.
