Starting from the characteristics of fruit picking,the characteristics of fruit picking robot manipulators and the research state at home and abroad are reviewed.The analysis summarizes the difficulties in fruit picki...Starting from the characteristics of fruit picking,the characteristics of fruit picking robot manipulators and the research state at home and abroad are reviewed.The analysis summarizes the difficulties in fruit picking robotic arm research.Aiming at the configuration of the manipulator,the structure and characteristics of the manipulator with redundant degrees of freedom are introduced,and the feasibility of the redundant mechanism is demonstrated through the current research state of the manipulator.展开更多
The increase in precision agriculture has promoted the development of picking robot technology,and the visual recognition system at its core is crucial for improving the level of agricultural automation.This paper rev...The increase in precision agriculture has promoted the development of picking robot technology,and the visual recognition system at its core is crucial for improving the level of agricultural automation.This paper reviews the progress of visual recognition tech-nology for picking robots,including image capture technology,target detection algorithms,spatial positioning strategies and scene un-derstanding.This article begins with a description of the basic structure and function of the vision system of the picking robot and em-phasizes the importance of achieving high-efficiency and high-accuracy recognition in the natural agricultural environment.Sub-sequently,various image processing techniques and vision algorithms,including color image analysis,three-dimensional depth percep-tion,and automatic object recognition technology that integrates machine learning and deep learning algorithms,were analysed.At the same time,the paper also highlights the challenges of existing technologies in dynamic lighting,occlusion problems,fruit maturity di-versity,and real-time processing capabilities.This paper further discusses multisensor information fusion technology and discusses methods for combining visual recognition with a robot control system to improve the accuracy and working rate of picking.At the same time,this paper also introduces innovative research,such as the application of convolutional neural networks(CNNs)for accurate fruit detection and the development of event-based vision systems to improve the response speed of the system.At the end of this paper,the future development of visual recognition technology for picking robots is predicted,and new research trends are proposed,including the refinement of algorithms,hardware innovation,and the adaptability of technology to different agricultural conditions.The purpose of this paper is to provide a comprehensive analysis of visual recognition technology for researchers and practitioners in the field of agricul-tural robotics,including current achievements,existing challenges and future development prospects.展开更多
With the development of tree-climbing robots and robotic end-effectors,it is possible to develop automated coconutpicking robots with the help of machine vision technology.Coconuts grow in clusters in the canopy and a...With the development of tree-climbing robots and robotic end-effectors,it is possible to develop automated coconutpicking robots with the help of machine vision technology.Coconuts grow in clusters in the canopy and are easily occluded by leaves.Therefore,the detection of multi-class coconut clusters according to the occlusion condition is necessary for robots to develop picking strategies.The coconut detection model,named YOLO-Coco,was developed based on the YOLOv7-tiny network.It detected coconuts in different conditions such as not-occluded,leaves-occluded,and trunk-occluded fruit.The developed model used Efficient Channel Attention(ECA)to enhance the feature weights extracted by the backbone network.Re-parameterization Convolution(RepConv)made the model convolution layers deeper and provided more semantic information for the detection head.Finally,the Bi-directional Feature Pyramid Network(BiFPN)was used to optimize the head network structure of YOLO-Coco to achieve the balanced fusion of multi-scale features.The results showed that the mean average precision(mAP)of YOLO-Coco for detecting multi-class coconut clusters was 93.6%,and the average precision(AP)of not-occluded,leaves-occluded,and trunk-occluded fruit were 90.5%,93.8%,and 96.4%,respectively.The detection accuracy of YOLO-Coco for yellow coconuts was 5.1%higher than that for green coconuts.Compared with seven mainstream deep learning networks,YOLO-Coco achieved the highest detection accuracy in detecting multi-class coconut clusters,while maintaining advantages in detection speed and model size.The developed model can accurately detect coconuts in complex canopy environments,providing technical support for the visual system of coconut-picking robots.展开更多
In the unstructured litchi orchard,precise identification and localization of litchi fruits and picking points are crucial for litchi-picking robots.Most studies adopt multi-step methods to detect fruit and locate pic...In the unstructured litchi orchard,precise identification and localization of litchi fruits and picking points are crucial for litchi-picking robots.Most studies adopt multi-step methods to detect fruit and locate picking points,which are slow and struggle to cope with complex environments.This study proposes a YOLOv8-iGR model based on YOLOv8n-pose improvement,integrating end-to-end network for both object detection and key point detection.Specifically,this study considers the influence of auxiliary points on picking point and designs four litchi key point strategies.Secondly,the architecture named iSaE is proposed,which combines the capabilities of CNN and attention mechanism.Subsequently,C2f is replaced by Generalized Efficient Layer Aggregation Network(GELAN)to reduce model redundancy and improve detection accuracy.Finally,based on RFAConv,RFAPoseHead is designed to address the issue of parameter sharing in large convolutional kernels,thereby more effectively extracting feature information.Experimental results demonstrate that YOLOv8-iGR achieves an AP of 95.7%in litchi fruit detection,and the Euclidean distance error of picking points is less than 8 pixels across different scenes,meeting the requirements of litchi picking.Additionally,the GFLOPs of the model are reduced by 10.71%.The accuracy of the model’s localization for picking points was tested through field picking experiments.In conclusion,YOLOv8-iGR exhibits outstanding detection performance along with lower model complexity,making it more feasible for implementation on robots.This will provide technical support for the vision system of the litchi-picking robot.展开更多
The fruit and vegetable picking has posed a great challenge on the production and markets during the harvest season.Manual picking cannot fully meet the rapid requirements of each market,mainly due to the high labor-i...The fruit and vegetable picking has posed a great challenge on the production and markets during the harvest season.Manual picking cannot fully meet the rapid requirements of each market,mainly due to the high labor-intensive and time-consuming tasks,even the aging and shortage of agricultural labor force in recent years.Alternatively,smart robotics can be an efficient solution to increase the planting areas for the markets in combination with changes in cultivation,preservation,and processing technology.However,some improvements still need to be performed on these picking robots.To document the progress in and current status of this field,this study performed a bibliometric analysis.This analysis evaluated the current performance characteristics of various fruit and vegetable picking robots for better prospects in the future.Five perspectives were proposed covering the robotic arms,end effectors,vision systems,picking environments,and picking performance for the large-scale mechanized production of fruits and vegetables in modern agriculture.The current problems of fruit and vegetable picking robots were summarized.Finally,the outlook of the fruit and vegetable picking robots prospected from four aspects:structured environment for fruit planting,the algorithm of recognition and positioning,picking efficiency,and cost-saving picking robots.This study comprehensively assesses the current research status,thus helping researchers steer their projects or locate potential collaborators.展开更多
The important indicators to measure the goodness of rigid fruit and vegetable picking robot have two aspects,the first is the reasonable structural design of the end-effector,and the second is having a high precision ...The important indicators to measure the goodness of rigid fruit and vegetable picking robot have two aspects,the first is the reasonable structural design of the end-effector,and the second is having a high precision positioning recognition method.Many researchers have done a lot of work in these two aspects,and a variety of end-effector structures and advanced position recognition methods are constantly appearing in people’s view.The working principle,structural characteristics,advantages and disadvantages of each end-effector are summarized to help us design better fruit and vegetable picking robot.The authors start from the rigid fruit and vegetable picking robot grasping methods,separation methods,and position recognition methods,firstly introduce three different grasping methods and the characteristics of the two separation methods,then introduce the under-driven picking robot developed on the basis of the traditional rigid picking robot,then explain the single special,multi-feature,and deep learning location position recognition methods currently used,and finally make a summary and outlook on the rigid fruit and vegetable picking robot from the structural development and position recognition methods.展开更多
Accurate detection of citrus in the natural orchard is crucial for citrus-picking robots.However,it has become a challenging task due to the influence of illumination,severe shading of branches and leaves,as well as o...Accurate detection of citrus in the natural orchard is crucial for citrus-picking robots.However,it has become a challenging task due to the influence of illumination,severe shading of branches and leaves,as well as overlapping of citrus.To this end,a Dense-TRU-YOLO model was proposed,which integrated the Denseblock with the Transformer and used UNet++network as the neck structure.First of all,the Denseblock structure was incorporated into YOLOv5,which added shallow semantic information to the deep part of the network and improved the flow of information and gradients.Secondly,the deepest Cross Stage Partial Connections(CSP)bottleneck with the 3 convolutions module of the backbone was replaced by the CSP Transformer with 3 convolutions module,which increased the semantic resolution and improved the detection accuracy of occlusion.Finally,the neck of the original network was replaced by the combined structure of UNet++feature pyramid networks(UNet++-FPN),which not only added cross-weighted links between nodes with the same size but also enhanced the feature fusion ability between nodes with different sizes,making the regression of the network to the target boundary more accurate.Ablation experiments and comparison experiments showed that the Dense-TRU-YOLO can effectively improve the detection accuracy of citrus under severe occlusion and overlap.The overall accuracy,recall,mAP@0.5,and F1 were 90.8%,87.6%,90.5%,and 87.9%,respectively.The precision of Dense-TRU-YOLO was the highest,which was 3.9%,6.45%,1.9%,7.4%,3.3%,4.9%,and 9.9%higher than that of the YOLOv5-s,YOLOv3,YOLOv5-n,YOLOv4-tiny,YOLOv4,YOLOX,and YOLOF,respectively.In addition,the reasoning speed was 9.2 ms,1.7 ms,10.5 ms,and 2.3 ms faster than that of YOLOv3,YOLOv5-n,YOLOv4,and YOLOX.Dense TRU-YOLO is designed to enhance the accuracy of fruit recognition in natural settings and boost the detection capabilities for small targets at extended ranges.展开更多
Apple fruits on trees tend to swing because of wind or other natural causes,therefore reducing the accuracy of apple picking by robots.To increase the accuracy and to speed up the apple tracking and identifying proces...Apple fruits on trees tend to swing because of wind or other natural causes,therefore reducing the accuracy of apple picking by robots.To increase the accuracy and to speed up the apple tracking and identifying process,tracking and recognition method combined with an affine transformation was proposed.The method can be divided into three steps.First,the initial image was segmented by Otsu’s thresholding method based on the two times Red minus Green minus Blue(2R-G-B)color feature;after improving the binary image,the apples were recognized with a local parameter adaptive Hough circle transformation method,thus improving the accuracy of recognition and avoiding the long,time-consuming process and excessive fitted circles in traditional Hough circle transformation.The process and results were verified experimentally.Second,the Shi-Tomasi corners detected and extracted from the first frame image were tracked,and the corners with large positive and negative optical flow errors were removed.The affine transformation matrix between the two frames was calculated based on the Random Sampling Consistency algorithm(RANSAC)to correct the scale of the template image and predict the apple positions.Third,the best positions of the target apples within 1.2 times of the prediction area were searched with a de-mean normalized cross-correlation template matching algorithm.The test results showed that the running time of each frame was 25 ms and 130 ms and the tracking error was more than 8%and 20%in the absence of template correction and apple position prediction,respectively.In comparison,the running time of our algorithm was 25 ms,and the tracking error was less than 4%.Therefore,test results indicate that speed and efficiency can be greatly improved by using our method,and this strategy can also provide a reference for tracking and recognizing other oscillatory fruits.展开更多
基金National Natural Science Foundation of China(51305402)。
文摘Starting from the characteristics of fruit picking,the characteristics of fruit picking robot manipulators and the research state at home and abroad are reviewed.The analysis summarizes the difficulties in fruit picking robotic arm research.Aiming at the configuration of the manipulator,the structure and characteristics of the manipulator with redundant degrees of freedom are introduced,and the feasibility of the redundant mechanism is demonstrated through the current research state of the manipulator.
基金supported by the National Natural Science Foundation of China(Nos.62027810 and 61733004)the National Key Research and Development Program of China(No.2020YFB1712600)+1 种基金the Hunan Science and Technology Program of Hunan Province,China(Nos.2017XK2102 and 2018GK2022)supported by the Changsha Science and Technology Innovation Fund,China(No.kq2402079).
文摘The increase in precision agriculture has promoted the development of picking robot technology,and the visual recognition system at its core is crucial for improving the level of agricultural automation.This paper reviews the progress of visual recognition tech-nology for picking robots,including image capture technology,target detection algorithms,spatial positioning strategies and scene un-derstanding.This article begins with a description of the basic structure and function of the vision system of the picking robot and em-phasizes the importance of achieving high-efficiency and high-accuracy recognition in the natural agricultural environment.Sub-sequently,various image processing techniques and vision algorithms,including color image analysis,three-dimensional depth percep-tion,and automatic object recognition technology that integrates machine learning and deep learning algorithms,were analysed.At the same time,the paper also highlights the challenges of existing technologies in dynamic lighting,occlusion problems,fruit maturity di-versity,and real-time processing capabilities.This paper further discusses multisensor information fusion technology and discusses methods for combining visual recognition with a robot control system to improve the accuracy and working rate of picking.At the same time,this paper also introduces innovative research,such as the application of convolutional neural networks(CNNs)for accurate fruit detection and the development of event-based vision systems to improve the response speed of the system.At the end of this paper,the future development of visual recognition technology for picking robots is predicted,and new research trends are proposed,including the refinement of algorithms,hardware innovation,and the adaptability of technology to different agricultural conditions.The purpose of this paper is to provide a comprehensive analysis of visual recognition technology for researchers and practitioners in the field of agricul-tural robotics,including current achievements,existing challenges and future development prospects.
基金financially supported by the Key R&D Projects in Hainan Province(Grant No.ZDYF2022XDNY231)the National Natural Science Foundation of China(Grant No.52265040)the Innovative Research Projects for Graduate Students in Hainan Province(Grant No.Qhyb2023-100).
文摘With the development of tree-climbing robots and robotic end-effectors,it is possible to develop automated coconutpicking robots with the help of machine vision technology.Coconuts grow in clusters in the canopy and are easily occluded by leaves.Therefore,the detection of multi-class coconut clusters according to the occlusion condition is necessary for robots to develop picking strategies.The coconut detection model,named YOLO-Coco,was developed based on the YOLOv7-tiny network.It detected coconuts in different conditions such as not-occluded,leaves-occluded,and trunk-occluded fruit.The developed model used Efficient Channel Attention(ECA)to enhance the feature weights extracted by the backbone network.Re-parameterization Convolution(RepConv)made the model convolution layers deeper and provided more semantic information for the detection head.Finally,the Bi-directional Feature Pyramid Network(BiFPN)was used to optimize the head network structure of YOLO-Coco to achieve the balanced fusion of multi-scale features.The results showed that the mean average precision(mAP)of YOLO-Coco for detecting multi-class coconut clusters was 93.6%,and the average precision(AP)of not-occluded,leaves-occluded,and trunk-occluded fruit were 90.5%,93.8%,and 96.4%,respectively.The detection accuracy of YOLO-Coco for yellow coconuts was 5.1%higher than that for green coconuts.Compared with seven mainstream deep learning networks,YOLO-Coco achieved the highest detection accuracy in detecting multi-class coconut clusters,while maintaining advantages in detection speed and model size.The developed model can accurately detect coconuts in complex canopy environments,providing technical support for the visual system of coconut-picking robots.
基金supported by Natural Science Foundation of Guangdong Province(Grant No.2025A1515011771)Guangzhou Science and Technology Plan Project(Grant No.2024E04J1242,2023B01J0046)+2 种基金Guangdong Provincial Department of Science and Technology(Grant No.2023A0505050130)Key Projects of Guangzhou Science and Technology Program(Grant No.2024B03J1357)Natural Science Foundation of China(Grant No.61863011,32071912).
文摘In the unstructured litchi orchard,precise identification and localization of litchi fruits and picking points are crucial for litchi-picking robots.Most studies adopt multi-step methods to detect fruit and locate picking points,which are slow and struggle to cope with complex environments.This study proposes a YOLOv8-iGR model based on YOLOv8n-pose improvement,integrating end-to-end network for both object detection and key point detection.Specifically,this study considers the influence of auxiliary points on picking point and designs four litchi key point strategies.Secondly,the architecture named iSaE is proposed,which combines the capabilities of CNN and attention mechanism.Subsequently,C2f is replaced by Generalized Efficient Layer Aggregation Network(GELAN)to reduce model redundancy and improve detection accuracy.Finally,based on RFAConv,RFAPoseHead is designed to address the issue of parameter sharing in large convolutional kernels,thereby more effectively extracting feature information.Experimental results demonstrate that YOLOv8-iGR achieves an AP of 95.7%in litchi fruit detection,and the Euclidean distance error of picking points is less than 8 pixels across different scenes,meeting the requirements of litchi picking.Additionally,the GFLOPs of the model are reduced by 10.71%.The accuracy of the model’s localization for picking points was tested through field picking experiments.In conclusion,YOLOv8-iGR exhibits outstanding detection performance along with lower model complexity,making it more feasible for implementation on robots.This will provide technical support for the vision system of the litchi-picking robot.
基金the Basic Public Welfare Research Project of Zhejiang Province(No.LGN20E050007,No.LGG19E050023)Xinjiang Boshiran Intelligent Agricultural Machinery Co.,Ltd.
文摘The fruit and vegetable picking has posed a great challenge on the production and markets during the harvest season.Manual picking cannot fully meet the rapid requirements of each market,mainly due to the high labor-intensive and time-consuming tasks,even the aging and shortage of agricultural labor force in recent years.Alternatively,smart robotics can be an efficient solution to increase the planting areas for the markets in combination with changes in cultivation,preservation,and processing technology.However,some improvements still need to be performed on these picking robots.To document the progress in and current status of this field,this study performed a bibliometric analysis.This analysis evaluated the current performance characteristics of various fruit and vegetable picking robots for better prospects in the future.Five perspectives were proposed covering the robotic arms,end effectors,vision systems,picking environments,and picking performance for the large-scale mechanized production of fruits and vegetables in modern agriculture.The current problems of fruit and vegetable picking robots were summarized.Finally,the outlook of the fruit and vegetable picking robots prospected from four aspects:structured environment for fruit planting,the algorithm of recognition and positioning,picking efficiency,and cost-saving picking robots.This study comprehensively assesses the current research status,thus helping researchers steer their projects or locate potential collaborators.
基金supported by the National Natural Science Foundation of China(Grant No.51775002)the 14th Five-Year Plan of Beijing Education Science(Grant No.CDDB21173).
文摘The important indicators to measure the goodness of rigid fruit and vegetable picking robot have two aspects,the first is the reasonable structural design of the end-effector,and the second is having a high precision positioning recognition method.Many researchers have done a lot of work in these two aspects,and a variety of end-effector structures and advanced position recognition methods are constantly appearing in people’s view.The working principle,structural characteristics,advantages and disadvantages of each end-effector are summarized to help us design better fruit and vegetable picking robot.The authors start from the rigid fruit and vegetable picking robot grasping methods,separation methods,and position recognition methods,firstly introduce three different grasping methods and the characteristics of the two separation methods,then introduce the under-driven picking robot developed on the basis of the traditional rigid picking robot,then explain the single special,multi-feature,and deep learning location position recognition methods currently used,and finally make a summary and outlook on the rigid fruit and vegetable picking robot from the structural development and position recognition methods.
基金supported by the special research project of the Innovation and Development Center for Ideological and Political Work in Colleges and Universities(Wuhan Donghu University)under the Ministry of Education in 2024(Practice Research on the Whole-process Fine Cultivation of New Engineering Talents with New Qualities in the Context of New Productive Forces,Grant No.WHDHSZZX2024085)the Humanities and Social Sciences Research project of the Chongqing Education Commission in 2024(Theory and Practice Research on Digital Portraits Enabling Comprehensive Quality Evaluation of College Students,Grant No.24SKGH100)the general project of the“14th Five-Year Plan”for Education Science in Chongqing in 2024(Research and Practice on the Construction of an Intelligent Recommendation Employment System for Person-Job Matching Enabled by Digital Portraits,Grant No.K24YG2060081).
文摘Accurate detection of citrus in the natural orchard is crucial for citrus-picking robots.However,it has become a challenging task due to the influence of illumination,severe shading of branches and leaves,as well as overlapping of citrus.To this end,a Dense-TRU-YOLO model was proposed,which integrated the Denseblock with the Transformer and used UNet++network as the neck structure.First of all,the Denseblock structure was incorporated into YOLOv5,which added shallow semantic information to the deep part of the network and improved the flow of information and gradients.Secondly,the deepest Cross Stage Partial Connections(CSP)bottleneck with the 3 convolutions module of the backbone was replaced by the CSP Transformer with 3 convolutions module,which increased the semantic resolution and improved the detection accuracy of occlusion.Finally,the neck of the original network was replaced by the combined structure of UNet++feature pyramid networks(UNet++-FPN),which not only added cross-weighted links between nodes with the same size but also enhanced the feature fusion ability between nodes with different sizes,making the regression of the network to the target boundary more accurate.Ablation experiments and comparison experiments showed that the Dense-TRU-YOLO can effectively improve the detection accuracy of citrus under severe occlusion and overlap.The overall accuracy,recall,mAP@0.5,and F1 were 90.8%,87.6%,90.5%,and 87.9%,respectively.The precision of Dense-TRU-YOLO was the highest,which was 3.9%,6.45%,1.9%,7.4%,3.3%,4.9%,and 9.9%higher than that of the YOLOv5-s,YOLOv3,YOLOv5-n,YOLOv4-tiny,YOLOv4,YOLOX,and YOLOF,respectively.In addition,the reasoning speed was 9.2 ms,1.7 ms,10.5 ms,and 2.3 ms faster than that of YOLOv3,YOLOv5-n,YOLOv4,and YOLOX.Dense TRU-YOLO is designed to enhance the accuracy of fruit recognition in natural settings and boost the detection capabilities for small targets at extended ranges.
基金This work was financially supported by Basic Public Welfare Research Project of Zhejiang Province(Grant No.LGN20E050007).
文摘Apple fruits on trees tend to swing because of wind or other natural causes,therefore reducing the accuracy of apple picking by robots.To increase the accuracy and to speed up the apple tracking and identifying process,tracking and recognition method combined with an affine transformation was proposed.The method can be divided into three steps.First,the initial image was segmented by Otsu’s thresholding method based on the two times Red minus Green minus Blue(2R-G-B)color feature;after improving the binary image,the apples were recognized with a local parameter adaptive Hough circle transformation method,thus improving the accuracy of recognition and avoiding the long,time-consuming process and excessive fitted circles in traditional Hough circle transformation.The process and results were verified experimentally.Second,the Shi-Tomasi corners detected and extracted from the first frame image were tracked,and the corners with large positive and negative optical flow errors were removed.The affine transformation matrix between the two frames was calculated based on the Random Sampling Consistency algorithm(RANSAC)to correct the scale of the template image and predict the apple positions.Third,the best positions of the target apples within 1.2 times of the prediction area were searched with a de-mean normalized cross-correlation template matching algorithm.The test results showed that the running time of each frame was 25 ms and 130 ms and the tracking error was more than 8%and 20%in the absence of template correction and apple position prediction,respectively.In comparison,the running time of our algorithm was 25 ms,and the tracking error was less than 4%.Therefore,test results indicate that speed and efficiency can be greatly improved by using our method,and this strategy can also provide a reference for tracking and recognizing other oscillatory fruits.
