Embodied learning for object-centric robotic manipulation is a rapidly developing and challenging area in embodied AI.It is crucial for advancing next-generation intelligent robots and has garnered significant interes...Embodied learning for object-centric robotic manipulation is a rapidly developing and challenging area in embodied AI.It is crucial for advancing next-generation intelligent robots and has garnered significant interest recently.Unlike data-driven machine learning methods,embodied learning focuses on robot learning through physical interaction with the environment and perceptual feedback,making it especially suitable for robotic manipulation.In this paper,we provide a comprehensive survey of the latest advancements in this field and categorize the existing work into three main branches:1)Embodied perceptual learning,which aims to predict object pose and affordance through various data representations;2)Embodied policy learning,which focuses on generating optimal robotic decisions using methods such as reinforcement learning and imitation learning;3)Embodied task-oriented learning,designed to optimize the robot′s performance based on the characteristics of different tasks in object grasping and manipulation.In addition,we offer an overview and discussion of public datasets,evaluation metrics,representative applications,current challenges,and potential future research directions.A project associated with this survey has been established at https://github.com/RayYoh/OCRM_survey.展开更多
Gesture-based interactions in a virtual reality(VR)setting can enhance our experience of traditional practices as part of preserving and communicating heritage.Cultural experiences embodied within VR environments are ...Gesture-based interactions in a virtual reality(VR)setting can enhance our experience of traditional practices as part of preserving and communicating heritage.Cultural experiences embodied within VR environments are suggested to be an effective approach for experiencing intangible cultural heritage.Ceremonies,rituals,and related ancestral enactments are important for preserving cultural heritage.Kāi BǐLǐ,also known as the First Writing Ceremony,is traditionally held for Chinese children before their first year of elementary school.However,gesture-based immersive VR for learning this tradition is new,and have not been developed within the community.This study focused on how users experienced learning cultural practices using gesture-based interactive VR across different age groups and hardware platforms.We first conducted an experiment with 60 participants(30 young adults and 30 children)using the First Writing Ceremony as a case study in which gestural interactions were elicited,designed,implemented,and evaluated.The study showed significant differences in play time and presence between the head-mounted display VR and desktop VR.In addition,children were less likely to experience fatigue than young adults.Following this,we conducted another study after eight months to investigate the VR systems’long-term learning effectiveness.This showed that children outperformed young adults in demonstrating greater knowledge retention.Our results and findings contribute to the design of gesture-based VR for different age groups across different platforms for experiencing,learning,and practicing cultural activities.展开更多
Large language models(LLMs)have been widely adopted in robotic applications in recent years,but their ability in task planning of long-horizon and complex tasks remains a challenge.In this work,we present a gradual le...Large language models(LLMs)have been widely adopted in robotic applications in recent years,but their ability in task planning of long-horizon and complex tasks remains a challenge.In this work,we present a gradual learning method to address this challenge and explore its usability in surgical training tasks that require high levels of reasoning,such as peg transfer and the sliding puzzle task.Experiments were conducted using the da Vinci Research Kit(dVRK),with environment feedback initiating follow-up prompts for the LLM when necessary,as well as in a simulation environment.Results showed that for complex tasks,the gradual learning method outperformed the direct approach in the LLM's task and motion planning,requiring fewer follow-up prompts and leading to higher success rates with faster execution.This suggests that for complex pseudo-surgical tasks,it is more efficient to have the LLM solve simpler versions of the task while incrementally increasing complexity,rather than tackling the full complex task at once.The approach shows promise for enhancing robot-assisted surgery where tasks are complex,long-horizon,and demand high-reasoning abilities.展开更多
基金supported in part by the National Natural Science Foundation of China(No.62106236).
文摘Embodied learning for object-centric robotic manipulation is a rapidly developing and challenging area in embodied AI.It is crucial for advancing next-generation intelligent robots and has garnered significant interest recently.Unlike data-driven machine learning methods,embodied learning focuses on robot learning through physical interaction with the environment and perceptual feedback,making it especially suitable for robotic manipulation.In this paper,we provide a comprehensive survey of the latest advancements in this field and categorize the existing work into three main branches:1)Embodied perceptual learning,which aims to predict object pose and affordance through various data representations;2)Embodied policy learning,which focuses on generating optimal robotic decisions using methods such as reinforcement learning and imitation learning;3)Embodied task-oriented learning,designed to optimize the robot′s performance based on the characteristics of different tasks in object grasping and manipulation.In addition,we offer an overview and discussion of public datasets,evaluation metrics,representative applications,current challenges,and potential future research directions.A project associated with this survey has been established at https://github.com/RayYoh/OCRM_survey.
基金support from the National Natural Science Foundation of China,No.62207022the Minjiang Scholars Program at Quanzhou Normal University.
文摘Gesture-based interactions in a virtual reality(VR)setting can enhance our experience of traditional practices as part of preserving and communicating heritage.Cultural experiences embodied within VR environments are suggested to be an effective approach for experiencing intangible cultural heritage.Ceremonies,rituals,and related ancestral enactments are important for preserving cultural heritage.Kāi BǐLǐ,also known as the First Writing Ceremony,is traditionally held for Chinese children before their first year of elementary school.However,gesture-based immersive VR for learning this tradition is new,and have not been developed within the community.This study focused on how users experienced learning cultural practices using gesture-based interactive VR across different age groups and hardware platforms.We first conducted an experiment with 60 participants(30 young adults and 30 children)using the First Writing Ceremony as a case study in which gestural interactions were elicited,designed,implemented,and evaluated.The study showed significant differences in play time and presence between the head-mounted display VR and desktop VR.In addition,children were less likely to experience fatigue than young adults.Following this,we conducted another study after eight months to investigate the VR systems’long-term learning effectiveness.This showed that children outperformed young adults in demonstrating greater knowledge retention.Our results and findings contribute to the design of gesture-based VR for different age groups across different platforms for experiencing,learning,and practicing cultural activities.
基金Canada Foundation for Innovation(CFI)Natural Sciences and Engineering Research Council(NSERC)of Canada+1 种基金Canadian Institutes of Health Research(CIHR)Alberta Innovates,Canada。
文摘Large language models(LLMs)have been widely adopted in robotic applications in recent years,but their ability in task planning of long-horizon and complex tasks remains a challenge.In this work,we present a gradual learning method to address this challenge and explore its usability in surgical training tasks that require high levels of reasoning,such as peg transfer and the sliding puzzle task.Experiments were conducted using the da Vinci Research Kit(dVRK),with environment feedback initiating follow-up prompts for the LLM when necessary,as well as in a simulation environment.Results showed that for complex tasks,the gradual learning method outperformed the direct approach in the LLM's task and motion planning,requiring fewer follow-up prompts and leading to higher success rates with faster execution.This suggests that for complex pseudo-surgical tasks,it is more efficient to have the LLM solve simpler versions of the task while incrementally increasing complexity,rather than tackling the full complex task at once.The approach shows promise for enhancing robot-assisted surgery where tasks are complex,long-horizon,and demand high-reasoning abilities.
